In the previous article I showed how to build a webapp in the Azure pipeline. In this article I will explain to you how you run tests in your pipeline. I will show you how to do this for both Angular and Node.js.
Adding tests frontend – Angular
Adding tests to the frontend is a bit harder than for the backend, so will we start out with this.
This article helped me out a lot by adding tests to Angular. I will cover what Oliver said in that article and add some more about how to improve performance.
Adding code coverage
First we will want to add the Cobertura reporter to karma.conf.js by doing the following
module.exports = function (config) {
...
coverageIstanbulReporter: {
dir: require('path').join(__dirname, '../coverage'),
reports: ['html', 'lcovonly', 'text-summary', 'cobertura'],
fixWebpackSourcePaths: true
},
...
};
Next you will want to install the following three packages as dev dependencies, so the tests can run headless.
- Puppeteer
- Karma-jasmine-html-reporter
- Karma-spec-reporter
Now you will want to edit the karma.conf.js again so it uses puppeteer. Make it so it looks something like this. The highlighted line are the lines that have been changed.
module.exports = function (config) {
const puppeteer = require('puppeteer');
process.env.CHROME_BIN = puppeteer.executablePath();
config.set({
basePath: '',
frameworks: ['jasmine', '@angular-devkit/build-angular'],
plugins: [
require('karma-jasmine'),
require('karma-chrome-launcher'),
require('karma-jasmine-html-reporter'),
require('karma-coverage-istanbul-reporter'),
require('@angular-devkit/build-angular/plugins/karma')
],
client: {
clearContext: false // leave Jasmine Spec Runner output visible in browser
},
coverageIstanbulReporter: {
dir: require('path').join(__dirname, '../coverage'),
reports: ['html', 'lcovonly', 'text-summary', 'cobertura'],
fixWebpackSourcePaths: true
},
reporters: ['progress', 'kjhtml'],
port: 9876,
colors: true,
logLevel: config.LOG_INFO,
autoWatch: true,
browsers: ['ChromeHeadless'],
singleRun: false
});
};
The code coverage problem
For the code coverage, we want the backend as well as the frontend reported. The problem is, that azure overrides the code coverage files when a new coverage file is uploaded. See here. Now there’s a solution for that (until Microsoft fixes this issue). We can use an custom reporter made by a guy named Daniel palm. This reporter can merge files, in contrary to Azure’s reporter.
You can install his report generator from the marketplace from here. Now you can use it in your pipeline.
Run the tests and create the code coverage file like so:
- task: DeleteFiles@1
displayName: 'Delete JUnit files'
inputs:
SourceFolder: '$(System.DefaultWorkingDirectory)/frontend/junit'
Contents: 'TESTS*.xml'
- task: Npm@1
displayName: 'Test Angular'
inputs:
command: custom
customCommand: run test -- --watch=false --code-coverage
workingDir: '$(System.DefaultWorkingDirectory)/frontend'
- task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator@4
displayName: ReportGenerator
inputs:
reports: '$(Build.SourcesDirectory)/frontend/coverage/cobertura-coverage.xml'
targetdir: '$(Build.SourcesDirectory)/frontend/coverage'
reporttypes: 'Cobertura'
assemblyfilters: '-xunit*'
- task: PublishPipelineArtifact@1
inputs:
targetPath: '$(Build.SourcesDirectory)/frontend/coverage'
artifact: 'frontendCoverage'
We’re first running a deleteFiles task, to make sure the leftover files from the previous build don’t mess up the new results.
Next we’re publishing the coverage as an artifact (yes this is just one file), so we can use it later on when we’re publishing the codeCoverage.
Adding the test summary
For the tests we will just use the ng test command. We’re going to use a junit reporter. Install it by running
npm install karma-junit-reporter --save-dev
Next, add the junit reporter to the karma config by adding the following to the karma.conf.js:
require('karma-jasmine'),
require('karma-chrome-launcher'),
require('karma-jasmine-html-reporter'),
require('karma-coverage-istanbul-reporter'),
require('@angular-devkit/build-angular/plugins/karma'),
require('karma-junit-reporter')
reporters: ['progress', 'kjhtml', 'junit'],
junitReporter: {
outputDir: './junit'
},
Last, add the tests results to the pipeline by adding this to the azure-pipelines.yml:
- task: PublishTestResults@2
displayName: 'Publish Angular test results'
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '$(System.DefaultWorkingDirectory)/frontend/junit/TESTS-*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/frontend/junit'
testRunTitle: 'Frontend mocha tests'
Improving performance
In my experience, the performance of the tests is very, very slow. For me it sometimes takes up to two whole minutes to complete the tests. Sadly, this is mainly caused by the angular compiler itself. There are some ways to reduce the build time a bit, but for me it didn’t help much. The test time was still reduced by 20 seconds though. So you might consider it.
This article explains very well how why the tests are so slow in detail. It also explains how to solve this. This article also shows some ways to speed the process up.
Basically the main thing is that for every test, Angular creates an TestBed. You can also instantiate this TestBed just one time and run it for every test. In the provided article zshuffield used Ng-Bullet. You can use Ng-bullet by installing it and changing the test code to look like the following:
The disadvantage of creating a TestBed just one time, is that code can mess up at runtime. I would only recommend trying to increase the test time if you have a lot of tests. Think of thousands of tests.
Adding tests backend – Node.js
Adding tests to the backend is considerably easier than for the frontend. To accomplish this I used the answer from crimbo from this stackoverflow post.
If you’ve not installed jest yet, do so now and add some tests.
Then install the jest-junit package in the backend folder
npm add --save-dev jest-junit
Adding code coverage
We will make a script in package.json that runs the test coverage. Add the following under scripts:
"test-azure-coverage": "jest --coverage --coverageReporters=cobertura --forceExit"
Now add the reporters to the jest config. Check the documentation here.
"jest": {
"reporters": [
"default",
"jest-junit"
],
Next you add the script, run the reporter (the palm reporter), add it to the pipeline and publish it. You can do this by adding the following to the azure-pipelines.yml:
- task: Npm@1
displayName: run test backend coverage
inputs:
command: 'custom'
workingDir: 'backend'
customCommand: 'run test-azure-coverage'
- task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator@4
displayName: ReportGenerator
inputs:
reports: '$(Build.SourcesDirectory)/backend/coverage/cobertura-coverage.xml'
targetdir: '$(Build.SourcesDirectory)/backend/coverage'
reporttypes: 'Cobertura'
assemblyfilters: '-xunit*'
- task: PublishPipelineArtifact@1
inputs:
targetPath: '$(Build.SourcesDirectory)/backend/coverage'
artifact: 'backendCoverage'
Adding the test summary
To add the test summary we again add a script to the package.json
"test-azure-report": "jest --ci --reporters=default --reporters=jest-junit --forceExit",
This should run the summary. The ci flag is for the continuous integration. I added the forceExit, so it exits when it’s done. See here for the documentation
Next add the test command to the pipeline and publish the result by adding the following to the azure-pipelines.yml:
- task: Npm@1
displayName: run test backend report
inputs:
command: 'custom'
workingDir: 'backend'
customCommand: 'run test-azure-report'
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'junit.xml'
searchFolder: '$(Build.SourcesDirectory)/backend'
testRunTitle: 'Run backend jest tests'
Publishing the code coverage
We’ve published both the frontend and the backend as an artifact. Now we’re going to use these artifacts to publish the coverage altogether.
We do so by adding the following steps to a new job in the azure-pipelines.yml:
- job: codeCoverage
displayName: publish code coverage
dependsOn:
- Frontend
- Backend
steps:
- checkout: none
- task: DownloadPipelineArtifact@2
inputs:
artifact: backend
path: $(Pipeline.Workspace)/backend/backend
itemPattern: '!node_modules/**'
- task: DownloadPipelineArtifact@2
inputs:
artifact: frontend
path: $(Pipeline.Workspace)/frontend/frontend
itemPattern: '!node_modules/**'
- download: current
artifact: backendCoverage
- download: current
artifact: frontendCoverage
- task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator@4
displayName: ReportGenerator
inputs:
reports: '$(Pipeline.Workspace)/backendCoverage/Cobertura.xml;$(Pipeline.Workspace)/frontendCoverage/Cobertura.xml'
targetdir: '$(Pipeline.Workspace)/coverage'
reporttypes: 'HtmlInline_AzurePipelines;Cobertura'
- task: PublishCodeCoverageResults@1
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: '$(Pipeline.Workspace)/coverage/*.xml'
reportDirectory: '$(Pipeline.Workspace)/coverage'
This job downloads the two source code artifacts (this is needed for reference in the code coverage report later on) and downloads the two generated report files.
Notice how I don’t download the node_modules folder, because this is not necessary and is a very large folder.
We use the report generator job from Palm again to generate the final report in html. We publish this report with the PublishCodeCoverageResults task. Your result after running the pipeline should look something like this:
Notice the (2x) at the parser, this indicates that it has combined two reports into 1 report.
And we’re done! This should add backend and frontend tests and code coverage to your Azure pipeline. In the next article I will explain how you can combine on-premise VM’s with the multiple stage environment, so you can have your environments set up so deployments can only be done by manual approval.
Your code in the end should now look like this. I highlighted the lines that have been added compared to the previous file. Don’t forgot to make the changes to the other files as well!
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: Build
jobs:
- job: Frontend
steps:
- task: Npm@1
inputs:
command: 'ci'
workingDir: '$(System.DefaultWorkingDirectory)/frontend'
- task: Bash@3
inputs:
targetType: 'inline'
script: 'npm run ng build'
workingDirectory: '$(System.DefaultWorkingDirectory)/frontend'
- task: DeleteFiles@1
displayName: 'Delete JUnit files'
inputs:
SourceFolder: '$(System.DefaultWorkingDirectory)/frontend/junit'
Contents: 'TESTS*.xml'
- task: Npm@1
displayName: 'Test Angular'
inputs:
command: custom
customCommand: run test -- --watch=false --code-coverage
workingDir: '$(System.DefaultWorkingDirectory)/frontend'
- task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator@4
displayName: ReportGenerator
inputs:
reports: '$(Build.SourcesDirectory)/frontend/coverage/cobertura-coverage.xml'
targetdir: '$(Build.SourcesDirectory)/frontend/coverage'
reporttypes: 'Cobertura'
assemblyfilters: '-xunit*'
- task: PublishTestResults@2
displayName: 'Publish Angular test results'
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '$(System.DefaultWorkingDirectory)/frontend/junit/TESTS-*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/frontend/junit'
testRunTitle: 'Frontend mocha tests'
- task: PublishPipelineArtifact@1
inputs:
targetPath: '$(Build.SourcesDirectory)/frontend/coverage'
artifact: 'frontendCoverage'
- task: PublishPipelineArtifact@1
inputs:
targetPath: '$(Pipeline.Workspace)/frontend'
artifact: 'frontend'
publishLocation: 'pipeline'
- job: Backend
steps:
- task: Npm@1
inputs:
command: 'ci'
workingDir: '$(System.DefaultWorkingDirectory)/backend'
- task: Bash@3
inputs:
targetType: 'inline'
script: 'npm run build'
workingDirectory: '$(System.DefaultWorkingDirectory)/backend'
- task: Npm@1
displayName: run test backend report
inputs:
command: 'custom'
workingDir: 'backend'
customCommand: 'run test-azure-report'
- task: Npm@1
displayName: run test backend coverage
inputs:
command: 'custom'
workingDir: 'backend'
customCommand: 'run test-azure-coverage'
- task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator@4
displayName: ReportGenerator
inputs:
reports: '$(Build.SourcesDirectory)/backend/coverage/cobertura-coverage.xml'
targetdir: '$(Build.SourcesDirectory)/backend/coverage'
reporttypes: 'Cobertura'
assemblyfilters: '-xunit*'
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'junit.xml'
searchFolder: '$(Build.SourcesDirectory)/backend'
testRunTitle: 'Run backend jest tests'
- task: PublishPipelineArtifact@1
inputs:
targetPath: '$(Build.SourcesDirectory)/backend/coverage'
artifact: 'backendCoverage'
- task: PublishPipelineArtifact@1
inputs:
targetPath: '$(Pipeline.Workspace)/backend'
artifact: 'backend'
publishLocation: 'pipeline'
- job: codeCoverage
displayName: publish code coverage
dependsOn:
- Frontend
- Backend
steps:
- checkout: none
- task: DownloadPipelineArtifact@2
inputs:
artifact: backend
path: $(Pipeline.Workspace)/backend/backend
itemPattern: '!node_modules/**'
- task: DownloadPipelineArtifact@2
inputs:
artifact: frontend
path: $(Pipeline.Workspace)/frontend/frontend
itemPattern: '!node_modules/**'
- download: current
artifact: backendCoverage
- download: current
artifact: frontendCoverage
- task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator@4
displayName: ReportGenerator
inputs:
reports: '$(Pipeline.Workspace)/backendCoverage/Cobertura.xml;$(Pipeline.Workspace)/frontendCoverage/Cobertura.xml'
targetdir: '$(Pipeline.Workspace)/coverage'
reporttypes: 'HtmlInline_AzurePipelines;Cobertura'
- task: PublishCodeCoverageResults@1
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: '$(Pipeline.Workspace)/coverage/*.xml'
reportDirectory: '$(Pipeline.Workspace)/coverage'
