Using parallel jobs in Azure Pipelines

Today we are going to investigate parallel jobs in Azure DevOps, and learn how we can use them to speed up our pipelines. We have known about parallel jobs for quite a while, but we haven’t used them, theorizing that they wouldn’t make much of a difference – boy, were we wrong!

What you need to use parallel jobs

A few requirements first, especially if you rely on the Microsoft-hosted builds, you may need to purchase parallel jobs. You can view your current parallel jobs total in the project settings. Here we see that in the free tier, only one job is provided in the Microsoft-hosted agents and Self-hosted is limited by license. You can always purchase more jobs, for about $40US a month each. Our project is a public project, which provides extra benefits – we have 10 free Microsoft-hosted parallel jobs!

In our current multi-stage YAML pipeline, each stage has a single job. This job can only run one task at a time. However, some of our tasks could be run in parallel to reduce duration. Note that jobs don’t solve everything, there is overhead to creating each additional job, (roughly 10-20 seconds). You may need to experiment, as sometimes adding too many jobs will slow our pipelines down.

Speeding up our build

Let’s look out our build first. Currently this is taking about 8 minutes in just one job.

If we dive into the details of what our build is doing, we can see that the bulk of the build duration, 5 minutes of the 8, is used in five tasks, restoring .NET, building .NET, running .NET automated tests, SolarCloud analysis, and security scanning. The diagram below summarizes all of the tasks and their timing, all contained within the single job.

With some experimentation, we were able to move many of these tasks into their own jobs. Shown in the diagram below, we can see that by using 5 jobs, we can halve the build time. These jobs don’t have any dependencies, when the build starts, five build agents are used to run the jobs, and when all of the jobs are complete, the build is complete.

The YAML for each build job looks very similar to code below.

• job: The job id. This needs to be unique within the stage
• displayName: a friendly description of the job. This shows in the pipeline output
• pool: The type of pool to use – in this case the latest windows image
• variables: Any variables needed
• steps: all of the individual tasks

There are dependencies that can’t be split – for example, SonarCloud needs a build, and each of the .NET projects need a restore, but by splitting each project into it’s own job, we are reducing the amount of code to restore and build, and therefore reducing the overall build time.

Note that if you have artifacts that need to be used in subsequent jobs, you will need to publish the artifacts to your drop folder and downloaded in a future task. The complete build YAML can be viewed in the repo.

Deployments

Let’s look at our deployments next. As you can see, these are currently taking 12-16 minutes to run.

The expensive tasks seem to be the ARM template deployment, SQL database DACPAC deployment, and slot swaps, (warming up the slot takes some time), all contained in one job, as we can see in the diagram below. While we plan to spend some time looking at optimizing these individual steps in the future, for now, we will look at what we can run in parallel.

The new new plan, summarized in the diagram below, shows that the deployment is vastly more complex than the build stage, splitting 8 jobs up over 4 phases.

• As the ARM template deployment is infrastructure as code, we can’t deploy anything else until this is done. This is the only job in the first phase.
• Next, in the second phase, we can deploy the DACPAC to the database, web service, and website to our staging slot. None of these jobs start until the ARM template job is complete.
• The third phase is to run our functional tests, which act as a smoke test to ensure everything is working, running tests directly on the staging slots. This job doesn’t start until all of the jobs in “phase 2” are complete.
• Finally, we head to the 4th phase, where we individually swap the slots, before wrapping up the deployment. These last two jobs don’t start until the functional tests jobs have completed successfully.

The deployment YAML is very similar to the build YAML, with a few key differences:

• deployment: Replaces “job”, but still acts as the job id. This needs to be unique within the stage.
• environment: The location the code is being deployed to
• dependsOn: jobs that are required to be completed successfully before this job starts. In the example below we are running functional tests, which requires 4 previous jobs to complete.
• strategy: The deployment strategy. We talked more about this (and environments) last week in a separate blog post.
• steps: all of the individual tasks

This brings our deployment time down to a consistent ~10-11 minutes

The complete deployment YAML can be viewed in the repo.

Wrap-up

Today we have learned about how to build parallel jobs, and some of the benefits of using them. We’ve seen significant improvement, our complete build/release went from ~45 minutes to ~34 minutes, with a 4 minute build and 10 minute environment deployments.

This helps us to shift left significantly. When we run a pull request build, which runs both a build and deployment to dev, we can receive comprehensive feedback and run ALL of our tests in less than 15 minutes, and can receive feedback about 90% of tests in less than 5 minutes – faster than we can make a cup of coffee.

References

• Parallel jobs: https://docs.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent-jobs?view=azure-devops
• Job definition: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
• Featured image credit: https://www.jobrouter.com/files/_/b/b/csm_Parallel_workflows_fe96f99dd1.png