Microsoft’s new Enterprise Live Migrations (ELM) 🔗 tool (currently in public preview) makes moving repos from Azure DevOps (ADO) to GitHub Enterprise (GHE) easy, and it can even rewire your existing pipelines, as long as they stay in ADO. But is that the way to go? Do your pipelines work how you want them to after the move? In our case, no. We wanted to standardise on GHE’s repo features, so we went hybrid: CI in GHE, CD in ADO. A hybrid approach was suggested to us by the GitHub team and is well known in the community, but everyone draws the line differently. What most of the online guidance doesn’t cover is data residency. ELM only supports GHE.com, and there’s no UK region, which changes what a hybrid setup looks like for UK regulated environments.
This is my take on a potential implementation, with the decisions that we made and the trade-offs we had to accept. This isn’t meant to be a one size fits all solution, but it may help you think about your own migration.
What’s moving to GitHub Actions
We chose to move our CI over to GitHub Actions for a few reasons, primarily to standardise our CI and to get closer to the code.
Putting it closer to the code makes it easier for developers to see the results of their changes, without having to jump between platforms. It also makes it easier to manage the CI builds across our projects. Reusable workflows are built for this. They mean that we can have a simple CI.yaml in each repo, which in turn calls out to a shared workflow with the actual build steps. This simplifies it for developers as they don’t need to worry about the details of the workflow. It also makes it easier from a platform perspective, we can enforce certain steps centrally, and making changes is done in one place and affects all repos.
Make sure you version your shared workflows. We have chosen to start with a v1 tag. While we are iterating on the current workflows, we move that tag to the latest commit to make changes, and create a minor version tag as well to track changes. Our CI builds reference the v1 tag, so they will always use the latest version of the shared workflow. When we make a breaking change, we will create a v2 tag, and update the CI builds to reference that instead. Doing it this way allows us to update them all at once, but have control for when we need to make a breaking change which can then be rolled out incrementally.
So why not deploy from GitHub too?
Ideally you want to have your CI and CD in the same place, but there are some constraints that affected us and our ability to do that. The primary one, data residency. We have our own <name>.ghe.com instance, which is hosted in Europe as that is the closest region available. Our resources are all in UK Azure regions, and we want to be able to deploy privately. GitHub Enterprise has VNet integration for runners, simple right? Wrong, UK South and UK West are not supported regions 🔗 for VNet integration, so we cannot deploy privately from GitHub Actions.
We could instead use the self-hosted runners and deploy them inside our network boundary, but that would mean we would have to manage the runners ourselves, and we would lose the benefits of using managed runners.
Why Azure DevOps and Managed DevOps Pools?
Firstly, why ADO? We already use it, and we’ve made the decision that we will keep it for boards and use the GitHub -> Azure Boards integration. ADO Boards are a much more mature product, especially for teams and organisations that want strict work planning processes.
But the big reason to keep CD in ADO is that we can use Managed DevOps Pools (MDPs) 🔗. MDPs are a resource that you deploy into your own Azure subscription, and the agents are injected into your network through a delegated subnet. They are a managed set of runners that can be used by any ADO pipeline. MDPs give us the private network without the maintenance.
Why not ADO self-hosted agents? Well, the reason is the same as on the GitHub side, self-hosted vs managed. They are a great option, but you have to patch, scale and look after them yourself, and that’s the work MDPs take off our plate.
How the hand-off works
The workflow that we settled on is to do CI and create release artifacts in GHE, and then do the deployment stage in ADO. The hand-off between the two is done through an immutable release in GHE. The release is created by a CD workflow in GHE, and publishes the artifacts to GHE releases. We then trigger our ADO pipeline to pull this release and use it for deployment.
This is a simplified example rather than our full end to end process, but it shows the key hand-offs between the two systems.
To get from GHE to ADO, we use the Azure/pipelines 🔗 action, which triggers an Azure Pipelines run straight from a GitHub Actions workflow. It’s as simple as giving it your project URL and pipeline name, but you will need a PAT for it to authenticate. I’ll come back to that in the trade-offs section. Once triggered, our ADO pipeline uses the GHE API to pull the release artifacts. For ease, I then publish those artifacts on the ADO side as a pipeline artifact, this way they are there for easy reuse, especially between environments. A note on the ADO side: the built-in task for downloading a GitHub release doesn’t support a GHE.com service connection, so I had to fall back to a custom script that calls the GHE API directly. It works, but it’s code we now own and maintain, and it needs a credential passed in. Not ideal, but at the time of writing it was the only option I found. If the built-in task gains support for GHE.com, I’d switch back in a heartbeat.
Now, there are a few things to consider when it comes to the hand-off between the two systems. The first is making sure that the release is immutable. GHE releases support this, but it isn’t on by default, so you need to enable release immutability, either per repo or as an org-wide policy. It only applies to releases created after you turn it on.
Secondly, what is the trigger point, not the technical how, the business why. Some of this will depend on your stakeholders and the business requirement. Equally, you must make sure that the right build gets to the right environments, for example you may choose to push to dev every time a PR is completed into main so that the development team can test it. But then when do you push onto test for stakeholders to see it? Technically there are a few ways to handle the outcome of this decision, firstly it can be workflow dispatch (a manual trigger), no automation and no help for the development teams. Next, you can use tags, the team create a tag on a commit and this is then the GHE action trigger for the process to start. Finally, you can use a release branch, either long or short-lived, and merging into this branch is the start of the process. The important thing is to make sure that the right people are involved in the decision and that it is documented. There are advantages and disadvantages to each of these approaches, and it is important to make sure these are all considered and that everyone is aligned before you make the decision.
Why the immutable release matters
Immutability in releases is a primary concern when designing any release process, and something I’ve talked about in my post on immutable deployments. The key point is that you want to always be able to be sure of what was built is what is deployed, and that if you roll back you have the exact same artifact to redeploy. This is especially important when you have a hand-off between two systems, as it is easy for the wrong artifact to be deployed if you don’t have a clear contract between the two systems.
In our implementation we know that what ADO is pulling is exactly what GHE built and published, and therefore the state it was in, tests passing etc, is the state that will be deployed. If we did not approach it with this design, then ADO would be rebuilding the code and creating a new artifact, which could be different to what was built in GHE. How different is a very open-ended question, and it could be as simple as a dependency being updated, or a test being fixed, or a build step being changed. The point is that you don’t know what has changed, and therefore you don’t know what is being deployed. This is a risk that we are not willing to take.
When it comes to traceability, especially in an incident, we can track back from our production deployment to the exact release and commit that was deployed. This is a key requirement for us, and hopefully everyone doing this! You need to be sure of what was deployed so you can track what changed and why. A great help on this, compared to our previous position, is the built-in release note generation in GitHub releases. It automatically generates the release notes based on the gap between the two commits and any PRs that have been merged. This helps for operations but also is a great help for stakeholder management.
The trade-offs
With every solution, there are always some trade-offs and this is no different. The main trade-off is that we have two systems to manage, however in our position, this would have been the case whether we chose to split deployment or not due to the use of Azure Boards. As part of the two systems, there are two sets of auth to manage. Especially with these kinds of hybrid setups as all of the auth between the systems is done through PATs, which are a pain to manage. PATs expire, have to be stored somewhere safe, and are normally tied to the user who created them. To avoid pinning them to a single person, we are choosing to use a service account with shared ownership, so nothing breaks when someone moves on. We have an ADO PAT stored in GHE so it can trigger the pipeline, and a GHE PAT stored in ADO so it can pull the release from the GHE API. That’s still two long-lived secrets to scope tightly and rotate before they expire, which matters a lot in a regulated environment. You also have teams having to look in two systems to see the state of an application. In my opinion, this leads really heavily into a discussion of who owns what parts of the process. Personally I think that the CI process should be a developer-owned process, and the CD process should be an operations-owned process. This doesn’t mean that as the DevOps side we can’t implement CI, creating reusable workflows and templates for example, but the outcome of that action is a developer problem. Having said that, traceability is better with immutable releases, but it does mean that you have to manage the release history in two systems. This is a trade-off that we are willing to make, as we feel that the benefits of having immutable releases outweigh the cost of managing the release history in two systems.
When this changes
This solution probably isn’t the end-game for us, and anyone else in this situation. If, and hopefully when, GitHub’s private networking reaches UK regions, we will 100% be looking to collapse our deployments back into GitHub Actions and have one platform.
When this does hopefully happen, the move should be small and can be staged, primarily because of the enforcement of immutable releases. The contract between our build and deployment is that artifact, so as long as that remains we can repoint either side of the process.
Personally the migration from ADO to GHE has been a great experience, it’s still on-going and is going to be for a while. I’m learning a ton and solving some really interesting challenges, both technical and business.
