This article introduces Build Pipelines and Code Repositories in OCI DevOps (released on October 26th 2021) – complementing the Artifact Registry and Deployment Pipelines that were first launched in July of 2021 and that I discussed in this article.
Build [ing software] is an apparently clear indication of a specific process. With a clear result. Or is it? Build is typically an automated activity that is triggered sometimes manually but usually automatically by specific changes in a code repository. A commit or (more commonly) a pull request for a branch or the creation of a release tag will start a process we call the build process. What exactly the build process will do varies. Common elements though are:
- gather resources that are required for the process – from code repositories, library repositories (NPM, Maven, PyPI), container (image) registries
- validate digital signatures for sources and artifacts
- automated source code inspection and improvement such as linting and static code analysis
- transpilation
- compilation
- automated (unit) tests (tests that do not require deployment)
- creation of [optimized, minimized] deployable artifacts – such as container image or JAR, ZIP, Wheel, native binary (for example with GraalVM)
- validate/sign artifacts
- store artifact in a artifact repository – such as a container image registry, plain object storage or a dedicated artifact registry
- trigger a deployment pipeline – to deploy to a testing environment and do extensive (automated) testing or to deploy to a production environment (canary or even full release)
Many tools exist to help implement an automated build process. Once upon a time Sun Microsystem’s Hudson was a huge name in the Java community – then Jenkins took over as leader. CircleCI, GitHub Actions, GitLab, Azure DevOps, AWS CodeBuild, Travis C, Bamboo, Codeship, TeamCity are other prominent offerings. Oracle Cloud had Developer Cloud Service that later was rolled into Visual Builder Studio (a service with many facilities for development and DevOps teams- including Wiki, Storyboards and Workitem management as well as CI and CD features).
OCI DevOps is a recently introduced service on Oracle Cloud Infrastructure (Summer 2021). It is a continuous integration/continuous delivery (CI/CD) service that automates the delivery and deployment of software to Oracle Cloud Infrastructure (OCI) compute platforms. The build pipelines and code repository introduced in this article are most (if not only) meaningful for teams that want to build software that will run on OCI. At present, deployment is done to Compute VM, OKE Kubernetes Cluster or Oracle Function: only code targeted at these platforms benefit from OCI DevOps. OCI DevOps is a specialized (niche) offering that supports cloud native and server light and serverless software engineering.
OCI DevOps Continuous Integration
CI on OCI DevOps consists of two main parts – Code Repositories and Build Pipelines – that are linked to the components introduced earlier for CD: Artifact Registry and Deployment Pipelines. The Build Pipelines make use of other OCI services, such as Vault, Notification and Logging and build on the same underlying framework of IAM (users, credentials, permissions, dynamic groups) and IaaS (Network, Compute VM).
OCI Code Repositories
OCI DevOps offers Git style code repositories – similar to GitHub, GitLab or Azure DevOps. You pay for storage only – no additional charges for the Git repo overhead. You get a Git repo that you access in a secure way – from your command line or local Git GUI tool just like you are used to. Or through a simple, straightforward browser UI that for example allows searching for sources, commits and inspecting pull requests. This UI is not meant to be your next Git power tool.
An OCI Code Repository can be set up as a mirror for another Git repository on GitHub or GitLab. This means that changes to your existing Git repo are replicated to the Code Repository on OCI. The main benefit of doing that is speeding up the build process: when the build needs to fetch the sources from the repo, it will be able to do so much faster from a nearby OCI Code Repository than from a farther removed GitHub or GitLab repository.
OCI Build Pipelines
A build pipeline is the process that is triggered – manually or by an event in the code repository – and that will do whatever it is that needs doing. While that sounds vague, it is quite up to you what you want the build pipeline to do. Four types of stages can be defined in the pipeline: managed build stage, deliver artifact, trigger deployment and wait. You can add multiple stages to a pipeline, to be executed in a sequence or in parallel.
Delivery of artifact: take the outputs from a managed build stage and store them in an artifact repository, either the OCI Container Registry or the Artifact Registry; at present you can not push your container image to an external container registry (that would have to be a build stage or perhaps a Function action in a Deployment Pipeline).
Trigger deployment: start a OCI DevOps Deployment Pipeline, passing exported variables from the build stage(s) for use in the deployment
Managed Build: well, that is of course the most interesting step. In a managed build stage:
- a build server is fired up
- the triggering sources are cloned from the code repository to the build server; these should contain a build specification file – a yaml file that describes the build stage and its actions
- environment variables are set up – based on Secrets in OCI Vault, standard and user defined input parameters to build pipeline or exported variables from previous build stages
- additional artifacts required for the build process are downloaded to the build server (either outcomes from previous build stages or external resources)
- Linux Shell commands are executed to produce the output – and do anything else you want to see done; such a command can run a shell script that in turn contains many commands; however, OCI DevOps can keep track of individual command steps it knows about, not the individual steps in your own script
At the time of writing, there is only one type of build server available: an OCI Compute Instance (AMD) with 1 OCPU and 8 GB RAM with Oracle Linux 7 and some tooling preinstalled (such as Docker for building containers and the OCI CLI). The build server is a clean sheet when it starts running. Everything you need to do the build thing needs to be installed at the beginning of the build. If you need a Java JDK or a Node and NPM environment – it needs to be installed. And when you do your next run, you start all over again. While this guarantees a fresh build under the same conditions every time the pipeline is executed – it also means that builds take [much] longer than they would have to in a properly prepared build server.
One option to speed things up a little could be to use a container in which to do the actual build – tailor the container to your exact and repeated needs and store the image in the OCI Container Registry. You can retrieve this image as input artifact at the start of the build stage.
Another strategy is to have multiple build stages – some working in parallel on different artifacts – with a build stage further down the pipeline collecting the output produced by earlier stages to cook up the final output that is to be published to the artifact repository.
The OCI DevOps team indicates that in the (near) future there will be more flexibility regarding the build server – both its size and its starting composition. Teams should be able to compose a custom build server to use in specific build stages – preinstalling runtimes (Node, NPM, JDK, GraalVM, Maven, Python,…) and tooling (JUnit, Jest, SonarQube, …) they want to use in specific build pipelines.
Note that while OCI DevOps itself is a free service, you do have to pay for the build server – just the pay per use compute costs and the network traffic to and from outside OCI boundaries. .
Resources
Build specification reference – describing the format of the build specification file that drives the actual build on the build server
Example of build specification for building a Node application
OCI Documentation on Build Pipelines
OCI Documentation on Code Repositories
Jonathan Schreiber’s GitHub Repo with Demo of Node application through Build and Deployment through OCI DevOps
