“Continuous Delivery (CD) is a software engineering approach in which teams keep producing valuable software in short cycles and ensure that the software can be reliably released at any time.” (from here)
Software artefacts are developed using a build pipeline. This pipeline consists of several steps to provide quick feedback on software quality by means of code quality checks, automated tests, test coverage checks, etc. When the software is done (adhering to a Definition of Done (DoD)), it is released. This release as a whole is then tested and promoted until it reaches a production environment. In the meantime, work on a next release has already started. This process is shown in the below image. This is a minimal example (especially on the test part). In the below image, you can see there are 3 releases in progress. This is only in order to illustrate the process. You should of course try to limit the number of releases which are not in production to reduce the overhead of fixes on those releases.
Automation of the release process is often a challenge. Why is this difficult? One of the reasons is the identification of what should be in a release. This is especially so when the process of creating a release is not automated. There is a transition phase (in the image between Test phase 2 and Test phase 3) when the unit (artefact) build pipeline stops and when the release as a whole continues to be promoted through the environments. In the image of the process above, you can easily identify where you can automate the construction of a release; at the end of the unit build pipeline. This is where you can identify the unit which has been approved by means of the different test phases / quality checks in the unit build pipeline and you know the release the unit has to be put in to be propagated as a whole. Why not add the unit to the release in an automated fashion there?
Automation
Artifact repository, Maven POM and dependencies
A common practice is to put artifacts in an artifact repository. Artifact repositories are often Maven compliant (i.e. Nexus, Artifactory) so you can identify your artifact with Maven GAV coordinates (groupId, artifactId, version). What better way to describe a collection of artifacts which can be identified with GAV attributes than a Maven POM file? You can define your artifacts as dependencies in your POM file. The Maven assembly plugin can then download those artifacts and put them in a specific structure to allow easy deployment. In case a dependency is already there, you want to update the version in the POM. If it is not there, you want to add it.
Automating adding dependencies to a POM
Below is a short Python (2.7) script to add dependencies to a POM file if the dependency is not there yet or to update the version of the dependency if it already is there.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | import os import xml.etree.ElementTree as ET import xml.dom.minidom as minidom import sys,re import argparse #script updates a pom.xml file with a specific artifactid/groupid/version/type/classifier dependency #if the dependency is already there, the version is checked and updated if needed #if the dependency is not there, it is added #the comparison of dependencies is based on artifactid/groupid/type (and optionally classifier). other fields are ignored #the pom file should be in UTF-8 #set the default namespace of the pom.xml file pom_ns = dict(pom='http://maven.apache.org/POM/4.0.0') ET.register_namespace('',pom_ns.get('pom')) #parse the arguments parser = argparse.ArgumentParser(description='Update pom.xml file with dependency') parser.add_argument('pomlocation', help='Location on the filesystem of the pom.xml file to update') parser.add_argument('artifactid', help='ArtifactId of the artifact to update') parser.add_argument('groupid', help='GroupId of the artifact to update') parser.add_argument('version', help='Version of the artifact to update') parser.add_argument('type', help='Type of the artifact to update') parser.add_argument('--classifier', help='Classifier of the artifact to update',default=None) args = parser.parse_args() pomlocation=args.pomlocation artifactid=args.artifactid groupid=args.groupid version=args.version type=args.type classifier=args.classifier #read a file and return a ElementTree def get_tree_from_xmlfile(filename): if os.path.isfile(filename): tree = ET.parse(filename) return tree else: raise Exception('Error opening '+filename) #obtain a specific element from an ElementTree based on an xpath def get_xpath_element_from_tree(tree,xpath,namespaces): return tree.find(xpath, namespaces) #returns the content of an element as a string def element_to_str(element): return ET.tostring(element, encoding='utf8', method='xml') #returns an ElementTree as a pretty printed string def elementtree_to_str(et): root=et.getroot() ugly_xml = ET.tostring(root, encoding='utf8', method='xml') dom=minidom.parseString(ugly_xml) prettyXML=dom.toprettyxml('\t','\n','utf8') trails=re.compile(r'\s+\n') prettyXML=re.sub(trails,"\n",prettyXML) return prettyXML #creates an Element object with artifactId, groupId, version, type, classifier elements (used to append a new dependency). classifier is left out if None def create_dependency(param_groupid,param_artifactid,param_version,param_type,param_classifier): dependency_element = ET.Element("dependency") groupid_element = ET.Element("groupId") groupid_element.text = param_groupid dependency_element.append(groupid_element) artifactid_element = ET.Element("artifactId") artifactid_element.text = param_artifactid dependency_element.append(artifactid_element) version_element = ET.Element("version") version_element.text = param_version dependency_element.append(version_element) type_element = ET.Element("type") type_element.text = param_type dependency_element.append(type_element) if param_classifier is not None: classifier_element = ET.Element("classifier") classifier_element.text = param_classifier dependency_element.append(classifier_element) return dependency_element #adds a dependency element to a pom ElementTree. the dependency element can be created with create_dependency def add_dependency(pom_et,dependency_element): pom_et.find('pom:dependencies',pom_ns).append(dependency_element) return pom_et #update the version of a dependency in the pom ElementTree if it is already present. else adds the dependency #returns the updated ElementTree and a boolean indicating if the pom ElementTree has been updated def merge_dependency(pom_et,param_artifactid,param_groupid,param_type,param_version,param_classifier): artifactfound=False pom_et_changed=False for dependency_element in pom_et.findall('pom:dependencies/pom:dependency',pom_ns): checkgroupid = get_xpath_element_from_tree(dependency_element,'pom:groupId',pom_ns).text checkartifactid = get_xpath_element_from_tree(dependency_element,'pom:artifactId',pom_ns).text checktype = get_xpath_element_from_tree(dependency_element,'pom:type',pom_ns).text if param_classifier is not None: checkclassifier_el = get_xpath_element_from_tree(dependency_element,'pom:classifier',pom_ns) if checkclassifier_el is not None: checkclassifier=checkclassifier_el.text else: checkclassifier=None else: checkclassifier = None if (checkgroupid == param_groupid and checkartifactid == param_artifactid and checktype == param_type and (checkclassifier == param_classifier or param_classifier is None)): artifactfound=True print 'Artifact found in '+pomlocation pomversion=dependency_element.find('pom:version',pom_ns).text if pomversion != param_version: print "Artifact has different version in "+pomlocation+". Updating" dependency_element.find('pom:version',pom_ns).text=param_version pom_et_changed=True else: print "Artifact already in "+pomlocation+" with correct version. Update not needed" if not artifactfound: print 'Artifact not found in pom. Adding' dependency_element = create_dependency(param_groupid,param_artifactid,param_version,param_type,param_classifier) pom_et = add_dependency(pom_et,dependency_element) pom_et_changed=True return pom_et,pom_et_changed #read the file at the pomlocation parameter pom_et = get_tree_from_xmlfile(pomlocation) #merge the dependency into the obtained ElementTree pom_et,pom_et_changed=merge_dependency(pom_et,artifactid,groupid,type,version,classifier) #overwrite the pomlocation if it has been changed if pom_et_changed: print "Overwriting "+pomlocation+" with changes" target = open(pomlocation, 'w') target.truncate() target.write(elementtree_to_str(pom_et)) target.close() else: print pomlocation+" does not require changes" |
The script can deal with an optional classifier. When not specified it updates dependencies without looking at the classifier so be careful with this. Also the script does some pretty printing when updating. This makes it easy to compare the POM file after a version control commit to for example compare different releases.
Seeing it work
Example pom.xml file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | <?xml version="1.0" encoding="utf8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>nl.amis.smeetsm.release</groupId> <artifactId>Release</artifactId> <packaging>pom</packaging> <version>1.0</version> <dependencies> <dependency> <groupId>nl.amis.smeetsm.functionalunit.HelloWorld</groupId> <artifactId>HelloWorld_FU</artifactId> <version>1.0</version> <type>pom</type> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-assembly-plugin</artifactId> <version>2.5.4</version> <configuration> <descriptors> <descriptor>release-assembly.xml</descriptor> </descriptors> </configuration> </plugin> </plugins> </build> </project> |
The artifact HelloWorld_FU contains dependencies to other artifacts ending in SCA or SB to indicate if it is a SOA Suite SCA composite artifact or a Service Bus artifact. The release-assembly.xml file below puts the different types in different directories and zips the result. This way a release zip file is created.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | <assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd"> <id>release</id> <formats> <format>zip</format> </formats> <dependencySets> <dependencySet> <outputDirectory>/composite</outputDirectory> <includes> <include>nl.amis.smeetsm.*:*_SCA</include> </includes> </dependencySet> <dependencySet> <outputDirectory>/servicebus</outputDirectory> <includes> <include>nl.amis.smeetsm.*:*_SB</include> </includes> </dependencySet> </dependencySets> </assembly> |
Updating a dependency version: releasescript.py pom.xml HelloWorld_FU nl.amis.smeetsm.functionalunit.HelloWorld 2.0 pom
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | <?xml version="1.0" encoding="utf8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>nl.amis.smeetsm.release</groupId> <artifactId>Release</artifactId> <packaging>pom</packaging> <version>1.0</version> <dependencies> <dependency> <groupId>nl.amis.smeetsm.functionalunit.HelloWorld</groupId> <artifactId>HelloWorld_FU</artifactId> <version>2.0</version> <type>pom</type> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-assembly-plugin</artifactId> <version>2.5.4</version> <configuration> <descriptors> <descriptor>release-assembly.xml</descriptor> </descriptors> </configuration> </plugin> </plugins> </build> </project> |
Adding a dependency version: releasescript.py pom.xml ByeWorld_FU nl.amis.smeetsm.functionalunit.ByeWorld 2.0 pom
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | <?xml version="1.0" encoding="utf8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>nl.amis.smeetsm.release</groupId> <artifactId>Release</artifactId> <packaging>pom</packaging> <version>1.0</version> <dependencies> <dependency> <groupId>nl.amis.smeetsm.functionalunit.HelloWorld</groupId> <artifactId>HelloWorld_FU</artifactId> <version>2.0</version> <type>pom</type> </dependency> <dependency> <groupId>nl.amis.smeetsm.functionalunit.ByeWorld</groupId> <artifactId>ByeWorld_FU</artifactId> <version>2.0</version> <type>pom</type> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-assembly-plugin</artifactId> <version>2.5.4</version> <configuration> <descriptors> <descriptor>release-assembly.xml</descriptor> </descriptors> </configuration> </plugin> </plugins> </build> </project> |
Finally
When you want to start with the next release, you should create a branch in your version control system of the current release. This way you can separate releases in version control and can also easily create fixes on existing releases.
