Last year I was consulting a client on their release management strategy. During this assignment, I created Dream - a web based tool for managing and releasing Maven projects and their dependencies, when you have dozens or hundreds of them.
This was in an area of multiple complex trading systems. The teams were maintaining over 50 startable runtime components, consisting of over 200 individual Java components. All the Java projects were built with the help of Maven already, with each of the projects having its own lifecycle:
Obviously, with such a huge number of components, a manual versioning and release management is a job for Sisyphus. Versioning, tagging, branching, … all these basic things were not done in a systematical way, because of the lack of tools and too much manual work required. Also, it was hard to keep an overview over the dependencies between all those projects: Which other components are using the one I am working on? This question was not always easy to answer for developers.
One of the first things I asked, of course, was: Do you really need so many components with their individual life cycles? Can't we simply collapse everything to 5-10 multi-module projects?
The clear answer was: No, we still consider fine-grained projects as a requirement, because it does have advantages such as:
Not at all easy, because:
We wanted to solve this problem in a more generic way, and decided to create Dream - the Dependency And Release Management System. It should standardize the release process and allow for central, yet fine grained control over projects, dependencies, artifacts.
Before going into more details, please keep in mind the basics of Maven versioning:
Dream keeps a project and dependency database up-to-date, to have the full overview about all projects and their dependencies.
It checks out and updates (every 15 min) the whole source code repository and analyzes its structure. It identifies Maven projects, and reads details about the artifact and dependency situation from the POM files (using embedded Maven, like for example the Maven Eclipse plugin m2e does). And it stores the dependency situation in its own database.
First of all, a web UI allows for analyzing this information. Developers can use Dream to easily answer questions like: Which projects depend on the framework library XY in version 1.2-SNAPSHOT?
Dream allows for bulk POM changes, e.g. aligning project dependencies, globally upgrading specific dependencies, ... - all with just a few clicks instead of changing hundreds of POM files manually.
New projects will be set up on the build server with a few clicks in Dream.
The most important feature, however, is controlling releases. After all, this is why Dream was created! A wizard allows to select components that need to be released. Dream will automatically detect which other components need to be released, too (because they are SNAPSHOT versions). The release builds for each individual project are created on the Bamboo build server and all available agents with the help of the Maven release plugin. Dream sets up Bamboo build plans as required and controls them: It executes them in the correct order, in parallel whenever possible, and monitors them for errors.
Sooner or later after a release, there will be the need for patches of the production system. Dream can be used to create maintenance branches including setting them up on the build server. When a bug in a base library needs to be fixed, Dream will suggest to also create branches for projects that depend on this base library - as they are potentially affected.
The beauty of Dream is: Its usage is always optional. The POMs in SVN stay the final source of truth! Anything that Dream does can be done manually if required - Dream will pick it up automatically.
Dream was built within 3 or 4 months, based on lightweight technologies such as the excellent Vaadin web framework, Spring including Spring Data, and Postgres.
Within the first year of its usage, over 300 releases for multiple systems were created with the help of Dream. Some of them major releases with 100+ involved components, some of them small patch release of just 1 or 2 components. All of them with proper versioning, and full and easy traceability from the versions running on the test and production systems back to the source code.
I know that Dream might be interesting for others, and it was implemented in a rather generic way: The adapters to SVN and Bamboo could be easily replaced with adapters to other SCM systems and build servers. Unfortunately, as Dream was created dring one client's project assignment, I cannot simply open source it.
But I would appreciate to read from you about your thoughts on such a tool.