Dream - Dependency And Release Management System



2013-10-14

Last year I was consulting a client on their release management strategy. During this assignment, I created Dream - a web based tool for managing and releasing Maven projects and their dependencies, when you have dozens or hundreds of them.

200+ Maven projects with individual lifecycles

This was in an area of multiple complex trading systems. The teams were maintaining over 50 startable runtime components, consisting of over 200 individual Java components. All the Java projects were built with the help of Maven already, with each of the projects having its own lifecycle:

  • each project with its own version number
  • each project with its own Maven build (only few multi-module builds)
  • each project with its own Subversion trunk, branches, tags

Obviously, with such a huge number of components, a manual versioning and release management is a job for Sisyphus. Versioning, tagging, branching, … all these basic things were not done in a systematical way, because of the lack of tools and too much manual work required. Also, it was hard to keep an overview over the dependencies between all those projects: Which other components are using the one I am working on? This question was not always easy to answer for developers.

The advantages of fine grained projects

One of the first things I asked, of course, was: Do you really need so many components with their individual life cycles? Can't we simply collapse everything to 5-10 multi-module projects?

The clear answer was: No, we still consider fine-grained projects as a requirement, because it does have advantages such as:

  • Base libraries can be re-used across multiple systems, but not all of them in each system
  • Clear separation of logical sub-components
  • Enforce the architecture through Maven dependencies, e.g. it is impossible for a framework library to refer to application code - other big projects need tools like Sonargraph to ensure this

How to manage such a high number of projects with their individual lifecycles?

Not at all easy, because:

  • The standard Maven tooling, such as the Maven Release Plugin, is focussing on the individual project: To be able to release one project, all its dependencies must have been released earlier.
  • Many Continuous Integration (CI) Servers are Maven-aware and know about the dependencies between the projects, but they focus on the CI builds: E.g. Atlassian Bamboo would allow running the release build for one project multiple times which would be a problem.
  • Web research results were: No standard tools seem to be on the market that help managing a high number of inter-dependent projects, but many people seem to build their own custom tooling for this. Often very project-specific scripts.

We wanted to solve this problem in a more generic way, and decided to create Dream - the Dependency And Release Management System. It should standardize the release process and allow for central, yet fine grained control over projects, dependencies, artifacts.

Maven Versioning

Before going into more details, please keep in mind the basics of Maven versioning:

  • While developing a component, you are working on "Snapshot" versions, e.g. 1.0-SNAPSHOT, i.e. snapshots on the way towards 1.0.
  • When the component is released, a version 1.0 is built and released.
  • After the release you are working towards the next version, i.e. 1.1-SNAPSHOT or 2.0-SNAPSHOT.
  • While snapshot versions change all the time, a released version is stable - it won't be changed afterwards.

What's the idea behind Dream?

Dream keeps a project and dependency database up-to-date, to have the full overview about all projects and their dependencies.

It checks out and updates (every 15 min) the whole source code repository and analyzes its structure. It identifies Maven projects, and reads details about the artifact and dependency situation from the POM files (using embedded Maven, like for example the Maven Eclipse plugin m2e does). And it stores the dependency situation in its own database.

What to do with this information?

First of all, a web UI allows for analyzing this information. Developers can use Dream to easily answer questions like: Which projects depend on the framework library XY in version 1.2-SNAPSHOT?

Dream allows for bulk POM changes, e.g. aligning project dependencies, globally upgrading specific dependencies, ... - all with just a few clicks instead of changing hundreds of POM files manually.

New projects will be set up on the build server with a few clicks in Dream.

The most important feature, however, is controlling releases. After all, this is why Dream was created! A wizard allows to select components that need to be released. Dream will automatically detect which other components need to be released, too (because they are SNAPSHOT versions). The release builds for each individual project are created on the Bamboo build server and all available agents with the help of the Maven release plugin. Dream sets up Bamboo build plans as required and controls them: It executes them in the correct order, in parallel whenever possible, and monitors them for errors.

Sooner or later after a release, there will be the need for patches of the production system. Dream can be used to create maintenance branches including setting them up on the build server. When a bug in a base library needs to be fixed, Dream will suggest to also create branches for projects that depend on this base library - as they are potentially affected.

The beauty of Dream is: Its usage is always optional. The POMs in SVN stay the final source of truth! Anything that Dream does can be done manually if required - Dream will pick it up automatically.

Background

Dream was built within 3 or 4 months, based on lightweight technologies such as the excellent Vaadin web framework, Spring including Spring Data, and Postgres.

Within the first year of its usage, over 300 releases for multiple systems were created with the help of Dream. Some of them major releases with 100+ involved components, some of them small patch release of just 1 or 2 components. All of them with proper versioning, and full and easy traceability from the versions running on the test and production systems back to the source code.

I know that Dream might be interesting for others, and it was implemented in a rather generic way: The adapters to SVN and Bamboo could be easily replaced with adapters to other SCM systems and build servers. Unfortunately, as Dream was created dring one client's project assignment, I cannot simply open source it.

But I would appreciate to read from you about your thoughts on such a tool.