0 Comments

It makes me highly uncomfortable if someone suggests that I support a piece of software without an automated build process.

In my opinion its one of the cornerstones on top of which software delivery is built. It just makes everything that comes afterwards easier, and enables you to think and plan at a much higher level, allowing you to worry about much more complicated topics.

Like continuous delivery.

But lets shy away from that for a moment, because sometimes you have to deal with the basics before you can do anything else.

In my current position I’m responsible for the engineering behind all of the legacy products in our organization. At this point in time those products make almost all of the money (yay!), but contain 90%+ of the technical terror (boo!) so the most important thing from my point of view is to ensure that we can at least deliver them reliably and regularly.

Now, some of the products are in a good place regarding delivery.

Some, however, are not.

Someones Always Playing Continuous Integration Games

One specific product comes to mind. Now, that’s not to say that the other products are perfect (far from it in fact), but this product in particular is lacking some of the most fundamental elements of good software delivery, and it makes me uneasy.

In fairness, the product is still quite successful (multiple millions of dollars of revenue), but from an engineering point of view, that’s only because of the heroic efforts of the individuals involved.

With no build process, you suffer from the following shortcomings:

  • No versioning (or maybe ad-hoc versioning if you’re lucky). This makes it hard to reason about what variety of software the customer has, and can make support a nightmare. Especially true when you’re dealing with desktop software.
  • Mysterious or arcane build procedures. If no-one has taken the time to recreate the build environment (assuming there is one), then it probably has all sorts of crazy dependencies. This has the side effect of making it really hard to get a new developer involved as well.
  • No automated tests. With no build process running the tests, if you do have tests, they are probably not being run regularly. That’s if you have tests at all of course, because with no process running them, people probably aren’t writing them.
  • A poor or completely ad-hoc distribution mechanism. Without a build process to form the foundation of such a process, the one that does exist is mostly ad-hoc and hard to follow.

But there is no point in dwelling on what we don’t have.

Instead, lets do something about it.

Who Cares They’re Always Changing Continuous Integration Names

The first step is a build script.

Now, as I’ve mentioned before on this blog, I’m a big fan of including the build script into the repository, so that anyone with the appropriate dependencies can just clone the repo and run the script to get a deliverable. Release candidates will be built on some sort of controlled build server obviously, but I’ve found its important to be able to execute the same logic both locally and remotely in order to be able to react to unexpected issues.

Of course, the best number of dependencies outside of the repository is zero, but sometimes that’s not possible. Aim to minimise them at least, either by isolating them and including them directly, or by providing some form of automated bootstrapping.

This particular product is built in a mixture of Delphi (specifically Delphi 7) and .NET, so it wasn’t actually all that difficult to use our existing build framework (a horrific aberration built in Powershell) to get something up and running fairly quickly.

The hardest past was figuring out how to get the Delphi compiler to work from the command line, while still producing the same output as it would if you just followed the current build process (i.e. compilation from within the IDE).

With the compilation out of the way, the second hardest part was creating an artifact that looked and acted like the artifact that was being manually created. This comes in the form of a self-extracting zip file containing an assortment of libraries and executables that make up the “update” package.

Having dealt with both of those challenges, its nothing but smooth sailing.

We Just Want to Dance Here, But We Need An AMI

Ha ha ha ha no.

Being a piece of legacy software, the second step to was to create a build environment that could be used from TeamCIty.

This means an AMI with everything required in order to execute the build script.

For Delphi 7, that means an old version of the Delphi IDE and build tools. Good thing we still had the CD that the installer came on, so we just made an ISO and remotely mounted it in order to install the required software.

Then came the multitude of library and tool dependencies specific to this particular piece of software. Luckily, someone had actually documented enough instructions on how to set up a development environment, so we used that information to complete the configuration of the machine.

A few minor hiccups later and we had a build artifact coming out of TeamCity for this product for the very first time.

A considerable victory.

But it wasn’t versioned yet.

They Call Us Irresponsible, The Versioning Is A Lie

This next step is actually still under construction, but the plan is to use the TeamCity build number input and some static version configuration stored inside the repository to create a SemVer styled version for each build that passes through TeamCity.

Any build not passing through TeamCity, or being built from a branch should be tagged with an appropriate pre-release string (i.e. 1.0.0-[something]), allowing us to distinguish good release candidates (off master via TeamCity) from dangerous builds that should never be released to a real customer.

The abomination of a Powershell build framework allows for most of this sort of stuff, but assumes that a .NET style AssemblyInfo.cs file will exist somewhere in the source.

At the end of the day, we decided to just include such a file for ease of use, and then propagate that version generated via the script into the Delphi executables through means that I am currently unfamiliar with.

Finally, all builds automatically tag the appropriate commit in Git, but that’s pretty much built into TeamCity anyway, so barely worth mentioning.

Conclusion

Like I said at the start of the post, if you don’t have an automated build process, you’re definitely doing it wrong.

I managed to summarise the whole “lets construct a build process” journey into a single, fairly lightweight blog post, but a significant amount of work went into it over the course of a few months. I was only superficially involved (as is mostly the case these days), so I have to give all of the credit to my colleagues.

The victory that this build process represents cannot be understated though, as it will form a solid foundation for everything to come.

A small step in the greater scheme of things, but I’m sure everyone knows the quote at this point.