0 Comments

God that title is terrible. Sorry.

If you’ve ever read any of my blog before, you would know that I’ve put a significant amount of effort into making sure that our environments can be spun up and down easily. A good example of this is the environment setup inside the Solavirum.Testing.JMeter repository, which allows you to easily setup a group of JMeter worker machines for the purposes of load testing.

These environment setup scripts are all well and good, but I really don’t like the fact that I end up executing them from my local machine. This can breed a host of problems, in the same way that compiling and distributing code from your own machine can. Its fine for the rarely provisioned environments (like the JMeter workers I mentioned above) but its definitely not suitable for provisioning a CI environment for a web service or anything similar.

Additionally, when you provision environments from a developer machine you risk accidentally provisioning an environment with changes that have not been committed into source control. This can be useful for testing purposes, but ultimately leads to issues with configuration management. Its nice to tag the commit that the environment was provisioned from as well, on both sides (in source control and on the environment itself, just like versioning a library or executable).

Luckily we already have a platform in place for centralizing our compiling and packaging, and I should be able to use it to do environment management as well.

TeamCity.

I’m Not Sure Why Its Called TeamCity

If you’re unfamiliar with TeamCity, its similar to Jenkins. If you’re unfamiliar with Jenkins, its similar to TeamCity.

Ha.

Anyway, TeamCity is a CI (Continuous Integration) service. It allows you to setup build definitions and then run them on various triggers to produce artifacts (like installers, or Nuget packages). It does a lot more to be honest, but its core idea is about automating the build process in a controlled environment, free from the tyranny of developer machines.

The team here was using TeamCity before I started, so they already had some components being built, including a 20 step monstrosity that I will one day smooth out.

As a general rule, I love a good CI server, but I much prefer to automate in something like Powershell, so that it can be run locally if need be (for debugging/investigation purposes), so I’m wary of putting too much logic inside the actual CI server configuration itself. I definitely like to use the CI server for scheduling, history and other ancillary services (like tagging on successful builds and so on) though, the things that you don’t necessarily need when running a build locally.

Anyway, the meat of this blog post is about automating environment management using TeamCity, so I should probably talk about that now.

Most of my environment provisioning scripts have the same structure (with a few notable exceptions), so it was easy enough to create a Template is TeamCity to automate the destruction and recreation of an environment via scripts that already exist. The template was simple, a link to a git repository (configurable by repo name), a simple build step that just runs some Powershell, a trigger and some build parameters.

The only thing I can really copy here is the Powershell build step, so here it is:

try
{
    if ("%teamcity.build.branch.is_default%" -eq "false") 
    {
        Write-Error "Cannot create environment from non-default branch (i.e. not master)."
        exit 1
    }

    . ".\scripts\build\_Find-RootDirectory.ps1"

    $rootDirectory = Find-RootDirectory "."
    $rootDirectoryPath = $rootDirectory.FullName

    # Consider adding Migration check and run that instead if it exists.
    $environment = "%environment%"
    
    $restrictedEnvironmentRegex = "prod"
    if ($environment -match $restrictedEnvironmentRegex)
    {
        write-error "No. You've selected the environment named [$environment] to create, and it matches the regex [$restrictedEnvironmentRegex]. Think harder before you do this."
        Write-Host "##teamcity[buildProblem description='Restricted Environment Selected']"
        exit 1
    }
    
    $invokeNewEnvironmentPath = ".\scripts\environment\Invoke-NewEnvironment.ps1"
    $invokeDeleteEnvironmentPath = ".\scripts\environment\Invoke-DeleteEnvironment.ps1"

    if(-not (test-path $invokeNewEnvironmentPath) -or -not (test-path $invokeDeleteEnvironmentPath))
    {
        write-error "One of the expected environment management scripts (New: [$invokeNewEnvironmentPath], Delete: [$invokeDeleteEnvironmentPath]) could not be found."
        Write-Host "##teamcity[buildProblem description='Missing Environment Management Scripts']"
        exit 1
    }
    
    $bootstrapPath = ".\scripts\Functions-Bootstrap.ps1"
    if (-not (test-path $bootstrapPath))
    {
        Write-Warning "The bootstrap functions were not available at [$bootstrapPath]. This might not be important if everything is already present in the repository."
    }
    else
    {
        . $bootstrapPath
        Ensure-CommonScriptsAvailable
    }
    
    $octopusUrl = "%octopusdeploy-server-url%"
    $octopusApiKey = "%octopusdeploy-apikey%"
    $awsKey = "%environment-deployment-aws-key%"
    $awsSecret = "%environment-deployment-aws-secret%"
    $awsRegion =  "%environment-deployment-aws-region%"
    
    $arguments = @{}
    $arguments.Add("-Verbose", $true)
    $arguments.Add("-AwsKey", $awsKey)
    $arguments.Add("-AwsSecret", $awsSecret)
    $arguments.Add("-AwsRegion", $awsRegion)
    $arguments.Add("-OctopusApiKey", $octopusApiKey)
    $arguments.Add("-OctopusServerUrl", $octopusUrl)
    $arguments.Add("-EnvironmentName", $environment)
    
    try
    {
        Write-Host "##teamcity[blockOpened name='Delete Environment']"
        Write-Host "##teamcity[buildStatus text='Deleting $environment']"
        & $invokeDeleteEnvironmentPath @arguments
        Write-Host "##teamcity[buildStatus text='$environment Deleted']"
        Write-Host "##teamcity[blockClosed name='Delete Environment']"
    }
    catch
    {
        write-error $_
        Write-Host "##teamcity[buildProblem description='$environment Deletion Failed']"
        exit 1
    }

    try
    {
        $recreate = "%environment-recreate%"
        if ($recreate -eq "true")
        {
            Write-Host "##teamcity[blockOpened name='Create Environment']"
            Write-Host "##teamcity[buildStatus text='Creating $environment']"
            & $invokeNewEnvironmentPath @arguments
            Write-Host "##teamcity[buildStatus text='$environment Created']"
            Write-Host "##teamcity[blockClosed name='Create Environment']"
        }
    }
    catch
    {
        write-error $_
        Write-Host "##teamcity[buildProblem description='$environment Created Failed']"
        exit 1
    }
}
catch 
{
    write-error $_
    Write-Host "##teamcity[buildProblem description='$environment Created Failed']"
    exit 1
}

Once I had the template, I created new build configurations for each environment I was interested in, and filled them out appropriately.

Now I could recreate an entire environment just by clicking a button in TeamCity, and every successful recreation was tagged appropriately in Source Control, which was awesome. Now I had some traceability.

The final step was to schedule an automatic recreation of each CI environment every morning, to constantly validate our scripts and make sure they work appropriately.

Future Improvements

Alas, I ran into one of the most annoying parts of TeamCity. After the initial 20, licensing is partially based on build configurations. We already had a significant amount of configs, so I ran out before I could implement a build configuration to do a nightly tear down of environments that don’t need to exist overnight (for example all our CI environments). I had to settle for merely recreating them each morning (tear down followed by spin up), which at least verifies that the scripts continue to work.

If I could change build parameters based on a Trigger in TeamCity that would also work, but that’s a missing feature for now. I could simply set up two triggers, one for the morning to recreate and the other for the evening to tear down (where they both execute the same script, just with different inputs). This has been a requested feature of TeamCity for a while now, so I hope they get to it at some stage.

I’ll rectify this as soon as we get more build configurations. Which actually leads nicely into my next point.

So, What’s It Cost

Its free! Kinda.

Its free for 3 build agents and 20 build configurations. You can choose to buy another agent + 10 build configs for a token amount (currently $300 US), or you can choose to buy unlimited build configurations (the Enterprise edition) for another token amount (currently $2000 US).

If you’re anything like me, and you love to automate all the things, you will run out of build configurations far before you need more build agents.

I made the mistake of getting two build agent + configs packs through our organization before I realized that I should have just bought the Enterprise edition, and now I’m having a hard time justifying its purchase to the people what control the money. Unfortunate, but I’m sure I’ll convince them in time, and we’ve got an extra 2 build agents as well, so that’s always nice.

Jetbrains (the creators of TeamCity) were kind of annoying in this situation actually. We wanted to switch to Enterprise, and realized we didn’t need the build agents (yet), but they wouldn’t do us a deal. I can understand that its just probably their policy, but its still annoying.

Summary

I’m never happy unless what I’ve done can be executed on a machine completely different from my own, without a lot of futzing about. I’m a big proponent for “it should just work”, and having the environments triggered from TeamCity enforces that sort of thinking. Our build agents are pretty vanilla as far as our newer code is concerned (our legacy code has some nasty installed dependencies that I won’t go into detail about), so being able to execute the environment provisioning through TeamCity constantly validates that the code works straight out of source control.

It also lets other people create environments too, and essentially documents the usage of the environment provisioning scripts.

I get some nice side effects from doing environment management in TeamCtiy as well, the most important of which is the ability to easily tag when environments were provisioned (and from what commit) in source control.

Now I just need more build configurations…