That Cloud looks like a Staging Environment, Part 02
- Posted in:
- automation
- amazon
It’s been a while since I posted the first part of this blog post, but now its time for the thrilling conclusion! Note: Conclusion may not actually be thrilling.
Last time I gave a general outline of the problem we were trying to solve (automatic deployment of an API, into a controlled environment), went through our build process (TeamCity) and quickly ran through how we were using Amazon CloudFormation to setup environments.
This time I will be going over some additional pieces of the environment setup (including distributing dependencies and using Powershell Desired State Configuration for machine setup) and how we are using Octopus Deploy for deployment.
Like the last blog post, this one will be mostly explanation and high level descriptions, as opposed to a copy of the template itself and its dependencies (Powershell scripts, config files, etc). In other words, there won’t be a lot of code, just words.
Distribution Network
The environment setup has a number of dependencies, as you would expect, like scripts, applications (for example, something to zip and unzip archives), configuration files, etc. These dependencies are needed in a number of places. One of those places is wherever the script to create the environment is executed (a development machine or maybe even a CI machine) and another place is the actual AWS instances themselves, way up in the cloud, as they are being spun up and configured.
The most robust way to deal with this, to ensure that the correct versions of the dependencies are being used, is to deploy the dependencies as part of the execution of the script. This way you ensure that you are actually executing what you think you’re executing, and can make local changes and have those changes be used immediately, without having to upload or update scripts stored in some web accessible location (like S3 or an FTP site or something). I’ve seen approaches where dependencies are uploaded to some location once and then manually updated, but I think that approach is risky from a dependency management point of view, so I wanted to improve on it.
I did this in a similar way to what I did when automating the execution of our functional tests. In summary, all of the needed dependencies are collected and compressed during the environment creation script, and uploaded to a temporary location in S3, ready to be downloaded during the execution of the Cloud Formation template within Amazon’s infrastructure.
The biggest issue I had with distributing the dependencies via S3 was getting the newly created EC2 instances specified in the CloudFormation template access to the S3 bucket where the dependencies were. I first tried to use IAM roles (which I don’t really understand), but I didn’t have much luck, probably as a result of inexperience. In the end I went with supplying 3 pieces of information as parameters in the CloudFormation template. The S3 path to the dependencies archive, and a key and secret for a pre-configured AWS user that had guaranteed access to the bucket (via a Bucket Policy).
Within the template, inside the LaunchConfiguration (which defines the machines to be spun up inside an Auto Scaling Group) there is a section for supplying credentials to be used when accessing files stored in an S3 bucket, and the parameters are used there.
"LaunchConfig" : {
"Type" : "AWS::AutoScaling::LaunchConfiguration",
"Metadata" : {
"AWS::CloudFormation::Authentication" : {
"S3AccessCreds" : {
"type" : "S3",
"accessKeyId" : { "Ref" : "S3AccessKey" },
"secretKey" : { "Ref": "S3SecretKey" },
"buckets" : [ { "Ref":"S3BucketName" } ]
}
}
}
}I’m not a huge fan of the approach I had to take in the end, as I feel the IAM roles are a better way to go about it, I’m just not experienced enough to know how to implement them. Maybe next time.
My Wanton Desires
I’ll be honest, I don’t have a lot of experience with Powershell Desired State Configuration (DSC from here on). What I have observed so far, is that it is very useful, as it allows you to take a barebones Windows machine and specify (using a script, so its repeatable) what components you would like to be installed and how you want them to be configured. Things like the .NET Framework, IIS and even third party components like Octopus Tentacles.
When working with virtualisation in AWS, this allows you to skip the part where you have to configure an AMI of your very own, and to instead use one of the pre-built and maintained Amazon AMI’s. This allows you to easily update to the latest, patched version of the OS whenever you want, because you can always just run the same DSC on the new machine to get it into the state you want. You can even switch up the OS without too much trouble, flipping to a newer, greater version or maybe even dropping back to something older.
Even though I don’t have anything to add about DSC, I thought I’d mention it here as a record of the method we are using to configure the Windows instances in AWS during environment setup. Essentially what happens is that AWS gives you the ability to execute arbitrary scripts during the creation of a new EC2 instance. We use a barebones Windows Server 2012 AMI, so during setup it executes a series of scripts (via CloudFormation) one of which is the Powershell DSC script that installs IIS, .NET and an Octopus Tentacle.
I didn’t write the DSC script that we using, I just copied it from someone else in the organisation. It does what I need it to do though, so I haven’t spent any time trying to really understand it. I’m sure I’ll have to dig into it in more depth at some future point, and when I do, I’ll make sure to write about it.
Many Arms Make Light Work
With the environment itself sorted (CloudFormation + dependencies dynamically uploaded to S3 + Powershell DSC for configuration + some custom scripts) now its time to talk about deployment/release management.
Octopus Deploy is all about deployment/release management, and is pretty amazing.
I hadn’t actually used Octopus before this, but I had heard of it. I saw Paul speak at DDD Brisbane 2014, and had a quick chat with him after, so I knew what the product was for and approximately how it worked, but I didn’t realise how good it actually was until I started using it myself.
I think the thing that pleases me most about Octopus is that it treats automation and programmability as a first class citizen, rather than an afterthought. You never have to look too far or dig too deep to figure out how you are going to incorporate your deployment in an automated fashion. Octopus supplies a lot of different programmability options, from an executable to a set of .NET classes to a REST API, all of which let you do anything that you could do through the Octopus website.
Its great, and I wish more products were implemented in the same way.
Our usage of Octopus is very straightforward.
During environment setup Octopus Tentacles are installed on the machines with the appropriate configuration (i.e. in the appropriate role, belonging to the appropriate environment). These tentacles provide the means by which Octopus deploys releases.
One of the outputs from our build process is a NuGet Package containing everything necessary to run the API as a web service. We use Octopack for this (another awesome little tool supplied by Octopus Deploy), which is a simple NuGet Package that you add to your project, and then execute using an extra MSBuild flag at build time. Octopack takes care of expanding dependencies, putting everything in the right directory structure and including any pre and post deploy scripts that you have specified in the appropriate place for execution during deployment.
The package containing our application (versioned appropriately) is uploaded to a NuGet feed, and then we have a project in Octopus Deploy that contains some logic for how to deploy it (stock standard stuff relating to IIS setup). At the end of a CI build (via TeamCity) we create a new release in Octopus Deploy for the version that was just built and then automatically publish it to our CI environment.
This automatic deployment during build is done via a Powershell script.
if ($isCI)
{
Get-ChildItem -Path ($buildDirectory.FullName) | NuGet-Publish -ApiKey $octopusServerApiKey -FeedUrl "$octopusServerUrl/nuget/packages"
. "$repositoryRootPath\scripts\common\Functions-OctopusDeploy.ps1"
$octopusProject = "{OCTOPUS PROJECT NAME]"
New-OctopusRelease -ProjectName $octopusProject -OctopusServerUrl $octopusServerUrl -OctopusApiKey $octopusServerApiKey -Version $versionChangeResult.New -ReleaseNotes "[SCRIPT] Automatic Release created as part of Build."
New-OctopusDeployment -ProjectName $octopusProject -Environment "CI" -OctopusServerUrl $octopusServerUrl -OctopusApiKey $octopusServerApiKey
}Inside the snippet above, the $versionChangeResult is a local variable that defines the version information for the build, with the New property being the new version that was just generated. The NuGet-Publich function is a simple wrapper around NuGet.exe and we’ve abstracted the usage of Octo.exe to a set of functions (New-OctopusRelease, New-OctopusDeployment).
function New-OctopusRelease
{
[CmdletBinding()]
param
(
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$octopusServerUrl,
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$octopusApiKey,
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$projectName,
[string]$releaseNotes,
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$version
)
if ($repositoryRoot -eq $null) { throw "repositoryRoot script scoped variable not set. Thats bad, its used to find dependencies." }
$octoExecutable = Get-OctopusToolsExecutable
$octoExecutablePath = $octoExecutable.FullName
$command = "create-release"
$arguments = @()
$arguments += $command
$arguments += "--project"
$arguments += $projectName
$arguments += "--server"
$arguments += $octopusServerUrl
$arguments += "--apiKey"
$arguments += $octopusApiKey
if (![String]::IsNullOrEmpty($releaseNotes))
{
$arguments += "--releasenotes"
$arguments += "`"$releaseNotes`""
}
if (![String]::IsNullOrEmpty($version))
{
$arguments += "--version"
$arguments += $version
$arguments += "--packageversion"
$arguments += $version
}
(& "$octoExecutablePath" $arguments) | Write-Verbose
$octoReturn = $LASTEXITCODE
if ($octoReturn -ne 0)
{
throw "$command failed. Exit code [$octoReturn]."
}
}The only tricksy thing in the script above is the Get-OctopusToolsExecutable function. All this function does is ensure that the executable exists. It looks inside a known location (relative to the global $repositoryRoot variable) and if it can’t find the executable it will download the appropriate NuGet package.
The rest of the Octopus functions look very similar.
The End
I would love to be able to point you towards a Github repository containing the sum total of all that I have written about with regards to environment setup, deployment, publishing, etc, but I can’t. The first reason is because its very much tied to our specific requirements, and stripping out the sensitive pieces would take me too long. The second reason is that everything is not quite encapsulated in a single repository, which disappointments me greatly. There’s some pieces in TeamCity, some in Octopus Deploy with the rest in the repository. Additionally, the process is dependent on a few external components, like Octopus Deploy. This makes me uncomfortable actually, as I like for everything related to an application to be self contained, ideally within the one repository, including build, deployment, environment setup, etc.
Ignoring the fact that I can’t share the detail of the environment setup (sorry!), I consider this whole adventure to be a massive success.
This is the first time that I’ve felt comfortable with the amount of control that has gone into the setup of an environment. Its completely repeatable, and even has some allowance for creating special, one-off environments for specific purposes. Obviously there would be a number of environments that are long lived (CI, Staging and Production at least) but the ability to create a temporary environment in a completely automated fashion, maybe to do load testing or something similar, is huge for me.
Mostly I just like the fact that the entire environment setup is codified, written into scripts and configuration files that can then be versioned and controlled via some sort of source control (we use Git, you should too). This gives a lot of traceability to the whole process, especially when something breaks, and you never have to struggle with trying to understand exactly what has gone into the environment. Its written right there.
For me personally, a lot of the environment setup was completely new, including CloudFormation, Powershell DSC and Octopus Deploy, so this was a fantastic learning experience.
I assume that I will continue to gravitate towards this sort of DevOps work in the future, and I’m not sad about this at all. Its great fun, if occasionally frustrating.