Building A Better Beast, Part 2
- Posted in:
- logging
- elk
- aws
- environment
- powershell
With all of the general context and solution outlining done for now, its time to delve into some of the details. Specifically, the build/test/deploy pipeline for the log stack environments.
Unfortunately, we use the term environmentto describe two things. The first is an Octopus environment, which is basically a grouping construct inside Octopus Deploy, like CI or prod-green. The second is a set of infrastructure intended for a specific purpose, like an Auto Scaling Group and Load Balancer intended to host an API. In the case of the log stack, we have distinct environments for the infrastructure for each layer, like the Broker and the Indexer.
Our environments are all conceptually similar, there is a Git repository that contains everything necessary to create or update the infrastructure (CloudFormation templates, Powershell scripts, etc), along with the logic for what it means to build and validate a Nuget package that can be used to manage the environment. The repository is hooked up to a Build Configuration in TeamCity which runs the build script and the resulting versioned package is uploaded to our Nuget server. The package is then used in TeamCity via other Build Configurations to allow us to Create, Delete, Migrate and otherwise interact with the environment in question.
The creation of this process has happened in bits and pieces over the last few years, most of which I’ve written about on this blog.
Its a decent system, and I’m proud of how far we’ve come and how much automation is now in place, but it’s certainly not without its flaws.
Bestial Rage
The biggest problem with the current process is that while the environment is fully encapsulated as code inside a validated and versioned Nuget package, actually using that package to create or delete an environment is not as simple as it could be. As I mentioned above, we have a set of TeamCity Build Configurations for each environment that allow for the major operations like Create and Delete. If you’ve made changes to an environment and want to deploy them, you have to decide what sort of action is necessary (i.e. “its the first time, Create” or “it already exists, Migrate”) and “run”the build, which will download the package and run the appropriate script.
This is where is gets a bit onerous, especially for production. If you want to change any of the environment parameters from the default values the package was built with, you need to provide a set of parameter overrideswhen you run the build. For production, this means you often end up overriding everything (because production is a separate AWS account) which can be upwards of 10 different parameters, all of which are only visible if you go a look at the source CloudFormation template. You have to do this every time you want to execute that operation (although you can copy the parameters from previous runs, which acts a small shortcut).
The issue with this is that it means production deployments become vulnerable to human error, which is one of the things we’re trying to avoid by automating in the first place!
Another issue is that we lack a true “Update” operation. We only have Create, Delete, Clone and Migrate.
This is entirely my fault, because when I initially put the system together I had a bad experience with the CloudFormation Update command where I accidentally wiped out an S3 bucket containing customer data. As is often the case, that fear then led to an alternate (worse) solution involving cloning, checking, deleting, cloning, checking and deleting (in that order). This was safer, but incredibly slow and prone to failure.
The existence of these two problems (hard to deploy, slow failure-prone deployments) is reason enough for me to consider exploring alternative approaches for the log stack infrastructure.
Fantastic Beasts And Where To Find Them
The existing process does do a number of things well though, and has:
- A Nuget package that contains everything necessary to interact with the environment.
- Environment versioning, because that’s always important for traceability.
- Environment validation via tests executed as part of the build (when possible).
Keeping those three things in mind, and combining them with the desire to ease the actual environment deployment, an improved approach looks a lot like our typical software development/deployment flow.
- Changes to environment are checked in
- Changes are picked up by TeamCity, and a build is started
- Build is tested (i.e. a test environment is created, validated and destroyed)
- Versioned Nuget package is created
- Package is uploaded to Octopus
- Octopus Release is created
- Octopus Release is deployed to CI
- Secondary validation (i.e. test CI environment to make sure it does what it’s supposed to do after deployment)
- [Optional] Propagation of release to Staging
In comparison to our current process, the main difference is the deployment. Prior to this, we were treating our environments as libraries (i.e. they were built, tested, packaged and uploaded to MyGet to be used by something else). Now we’re treating them as self contained deployable components, responsible for knowing how to deploy themselves.
With the approach settled, all that’s left is to come up with an actual deployment process for an environment.
Beast Mastery
There are two main cases we need to take care of when deploying a CloudFormation stack to an Octopus environment.
The first case is what to do when the CloudFormation stack doesn’t exist.
This is the easy case, all we need to do is execute New-CFNStack with the appropriate parameters and then wait for the stack to finish.
The second case is what we should do when the CloudFormation stack already exists, which is the case that is not particularly well covered by our current environment management process.
Luckily, CloudFormation makes this relatively easy with the Update-CFNStack command. Updates are dangerous (as I mentioned above), but if you’re careful with resources that contain state, they are pretty efficient. The implementation of the update is quite smart as well, and will only update the things that have changed in the template (i.e. if you’ve only changed the Load Balancer, it won’t recreate all of your EC2 instances).
The completed deployment script is shown in full below.
[CmdletBinding()]
param
(
)
$here = Split-Path $script:MyInvocation.MyCommand.Path;
$rootDirectory = Get-Item ($here);
$rootDirectoryPath = $rootDirectory.FullName;
$ErrorActionPreference = "Stop";
$component = "unique-stack-name";
if ($OctopusParameters -ne $null)
{
$parameters = ConvertFrom-StringData ([System.IO.File]::ReadAllText("$here/cloudformation.parameters.octopus"));
$awsKey = $OctopusParameters["AWS.Deployment.Key"];
$awsSecret = $OctopusParameters["AWS.Deployment.Secret"];
$awsRegion = $OctopusParameters["AWS.Deployment.Region"];
}
else
{
$parameters = ConvertFrom-StringData ([System.IO.File]::ReadAllText("$here/cloudformation.parameters.local"));
$path = "C:\creds\credentials.json";
Write-Verbose "Attempting to load credentials (AWS Key, Secret, Region, Octopus Url, Key) from local, non-repository stored file at [$path]. This is done this way to allow for a nice development experience in vscode"
$creds = ConvertFrom-Json ([System.IO.File]::ReadAllText($path));
$awsKey = $creds.aws."aws-account".key;
$awsSecret = $creds.aws."aws-account".secret;
$awsRegion = $creds.aws."aws-account".region;
$parameters["OctopusAPIKey"] = $creds.octopus.key;
$parameters["OctopusServerURL"] = $creds.octopus.url;
}
$parameters["Component"] = $component;
$environment = $parameters["OctopusEnvironment"];
. "$here/scripts/common/Functions-Aws.ps1";
. "$here/scripts/common/Functions-Aws-CloudFormation.ps1";
Ensure-AwsPowershellFunctionsAvailable
$tags = @{
"environment"=$environment;
"environment:version"=$parameters["EnvironmentVersion"];
"application"=$component;
"function"="logging";
"team"=$parameters["team"];
}
$stackName = "$environment-$component";
$exists = Test-CloudFormationStack -AwsKey $awsKey -AwsSecret $awsSecret -AwsRegion $awsRegion -StackName $stackName
$cfParams = ConvertTo-CloudFormationParameters $parameters;
$cfTags = ConvertTo-CloudFormationTags $tags;
$args = @{
"-StackName"=$stackName;
"-TemplateBody"=[System.IO.File]::ReadAllText("$here\cloudformation.template");
"-Parameters"=$cfParams;
"-Tags"=$cfTags;
"-AccessKey"=$awsKey;
"-SecretKey"=$awsSecret;
"-Region"=$awsRegion;
"-Capabilities"="CAPABILITY_IAM";
};
if ($exists)
{
Write-Verbose "The stack [$stackName] exists, so I'm going to update it. Its better this way"
$stackId = Update-CFNStack @args;
$desiredStatus = [Amazon.CloudFormation.StackStatus]::UPDATE_COMPLETE;
$failingStatuses = @(
[Amazon.CloudFormation.StackStatus]::UPDATE_FAILED,
[Amazon.CloudFormation.StackStatus]::UPDATE_ROLLBACK_IN_PROGRESS,
[Amazon.CloudFormation.StackStatus]::UPDATE_ROLLBACK_COMPLETE
);
Wait-CloudFormationStack -StackName $stackName -DesiredStatus $desiredStatus -FailingStates $failingStatuses -AwsKey $awsKey -AwsSecret $awsSecret -AwsRegion $awsRegion
Write-Verbose "Stack [$stackName] Updated";
}
else
{
Write-Verbose "The stack [$stackName] does not exist, so I'm going to create it. Just watch me"
$args.Add("-DisableRollback", $true);
$stackId = New-CFNStack @args;
$desiredStatus = [Amazon.CloudFormation.StackStatus]::CREATE_COMPLETE;
$failingStatuses = @(
[Amazon.CloudFormation.StackStatus]::CREATE_FAILED,
[Amazon.CloudFormation.StackStatus]::ROLLBACK_IN_PROGRESS,
[Amazon.CloudFormation.StackStatus]::ROLLBACK_COMPLETE
);
Wait-CloudFormationStack -StackName $stackName -DesiredStatus $desiredStatus -FailingStates $failingStatuses -AwsKey $awsKey -AwsSecret $awsSecret -AwsRegion $awsRegion
Write-Verbose "Stack [$stackName] Created";
}Other than the Create/Update logic that I’ve already talked about, the only other interesting thing in the deployment script is the way that it deals with parameters.
Basically if the script detects that its being run from inside Octopus Deploy (via the presence of an $OctopusParameters variable), it will load all of its parameters (as a hashtable) from a particular local file. This file leverages the Octopus variable substitution feature, so that when we deploy the infrastructure to the various environments, it gets the appropriate values (like a different VPC because prod is a separate AWS account to CI). When its not running in Octopus, it just uses a different file, structured very similarly, with test/scratch values in it.
With the deployment script in place, we plug the whole thing into our existing “deployable” component structure and we have automatic deployment of tested, versioned infrastructure via Octopus Deploy.
Conclusion
Of course, being a first version, the deployment logic that I’ve described above is not perfect. For example, there is no support for deploying to an environment where the stack is in error (failing stacks can’t be updated, but they already exist, so you have to delete it and start again) and there is little to no feedback available if a stack creation/update fails for some reason.
Additionally, the code could benefit from being extracted to a library for reuse.
All in all, the deployment process I just described is a lot simpler than the one I described at the start of this post, and its managed by Octopus, which makes it consistent with the way that we do everything else, which is nice.
With a little bit more polish, and some pretty strict usage of the CloudFormation features that stop you accidentally deleting databases full of valuable data, I think it will be a good replacement for what we do now.
