That Neat Little Symbol From Half-Life, Part 4
Another week, another piece of the puzzle.
This time around, I’ll go through how we’ve setup the build and deployment pipeline for the Lambda function code that processes the ELB log files. Its not a particularly complex system, but it was something completely different compared to things we’ve done in the past.
In general terms, building software is relatively straightforward, as long as you have two things:
- A mechanism of tracking changes to the source code, i.e. source control.
- The ability to validate the code and create a versioned package, where the behaviour of the code can be reasoned about.
With at least those two pieces in place (preferably fully automated), your builds are taken care of. Granted, the accomplishing those things is not as simple as two dot points would lead you to believe, but conceptually there is not a lot going on.
Once you have versioned packages that can be reasoned about, all you need to do is figure out how to deliver them to the appropriate places. Again, ideally the whole process is automated. You shouldn’t have to remote onto a machine and manually copy files around, as that hardly ever ends well. This can get quite complicated depending on exactly what it is that you are deploying: on-premises software can be notoriously painful to deploy without some supporting system, but deploying static page websites to S3 is ridiculously easy.
My team uses TeamCity, Nuget and Powershell to accomplish the build part and Octopus Deploy to deploy almost all of our code, and I don’t plan on changing any of that particularly soon.
Some people seem to think that because its so easy to manage Lambda functions from the AWS management website that they don’t have to have a build and deployment pipeline. After all, just paste the code into the website and you’re good to go, right?
I disagree vehemently.
Power Up
Our ELB logs processor Lambda function follows our normal repository structure, just like any other piece of software we write.
The code for the Lambda function goes into the /src folder, along with a Nuspec file that describes how to construct the resulting versioned package at build time.
Inside a /scripts folder is a build script, written in Powershell, containing all of the logic necessary to build and verify a deployable package. It mostly just leverages a library of common functions (so our builds are consistent), and its goal is to facilitate all of the parts of the pipeline, like compilation (ha Javascript), versioning, testing, packaging and deployment.
Some build systems are completely encapsulated inside the software that does the build, like Jenkins or TeamCity. I don’t like this approach, because you can’t easily run/debug the build on a different machine for any reason. I much prefer the model where the repository has all of the knowledge necessary to do the entire build and deployment process, barring the bits that it needs to accomplish via an external system.
The build script the ELB logs processor function is included below, but keep in mind, this is just the entry point, and a lot of the bits and pieces are inside the common functions that you can see referenced.
[CmdletBinding()]
param
(
[switch]$deploy,
[string]$octopusServerUrl,
[string]$octopusApiKey,
[string]$component,
[string]$commaSeparatedDeploymentEnvironments,
[string[]]$projects,
[int]$buildNumber,
[switch]$prerelease,
[string]$prereleaseTag
)
$error.Clear()
$ErrorActionPreference = "Stop"
$here = Split-Path $script:MyInvocation.MyCommand.Path
. "$here\_Find-RootDirectory.ps1"
$rootDirectory = Find-RootDirectory $here
$rootDirectoryPath = $rootDirectory.FullName
. "$rootDirectoryPath\scripts\Invoke-Bootstrap.ps1"
. "$rootDirectoryPath\scripts\common\Functions-Build.ps1"
$arguments = @{}
$arguments.Add("Deploy", $deploy)
$arguments.Add("CommaSeparatedDeploymentEnvironments", $commaSeparatedDeploymentEnvironments)
$arguments.Add("OctopusServerUrl", $octopusServerUrl)
$arguments.Add("OctopusServerApiKey", $octopusApiKey)
$arguments.Add("Projects", $projects)
$arguments.Add("VersionStrategy", "SemVerWithPatchFilledAutomaticallyWithBuildNumber")
$arguments.Add("buildNumber", $buildNumber)
$arguments.Add("Prerelease", $prerelease)
$arguments.Add("PrereleaseTag", $prereleaseTag)
$arguments.Add("BuildEngineName", "nuget")
Build-DeployableComponent @argumentsI’m pretty sure I’ve talked about our build process and common scripts before, so I’m not going to go into any more detail.
Prior to deployment, the only interesting output is the versioned Nuget file, containing all of the dependencies necessary to run the Lambda function (which in our case is just the file in my previous post).
On A Rail
When it comes to deploying the Lambda function code, its a little bit more complicated that our standard software deployments via Octopus Deploy.
In most cases, we create a versioned package containing all of the necessary logic to deploy the software, so in the case of an API it contains a deploy.ps1 script that gets run automatically during deployment, responsible for creating a website and configuring IIS on the local machine. The most important thing Octopus does for us to provide mechanisms to get this package to the place where it needs to be.
It usually does this via an Octopus Tentacle, a service that runs on the deployment target and allows for communication between the Octopus server and the machine in question.
This system kind of falls apart when you’re trying to deploy to an AWS Lambda function, which cannot have a tentacle installed on it.
Instead, we rely on the AWS API and what amounts to a worker machine sitting in each environment.
When we do a deployment of our Lambda function project, it gets copied to the worker machine (which is actually just the Octopus server) and it runs the deployment script backed into the package. This script then uses Octopus variables to package the code again (in a way that AWS likes, a simple zip file) and uses the AWS API to upload the changed code to the appropriate Lambda function (by following a naming convention).
The deployment script is pretty straightforward:
function Get-OctopusParameter
{
[CmdletBinding()]
param
(
[string]$key
)
if ($OctopusParameters -eq $null)
{
throw "No variable called OctopusParameters is available. This script should be executed as part of an Octopus deployment."
}
if (-not($OctopusParameters.ContainsKey($key)))
{
throw "The key [$key] could not be found in the set of OctopusParameters."
}
return $OctopusParameters[$key]
}
$VerbosePreference = "Continue"
$ErrorActionPreference = "Stop"
$here = Split-Path -Parent $MyInvocation.MyCommand.Path
. "$here\_Find-RootDirectory.ps1"
$rootDirectory = Find-RootDirectory $here
$rootDirectoryPath = $rootDirectory.FullName
$awsKey = $OctopusParameters["AWS.Deployment.Key"]
$awsSecret = $OctopusParameters["AWS.Deployment.Secret"]
$awsRegion = $OctopusParameters["AWS.Deployment.Region"]
$environment = $OctopusParameters["Octopus.Environment.Name"]
$version = $OctopusParameters["Octopus.Release.Number"]
. "$rootDirectoryPath\scripts\common\Functions-Aws.ps1"
$aws = Get-AwsCliExecutablePath
$env:AWS_ACCESS_KEY_ID = $awsKey
$env:AWS_SECRET_ACCESS_KEY = $awsSecret
$env:AWS_DEFAULT_REGION = $awsRegion
$functionPath = "$here\src\function"
Write-Verbose "Compressing lambda code file"
Add-Type -AssemblyName System.IO.Compression.FileSystem
[system.io.compression.zipfile]::CreateFromDirectory($functionPath, "index.zip")
Write-Verbose "Updating Log Processor lambda function to version [$environment/$version]"
(& $aws lambda update-function-code --function-name $environment-ELBLogProcessorFunction --zip-file fileb://index.zip) | Write-VerboseNothing fancy, just using the AWS CLI to deploy code to a known function.
Apprehension
AWS Lambda does provide some other mechanisms to deploy code, and we probably could have used them to accomplish the same sort of thing, but I like our patterns to stay consistent and I’m a big fan of the functionality that Octopus Deploy provides, so I didn’t want to give that up.
We had to make a few changes to our environment pattern to allow for non-machine based deployment, like:
- Ensuring every automatically created environment has a machine registered in it representing the Octopus server (for running scripts that effect external systems)
- This meant that our environment cleanup also needed to take this into account, deregistering the machine before trying to remove the Octopus environment
- Providing a section at the end of the environment setup to deploy Octopus projects that aren’t related to specific machines
- Previously all of our deployments happened via cfn-init on the EC2 instances in question
Once all of that was in place, it was pretty easy to deploy code to a Lambda function as part of setting up the environment, just like we would deploy code to an EC2 instance. It was one of those cases where I’m glad we wrap our usage of CloudFormation in Powershell, because if we were just using raw CloudFormation, I’m not sure how we would have integrated the usage of Octopus Deploy.
To Be Summarised
I’ve only got one more post left in this series, which will summarise the entire thing and link back to all the constituent parts.
Until then, I don’t really have anything else to say.