AWS Lambda and .NET Core, Two Great Tastes That Taste Great Together, Part 3
Last week I wrote a high level overview of the build, package, test and deploy process that we created in order to get .NET Core C# into AWS Lambda.
As part of that post, I briefly touched on the validation part of the proces, which basically amounted to “we wrote some tests in C# using XUnit, then used the dotnet command line interface to run them as part of the build”.
Considering that the testing took a decent amount of the total time, that amount of coverage isn’t really appropriate, so this post will elaborate further, and explain some of the challenges we encountered and how we overcame them.
I’m A Little Testy
At a high level, testing a .NET Core AWS Lambda function written in C# is not really any more difficult than testing other C# code. Like the stuff I’ve been writing for years.
When you write a lambda function, all you really need is a single class with a method on it (Handler::Handler or equivalent). You could, if you wanted, put literally all of your logic into this class/method combination and have a fully functional lambda function.
That would probably make it harder to test though.
A better idea is to break it down into its constituent parts, ideally taking the form of small classes with a single responsibility, working together to accomplish a greater goal.
For the ELB Logs Processor, there are three clear responsibilities:
- Read a file from S3, and expose the information therein as lines of text
- Take a line of text and transform it into some sort of structure, potentially breaking it down and adding meta information
- Take a structure and output it to a Logstash endpoint
Technically there is also a fourth responsibility which is to tie the three things above together (i.e. orchestration), but we’ll get to that in a minute.
Anyway, if the responsibilities are clearly separated, we’re in unit testing territory, which is familiar enough that I don’t really want to talk about it too much. Grab your testing framework of choice (XUnit is a good pick for .NET Core, NUnit didn’t seem to work too well in Visual Studio + Resharper), mock out your dependencies and test away.
You can apply the same sort of testing to the orchestrator (the Handler) by giving it its dependencies through its constructor. Make sure you give it a parameterless constructor with default values though, or you’ll have issues running it in AWS.
This is basically just a poor mans dependency injection, but my favour dependency injection library, Ninject, does not exist in .NET Core and I didn’t want to invest the time to learn one of the other ones (like AutoFac) for such a small project.
Of course, while unit tests are valuabl, they don’t really tell us whether or not everything will work when you put it all together.
That’s where integration tests come in.
Integration Functions
In my experience, I’ve consistently found there to be two distinct flavours of integration test.
The first is the test that validates whether or not everything works when all of your things are put together. This tends to end up being a test of your dependency injection configuration (if you’re using it), where you create some top level object using your container of choice, then execute a series of methods on it (and its children) echoing what would normally happen in your program execution. You might mock out the very edges of the system (like databases, API dependencies, disk-based configuration, etc), but in general you want to validate that your tens (hundreds? thousands?) of classes play nice together and actually accomplish their overarching goal.
The second type of test is the one that validates whether or not a class that relies on an external system works as expected. These are very close to unit tests in appearance (single class, some mocked out dependencies), but are much slower and more fragile (due to the reliance on something outside of the current code base). They do have a lot of value though, because beyond application failures, they inform you when your integration with the external dependency breaks for whatever reason.
The ELB Logs Processor features both types of integration test, which is nice, because it makes for a comprehensive blog post.
The simpler integration test is the one for the component that integrates with S3 to download a file and expose it as a series of lines. All you need to do is put a file in a real S3 bucket, then run the class with an appropriate set of parameters to find that file, download it and expose it in the format that is desired. You’re free to simulate missing files (i.e. go get this file that doesn’t exist) and other error cases as desired as well, which is good.
The other integration test involves the Handler, using its default configuration and a realistic input. The idea would be to run the Handler in a way that is as close to how it will be run in reality as possible, and validate that it does what it advertises. This means we need to put some sort of sample file in S3, fire an event at the Handler describing that file (i.e. the same event that it would receive during a normal trigger in AWS), validate that the Handler ran as expected (perhaps measuring log output or something) and then validate that the events from the sample file were available in the destination ELK stack. A complicated test for sure, but one with lots of value.
As is always the case, while both of those tests are conceptually fairly simple and easy to spell out, they both have to surmount the same challenge.
AWS Authentication and Authorization.
401, Who Are You?
The various libraries supplied by AWS (Javascript, Powershell, .NET, Python, etc) all handle credential management in a similar way.
They use a sort of priority system for determining the source of their credentials, and it looks something like this:
- If credentials have been supplied directly in the code (i.e. key/secret), use those
- Else, if a profile has been supplied, look for that and use it
- Else, if no profile has been supplied, use the default profile
- Else, if environment variables have been set, use those
- Else, use information extracted from the local machine (i.e. IAM profiles or EC2 meta information)
The specific flow differs from language to language (.NET specific rules can be found here for example), but the general flow is pretty consistent.
What does that mean for our tests?
Well, the actual Lambda function code doesn’t need credentials at all, because the easiest way to run it is with an IAM profile applied to the Lambda function itself in AWS. Assuming no other credentials are available, this kicks in at option 5 in the list above, and is by far the least painful (and most secure) option to use.
When running the tests during development, or running them on the build server, there is no ambient IAM profile available, so we need to find some other way to get credentials to the code.
Luckily, the .NET AWS library allows you to override the logic that I described above, which is a pretty great (even though it does it with static state, which makes me sad).
With a little work we can nominate an AWS profile with a specific name to be our current “context”, and then use those credentials to both setup a temporary S3 bucket with a file in it, and to run the code being tested, like in the following Handler test.
/// <summary>
/// This test is reliant on an AWS profile being setup with the name "elb-processor-s3-test" that has permissions
/// to create and delete S3 buckets with prefixes oth.console.elb.processor.tests. and read items in those buckets
/// </summary>
[Fact]
public void WhenHandlingAValidS3Event_ConnectsToS3ToDownloadTheFile_AndOutputsEventsToLogstash()
{
var inMemoryLogger = new InMemoryLogger();
var xunitLogger = new XUnitLogger(_output);
var logger = new CompositeLogger(inMemoryLogger, xunitLogger);
var application = Guid.NewGuid().ToString();
var config = new Dictionary<string, string>
{
{ Configuration.Keys.Event_Environment, Guid.NewGuid().ToString() },
{ Configuration.Keys.Event_Application, application},
{ Configuration.Keys.Event_Component, Guid.NewGuid().ToString() },
{ Configuration.Keys.Logstash_Url, "{logstash-url}" }
};
using (AWSProfileContext.New("elb-processor-s3-test"))
{
logger.Log("Creating test bucket and uploading test file");
var s3 = new AmazonS3Client(new AmazonS3Config { RegionEndpoint = RegionEndpoint.APSoutheast2 });
var testBucketManager = new TestS3BucketManager(s3, "oth.console.elb.processor.tests.");
using (var bucket = testBucketManager.Make())
{
var templateFile = Path.Combine(ApplicationEnvironment.ApplicationBasePath, @"Helpers\Data\large-sample.log");
var altered = File.ReadAllText(templateFile).Replace("@@TIMESTAMP", DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ss.ffffffZ"));
var stream = new MemoryStream(Encoding.UTF8.GetBytes(altered));
var putResult = s3.PutObjectAsync(new PutObjectRequest { BucketName = bucket.Name, InputStream = stream, Key = "test-file" }).Result;
var message = new S3EventBuilder().ForS3Object(bucket.Name, "test-file").Build();
logger.Log("Executing ELB logs processor");
var handler = new Handler(logger, config);
handler.Handle(message);
}
}
// Check that there are some events in Elasticsearch with our Application
Func<long> query = () =>
{
var client = new HttpClient {BaseAddress = new Uri("{elasticsearch-url}")};
var raw = client.GetStringAsync($"/logstash-*/_search?q=Application:{application}").Result;
dynamic result = JsonConvert.DeserializeObject(raw);
return (long) result.hits.total;
};
query.ResultShouldEventuallyBe(hits => hits.Should().BeGreaterThan(0, $"because there should be some documents in Elasticsearch with Application:{application}"));
}The code for the AWSProfileContext class is relatively simple, in that while it is in effect it inserts the profile into the first position of the credential fallback process, so that it takes priority over everything else. When disposed, it removes that override, returning everything to the way it was. A similar dispose-delete pattern is used for the temporary S3 bucket.
With a profile override in place, you can run the tests in local development just by ensuring you have a profile available with the correct name, which allows for a nice smooth development experience once a small amount of manual setup has been completed.
To run the tests on the build servers, we put together a simple helper in Powershell that creates an AWS profile, runs a script block and then removes the profile. The sensitive parts of this process (i.e. the credentials themselves) are stored in TeamCity, and supplied when the script is run, so nothing dangerous needs to go into source control at all.
To Be Continued
The most complicated part of testing the ELB Logs Processor was getting the AWS credentials working consistently across a few different execution strategies, while not compromising the safety of the credentials themselves. The rest of the testing is pretty just vanilla C# testing.
The only other thing worth mentioning is that the AWS libraries available for .NET Core aren’t quite the same as the normal .NET libraries, so some bits and pieces didn’t work as expected. The biggest issue I ran into was the how a lot of the functionality for managing credential is typically configured through the app.config, and that sort of thing doesn’t even seem to exist in .NET Core. Regardless, we still got it work in the end, even though we did have to use the obsolete StoredProfileAWSCredentials class.
Next week will be the last post in this series, drawing everything together and focusing on actually running the new .NET Core ELB Logs Processor in a production environment.