We’ve spent a significant amount of effort recently ensuring that our software components are automatically built and deployed. Its not something new, and its certainly something that some of our components already had in place, but nothing was ever made generic enough to reuse. The weak spot in our build/deploy pipeline is definitely tests though. We’ve had a few attempts in the past to get test automation happening as part of the build, and while it has worked on an individual component basis, we’ve never really taken a holistic look at the process and made it easy to apply to a range of components.

I’ve mentioned this before but to me tests fall into 3 categories, Unit, Integration and Functional. Unit tests cover the smallest piece of functionality, usually algorithms or classes with all dependencies stubbed or mocked out. Integration tests cover whether all of the bits are configured to work together properly, and can be used to verify features in a controlled environment. Functional tests cover the application from a feature point of view. For example, functional tests for a web service would be run on it after it is deployed, verifying users can interact with it as expected.

From my point of view, the ideal flow is as follows:

Checkin – Build – Unit and Integration Tests – Deploy (CI) – Functional Tests – Deploy (Staging)

Obviously I’m talking about web components here (sites, services, etc), but you could definitely apply it to any component if you tried hard enough.

The nice part of this flow is that you can do any manual testing/exploration/early integration on the Staging environment, with the guarantee that it will probably not be broken by a bad deploy (because the functional tests will protect against that and prevent the promotion to staging).

Aren’t All Cities Full of Teams

We use Team City as our build platform and Octopus as our deployment platform, and thanks to these components we have the checkin, build and deployment parts of the pipeline pretty much taken care of.

My only issue with these products is that they are so configurable and powerful that people often use them to store complex build/deployment logic. This makes me sad, because that logic belongs as close to the code as possible, ideally in the same repository. I think you should be able to grab a repository and build it, without having the use an external tool to put all the pieces together. Its also an issue if you need to change your build logic, but still allow for older builds (maybe a hotfix branch or something). If you stored your build logic in source control, then this situation just works, because the logic is right there with the code.

So I mostly use Team City to trigger builds and collect history about previous builds (and their output), which it does a fine job at. Extending that thought I use Octopus to manage environments and machines, but all the logic for how to install a component lives in the deployable package (which can be built with minimal fuss from the repository).

I do have to mention that these tools do have elements of change control, and do allow you to version your Build Configurations (TeamCity)/Projects (Octopus). I just prefer that this logic lives with the source, because then the same version is applied to everything.

All of our build and deployment logic lives in source control, right next to the code. There is a single powershell script (unsurprisingly called build.ps1) per repository, acting as the entry point. The build script in each repository is fairly lightweight, leveraging a set of common scripts downloaded from our Nuget server, to avoid duplicating logic.

Team City calls this build script with some appropriate parameters, and it takes care of the rest.

Testy Testy Test Test

Until recently, our generic build script didn’t automatically execute tests, which was an obvious weakness. Being that we are in the process of setting up a brand new service, I thought this would be the ideal time to fix that.

To tie in with the types of tests I mentioned above, we generally have 2 projects that live in the same solution as the main body of code (X.Tests.Unit and X.Tests.Integration, where X is the component name), and then another project that lives in parallel called X.Tests.Functional. The Functional tests project is kind of a new thing that we’re trying out, so is still very much in flux. The other two projects are well accepted at this point, and consistently applied.

Both Unit and Integration tests are written using NUnit. We went with NUnit over MSTEST for reasons that seemed valid at the time, but which I can no longer recall with any level of clarity. I think it might have been something about the support for data driven tests, or the ability to easily execute the tests from the command line? MSTEST offers both of those things though, so I’m honestly not sure. I’m sure we had valid reasons though.

The good thing about NUnit, is that the NUnit Runner is a NuGet package of its own, which fits nicely into our dependency management strategy. We’ve written powershell scripts to manage external components (like Nuget, 7Zip, Octopus Command Line Tools, etc) and the general pattern I’ve been using is to introduce a Functions-Y.ps1 file into our CommonDeploymentScripts package, where Y is the name of the external component. This powershell file contains functions that we need from the external component (for example for Nuget it would be Restore, Install, etc) and also manages downloading the dependent package and getting a reference to the appropriate executable.

This approach has worked fairly well up to this point, so my plan was to use the same pattern for test execution. I’d need to implement functions to download and get a reference to the NUnit runner, as well as expose something to run the tests as appropriate. I didn’t only require a reference to NUnit though, as we also use OpenCover (and ReportGenerator) to get code coverage results when running the NUnit tests. Slightly more complicated, but really just another dependency to manage just like NUnit.

Weirdly Smooth

In a rare twist of fate, I didn’t actually encounter any major issues implementing the functions for running tests. I was surprised, as I always run into some crazy thing that saps my time and will to live. It was nice to have something work as intended, but it was probably primarily because this was a refactor of existing functionality. We already had the script that ran the tests and got the coverage metrics, I was just restructuring it and moving it into a place where it could be easily reused.

I wrote some very rudimentary tests to verify that the automatic downloading of the dependencies was working, and then set to work incorporating the execution of the tests into our build scripts.

function FindAndExecuteNUnitTests
{
    [CmdletBinding()]
    param
    (
        [System.IO.DirectoryInfo]$searchRoot,
        [System.IO.DirectoryInfo]$buildOutput
    )

    Write-Host "##teamcity[blockOpened name='Unit and Integration Tests']"

    if ($rootDirectory -eq $null) { throw "rootDirectory script scoped variable not set. Thats bad, its used to find dependencies." }
    $rootDirectoryPath = $rootDirectory.FullName

    . "$rootDirectoryPath\scripts\common\Functions-Enumerables.ps1"
    . "$rootDirectoryPath\scripts\common\Functions-OpenCover.ps1"

    $testAssemblySearchPredicate = { 
            $_.FullName -like "*release*" -and 
            $_.FullName -notlike "*obj*" -and
            (
                $_.Name -like "*integration*" -or 
                $_.Name -like "*unit*"
            )
        }
    Write-Verbose "Locating test assemblies using predicate [$testAssemblySearchPredicate]."
    $testLibraries = Get-ChildItem -File -Path $srcDirectoryPath -Recurse -Filter "*.Test*.dll" |
        Where $testAssemblySearchPredicate
            
    $failingTestCount = 0
    foreach ($testLibrary in $testLibraries)
    {
        $testSuiteName = $testLibrary.Name
        Write-Host "##teamcity[testSuiteStarted name='$testSuiteName']"
        $result = OpenCover-ExecuteTests $testLibrary
        $failingTestCount += $result.NumberOfFailingTests
        $newResultsPath = "$($buildDirectory.FullName)\$($result.LibraryName).TestResults.xml"
        Copy-Item $result.TestResultsFile "$newResultsPath"
        Write-Host "##teamcity[importData type='nunit' path='$newResultsPath']"

        Copy-Item $result.CoverageResultsDirectory "$($buildDirectory.FullName)\$($result.LibraryName).CodeCoverageReport" -Recurse

        Write-Host "##teamcity[testSuiteFinished name='$testSuiteName']"
    }

    write-host "##teamcity[publishArtifacts '$($buildDirectory.FullName)']"
    Write-Host "##teamcity[blockClosed name='Unit and Integration Tests']"

    if ($failingTestCount -gt 0)
    {
        throw "[$failingTestCount] Failing Tests. Aborting Build."
    }
}

As you can see, its fairly straightforward. After a successful build the source directory is searched for all DLLs with Tests in their name, that also appear in the release directory and are also named with either Unit or Integration. These DLLs are then looped through, and the tests executed on each one (using the OpenCover-ExecuteTests function from the Functions-OpenCover.ps1 file), with the results being added to the build output directory. A record of the number of failing tests is kept and if we get to the end with any failing tests an exception is thrown, which is intended to prevent the deployment of faulty code.

The build script that I extracted the excerpt above from lives inside our CommonDeploymentScripts package, which I have replicated into this Github repository.

I also took this opportunity to write some tests to verify that the build script was working as expected. In order to do that, I had to create a few dummy Visual Studio projects (one for a deployable component via Octopack and another for a simple library component). At the start of each test, these dummy projects are copied to a working directory, and then mutated as necessary in order to provide the appropriate situation that the test needs to verify.

The best example of this is the following test:

Describe {
    Context "When deployable component with failing tests supplied and valid deploy" {
        It "An exception is thrown indicating build failure" {
            $creds = Get-OctopusCredentials

            $testDirectoryPath = Get-UniqueTestWorkingDirectory
            $newSourceDirectoryPath = "$testDirectoryPath\src"
            $newBuildOutputDirectoryPath = "$testDirectoryPath\build-output"

            $referenceDirectoryPath = "$rootDirectoryPath\src\TestDeployableComponent"
            Copy-Item $referenceDirectoryPath $testDirectoryPath -Recurse

            MakeTestsFail $testDirectoryPath
            
            $project = "TEST_DeployableComponent"
            $environment = "CI"
            try
            {
                $result = Build-DeployableComponent -deploy -environment $environment -OctopusServerUrl $creds.Url -OctopusServerApiKey $creds.ApiKey -projects @($project) -DI_sourceDirectory { return $testDirectoryPath } -DI_buildOutputDirectory { return $newBuildOutputDirectoryPath }
            }
            catch 
            {
                $exception = $_
            }

            $exception | Should Not Be $null

            . "$rootDirectoryPath\scripts\common\Functions-OctopusDeploy.ps1"

            $projectRelease = Get-LastReleaseToEnvironment -ProjectName $project -EnvironmentName $environment -OctopusServerUrl $creds.Url -OctopusApiKey $creds.ApiKey
            $projectRelease | Should Not Be $result.VersionInformation.New
        }
    }
}

As you can see, there is a step in this test to make the dummy tests fail. All this does is rewrite one of the classes to return a different value than is expected, but its enough to fail the tests in the solution. By doing this, we can verify that yes a failing does in fact lead to no deployment.

Summary

Nothing that I’ve said or done above is particularly ground-breaking. Its all very familiar to anyone who is doing continuous integration/deployment. Having tests is fantastic, but unless they take part in your build/deploy pipeline they are almost useless. That’s probably a bit harsh, but if you can deploy code without running the tests on it, you will (with the best of intentions no doubt) and that doesn’t lead anywhere good.

Our approach doesn’t leverage the power of TeamCity directly, due to my reluctance to store complex logic there. There are upsides and downsides to this, mostly that you trade off owning the implementation of the test execution against keeping all your logic in one place.

Obviously I prefer the second approach, but your mileage may vary.

Like me, I assume you get into a situation sometimes where you want to execute a script, but you definitely don’t want some of its more permanent side effects to happen. Scripts can do all sorts of crazy things, like commit files into git, make changes to the local machine or publish files into production and you definitely don’t want those things to happen when you’re not intending them to.

This becomes even more important when you want to start writing tests for your scripts (or at least their components, like functions). You definitely don’t want the execution of your tests to change something permanently, especially if it changes the behaviour of the system under test, because then the next time you run the test its giving you a different result or executing a different code path. All things you want to avoid to get high quality tests.

In my explorations of the issue, I have come across two solutions. One helps to deal with side effects during tests and the other gives you greater control over your scripts, allowing you to develop them with some security that you are not making unintended changes.

Testing the Waters

I’ve recently started using Pester to test my Powershell functions.

The lack of testing (other than manual testing of course) in my script development process was causing me intense discomfort, coming from a strong background where I always wrote tests (before, after, during) whenever I was developing a feature in C#.

Its definitely improved my ability to write Powershell components quickly and in a robust way, and has improved my ability to refactor, safe in the knowledge that if I mess it up (and in a dynamic scripting language like Powershell with no Visual Studio calibre IDE, you will mess it up) the tests will at least catch the most boneheaded mistakes. Maybe even the more subtle mistakes too, if you write good tests.

Alas, I haven’t quite managed to get a handle on how to accomplish dependency injection in Powershell, but I have some ideas that may turn up in a future blog post. Or they may not, because it might be a terrible idea, only time will tell.

To tie this apparently pointless and unrelated topic back into the blog post, sometimes you need a way to control the results returned from some external call or to make sure that some external call doesn’t actually do anything. Luckily, Powershell being a dynamic language, you can simply overwrite a function definition in the scope of your test. I think.

I had to execute a Powershell script from within TestComplete recently, and was surprised when my trusty calls to write-host (for various informational logging messages) would throw errors when the script was executed via the Powershell object in System.Management.Automation. The behaviour makes perfect sense when you think about it, as that particular implementation of a host environment simply does not provide a mechanism to output anything not through the normal streams. I mention this because it was a problem that I managed to solve (albeit in a hacky way) by providing a blank implementation of write-host in the same session as my script, effectively overriding the implementation that was throwing errors.

Pester provides a mechanism for doing just this, through the use of Mocks.

I’d love to write a detailed example of how to use Mocks in Pester here, but to be honest, I haven’t had the need as of yet (like I said, I’ve only very recently started using Pester). Luckily the Pester wiki is pretty great, so there’s enough information there if you want to have a read.

I’m very familiar with mocking as a concept though, as I use it all the time in my C# tests. I personally am a fan of NSubstitute, but I’ve used Moq as well.

The only point I will make is that without dependency injection advertising what your components dependencies are, you have to be aware of its internal implementation in order to Mock out its dependencies. This makes me a little bit uncomfortable, but still, being able to Mock those dependencies out instead of having them hardcoded is much preferred.

Zhu Li, Do the Thing (But Not the Other Thing)

The second approach that I mentioned is a concept that is built into Powershell, which I have stolen and bastardised for my own personal gain.

A lot of the pre-installed Powershell components allow you to execute the script in –WhatIf mode.

WhatIf mode essentially runs the script as normal, except it doesn’t allow it to actually make any permanent changes. It’s up to the component exactly what it considers to be permanent changes, but its typically things like changing system settings and interacting with the file system. Instead it just writes out messages to whatever appropriate stream stating the action that would have normally occurred. I imagine that depending on how your component is written it might not react well to files it asks to be created or deleted not occurring as expected, but its still an interesting feature all the same.

In my case, I had a build and publish script that changed the AssemblyInfo of a C# project and then committed those changes to git, as well as tagging git with a build number when the publish completed successfully. I had to debug some issues with the script recently, and I wanted to run it without any of those more permanent changes happening.

This is where I leveraged the –WhatIf switch, even though I used it in a slightly different way, and didn’t propagate the switch down to any components being used by my script. Those components were mostly non-powershell, so it wouldn’t have helped anyway (things like git, MSBuild and robocopy).

I used the switch to turn off the various bits that made more permanent changes, and to instead output a message through the host describing the action that would have occurred. I left in the parts that made permanent changes to the file system (i.e. the files output from MSbuild) because those don’t have any impact on the rest of the system.

Of course you still need to test the script as a whole, which is why we have a fully fledged development environment that we can freely publish to as much as we like, but its still nice to execute the script safe in the knowledge that it’s not going to commit something to git.

I’ve found the WhatIf approach to be very effective, but it relies entirely on the author of the script to select the bits that they thing are permanent system changes and distinguish them from ones that are not as permanent (or at least easier to deal with, like creating new files). Without a certain level of analysis and thinking ahead, the approach obviously doesn’t work.

I’ve even considered defaulting WhatIf to on, to ensure that its a conscious effort to make permanent changes, just as a safety mechanism, mostly to protect future me from running the script in a stupid way.

Summary

When programming, its important to be aware of and to limit any side effects of the code that you have written, both for testing and development. The same holds true of scripts. The complication here is that scripts tend to bundle up lots of changes to the system being acted upon as their entire purpose, so you have to be careful with selecting which effects you want to minimise while developing.

In other news, I’ve been working on setting up a walking skeleton for a new service that my team is writing. Walking skeleton is a term referenced in Growing Object Oriented Software, Guided By Tests, and describes writing the smallest piece of functionality possible first, and then ensuring that the entire build and deployment process is created and working before doing anything else.

I suspect I will make a series of blog posts about that particular adventure.

Spoiler alert, it involves AWS and Powershell.

I’ve been doing a lot of Test Automation lately. In a weird co-incidence, a lot of swearing and gnashing of teeth as well. Strange that.

One of the automation tools that I’ve been using is TestComplete, and I’ve been pleasantly surprised to find that it doesn’t get in my way nearly as much as I would have expected. In comparison, I’ve had far more problems with the behaviour of the application under test rather than the execution environment, which says a lot for TestComplete’s ability to interact with horrifying VB6/WinForms chimera applications.

Its not all puppies and roses though. TestComplete is very much a tabs and text boxes kind of application and while parts of the tool have some nice Intellisense (more than I would have expected) its certainly no Visual Studio. In addition, binary files make meaningful diffs impossible, its licensing model is draconian at best and the price is insane.

But aside from those gripes, it certainly does get the job done.

When I decided that in order to test a piece of functionality involving the database, I would need to leverage some AWS EC2 images, I was unsurprised to find that it had support for executing Powershell scripts. I seems to be able to do just about anything really.

Well, I suppose support is a strong word.

Scripted Interactions

TestComplete supports a number of scripting languages for programmatically writing tests. There is also a very useful record-replay GUI which actually creates a programmatically accessible map of your application, which can then be used in scripts if you want.

When you create a TestComplete project, you can choose from VBScript, JScript and two other scripting varieties that slip my mind. After you’ve created the project, you don’t seem to be able to change this choice, so make sure you pick one you want to use.

I did not create the project.

The person who did picked JScript, which I have never written before. But at least it wasn't VBScript I suppose.

My understanding of JScript is limited, but I believe it is a Microsoft implementation of some Javascript standard. I have no idea which one, and its certainly not recent.

Anyway, I’m still not entirely sure how it all fits together, but from the JScript code you can access some of the .NET Framework libraries, like System.Management.Automation.

Using this namespace, you can create an object that represents a Powershell process, configure it with some commands, and then execute it.

Hence Powershell support.

I Have the Power

I’ve written about my growing usage of Powershell before, but I’ve found it to be a fantastically powerful tool. It really can do just about anything. In this case, I gravitated towards it because I’d already written a bunch of scripts for initialising AWS EC2 instances, and I wanted to reuse them to decrease the time it would take me to put together this functional test. For this test I would need to create an AMI containing a database in the state that I wanted it to be in, and then spin up instances of that AMI whenever I wanted to run the test.

In the end, as it is with these sorts of things, it wasn’t setting up the EC2 instances that took all of my time. That yak had already been shaved.

The first problem I had to solve was actually being able to run a Powershell script file using the component above.

Anyone who has ever used Powershell is probably very familiar with the Execution Policy of the system in question. Typically this defaults to Restricted, which means you are not allowed to execute scripts.

Luckily you can override this purely for the process you are about to execute, without impacting on the system at large. This, of course, means that you can change the Execution Policy without needing Administrative privileges.

Set-ExecutionPolicy RemoteSigned -Scope Process -Force

The second problem I had to solve was getting results out of the Powershell component in TestComplete. In my case I needed to get the Instance ID that was returned by the script, and then store it in a TestComplete variable for use later on. I’d need to do the same thing with the IP address, obtained after waiting for the instance to be ready.

Waiting for an instance to be ready is actually kind of annoying. There are two things you need to check. The first is whether or not the instance is considered running. The second is whether or not the instance is actually contactable through its network interface. Waiting for an instance to be running only takes about 15-30 seconds. Waiting for an instance to be contactable takes 5-10 minutes depending on how capricious AWS is feeling. As you can imagine, this can make testing your “automatically spin up an instance” script a very frustrating experience. Long feedback loops suck.

When you execute a series of commands through the Powershell component, the return value is a collection of PSObjects. These objects are what would have normally been returned via the output stream. In my case I was returning a single string, so I needed to get the first entry in the collection, and then get its ImmediateBaseObject property. To get the string value I then had to get the OleValue property.

Tying both of the above comments together, here is the collection of functions that I created to launch Powershell from TestComplete.

function CreatePowershellExecutionObject()
{
  var powershell = dotNET.System_Management_Automation.Powershell.Create();
  // This is set before every script because otherwise we can't execute script files,
  // Thats where most of our powershell lives.
  powershell.AddScript("Set-ExecutionPolicy RemoteSigned -Scope Process -Force");
  
  // This redirects the write-host function to nothingness, because otherwise any
  // script executed through this componenent will fail if it has the audacity to
  // write-host. This is because its non-interactive and write-host is like a special
  // host channel thing that needs to be implemented.
  powershell.AddScript("function write-host { }");
  
  return powershell;
}

function InvokePowershellAndThrowIfErrors(powershell)
{
  var result = powershell.Invoke();
  
  if (powershell.HadErrors)
  {
    var firstError = powershell.Streams.Error.Item(0);
    if (firstError.ErrorDetails != null)
    {
      throw new Error(firstError.ErrorDetails);
    }
    else
    {
      throw new Error(firstError.Exception.ToString());
    }
  }
  
  return result;
}

function GetCommonPowershellAwsScriptArguments()
{
  var awsKey = Project.Variables.AwsKey;
  var awsSecret = Project.Variables.AwsSecret;
  var awsRegion = Project.Variables.AwsRegion;
  return "-SuppliedAwsKey " + awsKey + " -SuppliedAwsSecret " + awsSecret + " -SuppliedAwsRegion " + awsRegion + " "
}

function StartAwsVirtualMachineWithFullDatabaseAndSetInstanceId()
{
  var gatewayVersion = GetVersionOfGatewayInstalled();
  var scriptsPath = GetScriptsDirectoryPath();
  var executeInstanceCreationScript = "& \"" + 
    scriptsPath + 
    "\\functional-tests\\create-new-max-size-db-ec2-instance.ps1\" " + 
    GetCommonPowershellAwsScriptArguments() +
    "-BuildIdentifier " + gatewayVersion;

  var powershell = CreatePowershellExecutionObject();
  powershell.AddScript(executeInstanceCreationScript);
  
  Indicator.PushText("Spinning up AWS EC2 Instance containing Test Database");
  var result = InvokePowershellAndThrowIfErrors(powershell);
  Indicator.PopText();
  
  var instanceId = result.Item(0).ImmediateBaseObject.OleValue; 
  
  KeywordTests.WhenADatabaseIsAtMaximumSizeForExpress_ThenTheFilestreamConversionStillWorks.Variables.FullDbEc2InstanceId = instanceId;
}

Notice that I split the instance creation from the waiting for the instance to be ready. This is an optimisation. I create the instance right at the start of the functional test suite, and then execute other tests while that instance is being setup in AWS. By the time I get to it, I don’t have to wait for it to be ready at all. Less useful when testing the database dependent test by itself, but it shaves 6+ minutes off the test suite when run together. Every little bit counts.

Application Shmaplication

Now that the test database was being setup as per my wishes, it was a simple matter to record the actions I wanted to take in the application and make some checkpoints for verification purposes.

Checkpoints in TestComplete are basically asserts. Record a value (of many varied types, ranging from Onscreen Object property values to images) and then check that the application matches those values when it is run.

After recording the test steps, I broke them down into reusable components (as is good practice) and made sure the script was robust in the face of failures and unexpected windows (and other things).

The steps for the executing the test using the application itself were easy enough, thanks to TestComplete.

Tidbits

I did encounter a few other things while setting this test up that I think are worth mentioning.

The first was that the test needed to be able to be run from inside our VPC (Virtual Private Cloud) in AWS as well as from our local development machines. Actually running the test was already a solved problem (solved when I automated the execution of the functional tests late last year), but making a connection to the virtual machine hosting the database in AWS was a new problem.

Our AWS VPC is fairly locked down (for good reason) so machines in there generally can’t access machines in the outside word except over a few authorised channels (HTTP and HTTPS through a proxy for example). Even though the database machine was sitting in the same VPC as the functional tests worker, I had planned to only access it through its public IP address (for simplicity). This wouldn’t work without additional changes to our security model, which would have been a pain (I have no control over those policies).

This meant that in one case I needed to use the Public IP Address of the instance (when it was being run from our development machines) and in the other I needed to use the Private IP Address.

Code to select the correct IP address fit nicely into my Powershell script to wait for the instance to be ready, which already returned an IP address. All I had to do was test the public IP over a specific port and depending on whether or not it worked, return the appropriate value. I did this using the TCPClient class in the .NET framework.

The second thing I had to deal with was a dependency change.

Previously our TestComplete project was not dependent on anything except itself. It could be very easily compressed into a single file and tossed around the internet as necessary. Now that I had added a dependency on a series of Powershell scripts I had to change the execution script for our Functional Tests to pack up and distribute additional directories. Nothing too painful luckily, as it was as simple enough matter to include more things in the compressed archive.

The final problem I ran into was with the application under test.

Part of the automated Functional Test is to open the database. When you ask the application to do that, it will pop up a fairly standard dialog with server & database name.

Being the helpful VB6 application that it is, it also does a search of the network to find any database servers that are available, so you can quickly select them from a dropdown. Being a single threaded application without a great amount of thought being put into user experience, the dialog freezes for a few moments while it does the search.

If you try to find that Window or any of its components using TestComplete while its frozen, the mapping for the dialog box changes from what you would expected (Aliases.Application.DialogName) to something completely different (Process(“ApplicationName”).Window(“DialogName”).Control(“Name”)). Since the test was recorded using the first alias, it then times out when looking for the control it expects, and fails.

I got around this by introducing a delay before even attempting to look for any controls on that dialog.

Depressingly, this is a common solution to solving automation problems like that.

Conclusion

If you were expecting this section to be where I tie everything together and impart on you some TL;DR lessons, then prepare yourself for disappointment.

The one thing I will say though is that setting up good automated Functional Tests takes a hell of a lot of effort. it gets easier as you spend more time doing it, but sometimes I question the benefits. Certainly its an interesting exercise and you learn a lot about the system under test, but the tests you create are usually fragile and overly complex.

Functional Tests that automate an application definitely shouldn’t be your only variety of test that’s for sure.

Regardless of the adventure above, TestComplete is pretty great, and it certainly makes the whole automated testing process much easier than it would be if you used something else.

Like CodedUI.

Which I have done before.

I don’t recommend it.

So, we decided to automate the execution of our Functional tests. After all, tests that aren't being run are effectively worthless.

In Part 1, I gave a little background, mentioned the scripting language we were going to use (Powershell), talked about programmatically spinning up the appropriate virtual machine in AWS for testing purposes and then how to communicate with it.

This time I will be talking about automating the installation of the software under test, running the actual tests and then reporting the results.

Just like last time, here is a link to the GitHub repository with sanitized versions of all the scripts (which have been updated significantly), so you don't have to piece them together from the snippets I’m posting throughout this post.

Installed into Power

Now that I could create an appropriate virtual machine as necessary and execute arbitrary code on it remotely, I needed to install the actual software under test.

First I needed to get the installer onto the machine though, as we do not make all of our build artefacts publically available on every single build (so no just downloading it from the internet). Essentially I just needed a mechanism to transfer files from one machine to another. Something that wouldn’t be too difficult to setup and maintain.

I tossed up a number of options:

• FTP. I could setup an FTP server on the virtual machine and transfer the files that way. The downside of this is that I would have to setup an FTP server on the virtual machine, and make sure it was secure; and configured correctly. I haven’t setup a lot of FTP servers before, so I decided not to do this one.
• SSH + File Transfer. Similar to the FTP option, I could install an SSH server on the virtual machine and then use something like SCP to securely copy the files to the machine. This would have been the easiest option if the machine was Linux based, but being a Windows machine it was more effort than it was worth.
• Use an intermediary location, like an Amazon S3 bucket. This is the option I ended up going with.

Programmatically copying files to an Amazon S3 bucket using Powershell is fairly straightforward, although I did run into two issues.

Folders? What Folders?

Even though its common for GUI tools that sit on top of Amazon S3 to present the information as a familiar folder/file directory structure, that is entirely not how it actually works. In fact, thinking of the information that you put in S3 in that way will just get you into trouble.

Instead, its much more accurate to think of the information you upload to S3 to be key/value pairs, where the key tends to look like a fully qualified file path.

I made an interesting error at one point and uploaded 3 things to S3 with the following keys, X, X\Y and X\Z. The S3 website interpreted the X as a folder, which meant that I was no longer able to access the file that I had actually stored at X, at least through the GUI anyway. This is one example of why thinking about S3 as folders/files can get you in trouble.

Actually uploading files to S3 using Powershell is easy enough. Amazon supply a set of cmdlets that allow you to interact with S3, and those cmdlets are pre-installed on machines originally created using an Amazon supplied AMI.

With regards to credentials, you can choose to store the credentials in configuration, allowing you to avoid having to enter them for every call, or you can supply then whenever you call the cmdlets. Because this was a script, I chose to supply them on each call, so that the script would be self contained. I’m not a fan of global settings in general, they make me uncomfortable. I feel that it makes the code harder to understand, and in this case, it would have obfuscated how the cmdlets were authenticating to the service.

The function that uploads things to S3 is as follows:

function UploadFileToS3
{
    param
    (
        [string]$awsKey,
        [string]$awsSecret,
        [string]$awsRegion,
        [string]$awsBucket,
        [System.IO.FileInfo]$file,
        [string]$S3FileKey
    )

    write-host "Uploading [$($file.FullName)] to [$($awsRegion):$($awsBucket):$S3FileKey]."
    Write-S3Object -BucketName $awsBucket -Key $S3FileKey -File "$($file.FullName)" -Region $awsRegion -AccessKey $awsKey -SecretKey $awsSecret

    return $S3FileKey
}

The function that downloads things is:

function DownloadFileFromS3
{
    param
    (
        [string]$awsKey,
        [string]$awsSecret,
        [string]$awsRegion,
        [string]$awsBucket,
        [string]$S3FileKey,
        [string]$destinationPath
    )

    $destinationFile = new-object System.IO.FileInfo($destinationPath)
    if ($destinationFile.Exists)
    {
        write-host "Destination for S3 download of [$S3FileKey] ([$($destinationFile.FullName)]) already exists. Deleting."
        $destinationFile.Delete()
    }

    write-host "Downloading [$($awsRegion):$($awsBucket):$S3FileKey] to [$($destinationFile.FullName)]."
    Read-S3Object -BucketName $awsBucket -Key $S3FileKey -File "$($destinationFile.FullName)" -Region $awsRegion -AccessKey $awsKey -SecretKey $awsSecret | write-host

    $destinationFile.Refresh()

    return $destinationFile
}

Once the installer was downloaded on the virtual machine, it was straightforward to install it silently.

if (!$installerFile.Exists)
{
    throw "The Installer was supposed to be located at [$($installerFile.FullName)] but could not be found."
}

write-host "Installing Application (silently) from the installer [$($installerFile.FullName)]"
# Piping the results of the installer to the output stream forces it to wait until its done before continuing on
# with the remainder of the script. No useful output comes out of it anyway, all we really care about
# is the return code.
& "$($installerFile.FullName)" /exenoui /qn /norestart | write-host
if ($LASTEXITCODE -ne 0)
{
    throw "Failed to Install Application."
}

We use Advanced Installer, which in turn uses an MSI, so you’ll notice that there are actually a number of switches being used above to get the whole thing to install without human interaction. Also note the the piping to write-host, which ensures that Powershell actually waits for the installer process to finish, instead of just starting it and then continuing on its merry way. I would have piped to write-output, but then the uncontrolled information from the installer would go to the output stream and mess up my return value.

Permissions Denied

God. Damn. Permissions.

I had so much trouble with permissions on the files that I uploaded to S3. I actually had a point where I was able to upload files, but I couldn’t download them using the same credentials that I used to upload them! That makes no goddamn sense.

To frame the situation somewhat, I created a user within our AWS account specifically for all of the scripted interactions with the service. I then created a bucket to contain the temporary files that are uploaded and downloaded as part of the functional test execution.

The way that S3 defines permissions on buckets is a little strange in my opinion. I would expect to be able to define permissions on a per user basis for the bucket. Like, user X can read from and write to this bucket, user Y can only read, etc. I would also expect files that are uploaded to this bucket to then inherit those permissions, like a folder, unless I went out of my way to change them. That’s the mental model of permissions that I have, likely as a result of using Windows for many years.

This is not how it works.

Yes you can define permissions on a bucket, but not for users within your AWS account. It doesn’t even seem to be able to define permissions for other specific AWS accounts either. There’s a number of groups available, one of which is Authenticated Users, which I originally set with the understanding that it would give authenticated users belonging to my AWS account the permissions I specified. Plot twist, Authenticated Users means any AWS user. Ever. As long as they are authenticating. Obviously not what I wanted. At least I could upload files though (but not download them).

Permissions are not inherited when set through the simple options I mentioned above, so any file I uploaded had no permissions set on it.

The only way to set permissions with the granularity necessary and to have them inherited automatically is to use a Bucket Policy.

Setting up a Bucket Policy is not straight forward, at least to a novice like myself.

After some wrangling with the helper page, and some reading of the documentation, here is the bucket policy I ended up using, with details obscured to protect the guilty.

{
    "Version": "2008-10-17",
    "Id": "Policy1417656003414",
    "Statement": [
        {
            "Sid": "Stmt1417656002326",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::[USER IDENTIFIER]"
            },
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::[BUCKET NAME]/*"
        },
        {
            "Sid": "Stmt14176560023267",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::[USER IDENTIFIER]"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::[BUCKET NAME]"
        }
    ]
}

The policy actually reads okay once you have it, but I’ll be honest, I still don’t quite understand it. I know that I’ve specifically given the listed permissions on the contents of the bucket to my user, and also given List permissions on the bucket itself. This allowed me to upload and download files with the user I created, which is what I wanted. I’ll probably never need to touch Bucket Policies again, but if I do, I’ll make more of an effort to understand them.

Testing My Patience

Just like the installer, before I can run the tests I need to actually have the tests to run.

We currently use TestComplete as our Functional test framework. I honestly haven’t looked into TestComplete all that much, apart from just running the tests, but it seems to be solid enough. TestComplete stores your functional tests in a structure similar to a Visual Studio project, with a project file and a number of files under that that define the actual tests and the order to run them in.

For us, our functional tests are stored in the same Git repository as our code. We use feature branches, so it makes sense to run the functional tests that line up with the code that the build was made from. The Build Agent that builds the installer has access to the source (obviously), so its a fairly simple matter to just zip up the definitions and any dependent files, and push that archive to S3 in the exact same manner as the installer, ready for the virtual machine to download.

Actually running the tests though? That was a bastard.

As I mentioned in Part 1, after a virtual machine is spun up as part of this testing process, I use Powershell remote execution to push a script to the machine for execution.

As long as you don’t want to do anything with a user interface, this works fantastically.

Functional tests being based entirely around interaction with the user interface therefore prove problematic.

The Powershell remote session that is created when executing the script remotely does not have any ability for user-interactivity, and cannot to my knowledge. Its just something that Powershell can’t do.

However, you can remotely execute another tool, PSExec, and specify that it run a process locally in an interactive session.

$testExecute = 'C:\Program Files (x86)\SmartBear\TestExecute 10\Bin\TestExecute.exe'
$testExecuteProjectFolderPath = "$functionalTestDefinitionsDirectoryPath\Application"
$testExecuteProject = "$testExecuteProjectFolderPath\ApplicationTests.pjs"
$testExecuteResultsFilePath = "$functionalTestDefinitionsDirectoryPath\TestResults.mht"

write-host "Running tests at [$testExecuteProject] using TestExecute at [$testExecute]. Results going to [$testExecuteResultsFilePath]."
# Psexec does a really annoying thing where it writes information to STDERR, which Powershell detects as an error
# and then throws an exception. The 2>&1 redirects all STDERR to STDOUT to get around this.
# Bit of a dirty hack here. The -i 2 parameter executes the application in interactive mode specifying
# a pre-existing session with ID 2. This is the session that was setup by creating a remote desktop
# session before this script was executed. Sorry.
& "C:\Tools\sysinternals\psexec.exe" -accepteula -i 2 -h -u $remoteUser -p $remotePassword "$testExecute" "$testExecuteProject" /run /SilentMode /exit /DoNotShowLog /ExportLog:$testExecuteResultsFilePath 2>&1 | write-host
[int]$testExecuteExitCode = $LASTEXITCODE

The –i [NUMBER] in the command above tells PSExec to execute the process in an interactive user session, specifically the session with the ID specified. I’ve hardcoded mine to 2, which isn’t great, but works reliably in this environment because the remote desktop session I create after spinning up the instance always ends up with ID 2. Hacky.

Remote desktop session you may ask? In order for TestComplete (well TestExecute technically) to execute the tests correctly you need to actually have a desktop session setup. I assume this is related to it hooking into the UI user mouse and keyboard hooks or something, I don’t really know. All I know is that it didn't work without a remote desktop session of some sort.

On the upside, you can automate the creation of a remote desktop session with a little bit of effort, although there are two hard bits to be aware of.

Who Are You?

There is no obvious way to supply credentials to the Remote Desktop client (mstsc.exe). You can supply the machine that you want to make the connection to (thank god), but not credentials. I think there might be support for storing this information in an RDP file though, which seem to be fairly straightforward. As you can probably guess from my lack of knowledge about that particular approach, that’s not what I ended up doing.

I still don’t fully understand the solution, but you can use the built in windows utility cmdkey to store credentials for things. If you store credentials for the remote address that you are connecting to, the Remote Desktop client will happily use them.

There is one thing you need to be careful with when using this utility to automate Remote Desktop connections. If you clean up after yourself (by deleting the stored credentials after you use them) make sure you wait until the remote session is established. If you delete the credentials before the client actually uses them you will end up thinking that the stored credentials didn’t work, and waste a day investigating and trialling VNC solutions which ultimately don’t work as well as Remote Desktop before you realise the stupidity of your mistake. This totally happened to a friend of mine. Not me at all.

Anyway, the entirety of the remote desktop script (from start-remote-session.ps1):

param (
    [Parameter(Mandatory=$true,Position=0)]
    [Alias("CN")]
    [string]$ComputerNameOrIp,
    [Parameter(Mandatory=$true,Position=1)]
    [Alias("U")] 
    [string]$User,
    [Parameter(Mandatory=$true,Position=2)]
    [Alias("P")] 
    [string]$Password
)

& "$($env:SystemRoot)\system32\cmdkey.exe" /generic:$ComputerNameOrIp /user:$User /pass:$Password | write-host

$ProcessInfo = new-object System.Diagnostics.ProcessStartInfo

$ProcessInfo.FileName = "$($env:SystemRoot)\system32\mstsc.exe" 
$ProcessInfo.Arguments = "/v $ComputerNameOrIp"

$Process = new-object System.Diagnostics.Process
$Process.StartInfo = $ProcessInfo
$startResult = $Process.Start()

Start-Sleep -s 15

& "$($env:SystemRoot)\system32\cmdkey.exe" /delete:$ComputerNameOrIp | write-host

return $Process.Id

Do You Trust Me?

The second hard thing with the Remote Desktop client is that it will ask you if you trust the remote computer if you don’t have certificates setup. Now I would typically advise that you setup certificates for this sort of thing, especially if communicating over the internet, but in this case I was remoting into a machine that was isolated from the internet within the same Amazon virtual network, so it wasn’t necessary.

Typically, clicking “yes” on an identity verification dialog isn't a problem, even if it is annoying. Of course, in a fully automated environment, where I am only using remote desktop because I need a desktop to actually be rendered to run my tests, it’s yet another annoying thing I need to deal with without human interaction.

Luckily, you can use a registry script to disable the identity verification in Remote Desktop. This had to be done on the build agent instance (the component that actually executes the functional tests).

Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client]
    "AuthenticationLevelOverride"=dword:00000000

Report Soldier!

With the tests actually running reliably (after all the tricks and traps mentioned above), all that was left was to report the results to TeamCity.

It was trivial to simply get an exit code from TestExecute (0 = Good, 1 = Warnings, 2 = Tests Failed, 3 = Tests Didn’t Run). You can then use this exit code to indicate to TeamCity whether or not the tests succeeded.

if ($testResult -eq $null)
{
    throw "No result returned from remote execution."
}

if ($testResult.Code -ne 0)
{
    write-host "##teamcity[testFailed name='$teamCityFunctionalTestsId' message='TestExecute returned error code $($testResult.Code).' details='See artifacts for TestExecute result files']"
}
else
{
    write-host "##teamcity[testFinished name='$teamCityFunctionalTestsId'"
}

That's enough to pass or fail a build.

Of course, if you actually have failing functional tests you want a hell of a lot more information in order to find out whythey failed. Considering the virtual machine on which the tests were executed will have been terminated at the end of the test run, we needed to extract the maximum amount of information possible.

TestComplete (TestExecute) has two reporting mechanisms, not including the exit code from above.

The first is an output file, which you can specify when running the process. I chose an MHT file, which is a nice HTML document showing the tests that ran (and failed), and which has embedded screenshots taken on the failing steps. Very useful.

The second is the actual execution log, which is attached to the TestComplete project. This is a little harder to use, as you need to take the entire project and its log file and open it in TestComplete, but is great for in depth digging, as it gives a lot of information about which steps are failing and has screenshots as well.

Both of these components are zipped up on the functional tests worker and then placed into S3, so the original script can download them and attach them the TeamCity build artefacts. This is essentially the same process as for getting the test definitions and installer to the functional tests worker, but in reverse, so I won’t go into any more detail about it.

Summary

So, after all is said and done, I had automated:

• The creation of a virtual machine for testing.
• The installation of the latest build on that virtual machine.
• The execution of the functional tests.
• The reporting of the results, in both a programmatic (build pass/fail) and human readable way.

There were obviously other supporting bits and pieces around the core processes above, but there is little point in mentioning them here in the summary.

Conclusion

All up, I spent about 2 weeks of actual time on automating the functional tests.

A lot of the time was spent familiarising myself with a set of tools that I’d never (or barely) used before, like TestComplete (and TestExecute), the software under test, Powershell, programmatic access to Amazon EC2 and programmatic access to Amazon S3.

As with all things technical, I frequently ran into roadblocks and things not working the way that I would have expected them too out of the box. These things are vicious time sinks, involving scouring the internet for other people who’ve had the same issue and hoping to all that is holy that they solved their problem and then remembered to come back and share their solution.

Like all things involving software, I fought complexity every step of way. The two biggest offenders were the complexity in handling errors in Powershell in a robust way (so I could clean up my EC2 instances) and actually getting TestExecute to run the damn tests because of its interactivity requirements.

When all was said and done though, the functional tests are now an integral part of our build process, which means there is far more incentive to adding to them and maintaining them. I do have some concerns about their reliability (UI focused tests are a bit like that), but that can be improved over time.

In my last blog post, I mentioned the 3 classifications that I think tests fall into, Unit, Integration and Functional.

Of course, regardless of classification, all tests are only valuable if they are actually being executed. Its wonderful to say you have tests, but if you’re not running them all the time, and actually looking at the results, they are worthless. Worse than worthless if you think about it, because the presence of tests gives a false sense of security about your system.

Typically executing Unit Tests (and Integration Tests if they are using the same framework) is trivial, made vasty easier by having a build server. Its not that bad even if you don’t have a build server, because those sorts of tests can typically be run on a developers machine, without a huge amount of fanfare. The downside of not having a build server, is that the developers in question need to remember to run the tests. As creative people, following a checklist that includes “wait for tests to run” is sometimes not our strongest quality.

Note that I’m not saying developers should not be running tests on their own machines, because they definitely should be. I would usually limit this to Unit tests though, or very self-contained Integration tests. You need to be very careful about complicating the process of actually writing and committing code if you want to produce features and improvements in a reasonable amount of time. Its very helpful to encourage people to run the tests themselves regularly, but to also have a fallback position. Just in case.

Compared to running Unit and Integration tests, Functional tests are a different story. Regardless of your software, you’ll want to run your Functional tests in a controlled environment, and this usually involves spinning up virtual machines, installing software, configuring the software and so on. To get good test results, and to lower the risk that the results have been corrupted by previously test runs, you’ll want to use a clean environment each time you run the tests. Setting up and running the tests then becomes a time consuming and boring task, something that developers hate.

What happens when you give a developer a task to do that is time consuming and boring?

Automation happens.

Procedural Logic

Before you start doing anything, its helpful to have a high level overview of what you want to accomplish.

At a high level, the automated execution of functional tests needed to:

• Set up a test environment.
  • Spin up a fresh virtual machine.
  • Install the software under test.
  • Configure software under test.
• Execute the functional tests.
• Report the results.

Fairly straightforward. As with everything related to software though, the devil is in the details.

For anyone who doesn’t want to listen to me blather, here is a link to a GitHub repository containing sanitized versions of the scripts. Note that the scripts were not complete at the time of this post, but will be completed later.

Now, on to the blather!

Automatic Weapons

In order to automate any of the above, I would need to select a scripting language.

It would need to be able to do just about anything (which is true of most scripting languages), but would also have to be able to allow me to remotely execute a script on a machine without having to log onto it or use the UI in any way.

I’ve been doing a lot of work with Powershell recently, mostly using it to automate build, package and publish processes. I’d hesitated to learn Powershell for a long time, because every time I encountered something that I thought would have been made easier by using Powershell, I realised I would have to spend a significant amount of time learning just the basics of Powershell before I could do anything useful. I finally bit the bullet and did just that, and its snowballed from there.

Powershell is the hammer and everything is a nail now.

Obviously being a well established scripting language and is installed on basically every modern version of Windows. Powerful by itself, it’s integration with the .NET framework allows a C# developer like me the power to fall back to the familiar .NET BCL for anything I can’t accomplish using just Powershell and its cmdlets. Finally, Powershell Remote Execution allows you to configure a machine and allow authenticated users to remotely execute scripts on it.

So, Powershell it was.

A little bit more about Powershell Remote Execution. It leverages the Windows Remoting Framework (WinRM), and once you’ve got all the bits and pieces setup on the target machine, is very easy to use.

A couple of things to be aware of with remote execution:

1. By default the Windows Remoting Service is not enabled on some versions of Windows. Obviously this needs to be running.
2. Powershell Remote Execution communicates over port 5985 (HTTP) and 5986 (HTTPS). Earlier versions used 80 and 443. These ports need to be configured in the Firewall on the machine in question.
3. The user you are planning on using for the remote execution (and I highly suggest using a brand new user just for this purpose) needs to be a member of the [GROUP HERE] group.

Once you’ve sorted the things above, actually remotely executing a script can be accomplish using the Invoke-Command cmdlet, like so:

$pw = ConvertTo-SecureString '[REMOTE USER PASSWORD' -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential('[REMOTE USERNAME]', $pw)
$session = New-PSSession -ComputerName $ipaddress -Credential $cred 

write-host "Beginning remote execution on [$ipaddress]."

$testResult = Invoke-Command -Session $session -FilePath "$root\remote-download-files-and-run-functional-tests.ps1" -ArgumentList $awsKey, $awsSecret, $awsRegion, $awsBucket, $buildIdentifier

Notice that I don’t have to use a machine name at all. IP Addresses work fine in the ComputerName parameter. How do I know the IP address? That information is retrieved when starting the Amazon EC2 instance.

Environmental Concerns

In order to execute the functional tests, I wanted to be able to create a brand new, clean virtual machine without any human interaction. As I’ve stated previously, we primarily use Amazon EC2 for our virtualisation needs.

The creation of a virtual machine for functional testing would need to be done from another AWS EC2 instance, the one running the TeamCity build agent. The idea being that the build agent instance is responsible for building the software/installer, and would in turn farm out the execution of the functional tests to a completely different machine, to keep a good separation of concerns.

Amazon supplies two methods of interacting with AWS EC2 (Elastic Compute Cloud) via Powershell on a Windows machine.

The first is a set of cmdlets (Get-EC2Instance, New-EC2Instance, etc).

The second is the classes available in the .NET SDK for AWS.

The upside of running on an EC2 instance that was based off an Amazon supplied image is that both of those methods are already installed, so I didn’t have to mess around with any dependencies.

I ended up using a combination of both (cmdlets and .NET SDK objects) to get an instance up and running, mostly because the cmdlets didn’t expose all of the functionality that I needed.

There were 3 distinct parts to using Amazon EC2 for the test environment. Creation, Configuration and Waiting and Clean Up. All of these needed to be automated.

Creation

Obviously an instance needs to be created. The reason this part is split from the Configuration and Waiting is because I’m still not all that accomplished at error handling and returning values in Powershell. Originally I had creation and configuration/waiting in the same script, but if the call to New-EC2Instance returned successfully and then something else failed, I had a hard time returning the instance information in order to terminate it in the finally block of the wrapping script.

The full content of the creation script is available at create-new-ec2-instance.ps1. Its called from the main script (functional-tests.ps1).

Configuration and Waiting

Beyond the configuration done as part of creation, instances can be tagged to add additional information. Also, the script needs to wait on a number of important indicators to ensure that the instance is ready to be interacted with. It made sense to do these two things together for reasons.

The tags help to identify the instance (the name) and also mark the instance as being acceptable to be terminated as part of a scheduled cleanup script that runs over all of our EC2 instances in order to ensure we don’t run expensive instances longer than we expected to.

As for the waiting indicators, the first indicator is whether or not the instance is running. This is an easy one, as the state of the instance is very easy to get at. You can see the function below, but all it does is poll the instance every 5 seconds to check whether or not it has entered the desired state yet.

The second indicator is a bit harder to get at, but it actually much more important. EC2 instances can be configured with status checks, and one of those status checks is whether or not the instance is actually reachable. I’m honestly not sure if this is something that someone before me setup, or if it is standard on all EC2 instances, but its extremely useful.

Anyway, accessing this status check is a bit of a rabbit hole. You can see the function below, but it uses a similar approach to the running check. It polls some information about the instance every 5 seconds until it meets certain criteria. This is the one spot in the entire script that I had to use the .NET SDK classes, as I couldn’t find a way to get this information out of a cmdlet.

The full content of the configuration and wait script is available at tag-and-wait-for-ec2-instance.ps1, and is just called from the main script.

Clean Up

Since you don’t want to leave instances hanging around, burning money, the script needs to clean up after it was done.

Programmatically terminating an instance is quite easy, but I had a lot of issues around the robustness of the script itself, as I couldn’t quite grasp the correct path to ensure that a clean up was always run if an instance was successfully created. The solution to this was to split the creation and tag/wait into different scripts, to ensure that if creation finished it would always return identifying information about the instance for clean up.

Termination happens in the finally block of the main script (functional-tests.ps1).

Instant Machine

Of course all of the instance creation above is dependent on actually having an AMI (Amazon Machine Image) available that holds all of the baseline information about the instance to be created, as well as other things like VPC (Virtual Private Cloud, basically how the instance fits into a network) and security groups (for defining port accessibility). I’d already gone through this process last time I was playing with EC2 instances, so it was just a matter of identifying the various bits and pieces that needs to be done on the machine in order to make it work, while keeping it as clean as possible in order to get good test results.

I went through the image creation process a lot as I evolved the automation script. One thing I found to be useful was to create a change log for the machine in question (I used a page in Confluence) and to version any images made. This helped me to keep the whole process repeatable, as well as documenting the requirements of a machine able to perform the functional tests.

To Be Continued

I think that’s probably enough for now, so next time I’ll continue and explain about automating the installation of the software under test and then actually running the tests and reporting the results.

Until next time!