I’ve been doing a lot of Test Automation lately. In a weird co-incidence, a lot of swearing and gnashing of teeth as well. Strange that.

One of the automation tools that I’ve been using is TestComplete, and I’ve been pleasantly surprised to find that it doesn’t get in my way nearly as much as I would have expected. In comparison, I’ve had far more problems with the behaviour of the application under test rather than the execution environment, which says a lot for TestComplete’s ability to interact with horrifying VB6/WinForms chimera applications.

Its not all puppies and roses though. TestComplete is very much a tabs and text boxes kind of application and while parts of the tool have some nice Intellisense (more than I would have expected) its certainly no Visual Studio. In addition, binary files make meaningful diffs impossible, its licensing model is draconian at best and the price is insane.

But aside from those gripes, it certainly does get the job done.

When I decided that in order to test a piece of functionality involving the database, I would need to leverage some AWS EC2 images, I was unsurprised to find that it had support for executing Powershell scripts. I seems to be able to do just about anything really.

Well, I suppose support is a strong word.

Scripted Interactions

TestComplete supports a number of scripting languages for programmatically writing tests. There is also a very useful record-replay GUI which actually creates a programmatically accessible map of your application, which can then be used in scripts if you want.

When you create a TestComplete project, you can choose from VBScript, JScript and two other scripting varieties that slip my mind. After you’ve created the project, you don’t seem to be able to change this choice, so make sure you pick one you want to use.

I did not create the project.

The person who did picked JScript, which I have never written before. But at least it wasn't VBScript I suppose.

My understanding of JScript is limited, but I believe it is a Microsoft implementation of some Javascript standard. I have no idea which one, and its certainly not recent.

Anyway, I’m still not entirely sure how it all fits together, but from the JScript code you can access some of the .NET Framework libraries, like System.Management.Automation.

Using this namespace, you can create an object that represents a Powershell process, configure it with some commands, and then execute it.

Hence Powershell support.

I Have the Power

I’ve written about my growing usage of Powershell before, but I’ve found it to be a fantastically powerful tool. It really can do just about anything. In this case, I gravitated towards it because I’d already written a bunch of scripts for initialising AWS EC2 instances, and I wanted to reuse them to decrease the time it would take me to put together this functional test. For this test I would need to create an AMI containing a database in the state that I wanted it to be in, and then spin up instances of that AMI whenever I wanted to run the test.

In the end, as it is with these sorts of things, it wasn’t setting up the EC2 instances that took all of my time. That yak had already been shaved.

The first problem I had to solve was actually being able to run a Powershell script file using the component above.

Anyone who has ever used Powershell is probably very familiar with the Execution Policy of the system in question. Typically this defaults to Restricted, which means you are not allowed to execute scripts.

Luckily you can override this purely for the process you are about to execute, without impacting on the system at large. This, of course, means that you can change the Execution Policy without needing Administrative privileges.

Set-ExecutionPolicy RemoteSigned -Scope Process -Force

The second problem I had to solve was getting results out of the Powershell component in TestComplete. In my case I needed to get the Instance ID that was returned by the script, and then store it in a TestComplete variable for use later on. I’d need to do the same thing with the IP address, obtained after waiting for the instance to be ready.

Waiting for an instance to be ready is actually kind of annoying. There are two things you need to check. The first is whether or not the instance is considered running. The second is whether or not the instance is actually contactable through its network interface. Waiting for an instance to be running only takes about 15-30 seconds. Waiting for an instance to be contactable takes 5-10 minutes depending on how capricious AWS is feeling. As you can imagine, this can make testing your “automatically spin up an instance” script a very frustrating experience. Long feedback loops suck.

When you execute a series of commands through the Powershell component, the return value is a collection of PSObjects. These objects are what would have normally been returned via the output stream. In my case I was returning a single string, so I needed to get the first entry in the collection, and then get its ImmediateBaseObject property. To get the string value I then had to get the OleValue property.

Tying both of the above comments together, here is the collection of functions that I created to launch Powershell from TestComplete.

function CreatePowershellExecutionObject()
{
  var powershell = dotNET.System_Management_Automation.Powershell.Create();
  // This is set before every script because otherwise we can't execute script files,
  // Thats where most of our powershell lives.
  powershell.AddScript("Set-ExecutionPolicy RemoteSigned -Scope Process -Force");
  
  // This redirects the write-host function to nothingness, because otherwise any
  // script executed through this componenent will fail if it has the audacity to
  // write-host. This is because its non-interactive and write-host is like a special
  // host channel thing that needs to be implemented.
  powershell.AddScript("function write-host { }");
  
  return powershell;
}

function InvokePowershellAndThrowIfErrors(powershell)
{
  var result = powershell.Invoke();
  
  if (powershell.HadErrors)
  {
    var firstError = powershell.Streams.Error.Item(0);
    if (firstError.ErrorDetails != null)
    {
      throw new Error(firstError.ErrorDetails);
    }
    else
    {
      throw new Error(firstError.Exception.ToString());
    }
  }
  
  return result;
}

function GetCommonPowershellAwsScriptArguments()
{
  var awsKey = Project.Variables.AwsKey;
  var awsSecret = Project.Variables.AwsSecret;
  var awsRegion = Project.Variables.AwsRegion;
  return "-SuppliedAwsKey " + awsKey + " -SuppliedAwsSecret " + awsSecret + " -SuppliedAwsRegion " + awsRegion + " "
}

function StartAwsVirtualMachineWithFullDatabaseAndSetInstanceId()
{
  var gatewayVersion = GetVersionOfGatewayInstalled();
  var scriptsPath = GetScriptsDirectoryPath();
  var executeInstanceCreationScript = "& \"" + 
    scriptsPath + 
    "\\functional-tests\\create-new-max-size-db-ec2-instance.ps1\" " + 
    GetCommonPowershellAwsScriptArguments() +
    "-BuildIdentifier " + gatewayVersion;

  var powershell = CreatePowershellExecutionObject();
  powershell.AddScript(executeInstanceCreationScript);
  
  Indicator.PushText("Spinning up AWS EC2 Instance containing Test Database");
  var result = InvokePowershellAndThrowIfErrors(powershell);
  Indicator.PopText();
  
  var instanceId = result.Item(0).ImmediateBaseObject.OleValue; 
  
  KeywordTests.WhenADatabaseIsAtMaximumSizeForExpress_ThenTheFilestreamConversionStillWorks.Variables.FullDbEc2InstanceId = instanceId;
}

Notice that I split the instance creation from the waiting for the instance to be ready. This is an optimisation. I create the instance right at the start of the functional test suite, and then execute other tests while that instance is being setup in AWS. By the time I get to it, I don’t have to wait for it to be ready at all. Less useful when testing the database dependent test by itself, but it shaves 6+ minutes off the test suite when run together. Every little bit counts.

Application Shmaplication

Now that the test database was being setup as per my wishes, it was a simple matter to record the actions I wanted to take in the application and make some checkpoints for verification purposes.

Checkpoints in TestComplete are basically asserts. Record a value (of many varied types, ranging from Onscreen Object property values to images) and then check that the application matches those values when it is run.

After recording the test steps, I broke them down into reusable components (as is good practice) and made sure the script was robust in the face of failures and unexpected windows (and other things).

The steps for the executing the test using the application itself were easy enough, thanks to TestComplete.

Tidbits

I did encounter a few other things while setting this test up that I think are worth mentioning.

The first was that the test needed to be able to be run from inside our VPC (Virtual Private Cloud) in AWS as well as from our local development machines. Actually running the test was already a solved problem (solved when I automated the execution of the functional tests late last year), but making a connection to the virtual machine hosting the database in AWS was a new problem.

Our AWS VPC is fairly locked down (for good reason) so machines in there generally can’t access machines in the outside word except over a few authorised channels (HTTP and HTTPS through a proxy for example). Even though the database machine was sitting in the same VPC as the functional tests worker, I had planned to only access it through its public IP address (for simplicity). This wouldn’t work without additional changes to our security model, which would have been a pain (I have no control over those policies).

This meant that in one case I needed to use the Public IP Address of the instance (when it was being run from our development machines) and in the other I needed to use the Private IP Address.

Code to select the correct IP address fit nicely into my Powershell script to wait for the instance to be ready, which already returned an IP address. All I had to do was test the public IP over a specific port and depending on whether or not it worked, return the appropriate value. I did this using the TCPClient class in the .NET framework.

The second thing I had to deal with was a dependency change.

Previously our TestComplete project was not dependent on anything except itself. It could be very easily compressed into a single file and tossed around the internet as necessary. Now that I had added a dependency on a series of Powershell scripts I had to change the execution script for our Functional Tests to pack up and distribute additional directories. Nothing too painful luckily, as it was as simple enough matter to include more things in the compressed archive.

The final problem I ran into was with the application under test.

Part of the automated Functional Test is to open the database. When you ask the application to do that, it will pop up a fairly standard dialog with server & database name.

Being the helpful VB6 application that it is, it also does a search of the network to find any database servers that are available, so you can quickly select them from a dropdown. Being a single threaded application without a great amount of thought being put into user experience, the dialog freezes for a few moments while it does the search.

If you try to find that Window or any of its components using TestComplete while its frozen, the mapping for the dialog box changes from what you would expected (Aliases.Application.DialogName) to something completely different (Process(“ApplicationName”).Window(“DialogName”).Control(“Name”)). Since the test was recorded using the first alias, it then times out when looking for the control it expects, and fails.

I got around this by introducing a delay before even attempting to look for any controls on that dialog.

Depressingly, this is a common solution to solving automation problems like that.

Conclusion

If you were expecting this section to be where I tie everything together and impart on you some TL;DR lessons, then prepare yourself for disappointment.

The one thing I will say though is that setting up good automated Functional Tests takes a hell of a lot of effort. it gets easier as you spend more time doing it, but sometimes I question the benefits. Certainly its an interesting exercise and you learn a lot about the system under test, but the tests you create are usually fragile and overly complex.

Functional Tests that automate an application definitely shouldn’t be your only variety of test that’s for sure.

Regardless of the adventure above, TestComplete is pretty great, and it certainly makes the whole automated testing process much easier than it would be if you used something else.

Like CodedUI.

Which I have done before.

I don’t recommend it.

So, we decided to automate the execution of our Functional tests. After all, tests that aren't being run are effectively worthless.

In Part 1, I gave a little background, mentioned the scripting language we were going to use (Powershell), talked about programmatically spinning up the appropriate virtual machine in AWS for testing purposes and then how to communicate with it.

This time I will be talking about automating the installation of the software under test, running the actual tests and then reporting the results.

Just like last time, here is a link to the GitHub repository with sanitized versions of all the scripts (which have been updated significantly), so you don't have to piece them together from the snippets I’m posting throughout this post.

Installed into Power

Now that I could create an appropriate virtual machine as necessary and execute arbitrary code on it remotely, I needed to install the actual software under test.

First I needed to get the installer onto the machine though, as we do not make all of our build artefacts publically available on every single build (so no just downloading it from the internet). Essentially I just needed a mechanism to transfer files from one machine to another. Something that wouldn’t be too difficult to setup and maintain.

I tossed up a number of options:

• FTP. I could setup an FTP server on the virtual machine and transfer the files that way. The downside of this is that I would have to setup an FTP server on the virtual machine, and make sure it was secure; and configured correctly. I haven’t setup a lot of FTP servers before, so I decided not to do this one.
• SSH + File Transfer. Similar to the FTP option, I could install an SSH server on the virtual machine and then use something like SCP to securely copy the files to the machine. This would have been the easiest option if the machine was Linux based, but being a Windows machine it was more effort than it was worth.
• Use an intermediary location, like an Amazon S3 bucket. This is the option I ended up going with.

Programmatically copying files to an Amazon S3 bucket using Powershell is fairly straightforward, although I did run into two issues.

Folders? What Folders?

Even though its common for GUI tools that sit on top of Amazon S3 to present the information as a familiar folder/file directory structure, that is entirely not how it actually works. In fact, thinking of the information that you put in S3 in that way will just get you into trouble.

Instead, its much more accurate to think of the information you upload to S3 to be key/value pairs, where the key tends to look like a fully qualified file path.

I made an interesting error at one point and uploaded 3 things to S3 with the following keys, X, X\Y and X\Z. The S3 website interpreted the X as a folder, which meant that I was no longer able to access the file that I had actually stored at X, at least through the GUI anyway. This is one example of why thinking about S3 as folders/files can get you in trouble.

Actually uploading files to S3 using Powershell is easy enough. Amazon supply a set of cmdlets that allow you to interact with S3, and those cmdlets are pre-installed on machines originally created using an Amazon supplied AMI.

With regards to credentials, you can choose to store the credentials in configuration, allowing you to avoid having to enter them for every call, or you can supply then whenever you call the cmdlets. Because this was a script, I chose to supply them on each call, so that the script would be self contained. I’m not a fan of global settings in general, they make me uncomfortable. I feel that it makes the code harder to understand, and in this case, it would have obfuscated how the cmdlets were authenticating to the service.

The function that uploads things to S3 is as follows:

function UploadFileToS3
{
    param
    (
        [string]$awsKey,
        [string]$awsSecret,
        [string]$awsRegion,
        [string]$awsBucket,
        [System.IO.FileInfo]$file,
        [string]$S3FileKey
    )

    write-host "Uploading [$($file.FullName)] to [$($awsRegion):$($awsBucket):$S3FileKey]."
    Write-S3Object -BucketName $awsBucket -Key $S3FileKey -File "$($file.FullName)" -Region $awsRegion -AccessKey $awsKey -SecretKey $awsSecret

    return $S3FileKey
}

The function that downloads things is:

function DownloadFileFromS3
{
    param
    (
        [string]$awsKey,
        [string]$awsSecret,
        [string]$awsRegion,
        [string]$awsBucket,
        [string]$S3FileKey,
        [string]$destinationPath
    )

    $destinationFile = new-object System.IO.FileInfo($destinationPath)
    if ($destinationFile.Exists)
    {
        write-host "Destination for S3 download of [$S3FileKey] ([$($destinationFile.FullName)]) already exists. Deleting."
        $destinationFile.Delete()
    }

    write-host "Downloading [$($awsRegion):$($awsBucket):$S3FileKey] to [$($destinationFile.FullName)]."
    Read-S3Object -BucketName $awsBucket -Key $S3FileKey -File "$($destinationFile.FullName)" -Region $awsRegion -AccessKey $awsKey -SecretKey $awsSecret | write-host

    $destinationFile.Refresh()

    return $destinationFile
}

Once the installer was downloaded on the virtual machine, it was straightforward to install it silently.

if (!$installerFile.Exists)
{
    throw "The Installer was supposed to be located at [$($installerFile.FullName)] but could not be found."
}

write-host "Installing Application (silently) from the installer [$($installerFile.FullName)]"
# Piping the results of the installer to the output stream forces it to wait until its done before continuing on
# with the remainder of the script. No useful output comes out of it anyway, all we really care about
# is the return code.
& "$($installerFile.FullName)" /exenoui /qn /norestart | write-host
if ($LASTEXITCODE -ne 0)
{
    throw "Failed to Install Application."
}

We use Advanced Installer, which in turn uses an MSI, so you’ll notice that there are actually a number of switches being used above to get the whole thing to install without human interaction. Also note the the piping to write-host, which ensures that Powershell actually waits for the installer process to finish, instead of just starting it and then continuing on its merry way. I would have piped to write-output, but then the uncontrolled information from the installer would go to the output stream and mess up my return value.

Permissions Denied

God. Damn. Permissions.

I had so much trouble with permissions on the files that I uploaded to S3. I actually had a point where I was able to upload files, but I couldn’t download them using the same credentials that I used to upload them! That makes no goddamn sense.

To frame the situation somewhat, I created a user within our AWS account specifically for all of the scripted interactions with the service. I then created a bucket to contain the temporary files that are uploaded and downloaded as part of the functional test execution.

The way that S3 defines permissions on buckets is a little strange in my opinion. I would expect to be able to define permissions on a per user basis for the bucket. Like, user X can read from and write to this bucket, user Y can only read, etc. I would also expect files that are uploaded to this bucket to then inherit those permissions, like a folder, unless I went out of my way to change them. That’s the mental model of permissions that I have, likely as a result of using Windows for many years.

This is not how it works.

Yes you can define permissions on a bucket, but not for users within your AWS account. It doesn’t even seem to be able to define permissions for other specific AWS accounts either. There’s a number of groups available, one of which is Authenticated Users, which I originally set with the understanding that it would give authenticated users belonging to my AWS account the permissions I specified. Plot twist, Authenticated Users means any AWS user. Ever. As long as they are authenticating. Obviously not what I wanted. At least I could upload files though (but not download them).

Permissions are not inherited when set through the simple options I mentioned above, so any file I uploaded had no permissions set on it.

The only way to set permissions with the granularity necessary and to have them inherited automatically is to use a Bucket Policy.

Setting up a Bucket Policy is not straight forward, at least to a novice like myself.

After some wrangling with the helper page, and some reading of the documentation, here is the bucket policy I ended up using, with details obscured to protect the guilty.

{
    "Version": "2008-10-17",
    "Id": "Policy1417656003414",
    "Statement": [
        {
            "Sid": "Stmt1417656002326",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::[USER IDENTIFIER]"
            },
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::[BUCKET NAME]/*"
        },
        {
            "Sid": "Stmt14176560023267",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::[USER IDENTIFIER]"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::[BUCKET NAME]"
        }
    ]
}

The policy actually reads okay once you have it, but I’ll be honest, I still don’t quite understand it. I know that I’ve specifically given the listed permissions on the contents of the bucket to my user, and also given List permissions on the bucket itself. This allowed me to upload and download files with the user I created, which is what I wanted. I’ll probably never need to touch Bucket Policies again, but if I do, I’ll make more of an effort to understand them.

Testing My Patience

Just like the installer, before I can run the tests I need to actually have the tests to run.

We currently use TestComplete as our Functional test framework. I honestly haven’t looked into TestComplete all that much, apart from just running the tests, but it seems to be solid enough. TestComplete stores your functional tests in a structure similar to a Visual Studio project, with a project file and a number of files under that that define the actual tests and the order to run them in.

For us, our functional tests are stored in the same Git repository as our code. We use feature branches, so it makes sense to run the functional tests that line up with the code that the build was made from. The Build Agent that builds the installer has access to the source (obviously), so its a fairly simple matter to just zip up the definitions and any dependent files, and push that archive to S3 in the exact same manner as the installer, ready for the virtual machine to download.

Actually running the tests though? That was a bastard.

As I mentioned in Part 1, after a virtual machine is spun up as part of this testing process, I use Powershell remote execution to push a script to the machine for execution.

As long as you don’t want to do anything with a user interface, this works fantastically.

Functional tests being based entirely around interaction with the user interface therefore prove problematic.

The Powershell remote session that is created when executing the script remotely does not have any ability for user-interactivity, and cannot to my knowledge. Its just something that Powershell can’t do.

However, you can remotely execute another tool, PSExec, and specify that it run a process locally in an interactive session.

$testExecute = 'C:\Program Files (x86)\SmartBear\TestExecute 10\Bin\TestExecute.exe'
$testExecuteProjectFolderPath = "$functionalTestDefinitionsDirectoryPath\Application"
$testExecuteProject = "$testExecuteProjectFolderPath\ApplicationTests.pjs"
$testExecuteResultsFilePath = "$functionalTestDefinitionsDirectoryPath\TestResults.mht"

write-host "Running tests at [$testExecuteProject] using TestExecute at [$testExecute]. Results going to [$testExecuteResultsFilePath]."
# Psexec does a really annoying thing where it writes information to STDERR, which Powershell detects as an error
# and then throws an exception. The 2>&1 redirects all STDERR to STDOUT to get around this.
# Bit of a dirty hack here. The -i 2 parameter executes the application in interactive mode specifying
# a pre-existing session with ID 2. This is the session that was setup by creating a remote desktop
# session before this script was executed. Sorry.
& "C:\Tools\sysinternals\psexec.exe" -accepteula -i 2 -h -u $remoteUser -p $remotePassword "$testExecute" "$testExecuteProject" /run /SilentMode /exit /DoNotShowLog /ExportLog:$testExecuteResultsFilePath 2>&1 | write-host
[int]$testExecuteExitCode = $LASTEXITCODE

The –i [NUMBER] in the command above tells PSExec to execute the process in an interactive user session, specifically the session with the ID specified. I’ve hardcoded mine to 2, which isn’t great, but works reliably in this environment because the remote desktop session I create after spinning up the instance always ends up with ID 2. Hacky.

Remote desktop session you may ask? In order for TestComplete (well TestExecute technically) to execute the tests correctly you need to actually have a desktop session setup. I assume this is related to it hooking into the UI user mouse and keyboard hooks or something, I don’t really know. All I know is that it didn't work without a remote desktop session of some sort.

On the upside, you can automate the creation of a remote desktop session with a little bit of effort, although there are two hard bits to be aware of.

Who Are You?

There is no obvious way to supply credentials to the Remote Desktop client (mstsc.exe). You can supply the machine that you want to make the connection to (thank god), but not credentials. I think there might be support for storing this information in an RDP file though, which seem to be fairly straightforward. As you can probably guess from my lack of knowledge about that particular approach, that’s not what I ended up doing.

I still don’t fully understand the solution, but you can use the built in windows utility cmdkey to store credentials for things. If you store credentials for the remote address that you are connecting to, the Remote Desktop client will happily use them.

There is one thing you need to be careful with when using this utility to automate Remote Desktop connections. If you clean up after yourself (by deleting the stored credentials after you use them) make sure you wait until the remote session is established. If you delete the credentials before the client actually uses them you will end up thinking that the stored credentials didn’t work, and waste a day investigating and trialling VNC solutions which ultimately don’t work as well as Remote Desktop before you realise the stupidity of your mistake. This totally happened to a friend of mine. Not me at all.

Anyway, the entirety of the remote desktop script (from start-remote-session.ps1):

param (
    [Parameter(Mandatory=$true,Position=0)]
    [Alias("CN")]
    [string]$ComputerNameOrIp,
    [Parameter(Mandatory=$true,Position=1)]
    [Alias("U")] 
    [string]$User,
    [Parameter(Mandatory=$true,Position=2)]
    [Alias("P")] 
    [string]$Password
)

& "$($env:SystemRoot)\system32\cmdkey.exe" /generic:$ComputerNameOrIp /user:$User /pass:$Password | write-host

$ProcessInfo = new-object System.Diagnostics.ProcessStartInfo

$ProcessInfo.FileName = "$($env:SystemRoot)\system32\mstsc.exe" 
$ProcessInfo.Arguments = "/v $ComputerNameOrIp"

$Process = new-object System.Diagnostics.Process
$Process.StartInfo = $ProcessInfo
$startResult = $Process.Start()

Start-Sleep -s 15

& "$($env:SystemRoot)\system32\cmdkey.exe" /delete:$ComputerNameOrIp | write-host

return $Process.Id

Do You Trust Me?

The second hard thing with the Remote Desktop client is that it will ask you if you trust the remote computer if you don’t have certificates setup. Now I would typically advise that you setup certificates for this sort of thing, especially if communicating over the internet, but in this case I was remoting into a machine that was isolated from the internet within the same Amazon virtual network, so it wasn’t necessary.

Typically, clicking “yes” on an identity verification dialog isn't a problem, even if it is annoying. Of course, in a fully automated environment, where I am only using remote desktop because I need a desktop to actually be rendered to run my tests, it’s yet another annoying thing I need to deal with without human interaction.

Luckily, you can use a registry script to disable the identity verification in Remote Desktop. This had to be done on the build agent instance (the component that actually executes the functional tests).

Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client]
    "AuthenticationLevelOverride"=dword:00000000

Report Soldier!

With the tests actually running reliably (after all the tricks and traps mentioned above), all that was left was to report the results to TeamCity.

It was trivial to simply get an exit code from TestExecute (0 = Good, 1 = Warnings, 2 = Tests Failed, 3 = Tests Didn’t Run). You can then use this exit code to indicate to TeamCity whether or not the tests succeeded.

if ($testResult -eq $null)
{
    throw "No result returned from remote execution."
}

if ($testResult.Code -ne 0)
{
    write-host "##teamcity[testFailed name='$teamCityFunctionalTestsId' message='TestExecute returned error code $($testResult.Code).' details='See artifacts for TestExecute result files']"
}
else
{
    write-host "##teamcity[testFinished name='$teamCityFunctionalTestsId'"
}

That's enough to pass or fail a build.

Of course, if you actually have failing functional tests you want a hell of a lot more information in order to find out whythey failed. Considering the virtual machine on which the tests were executed will have been terminated at the end of the test run, we needed to extract the maximum amount of information possible.

TestComplete (TestExecute) has two reporting mechanisms, not including the exit code from above.

The first is an output file, which you can specify when running the process. I chose an MHT file, which is a nice HTML document showing the tests that ran (and failed), and which has embedded screenshots taken on the failing steps. Very useful.

The second is the actual execution log, which is attached to the TestComplete project. This is a little harder to use, as you need to take the entire project and its log file and open it in TestComplete, but is great for in depth digging, as it gives a lot of information about which steps are failing and has screenshots as well.

Both of these components are zipped up on the functional tests worker and then placed into S3, so the original script can download them and attach them the TeamCity build artefacts. This is essentially the same process as for getting the test definitions and installer to the functional tests worker, but in reverse, so I won’t go into any more detail about it.

Summary

So, after all is said and done, I had automated:

• The creation of a virtual machine for testing.
• The installation of the latest build on that virtual machine.
• The execution of the functional tests.
• The reporting of the results, in both a programmatic (build pass/fail) and human readable way.

There were obviously other supporting bits and pieces around the core processes above, but there is little point in mentioning them here in the summary.

Conclusion

All up, I spent about 2 weeks of actual time on automating the functional tests.

A lot of the time was spent familiarising myself with a set of tools that I’d never (or barely) used before, like TestComplete (and TestExecute), the software under test, Powershell, programmatic access to Amazon EC2 and programmatic access to Amazon S3.

As with all things technical, I frequently ran into roadblocks and things not working the way that I would have expected them too out of the box. These things are vicious time sinks, involving scouring the internet for other people who’ve had the same issue and hoping to all that is holy that they solved their problem and then remembered to come back and share their solution.

Like all things involving software, I fought complexity every step of way. The two biggest offenders were the complexity in handling errors in Powershell in a robust way (so I could clean up my EC2 instances) and actually getting TestExecute to run the damn tests because of its interactivity requirements.

When all was said and done though, the functional tests are now an integral part of our build process, which means there is far more incentive to adding to them and maintaining them. I do have some concerns about their reliability (UI focused tests are a bit like that), but that can be improved over time.

In my last blog post, I mentioned the 3 classifications that I think tests fall into, Unit, Integration and Functional.

Of course, regardless of classification, all tests are only valuable if they are actually being executed. Its wonderful to say you have tests, but if you’re not running them all the time, and actually looking at the results, they are worthless. Worse than worthless if you think about it, because the presence of tests gives a false sense of security about your system.

Typically executing Unit Tests (and Integration Tests if they are using the same framework) is trivial, made vasty easier by having a build server. Its not that bad even if you don’t have a build server, because those sorts of tests can typically be run on a developers machine, without a huge amount of fanfare. The downside of not having a build server, is that the developers in question need to remember to run the tests. As creative people, following a checklist that includes “wait for tests to run” is sometimes not our strongest quality.

Note that I’m not saying developers should not be running tests on their own machines, because they definitely should be. I would usually limit this to Unit tests though, or very self-contained Integration tests. You need to be very careful about complicating the process of actually writing and committing code if you want to produce features and improvements in a reasonable amount of time. Its very helpful to encourage people to run the tests themselves regularly, but to also have a fallback position. Just in case.

Compared to running Unit and Integration tests, Functional tests are a different story. Regardless of your software, you’ll want to run your Functional tests in a controlled environment, and this usually involves spinning up virtual machines, installing software, configuring the software and so on. To get good test results, and to lower the risk that the results have been corrupted by previously test runs, you’ll want to use a clean environment each time you run the tests. Setting up and running the tests then becomes a time consuming and boring task, something that developers hate.

What happens when you give a developer a task to do that is time consuming and boring?

Automation happens.

Procedural Logic

Before you start doing anything, its helpful to have a high level overview of what you want to accomplish.

At a high level, the automated execution of functional tests needed to:

• Set up a test environment.
  • Spin up a fresh virtual machine.
  • Install the software under test.
  • Configure software under test.
• Execute the functional tests.
• Report the results.

Fairly straightforward. As with everything related to software though, the devil is in the details.

For anyone who doesn’t want to listen to me blather, here is a link to a GitHub repository containing sanitized versions of the scripts. Note that the scripts were not complete at the time of this post, but will be completed later.

Now, on to the blather!

Automatic Weapons

In order to automate any of the above, I would need to select a scripting language.

It would need to be able to do just about anything (which is true of most scripting languages), but would also have to be able to allow me to remotely execute a script on a machine without having to log onto it or use the UI in any way.

I’ve been doing a lot of work with Powershell recently, mostly using it to automate build, package and publish processes. I’d hesitated to learn Powershell for a long time, because every time I encountered something that I thought would have been made easier by using Powershell, I realised I would have to spend a significant amount of time learning just the basics of Powershell before I could do anything useful. I finally bit the bullet and did just that, and its snowballed from there.

Powershell is the hammer and everything is a nail now.

Obviously being a well established scripting language and is installed on basically every modern version of Windows. Powerful by itself, it’s integration with the .NET framework allows a C# developer like me the power to fall back to the familiar .NET BCL for anything I can’t accomplish using just Powershell and its cmdlets. Finally, Powershell Remote Execution allows you to configure a machine and allow authenticated users to remotely execute scripts on it.

So, Powershell it was.

A little bit more about Powershell Remote Execution. It leverages the Windows Remoting Framework (WinRM), and once you’ve got all the bits and pieces setup on the target machine, is very easy to use.

A couple of things to be aware of with remote execution:

1. By default the Windows Remoting Service is not enabled on some versions of Windows. Obviously this needs to be running.
2. Powershell Remote Execution communicates over port 5985 (HTTP) and 5986 (HTTPS). Earlier versions used 80 and 443. These ports need to be configured in the Firewall on the machine in question.
3. The user you are planning on using for the remote execution (and I highly suggest using a brand new user just for this purpose) needs to be a member of the [GROUP HERE] group.

Once you’ve sorted the things above, actually remotely executing a script can be accomplish using the Invoke-Command cmdlet, like so:

$pw = ConvertTo-SecureString '[REMOTE USER PASSWORD' -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential('[REMOTE USERNAME]', $pw)
$session = New-PSSession -ComputerName $ipaddress -Credential $cred 

write-host "Beginning remote execution on [$ipaddress]."

$testResult = Invoke-Command -Session $session -FilePath "$root\remote-download-files-and-run-functional-tests.ps1" -ArgumentList $awsKey, $awsSecret, $awsRegion, $awsBucket, $buildIdentifier

Notice that I don’t have to use a machine name at all. IP Addresses work fine in the ComputerName parameter. How do I know the IP address? That information is retrieved when starting the Amazon EC2 instance.

Environmental Concerns

In order to execute the functional tests, I wanted to be able to create a brand new, clean virtual machine without any human interaction. As I’ve stated previously, we primarily use Amazon EC2 for our virtualisation needs.

The creation of a virtual machine for functional testing would need to be done from another AWS EC2 instance, the one running the TeamCity build agent. The idea being that the build agent instance is responsible for building the software/installer, and would in turn farm out the execution of the functional tests to a completely different machine, to keep a good separation of concerns.

Amazon supplies two methods of interacting with AWS EC2 (Elastic Compute Cloud) via Powershell on a Windows machine.

The first is a set of cmdlets (Get-EC2Instance, New-EC2Instance, etc).

The second is the classes available in the .NET SDK for AWS.

The upside of running on an EC2 instance that was based off an Amazon supplied image is that both of those methods are already installed, so I didn’t have to mess around with any dependencies.

I ended up using a combination of both (cmdlets and .NET SDK objects) to get an instance up and running, mostly because the cmdlets didn’t expose all of the functionality that I needed.

There were 3 distinct parts to using Amazon EC2 for the test environment. Creation, Configuration and Waiting and Clean Up. All of these needed to be automated.

Creation

Obviously an instance needs to be created. The reason this part is split from the Configuration and Waiting is because I’m still not all that accomplished at error handling and returning values in Powershell. Originally I had creation and configuration/waiting in the same script, but if the call to New-EC2Instance returned successfully and then something else failed, I had a hard time returning the instance information in order to terminate it in the finally block of the wrapping script.

The full content of the creation script is available at create-new-ec2-instance.ps1. Its called from the main script (functional-tests.ps1).

Configuration and Waiting

Beyond the configuration done as part of creation, instances can be tagged to add additional information. Also, the script needs to wait on a number of important indicators to ensure that the instance is ready to be interacted with. It made sense to do these two things together for reasons.

The tags help to identify the instance (the name) and also mark the instance as being acceptable to be terminated as part of a scheduled cleanup script that runs over all of our EC2 instances in order to ensure we don’t run expensive instances longer than we expected to.

As for the waiting indicators, the first indicator is whether or not the instance is running. This is an easy one, as the state of the instance is very easy to get at. You can see the function below, but all it does is poll the instance every 5 seconds to check whether or not it has entered the desired state yet.

The second indicator is a bit harder to get at, but it actually much more important. EC2 instances can be configured with status checks, and one of those status checks is whether or not the instance is actually reachable. I’m honestly not sure if this is something that someone before me setup, or if it is standard on all EC2 instances, but its extremely useful.

Anyway, accessing this status check is a bit of a rabbit hole. You can see the function below, but it uses a similar approach to the running check. It polls some information about the instance every 5 seconds until it meets certain criteria. This is the one spot in the entire script that I had to use the .NET SDK classes, as I couldn’t find a way to get this information out of a cmdlet.

The full content of the configuration and wait script is available at tag-and-wait-for-ec2-instance.ps1, and is just called from the main script.

Clean Up

Since you don’t want to leave instances hanging around, burning money, the script needs to clean up after it was done.

Programmatically terminating an instance is quite easy, but I had a lot of issues around the robustness of the script itself, as I couldn’t quite grasp the correct path to ensure that a clean up was always run if an instance was successfully created. The solution to this was to split the creation and tag/wait into different scripts, to ensure that if creation finished it would always return identifying information about the instance for clean up.

Termination happens in the finally block of the main script (functional-tests.ps1).

Instant Machine

Of course all of the instance creation above is dependent on actually having an AMI (Amazon Machine Image) available that holds all of the baseline information about the instance to be created, as well as other things like VPC (Virtual Private Cloud, basically how the instance fits into a network) and security groups (for defining port accessibility). I’d already gone through this process last time I was playing with EC2 instances, so it was just a matter of identifying the various bits and pieces that needs to be done on the machine in order to make it work, while keeping it as clean as possible in order to get good test results.

I went through the image creation process a lot as I evolved the automation script. One thing I found to be useful was to create a change log for the machine in question (I used a page in Confluence) and to version any images made. This helped me to keep the whole process repeatable, as well as documenting the requirements of a machine able to perform the functional tests.

To Be Continued

I think that’s probably enough for now, so next time I’ll continue and explain about automating the installation of the software under test and then actually running the tests and reporting the results.

Until next time!

As I mentioned in a previous post, I recently started a new position at Onthehouse.

Onthehouse uses Amazon EC2 for their cloud based virtualisation, including that of the build environment (TeamCity). Its common for a build environment to be largely ignored as long as it is still working, until the day it breaks and then it all goes to hell.

Luckily that is not what happened.

Instead, the team identified that the build environment needed some maintenance, specifically around one of the application specific Build Agents.

Its an ongoing process, but the reason for there being an application specific Build Agent is because the application has a number of arcane, installed, licenced third-party components. Its VB6, so its hard to manage those dependencies in a way that is mobile. Something to work on in the future, but not a priority for right now.

My first task at Onthehouse, was to ensure that changes made to the running Instance of the Build Agent had been appropriately merged into the base Image. As someone who had never before used the Amazon virtualisation platform (Amazon EC2) I was somewhat confused.

This post follows my journey through that confusion and out the other side into understanding and I hope it will be of use to someone else out there.

As an aside, I think that getting new developers to start with build services is a great way to familiarise them with the most important part of an application, how to build it. Another fantastic first step is to get them to fix bugs.

Virtually Awesome

As I mentioned previously, I’ve never used AWS (Amazon Web Services) before, other than uploading some files to an Amazon S3 account, let alone the virtualization platform (Amazon EC2).

My main experience with virtualisation comes from using Virtual Box on my own PC. Sure, I’ve used Azure to spin up machines and websites, and I’ve occasionally interacted with VMWare and Hyper-V, but Virtual Box is what I use every single day to build, create, maintain and execute sandbox environments for testing, exploration and all sorts of other things.

I find Virtual Box straightforward.

You have a virtual machine (which has settings, like CPU Cores, Memory, Disks, etc) and each machine has a set of snapshots.

Snapshots are a record of the state of the virtual machine and its settings at a point in time chosen by the user. I take Snapshots all the time, and I use them to easily roll back to important moments, like a specific version of an application, or before I wrecked everything by doing something stupid and destructive (it happens more than you think).

Thinking about this now, I’m not sure how Virtual Box and Snapshots interact with multiple disks. Snapshots seems to be primarily machine based, not disk based, encapsulating all of the things about the machine. I suppose that probably includes the disks. I guess I don’t tend to use multiple disks in the machines I’m snapshotting all the time, only using them rarely for specific tasks.

Images and Instances

Amazon EC2 (Elastic Compute) does not work the same way as Virtual Box.

I can see why its different to Virtual Box as they have entirely different purposes. Virtual Box is intended to facilitate virtualisation for the single user. EC2 is about using virtualisation to leverage the power of the cloud. Single users are a completely foreign concept. Its all about concurrency and scale.

In Amazon EC2 the core concept is an Image(or Amazon Machine Image, AMI). Images describe everything about a virtual machine, kind of like a Virtual Machine in Virtual Box. However, in order to actually use an Image, you must spin up an Instance of that Image.

At the point in time you spin up an Instance of an Image, they have diverged. The Instance typically contains a link back to its Image, but its not a hard link. The Instance and Image are distinctly separate and you can delete the Image (which if you are using an Amazon supplied Image, will happen regularly) without negatively impacting on the running instance.

Instances generally have Volumes, which I think are essentially virtual disks. Snapshots come into play here as well, but I don’t understand Volumes and Snapshots all that well at this point in time, so I’m going to conveniently gloss over them. Snapshots definitely don’t work like VirtualBox snapshots though, I know that much.

Instances can generally be rebooted, stopped, started and terminated.

Reboot, stop and start do what you expect.

Terminating an instance kills it forever. It also kills the Volume attached to the instance if you have that option selected. If you don’t have the Image that the Instance was created from, you’re screwed, its gone for good. Even if you do, you will have lost any change made to that Image since the Instance began running.

Build It Up

Back to the Build environment.

The application specific Build Agent had an Image, and an active Instance, as normal.

This Instance had been tweaked, updated and changed in various ways since the Image was made, so much so that no-one could remember exactly what had been done. Typically this wouldn’t be a major issue, as Instances don’t just up and disappear.

Except this Instance could, and had in the past.

The reason for its apparently ephemeral nature was because Amazon offers a spot pricing option for Instances. Spot pricing allows you to create a spot request and set your own price for an hour of compute time. As long as the spot price is below that price, your Instance will run. If the spot price goes above your price, your Instance dies. You can setup your spot price request to be reoccurring, such that the Instance will restart when the price goes down again, but you will have lost all information not on the baseline Image (an event like that is equivalent to terminating the instance and starting another one).

Obviously we needed to ensure that the baseline Image was completely able to run a build of the application in question, requiring the minimal amount of configuration on first start.

Thus began a week long adventure to take the current base Image, create an Instance from it, and get a build working, so we could be sure that if our Instance was terminated it would come back and we wouldn’t have to spend a week getting the build working again.

I won’t go into detail about the whole process, but it mostly involved lots of manual steps to find out what was thing was wrong this time, fixing it in as nice a way as time permitted and then trying again. It mostly involved waiting. Waiting for instances, waiting for builds, waiting for images. Not very interesting.

A Better Approach

Knowing what I know now (and how long the whole process would take), my approach would be slightly different.

Take a snapshot of the currently running Instance, spin up an Instance of it, change all of the appropriate unique settings to be invalid (Build Agent name mostly) and then take another Image. That’s your baseline.

Don’t get me wrong, it was a much better learning experience the first way, but it wasn’t exactly an excellent return on investment from the point of view of the organisation.

Ah well, hindsight.

A Better Architecture

The better architecture is to have TeamCity managed the lifetime of its Build Agents, which it is quite happy to do via Amazon EC2. TeamCity can then manage the instances as it sees fit, spinning them down during idle periods, and even starting more during periods of particularly high load (I’m looking at you, end of the iteration crunch time).

I think this is definitely the approach we will take in the future, but that’s a task for another day.

Conclusion

Honestly, the primary obstacle in this particular task was learning how Amazon handles virtualization, and wrapping my head around the differences between that and Virtual Box (which is where my mental model was coming from). After I got my head around that I was in mostly familiar territory, diagnosing build issues and determining the best fix that would maximise mobility in the future, while not requiring a massive amount of time.

From the point of view of me, a new starter, this exercise was incredibly valuable. It taught me an immense amount about the application, its dependencies, the way its built and all sorts of other, half-forgotten tidbits.

From the point of view of the business, I should have definitely realized that there was a quicker path to the end goal (make sure we can recover from a lost Build Agent instance) and taken that into consideration, rather than try to work my way through the arcane dependencies of the application. There’s always the risk that I missed something subtle as well, which will rear its ugly head next time we lose the Build Agent instance.

Which could happen.

At any moment.

(Cue Ominous Music)