TestComplete is Kind of Amazing
- Posted in:
- testing
- powershell
- amazon
I’ve been doing a lot of Test Automation lately. In a weird co-incidence, a lot of swearing and gnashing of teeth as well. Strange that.
One of the automation tools that I’ve been using is TestComplete, and I’ve been pleasantly surprised to find that it doesn’t get in my way nearly as much as I would have expected. In comparison, I’ve had far more problems with the behaviour of the application under test rather than the execution environment, which says a lot for TestComplete’s ability to interact with horrifying VB6/WinForms chimera applications.
Its not all puppies and roses though. TestComplete is very much a tabs and text boxes kind of application and while parts of the tool have some nice Intellisense (more than I would have expected) its certainly no Visual Studio. In addition, binary files make meaningful diffs impossible, its licensing model is draconian at best and the price is insane.
But aside from those gripes, it certainly does get the job done.
When I decided that in order to test a piece of functionality involving the database, I would need to leverage some AWS EC2 images, I was unsurprised to find that it had support for executing Powershell scripts. I seems to be able to do just about anything really.
Well, I suppose support is a strong word.
Scripted Interactions
TestComplete supports a number of scripting languages for programmatically writing tests. There is also a very useful record-replay GUI which actually creates a programmatically accessible map of your application, which can then be used in scripts if you want.
When you create a TestComplete project, you can choose from VBScript, JScript and two other scripting varieties that slip my mind. After you’ve created the project, you don’t seem to be able to change this choice, so make sure you pick one you want to use.
I did not create the project.
The person who did picked JScript, which I have never written before. But at least it wasn't VBScript I suppose.
My understanding of JScript is limited, but I believe it is a Microsoft implementation of some Javascript standard. I have no idea which one, and its certainly not recent.
Anyway, I’m still not entirely sure how it all fits together, but from the JScript code you can access some of the .NET Framework libraries, like System.Management.Automation.
Using this namespace, you can create an object that represents a Powershell process, configure it with some commands, and then execute it.
Hence Powershell support.
I Have the Power
I’ve written about my growing usage of Powershell before, but I’ve found it to be a fantastically powerful tool. It really can do just about anything. In this case, I gravitated towards it because I’d already written a bunch of scripts for initialising AWS EC2 instances, and I wanted to reuse them to decrease the time it would take me to put together this functional test. For this test I would need to create an AMI containing a database in the state that I wanted it to be in, and then spin up instances of that AMI whenever I wanted to run the test.
In the end, as it is with these sorts of things, it wasn’t setting up the EC2 instances that took all of my time. That yak had already been shaved.
The first problem I had to solve was actually being able to run a Powershell script file using the component above.
Anyone who has ever used Powershell is probably very familiar with the Execution Policy of the system in question. Typically this defaults to Restricted, which means you are not allowed to execute scripts.
Luckily you can override this purely for the process you are about to execute, without impacting on the system at large. This, of course, means that you can change the Execution Policy without needing Administrative privileges.
Set-ExecutionPolicy RemoteSigned -Scope Process -Force |
The second problem I had to solve was getting results out of the Powershell component in TestComplete. In my case I needed to get the Instance ID that was returned by the script, and then store it in a TestComplete variable for use later on. I’d need to do the same thing with the IP address, obtained after waiting for the instance to be ready.
Waiting for an instance to be ready is actually kind of annoying. There are two things you need to check. The first is whether or not the instance is considered running. The second is whether or not the instance is actually contactable through its network interface. Waiting for an instance to be running only takes about 15-30 seconds. Waiting for an instance to be contactable takes 5-10 minutes depending on how capricious AWS is feeling. As you can imagine, this can make testing your “automatically spin up an instance” script a very frustrating experience. Long feedback loops suck.
When you execute a series of commands through the Powershell component, the return value is a collection of PSObjects. These objects are what would have normally been returned via the output stream. In my case I was returning a single string, so I needed to get the first entry in the collection, and then get its ImmediateBaseObject property. To get the string value I then had to get the OleValue property.
Tying both of the above comments together, here is the collection of functions that I created to launch Powershell from TestComplete.
function CreatePowershellExecutionObject()
{
var powershell = dotNET.System_Management_Automation.Powershell.Create();
// This is set before every script because otherwise we can't execute script files,
// Thats where most of our powershell lives.
powershell.AddScript("Set-ExecutionPolicy RemoteSigned -Scope Process -Force");
// This redirects the write-host function to nothingness, because otherwise any
// script executed through this componenent will fail if it has the audacity to
// write-host. This is because its non-interactive and write-host is like a special
// host channel thing that needs to be implemented.
powershell.AddScript("function write-host { }");
return powershell;
}
function InvokePowershellAndThrowIfErrors(powershell)
{
var result = powershell.Invoke();
if (powershell.HadErrors)
{
var firstError = powershell.Streams.Error.Item(0);
if (firstError.ErrorDetails != null)
{
throw new Error(firstError.ErrorDetails);
}
else
{
throw new Error(firstError.Exception.ToString());
}
}
return result;
}
function GetCommonPowershellAwsScriptArguments()
{
var awsKey = Project.Variables.AwsKey;
var awsSecret = Project.Variables.AwsSecret;
var awsRegion = Project.Variables.AwsRegion;
return "-SuppliedAwsKey " + awsKey + " -SuppliedAwsSecret " + awsSecret + " -SuppliedAwsRegion " + awsRegion + " "
}
function StartAwsVirtualMachineWithFullDatabaseAndSetInstanceId()
{
var gatewayVersion = GetVersionOfGatewayInstalled();
var scriptsPath = GetScriptsDirectoryPath();
var executeInstanceCreationScript = "& \"" +
scriptsPath +
"\\functional-tests\\create-new-max-size-db-ec2-instance.ps1\" " +
GetCommonPowershellAwsScriptArguments() +
"-BuildIdentifier " + gatewayVersion;
var powershell = CreatePowershellExecutionObject();
powershell.AddScript(executeInstanceCreationScript);
Indicator.PushText("Spinning up AWS EC2 Instance containing Test Database");
var result = InvokePowershellAndThrowIfErrors(powershell);
Indicator.PopText();
var instanceId = result.Item(0).ImmediateBaseObject.OleValue;
KeywordTests.WhenADatabaseIsAtMaximumSizeForExpress_ThenTheFilestreamConversionStillWorks.Variables.FullDbEc2InstanceId = instanceId;
}Notice that I split the instance creation from the waiting for the instance to be ready. This is an optimisation. I create the instance right at the start of the functional test suite, and then execute other tests while that instance is being setup in AWS. By the time I get to it, I don’t have to wait for it to be ready at all. Less useful when testing the database dependent test by itself, but it shaves 6+ minutes off the test suite when run together. Every little bit counts.
Application Shmaplication
Now that the test database was being setup as per my wishes, it was a simple matter to record the actions I wanted to take in the application and make some checkpoints for verification purposes.
Checkpoints in TestComplete are basically asserts. Record a value (of many varied types, ranging from Onscreen Object property values to images) and then check that the application matches those values when it is run.
After recording the test steps, I broke them down into reusable components (as is good practice) and made sure the script was robust in the face of failures and unexpected windows (and other things).
The steps for the executing the test using the application itself were easy enough, thanks to TestComplete.
Tidbits
I did encounter a few other things while setting this test up that I think are worth mentioning.
The first was that the test needed to be able to be run from inside our VPC (Virtual Private Cloud) in AWS as well as from our local development machines. Actually running the test was already a solved problem (solved when I automated the execution of the functional tests late last year), but making a connection to the virtual machine hosting the database in AWS was a new problem.
Our AWS VPC is fairly locked down (for good reason) so machines in there generally can’t access machines in the outside word except over a few authorised channels (HTTP and HTTPS through a proxy for example). Even though the database machine was sitting in the same VPC as the functional tests worker, I had planned to only access it through its public IP address (for simplicity). This wouldn’t work without additional changes to our security model, which would have been a pain (I have no control over those policies).
This meant that in one case I needed to use the Public IP Address of the instance (when it was being run from our development machines) and in the other I needed to use the Private IP Address.
Code to select the correct IP address fit nicely into my Powershell script to wait for the instance to be ready, which already returned an IP address. All I had to do was test the public IP over a specific port and depending on whether or not it worked, return the appropriate value. I did this using the TCPClient class in the .NET framework.
The second thing I had to deal with was a dependency change.
Previously our TestComplete project was not dependent on anything except itself. It could be very easily compressed into a single file and tossed around the internet as necessary. Now that I had added a dependency on a series of Powershell scripts I had to change the execution script for our Functional Tests to pack up and distribute additional directories. Nothing too painful luckily, as it was as simple enough matter to include more things in the compressed archive.
The final problem I ran into was with the application under test.
Part of the automated Functional Test is to open the database. When you ask the application to do that, it will pop up a fairly standard dialog with server & database name.
Being the helpful VB6 application that it is, it also does a search of the network to find any database servers that are available, so you can quickly select them from a dropdown. Being a single threaded application without a great amount of thought being put into user experience, the dialog freezes for a few moments while it does the search.
If you try to find that Window or any of its components using TestComplete while its frozen, the mapping for the dialog box changes from what you would expected (Aliases.Application.DialogName) to something completely different (Process(“ApplicationName”).Window(“DialogName”).Control(“Name”)). Since the test was recorded using the first alias, it then times out when looking for the control it expects, and fails.
I got around this by introducing a delay before even attempting to look for any controls on that dialog.
Depressingly, this is a common solution to solving automation problems like that.
Conclusion
If you were expecting this section to be where I tie everything together and impart on you some TL;DR lessons, then prepare yourself for disappointment.
The one thing I will say though is that setting up good automated Functional Tests takes a hell of a lot of effort. it gets easier as you spend more time doing it, but sometimes I question the benefits. Certainly its an interesting exercise and you learn a lot about the system under test, but the tests you create are usually fragile and overly complex.
Functional Tests that automate an application definitely shouldn’t be your only variety of test that’s for sure.
Regardless of the adventure above, TestComplete is pretty great, and it certainly makes the whole automated testing process much easier than it would be if you used something else.
Like CodedUI.
Which I have done before.
I don’t recommend it.