Almost a year ago I wrote a blog post detailing an adventure I had with ClickOnce, certificates and trusted software. It was exhausting and like most things certificate related, frustrating and confusing. I got it working in the end, and everything was good.

For about a year.

I wrote the blog post a few months after I dealt with the issue, which was a few months after the certificate had been issued.

Thus, a few months ago, the certificate expired.

The renewal was remarkably painless, all things considered. I still had access to the same Windows Server virtual machine that I used for the last certificate, which was nice, and more importantly, I wrote down everything I did last time so I wouldn’t have to discover it all again.

I got the new certificate, incorporated it into the build/publish process and everything was good again.

Or so I thought.

Testing Is Important

This is one of those cases where you realise that testing really is important, and that you shouldn’t slack on it.

It’s a little bit hard in this project, because I don’t technically actively work on it anymore, and it was pretty barebones anyway. No build server or anything nice like that, no automated execution of functional tests, just a git repository, Visual Studio solution and some build scripts.

When I incorporated the new certificate, I did a build/publish to development to make sure that everything was okay. I validated the development build on a Windows 7 virtual machine that I had setup for just that task. It seemed fine. Installed correctly over the top of the old version, kept settings, no errors or scary warning screens about untrusted software, so I didn’t think any more of it.

I didn’t need to re-publish to production, because the old build was timestamped, so it was still valid with the old certificate. Other than the certificate changes, there was nothing else, so no need to push a new version.

What I didn’t test was Windows 7 with .NET Framework 4.0 (or Windows XP, which can only have .NET Framework 4.0.

Last week I did some reskinning work for the same project, giving the WPF screens a minor facelift to match the new style guide that had been implemented on the website. Pretty straightforward stuff in the scheme of things, thanks to WPF. I switched up some button styles, tweaked a few layouts and that was that.

Build publish to all 3 environments, validated that it worked (again on my normal Windows 7 test instance) and I moved on to other things.

Then the bug reports started coming in.

Software That Doesn’t Work Is Pretty Bad

Quite a few people were unable to install the new version. As a result, they were unable to run the software at all, because that’s how ClickOnce rolls. The only way they could start up, was to disconnect from the internet (so it would skip the step to check for new versions) and then reconnect after the software had started.

The reports all featured the same error screen:

That’s basically the default ClickOnce failure screen, so nothing particularly interesting there. It complains about the format of the application being wrong. Generic error, nothing special. The log though, that was interesting:

Below is a summary of the errors, details of these errors are listed later in the log.

* Activation of [REDACTED – Installation Location] resulted in exception. Following failure messages were detected:
+ Exception reading manifest from [REDACTED – Manifest Location]: the manifest may not be valid or the file could not be opened.
+ Manifest XML signature is not valid.
+ SignatureDescription could not be created for the signature algorithm supplied.

So apparently I have an invalid signature, which, of course, is related to certificates. Considering it all worked fine on newer/updated machines, my mind immediately went to the predictable thorn in my side, the differences between .NET 4 and .NET 4.5. Windows XP can never have .NET 4.5, and Windows 7 boxes that haven’t been updated to the latest version won’t have .NET 4.5 either.

A quick search showed that .NET only supports SHA-1 and my manifests were signed with SHA-256.

But what did I do that caused it to stop working? I worked perfectly well before I published my reskinning changes, and they were pure XAML, nothing special.

If you’ve been paying attention, what I did was obvious. I changed the certificate.

To me, that wasn’t obvious, because I changed the certificate a few months ago, and I’d already forgotten about it.

After a few hours of investigation, I came across some reports of Visual Studio 2013 not signing ClickOnce applications targeting .NET 4 with the proper algorithm. My new certificate was SHA-256, so this sounded plausible. The problem was, this particular bug was fixed in Visual Studio 2013 Update 3, and I was running Update 4.

This StackOverflow post in particular was helpful, because it reminded me that I was using Mage to re-sign the manifests after I used SignTool to sign the assembly.

Apparently the fix they made in Visual Studio Update 3 was never incorporated into Mage, so you quite literally cannot choose to sign the manifests with SHA-1 using an SHA-256 certificate, even if you have to.

Fixing the Match

The fix was actually pretty elegant, and I’m not sure why I didn’t do the whole signing thing this way in the first place.

Instead of writing custom code to sign the manifests again when signing the executable after building/publishing via MSBuild, I can just call MSBuild twice, once for the build and once for the publish. In between those two calls, I use SignTool to sign the executable, and everything just works.

$msbuild = "C:\Program Files (x86)\MSBuild\12.0\bin\msbuild.exe"
$solutionFile = $PSScriptRoot + "\[SOLUTION_FILE_NAME]"

.\tools\nuget.exe restore $solutionFile

$msbuildArgs = '"' + $solutionFile + '" ' + '/t:clean,rebuild /v:minimal /p:Configuration="' + $configuration + '"'
& $msbuild $msbuildArgs

if($LASTEXITCODE -ne 0)
{
    write-host "Build FAILURE" -ForegroundColor Red
    exit 1
}

$appPath = "$msbuildOutput\$applicationName.application"
$timestampServerUrl = "http://timestamp.comodoca.com/authenticode"

$certificateFileName = "[CERTIFICATE_NAME].pfx"
$intermediateCertificateFileName = "[INTERMEDIATE_CERTIFICATE_NAME].cer"
$executableToSign = "[PATH_TO_EXECUTABLE]"

$certificatesDirectory = $PSScriptRoot + "[CERTIFICATES_DIRECTORY]"
$certificateFilePath = "$certificatesDirectory\$certificateFileName"
$intermediateCertificateFilePath = "$certificatesDirectory\$intermediateCertificateFileName"

& tools\signtool.exe sign /f "$certificateFilePath" /p "$certificatePassword" /ac "$intermediateCertificateFilePath" -t $timestampServerUrl "$executableToSign"
if($LASTEXITCODE -ne 0)
{
    write-error "Signing Failure"
    exit 1
}

$msbuildArgs = '"' + $solutionFile + '" ' + '/t:publish /v:minimal /p:Configuration="' + $configuration + '";PublishDir=' + $msbuildOutput + ';InstallUrl="' + $installUrl + '";IsWebBootstrapper=true;InstallFrom=Web'
& $msbuild $msbuildArgs

The snippet above doesn’t contain definitions for all of the variables, as it is an excerpt from a larger script. It demonstrates the concept though, even if it wouldn’t execute by itself.

Using this approach, Visual Studio takes care of using the correct algorithm for the target framework, the executable is signed appropriately and everyone is happy.

This meant that I could delete a bunch of scripts that dealt with signing and re-signing, which is my favourite part of software development. Nothing feels better than deleting code that you don’t need anymore because you’ve found a better way.

Conclusion

Certificates are definitely one of those things in software that I never quite get a good handle on, just because I don’t deal with them every day. I have a feeling that if I was dealing with certificates more often, I probably would see these sorts of things coming, and be able to prepare accordingly.

More importantly though, this issue would have been discovered immediately if we had a set of functional tests that validate that the software can be installed correctly on a variety of operation systems, specifically those that we officially support. Obviously it would be better if the tests were automated, but even having a manual set of tests would have made the issue obvious as soon as I made the change, instead of months later when I make an unrelated change.

Of course, I should have run such tests at the time, but that’s why automation is important, to prevent that exact situation.

That or hiring a dedicated tester who likes to do that sort of thing manually.

Because I don’t

Update: I ran into an issue with the script used in this post to do the signing when using an SHA-256 certificate (i.e. a newer one). You wrote another post describing the issue and solution here.

God I hate certificates.

Everything involving them always seems to be painful. Then you finally get the certificate thing done after hours of blood, sweat and pain, put it behind you, and some period of time later, the certificate expires and it all happens again. Of course, you’ve forgotten how you dealt with it the first time.

I’ve blogged before about the build/publish script I made for a ClickOnce WPF application, but I neglected to mention that there was a certificate involved.

Signing is important when distributing an application through ClickOnce, as without a signed installer, whenever anyone tries to install your application they will get warnings. Warnings like this one.

For a commercial application, that’s a terrible experience. Nobody will want to install a piece of software when their screen is telling them that “the author of the software is unknown”. And its red! Red bad. Earlier versions of Internet Explorer weren’t quite as hostile, but starting in IE9 (I think) the warning dialog was made significantly stronger. Its hard to even find the button to override the warning and just install the damn thing (Options –> More Options –> Run Anyway, which is a really out of the way).

As far as I can tell, all ClickOnce applications have a setup.exe file. I have no idea if you can customise this, but its essentially just a bootstrapper for the .application file which does some addition checks (like .NET Framework version).

Anyway, the appropriate way to deal with the above issue is by signing the ClickOnce manifests.

You need to use an Authenticode Code Signing Certificate, from a trusted Certificate Authority. These can range in price from $100 US to $500+ US. Honestly, I don’t understand the difference. For this project, we picked up one from Thawte for reasons I can no longer remember.

There’s slightly more to the whole signing process than just having the appropriate Certificate and Signing the installer. Even with a fully signed installer, Internet Explorer (via SmartScreen) will still give a warning to your users when they try to install, saying that “this application is not commonly downloaded”. The only way around this is to build up reputation with SmartScreen, and the only way to do that is slowly, over time, as more and more people download your installer. The kicker here is that without a certificate the reputation is tied to the specific installer, so if you ever make a new installer (like for a version update) all that built up reputation will go away. If you signed it however, the reputation accrues on the Certificate instead.

Its all very convoluted.

Enough time has passed between now and when I bought and setup the certificate for me to have completely forgotten how I went about it. I remember it being an extremely painful process. I vaguely recall having to generate a CSR (Certificate Signing Request), but I did it from Windows 7 first accidentally, and you can’t easily get the certificate out if you do that, so I had to redo the whole process on Windows Server 2012 R2. Thawte took ages to process the order as well, getting stuck on parts of the certification process a number of times.

Once I exported the certificate (securing the private key with a password) it was easy to incorporate it into the actual publish process though. Straightforward configuration option inside the Project properties, under Signing. The warning went from red (bad) to orange (okayish). This actually gives the end-user a Run button, instead of strongly recommending to just not run this thing. We also started gaining reputation against our Certificate, so that one day it would eventually be green (yay!).

Last week, someone tried to install the application on Windows 8, and it all went to hell again.

I incorrectly assumed that once installed, the application would be trusted, which was true in Windows 7. This is definitely not the case in Windows 8.

Because the actual executable was not signed, the user got to see the following wonderful screen immediately after successfully installing the application (when it tries to automatically start).

Its the same sort of thing as what happens when you run the installer, except it takes over the whole screen to try and get the message across. The Run Anyway command is not quite as hidden (click on More Info) but still not immediately apparent.

The root cause of the problem was obvious (I just hadn’t signed the executable), but fixing it took me at least a day of effort, which is a day of my life I will never get back. That I had to spend in Certificate land. Again.

First Stab

At first I thought I would just be able to get away with signing the assembly. I mean, that option is directly below the configuration option for signing the ClickOnce manifests, so they must be related, right?

I still don’t know, because I spent the next 4 hours attempting to use my current Authenticode Code Signing Certificate as the strong name key file for signing the assembly.

I got an extremely useful error message.

Error during Import of the Keyset: Object already exists.

After a bit of digging it turns out that if you did not use KeySpec=2 (AT_SIGNATURE) during enrollment (i.e. when generating the CSR) you can’t use the resulting certificate for strong naming inside Visual Studio. I tried a number of things, including re-exporting, deleting and then importing the Certificate trying to force AT_SIGNATURE to be on, but I did not have any luck at all. Thawte support was helpful, but in the end unable to do anything about it.

Second Stab

Okay, what about signing the actual executable? Surely I can use my Authenticode Code Signing Certificate to sign the damn executable.

You can sign an executable (not just executables, other stuff too) using the CodeSign tool, which is included in one of the Windows SDKs. I stole mine from “C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\bin”. The nice thing is that its an (entirely?) standalone application, so you can include it in your repository in the tools directory so that builds/publish

Of course, because I’m publishing the application through ClickOnce its not just as simple as “sign the executable”. ClickOnce uses the hash of files included in the install in the generation of its .manifest file, so if you sign the executable after ClickOnce has published to a local directory (before pushing it to the remote location, like I was doing) it changes the hash of the file and the .manifest is no longer valid.

With my newfound Powershell skills (and some help from this awesome StackOverflow post), I wrote the following script.

param
(
    $certificatesDirectory,
    $workingDirectory,
    $certPassword
)

if ([string]::IsNullOrEmpty($certificatesDirectory))
{
    write-error "The supplied certificates directory is empty. Terminating."
    exit 1
}

if ([string]::IsNullOrEmpty($workingDirectory))
{
    write-error "The supplied working directory is empty. Terminating."
    exit 1
}

if ([string]::IsNullOrEmpty($certPassword))
{
    write-error "The supplied certificate password is empty. Terminating."
    exit 1
}

write-output "The root directory of all files to be deployed is [$workingDirectory]."

$appFilesDirectoryPath = Convert-Path "$workingDirectory\Application Files\[PUBLISH DIRECTORY ROOT NAME]_*\"

write-output "The application manifest and all other application files are located in [$appFilesDirectoryPath]."

if ([string]::IsNullOrEmpty($appFilesDirectoryPath))
{
    write-error "Application Files directory is empty. Terminating."
    exit 1
}

#Need to resign the application manifest, but before we do we need to rename all the files back to their original names (remove .deploy)
Get-ChildItem "$appFilesDirectoryPath\*.deploy" -Recurse | Rename-Item -NewName { $_.Name -replace '\.deploy','' }

$certFilePath = "$certificatesDirectory\[CERTIFICATE FILE NAME]"

write-output "All code signing will be accomplished using the certificate at [$certFilePath]."

$appManifestPath = "$appFilesDirectoryPath\[MANIFEST FILE NAME]"
$appPath = "$workingDirectory\[APPLICATION FILE NAME]"
$timestampServerUrl = "http://timestamp.globalsign.com/scripts/timstamp.dll"

& tools\signtool.exe sign /f "$certFilePath" /p "$certPassword" -t $timestampServerUrl "$appFilesDirectoryPath\[EXECUTABLE FILE NAME]"
if($LASTEXITCODE -ne 0)
{
    write-error "Signing Failure"
    exit 1
}

# mage -update sets the publisher to the application name (overwriting any previous setting)
# We could hardcode it here, but its more robust if we get it from the manifest before we
# mess with it.
[xml] $xml = Get-Content $appPath
$ns = New-Object System.Xml.XmlNamespaceManager($xml.NameTable)
$ns.AddNamespace("asmv1", "urn:schemas-microsoft-com:asm.v1")
$ns.AddNamespace("asmv2", "urn:schemas-microsoft-com:asm.v2")
$publisher = $xml.SelectSingleNode('//asmv1:assembly/asmv1:description/@asmv2:publisher', $ns).Value
write-host "Publisher extracted from current .application file is [$publisher]."

# It would be nice to check the results from mage.exe for errors, but it doesn't seem to return error codes :(
& tools\mage.exe -update $appManifestPath -certFile "$certFilePath" -password "$certPassword"
& tools\mage.exe -update $appPath -certFile "$certFilePath" -password "$certPassword" -appManifest "$appManifestPath" -pub $publisher -ti $timestampServerUrl

#Rename files back to the .deploy extension, skipping the files that shouldn't be renamed
Get-ChildItem -Path "$appFilesDirectoryPath"  -Recurse | Where-Object {!$_.PSIsContainer -and $_.Name -notlike "*.manifest"} | Rename-Item -NewName {$_.Name + ".deploy"}

Its not the most fantastic thing I’ve ever written, but it gets the job done. Note that the password for the certificate is supplied to the script as a parameter (don’t include passwords in scripts, that’s just stupid). Also note that I’ve replaced some paths/names with tokens in all caps (like [PUBLISH DIRECTORY ROOT NAME]) to protect the innocent.

The meat of the script does the following:

• Locates the publish directory (which will have a name like [PROJECT NAME]_[VERSION]).
• Removes all of the .deploy suffixes from the files in the publish directory. ClickOnce appends .deploy to all files that are going to be deployed. I do not actually know why.
• Signs the executable.
• Extracts the current publisher from the manifest.
• Updates the .manifest file.
• Updates the .application file.
• Restores the previously removed .deploy suffix.

You may be curious as to why the publisher is extracted from the .manifest file and then re-supplied. This is because if you update a .manifest file and you don’t specify a publisher, it overwrites whatever publisher was there before with the application name. Obviously, this is bad.

Anyway, the signing script is called after a build/publish but before the copy to the remote location in the publish script for the application.

Conclusion

After signing the executable and ClickOnce manifest, Windows 8 no longer complains about the application, and the installation process is much more pleasant. Still not green, but getting closer.

I really do hate every time I have to interact with a certificate though. Its always complicated, complex and confusing and leaves me frustrated and annoyed at the whole thing. Every time I learn just enough to get through the problem, but I never feel like I understand the intricacies enough to really be able to do this sort of thing with confidence.

Its one of those times in software development where I feel like the whole process is too complicated, even for a developer. It doesn’t help that not only is the technical process of using a certificate complicated, but even buying one is terrible, with arbitrary price differences (why are they different?) and terrible processes that you have to go through to even get a certificate.

At least this time I have a blog, and I’ve written this down so I can find it when it all happens again and I’ve purged all the bad memories from my head.