0 Comments

Like any normal, relatively competent group of software developers, we write libraries to enable reuse, mostly so we don’t have to write the same code a thousand times. Typically each library is small and relatively self contained, offering only a few pieces of highly cohesive functionality that can be easily imported into something with little to no effort. Of course, we don’t always get this right, and there are definitely a few “Utility” or “Common” packages hanging around, but we do our best.

Once you start building and working with libraries in .NET, the obvious choice for package management is Nuget. Our packages don’t tend to be released to the greater public, so we don’t publish to Nuget.org directly, we just use the built-in Nuget feed supplied by TeamCity.

Well we did, up until recently, when we discovered that the version of TeamCity we were using did not support the latest Nuget API and Visual Studio 2015 didn’t like the old version of the Nuget feed API. This breaking change was opportune, because while it was functional, the TeamCity Nuget feed had been bugging us for a while. Its difficult to add packages that weren’t created by a Buidl Configuration and packages are intrinsically tied to the Artefacts of a Build Configuration, so they can be accidentally deleted when cleaning up. So we decided to move to an actual Nuget server, and we chose Myget because we didn’t feel like rolling our own.

That preface should set the stage for the main content of this post, which is about dealing with the difficulties in debugging when a good chunk of your code is in a library that was prepared earlier.

Step In

To me, debugging is a core part of programming.

Sure you can write tests to verify functionality and you can view your resulting artefact(webpage, application, whatever), but at some pointits incredibly helpful to be able to step through your code, line by line, in order to get to the bottom of something (usually a bug).

I’m highly suspicious of languages and development environments that don’t offer an easy way to debug, and of developers that rely on interpreting log statements to understand their code. I imagine this is probably because of a history of using Visual Studio, which has had all of these features forever, but trying to do anything but the most basic actions without a good IDE is hellish. Its one of the many reasons why I hate trying to debug anything in a purely command line driven environment (like some flavours of Linux or headless Windows).

I spend most of my time dealing with two languages, C# and Powershell, which are both relatively easy to get a good debugging experience. Powershell ISE is functional (though not perfect) and Visual Studio is ridiculously feature rich when it comes to debugging.

To tie all of this back in with the introduction though, debugging into C# libraries can be somewhat challenging.

In order to debug C# code you need debugging information in the form of PDB files. These files either need to be generated at compile time, or they need to be generated from the decompiled DLL before debugging. PDB files give the IDE all of the information it needs in order to identify the constituent components of the code, including some information about the source files the code was compiled from. What these files don’t give, however, is the actual source code itself. So while you might have the knowledge that you are in X function, you have no idea what the code inside that function is, nor what any of the variables in play are.

Packaging the PDB files along with the compiled binaries is easy enough using Nuget, assuming you have the option enabled to generate them for your selected Build Configuration.

Packaging source files is also pretty easy.

The Nuget executable provides a –symbols parameter that will automatically create two packages, the normal one and another one containing all of the necessary debug information + all of your source files as of the time of compile.

What do you do with it though?

Source Of All Evil

Typically, if you were uploading your package to Nuget.org, the Nuget executable would automatically upload the symbols package to SymbolSource. SymbolSource is a service that can be configured as a symbol server inside Visual Studio, allowing the application to automatically download debugging information as necessary (including source files).

In our case, we were just including the package as an artefact inside TeamCity, which automatically includes it in the TeamCity Nuget feed. No symbol server was involved.

In fact, I have a vague memory of trying to include both symbol and non-symbol packages and having it not work at all.

At the time, the hacky solution was put in place to only include the symbols package in the artefacts for the Build Configuration in TeamCity, but to rename it to not include the word symbols in the filename. This kept TeamCity happy enough, and it allowed us to debug into our libraries when we needed to (the PDB files would be automatically loaded and it was a simple matter to select the source file inside the appropriate packages directory whenever the IDE asked for them).

This worked up until the point where we switched to Myget.

Packages restored from Myget no longer had the source files inside them, leaving us in a position where our previous approach no longer worked. Honestly, our previous approach was a little hacky, so its probably for the best that it stopped working.

Sourcery

It turns out that Myget is a bit smarter than TeamCity when it comes to being a Nuget feed.

It expects you to upload both the normal package and the symbols package, and then it deals with all the fun stuff on the server. For example, it will automatically interpret and edit the PDB files to say that the source can be obtained from the Myget server, rather than from wherever the files were when it was compiled and packaged.

Our hacky solution to get Symbols and Source working when using TeamCity was getting in the way of a real product.

Luckily, this was an easy enough problem to solve while also still maintaining backwards compatibility.

Undo the hack that enforced the generation of a single output from Nuget (the renamed symbols package), upload both files correctly to Myget and then upload just a renamed symbols package to TeamCity separately.

Now anything that is still using TeamCity will keep working in the same way as it always was (manual location of source files as necessary) and all of the projects that have been upgraded to use our new Myget feed will automatically download both symbols and source files whenever its appropriate (assuming you have the correct symbol server setup in Visual Studio).

Conclusion

Being able to debug into the source files of any of your packages automatically is extremely useful, but more importantly, it provides a seamless experience to the developer, preventing the waste of time and focus that occurs when flow gets interrupted in order to deal with something somewhat orthogonal to the original task.

The other lesson learned (which I should honestly have engraved somewhere highly visible by now) is that hacked solutions always have side effects that you can’t foresee. Its one of those unfortunate facts of being a software developer.

From a best practices point of view, its almost always better to fully understand the situation before applying any sort of fix, but you have to know how to trade that off against speed. We could have chosen to implement a proper Nuget server when we first ran across the issue with debugging our libraries, rather than hacking together a solution to make TeamCity work approximately like we wanted, but we would have paid some sort of opportunity cost in order to do that.

At the least, what we should have done was treat symbols package rename for TeamCity in a isolated fashion, so the build still output the correct packages.

This would have made it obvious that something special was happening for TeamCity and for TeamCity only, making it easier to work with a completely compliant system at a later date.

I suppose the lesson is, if you’re going to put a hack into place, make sure its a well constructed hack.

Which seems like a bit of an oxymoron.