If you’re writing line of business software of almost any description, you probably need to generate reports.

In the industry that I’m current working in, these reports are typically financial, showing the movement of money relating to a trust account.

Our legacy product has been around for a very very long time, so it has all the reports you could need. Unfortunately, they are generated using an ancient version of crystal reports, which I’m pretty sure is possessed by a demonic entity, so they can be a bit of a nightmare to maintain.

For our new cloud product, I’m not entirely familiar with how we deal with reports, but I think that they are generated using HTML and then converted to PDF. Its enough of a feature that there’s a whole reporting subsystem dedicated to the task).

Unfortunately, our most recent development efforts in the legacy product fall somewhere in the middle between terrifying ancient evil and relatively modern report generation processes.

The crystal reports component is VB6 native, and all of our new functionality is written in C# (using WPF for the view layer). We could call back into the VB6 to generate a report, but honestly, I don’t want to touch that with a ten-foot pole. We can’t easily leverage the HTML/PDF generation capabilities of the new cloud product either, as it was never built to be reused by an entirely different domain.

As a result, we’ve mostly just shied away from doing reports as a part of new features.

Our latest feature is a little different though, as it is an automated receipting solution, and a report of some description is no longer optional.

Last responsible moment and all that.

Forming An Opinion

If you’re trying to build out a report in WPF, you’d think that there would be a component there all ready to go.

You’d be mostly wrong, at least as far as native WPF is concerned. There are a few bits and pieces around, but nothing particularly concrete or well put together (at least as far as we could determine anyway).

Instead, most people recommend that you use the Windows Forms Report Viewer, and the systems that it is built on.

We poked at this for a little while, but it just seemed so…archaic and over complicated. All we wanted was to take a View Model describing the report (of our design) and Bind it, just like we would for a normal WPF view.

Enter the FlowDocument.

In The Flow

I’ll be honest, I didn’t actually do the work to build a report out in WPF using FlowDocuments, so most of what I’m writing here is second hand knowledge. I didn’t have to live through the pain, but the colleague that did it assures me that it was quite an ordeal.

At their core, FlowDocuments allow you to essentially create a document (like a newspaper article or something similar) in WPF. They handle things like sizing the content to the available area, scrolling and whatnot, all with the capability to render normal XAML controls alongside textual constructs (paragraphs, images, etc).

There are a few things that they don’t do out of the box though:

• Pagination when printing. The default paginator is mostly concerned with making pages for a screen (rather than a printed document), and doesn’t allow for headers or footers at all. As a result, we implemented a custom DocumentPaginator that did what we needed it to do.
• Templated content repetition. If you’re using WPF, and MVVM, you’re probably familiar with the ItemsControl (or its equivalents). If you want to do something similar in a FlowDocument though (i.e. bind to a list of things), you’ll need to put together a custom templating system. This is relevant to us because our report is mostly tabular, so we just wanted a relatively simple repeater.

With those bits and pieces out of the way though, what you get is a decent component that you can use on top of a view model that displays the report to the user in the application (i.e. scrolling, text selection, etc) with the capability to print it out to any of the printers they have available.

Its not exactly a ground breaking advance in the high-tech field of report generation, but it gets the job done without any heavyweight components or painful integrations.

Conclusion

I’m sure there are hardcore reporting components out there that are built in pure WPF and do exactly what we want, but we just couldn’t find them.

Instead, we settled for knocking together some extensions to the existing FlowDocument functionality that accomplished exactly what we needed and no more.

With a little bit of effort, we could probably make them more generic and reusable, and I might even look into doing that at some point in the future, but to be honest now that we’ve done the one report that we needed to do, we’ll probably mostly forget about it.

Until the next time of course, then we’ll probably wonder why we didn’t make it generic and reusable in the first place.

As I’ve written about a bunch of times before, we have a data synchronization process for freeing customer data from restrictive confines of their office. We built it to allow the user to get at their valuable information when they are on the road, as our users tend to be, as something of a stopgap while we continue to develop and improve our cloud platform.

A fantastic side effect of having the data is that we can make it extremely easy for our customers to migrate from our legacy product to the new cloud platform, because we’ve already dealt with the step of getting their information into a place where we can easily access it. There are certainly challenges inherent in this approach, but anything we can do to make it easier to move from one system to another is going to make our customers happier when they do decide to switch.

And happy customers are the best customers.

As is always the case with software though, while the new cloud platform is pretty amazing, its still competing with a piece of software that’s been actively developed for almost 20 years, and not everyone wants to make the plunge just yet.

In the meanwhile, it would be hugely beneficial to let our customers use at least some part of the new system that we’ve been developing both because we honestly believe its a simpler and more focused experience and because the more people that use the new system, the more feedback we get, and that equals a better product for everyone.

Without a link between the two systems though, its all very pointless. Customers aren’t going to want to maintain two sets of identical information, and rightly so.

But what if we did it for them?

Data…Remember Who You Were

We already have a decent automated migration process. Its not perfect, but it ticks a lot of boxes with regards to bringing across the core entities that our customers rely on on a day to day basis.

All we need to do is execute that migration process on a regular basis, following some sort of cadence and acceptable latency that we figure out with the help of some test users.

This is all very good in theory, but the current migration process results in a brand new cloud account every time its executed on a customers data, which is pretty unusable for if we’re trying to make the customers lives easier. Ideally we want to re-execute the migration, but targeting an existing account. New entities get created, existing entities get updated and deleted entities get deleted.

Its basically another sync algorithm, except this time its more complicated. The current on-premises to remote sync is a relatively simple table by table, row by row process, because both sides need to look identical at the end. This time though, transforms occur, entities are broken down (or consolidated) and generally there is no one-to-one relationship that’s easy to follow.

On the upside, the same transformations are executed for both the initial migration and the update, so we can deal with the complexity somewhat by simply remembering the outcome of the previous process (actually all previous processes), and using that to identify what the destination was for all previously migrated entities.

What we’re left with is an update process that can be executed on top of any previous migration, which will:

• Identify any previously migrated entities by their ID’s, and perform appropriate update or replacement operations
• Identify new entities and perform appropriate creation operations, storing the ID mapping (i.e. original to new) in a place where it can be retrieved later

But what about deletions?

Well, at this point in time we’re just prototyping, so we kind of just swept that under the rug.

I’m sure it will be easy.

I mean, look at how easy deletions were to handle in the original data synchronization algorithm.

Its All Just Too Much

Unfortunately, while the prototype I described above works perfectly fine to demonstrate that an account in the cloud system can be kept up to date with data from the legacy product, we can’t give it to a customer yet because every migration “update” is a full migration of all of the customers data, not just the stuff that’s changed

Incredibly wasteful.

For our test data set, this isn’t really a problem. A migration only takes a minute at most, usually less, and its only really moving tends of entities around.

For a relatively average customer though, a migration takes like 45 minutes and moves thousands to tens of thousands of entities. Its never really been a huge problem before, because we only did it once or twice while moving them to the new cloud platform, but it quickly becomes unmanageable if you want to do it in some sort of constant loop.

Additionally, we can’t easily use a dumb timer to automate the process or we’ll get into problems with re-entrancy, where two migrations for the same data are happening at the same time. Taking that into account though, even if we started the next migration the moment the previous one finished (in some sort of eternal chain), we’re still looking at a total latency that is dependent on the size of the customers complete data set, not just the stuff they are actively working with. Also, we’d be using resources pointlessly, which is a scaling nightmare.

We should only be acting on a delta, using only the things that have changed since the last “update”.

A solvable problem, but it does add to the overall complexity, as we can’t fall back on the row based versioning that we used for the core data synchronization algorithm because of the transforms, so we have to create some sort of aggregate version from the constituent elements and store that for later use.

Its Time To Move On

Even if we do limit our migration updates to only the things that have changed since last time, we still have a problem if we’re just blindly running the process on a timer (or eternal chain). If there are no changes, why do a bunch of work for nothing?

Instead, we should be able to reactto the changes as the existing synchronization system detects them.

To be clear, the existing process is also timer based, so its not perfect. It would be nice to not add another timer to the entire thing though.

With some effort we should be able to produce a stream of information from our current sync API that describes the changes being made. We can then consume that stream in order to optimise the migration update process, focusing only on those entities that have changed, and more importantly, only doing work when changes actually occur.

Conclusion

Ignoring the limitations for a moment, the good news is that the prototype was enough to prove to interested stakeholders that the approach has merit. At least enough merit to warrant a second prototype, under the assumption that the second prototype  will alleviate some of the limitations and we can get it in front of some real customers for feedback.

I have no doubt that there will be some usability issues that come out during the user testing, but that’s to be expected when we do something as lean as this. Whether or not those usability issues are enough to cause us to run away from the idea altogether is the different question, but we won’t know until we try.

Even if the idea of using two relatively disparate systems on a day to day basis is unpalatable, we still get some nice things from being able to silently migrate our customers into our cloud platform, and keep that information up to date

I mean, what better way is there to demo a new piece of software to a potential customer than to do it with their own data?

A short post this week to talk about something that should be obvious from a big picture point of view, and is topical because I’ve experienced it myself recently.

If you are sick, don’t come into work.

If you have been sick, but think you’re better now, still don’t come into work.

The world will continue to turn without you for a little while, and even if it doesn’t, maybe its better to find that our now, rather than later.

Patient Zero

To be clear, I am definitely not a qualified epidemiologist, but for arguments sake lets assume it is impossible to spread disease if you are not near other people.

The engineer in me knows that there are edge cases that circumvent this “rule”, for example indirect infection vectors like contaminated food and other things, but the engineer in me should learn to be quiet.

If you’re trying to maintain velocity in a development team (or throughput, choose your poison), then the spread of sickness can be one of your worst enemies. If you’re lucky everyone gets sick at once, and its all over quickly, but if you’re not its drawn out over the course of a few weeks (or months!) and is hugely disruptive to getting anything done.

As a sick person, the best thing you can do for your team is not make them sick. Sure, if you completely extract yourself from the situation then whatever it is that you are working on will suffer. That’s a fair call.

But all the other things? They will keep trucking along.

Honestly, if the thing you were working on was important enough, someone else will do it anyway, especially if its blocking their work. This is the worst case scenario though (at least as far as work-in-progress is concerned), and all it really results in is some repetition of work already done.

I’d much rather pay that price than the cascading effect that repeated waves of sickness have on development.

Centre for Disease Control

If you’re in a position of leadership, you need to make sure that your messaging is clear, and your follow up behaviour is consistent. If you’re spouting the rhetoric of “if you’re sick, stay away” but you then get annoyed when a deadline is missed or work is not progressing, you’re doing it wrong.

You need to play the long game, and trust in the fact that even though something might suffer now, you’ll be better off in the long run in terms of predictability of delivery if everyone is playing by the same rules

Organizationally, the capability to reliably and effectively work remotely is also a huge boon in this situation and can help mitigate the effect of not being co-located during the period of sickness. Don’t get me wrong, there is still a significant amount of value in reinforcing that your people should take care of themselves first, before they think about work, but the ability to work remotely allows them to ease themselves back in, even if they are not feeling 100%, and still prevents further infection.

Other things that help are:

• Small work items and low cycle time, which mostly limit the impact of a single person disappearing for a period of time
• Consistent communication, such that everyone on the team is generally aware of everything else that is going on
• Well rounded people who can do anything (even if they might be better at a small number of things), because you’re going to hurt if the only person who knows how to work on system X is quarantining themselves for a few weeks

Conclusion

In summary, if you feel sick, or if you think you might be sick at all, just stay the hell away.

When you feel better, work remotely until at least a few days have passed, just in case.

Of course, everything that I’ve written here assumes that your people use the system honestly and fairly, but really, if you find yourself in a situation where you can’t trust them in that way, then I don’t understand why they would be working with you in the first place.

To put it into context, think about how stupid the following sentence is:

“I don’t trust you to be honest that you’re sick, but I trust you to write this million dollar software critical to the existence of our business”.

You've run across a blog post that has moved!

I've got a new blog now and I'm moving over bits and pieces of content depending on my need. Also, I don't have an easy mechanism to do a permanent redirect, so I have to settle for this instead.

Click here to see the exactly same content that used to be here, except now it also has a picture.

I am sick.

I’m not really all that sad though.

Just sick.

Posts resume next week (probably).