A new year means means more blog posts, and there is no better time to start than now.

Or maybe a week ago I suppose when the new year actually started, but I was on holidays, and writing blog posts while I’m on holidays just seems wrong. Blog posts are written on the train on my way to work, and that pattern is far too ingrained to do anything about now.

Anyway, that’s probably enough rambling, so lets get on with the show and discuss software prototypes, because I have opinions and this is the internet.

Its Code, But You Throw It Away

Software prototyping is simple in concept, but quickly gets complicated in execution.

Typically a prototype consists of some engineers throwing something together, probably ignoring normal engineering practices, to prove an idea or approach. Then, once its served its purpose, those same engineers toss it in the garbage.

The goal is to learn, not to create a long lasting artefact, and that is often where prototypes become dangerous. If a business sees something working (and a prototype probably works, even though it might have rough edges), then it might be inclined to make plans based on that. Perhaps attempt to push it out to a larger audience than was originally intended, or to start making claims that a feature is complete and ready to use.

Its a horrible feeling, watching the terrifying hacked together piece of code meant to prove a possibility become a core part of a business process. Especially so when you’re the one responsible for maintaining it, probably because you’re the only one who knows how it works.

Like I said, simple in concept, but complicated in the long run.

Of course, a prototype does not strictly have to be thrown away, but in my opinion, if you’re not throwing it away at the end, you’re probably just doing iterative development. If that’s the case, you really should be following good engineering practices all the way through instead of hacking something together and then trying to build on top of unstable foundations later.

Building Things To Answer The Wrong Questions

This blog post exists because we built a prototype recently.

I’m sure you think that the next paragraph is going to describe the situation where it “accidentally” became a core part of the business and its causing all sorts of problems, but that is surprisingly not the case. Everyone involved understood the purpose and limitations of the prototype and it was abandoned at the appropriate time, once it had served its purpose.

I had a completely different issue with our prototype experience; we probably shouldn’t have built one at all, and the construction of the prototype felt like it was wasted effort.

The situation we found ourselves in was that we wanted to provide some new functionality to the users of our legacy application that leveraged our new and shiny cloud platform. Kind of like a typical integration, with two different systems working in tandem, but we had a lot of control over both sides.

We prototyped the process for getting the two systems to talk to each other, with the plan that once we had that working at least partially, we could go and have early conversations with customers to see if they wanted to use it and how.

The reality was that the actual data flow between the two systems never really came up in any of those early conversations, as the topics covered were almost entirely focused around the new features available. We already had a one-off data migration process that would initialize the cloud system with information from the legacy software, and honestly, that would have been enough to start the conversation.

So the first mark against the prototype was that it just didn’t feel like its existence made a difference.

Hindsight Is Misleading

To be fair, I could very well be suffering from the curse of hindsight. Being able to look back at a situation with current knowledge and see a much more efficient way to do it is not really surprising after all. That’s how learning works.

Or it could be that we simply held on to the prototype for too long, and should have switched into constructing it (iteratively) for real sooner. Possibly as soon as we had answered the question “is it even possible?”.

Instead we held on to the prototype while we engaged with customers because we wanted to give them a sense of how the system would work in practice. Of course, because we were asking them to do real work in an environment that would one day be thrown away, they were rightly resistant, so not only did we gain little to nothing from throwing together the process from a customer conversation point of view, we actually made it harder to engage with them in relation to trying out the system for real.

If we had of simply started building the process out, piece by piece, following good engineering practices, we probably would have ended up at the same place in the end. It might have taken us longer in terms of constructing the real version (not having the lessons of the prototype to build on top of), but total time spent would probably have been less. Not only that, but the customers would have been able to try it out for real sooner, which would have given us the feedback that we needed sooner as well.

That’s not to say that a prototype is never beneficial, just that in my most recent experience, it didn’t really feel like it generated an appropriate amount of value.

Conclusion

Unlike the technical posts that I make, this one feels much more like a series of vaguely connected musings.

I don’t really have a concrete conclusion or lesson to take away, I’m just left with vague sense that our most recent experiment with building a prototype was a waste of time and effort that could have been better spent elsewhere.

Of course, there’s always the possibility that the specific situation we found ourselves in was simply a bad place to apply a prototype (which seems likely looking back), or maybe the prototype actually generated a huge amount of value and its just hard to see it in hindsight, because we have that knowledge now and its hard to analyse the situation without it.

Perhaps I’ll be making another post in a few months about a situation where I wished we had built a prototype…

When building a series of services to allow clients to access their own (previously office locked) data over the greater internet, there are a number of considerations to be made.

The old way was simple. There is a database. Stuff is in the database. When you want stuff, access the database. As long as the database in one office was powerful enough for the users in that office, you would be fine.

Moving all of that information into the cloud though…

Now everyone needs to access all their stuff at the same time. Now efficiency and isolation matters.

Well technically it mattered before as well, just not as much to the people who came before me.

I’m going to be talking about two things briefly in this post.

The first is isolating our upload and synchronization process from the actual service that needs to be queried.

The second is isolating binary data from all other requests.

Data Coming Right Up

In order to grant remote access to previously on-premises locked data we need to get that data out somehow. Unfortunately, for this system, the source of truth needs to stay on-premises for a number of different reasons that I won’t go into in too much detail. What we’re focusing on is allowing authenticated read-only access to the data from external systems.

The simplest solution to this is to have a replica of the data available in the cloud, and use that data for all incoming remote requests. Obviously this isn’t perfect (its an eventually consistent model), but because its read-only and we have some allowances for data latency (i.e. its okay if a mobile application doesn’t see exactly what is in the on-premises data the moment that it’s changed).

Of course, all of this data constantly being uploaded can cause a considerable amount of strain on the system as a whole, so we need to make sure that if there is a surge in the quantity of synchronization requests that the service responding to queries (get me the last 100 X entities) is not negatively impacted.

Easiest solution? Simply separate the two services, and share the data via a common store of some sort (our initial implementation will have a database).

With this model we gain some protection from load on on side impacting the other.

Its not perfect mind you, but the early separation gives us a lot of power moving forward if we need to change. For example, we could queue all synchronization requests to the sync service fairly easily, or split the shared database into a master and a number of read replicas. We don’t know if we’ll have a problem or what the solution to that problem will be, the important part is that we’ve isolated the potential danger, allowing for future change without as much effort.

10 Types of People

The system that we are constructing involves a moderate amount of binary data. I say moderate, but in reality, for most people who have large databases on premises, a good percentage of that data is binary data in various forms. Mostly images, but there are a lot of documents of various types as well (ranging from small and efficient PDF files to monstrous Word document abominations with embedded Excel spreadsheets).

Binary data is relatively problematic for a web service.

If you grant access to the binary data from a service, every request ties up one of your possible request handlers (whether it be threads, pseudo-threads or various other mechanisms of parallelism). This leaves less resources available for your other requests (data queries), which can make things difficult in the long run as the total number of binary data requests in flight at any particular moment slowly rises.

If you host the data outside the main service, you have to deal with the complexity of owning something else and making sure that it is secure (raw S3 would be ideal here, but then securing it is a pain).

In our case, our plan is to go with another service, purely for binary data. This allows us to leverage our existing authentication framework (so at least everything is secure), and allows us to use our existing logging tools to track access.

The benefits of isolating the access to binary data like this is that if there is a sudden influx of requests for images or documents or something, only that part of the system will be impacted (assuming there are no other shared components). Queries to get normal data will still complete in a timely fashion, and assuming we have written our integration well, some retry strategy will ensure that the binary data is delivered appropriately once the service resumes normal operation.

Summary

It is important to consider isolation concerns like I have outlined above when you are designing the architecture of a system.

You don’t necessarily have to implement all of your considerations straight away, but you at least need to know where your flex areas and where you can make changes without having to rewrite the entire thing. Understand how and when your architecture could adapt to potential changes, but don’t build it until you need it.

In our case, we also have a gateway/router sitting in front of everything, so we can remap URLs as we see fit moving into the future. In the case of the designs I’ve outlined above, they come from past (painful) experience. We’ve already encountered those issues in the past, while implementing similar systems, so we decided to go straight to the design that caters for them, rather than implement something we know would have problems down the track.

Its this sort of learning from your prior experiences that really makes a difference to the viability of an architecture in the long run.