0 Comments

This post is not as technical as some of my others. I really just want to bring attention to a tool for Elasticsearch that I honestly don’t think I could do without.

Cerebro.

From my experience, one of the hardest things to wrap my head around when working with Elasticsearch was visualizing how everything fit together. My background is primarily C# and .NET in a very Microsoft world, so I’m used to things like SQL Server, which comes with an excellent exploration and interrogation tool in the form of SQL Server Management studio. When it comes to Elasticsearch though, there seems to be no equiavelent, so I felt particularly blind.

Since starting to use Elasticsearch, I’ve become more and more fond of using the command line, so I’ve started to appreciate its amazing HTTP API more and more, but that initial learning curve was pretty vicious.

Anyway, to bring it back around, my first port of call when I started using Elasticsearch was to find a tool conceptually similar to SQL Server Management Studio. Something I could use to both visualize the storage system (however it worked) and possibly even query it as necessary.

I found Kopf.

Kopf did exactly what I wanted it to do. It provided a nice interface on top of Elasticsearch that helped me visualize how everything was structured and what sort of things I could do. To this day, if I attempt to visualize an Elasticsearch cluster in my head, the pictures that come to mind are of the Kopf interface. I can thank it for my understanding of the cluster, the nodes that make it up and the indexes stored therein, along with the excellent Elasticsearch documentation of course.

Later on I learnt that Kopf didn’t have to be used from the creators demonstration website (which is how I had been using it, connecting from my local machine to our ES ELK cluster), but could in fact be installed as a plugin inside Elasticsearch itself, which was even better, because you could access it from {es-url]}/plugins/_kopf, which was a hell of a lot easier.

Unfortunately, everything changed when the fire nation attacked…

No wait, that’s not right.

Everything changed when Elasticsearch 5 was released.

I’m The Juggernaut

Elasticsearch 5 deprecated site plugins. No more site plugins meant no more Kopf, or at least no more Kopf hosted within Elasticsearch. This made me sad, obviously, but I could still use the standalone site, so it wasn’t the end of the world.

My memory of the next bit is a little bit fuzzy, but I think even the standalone site stopped working properly when connecting to Elasticsearch 5. The creator of Kopf was no longer maintaining the project either, so it was unlikely that the problems would be solved.

I was basically blind.

Enter Cerebro.

No bones about it, Cerebro IS Kopf. It’s made by the same guy and is still being actively developed. Its pretty much a standalone Kopf (i.e. built in web server), and any differences between the two (other than some cosmetic stuff and the capability to easily save multiple Elasticsearch addresses) are lost on me.

As of this post, its up to 0.6.5, but as far as I can tell, it’s fully functional.

For my usage, I’ve incorporated Cerebro into our ELK stack, with a simple setup (ELB + single instance ASG), pre-configured with the appropriate Elasticsearch address in each environment that we spin up. As is the normal pattern, I’ve set it up on an AMI via Packer, and I deploy its configuration via Octopus deploy, but there is nothing particularly complicated there.

Kitty, Its Just A Phase

This post is pretty boring so far, so lets talk about Cerebro a little with the help of a screenshot.

This is the main screen of Cerebro, and it contains a wealth of information to help you get a handle on your Elasticsearch cluster.

It shows an overview of the cluster status, data nodes, indexes and their shards and replicas.

  • The cluster status is shown at the top of the screen, mostly via colour. Green good, yellow troublesome, red bad. Helpfully enough, the icon in the browser also changes colour according to the cluster status.
  • Data nodes are shown on the left, and display information like memory, cpu and disk, as well as IP address and name.
  • Indexes pretty much fill the rest of the screen, displaying important statistics like the number of documents and size, while allowing you to access things like mappings and index operations (like delete)
  • The intersection of index and data node gives information about shard/replica allocation. In the example above, we have 3 shards, 2 replicas and 3 nodes, so each node has a full copy of the data. Solid squares indicate the primary shard.
  • If you have unassigned or relocating shards, this information appears directly above the nodes, and shards currently being moved are shown in same place as normal shards, except blue.

Honestly, I don’t really use the other screens in Cerebro very much, or at least nowhere near as much as I use the overview screen. The dedicated nodes screen can be useful to view your master nodes (which aren’t shown on the overview), and to get a more performance focused display. I’ve also used the index templates screen for managing/viewing our logstash index template, but that’s mostly done through an Octopus deployment now.

There are others (including an ad-hoc query screen), but again, I haven’t really dug into them in depth. At least not enough to talk about them anyway.

That first screen though, the overview, is worth its weight in gold as far as I’m concerned.

Conclusion

I doubt I would understand Elasticsearch anywhere near as much as I do without Kopf/Cerebro. Realistically, I don’t really understand it much at all, but that little understanding I do have would be non-existent without these awesome tools.

Its not just a one horse town though. Elastic.co provides some equivalent tools as well (like Monitoring (formerly Marvel)) which offer similar capabilities, but they are mostly paid services as far as I can tell, so I’ve been hesitant to explore them in more depth.

I’m already spending way too much on the hardware for our log stack, so adding software costs on top of that is a challenging battle that I’m not quite ready to fight.

It doesn’t help that the last time I tried to price it, their answer for “How much for the things?” was basically “How much you got?”.

0 Comments

In my last blog post (Permaculture Paradise, the road so far…, an instant classic by the way), I mentioned that I found it to be quite an ordeal to create animations to show how my garden changed over time.

I’ve been taking weekly photos of various locations around my yard over the past year or so. They act as a form of metric, allowing me to compare two arbitrary points in time, and I use that ability to verify that I’m actually making things better. I use my phone to take the photos, which are then automatically synced to OneDrive (which is then in turn synced to my other computers). Handy.

Obviously I don’t have tripods or any other form of static photography setup in my yard (they’d just get in the way) so I have to manually do my best to align the photos to static points of reference, so that you can flip through them quickly to see change.

As an aside, I’ve thought before of making an app for my phone that would help me do the above. The simplest implementation I can think of would be to get the app to overlay the last image taken over the live feed of the camera screen (with a low opacity) so the user could align the static points in the last photo to now and make sure that the photo was taken from the same angle/position. I’m sure there’s also a whole bunch of complicated things you could do in order to programmatically match up static points of reference (like facial recognition algorithms maybe) but I feel that that sort of thing is a little beyond my grasp at this point in time. It would probably be really interesting though.

Anyway, back to the making animations thing. I thought that since I have a blog now, I’d be able to use those photos to show my legions of readers (ha) the changes happening in my garden. I assumed that stitching together a series of images into an animated gif would be easy. I mean, the internet is FULL of animated gifs, surely it can’t be that hard.

Turns out its pretty hard.

I should clarify that statement somewhat I suppose. Its actually really easy to stitch together a bunch of images into an animated gif. There are online services that do just that (makeagif is one for example, but there are others). The hard bit was the combination of my photos being quite large along with the fact that they didn’t necessarily transition cleanly from one picture to the next (what can I say, taking pictures in exactly the same position from week to week is hard).

The First Attempt

My first thought was to find a way to resize the images down to an appropriate size and then use one of the online services. Seems fair enough, any image editing program will let you do just that. I’m a lazy person however (to potential employers, remember, laziness is one of the great virtues of a programmer) so I didn’t want to have to go through the hundreds of photos I had and manually resize them. I was prepared to go through a significant amount of effort to avoid having to do the thing manually in fact.

Enter ImageMagick.

ImageMagick is a sweet little tool for doing command line manipulation of images, including resizing. It also makes gifs! Fantastic, two birds with one stone. To hell with using an online service, I’ll just use this tool to make my gifs myself. Much better.

I grabbed the installed version of ImageMagick, installed it on a Virtual Machine, stole the install directory (god I hate installing development software on my main machine sometimes, it always comes with a bunch of “helpful” stuff, like examples and tutorials and all sorts of other crap that’s better stored on the internet where I could actually search for it) and tossed it into my tools directory.

As a general rule of thumb, I’m very much in favour of portable applications when command line tools are concerned. I have a Tools directory in my OneDrive that contains cmder and a bunch of tools that I use from the command line (git, python). Whenever I find a new command line tool, I like to add it to my Tools/cmder/vendor directory and then customize the init.bat to add that directory (or whatever the appropriate bin directory is) to my PATH when running cmder.

As is sometimes the case with this sort of  “lets just steal the Program Files directory” approach, it didn’t work.

convert *.jpg composite.gif Invalid Parameter - composite.gif

Well, that’s a little weird. I mean, that’s the exact command that ImageMagick expects. A simple conversion of all jpgs in the directory to an animated gif.

It turns out that there is already a convert command defined in system32, and when I added ImageMagick to my path in cmder, I added it like this (which was consistent with all of the other path customisations).

@set PATH=%PATH%;%CMDER_ROOT%\vendor\ImageMagick-6.8.9-Q16-64-NP; 

That was wrong, because the ImageMagick convert command appeared later in the path than the built in Windows convert command

I have no idea what the built in convert command does, but I’m glad it was non-destructive when pointed at my image files. I mean I copied them to a working directory just in case I messed them up, but still, it could have done almost anything.

Anyway, once I fixed that by pre-pending the ImageMagick directory to my path (instead of appending it), I got a new error.

convert *.jpg composite.gif convert.exe RegistryKeyLookupFailed CoderModulesPath @ error/module.c/GetMagickModulePath/662 no devcode delegate for this image format JPEG @ error/constitute.c/ReadImage/501

There were a lot of errors like that.

My own fault really, I should know better than to assume an installed app will just work if you move it around. The installed version of ImageMagick is dependent on some registry entries, which seem to be where it defines the location for the modules for dealing with files of a particular format. Something like that anyway. Ugh, registry entries.

I went back to the ImageMagick site and grabbed the portable build instead and tried again.

convert *.jpg composite.gif convert.exe Memory allocation failed composite.gif @ error/quantize.c/QuantizeImage/2733 error/gif.c/WriteGIFImage/1642

Yessss, progress.

Nooooo, memory allocation failure.

I’ll save you the long rambling investigation here. I had plenty of memory available, but the portable build of ImageMagick is 32 bit and the images I was trying to convert into a gif were pushing the 1.9GB memory limit for a 32 bit process almost straight away. I tried resizing the files to be smaller, but any time I tried to make a gif with all 47 of the images I just kept running into the memory limit.

The Tangent

I’m a developer! I should be able to take the ImageMagick source and create a portable build targeted to a 64 bit environment. There’s even instructions on the ImageMagick website about making a build of your own, how hard can it be?

About a day later I gave up. Its C++ and old and Visual Studio 2013 just could not cope with it. While I’m sure I would have been able to get there in the end, the motivation just wasn’t there. Sometimes you just have to accept that maybe you shouldn’t be shaving that particular yak.

The Second Attempt

I installed the 64 bit version of ImageMagick and composed the gif from the raw photos.

It looked terrible.

My photos had enough difference between them that the resulting gif was jumping all over the place, and was just plain unpleasant to watch. I didn’t even look like it had included all of the images either. The first few images in the series were from my old phone as well, so they had completely different dimensions, which really messed everything up (and may have been the reason for it not including all of the images in the gif).

The Third Attempt

Good old Paint.NET. You have an plugin for dealing with animated gifs as well.

That Paint.NET website is a horrible horrible piece of work, with a ton of “download” links that are actually just advertisements. I’ve been using Paint.NET 3.5.10 for a while now, but it looks like the latest version is 4.0.3 on the website. Here is a direct download link for that version. Not sure how long it will be valid for.

I used Paint.NET to load all of the photos for a single location in the garden as layers (Layers –> Import from File). I then resized the canvas so that there was some breathing room around all of the images and hid all of the layers except for the first one. Finally I went through each layer, made it visible, set its opacity to approximately 50% and manually moved it into position. When moving the image, I picked a static point of reference (this time it was my compost bins, which was nice because they were in the centre) and aligned that point on the first layer to each additional layer. I did this to mitigate the jumping effect that I noticed after making the gif using ImageMagick. With a point of reference for the human eye to latch on to, the resulting gif appears much smoother than it actually is.

Finally, to deal with the fact that not all of my images were the same dimensions, I cropped everything that didn’t contain content in every layer (make selection, Image –> Crop to Selection).

Lastly, I saved as an animated gif.

Victory!

Well, almost. The resulting gif was pretty huge (100+MB), because I forgot to resize the photos down. At least with all the images in the same document the fix was simple though, Image –> Resize –> X%, repeat until I got the resulting gif down to a size I was more comfortable with (14MB).

Bonus Attempt!

I did question whether or not an animated gif was really the best way to do the change animation, so I tried Windows Movie Maker. While it did let me make a movie of the images stitched together fairly easily, I really couldn’t do anything more than that (and I really needed to reposition the images to make the resulting animation look acceptable). There were a few other video making tools that I considered as well (Lightworks was one) but they all had fairly steep learning curves, and I while I wanted to make a nice animation, I didn’t have an infinite amount of time available.

The Delivery

14MB is still too large for someone to download via a blog post though. Its fine for my own personal use, but that’s a hell of a lot of data to pull down just to look at how my garden changed over time. It would also mean that I would have to host that file as well, which would have a direct impact on my own bandwidth costs (I think anyway. I mean this blog is hosted on Azure. I should check that…)

After a bit of searching, I found gfycat. Its a nice little service that allows you to upload a gif and then either embed it or link to it from your website. The gif is actually converted to an MP4 (I think) and when embedded or linked, will be displayed as either a HTML5 video or directly as an animated gif. The whole service is powered by Amazon as well, so I don’t need to worry about content location or anything like that. It just works.

I won’t embed the image again (because its already in my Permaculture Paradise, the road so far… post) but its directly available at http://gfycat.com/EnchantedDearestIrukandjijellyfish if you want to have a look.

It ended up being 4MB in gfy format in the end, which is still incredibly large (sorry mobile users). I think I’ll need to considering making future animations at a lower resolution or reducing the quality in some way in order to drop it below 1MB at least. Its an ongoing process.

Gfycat has some instructions for embedding a gfy into your medium (website, blog post, whatever). It involves adding some script to your page, and then including an img tag with specific class and data-id attributes. Fairly straightforward.

The Javascript they ask you to include is:

(function(d, t) {
    var g = d.createElement(t),
        s = d.getElementsByTagName(t)[0];
    g.src = 'http://assets.gfycat.com/js/gfyajax-0.517d.js';
    s.parentNode.insertBefore(g, s);
}(document, 'script'));

A little bit strange actually, because all it does is insert the contents of the specified JavaScript file into your page when the JavaScript is run. Kind of pointless. Why not just use the src attribute of the script tag?

Anyway, you combine that piece of JavaScript with an img tag that looks like the following:

<img class="gfyitem" data-id="[KEY GOES HERE]" />

I tried to read through the contents of the JS file, but didn’t spend too much time on it. I assume what it does is find all img tags that have a class of gfyitem and then mutate them to be either the HTML5 video tag (if supported by the browser) or a link to the plain old gif, for whatever key you’ve referenced.

I’m using a customised version of MiniBlog, so I just added a script tag containing a reference to the JS file to the end of the _Layout.cshtml file of my chosen theme (OneColumn forever!), pushed to Azure (Git deployment is amazing) and the img tag that I’d inserted into my blog post was magically transformed into a gfy.

I like it when things just work.

I had to manually insert the image tag in my blog post using Windows Live Writer though, as it wasn’t an image link in the traditional sense, what with it not having a src and all. Its fairly easy to edit the source when using Windows Live Writer, but maybe a Plugin would be nice? I wonder how you write a Windows Live Writer plugin?

Conclusion

Making a nice looking gif from my photos was a lot harder and a lot more time consuming than I originally thought it would be.

At least some of that was caused by me, as I didn’t want to settle for a crappy looking gif, and I didn’t (originally) want to have to do the whole thing manually. My initial dream was to have the images smoothly morph from one to the other, but I quickly abandoned that and settled for just making a basic animation. That was probably for the best, as I still have no idea how I could accomplish the morphing. I did run across some techniques for doing it, but they looked complex enough for morphing just two images, let alone doing it for 47.

When it comes to manipulating images, I think you really do need to see what you’re doing in order to make something that looks good. I’m sure there’s a bunch of crazy people out there who could visualize all the images in their head and do the entire thing (including repositioning) through the command line, but I am not one of those people. In the end, I used the image editing software that I was most familiar with, and I thank God that someone made a plugin for exporting animated gifs from it.

I only ended up making a single animation (because of how time consuming the whole process was) but I will be creating more in the future. I’ve got at least 10 other locations in my garden that I can use, and I’m toying with applying the same change tracking process with my own body (I don’t think I. Hopefully I’ll get better at making the animations over time, because it took me at least an hour to put together the one I did make.

I hope that my long rambling journey through making a time-lapse animation will be of use to someone. I mean, it was worthwhile to me, because I learnt a lot about the whole process, and I hope I’ve managed to capture some of that experience in this post.