Faster S3! Clone! Clone!
A little over 4 months ago, I wrote a post about trying to improve the speed of cloning a large S3 bucket. At the time, I tried to simply parallelise the execution of the AWS CLI sync command, which actually proved to be much slower than simply leaving the CLI alone to do its job. It was an unsurprising result in retrospect, but you never know unless you try.
Unwilling to let the idea die, I decided to make it my focus during our recent hack days.
If you are unfamiliar with the concept of a hack day (or Hackathon as they are sometimes known), have a look at this Wikipedia article. At my current company, we’re only just starting to include hack days on a regular basis, but its a good sign of a healthy development environment.
Continuing on with the original train of thought (parallelise via prefixes), I needed to find a way to farm out the work to something (whether it was a pool of our own workers or some other mechanism). Continuing with that train of thought, I chose to use AWS Lambda.
Enter Node.js on Lambda.
At A High Level
AWS Lambda is a relatively new offering, allowing you to configure some code to automatically execute following a trigger from one of a number of different events, including an SNS Topic Notification, changes to an S3 bucket or a HTTP call. You can use Python, Java or Javascript (through Node.js) as code natively, but you can technically use anything you can compile into a Linux compatible executable and make accessible to the function via S3 or something similar.
Since Javascript seems to be everywhere now (even though its hard to call it a real language), it was a solid choice. No point being afraid of new things.
Realistically, I should have been at least a little afraid of new things.
Conceptually the idea can be explained as a simple divide and conquer strategy, managed by files in an S3 bucket (because S3 was the triggering mechanism I was most familiar with).
If something wants to trigger a clone, it writes a file into a known S3 bucket detailing the desired operation (source, destination, some sort of id) with a key of {id}-{source}-{destination}/clone-request.
In response, the Lambda function will trigger, segment the work and write a file for each segment with a key of {id}-{source}-{destination}/{prefix}-segment-request. When it has finished breaking down the work, it will write another file with the key {id}-{source}-{destination}/clone-response, containing a manifest of the breakdown, indicating that it is done with the division of work.
As each segment file is being written, another Lambda function will be triggered, doing the actual copy work and finally writing a file with the key {id}-{source}-{destination}/{prefix}-segment-response to indicate that its done.
File Formats Are Interesting
Each clone-request file looks like this:
{
id: {id},
source: {
name: {source-bucket-name}
},
destination: {
name: {destination-bucket-name}
}
}Its a relatively simple file that would be easy to extend as necessary (for example, if you needed to specify the region, credentials to access the bucket, etc).
The clone-response file (the manifest), looks like this:
{
id: {id},
source: {
name: {source-bucket-name}
},
destination: {
name: {destination-bucket-name}
},
segments: {
count: {number-of-segments},
values: [
{segment-key},
{segment-key}
...
]
}
}Again, another relatively simple file. The only additional information is the segments that the task was broken into. These segments are used for tracking purposes, as the code that requests a clone needs some way to know when the clone is done.
Each segment-request file looks like this:
{
id: {id},
source: {
name: {source-bucket-name},
prefix: {prefix}
},
destination: {
name: {destination-bucket-name}
}
}And finally, each segment-response file looks like this:
{
id: {id},
source: {
name: {source-bucket-name},
prefix: {prefix}
},
destination: {
name: {destination-bucket-name}
},
files: [
{key},
{key},
...
]
}Nothing fancy or special, just straight JSON files with all the information needed.
Breaking It All Down
First up, the segmentation function.
Each Javascript Lambda function already comes with access to the aws-sdk, which is super useful, because honestly if you’re using Lambda, you’re probably doing it because you need to talk to other AWS offerings.
The segmentation function has to read in the triggering file from S3, parse it (its Javascript and JSON so that’s trivial at least), iterate through the available prefixes (using a delimiter, and sticking with the default “/”), write out a file for each unique prefix and finally write out a file containing the manifest.
As I very quickly learned, using Node.js to accomplish the apparently simple task outlined above was made not simple at all thanks to its fundamentally asynchronous nature, and the fact that async calls don’t seem to return a traceable component (unlike in C#, where if you were using async tasks you would get a task object that could be used to track whether or not the task succeeded/failed).
To complicate this even further, the aws-sdk will only return a maximum of 1000 results when listing the prefixes in a bucket (or doing anything with a bucket really), which means you have to loop using the callbacks. This makes accumulating some sort of result set annoying difficult, especially if you want to know when you are done.
Anyway, the segmentation function is as follows:
console.log('Loading function');
var aws = require('aws-sdk');
var s3 = new aws.S3({ apiVersion: '2006-03-01' });
function putCallback(err, data)
{
if (err)
{
console.log('Failed to Upload Clone Segment ', err);
}
}
function generateCloneSegments(s3Source, command, commandBucket, marker, context, segments)
{
var params = { Bucket: command.source.name, Marker: marker, Delimiter: '/' };
console.log("Listing Prefixes: ", JSON.stringify(params));
s3Source.listObjects(params, function(err, data) {
if (err)
{
context.fail(err);
}
else
{
for (var i = 0; i < data.CommonPrefixes.length; i++)
{
var item = data.CommonPrefixes[i];
var segmentRequest = {
id: command.id,
source : {
name: command.source.name,
prefix: item.Prefix
},
destination : {
name: command.destination.name
}
};
var segmentKey = command.id + '/' + item.Prefix.replace('/', '') + '-segment-request';
segments.push(segmentKey);
console.log("Uploading: ", segmentKey);
var segmentUploadParams = { Bucket: commandBucket, Key: segmentKey, Body: JSON.stringify(segmentRequest), ContentType: 'application/json'};
s3.putObject(segmentUploadParams, putCallback);
}
if(data.IsTruncated)
{
generateCloneSegments(s3Source, command, commandBucket, data.NextMarker, context, segments);
}
else
{
// Write a clone-response file to the commandBucket, stating the segments generated
console.log('Total Segments: ', segments.length);
var cloneResponse = {
segments: {
count: segments.length,
values: segments
}
};
var responseKey = command.id + '/' + 'clone-response';
var cloneResponseUploadParams = { Bucket: commandBucket, Key: responseKey, Body: JSON.stringify(cloneResponse), ContentType: 'application/json'};
console.log("Uploading: ", responseKey);
s3.putObject(cloneResponseUploadParams, putCallback);
}
}
});
}
exports.handler = function(event, context) {
//console.log('Received event:', JSON.stringify(event, null, 2));
var commandBucket = event.Records[0].s3.bucket.name;
var key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
var params = {
Bucket: commandBucket,
Key: key
};
s3.getObject(params, function(err, data)
{
if (err)
{
context.fail(err);
}
else
{
var command = JSON.parse(data.Body);
var s3Source = new aws.S3({ apiVersion: '2006-03-01', region: 'ap-southeast-2' });
var segments = [];
generateCloneSegments(s3Source, command, commandBucket, '', context, segments);
}
});
};
I’m sure some improvements could be made to the Javascript (I’d love to find a way automate tests on it), but its not bad for being written directly into the AWS console.
Hi Ho, Hi Ho, Its Off To Work We Go
The actual cloning function is remarkably similar to the segmenting function.
It still has to loop through items in the bucket, except it limits itself to items that match a certain prefix. It still has to do something for each item (execute a copy and add the key to its on result set) and it still has to write a file right at the end when everything is done.
console.log('Loading function');
var aws = require('aws-sdk');
var commandS3 = new aws.S3({ apiVersion: '2006-03-01' });
function copyCallback(err, data)
{
if (err)
{
console.log('Failed to Copy ', err);
}
}
function copyFiles(s3, command, commandBucket, marker, context, files)
{
var params = { Bucket: command.source.name, Marker: marker, Prefix: command.source.prefix };
s3.listObjects(params, function(err, data) {
if (err)
{
context.fail(err);
}
else
{
for (var i = 0; i < data.Contents.length; i++)
{
var key = data.Contents[i].Key;
files.push(key);
console.log("Copying [", key, "] from [", command.source.name, "] to [", command.destination.name, "]");
var copyParams = {
Bucket: command.destination.name,
CopySource: command.source.name + '/' + key,
Key: key
};
s3.copyObject(copyParams, copyCallback);
}
if(data.IsTruncated)
{
copyFiles(s3, command, commandBucket, data.NextMarker, context, segments);
}
else
{
// Write a segment-response file
console.log('Total Files: ', files.length);
var segmentResponse = {
id: command.id,
source: command.source,
destination : {
name: command.destination.name,
files: {
count: files.length,
files: files
}
}
};
var responseKey = command.id + '/' + command.source.prefix.replace('/', '') + '-segment-response';
var segmentResponseUploadParams = { Bucket: commandBucket, Key: responseKey, Body: JSON.stringify(segmentResponse), ContentType: 'application/json'};
console.log("Uploading: ", responseKey);
commandS3.putObject(segmentResponseUploadParams, function(err, data) { });
}
}
});
}
exports.handler = function(event, context) {
//console.log('Received event:', JSON.stringify(event, null, 2));
var commandBucket = event.Records[0].s3.bucket.name;
var key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
var params = {
Bucket: commandBucket,
Key: key
};
commandS3.getObject(params, function(err, data)
{
if (err)
{
context.fail(err);
}
else
{
var command = JSON.parse(data.Body);
var s3 = new aws.S3({ apiVersion: '2006-03-01', region: 'ap-southeast-2' });
var files = [];
copyFiles(s3, command, commandBucket, '', context, files);
}
});
};
Tricksy Trickses
You may notice that there is no mention of credentials in the code above. That’s because the Lambda functions run under a role with a policy that gives them the ability to list, read and put into any bucket in our account. Roles are handy for accomplishing things in AWS, avoiding the new to supply credentials. When applied to the resource, and no credentials are supplied, the aws-sdk will automatically generate a short term token using the role, reducing the likelihood of leaked credentials.
As I mentioned above, The asynchronous nature of Node.js made everything a little but more difficult than expected. It was hard to determine when anything was done (somewhat important for writing manifest files). Annoyingly enough, it was even hard to determine when the function itself was finished. I kept running into issues where the function execution had finished, and it looked like it had done all of the work I expected it to do, but AWS Lambda was reporting that it did not complete successfully.
In the initial version of Node.js I was using (v0.10.42), the AWS supplied context object had a number of methods on it to indicate completion (whether success or failure). If I called the Succeed method after I setup my callbacks, the function would terminate without doing anything, because it didn’t automatically wait for the callbacks to complete. If I didn’t call it, the function would be marked as “did not complete successfully”. Extremely annoying.
As is often the case with AWS though, on literally the second hack day, AWS released support for Node.js v4.3, which automatically waits for all pending callbacks to complete before completing the function, completely changing the interaction model for the better. I did upgrade to the latest version during the second hack day (after I had accepted that my function was going to error out in the control panel but actually do all the work it needed to), but it wasn’t until later that I realised that the upgrade had fixed my problem.
The last tripwire I ran into was related to AWS Lambda not being available in all regions yet. Specifically, its not in ap-southeast-2 (Sydney), which is where all of our infrastructure lives. S3 is weird in relation to regions, as buckets are globally unique and accessible, but they do actually have a home region. What does this have to do with Lambda? Well, the S3 bucket triggers I used as the impetus for the function execution only work if the S3 bucket is in the same region as the Lambda function (so us-west-1), even though once you get inside the Lambda function you can read/write to any bucket you like. Weird.
Conclusion
I’ve omitted the Powershell code responsible for executing the clone for brevity. It writes the request to the bucket, reads the response and then polls waiting for all of the segments to be completed, so its not particularly interesting, although the polling for segment completion was my first successful application of the Invoke-Parallel function from Script Center.
Profiling the AWS Lambda approach versus the original AWS CLI sync command approach over a test bucket (7500 objects, 195 distinct prefixes, 8000 MB of data) showed a decent improvement in performance. The sync approach took 142 seconds and the Lambda approach took 55 seconds, approximately a third of the time, which was good to see considering the last time I tried to parallelise the clone it actually decreased the performance. I think with some tweaking the Lambda approach could be improved further, with tighter polling tolerances and an increased number of parallel Lamda executions allowed.
Unfortunately, I have not had the chance to execute the AWS Lambda implementation on the huge bucket that is the entire reason it exists, but I suspect that it won’t work.
Lambda allows at maximum 5 minutes of execution time per function, and I suspect that the initial segmentation for a big enough bucket will probably take longer than that. It might be possible to chain lambda functions together (i.e. trigger one from the next one, perhaps per 1000 results returned from S3, but I’m not entirely sure how to do that yet (maybe using SNS notifications instead of S3?). Additionally, with a big enough bucket, the manifest file itself (detailed the segments) might become unwieldy. I think the problem bucket has something like 200K unique prefixes, so the size of the manifest file can add up quickly.
Regardless, the whole experience was definitely useful from a technical growth point of view. Its always a good idea to remove yourself from your comfort zone and try some new things, and AWS Lambda + Node.js are definitely well outside my comfort zone.
A whole different continent in fact.