build the blog with nix

1 files changed, 100 insertions, 0 deletions

diff --git a/src/_posts/2013-10-08-generations.md b/src/_posts/2013-10-08-generations.md
new file mode 100644
index 0000000..c1c433d
--- /dev/null
+++ b/src/_posts/2013-10-08-generations.md
@@ -0,0 +1,100 @@
+---
+title: Generations
+description: >-
+    A simple file distribution strategy for very large scale, high-availability
+    file-services.
+---
+
+## The problem
+
+At [cryptic.io][cryptic] we plan on having millions of different
+files, any of which could be arbitrarily chosen to be served any given time.
+These files are uploaded by users at arbitrary times.
+
+Scaling such a system is no easy task. The solution I've seen implemented in the
+past involves shuffling files around on a nearly constant basis, making sure
+that files which are more "popular" are on fast drives, while at the same time
+making sure that no drives are at capicty and at the same time that all files,
+even newly uploaded ones, are stored redundantly.
+
+The problem with this solution is one of coordination. At any given moment the
+app needs to be able to "find" a file so it can give the client a link to
+download the file from one of the servers that it's on. Full-filling this simple
+requirement means that all datastores/caches where information about where a
+file lives need to be up-to-date at all times, and even then there are
+race-conditions and network failures to contend with, while at all times the
+requirements of the app evolve and change.
+
+## A simpler solution
+
+Let's say you want all files which get uploaded to be replicated in triplicate
+in some capacity. You buy three identical hard-disks, and put each on a separate
+server.  As files get uploaded by clients, each file gets put on each drive
+immediately. When the drives are filled (which should be at around the same
+time), you stop uploading to them.
+
+That was generation 0.
+
+You buy three more drives, and start putting all files on them instead. This is
+going to be generation 1. Repeat until you run out of money.
+
+That's it.
+
+### That's it?
+
+It seems simple and obvious, and maybe it's the standard thing which is done,
+but as far as I can tell no-one has written about it (though I'm probably not
+searching for the right thing, let me know if this is the case!).
+
+### Advantages
+
+* It's so simple to implement, you could probably do it in a day if you're
+starting a project from scratch
+
+* By definition of the scheme all files are replicated in multiple places.
+
+* Minimal information about where a file "is" needs to be stored. When a file is
+uploaded all that's needed is to know what generation it is in, and then what
+nodes/drives are in that generation. If the file's name is generated
+server-side, then the file's generation could be *part* of its name, making
+lookup even faster.
+
+* Drives don't need to "know" about each other. What I mean by this is that
+whatever is running as the receive point for file-uploads on each drive doesn't
+have to coordinate with its siblings running on the other drives in the
+generation. In fact it doesn't need to coordinate with anyone. You could
+literally rsync files onto your drives if you wanted to. I would recommend using
+[marlin][0] though :)
+
+* Scaling is easy. When you run out of space you can simply start a new
+generation. If you don't like playing that close to the chest there's nothing to
+say you can't have two generations active at the same time.
+
+* Upgrading is easy. As long as a generation is not marked-for-upload, you can
+easily copy all files in the generation into a new set of bigger, badder drives,
+add those drives into the generation in your code, remove the old ones, then
+mark the generation as uploadable again.
+
+* Distribution is easy. You just copy a generation's files onto a new drive in
+Europe or wherever you're getting an uptick in traffic from and you're good to
+go.
+
+* Management is easy. It's trivial to find out how many times a file has been
+replicated, or how many countries it's in, or what hardware it's being served
+from (given you have easy access to information about specific drives).
+
+### Caveats
+
+The big caveat here is that this is just an idea. It has NOT been tested in
+production. But we have enough faith in it that we're going to give it a shot at
+[cryptic.io][cryptic]. I'll keep this page updated.
+
+The second caveat is that this scheme does not inherently support caching. If a
+file suddenly becomes super popular the world over your hard-disks might not be
+able to keep up, and it's probably not feasible to have an FIO drive in *every*
+generation. I think that [groupcache][1] may be the answer to this problem,
+assuming your files are reasonably small, but again I haven't tested it yet.
+
+[cryptic]: https://cryptic.io
+[0]: https://github.com/cryptic-io/marlin
+[1]: https://github.com/golang/groupcache