summaryrefslogtreecommitdiff
path: root/src/_posts
diff options
context:
space:
mode:
Diffstat (limited to 'src/_posts')
-rw-r--r--src/_posts/2013-04-09-erlang-tcp-socket-pull-pattern.md256
-rw-r--r--src/_posts/2013-07-11-goplus.md77
-rw-r--r--src/_posts/2013-10-08-generations.md100
-rw-r--r--src/_posts/2013-10-25-namecoind-ssl.md248
-rw-r--r--src/_posts/2014-01-11-diamond-square.md494
-rw-r--r--src/_posts/2014-10-29-erlang-pitfalls.md192
-rw-r--r--src/_posts/2015-03-11-rabbit-hole.md165
-rw-r--r--src/_posts/2015-07-15-go-http.md547
-rw-r--r--src/_posts/2015-11-21-happy-trees.md235
-rw-r--r--src/_posts/2017-09-06-brian-bars.md105
-rw-r--r--src/_posts/2018-10-25-rethinking-identity.md292
-rw-r--r--src/_posts/2018-11-12-viz-1.md54
-rw-r--r--src/_posts/2018-11-12-viz-2.md49
-rw-r--r--src/_posts/2019-08-02-program-structure-and-composability.md587
-rw-r--r--src/_posts/2020-04-26-trading-in-the-rain.md55
-rw-r--r--src/_posts/2020-05-30-denver-protests.md161
-rw-r--r--src/_posts/2020-07-07-viz-3.md154
-rw-r--r--src/_posts/2020-11-16-component-oriented-programming.md352
-rw-r--r--src/_posts/2021-01-01-new-year-new-resolution.md50
-rw-r--r--src/_posts/2021-01-09-ginger.md352
-rw-r--r--src/_posts/2021-01-14-the-web.md239
21 files changed, 4764 insertions, 0 deletions
diff --git a/src/_posts/2013-04-09-erlang-tcp-socket-pull-pattern.md b/src/_posts/2013-04-09-erlang-tcp-socket-pull-pattern.md
new file mode 100644
index 0000000..3e5f0af
--- /dev/null
+++ b/src/_posts/2013-04-09-erlang-tcp-socket-pull-pattern.md
@@ -0,0 +1,256 @@
+---
+title: "Erlang, tcp sockets, and active true"
+description: >-
+ Using `{active:once}` isn't always the best way to handle connections.
+---
+
+If you don't know erlang then [you're missing out][0]. If you do know erlang,
+you've probably at some point done something with tcp sockets. Erlang's highly
+concurrent model of execution lends itself well to server programs where a high
+number of active connections is desired. Each thread can autonomously handle its
+single client, greatly simplifying the logic of the whole application while
+still retaining [great performance characteristics][1].
+
+## Background
+
+For an erlang thread which owns a single socket there are three different ways
+to receive data off of that socket. These all revolve around the `active`
+[setopts][2] flag. A socket can be set to one of:
+
+* `{active,false}` - All data must be obtained through [recv/2][3] calls. This
+ amounts to syncronous socket reading.
+
+* `{active,true}` - All data on the socket gets sent to the controlling thread
+ as a normal erlang message. It is the thread's
+ responsibility to keep up with the buffered data in the
+ message queue. This amounts to asyncronous socket reading.
+
+* `{active,once}` - When set the socket is placed in `{active,true}` for a
+ single packet. That is, once set the thread can expect a
+ single message to be sent to when data comes in. To receive
+ any more data off of the socket the socket must either be
+ read from using [recv/2][3] or be put in `{active,once}` or
+ `{active,true}`.
+
+## Which to use?
+
+Many (most?) tutorials advocate using `{active,once}` in your application
+\[0]\[1]\[2]. This has to do with usability and security. When in `{active,true}`
+it's possible for a client to flood the connection faster than the receiving
+process will process those messages, potentially eating up a lot of memory in
+the VM. However, if you want to be able to receive both tcp data messages as
+well as other messages from other erlang processes at the same time you can't
+use `{active,false}`. So `{active,once}` is generally preferred because it
+deals with both of these problems quite well.
+
+## Why not to use `{active,once}`
+
+Here's what your classic `{active,once}` enabled tcp socket implementation will
+probably look like:
+
+```erlang
+-module(tcp_test).
+-compile(export_all).
+
+-define(TCP_OPTS, [
+ binary,
+ {packet, raw},
+ {nodelay,true},
+ {active, false},
+ {reuseaddr, true},
+ {keepalive,true},
+ {backlog,500}
+]).
+
+%Start listening
+listen(Port) ->
+ {ok, L} = gen_tcp:listen(Port, ?TCP_OPTS),
+ ?MODULE:accept(L).
+
+%Accept a connection
+accept(L) ->
+ {ok, Socket} = gen_tcp:accept(L),
+ ?MODULE:read_loop(Socket),
+ io:fwrite("Done reading, connection was closed\n"),
+ ?MODULE:accept(L).
+
+%Read everything it sends us
+read_loop(Socket) ->
+ inet:setopts(Socket, [{active, once}]),
+ receive
+ {tcp, _, _} ->
+ do_stuff_here,
+ ?MODULE:read_loop(Socket);
+ {tcp_closed, _}-> donezo;
+ {tcp_error, _, _} -> donezo
+ end.
+```
+
+This code isn't actually usable for a production system; it doesn't even spawn a
+new process for the new socket. But that's not the point I'm making. If I run it
+with `tcp_test:listen(8000)`, and in other window do:
+
+```bash
+while [ 1 ]; do echo "aloha"; done | nc localhost 8000
+```
+
+We'll be flooding the the server with data pretty well. Using [eprof][4] we can
+get an idea of how our code performs, and where the hang-ups are:
+
+```erlang
+1> eprof:start().
+{ok,<0.34.0>}
+
+2> P = spawn(tcp_test,listen,[8000]).
+<0.36.0>
+
+3> eprof:start_profiling([P]).
+profiling
+
+4> running_the_while_loop.
+running_the_while_loop
+
+5> eprof:stop_profiling().
+profiling_stopped
+
+6> eprof:analyze(procs,[{sort,time}]).
+
+****** Process <0.36.0> -- 100.00 % of profiled time ***
+FUNCTION CALLS % TIME [uS / CALLS]
+-------- ----- --- ---- [----------]
+prim_inet:type_value_2/2 6 0.00 0 [ 0.00]
+
+....snip....
+
+prim_inet:enc_opts/2 6 0.00 8 [ 1.33]
+prim_inet:setopts/2 12303599 1.85 1466319 [ 0.12]
+tcp_test:read_loop/1 12303598 2.22 1761775 [ 0.14]
+prim_inet:encode_opt_val/1 12303599 3.50 2769285 [ 0.23]
+prim_inet:ctl_cmd/3 12303600 4.29 3399333 [ 0.28]
+prim_inet:enc_opt_val/2 24607203 5.28 4184818 [ 0.17]
+inet:setopts/2 12303598 5.72 4533863 [ 0.37]
+erlang:port_control/3 12303600 77.13 61085040 [ 4.96]
+```
+
+eprof shows us where our process is spending the majority of its time. The `%`
+column indicates percentage of time the process spent during profiling inside
+any function. We can pretty clearly see that the vast majority of time was spent
+inside `erlang:port_control/3`, the BIF that `inet:setopts/2` uses to switch the
+socket to `{active,once}` mode. Amongst the calls which were called on every
+loop, it takes up by far the most amount of time. In addition all of those other
+calls are also related to `inet:setopts/2`.
+
+I'm gonna rewrite our little listen server to use `{active,true}`, and we'll do
+it all again:
+
+```erlang
+-module(tcp_test).
+-compile(export_all).
+
+-define(TCP_OPTS, [
+ binary,
+ {packet, raw},
+ {nodelay,true},
+ {active, false},
+ {reuseaddr, true},
+ {keepalive,true},
+ {backlog,500}
+]).
+
+%Start listening
+listen(Port) ->
+ {ok, L} = gen_tcp:listen(Port, ?TCP_OPTS),
+ ?MODULE:accept(L).
+
+%Accept a connection
+accept(L) ->
+ {ok, Socket} = gen_tcp:accept(L),
+ inet:setopts(Socket, [{active, true}]), %Well this is new
+ ?MODULE:read_loop(Socket),
+ io:fwrite("Done reading, connection was closed\n"),
+ ?MODULE:accept(L).
+
+%Read everything it sends us
+read_loop(Socket) ->
+ %inet:setopts(Socket, [{active, once}]),
+ receive
+ {tcp, _, _} ->
+ do_stuff_here,
+ ?MODULE:read_loop(Socket);
+ {tcp_closed, _}-> donezo;
+ {tcp_error, _, _} -> donezo
+ end.
+```
+
+And the profiling results:
+
+```erlang
+1> eprof:start().
+{ok,<0.34.0>}
+
+2> P = spawn(tcp_test,listen,[8000]).
+<0.36.0>
+
+3> eprof:start_profiling([P]).
+profiling
+
+4> running_the_while_loop.
+running_the_while_loop
+
+5> eprof:stop_profiling().
+profiling_stopped
+
+6> eprof:analyze(procs,[{sort,time}]).
+
+****** Process <0.36.0> -- 100.00 % of profiled time ***
+FUNCTION CALLS % TIME [uS / CALLS]
+-------- ----- --- ---- [----------]
+prim_inet:enc_value_1/3 7 0.00 1 [ 0.14]
+prim_inet:decode_opt_val/1 1 0.00 1 [ 1.00]
+inet:setopts/2 1 0.00 2 [ 2.00]
+prim_inet:setopts/2 2 0.00 2 [ 1.00]
+prim_inet:enum_name/2 1 0.00 2 [ 2.00]
+erlang:port_set_data/2 1 0.00 2 [ 2.00]
+inet_db:register_socket/2 1 0.00 3 [ 3.00]
+prim_inet:type_value_1/3 7 0.00 3 [ 0.43]
+
+.... snip ....
+
+prim_inet:type_opt_1/1 19 0.00 7 [ 0.37]
+prim_inet:enc_value/3 7 0.00 7 [ 1.00]
+prim_inet:enum_val/2 6 0.00 7 [ 1.17]
+prim_inet:dec_opt_val/1 7 0.00 7 [ 1.00]
+prim_inet:dec_value/2 6 0.00 10 [ 1.67]
+prim_inet:enc_opt/1 13 0.00 12 [ 0.92]
+prim_inet:type_opt/2 19 0.00 33 [ 1.74]
+erlang:port_control/3 3 0.00 59 [ 19.67]
+tcp_test:read_loop/1 20716370 100.00 12187488 [ 0.59]
+```
+
+This time our process spent almost no time at all (according to eprof, 0%)
+fiddling with the socket opts. Instead it spent all of its time in the
+read_loop doing the work we actually want to be doing.
+
+## So what does this mean?
+
+I'm by no means advocating never using `{active,once}`. The security concern is
+still a completely valid concern and one that `{active,once}` mitigates quite
+well. I'm simply pointing out that this mitigation has some fairly serious
+performance implications which have the potential to bite you if you're not
+careful, especially in cases where a socket is going to be receiving a large
+amount of traffic.
+
+## Meta
+
+These tests were done using R15B03, but I've done similar ones in R14 and found
+similar results. I have not tested R16.
+
+* \[0] http://learnyousomeerlang.com/buckets-of-sockets
+* \[1] http://www.erlang.org/doc/man/gen_tcp.html#examples
+* \[2] http://erlycoder.com/25/erlang-tcp-server-tcp-client-sockets-with-gen_tcp
+
+[0]: http://learnyousomeerlang.com/content
+[1]: http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1
+[2]: http://www.erlang.org/doc/man/inet.html#setopts-2
+[3]: http://www.erlang.org/doc/man/gen_tcp.html#recv-2
+[4]: http://www.erlang.org/doc/man/eprof.html
diff --git a/src/_posts/2013-07-11-goplus.md b/src/_posts/2013-07-11-goplus.md
new file mode 100644
index 0000000..5ee121e
--- /dev/null
+++ b/src/_posts/2013-07-11-goplus.md
@@ -0,0 +1,77 @@
+---
+title: Go+
+description: >-
+ A simple proof-of-concept script for doing go dependency management.
+---
+
+Compared to other languages go has some strange behavior regarding its project
+root settings. If you import a library called `somelib`, go will look for a
+`src/somelib` folder in all of the folders in the `$GOPATH` environment
+variable. This works nicely for globally installed packages, but it makes
+encapsulating a project with a specific version, or modified version, rather
+tedious. Whenever you go to work on this project you'll have to add its path to
+your `$GOPATH`, or add the path permanently, which could break other projects
+which may use a different version of `somelib`.
+
+My solution is in the form of a simple script I'm calling go+. go+ will search
+in currrent directory and all of its parents for a file called `GOPROJROOT`. If
+it finds that file in a directory, it prepends that directory's absolute path to
+your `$GOPATH` and stops the search. Regardless of whether or not `GOPROJROOT`
+was found go+ will passthrough all arguments to the actual go call. The
+modification to `$GOPATH` will only last the duration of the call.
+
+As an example, consider the following:
+```
+/tmp
+ /hello
+ GOPROJROOT
+ /src
+ /somelib/somelib.go
+ /hello.go
+```
+
+If `hello.go` depends on `somelib`, as long as you run go+ from `/tmp/hello` or
+one of its children your project will still compile
+
+Here is the source code for go+:
+
+```bash
+#!/bin/sh
+
+SEARCHING_FOR=GOPROJROOT
+ORIG_DIR=$(pwd)
+
+STOPSEARCH=0
+SEARCH_DIR=$ORIG_DIR
+while [ $STOPSEARCH = 0 ]; do
+
+ RES=$( find $SEARCH_DIR -maxdepth 1 -type f -name $SEARCHING_FOR | \
+ grep -P "$SEARCHING_FOR$" | \
+ head -n1 )
+
+ if [ "$RES" = "" ]; then
+ if [ "$SEARCH_DIR" = "/" ]; then
+ STOPSEARCH=1
+ fi
+ cd ..
+ SEARCH_DIR=$(pwd)
+ else
+ export GOPATH=$SEARCH_DIR:$GOPATH
+ STOPSEARCH=1
+ fi
+done
+
+cd "$ORIG_DIR"
+exec go $@
+```
+
+## UPDATE: Goat
+
+I'm leaving this post for posterity, but go+ has some serious flaws in it. For
+one, it doesn't allow for specifying the version of a dependency you want to
+use. To this end, I wrote [goat][0] which does all the things go+ does, plus
+real dependency management, PLUS it is built in a way that if you've been
+following go's best-practices for code organization you shouldn't have to change
+any of your existing code AT ALL. It's cool, check it out.
+
+[0]: http://github.com/mediocregopher/goat
diff --git a/src/_posts/2013-10-08-generations.md b/src/_posts/2013-10-08-generations.md
new file mode 100644
index 0000000..c1c433d
--- /dev/null
+++ b/src/_posts/2013-10-08-generations.md
@@ -0,0 +1,100 @@
+---
+title: Generations
+description: >-
+ A simple file distribution strategy for very large scale, high-availability
+ file-services.
+---
+
+## The problem
+
+At [cryptic.io][cryptic] we plan on having millions of different
+files, any of which could be arbitrarily chosen to be served any given time.
+These files are uploaded by users at arbitrary times.
+
+Scaling such a system is no easy task. The solution I've seen implemented in the
+past involves shuffling files around on a nearly constant basis, making sure
+that files which are more "popular" are on fast drives, while at the same time
+making sure that no drives are at capicty and at the same time that all files,
+even newly uploaded ones, are stored redundantly.
+
+The problem with this solution is one of coordination. At any given moment the
+app needs to be able to "find" a file so it can give the client a link to
+download the file from one of the servers that it's on. Full-filling this simple
+requirement means that all datastores/caches where information about where a
+file lives need to be up-to-date at all times, and even then there are
+race-conditions and network failures to contend with, while at all times the
+requirements of the app evolve and change.
+
+## A simpler solution
+
+Let's say you want all files which get uploaded to be replicated in triplicate
+in some capacity. You buy three identical hard-disks, and put each on a separate
+server. As files get uploaded by clients, each file gets put on each drive
+immediately. When the drives are filled (which should be at around the same
+time), you stop uploading to them.
+
+That was generation 0.
+
+You buy three more drives, and start putting all files on them instead. This is
+going to be generation 1. Repeat until you run out of money.
+
+That's it.
+
+### That's it?
+
+It seems simple and obvious, and maybe it's the standard thing which is done,
+but as far as I can tell no-one has written about it (though I'm probably not
+searching for the right thing, let me know if this is the case!).
+
+### Advantages
+
+* It's so simple to implement, you could probably do it in a day if you're
+starting a project from scratch
+
+* By definition of the scheme all files are replicated in multiple places.
+
+* Minimal information about where a file "is" needs to be stored. When a file is
+uploaded all that's needed is to know what generation it is in, and then what
+nodes/drives are in that generation. If the file's name is generated
+server-side, then the file's generation could be *part* of its name, making
+lookup even faster.
+
+* Drives don't need to "know" about each other. What I mean by this is that
+whatever is running as the receive point for file-uploads on each drive doesn't
+have to coordinate with its siblings running on the other drives in the
+generation. In fact it doesn't need to coordinate with anyone. You could
+literally rsync files onto your drives if you wanted to. I would recommend using
+[marlin][0] though :)
+
+* Scaling is easy. When you run out of space you can simply start a new
+generation. If you don't like playing that close to the chest there's nothing to
+say you can't have two generations active at the same time.
+
+* Upgrading is easy. As long as a generation is not marked-for-upload, you can
+easily copy all files in the generation into a new set of bigger, badder drives,
+add those drives into the generation in your code, remove the old ones, then
+mark the generation as uploadable again.
+
+* Distribution is easy. You just copy a generation's files onto a new drive in
+Europe or wherever you're getting an uptick in traffic from and you're good to
+go.
+
+* Management is easy. It's trivial to find out how many times a file has been
+replicated, or how many countries it's in, or what hardware it's being served
+from (given you have easy access to information about specific drives).
+
+### Caveats
+
+The big caveat here is that this is just an idea. It has NOT been tested in
+production. But we have enough faith in it that we're going to give it a shot at
+[cryptic.io][cryptic]. I'll keep this page updated.
+
+The second caveat is that this scheme does not inherently support caching. If a
+file suddenly becomes super popular the world over your hard-disks might not be
+able to keep up, and it's probably not feasible to have an FIO drive in *every*
+generation. I think that [groupcache][1] may be the answer to this problem,
+assuming your files are reasonably small, but again I haven't tested it yet.
+
+[cryptic]: https://cryptic.io
+[0]: https://github.com/cryptic-io/marlin
+[1]: https://github.com/golang/groupcache
diff --git a/src/_posts/2013-10-25-namecoind-ssl.md b/src/_posts/2013-10-25-namecoind-ssl.md
new file mode 100644
index 0000000..2711a92
--- /dev/null
+++ b/src/_posts/2013-10-25-namecoind-ssl.md
@@ -0,0 +1,248 @@
+---
+title: Namecoin, A Replacement For SSL
+description: >-
+ If we use the namecoin chain as a DNS service we get security almost for
+ free, along with lots of other benefits.
+---
+
+At [cryptic.io][cryptic] we are creating a client-side, in-browser encryption
+system where a user can upload their already encrypted content to our storage
+system and be 100% confident that their data can never be decrypted by anyone
+but them.
+
+One of the main problems with this approach is that the client has to be sure
+that the code that's being run in their browser is the correct code; that is,
+that they aren't the subject of a man-in-the-middle attack where an attacker is
+turning our strong encryption into weak encryption that they could later break.
+
+A component of our current solution is to deliver the site's javascript (and all
+other assets, for that matter) using SSL encryption. This protects the files
+from tampering in-between leaving our servers and being received by the client.
+Unfortunately, SSL isn't 100% foolproof. This post aims to show why SSL is
+faulty, and propose a solution.
+
+## SSL
+
+SSL is the mechanism by which web-browsers establish an encrypted connection to
+web-servers. The goal of this connection is that only the destination
+web-browser and the server know what data is passing between them. Anyone spying
+on the connection would only see gibberish. To do this a secret key is first
+established between the client and the server, and used to encrypt/decrypt all
+data. As long as no-one but those parties knows that key, that data will never
+be decrypted by anyone else.
+
+SSL is what's used to establish that secret key on a per-session basis, so that
+a key isn't ever re-used and so only the client and the server know it.
+
+### Public-Private Key Cryptography
+
+SSL is based around public-private key cryptography. In a public-private key
+system, you have both a public key which is generated from a private key. The
+public key can be given to anyone, but the private key must remain hidden. There
+are two main uses for these two keys:
+
+* Someone can encrypt a message with your public key, and only you (with the
+ private key) can decrypt it.
+
+* You can sign a message with your private key, and anyone with your public key
+ can verify that it was you and not someone else who signed it.
+
+These are both extremely useful functions, not just for internet traffic but for
+any kind of communication form. Unfortunately, there remains a fundamental flaw.
+At some point you must give your public key to the other person in an insecure
+way. If an attacker was to intercept your message containing your public key and
+swap it for their own, then all future communications could be compromised. That
+attacker could create messages the other person would think are from you, and
+the other person would encrypt messages meant for you but which would be
+decrypt-able by the attacker.
+
+### How does SSL work?
+
+SSL is at its heart a public-private key system, but its aim is to be more
+secure against the attack described above.
+
+SSL uses a trust-chain to verify that a public key is the intended one. Your web
+browser has a built-in set of public keys, called the root certificates, that it
+implicitly trusts. These root certificates are managed by a small number of
+companies designated by some agency who decides on these things.
+
+When you receive a server's SSL certificate (its public key) that certificate
+will be signed by a root certificate. You can verify that signature since you
+have the root certificate's public key built into your browser. If the signature
+checks out then you know a certificate authority trusts the public key the site
+gave you, which means you can trust it too.
+
+There's a bit (a lot!) more to SSL than this, but this is enough to understand
+the fundamental problems with it.
+
+### How SSL doesn't work
+
+SSL has a few glaring problems. One, it implies we trust the companies holding
+the root certificates to not be compromised. If some malicious agency was to get
+ahold of a root certificate they could listen in on any connection on the
+internet by swapping a site's real certificate with one they generate on the
+fly. They could trivially steal any data we send on the internet.
+
+The second problem is that it's expensive. Really expensive. If you're running a
+business you'll have to shell out about $200 a year to keep your SSL certificate
+signed (those signatures have an expiration date attached). Since there's very
+few root authorities there's an effective monopoly on signatures, and there's
+nothing we can do about it. For 200 bucks I know most people simply say "no
+thanks" and go unencrypted. The solution is creating a bigger problem.
+
+## Bitcoins
+
+Time to switch gears, and propose a solution to the above issues: namecoins. I'm
+going to first talk about what namecoins are, how they work, and why we need
+them. To start with, namecoins are based on bitcoins.
+
+If you haven't yet checked out bitcoins, [I highly encourage you to do
+so][bitcoins]. They're awesome, and I think they have a chance of really
+changing the way we think of and use money in the future. At the moment they're
+still a bit of a novelty in the tech realm, but they're growing in popularity.
+
+The rest of this post assumes you know more or less what bitcoins are, and how
+they work.
+
+## Namecoins
+
+Few people actually know about bitcoins. Even fewer know that there's other
+crypto-currencies besides bitcoins. Basically, developers of these alternative
+currencies (altcoins, in the parlance of our times) took the original bitcoin
+source code and modified it to produce a new, separate blockchain from the
+original bitcoin one. The altcoins are based on the same idea as bitcoins
+(namely, a chain of blocks representing all the transactions ever made), but
+have slightly different characterstics.
+
+One of these altcoins is called namecoin. Where other altcoins aim to be digital
+currencies, and used as such (like bitcoins), namecoin has a different goal. The
+point of namecoin is to create a global, distributed, secure key-value store.
+You spend namecoins to claim arbitrary keys (once you've claimed it, you own it
+for a set period of time) and to give those keys arbitrary values. Anyone else
+with namecoind running can see these values.
+
+### Why use it?
+
+A blockchain based on a digital currency seems like a weird idea at first. I
+know when I first read about it I was less than thrilled. How is this better
+than a DHT? It's a key-value store, why is there a currency involved?
+
+#### DHT
+
+DHT stands for Distributed Hash-Table. I'm not going to go too into how they
+work, but suffice it to say that they are essentially a distributed key-value
+store. Like namecoin. The difference is in the operation. DHTs operate by
+spreading and replicating keys and their values across nodes in a P2P mesh. They
+have [lots of issues][dht] as far as security goes, the main one being that it's
+fairly easy for an attacker to forge the value for a given key, and very
+difficult to stop them from doing so or even to detect that it's happened.
+
+Namecoins don't have this problem. To forge a particular key an attacker would
+essentially have to create a new blockchain from a certain point in the existing
+chain, and then replicate all the work put into the existing chain into that new
+compromised one so that the new one is longer and other clients in the network
+will except it. This is extremely non-trivial.
+
+#### Why a currency?
+
+To answer why a currency needs to be involved, we need to first look at how
+bitcoin/namecoin work. When you take an action (send someone money, set a value
+to a key) that action gets broadcast to the network. Nodes on the network
+collect these actions into a block, which is just a collection of multiple
+actions. Their goal is to find a hash of this new block, combined with some data
+from the top-most block in the existing chain, combined with some arbitrary
+data, such that the first n characters in the resulting hash are zeros (with n
+constantly increasing). When they find one they broadcast it out on the network.
+Assuming the block is legitimate they receive some number of coins as
+compensation.
+
+That compensation is what keeps a blockchain based currency going. If there
+were no compensation there would be no reason to mine except out of goodwill, so
+far fewer people would do it. Since the chain can be compromised if a malicious
+group has more computing power than all legitimate miners combined, having few
+legitimate miners is a serious problem.
+
+In the case of namecoins, there's even more reason to involve a currency. Since
+you have to spend money to make changes to the chain there's a disincentive for
+attackers (read: idiots) to spam the chain with frivolous changes to keys.
+
+#### Why a *new* currency?
+
+I'll admit, it's a bit annoying to see all these altcoins popping up. I'm sure
+many of them have some solid ideas backing them, but it also makes things
+confusing for newcomers and dilutes the "market" of cryptocoin users; the more
+users a particular chain has, the stronger it is. If we have many chains, all we
+have are a bunch of weak chains.
+
+The exception to this gripe, for me, is namecoin. When I was first thinking
+about this problem my instinct was to just use the existing bitcoin blockchain
+as a key-value storage. However, the maintainers of the bitcoin clients
+(who are, in effect, the maintainers of the chain) don't want the bitcoin
+blockchain polluted with non-commerce related data. At first I disagreed; it's a
+P2P network, no-one gets to say what I can or can't use the chain for! And
+that's true. But things work out better for everyone involved if there's two
+chains.
+
+Bitcoin is a currency. Namecoin is a key-value store (with a currency as its
+driving force). Those are two completely different use-cases, with two
+completely difference usage characteristics. And we don't know yet what those
+characteristics are, or if they'll change. If the chain-maintainers have to deal
+with a mingled chain we could very well be tying their hands with regards to
+what they can or can't change with regards to the behavior of the chain, since
+improving performance for one use-case may hurt the performance of the other.
+With two separate chains the maintainers of each are free to do what they see
+fit to keep their respective chains operating as smoothly as possible.
+Additionally, if for some reason bitcoins fall by the wayside, namecoin will
+still have a shot at continuing operation since it isn't tied to the former.
+Tldr: separation of concerns.
+
+## Namecoin as an alternative to SSL
+
+And now to tie it all together.
+
+There are already a number of proposed formats for standardizing how we store
+data on the namecoin chain so that we can start building tools around it. I'm
+not hugely concerned with the particulars of those standards, only that we can,
+in some way, standardize on attaching a public key (or a fingerprint of one) to
+some key on the namecoin blockchain. When you visit a website, the server
+would then send both its public key and the namecoin chain key to be checked
+against to the browser, and the browser would validate that the public key it
+received is the same as the one on the namecoin chain.
+
+The main issue with this is that it requires another round-trip when visiting a
+website: One for DNS, and one to check the namecoin chain. And where would this
+chain even be hosted?
+
+My proposition is there would exist a number of publicly available servers
+hosting a namecoind process that anyone in the world could send requests for
+values on the chain. Browsers could then be made with a couple of these
+hardwired in. ISPs could also run their own copies at various points in their
+network to improve response-rates and decrease load on the globally public
+servers. Furthermore, the paranoid could host their own and be absolutely sure
+that the data they're receiving is valid.
+
+If the above scheme sounds a lot like what we currently use for DNS, that's
+because it is. In fact, one of namecoin's major goals is that it be used as a
+replacement for DNS, and most of the talk around it is focused on this subject.
+DNS has many of the same problems as SSL, namely single-point-of-failure and
+that it's run by a centralized agency that we have to pay arbitrarily high fees
+to. By switching our DNS and SSL infrastructure to use namecoin we could kill
+two horribly annoying, monopolized, expensive birds with a single stone.
+
+That's it. If we use the namecoin chain as a DNS service we get security almost
+for free, along with lots of other benefits. To make this happen we need
+cooperation from browser makers, and to standardize on a simple way of
+retrieving DNS information from the chain that the browsers can use. The
+protocol doesn't need to be very complex, I think HTTP/REST should suffice,
+since the meat of the data will be embedded in the JSON value on the namecoin
+chain.
+
+If you want to contribute or learn more please check out [namecoin][nmc] and
+specifically the [d namespace proposal][dns] for it.
+
+[cryptic]: http://cryptic.io
+[bitcoins]: http://vimeo.com/63502573
+[dht]: http://www.globule.org/publi/SDST_acmcs2009.pdf
+[nsa]: https://www.schneier.com/blog/archives/2013/09/new_nsa_leak_sh.html
+[nmc]: http://dot-bit.org/Main_Page
+[dns]: http://dot-bit.org/Namespace:Domain_names_v2.0
diff --git a/src/_posts/2014-01-11-diamond-square.md b/src/_posts/2014-01-11-diamond-square.md
new file mode 100644
index 0000000..665e07c
--- /dev/null
+++ b/src/_posts/2014-01-11-diamond-square.md
@@ -0,0 +1,494 @@
+---
+title: Diamond Square
+description: >-
+ Tackling the problem of semi-realistic looking terrain generation in
+ clojure.
+updated: 2018-09-06
+---
+
+![terrain][terrain]
+
+I recently started looking into the diamond-square algorithm (you can find a
+great article on it [here][diamondsquare]). The following is a short-ish
+walkthrough of how I tackled the problem in clojure and the results. You can
+find the [leiningen][lein] repo [here][repo] and follow along within that, or
+simply read the code below to get an idea.
+
+Also, Marco ported my code into clojurescript, so you can get random terrain
+in your browser. [Check it out!][marco]
+
+```clojure
+(ns diamond-square.core)
+
+; == The Goal ==
+; Create a fractal terrain generator using clojure
+
+; == The Algorithm ==
+; Diamond-Square. We start with a grid of points, each with a height of 0.
+;
+; 1. Take each corner point of the square, average the heights, and assign that
+; to be the height of the midpoint of the square. Apply some random error to
+; the midpoint.
+;
+; 2. Creating a line from the midpoint to each corner we get four half-diamonds.
+; Average the heights of the points (with some random error) and assign the
+; heights to the midpoints of the diamonds.
+;
+; 3. We now have four square sections, start at 1 for each of them (with
+; decreasing amount of error for each iteration).
+;
+; This picture explains it better than I can:
+; https://blog.mediocregopher.com/img/diamond-square/dsalg.png
+; (http://nbickford.wordpress.com/2012/12/21/creating-fake-landscapes/dsalg/)
+;
+; == The Strategy ==
+; We begin with a vector of vectors of numbers, and iterate over it, filling in
+; spots as they become available. Our grid will have the top-left being (0,0),
+; y being pointing down and x going to the right. The outermost vector
+; indicating row number (y) and the inner vectors indicate the column number (x)
+;
+; = Utility =
+; First we create some utility functions for dealing with vectors of vectors.
+
+(defn print-m
+ "Prints a grid in a nice way"
+ [m]
+ (doseq [n m]
+ (println n)))
+
+(defn get-m
+ "Gets a value at the given x,y coordinate of the grid, with [0,0] being in the
+ top left"
+ [m x y]
+ ((m y) x))
+
+(defn set-m
+ "Sets a value at the given x,y coordinat of the grid, with [0,0] being in the
+ top left"
+ [m x y v]
+ (assoc m y
+ (assoc (m y) x v)))
+
+(defn add-m
+ "Like set-m, but adds the given value to the current on instead of overwriting
+ it"
+ [m x y v]
+ (set-m m x y
+ (+ (get-m m x y) v)))
+
+(defn avg
+ "Returns the truncated average of all the given arguments"
+ [& l]
+ (int (/ (reduce + l) (count l))))
+
+; = Grid size =
+; Since we're starting with a blank grid we need to find out what sizes the
+; grids can be. For convenience the size (height and width) should be odd, so we
+; easily get a midpoint. And on each iteration we'll be halfing the grid, so
+; whenever we do that the two resultrant grids should be odd and halfable as
+; well, and so on.
+;
+; The algorithm that fits this is size = 2^n + 1, where 1 <= n. For the rest of
+; this guide I'll be referring to n as the "degree" of the grid.
+
+
+(def exp2-pre-compute
+ (vec (map #(int (Math/pow 2 %)) (range 31))))
+
+(defn exp2
+ "Returns 2^n as an integer. Uses pre-computed values since we end up doing
+ this so much"
+ [n]
+ (exp2-pre-compute n))
+
+(def grid-sizes
+ (vec (map #(inc (exp2 %)) (range 1 31))))
+
+(defn grid-size [degree]
+ (inc (exp2 degree)))
+
+; Available grid heights/widths are as follows:
+;[3 5 9 17 33 65 129 257 513 1025 2049 4097 8193 16385 32769 65537 131073
+;262145 524289 1048577 2097153 4194305 8388609 16777217 33554433 67108865
+;134217729 268435457 536870913 1073741825])
+
+(defn blank-grid
+ "Generates a grid of the given degree, filled in with zeros"
+ [degree]
+ (let [gsize (grid-size degree)]
+ (vec (repeat gsize
+ (vec (repeat gsize 0))))))
+
+(comment
+ (print-m (blank-grid 3))
+)
+
+; = Coordinate Pattern (The Tricky Part) =
+; We now have to figure out which coordinates need to be filled in on each pass.
+; A pass is defined as a square step followed by a diamond step. The next pass
+; will be the square/dimaond steps on all the smaller squares generated in the
+; pass. It works out that the number of passes required to fill in the grid is
+; the same as the degree of the grid, where the first pass is 1.
+;
+; So we can easily find patterns in the coordinates for a given degree/pass,
+; I've laid out below all the coordinates for each pass for a 3rd degree grid
+; (which is 9x9).
+
+; Degree 3 Pass 1 Square
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . . . 1 . . . .] (4,4)
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+
+; Degree 3 Pass 1 Diamond
+; [. . . . 2 . . . .] (4,0)
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [2 . . . . . . . 2] (0,4) (8,4)
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . . . 2 . . . .] (4,8)
+
+; Degree 3 Pass 2 Square
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . 3 . . . 3 . .] (2,2) (6,2)
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+; [. . 3 . . . 3 . .] (2,6) (6,6)
+; [. . . . . . . . .]
+; [. . . . . . . . .]
+
+; Degree 3 Pass 2 Diamond
+; [. . 4 . . . 4 . .] (2,0) (6,0)
+; [. . . . . . . . .]
+; [4 . . . 4 . . . 4] (0,2) (4,2) (8,2)
+; [. . . . . . . . .]
+; [. . 4 . . . 4 . .] (2,4) (6,4)
+; [. . . . . . . . .]
+; [4 . . . 4 . . . 4] (0,6) (4,6) (8,6)
+; [. . . . . . . . .]
+; [. . 4 . . . 4 . .] (2,8) (6,8)
+
+; Degree 3 Pass 3 Square
+; [. . . . . . . . .]
+; [. 5 . 5 . 5 . 5 .] (1,1) (3,1) (5,1) (7,1)
+; [. . . . . . . . .]
+; [. 5 . 5 . 5 . 5 .] (1,3) (3,3) (5,3) (7,3)
+; [. . . . . . . . .]
+; [. 5 . 5 . 5 . 5 .] (1,5) (3,5) (5,5) (7,5)
+; [. . . . . . . . .]
+; [. 5 . 5 . 5 . 5 .] (1,7) (3,7) (5,7) (7,7)
+; [. . . . . . . . .]
+
+; Degree 3 Pass 3 Square
+; [. 6 . 6 . 6 . 6 .] (1,0) (3,0) (5,0) (7,0)
+; [6 . 6 . 6 . 6 . 6] (0,1) (2,1) (4,1) (6,1) (8,1)
+; [. 6 . 6 . 6 . 6 .] (1,2) (3,2) (5,2) (7,2)
+; [6 . 6 . 6 . 6 . 6] (0,3) (2,3) (4,3) (6,3) (8,3)
+; [. 6 . 6 . 6 . 6 .] (1,4) (3,4) (5,4) (7,4)
+; [6 . 6 . 6 . 6 . 6] (0,5) (2,5) (4,5) (6,5) (8,5)
+; [. 6 . 6 . 6 . 6 .] (1,6) (3,6) (5,6) (7,6)
+; [6 . 6 . 6 . 6 . 6] (0,7) (2,7) (4,7) (6,7) (8,7)
+; [. 6 . 6 . 6 . 6 .] (1,8) (3,8) (5,8) (7,8)
+;
+; I make two different functions, one to give the coordinates for the square
+; portion of each pass and one for the diamond portion of each pass. To find the
+; actual patterns it was useful to first look only at the pattern in the
+; y-coordinates, and figure out how that translated into the pattern for the
+; x-coordinates.
+
+(defn grid-square-coords
+ "Given a grid degree and pass number, returns all the coordinates which need
+ to be computed for the square step of that pass"
+ [degree pass]
+ (let [gsize (grid-size degree)
+ start (exp2 (- degree pass))
+ interval (* 2 start)
+ coords (map #(+ start (* interval %))
+ (range (exp2 (dec pass))))]
+ (mapcat (fn [y]
+ (map #(vector % y) coords))
+ coords)))
+;
+; (grid-square-coords 3 2)
+; => ([2 2] [6 2] [2 6] [6 6])
+
+(defn grid-diamond-coords
+ "Given a grid degree and a pass number, returns all the coordinates which need
+ to be computed for the diamond step of that pass"
+ [degree pass]
+ (let [gsize (grid-size degree)
+ interval (exp2 (- degree pass))
+ num-coords (grid-size pass)
+ coords (map #(* interval %) (range 0 num-coords))]
+ (mapcat (fn [y]
+ (if (even? (/ y interval))
+ (map #(vector % y) (take-nth 2 (drop 1 coords)))
+ (map #(vector % y) (take-nth 2 coords))))
+ coords)))
+
+; (grid-diamond-coords 3 2)
+; => ([2 0] [6 0] [0 2] [4 2] [8 2] [2 4] [6 4] [0 6] [4 6] [8 6] [2 8] [6 8])
+
+; = Height Generation =
+; We now work on functions which, given a coordinate, will return what value
+; coordinate will have.
+
+(defn avg-points
+ "Given a grid and an arbitrary number of points (of the form [x y]) returns
+ the average of all the given points that are on the map. Any points which are
+ off the map are ignored"
+ [m & coords]
+ (let [grid-size (count m)]
+ (apply avg
+ (map #(apply get-m m %)
+ (filter
+ (fn [[x y]]
+ (and (< -1 x) (> grid-size x)
+ (< -1 y) (> grid-size y)))
+ coords)))))
+
+(defn error
+ "Returns a number between -e and e, inclusive"
+ [e]
+ (- (rand-int (inc (* 2 e))) e))
+
+; The next function is a little weird. It primarily takes in a point, then
+; figures out the distance from that point to the points we'll take the average
+; of. The locf (locator function) is used to return back the actual points to
+; use. For the square portion it'll be the points diagonal from the given one,
+; for the diamond portion it'll be the points to the top/bottom/left/right from
+; the given one.
+;
+; Once it has those points, it finds the average and applies the error. The
+; error function is nothing more than a number between -interval and +interval,
+; where interval is the distance between the given point and one of the averaged
+; points. It is important that the error decreases the more passes you do, which
+; is why the interval is used.
+;
+; The error function is what should be messed with primarily if you want to
+; change what kind of terrain you generate (a giant mountain instead of
+; hills/valleys, for example). The one we use is uniform for all intervals, so
+; it generates a uniform terrain.
+
+(defn- grid-fill-point
+ [locf m degree pass x y]
+ (let [interval (exp2 (- degree pass))
+ leftx (- x interval)
+ rightx (+ x interval)
+ upy (- y interval)
+ downy (+ y interval)
+ v (apply avg-points m
+ (locf x y leftx rightx upy downy))]
+ (add-m m x y (+ v (error interval)))))
+
+(def grid-fill-point-square
+ "Given a grid, the grid's degree, the current pass number, and a point on the
+ grid, fills in that point with the average (plus some error) of the
+ appropriate corner points, and returns the resultant grid"
+ (partial grid-fill-point
+ (fn [_ _ leftx rightx upy downy]
+ [[leftx upy]
+ [rightx upy]
+ [leftx downy]
+ [rightx downy]])))
+
+(def grid-fill-point-diamond
+ "Given a grid, the grid's degree, the current pass number, and a point on the
+ grid, fills in that point with the average (plus some error) of the
+ appropriate edge points, and returns the resultant grid"
+ (partial grid-fill-point
+ (fn [x y leftx rightx upy downy]
+ [[leftx y]
+ [rightx y]
+ [x upy]
+ [x downy]])))
+
+; = Filling in the Grid =
+; We finally compose the functions we've been creating to fill in the entire
+; grid
+
+(defn- grid-fill-point-passes
+ "Given a grid, a function to fill in coordinates, and a function to generate
+ those coordinates, fills in all coordinates for a given pass, returning the
+ resultant grid"
+ [m fill-f coord-f degree pass]
+ (reduce
+ (fn [macc [x y]] (fill-f macc degree pass x y))
+ m
+ (coord-f degree pass)))
+
+(defn grid-pass
+ "Given a grid and a pass number, does the square then the diamond portion of
+ the pass"
+ [m degree pass]
+ (-> m
+ (grid-fill-point-passes
+ grid-fill-point-square grid-square-coords degree pass)
+ (grid-fill-point-passes
+ grid-fill-point-diamond grid-diamond-coords degree pass)))
+
+; The most important function in this guide, does all the work
+(defn terrain
+ "Given a grid degree, generates a uniformly random terrain on a grid of that
+ degree"
+ ([degree]
+ (terrain (blank-grid degree) degree))
+ ([m degree]
+ (reduce
+ #(grid-pass %1 degree %2)
+ m
+ (range 1 (inc degree)))))
+
+(comment
+ (print-m
+ (terrain 5))
+)
+
+; == The Results ==
+; We now have a generated terrain, probably. We should check it. First we'll
+; create an ASCII representation. But to do that we'll need some utility
+; functions.
+
+(defn max-terrain-height
+ "Returns the maximum height found in the given terrain grid"
+ [m]
+ (reduce max
+ (map #(reduce max %) m)))
+
+(defn min-terrain-height
+ "Returns the minimum height found in the given terrain grid"
+ [m]
+ (reduce min
+ (map #(reduce min %) m)))
+
+(defn norm
+ "Given x in the range (A,B), normalizes it into the range (0,new-height)"
+ [A B new-height x]
+ (int (/ (* (- x A) new-height) (- B A))))
+
+(defn normalize-terrain
+ "Given a terrain map and a number of \"steps\", normalizes the terrain so all
+ heights in it are in the range (0,steps)"
+ [m steps]
+ (let [max-height (max-terrain-height m)
+ min-height (min-terrain-height m)
+ norm-f (partial norm min-height max-height steps)]
+ (vec (map #(vec (map norm-f %)) m))))
+
+; We now define which ASCII characters we want to use for which heights. The
+; vector starts with the character for the lowest height and ends with the
+; character for the heighest height.
+
+(def tiles
+ [\~ \~ \" \" \x \x \X \$ \% \# \@])
+
+(defn tile-terrain
+ "Given a terrain map, converts it into an ASCII tile map"
+ [m]
+ (vec (map #(vec (map tiles %))
+ (normalize-terrain m (dec (count tiles))))))
+
+(comment
+ (print-m
+ (tile-terrain
+ (terrain 5)))
+
+; [~ ~ " " x x x X % $ $ $ X X X X X X $ x x x X X X x x x x " " " ~]
+; [" ~ " " x x X X $ $ $ X X X X X X X X X X X X X X x x x x " " " "]
+; [" " " x x x X X % $ % $ % $ $ X X X X $ $ $ X X X X x x x x " " "]
+; [" " " x x X $ % % % % % $ % $ $ X X $ $ $ $ X X x x x x x x " " x]
+; [" x x x x X $ $ # % % % % % % $ X $ X X % $ % X X x x x x x x x x]
+; [x x x X $ $ $ % % % % % $ % $ $ $ % % $ $ $ $ X X x x x x x x x x]
+; [X X X $ % $ % % # % % $ $ % % % % $ % $ $ X $ X $ X X x x x X x x]
+; [$ $ X $ $ % $ % % % % $ $ $ % # % % % X X X $ $ $ X X X x x x x x]
+; [% X X % % $ % % % $ % $ % % % # @ % $ $ X $ X X $ X x X X x x x x]
+; [$ $ % % $ $ % % $ $ X $ $ % % % % $ $ X $ $ X X X X X X x x x x x]
+; [% % % X $ $ % $ $ X X $ $ $ $ % % $ $ X X X $ X X X x x X x x X X]
+; [$ $ $ X $ $ X $ X X X $ $ $ $ % $ $ $ $ $ X $ X x X X X X X x X X]
+; [$ $ $ $ X X $ X X X X X $ % % % % % $ X $ $ $ X x X X X $ X X $ $]
+; [X $ $ $ $ $ X X X X X X X % $ % $ $ $ X X X X X x x X X x X X $ $]
+; [$ $ X X $ X X x X $ $ X X $ % X X X X X X X X X x X X x x X X X X]
+; [$ $ X X X X X X X $ $ $ $ $ X $ X X X X X X X x x x x x x x X X X]
+; [% % % $ $ X $ X % X X X % $ $ X X X X X X x x x x x x x x x X X $]
+; [$ % % $ $ $ X X $ $ $ $ $ $ X X X X x X x x x x " x x x " x x x x]
+; [$ X % $ $ $ $ $ X X X X X $ $ X X X X X X x x " " " " " " " " x x]
+; [$ X $ $ % % $ X X X $ X X X x x X X x x x x x " " " " " ~ " " " "]
+; [$ $ X X % $ % X X X X X X X X x x X X X x x x " " " " " " ~ " " "]
+; [$ $ X $ % $ $ X X X X X X x x x x x x x x x " " " " " " " " " ~ ~]
+; [$ $ $ $ $ X X $ X X X X X x x x x x x x x " " " " " " " ~ " " " ~]
+; [$ % X X $ $ $ $ X X X X x x x x x x x x x x " " " " ~ " " ~ " " ~]
+; [% $ $ X $ X $ X $ X $ X x x x x x x x x x x " " " " ~ ~ ~ " ~ " ~]
+; [$ X X X X $ $ $ $ $ X x x x x x x x x x x " " " " ~ ~ ~ ~ ~ ~ ~ ~]
+; [X x X X x X X X X X X X X x x x x x x x x x " " " ~ ~ " " ~ ~ ~ ~]
+; [x x x x x x X x X X x X X X x x x x x x x " x " " " " " ~ ~ ~ ~ ~]
+; [x x x x x x x x X X X X $ X X x X x x x x x x x x " ~ ~ ~ ~ ~ ~ ~]
+; [" x x x x x X x X X X X X X X X X x x x x x x " " " " ~ ~ ~ ~ ~ ~]
+; [" " " x x x X X X X $ $ $ X X X X X X x x x x x x x x " " ~ ~ ~ ~]
+; [" " " " x x x X X X X X $ $ X X x X X x x x x x x x " " " " " ~ ~]
+; [~ " " x x x x X $ X $ X $ $ X x X x x x x x x x x x x x x " " " ~]
+)
+
+; = Pictures! =
+; ASCII is cool, but pictures are better. First we import some java libraries
+; that we'll need, then define the colors for each level just like we did tiles
+; for the ascii representation.
+
+(import
+ 'java.awt.image.BufferedImage
+ 'javax.imageio.ImageIO
+ 'java.io.File)
+
+(def colors
+ [0x1437AD 0x04859D 0x007D1C 0x007D1C 0x24913C
+ 0x00C12B 0x38E05D 0xA3A3A4 0x757575 0xFFFFFF])
+
+; Finally we reduce over a BufferedImage instance to output every tile as a
+; single pixel on it.
+
+(defn img-terrain
+ "Given a terrain map and a file name, outputs a png representation of the
+ terrain map to that file"
+ [m file]
+ (let [img (BufferedImage. (count m) (count m) BufferedImage/TYPE_INT_RGB)]
+ (reduce
+ (fn [rown row]
+ (reduce
+ (fn [coln tile]
+ (.setRGB img coln rown (colors tile))
+ (inc coln))
+ 0 row)
+ (inc rown))
+ 0 (normalize-terrain m (dec (count colors))))
+ (ImageIO/write img "png" (File. file))))
+
+(comment
+ (img-terrain
+ (terrain 10)
+ "resources/terrain.png")
+
+ ; https://blog.mediocregopher.com/img/diamond-square/terrain.png
+)
+
+; == Conclusion ==
+; There's still a lot of work to be done. The algorithm starts taking a
+; non-trivial amount of time around the 10th degree, which is only a 1025x1025px
+; image. I need to profile the code and find out where the bottlenecks are. It's
+; possible re-organizing the code to use pmaps instead of reduces in some places
+; could help.
+```
+
+[marco]: http://marcopolo.io/diamond-square/
+[terrain]: /img/diamond-square/terrain.png
+[diamondsquare]: http://www.gameprogrammer.com/fractal.html
+[lein]: https://github.com/technomancy/leiningen
+[repo]: https://github.com/mediocregopher/diamond-square
diff --git a/src/_posts/2014-10-29-erlang-pitfalls.md b/src/_posts/2014-10-29-erlang-pitfalls.md
new file mode 100644
index 0000000..32a8095
--- /dev/null
+++ b/src/_posts/2014-10-29-erlang-pitfalls.md
@@ -0,0 +1,192 @@
+---
+title: Erlang Pitfalls
+description: >-
+ Common pitfalls that people may run into when designing and writing
+ large-scale erlang applications.
+---
+
+I've been involved with a large-ish scale erlang project at Grooveshark since
+sometime around 2011. I started this project knowing absolutely nothing about
+erlang, but now I feel I have accumulated enough knowlege over time that I could
+conceivably give some back. Specifically, common pitfalls that people may run
+into when designing and writing a large-scale erlang application. Some of these
+may show up when searching for them, but some of them you may not even know you
+need to search for.
+
+## now() vs timestamp()
+
+The cononical way of getting the current timestamp in erlang is to use
+`erlang:now()`. This works great at small loads, but if you find your
+application slowing down greatly at highly parallel loads and you're calling
+`erlang:now()` a lot, it may be the culprit.
+
+A property of this method you may not realize is that it is monotonically
+increasing, meaning even if two processes call it at the *exact* same time they
+will both receive different output. This is done through some locking on the
+low-level, as well as a bit of math to balance out the time getting out of sync
+in the scenario.
+
+There are situations where fetching always unique timestamps is useful, such as
+seeding RNGs and generating unique identifiers for things, but usually when
+people fetch a timestamp they just want a timestamp. For these cases,
+`os:timestamp()` can be used. It is not blocked by any locks, it simply returns
+the time.
+
+## The rpc module is slow
+
+The built-in `rpc` module is slower than you'd think. This mostly stems from it
+doing a lot of extra work for every `call` and `cast` that you do, ensuring that
+certain conditions are accounted for. If, however, it's sufficient for the
+calling side to know that a call timed-out on them and not worry about it any
+further you may benefit from simply writing your own rpc module. Alternatively,
+use [one which already exists](https://github.com/cloudant/rexi).
+
+## Don't send anonymous functions between nodes
+
+One of erlang's niceties is transparent message sending between two phsyical
+erlang nodes. Once nodes are connected, a process on one can send any message to
+a process on the other exactly as if they existed on the same node. This is fine
+for many data-types, but for anonymous functions it should be avoided.
+
+For example:
+
+```erlang
+RemotePid ! {fn, fun(I) -> I + 1 end}.
+```
+
+Would be better written as
+
+```erlang
+incr(I) ->
+ I + 1.
+
+RemotePid ! {fn, ?MODULE, incr}.
+```
+
+and then using an `apply` on the RemotePid to actually execute the function.
+
+This is because hot-swapping code messes with anonymous functions quite a bit.
+Erlang isn't actually sending a function definition across the wire; it's simply
+sending a reference to a function. If you've changed the code within the
+anonymous function on a node, that reference changes. The sending node is
+sending a reference to a function which may not exist anymore on the receiving
+node, and you'll get a weird error which Google doesn't return many results for.
+
+Alternatively, if you simply send atoms across the wire and use `apply` on the
+other side, only atoms are sent and the two nodes involved can have totally
+different ideas of what the function itself does without any problems.
+
+## Hot-swapping code is a convenience, not a crutch
+
+Hot swapping code is the bees-knees. It lets you not have to worry about
+rolling-restarts for trivial code changes, and so adds stability to your
+cluster. My warning is that you should not rely on it. If your cluster can't
+survive a node being restarted for a code change, then it can't survive if that
+node fails completely, or fails and comes back up. Design your system pretending
+that hot-swapping does not exist, and only once you've done that allow yourself
+to use it.
+
+## GC sometimes needs a boost
+
+Erlang garbage collection (GC) acts on a per-erlang-process basis, meaning that
+each process decides on its own to garbage collect itself. This is nice because
+it means stop-the-world isn't a problem, but it does have some interesting
+effects.
+
+We had a problem with our node memory graphs looking like an upwards facing
+line, instead of a nice sinusoid relative to the number of connections during
+the day. We couldn't find a memory leak *anywhere*, and so started profiling. We
+found that the memory seemed to be comprised of mostly binary data in process
+heaps. On a hunch my coworker Mike Cugini (who gets all the credit for this) ran
+the following on a node:
+
+```erlang
+lists:foreach(erlang:garbage_collect/1, erlang:processes()).
+```
+
+and saw memory drop in a huge way. We made that code run every 10 minutes or so
+and suddenly our memory problem went away.
+
+The problem is that we had a lot of processes which individually didn't have
+much heap data, but all-together were crushing the box. Each didn't think it had
+enough to garbage collect very often, so memory just kept going up. Calling the
+above forces all processes to garbage collect, and thus throw away all those
+little binary bits they were hoarding.
+
+## These aren't the solutions you are looking for
+
+The `erl` process has tons of command-line options which allow you to tweak all
+kinds of knobs. We've had tons of performance problems with our application, as
+of yet not a single one has been solved with turning one of these knobs. They've
+all been design issues or just run-of-the-mill bugs. I'm not saying the knobs
+are *never* useful, but I haven't seen it yet.
+
+## Erlang processes are great, except when they're not
+
+The erlang model of allowing processes to manage global state works really well
+in many cases. Possibly even most cases. There are, however, times when it
+becomes a performance problem. This became apparent in the project I was working
+on for Grooveshark, which was, at its heart, a pubsub server.
+
+The architecture was very simple: each channel was managed by a process, client
+connection processes subscribed to that channel and received publishes from it.
+Easy right? The problem was that extremely high volume channels were simply not
+able to keep up with the load. The channel process could do certain things very
+fast, but there were some operations which simply took time and slowed
+everything down. For example, channels could have arbitrary properties set on
+them by their owners. Retrieving an arbitrary property from a channel was a
+fairly fast operation: client `call`s the channel process, channel process
+immediately responds with the property value. No blocking involved.
+
+But as soon as there was any kind of call which required the channel process to
+talk to yet *another* process (unfortunately necessary), things got hairy. On
+high volume channels publishes/gets/set operations would get massively backed up
+in the message queue while the process was blocked on another process. We tried
+many things, but ultimately gave up on the process-per-channel approach.
+
+We instead decided on keeping *all* channel state in a transactional database.
+When client processes "called" operations on a channel, they really are just
+acting on the database data inline, no message passing involved. This means that
+read-only operations are super-fast because there is minimal blocking, and if
+some random other process is being slow it only affects the one client making
+the call which is causing it to be slow, and not holding up a whole host of
+other clients.
+
+## Mnesia might not be what you want
+
+This one is probably a bit controversial, and definitely subject to use-cases.
+Do your own testing and profiling, find out what's right for you.
+
+Mnesia is erlang's solution for global state. It's an in-memory transactional
+database which can scale to N nodes and persist to disk. It is hosted
+directly in the erlang processes memory so you interact with it in erlang
+directly in your code; no calling out to database drivers and such. Sounds great
+right?
+
+Unfortunately mnesia is not a very full-featured database. It is essentially a
+key-value store which can hold arbitrary erlang data-types, albeit in a set
+schema which you lay out for it during startup. This means that more complex
+types like sorted sets and hash maps (although this was addressed with the
+introduction of the map data-type in R17) are difficult to work with within
+mnesia. Additionally, erlang's data model of immutability, while awesome
+usually, can bite you here because it's difficult (impossible?) to pull out
+chunks of data within a record without accessing the whole record.
+
+For example, when retrieving the list of processes subscribed to a channel our
+application doesn't simply pull the full list and iterate over it. This is too
+slow, and in some cases the subscriber list was so large it wasn't actually
+feasible. The channel process wasn't cleaning up its heap fast enough, so
+multiple publishes would end up with multiple copies of the giant list in
+memory. This became a problem. Instead we chain spawned processes, each of which
+pull a set chunk of the subsciber list, and iterate over that. This is very
+difficult to implement in mnesia without pulling the full subscriber list into
+the process' memory at some point in the process.
+
+It is, however, fairly trivial to implement in redis using sorted sets. For this
+case, and many other cases after, the motto for performance improvements became
+"stick it in redis". The application is at the point where *all* state which
+isn't directly tied to a specific connection is kept in redis, encoded using
+`term_to_binary`. The performance hit of going to an outside process for data
+was actually much less than we'd originally thought, and ended up being a plus
+since we had much more freedom to do interesting hacks to speedup up our
+accesses.
diff --git a/src/_posts/2015-03-11-rabbit-hole.md b/src/_posts/2015-03-11-rabbit-hole.md
new file mode 100644
index 0000000..97c2b80
--- /dev/null
+++ b/src/_posts/2015-03-11-rabbit-hole.md
@@ -0,0 +1,165 @@
+---
+title: Rabbit Hole
+description: >-
+ Complex systems sometimes require complex debugging.
+---
+
+We've begun rolling out [SkyDNS][skydns] at my job, which has been pretty neat.
+We're basing a couple future projects around being able to use it, and it's made
+dynamic configuration and service discovery nice and easy.
+
+This post chronicles catching a bug because of our switch to SkyDNS, and how we
+discover its root cause. I like to call these kinds of bugs "rabbit holes"; they
+look shallow at first, but anytime you make a little progress forward a little
+more is always required, until you discover the ending somewhere totally
+unrelated to the start.
+
+## The Bug
+
+We are seeing *tons* of these in the SkyDNS log:
+
+```
+[skydns] Feb 20 17:21:15.168 INFO | no nameservers defined or name too short, can not forward
+```
+
+I fire up tcpdump to see if I can see anything interesting, and sure enough run
+across a bunch of these:
+
+```
+# tcpdump -vvv -s 0 -l -n port 53
+tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
+ ...
+ $fen_ip.50257 > $skydns_ip.domain: [udp sum ok] 16218+ A? unknown. (25)
+ $fen_ip.27372 > $skydns_ip.domain: [udp sum ok] 16218+ A? unknown. (25)
+ $fen_ip.35634 > $skydns_ip.domain: [udp sum ok] 59227+ A? unknown. (25)
+ $fen_ip.64363 > $skydns_ip.domain: [udp sum ok] 59227+ A? unknown. (25)
+```
+
+It appears that some of our front end nodes (FENs) are making tons of DNS
+fequests trying to find the A record of `unknown`. Something on our FENs is
+doing something insane and is breaking.
+
+## The FENs
+
+Hopping over to my favorite FEN we're able to see the packets in question
+leaving on a tcpdump as well, but that's not helpful for finding the root cause.
+We have lots of processes running on the FENs and any number of them could be
+doing something crazy.
+
+We fire up sysdig, which is similar to systemtap and strace in that it allows
+you to hook into the kernel and view various kernel activites in real time, but
+it's easier to use than both. The following command dumps all UDP packets being
+sent and what process is sending them:
+
+```
+# sysdig fd.l4proto=udp
+...
+2528950 22:17:35.260606188 0 php-fpm (21477) < connect res=0 tuple=$fen_ip:61173->$skydns_ip:53
+2528961 22:17:35.260611327 0 php-fpm (21477) > sendto fd=102(<4u>$fen_ip:61173->$skydns_ip:53) size=25 tuple=NULL
+2528991 22:17:35.260631917 0 php-fpm (21477) < sendto res=25 data=.r...........unknown.....
+2530470 22:17:35.261879032 0 php-fpm (21477) > ioctl fd=102(<4u>$fen_ip:61173->$skydns_ip:53) request=541B argument=7FFF82DC8728
+2530472 22:17:35.261880574 0 php-fpm (21477) < ioctl res=0
+2530474 22:17:35.261881226 0 php-fpm (21477) > recvfrom fd=102(<4u>$fen_ip:61173->$skydns_ip:53) size=1024
+2530476 22:17:35.261883424 0 php-fpm (21477) < recvfrom res=25 data=.r...........unknown..... tuple=$skydns_ip:53->$fen_ip:61173
+2530485 22:17:35.261888997 0 php-fpm (21477) > close fd=102(<4u>$fen_ip:61173->$skydns_ip:53)
+2530488 22:17:35.261892626 0 php-fpm (21477) < close res=0
+```
+
+Aha! We can see php-fpm is requesting something over udp with the string
+`unknown` in it. We've now narrowed down the guilty process, the rest should be
+easy right?
+
+## Which PHP?
+
+Unfortunately we're a PHP shop; knowing that php-fpm is doing something on a FEN
+narrows down the guilty codebase little. Taking the FEN out of our load-balancer
+stops the requests for `unknown`, so we *can* say that it's some user-facing
+code that is the culprit. Our setup on the FENs involves users hitting nginx
+for static content and nginx proxying PHP requests back to php-fpm. Since all
+our virtual domains are defined in nginx, we are able to do something horrible.
+
+On the particular FEN we're on we make a guess about which virtual domain the
+problem is likely coming from (our main app), and proxy all traffic from all
+other domains to a different FEN. We still see requests for `unknown` leaving
+the box, so we've narrowed the problem down a little more.
+
+## The Despair
+
+Nothing in our code is doing any direct DNS calls as far as we can find, and we
+don't see any places PHP might be doing it for us. We have lots of PHP
+extensions in place, all written in C and all black boxes; any of them could be
+the culprit. Grepping through the likely candidates' source code for the string
+`unknown` proves fruitless.
+
+We try xdebug at this point. xdebug is a profiler for php which will create
+cachegrind files for the running code. With cachegrind you can see every
+function which was ever called, how long spent within each function, a full
+call-graph, and lots more. Unfortunately xdebug outputs cachegrind files on a
+per-php-fpm-process basis, and overwrites the previous file on each new request.
+So xdebug is pretty much useless, since what is in the cachegrind file isn't
+necessarily what spawned the DNS request.
+
+## Gotcha (sorta)
+
+We turn back to the tried and true method of dumping all the traffic using
+tcpdump and perusing through that manually.
+
+What we find is that nearly everytime there is a DNS request for `unknown`, if
+we scroll up a bit there is (usually) a particular request to memcache. The
+requested key is always in the style of `function-name:someid:otherstuff`. When
+looking in the code around that function name we find this ominous looking call:
+
+```php
+$ipAddress = getIPAddress();
+$geoipInfo = getCountryInfoFromIP($ipAddress);
+```
+
+This points us in the right direction. On a hunch we add some debug
+logging to print out the `$ipAddress` variable, and sure enough it comes back as
+`unknown`. AHA!
+
+So what we surmise is happening is that for some reason our geoip extension,
+which we use to get the location data of an IP address and which
+`getCountryInfoFromIP` calls, is seeing something which is *not* an IP address
+and trying to resolve it.
+
+## Gotcha (for real)
+
+So the question becomes: why are we getting the string `unknown` as an IP
+address?
+
+Adding some debug logging around the area we find before showed that
+`$_SERVER['REMOTE_ADDR']`, which is the variable populated with the IP address
+of the client, is sometimes `unknown`. We guess that this has something to do
+with some magic we are doing on nginx's side to populate `REMOTE_ADDR` with the
+real IP address of the client in the case of them going through a proxy.
+
+Many proxies send along the header `X-Forwarded-For` to indicate the real IP of
+the client they're proxying for, otherwise the server would only see the proxy's
+IP. In our setup I decided that in those cases we should set the `REMOTE_ADDR`
+to the real client IP so our application logic doesn't even have to worry about
+it. There are a couple problems with this which render it a bad decision, one
+being that if some misbahaving proxy was to, say, start sending
+`X-Forwarded-For: unknown` then some written applications might mistake that to
+mean the client's IP is `unknown`.
+
+## The Fix
+
+The fix here was two-fold:
+
+1) We now always set `$_SERVER['REMOTE_ADDR']` to be the remote address of the
+requests, regardless of if it's a proxy, and also send the application the
+`X-Forwarded-For` header to do with as it pleases.
+
+2) Inside our app we look at all the headers sent and do some processing to
+decide what the actual client IP is. PHP can handle a lot more complex logic
+than nginx can, so we can do things like check to make sure the IP is an IP, and
+also that it's not some NAT'd internal ip, and so forth.
+
+And that's it. From some weird log messages on our DNS servers to an nginx
+mis-configuration on an almost unrelated set of servers, this is one of those
+strange bugs that never has a nice solution and goes unsolved for a long time.
+Spending the time to dive down the rabbit hole and find the answer is often
+tedious, but also often very rewarding.
+
+[skydns]: https://github.com/skynetservices/skydns
diff --git a/src/_posts/2015-07-15-go-http.md b/src/_posts/2015-07-15-go-http.md
new file mode 100644
index 0000000..7da7d6b
--- /dev/null
+++ b/src/_posts/2015-07-15-go-http.md
@@ -0,0 +1,547 @@
+---
+title: Go's http package by example
+description: >-
+ The basics of using, testing, and composing apps built using go's net/http
+ package.
+---
+
+Go's [http](http://golang.org/pkg/net/http/) package has turned into one of my
+favorite things about the Go programming language. Initially it appears to be
+somewhat complex, but in reality it can be broken down into a couple of simple
+components that are extremely flexible in how they can be used. This guide will
+cover the basic ideas behind the http package, as well as examples in using,
+testing, and composing apps built with it.
+
+This guide assumes you have some basic knowledge of what an interface in Go is,
+and some idea of how HTTP works and what it can do.
+
+## Handler
+
+The building block of the entire http package is the `http.Handler` interface,
+which is defined as follows:
+
+```go
+type Handler interface {
+ ServeHTTP(ResponseWriter, *Request)
+}
+```
+
+Once implemented the `http.Handler` can be passed to `http.ListenAndServe`,
+which will call the `ServeHTTP` method on every incoming request.
+
+`http.Request` contains all relevant information about an incoming http request
+which is being served by your `http.Handler`.
+
+The `http.ResponseWriter` is the interface through which you can respond to the
+request. It implements the `io.Writer` interface, so you can use methods like
+`fmt.Fprintf` to write a formatted string as the response body, or ones like
+`io.Copy` to write out the contents of a file (or any other `io.Reader`). The
+response code can be set before you begin writing data using the `WriteHeader`
+method.
+
+Here's an example of an extremely simple http server:
+
+```go
+package main
+
+import (
+ "fmt"
+ "log"
+ "net/http"
+)
+
+type helloHandler struct{}
+
+func (h helloHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintf(w, "hello, you've hit %s\n", r.URL.Path)
+}
+
+func main() {
+ err := http.ListenAndServe(":9999", helloHandler{})
+ log.Fatal(err)
+}
+```
+
+`http.ListenAndServe` serves requests using the handler, listening on the given
+address:port. It will block unless it encounters an error listening, in which
+case we `log.Fatal`.
+
+Here's an example of using this handler with curl:
+
+```
+ ~ $ curl localhost:9999/foo/bar
+ hello, you've hit /foo/bar
+```
+
+
+## HandlerFunc
+
+Often defining a full type to implement the `http.Handler` interface is a bit
+overkill, especially for extremely simple `ServeHTTP` functions like the one
+above. The `http` package provides a helper function, `http.HandlerFunc`, which
+wraps a function which has the signature
+`func(w http.ResponseWriter, r *http.Request)`, returning an `http.Handler`
+which will call it in all cases.
+
+The following behaves exactly like the previous example, but uses
+`http.HandlerFunc` instead of defining a new type.
+
+```go
+package main
+
+import (
+ "fmt"
+ "log"
+ "net/http"
+)
+
+func main() {
+ h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintf(w, "hello, you've hit %s\n", r.URL.Path)
+ })
+
+ err := http.ListenAndServe(":9999", h)
+ log.Fatal(err)
+}
+```
+
+## ServeMux
+
+On their own, the previous examples don't seem all that useful. If we wanted to
+have different behavior for different endpoints we would end up with having to
+parse path strings as well as numerous `if` or `switch` statements. Luckily
+we're provided with `http.ServeMux`, which does all of that for us. Here's an
+example of it being used:
+
+```go
+package main
+
+import (
+ "fmt"
+ "log"
+ "net/http"
+)
+
+func main() {
+ h := http.NewServeMux()
+
+ h.HandleFunc("/foo", func(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintln(w, "Hello, you hit foo!")
+ })
+
+ h.HandleFunc("/bar", func(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintln(w, "Hello, you hit bar!")
+ })
+
+ h.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(404)
+ fmt.Fprintln(w, "You're lost, go home")
+ })
+
+ err := http.ListenAndServe(":9999", h)
+ log.Fatal(err)
+}
+```
+
+The `http.ServeMux` is itself an `http.Handler`, so it can be passed into
+`http.ListenAndServe`. When it receives a request it will check if the request's
+path is prefixed by any of its known paths, choosing the longest prefix match it
+can find. We use the `/` endpoint as a catch-all to catch any requests to
+unknown endpoints. Here's some examples of it being used:
+
+```
+ ~ $ curl localhost:9999/foo
+Hello, you hit foo!
+
+ ~ $ curl localhost:9999/bar
+Hello, you hit bar!
+
+ ~ $ curl localhost:9999/baz
+You're lost, go home
+```
+
+`http.ServeMux` has both `Handle` and `HandleFunc` methods. These do the same
+thing, except that `Handle` takes in an `http.Handler` while `HandleFunc` merely
+takes in a function, implicitly wrapping it just as `http.HandlerFunc` does.
+
+### Other muxes
+
+There are numerous replacements for `http.ServeMux` like
+[gorilla/mux](http://www.gorillatoolkit.org/pkg/mux) which give you things like
+automatically pulling variables out of paths, easily asserting what http methods
+are allowed on an endpoint, and more. Most of these replacements will implement
+`http.Handler` like `http.ServeMux` does, and accept `http.Handler`s as
+arguments, and so are easy to use in conjunction with the rest of the things
+I'm going to talk about in this post.
+
+## Composability
+
+When I say that the `http` package is composable I mean that it is very easy to
+create re-usable pieces of code and glue them together into a new working
+application. The `http.Handler` interface is the way all pieces communicate with
+each other. Here's an example of where we use the same `http.Handler` to handle
+multiple endpoints, each slightly differently:
+
+```go
+package main
+
+import (
+ "fmt"
+ "log"
+ "net/http"
+)
+
+type numberDumper int
+
+func (n numberDumper) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintf(w, "Here's your number: %d\n", n)
+}
+
+func main() {
+ h := http.NewServeMux()
+
+ h.Handle("/one", numberDumper(1))
+ h.Handle("/two", numberDumper(2))
+ h.Handle("/three", numberDumper(3))
+ h.Handle("/four", numberDumper(4))
+ h.Handle("/five", numberDumper(5))
+
+ h.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(404)
+ fmt.Fprintln(w, "That's not a supported number!")
+ })
+
+ err := http.ListenAndServe(":9999", h)
+ log.Fatal(err)
+}
+```
+
+`numberDumper` implements `http.Handler`, and can be passed into the
+`http.ServeMux` multiple times to serve multiple endpoints. Here's it in action:
+
+```
+ ~ $ curl localhost:9999/one
+Here's your number: 1
+ ~ $ curl localhost:9999/five
+Here's your number: 5
+ ~ $ curl localhost:9999/bazillion
+That's not a supported number!
+```
+
+## Testing
+
+Testing http endpoints is extremely easy in Go, and doesn't even require you to
+actually listen on any ports! The `httptest` package provides a few handy
+utilities, including `NewRecorder` which implements `http.ResponseWriter` and
+allows you to effectively make an http request by calling `ServeHTTP` directly.
+Here's an example of a test for our previously implemented `numberDumper`,
+commented with what exactly is happening:
+
+```go
+package main
+
+import (
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ . "testing"
+)
+
+func TestNumberDumper(t *T) {
+ // We first create the http.Handler we wish to test
+ n := numberDumper(1)
+
+ // We create an http.Request object to test with. The http.Request is
+ // totally customizable in every way that a real-life http request is, so
+ // even the most intricate behavior can be tested
+ r, _ := http.NewRequest("GET", "/one", nil)
+
+ // httptest.Recorder implements the http.ResponseWriter interface, and as
+ // such can be passed into ServeHTTP to receive the response. It will act as
+ // if all data being given to it is being sent to a real client, when in
+ // reality it's being buffered for later observation
+ w := httptest.NewRecorder()
+
+ // Pass in our httptest.Recorder and http.Request to our numberDumper. At
+ // this point the numberDumper will act just as if it was responding to a
+ // real request
+ n.ServeHTTP(w, r)
+
+ // httptest.Recorder gives a number of fields and methods which can be used
+ // to observe the response made to our request. Here we check the response
+ // code
+ if w.Code != 200 {
+ t.Fatalf("wrong code returned: %d", w.Code)
+ }
+
+ // We can also get the full body out of the httptest.Recorder, and check
+ // that its contents are what we expect
+ body := w.Body.String()
+ if body != fmt.Sprintf("Here's your number: 1\n") {
+ t.Fatalf("wrong body returned: %s", body)
+ }
+
+}
+```
+
+In this way it's easy to create tests for your individual components that you
+are using to build your application, keeping the tests near to the functionality
+they're testing.
+
+Note: if you ever do need to spin up a test server in your tests, `httptest`
+also provides a way to create a server listening on a random open port for use
+in tests as well.
+
+## Middleware
+
+Serving endpoints is nice, but often there's functionality you need to run for
+*every* request before the actual endpoint's handler is run. For example, access
+logging. A middleware component is one which implements `http.Handler`, but will
+actually pass the request off to another `http.Handler` after doing some set of
+actions. The `http.ServeMux` we looked at earlier is actually an example of
+middleware, since it passes the request off to another `http.Handler` for actual
+processing. Here's an example of our previous example with some logging
+middleware:
+
+```go
+package main
+
+import (
+ "fmt"
+ "log"
+ "net/http"
+)
+
+type numberDumper int
+
+func (n numberDumper) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintf(w, "Here's your number: %d\n", n)
+}
+
+func logger(h http.Handler) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ log.Printf("%s requested %s", r.RemoteAddr, r.URL)
+ h.ServeHTTP(w, r)
+ })
+}
+
+func main() {
+ h := http.NewServeMux()
+
+ h.Handle("/one", numberDumper(1))
+ h.Handle("/two", numberDumper(2))
+ h.Handle("/three", numberDumper(3))
+ h.Handle("/four", numberDumper(4))
+ h.Handle("/five", numberDumper(5))
+
+ h.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(404)
+ fmt.Fprintln(w, "That's not a supported number!")
+ })
+
+ hl := logger(h)
+
+ err := http.ListenAndServe(":9999", hl)
+ log.Fatal(err)
+}
+```
+
+`logger` is a function which takes in an `http.Handler` called `h`, and returns
+a new `http.Handler` which, when called, will log the request it was called with
+and then pass off its arguments to `h`. To use it we pass in our
+`http.ServeMux`, so all incoming requests will first be handled by the logging
+middleware before being passed to the `http.ServeMux`.
+
+Here's an example log entry which is output when the `/five` endpoint is hit:
+
+```
+2015/06/30 20:15:41 [::1]:34688 requested /five
+```
+
+## Middleware chaining
+
+Being able to chain middleware together is an incredibly useful ability which we
+get almost for free, as long as we use the signature
+`func(http.Handler) http.Handler`. A middleware component returns the same type
+which is passed into it, so simply passing the output of one middleware
+component into the other is sufficient.
+
+However, more complex behavior with middleware can be tricky. For instance, what
+if you want a piece of middleware which takes in a parameter upon creation?
+Here's an example of just that, with a piece of middleware which will set a
+header and its value for all requests:
+
+```go
+package main
+
+import (
+ "fmt"
+ "log"
+ "net/http"
+)
+
+type numberDumper int
+
+func (n numberDumper) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintf(w, "Here's your number: %d\n", n)
+}
+
+func logger(h http.Handler) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ log.Printf("%s requested %s", r.RemoteAddr, r.URL)
+ h.ServeHTTP(w, r)
+ })
+}
+
+type headerSetter struct {
+ key, val string
+ handler http.Handler
+}
+
+func (hs headerSetter) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set(hs.key, hs.val)
+ hs.handler.ServeHTTP(w, r)
+}
+
+func newHeaderSetter(key, val string) func(http.Handler) http.Handler {
+ return func(h http.Handler) http.Handler {
+ return headerSetter{key, val, h}
+ }
+}
+
+func main() {
+ h := http.NewServeMux()
+
+ h.Handle("/one", numberDumper(1))
+ h.Handle("/two", numberDumper(2))
+ h.Handle("/three", numberDumper(3))
+ h.Handle("/four", numberDumper(4))
+ h.Handle("/five", numberDumper(5))
+
+ h.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(404)
+ fmt.Fprintln(w, "That's not a supported number!")
+ })
+
+ hl := logger(h)
+ hhs := newHeaderSetter("X-FOO", "BAR")(hl)
+
+ err := http.ListenAndServe(":9999", hhs)
+ log.Fatal(err)
+}
+```
+
+And here's the curl output:
+
+```
+ ~ $ curl -i localhost:9999/three
+ HTTP/1.1 200 OK
+ X-Foo: BAR
+ Date: Wed, 01 Jul 2015 00:39:48 GMT
+ Content-Length: 22
+ Content-Type: text/plain; charset=utf-8
+
+ Here's your number: 3
+
+```
+
+`newHeaderSetter` returns a function which accepts and returns an
+`http.Handler`. Calling that returned function with an `http.Handler` then gets
+you an `http.Handler` which will set the header given to `newHeaderSetter`
+before continuing on to the given `http.Handler`.
+
+This may seem like a strange way of organizing this; for this example the
+signature for `newHeaderSetter` could very well have looked like this:
+
+```
+func newHeaderSetter(key, val string, h http.Handler) http.Handler
+```
+
+And that implementation would have worked fine. But it would have been more
+difficult to compose going forward. In the next section I'll show what I mean.
+
+## Composing middleware with alice
+
+[Alice](https://github.com/justinas/alice) is a very simple and convenient
+helper for working with middleware using the function signature we've been using
+thusfar. Alice is used to create and use chains of middleware. Chains can even
+be appended to each other, giving even further flexibility. Here's our previous
+example with a couple more headers being set, but also using alice to manage the
+added complexity.
+
+```go
+package main
+
+import (
+ "fmt"
+ "log"
+ "net/http"
+
+ "github.com/justinas/alice"
+)
+
+type numberDumper int
+
+func (n numberDumper) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintf(w, "Here's your number: %d\n", n)
+}
+
+func logger(h http.Handler) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ log.Printf("%s requested %s", r.RemoteAddr, r.URL)
+ h.ServeHTTP(w, r)
+ })
+}
+
+type headerSetter struct {
+ key, val string
+ handler http.Handler
+}
+
+func (hs headerSetter) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set(hs.key, hs.val)
+ hs.handler.ServeHTTP(w, r)
+}
+
+func newHeaderSetter(key, val string) func(http.Handler) http.Handler {
+ return func(h http.Handler) http.Handler {
+ return headerSetter{key, val, h}
+ }
+}
+
+func main() {
+ h := http.NewServeMux()
+
+ h.Handle("/one", numberDumper(1))
+ h.Handle("/two", numberDumper(2))
+ h.Handle("/three", numberDumper(3))
+ h.Handle("/four", numberDumper(4))
+
+ fiveHS := newHeaderSetter("X-FIVE", "the best number")
+ h.Handle("/five", fiveHS(numberDumper(5)))
+
+ h.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(404)
+ fmt.Fprintln(w, "That's not a supported number!")
+ })
+
+ chain := alice.New(
+ newHeaderSetter("X-FOO", "BAR"),
+ newHeaderSetter("X-BAZ", "BUZ"),
+ logger,
+ ).Then(h)
+
+ err := http.ListenAndServe(":9999", chain)
+ log.Fatal(err)
+}
+```
+
+In this example all requests will have the headers `X-FOO` and `X-BAZ` set, but
+the `/five` endpoint will *also* have the `X-FIVE` header set.
+
+## Fin
+
+Starting with a simple idea of an interface, the `http` package allows us to
+create for ourselves an incredibly useful and flexible (yet still rather simple)
+ecosystem for building web apps with re-usable components, all without breaking
+our static checks.
diff --git a/src/_posts/2015-11-21-happy-trees.md b/src/_posts/2015-11-21-happy-trees.md
new file mode 100644
index 0000000..8d36a91
--- /dev/null
+++ b/src/_posts/2015-11-21-happy-trees.md
@@ -0,0 +1,235 @@
+---
+title: Happy Trees
+description: >-
+ Visualizing a forest of happy trees.
+---
+
+Source code related to this post is available [here](https://github.com/mediocregopher/happy-tree).
+
+This project was inspired by [this video](https://www.youtube.com/watch?v=_DpzAvb3Vk4),
+which you should watch first in order to really understand what's going on.
+
+My inspiration came from his noting that happification could be done on numbers
+in bases other than 10. I immediately thought of hexadecimal, base-16, since I'm
+a programmer and that's what I think of. I also was trying to think of how one
+would graphically represent a large happification tree, when I realized that
+hexadecimal numbers are colors, and colors graphically represent things nicely!
+
+## Colors
+
+Colors to computers are represented using 3-bytes, encompassing red, green, and
+blue. Each byte is represented by two hexadecimal digits, and they are appended
+together. For example `FF0000` represents maximum red (`FF`) added to no green
+and no blue. `FF5500` represents maximum red (`FF`), some green (`55`) and no
+blue (`00`), which when added together results in kind of an orange color.
+
+## Happifying colors
+
+In base 10, happifying a number is done by splitting its digits, squaring each
+one individually, and adding the resulting numbers. The principal works the same
+for hexadecimal numbers:
+
+```
+A4F
+A*A + 4*4 + F*F
+64 + 10 + E1
+155 // 341 in decimal
+```
+
+So if all colors are 6-digit hexadecimal numbers, they can be happified easily!
+
+```
+FF5500
+F*F + F*F + 5*5 + 5*5 + 0*0 + 0*0
+E1 + E1 + 19 + 19 + 0 + 0
+0001F4
+```
+
+So `FF5500` (an orangish color) happifies to `0001F4` (a darker blue). Since
+order of digits doesn't matter, `5F50F0` also happifies to `0001F4`. From this
+fact, we can make a tree (hence the happification tree). I can do this process
+on every color from `000000` (black) to `FFFFFF` (white), so I will!
+
+## Representing the tree
+
+So I know I can represent the tree using color, but there's more to decide on
+than that. The easy way to represent a tree would be to simply draw a literal
+tree graph, with a circle for each color and lines pointing to its parent and
+children. But this is boring, and also if I want to represent *all* colors the
+resulting image would be enormous and/or unreadable.
+
+I decided on using a hollow, multi-level pie-chart. Using the example
+of `000002`, it would look something like this:
+
+![An example of a partial multi-level pie chart](/img/happy-tree/partial.png)
+
+The inner arc represents the color `000002`. The second arc represents the 15
+different colors which happify into `000002`, each of them may also have their
+own outer arc of numbers which happify to them, and so on.
+
+This representation is nice because a) It looks cool and b) it allows the
+melancoils of the hexadecimals to be placed around the happification tree
+(numbers which happify into `000001`), which is convenient. It's also somewhat
+easier to code than a circle/branch based tree diagram.
+
+An important feature I had to implement was proportional slice sizes. If I were
+to give each child of a color an equal size on that arc's edge the image would
+simply not work. Some branches of the tree are extremely deep, while others are
+very shallow. If all were given the same space, those deep branches wouldn't
+even be representable by a single pixel's width, and would simply fail to show
+up. So I implemented proportional slice sizes, where the size of every slice is
+determined to be proportional to how many total (recursively) children it has.
+You can see this in the above example, where the second level arc is largely
+comprised of one giant slice, with many smaller slices taking up the end.
+
+## First attempt
+
+My first attempt resulted in this image (click for 5000x5000 version):
+
+[![Result of first attempt](/img/happy-tree/happy-tree-atmp1-small.png)](/img/happy-tree/happy-tree-atmp1.png)
+
+The first thing you'll notice is that it looks pretty neat.
+
+The second thing you'll notice is that there's actually only one melancoil in
+the 6-digit hexadecimal number set. The innermost black circle is `000000` which
+only happifies to itself, and nothing else will happify to it (sad `000000`).
+The second circle represents `000001`, and all of its runty children. And
+finally the melancoil, comprised of:
+
+```
+00000D -> 0000A9 -> 0000B5 -> 000092 -> 000055 -> 00003 -> ...
+```
+
+The final thing you'll notice (or maybe it was the first, since it's really
+obvious) is that it's very blue. Non-blue colors are really only represented as
+leaves on their trees and don't ever really have any children of their own, so
+the blue and black sections take up vastly more space.
+
+This makes sense. The number which should generate the largest happification
+result, `FFFFFF`, only results in `000546`, which is primarily blue. So in effect
+all colors happify to some shade of blue.
+
+This might have been it, technically this is the happification tree and the
+melancoil of 6 digit hexadecimal numbers represented as colors. But it's also
+boring, and I wanted to do better.
+
+## Second attempt
+
+The root of the problem is that the definition of "happification" I used
+resulted in not diverse enough results. I wanted something which would give me
+numbers where any of the digits could be anything. Something more random.
+
+I considered using a hash instead, like md5, but that has its own problems.
+There's no gaurantee that any number would actually reach `000001`, which isn't
+required but it's a nice feature that I wanted. It also would be unlikely that
+there would be any melancoils that weren't absolutely gigantic.
+
+I ended up redefining what it meant to happify a hexadecimal number. Instead of
+adding all the digits up, I first split up the red, green, and blue digits into
+their own numbers, happified those numbers, and finally reassembled the results
+back into a single number. For example:
+
+```
+FF5500
+FF, 55, 00
+F*F + F*F, 5*5 + 5*5, 0*0 + 0*0
+1C2, 32, 00
+C23200
+```
+
+I drop that 1 on the `1C2`, because it has no place in this system. Sorry 1.
+
+Simply replacing that function resulted in this image (click for 5000x5000) version:
+
+[![Result of second attempt](/img/happy-tree/happy-tree-atmp2-small.png)](/img/happy-tree/happy-tree-atmp2.png)
+
+The first thing you notice is that it's so colorful! So that goal was achieved.
+
+The second thing you notice is that there's *significantly* more melancoils.
+Hundreds, even. Here's a couple of the melancoils (each on its own line):
+
+```
+00000D -> 0000A9 -> 0000B5 -> 000092 -> 000055 -> 000032 -> ...
+000D0D -> 00A9A9 -> 00B5B5 -> 009292 -> 005555 -> 003232 -> ...
+0D0D0D -> A9A9A9 -> B5B5B5 -> 929292 -> 555555 -> 323232 -> ...
+0D0D32 -> A9A90D -> B5B5A9 -> 9292B5 -> 555592 -> 323255 -> ...
+...
+```
+
+And so on. You'll notice the first melancoil listed is the same as the one from
+the first attempt. You'll also notice that the same numbers from the that
+melancoil are "re-used" in the rest of them as well. The second coil listed is
+the same as the first, just with the numbers repeated in the 3rd and 4th digits.
+The third coil has those numbers repeated once more in the 1st and 2nd digits.
+The final coil is the same numbers, but with the 5th and 6th digits offset one
+place in the rotation.
+
+The rest of the melancoils in this attempt work out to just be every conceivable
+iteration of the above. This is simply a property of the algorithm chosen, and
+there's not a whole lot we can do about it.
+
+## Third attempt
+
+After talking with [Mr. Marco](/members/#marcopolo) about the previous attempts
+I got an idea that would lead me towards more attempts. The main issue I was
+having in coming up with new happification algorithms was figuring out what to
+do about getting a number greater than `FFFFFF`. Dropping the leading digits
+just seemed.... lame.
+
+One solution I came up with was to simply happify again. And again, and again.
+Until I got a number less than or equal to `FFFFFF`.
+
+With this new plan, I could increase the power by which I'm raising each
+individual digit, and drop the strategy from the second attempt of splitting the
+number into three parts. In the first attempt I was doing happification to the
+power of 2, but what if I wanted to happify to the power of 6? It would look
+something like this (starting with the number `34BEEF`):
+
+```
+34BEEF
+3^6 + 4^6 + B^6 + E^6 + E^6 + E^6 + F^6
+2D9 + 1000 + 1B0829 + 72E440 + 72E440 + ADCEA1
+1AEB223
+
+1AEB223 is greater than FFFFFF, so we happify again
+
+1^6 + A^6 + E^6 + B^6 + 2^6 + 2^6 + 3^6
+1 + F4240 + 72E440 + 1B0829 + 40 + 40 + 2D9
+9D3203
+```
+
+So `34BEEF` happifies to `9D3203`, when happifying to the power of 6.
+
+As mentioned before the first attempt in this blog was the 2nd power tree,
+here's the trees for the 3rd, 4th, 5th, and 6th powers (each image is a link to
+a larger version):
+
+3rd power:
+[![Third attempt, 3rd power](/img/happy-tree/happy-tree-atmp3-pow3-small.png)](/img/happy-tree/happy-tree-atmp3-pow3.png)
+
+4th power:
+[![Third attempt, 4th power](/img/happy-tree/happy-tree-atmp3-pow4-small.png)](/img/happy-tree/happy-tree-atmp3-pow4.png)
+
+5th power:
+[![Third attempt, 5th power](/img/happy-tree/happy-tree-atmp3-pow5-small.png)](/img/happy-tree/happy-tree-atmp3-pow5.png)
+
+6th power:
+[![Third attempt, 6th power](/img/happy-tree/happy-tree-atmp3-pow6-small.png)](/img/happy-tree/happy-tree-atmp3-pow6.png)
+
+A couple things to note:
+
+* 3-5 are still very blue. It's not till the 6th power that the distribution
+ becomes random enough to become very colorful.
+
+* Some powers have more coils than others. Power of 3 has a lot, and actually a
+ lot of them aren't coils, but single narcissistic numbers. Narcissistic
+ numbers are those which happify to themselves. `000000` and `000001` are
+ narcissistic numbers in all powers, power of 3 has quite a few more.
+
+* 4 looks super cool.
+
+Using unsigned 64-bit integers I could theoretically go up to the power of 15.
+But I hit a roadblock at power of 7, in that there's actually a melancoil which
+occurs whose members are all greater than `FFFFFF`. This means that my strategy
+of repeating happifying until I get under `FFFFFF` doesn't work for any numbers
+which lead into that coil.
diff --git a/src/_posts/2017-09-06-brian-bars.md b/src/_posts/2017-09-06-brian-bars.md
new file mode 100644
index 0000000..2c56272
--- /dev/null
+++ b/src/_posts/2017-09-06-brian-bars.md
@@ -0,0 +1,105 @@
+---
+title: Brian Bars
+description: >-
+ Cheap and easy to make, healthy, vegan, high-carb, high-protein. "The Good
+ Stuff".
+updated: 2018-01-18
+---
+
+It actually blows my mind it's been 4 years since I used this blog. It was
+previously a tech blog, but then I started putting all my tech-related posts on
+[the cryptic blog](https://cryptic.io). As of now this is a lifestyle/travel
+blog. The me of 4 years ago would be horrified.
+
+Now I just have to come up with a lifestyle and do some traveling.
+
+## Recipe
+
+This isn't a real recipe because I'm not going to preface it with my entire
+fucking life story. Let's talk about the food.
+
+Brian bars:
+
+* Are like Clif Bars, but with the simplicity of ingredients that Larabars have.
+* Are easy to make, only needing a food processor (I use a magic bullet) and a
+ stovetop oven.
+* Keep for a long time and don't really need refrigerating (but don't mind it
+ neither)
+* Are paleo, vegan, gluten-free, free-range, grass-fed, whatever...
+* Are really really filling.
+* Are named after me, deal with it.
+
+I've worked on this recipe for a bit, trying to make it workable, and will
+probably keep adjusting it (and this post) as time goes on.
+
+### Ingredients
+
+Nuts and seeds. Most of this recipe is nuts and seeds. Here's the ones I used:
+
+* 1 cup almonds
+* 1 cup peanuts
+* 1 cup walnuts
+* 1 cup coconut flakes/shavings/whatever
+* 1/2 cup flax seeds
+* 1/2 cup sesame seeds
+
+For all of those above it doesn't _really_ matter what nuts/seeds you use, it's
+all gonna get ground up anyway. So whatever's cheap works fine. Also, avoid
+salt-added ones if you can.
+
+The other ingredients are:
+
+* 1 cup raisins/currants
+* 1.5 lbs of pitted dates (no added sugar! you don't need it!)
+* 2 cups oats
+
+### Grind up the nuts
+
+Throw the nuts into the food processor and grind them into a powder. Then throw
+that powder into a bowl along with the seeds, coconuts, raisins, and oats, and
+mix em good.
+
+I don't _completely_ grind up the nuts, instead leaving some chunks in it here
+and there, but you do you.
+
+### Prepare the dates
+
+This is the harder part, and is what took me a couple tries to get right. The
+best strategy I've found is to steam the dates a bit over a stove to soften
+them. Then, about a cup at a time, you can throw them in the food processor and
+turn them into a paste. You may have to add a little water if your processor is
+having trouble.
+
+Once processed you can add the dates to the mix from before and stir it all up.
+It'll end up looking something like cookie dough. Except unlike cookie dough
+it's completely safe to eat and maybe sorta healthy.
+
+### Bake it, Finish it
+
+Put the dough stuff in a pan of some sort, flatten it out, and stick it in the
+oven at like 250 or 300 for a few hours. You're trying to cook out the water you
+added earlier when you steamed the dates, as well as whatever little moisture
+the dates had in the first place.
+
+Once thoroughly baked you can stick the pan in the fridge to cool and keep,
+and/or cut it up into individual bars. Keep in mind that the bars are super
+filling and allow for pretty small portions. Wrap em in foil or plastic wrap and
+take them to-go, or keep them around for a snack. Or both. Or whatever you want
+to do, it's your food.
+
+### Cleanup
+
+Dates are simultaneously magical and the most annoying thing to work with, so
+there's cleanup problems you may run into with them:
+
+Protip #1: When cleaning your processed date slime off of your cooking utensils
+I'd recommend just letting them soak in water for a while. Dry-ish date slime
+will stick to everything, while soaked date slime will come right off.
+
+Protip #2: Apparently if you want ants, dates are a great way to get ants. My
+apartment has never had an ant problem until 3 hours after I made a batch of
+these and didn't wipe down my counter enough. I'm still dealing with the ants.
+Apparently there's enviromentally friendly ant poisons where the ants happily
+carry the poison back into the nest and the whole nest eats it and dies. Which
+feels kinda mean in some way, but is also pretty clever and they're just ants
+anyway so fuck it.
diff --git a/src/_posts/2018-10-25-rethinking-identity.md b/src/_posts/2018-10-25-rethinking-identity.md
new file mode 100644
index 0000000..d3520d7
--- /dev/null
+++ b/src/_posts/2018-10-25-rethinking-identity.md
@@ -0,0 +1,292 @@
+---
+title: Rethinking Identity
+description: >-
+ A more useful way of thinking about identity on the internet, and using that
+ to build a service which makes our online life better.
+---
+
+In my view, the major social media platforms (Facebook, Twitter, Instagram,
+etc...) are broken. They worked well at small scales, but billions of people are
+now exposed to them, and [Murphy's Law][murphy] has come into effect. The weak
+points in the platforms have been found and exploited, to the point where
+they're barely usable for interacting with anyone you don't already know in
+person.
+
+[murphy]: https://en.wikipedia.org/wiki/Murphy%27s_law
+
+On the other hand, social media, at its core, is a powerful tool that humans
+have developed, and it's not one to be thrown away lightly (if it can be thrown
+away at all). It's worthwhile to try and fix it. So that's what this post is
+about.
+
+A lot of moaning and groaning has already been done on how social media is toxic
+for the average person. But the average person isn't doing anything more than
+receiving and reacting to their environment. If that environment is toxic, the
+person in it becomes so as well. It's certainly possible to filter the toxicity
+out, and use a platform to your own benefit, but that takes work on the user's
+part. It would be nice to think that people will do more than follow the path of
+least resistance, but at scale that's simply not how reality is, and people
+shouldn't be expected to do that work.
+
+To identify what has become toxic about the platforms, first we need to identify
+what a non-toxic platform would look like.
+
+The ideal definition for social media is to give people a place to socialize
+with friends, family, and the rest of the world. Defining "socialize" is tricky,
+and probably an exercise only a socially awkward person who doesn't do enough
+socializing would undertake. "Expressing one's feelings, knowledge, and
+experiences to other people, and receiving theirs in turn" feels like a good
+approximation. A platform where true socializing was the only activity would be
+ideal.
+
+Here are some trends on our social media which have nothing to do with
+socializing: artificially boosted follower numbers on Instagram to obtain
+product sponsors, shills in Reddit comments boosting a product or company,
+russian trolls on Twitter spreading propaganda, trolls everywhere being dicks
+and switching IPs when they get banned, and [that basketball president whose
+wife used burner Twitter accounts to trash talk players][president].
+
+[president]: https://www.nytimes.com/2018/06/07/sports/bryan-colangelo-sixers-wife.html
+
+These are all examples of how anonymity can be abused on social media. I want
+to say up front that I'm _not_ against anonymity on the internet, and that I
+think we can have our cake and eat it too. But we _should_ acknowledge the
+direct and indirect problems anonymity causes. We can't trust that anyone on
+social media is being honest about who they are and what their motivation is.
+This problem extends outside of social media too, to Amazon product reviews (and
+basically any other review system), online polls and raffles, multiplayer games,
+and surely many other other cases.
+
+## Identity
+
+To fix social media, and other large swaths of the internet, we need to rethink
+identity. This process started for me a long time ago, when I watched [this TED
+talk][identity], which discusses ways in which we misunderstand identity.
+Crucially, David Birch points out that identity is not a name, it's more
+fundamental than that.
+
+[identity]: https://www.ted.com/talks/david_birch_identity_without_a_name
+
+In the context of online platforms, where a user creates an account which
+identifies them in some way, identity breaks down into 3 distinct problems
+which are often conflated:
+
+* Authentication: Is this identity owned by this person?
+* Differentiation: Is this identity unique to this person?
+* Authorization: Is this identity allowed to do X?
+
+For internet platform developers, authentication has been given the full focus.
+Blog posts, articles, guides, and services abound which deal with properly
+hashing and checking passwords, two factor authentication, proper account
+recovery procedure, etc... While authentication is not a 100% solved problem,
+it's had the most work done on it, and the problems which this post deals with
+are not affected by it.
+
+The problem which should instead be focused on is differentiation.
+
+## Differentiation
+
+I want to make very clear, once more, that I am _not_ in favor of de-anonymizing
+the web, and doing so is not what I'm proposing.
+
+Differentiation is without a doubt the most difficult identity problem to solve.
+It's not even clear that it's solvable offline. Take this situation: you are in
+a room, and you are told that one person is going to walk in, then leave, then
+another person will do the same. These two persons may or may not be the same
+person. You're allowed to do anything you like to each person (with their
+consent) in order to determine if they are the same person or not.
+
+For the vast, vast majority of cases you can simply look with your eyeballs and
+see if they are different people. But this will not work 100% of the time.
+Identical twins are an obvious example of two persons looking like one, but a
+malicious actor with a disguise might be one person posing as two. Biometrics
+like fingerprints, iris scanning, and DNA testing fail for many reasons (the
+identical twin case being one). You could attempt to give the first a unique
+marking on their skin, but who's to say they don't have a solvent, which can
+clean that marking off, waiting right outside the door?
+
+The solutions and refutations can continue on pedantically for some time, but
+the point is that there is likely not a 100% solution, and even the 90%
+solutions require significant investment. Differentiation is a hard problem,
+which most developers don't want to solve. Most are fine with surrogates like
+checking that an email or phone number is unique to the platform, but these
+aren't enough to stop a dedicated individual or organization.
+
+### Roll Your Own Differentiation
+
+If a platform wants to roll their own solution to the differentiation problem, a
+proper solution, it might look something like this:
+
+* Submit an image of your passport, or other government issued ID. This would
+ have to be checked against the appropriate government agency to ensure the
+ ID is legitimate.
+
+* Submit an image of your face, alongside a written note containing a code given
+ by the platform. Software to detect manipulated images would need to be
+ employed, as well as reverse image searching to ensure the image isn't being
+ reused.
+
+* Once completed, all data needs to be hashed/fingerprinted and then destroyed,
+ so sensitive data isn't sitting around on servers, but can still be checked
+ against future users signing up for the platform.
+
+* A dedicated support team would be needed to handle edge-cases and mistakes.
+
+None of these is trivial, nor would I trust an up-and-coming platform which is
+being bootstrapped out of a basement to implement any of them correctly.
+Additionally, going through with this process would be a _giant_ point of
+friction for a user creating a new account; they likely would go use a different
+platform instead, which didn't have all this nonsense required.
+
+### Differentiation as a Service
+
+This is the crux of this post.
+
+Instead of each platform rolling their own differentiation, what if there was a
+service for it. Users would still have to go through the hassle described above,
+but only once forever, and on a more trustable site. Then platforms, no matter
+what stage of development they're at, could use that service to ensure that
+their community of users is free from the problems of fake accounts and trolls.
+
+This is what the service would look like:
+
+* A user would have to, at some point, have gone through the steps above to
+ create an account on the differentiation-as-a-service (DaaS) platform. This
+ account would have the normal authentication mechanisms that most platforms
+ do (password, two-factor, etc...).
+
+* When creating an account on a new platform, the user would login to their DaaS
+ account (similar to the common "login with Google/Facebook/Twitter" buttons).
+
+* The DaaS then returns an opaque token, an effectively random string which
+ uniquely identifies that user, to the platform. The platform can then check in
+ its own user database for any other users using that token, and know if the
+ user already has an account. All of this happens without any identifying
+ information being passed to the platform.
+
+Similar to how many sites outsource to Cloudflare to handle DDoS protection,
+which is better handled en masse by people familiar with the problem, the DaaS
+allows for outsourcing the problem of differentiation. Users are more likely to
+trust an established DaaS service than a random website they're signing up for.
+And signing up for a DaaS is a one-time event, so if enough platforms are using
+the DaaS it could become worthwhile for them to do so.
+
+Finally, since the DaaS also handles authentication, a platform could outsource
+that aspect of identity management to it as well. This is optional for the
+platform, but for smaller platforms which are just starting up it might be
+worthwhile to save that development time.
+
+### Traits of a Successful DaaS
+
+It's possible for me to imagine a world where use of DaaS' is common, but
+bridging the gap between that world and this one is not as obvious. Still, I
+think it's necessary if the internet is to ever evolve passed being, primarily,
+a home for trolls. There are a number of traits of an up-and-coming DaaS which
+would aid it in being accepted by the internet:
+
+* **Patience**: there is a critical mass of users and platforms using DaaS'
+ where it becomes more advantageous for platforms to use the DaaS than not.
+ Until then, the DaaS and platforms using it need to take deliberate but small
+ steps. For example: making DaaS usage optional for platform users, and giving
+ their accounts special marks to indicate they're "authentic" (like Twitter's
+ blue checkmark); giving those users' activity higher weight in algorithms;
+ allowing others to filter out activity of non-"authentic" users; etc... These
+ are all preliminary steps which can be taken which encourage but don't require
+ platform users to use a DaaS.
+
+* **User-friendly**: most likely the platforms using a DaaS are what are going
+ to be paying the bills. A successful DaaS will need to remember that, no
+ matter where the money comes from, if the users aren't happy they'll stop
+ using the DaaS, and platforms will be forced to switch to a different one or
+ stop using them altogether. User-friendliness means more than a nice
+ interface; it means actually caring for the users' interests, taking their
+ privacy and security seriously, and in all other aspects being on their side.
+ In that same vein, competition is important, and so...
+
+* **No country/government affiliation**: If the DaaS was to be run by a
+ government agency it would have no incentive to provide a good user
+ experience, since the users aren't paying the bills (they might not even be in
+ that country). A DaaS shouldn't be exclusive to any one government or country
+ anyway. Perhaps it starts out that way, to get off the ground, but ultimately
+ the internet is a global institution, and is healthiest when it's connecting
+ individuals _around the world_. A successful DaaS will reach beyond borders
+ and try to connect everyone.
+
+Obviously actually starting a DaaS would be a huge undertaking, and would
+require proper management and good developers and all that, but such things
+apply to most services.
+
+## Authorization
+
+The final aspect of identity management, which I haven't talked about yet, is
+authorization. This aspect deals with what a particular identity is allowed to
+do. For example, is an identity allowed to claim they have a particular name, or
+are from a particular place, or are of a particular age? Other things like
+administration and moderation privileges also fall under authorization, but they
+are generally defined and managed within a platform.
+
+A DaaS has the potential to help with authorization as well, though with a giant
+caveat. If a DaaS were to not fingerprint and destroy the user's data, like
+their name and birthday and whatnot, but instead store them, then the following
+use-case could also be implemented:
+
+* A platform wants to know if a user is above a certain age, let's say. It asks
+ the DaaS for that information.
+
+* The DaaS asks the user, OAuth style, whether the user is ok with giving the
+ platform that information.
+
+* If so, the platform is given that information.
+
+This is a tricky situation. It adds a lot of liablity for the user, since their
+raw data will be stored with the DaaS, ripe for hacking. It also places a lot of
+trust with the DaaS to be responsible with users' data and not go giving it out
+willy-nilly to others, and instead to only give out the bare-minimum that the
+user allows. Since the user is not the DaaS' direct customer, this might be too
+much to ask. Nevertheless, it's a use-case which is worth thinking about.
+
+## Dapps
+
+The idea of decentralized applications, or dapps, has begun to gain traction.
+While not mainstream yet, I think they have potential, and it's necessary to
+discuss how a DaaS would operate in a world where the internet is no longer
+hosted in central datacenters.
+
+Consider an Ethereum-based dapp. If a user were to register one ethereum address
+(which are really public keys) with their DaaS account, the following use-case
+could be implemented:
+
+* A charity dapp has an ethereum contract, which receives a call from an
+ ethereum address asking for money. The dapp wants to ensure every person it
+ sends money to hasn't received any that day.
+
+* The DaaS has a separate ethereum contract it manages, where it stores all
+ addresses which have been registered to a user. There is no need to keep any
+ other user information in the contract.
+
+* The charity dapp's contract calls the DaaS' contract, asking it if the address
+ is one of its addresses. If so, and if the charity contract hasn't given to
+ that address yet today, it can send money to that address.
+
+There would perhaps need to be some mechanism by which a user could change their
+address, which would be complex since that address might be in use by a dapp
+already, but it's likely a solvable problem.
+
+A charity dapp is a bit of a silly example; ideally with a charity dapp there'd
+also be some mechanism to ensure a person actually _needs_ the money. But
+there's other dapp ideas which would become feasible, due to the inability of a
+person to impersonate many people, if DaaS use becomes normal.
+
+## Why Did I Write This?
+
+Perhaps you've gotten this far and are asking: "Clearly you've thought about
+this a lot, why don't you make this yourself and make some phat stacks of cash
+with a startup?" The answer is that this project would need to be started and
+run by serious people, who can be dedicated and thorough and responsible. I'm
+not sure I'm one of those people; I get distracted easily. But I would like to
+see this idea tried, and so I've written this up thinking maybe someone else
+would take the reins.
+
+I'm not asking for equity or anything, if you want to try; it's a free idea for
+the taking. But if it turns out to be a bazillion dollar Good Idea™, I won't say
+no to a donation...
diff --git a/src/_posts/2018-11-12-viz-1.md b/src/_posts/2018-11-12-viz-1.md
new file mode 100644
index 0000000..8fd9fd9
--- /dev/null
+++ b/src/_posts/2018-11-12-viz-1.md
@@ -0,0 +1,54 @@
+---
+title: >-
+ Visualization 1
+description: >-
+ Using clojurescript and quil to generate interesting visuals
+series: viz
+git_repo: https://github.com/mediocregopher/viz.git
+git_commit: v1
+---
+
+First I want to appologize if you've seen this already, I originally had this up
+on my normal website, but I've decided to instead consolidate all my work to my
+blog.
+
+This is the first of a series of visualization posts I intend to work on, each
+building from the previous one.
+
+<script src="/assets/viz/1/goog/base.js"></script>
+<script src="/assets/viz/1/cljs_deps.js"></script>
+<script>goog.require("viz.core");</script>
+<p align="center"><canvas id="viz"></canvas></p>
+
+This visualization follows a few simple rules:
+
+* Any point can only be occupied by a single node. A point may be alive (filled)
+ or dead (empty).
+
+* On every tick each live point picks from 0 to N new points to spawn, where N is
+ the number of empty adjacent points to it. If it picks 0, it becomes dead.
+
+* Each line indicates the parent of a point. Lines have an arbitrary lifetime of
+ a few ticks, and occupy the points they connect (so new points may not spawn
+ on top of a line).
+
+* When a dead point has no lines it is cleaned up, and its point is no longer
+ occupied.
+
+The resulting behavior is somewhere between [Conway's Game of
+Life](https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life) and white noise.
+Though each point operates independently, they tend to move together in groups.
+When two groups collide head on they tend to cancel each other out, killing most
+of both. When they meet while both heading in a common direction they tend to
+peacefully merge towards that direction.
+
+Sometimes their world becomes so cluttered there's hardly room to move.
+Sometimes a major coincidence of events leads to multiple groups canceling each
+other at once, opening up the world and allowing for an explosion of new growth.
+
+Some groups spiral about a single point, sustaining themselves and defending
+from outside groups in the same movement. This doesn't last for very long.
+
+The performance of this visualization is not very optimized, and will probably
+eat up your CPU like nothing else. Most of the slowness comes from drawing the
+lines; since there's so many individual small ones it's quite cumbersome to do.
diff --git a/src/_posts/2018-11-12-viz-2.md b/src/_posts/2018-11-12-viz-2.md
new file mode 100644
index 0000000..c3e342e
--- /dev/null
+++ b/src/_posts/2018-11-12-viz-2.md
@@ -0,0 +1,49 @@
+---
+title: >-
+ Visualization 2
+description: >-
+ Now in glorious technicolor!
+series: viz
+git_repo: https://github.com/mediocregopher/viz.git
+git_commit: v2
+---
+
+
+<script src="/assets/viz/2/goog/base.js"></script>
+<script src="/assets/viz/2/cljs_deps.js"></script>
+<script>goog.require("viz.core");</script>
+<p align="center"><canvas id="viz"></canvas></p>
+
+This visualization builds on the previous. Structurally the cartesian grid has
+been turned into an isometric one, but this is more of an environmental change
+than a behavioral one.
+
+Behavioral changes which were made:
+
+* When a live point is deciding its next spawn points, it first sorts the set of
+ empty adjacent points from closest-to-the-center to farthest. It then chooses
+ a number `n` between `0` to `N` (where `N` is the sorted set's size) and
+ spawns new points from the first `n` points of the sorted set. `n` is chosen
+ based on:
+
+ * The live point's linear distance from the center.
+
+ * A random multiplier.
+
+* Each point is spawned with an attached color, where the color chosen is a
+ slightly different hue than its parent. The change is deterministic, so all
+ child points of the same generation have the same color.
+
+The second change is purely cosmetic, but does create a mesmerizing effect. The
+first change alters the behavior dramatically. Only the points which compete for
+the center are able to reproduce, but by the same token are more likely to be
+starved out by other points doing the same.
+
+In the previous visualization the points moved around in groups aimlessly. Now
+the groups are all competing for the same thing, the center. As a result they
+congregate and are able to be viewed as a larger whole.
+
+The constant churn of the whole takes many forms, from a spiral in the center,
+to waves crashing against each other, to outright chaos, to random purges of
+nearly all points. Each form lasts for only a few seconds before giving way to
+another.
diff --git a/src/_posts/2019-08-02-program-structure-and-composability.md b/src/_posts/2019-08-02-program-structure-and-composability.md
new file mode 100644
index 0000000..b44c534
--- /dev/null
+++ b/src/_posts/2019-08-02-program-structure-and-composability.md
@@ -0,0 +1,587 @@
+---
+title: >-
+ Program Structure and Composability
+description: >-
+ Discussing the nature of program structure, the problems presented by
+ complex structures, and a pattern that helps in solving those problems.
+---
+
+## Part 0: Introduction
+
+This post is focused on a concept I call “program structure,” which I will try
+to shed some light on before discussing complex program structures. I will then
+discuss why complex structures can be problematic to deal with, and will finally
+discuss a pattern for dealing with those problems.
+
+My background is as a backend engineer working on large projects that have had
+many moving parts; most had multiple programs interacting with each other, used
+many different databases in various contexts, and faced large amounts of load
+from millions of users. Most of this post will be framed from my perspective,
+and will present problems in the way I have experienced them. I believe,
+however, that the concepts and problems I discuss here are applicable to many
+other domains, and I hope those with a foot in both backend systems and a second
+domain can help to translate the ideas between the two.
+
+Also note that I will be using Go as my example language, but none of the
+concepts discussed here are specific to Go. To that end, I’ve decided to favor
+readable code over “correct” code, and so have elided things that most gophers
+hold near-and-dear, such as error checking and proper documentation, in order to
+make the code as accessible as possible to non-gophers as well. As with before,
+I trust that someone with a foot in Go and another language can help me
+translate between the two.
+
+## Part 1: Program Structure
+
+In this section I will discuss the difference between directory and program
+structure, show how global state is antithetical to compartmentalization (and
+therefore good program structure), and finally discuss a more effective way to
+think about program structure.
+
+### Directory Structure
+
+For a long time, I thought about program structure in terms of the hierarchy
+present in the filesystem. In my mind, a program’s structure looked like this:
+
+```
+// The directory structure of a project called gobdns.
+src/
+ config/
+ dns/
+ http/
+ ips/
+ persist/
+ repl/
+ snapshot/
+ main.go
+```
+
+What I grew to learn was that this conflation of “program structure” with
+“directory structure” is ultimately unhelpful. While it can’t be denied that
+every program has a directory structure (and if not, it ought to), this does not
+mean that the way the program looks in a filesystem in any way corresponds to
+how it looks in our mind’s eye.
+
+The most notable way to show this is to consider a library package. Here is the
+structure of a simple web-app which uses redis (my favorite database) as a
+backend:
+
+```
+src/
+ redis/
+ http/
+ main.go
+```
+
+If I were to ask you, based on that directory structure, what the program does
+in the most abstract terms, you might say something like: “The program
+establishes an http server that listens for requests. It also establishes a
+connection to the redis server. The program then interacts with redis in
+different ways based on the http requests that are received on the server.”
+
+And that would be a good guess. Here’s a diagram that depicts the program
+structure, wherein the root node, `main.go`, takes in requests from `http` and
+processes them using `redis`.
+
+{% include image.html
+ dir="program-structure" file="diag1.jpg" width=519
+ descr="Example 1"
+ %}
+
+This is certainly a viable guess for how a program with that directory
+structure operates, but consider another answer: “A component of the program
+called `server` establishes an http server that listens for requests. `server`
+also establishes a connection to a redis server. `server` then interacts with
+that redis connection in different ways based on the http requests that are
+received on the http server. Additionally, `server` tracks statistics about
+these interactions and makes them available to other components. The root
+component of the program establishes a connection to a second redis server, and
+stores those statistics in that redis server.” Here’s another diagram to depict
+_that_ program.
+
+{% include image.html
+ dir="program-structure" file="diag2.jpg" width=712
+ descr="Example 2"
+ %}
+
+The directory structure could apply to either description; `redis` is just a
+library which allows for interaction with a redis server, but it doesn’t
+specify _which_ or _how many_ servers. However, those are extremely important
+factors that are definitely reflected in our concept of the program’s
+structure, and not in the directory structure. **What the directory structure
+reflects are the different _kinds_ of components available to use, but it does
+not reflect how a program will use those components.**
+
+
+### Global State vs Compartmentalization
+
+The directory-centric view of structure often leads to the use of global
+singletons to manage access to external resources like RPC servers and
+databases. In examples 1 and 2 the `redis` library might contain code which
+looks something like this:
+
+```go
+// A mapping of connection names to redis connections.
+var globalConns = map[string]*RedisConn{}
+
+func Get(name string) *RedisConn {
+ if globalConns[name] == nil {
+ globalConns[name] = makeRedisConnection(name)
+ }
+ return globalConns[name]
+}
+```
+
+Even though this pattern would work, it breaks with our conception of the
+program structure in more complex cases like example 2. Rather than the `redis`
+component being owned by the `server` component, which actually uses it, it
+would be practically owned by _all_ components, since all are able to use it.
+Compartmentalization has been broken, and can only be held together through
+sheer human discipline.
+
+**This is the problem with all global state. It is shareable among all
+components of a program, and so is accountable to none of them.** One must look
+at an entire codebase to understand how a globally held component is used,
+which might not even be possible for a large codebase. Therefore, the
+maintainers of these shared components rely entirely on the discipline of their
+fellow coders when making changes, usually discovering where that discipline
+broke down once the changes have been pushed live.
+
+Global state also makes it easier for disparate programs/components to share
+datastores for completely unrelated tasks. In example 2, rather than creating a
+new redis instance for the root component’s statistics storage, the coder might
+have instead said, “well, there’s already a redis instance available, I’ll just
+use that.” And so, compartmentalization would have been broken further. Perhaps
+the two instances _could_ be coalesced into the same instance for the sake of
+resource efficiency, but that decision would be better made at runtime via the
+configuration of the program, rather than being hardcoded into the code.
+
+From the perspective of team management, global state-based patterns do nothing
+except slow teams down. The person/team responsible for maintaining the central
+library in which shared components live (`redis`, in the above examples)
+becomes the bottleneck for creating new instances for new components, which
+will further lead to re-using existing instances rather than creating new ones,
+further breaking compartmentalization. Additionally the person/team responsible
+for the central library, rather than the team using it, often finds themselves
+as the maintainers of the shared resource.
+
+### Component Structure
+
+So what does proper program structure look like? In my mind the structure of a
+program is a hierarchy of components, or, in other words, a tree. The leaf
+nodes of the tree are almost _always_ IO related components, e.g., database
+connections, RPC server frameworks or clients, message queue consumers, etc.
+The non-leaf nodes will _generally_ be components that bring together the
+functionalities of their children in some useful way, though they may also have
+some IO functionality of their own.
+
+Let's look at an even more complex structure, still only using the `redis` and
+`http` component types:
+
+{% include image.html
+ dir="program-structure" file="diag3.jpg" width=729
+ descr="Example 3"
+ %}
+
+This component structure contains the addition of the `debug` component.
+Clearly the `http` and `redis` components are reusable in different contexts,
+but for this example the `debug` endpoint is as well. It creates a separate
+http server that can be queried to perform runtime debugging of the program,
+and can be tacked onto virtually any program. The `rest-api` component is
+specific to this program and is therefore not reusable. Let’s dive into it a
+bit to see how it might be implemented:
+
+```go
+// RestAPI is very much not thread-safe, hopefully it doesn't have to handle
+// more than one request at once.
+type RestAPI struct {
+ redisConn *redis.RedisConn
+ httpSrv *http.Server
+
+ // Statistics exported for other components to see
+ RequestCount int
+ FooRequestCount int
+ BarRequestCount int
+}
+
+func NewRestAPI() *RestAPI {
+ r := new(RestAPI)
+ r.redisConn := redis.NewConn("127.0.0.1:6379")
+
+ // mux will route requests to different handlers based on their URL path.
+ mux := http.NewServeMux()
+ mux.HandleFunc("/foo", r.fooHandler)
+ mux.HandleFunc("/bar", r.barHandler)
+ r.httpSrv := http.NewServer(mux)
+
+ // Listen for requests and serve them in the background.
+ go r.httpSrv.Listen(":8000")
+
+ return r
+}
+
+func (r *RestAPI) fooHandler(rw http.ResponseWriter, r *http.Request) {
+ r.redisConn.Command("INCR", "fooKey")
+ r.RequestCount++
+ r.FooRequestCount++
+}
+
+func (r *RestAPI) barHandler(rw http.ResponseWriter, r *http.Request) {
+ r.redisConn.Command("INCR", "barKey")
+ r.RequestCount++
+ r.BarRequestCount++
+}
+```
+
+
+In that snippet `rest-api` coalesced `http` and `redis` into a simple REST-like
+api using pre-made library components. `main.go`, the root component, does much
+the same:
+
+```go
+func main() {
+ // Create debug server and start listening in the background
+ debugSrv := debug.NewServer()
+
+ // Set up the RestAPI, this will automatically start listening
+ restAPI := NewRestAPI()
+
+ // Create another redis connection and use it to store statistics
+ statsRedisConn := redis.NewConn("127.0.0.1:6380")
+ for {
+ time.Sleep(1 * time.Second)
+ statsRedisConn.Command("SET", "numReqs", restAPI.RequestCount)
+ statsRedisConn.Command("SET", "numFooReqs", restAPI.FooRequestCount)
+ statsRedisConn.Command("SET", "numBarReqs", restAPI.BarRequestCount)
+ }
+}
+```
+
+One thing that is clearly missing in this program is proper configuration,
+whether from command-line or environment variables, etc. As it stands, all
+configuration parameters, such as the redis addresses and http listen
+addresses, are hardcoded. Proper configuration actually ends up being somewhat
+difficult, as the ideal case would be for each component to set up its own
+configuration variables without its parent needing to be aware. For example,
+`redis` could set up `addr` and `pool-size` parameters. The problem is that there
+are two `redis` components in the program, and their parameters would therefore
+conflict with each other. An elegant solution to this problem is discussed in
+the next section.
+
+## Part 2: Components, Configuration, and Runtime
+
+The key to the configuration problem is to recognize that, even if there are
+two of the same component in a program, they can’t occupy the same place in the
+program’s structure. In the above example, there are two `http` components: one
+under `rest-api` and the other under `debug`. Because the structure is
+represented as a tree of components, the “path” of any node in the tree
+uniquely represents it in the structure. For example, the two `http` components
+in the previous example have these paths:
+
+```
+root -> rest-api -> http
+root -> debug -> http
+```
+
+If each component were to know its place in the component tree, then it would
+easily be able to ensure that its configuration and initialization didn’t
+conflict with other components of the same type. If the `http` component sets
+up a command-line parameter to know what address to listen on, the two `http`
+components in that program would set up:
+
+```
+--rest-api-listen-addr
+--debug-listen-addr
+```
+
+So how can we enable each component to know its path in the component structure?
+To answer this, we’ll have to take a detour through a type, called `Component`.
+
+### Component and Configuration
+
+The `Component` type is a made-up type (though you’ll be able to find an
+implementation of it at the end of this post). It has a single primary purpose,
+and that is to convey the program’s structure to new components.
+
+To see how this is done, let's look at a couple of `Component`'s methods:
+
+```go
+// Package mcmp
+
+// New returns a new Component which has no parents or children. It is therefore
+// the root component of a component hierarchy.
+func New() *Component
+
+// Child returns a new child of the called upon Component.
+func (*Component) Child(name string) *Component
+
+// Path returns the Component's path in the component hierarchy. It will return
+// an empty slice if the Component is the root component.
+func (*Component) Path() []string
+```
+
+`Child` is used to create a new `Component`, corresponding to a new child node
+in the component structure, and `Path` is used retrieve the path of any
+`Component` within that structure. For the sake of keeping the examples simple,
+let’s pretend these functions have been implemented in a package called `mcmp`.
+Here’s an example of how `Component` might be used in the `redis` component’s
+code:
+
+```go
+// Package redis
+
+func NewConn(cmp *mcmp.Component, defaultAddr string) *RedisConn {
+ cmp = cmp.Child("redis")
+ paramPrefix := strings.Join(cmp.Path(), "-")
+
+ addrParam := flag.String(paramPrefix+"-addr", defaultAddr, "Address of redis instance to connect to")
+ // finish setup
+
+ return redisConn
+}
+```
+
+In our above example, the two `redis` components' parameters would be:
+
+```
+// This first parameter is for the stats redis, whose parent is the root and
+// therefore doesn't have a prefix. Perhaps stats should be broken into its own
+// component in order to fix this.
+--redis-addr
+--rest-api-redis-addr
+```
+
+`Component` definitely makes it easier to instantiate multiple redis components
+in our program, since it allows them to know their place in the component
+structure.
+
+Having to construct the prefix for the parameters ourselves is pretty annoying,
+so let’s introduce a new package, `mcfg`, which acts like `flag` but is aware
+of `Component`. Then `redis.NewConn` is reduced to:
+
+```go
+// Package redis
+
+func NewConn(cmp *mcmp.Component, defaultAddr string) *RedisConn {
+ cmp = cmp.Child("redis")
+ addrParam := mcfg.String(cmp, "addr", defaultAddr, "Address of redis instance to connect to")
+ // finish setup
+
+ return redisConn
+}
+```
+
+Easy-peasy.
+
+#### But What About Parse?
+
+Sharp-eyed gophers will notice that there is a key piece missing: When is
+`flag.Parse`, or its `mcfg` counterpart, called? When does `addrParam` actually
+get populated? It can’t happen inside `redis.NewConn` because there might be
+other components after `redis.NewConn` that want to set up parameters. To
+illustrate the problem, let’s look at a simple program that wants to set up two
+`redis` components:
+
+```go
+func main() {
+ // Create the root Component, an empty Component.
+ cmp := mcmp.New()
+
+ // Create the Components for two sub-components, foo and bar.
+ cmpFoo := cmp.Child("foo")
+ cmpBar := cmp.Child("bar")
+
+ // Now we want to try to create a redis sub-component for each component.
+
+ // This will set up the parameter "--foo-redis-addr", but bar hasn't had a
+ // chance to set up its corresponding parameter, so the command-line can't
+ // be parsed yet.
+ fooRedis := redis.NewConn(cmpFoo, "127.0.0.1:6379")
+
+ // This will set up the parameter "--bar-redis-addr", but, as mentioned
+ // before, redis.NewConn can't parse command-line.
+ barRedis := redis.NewConn(cmpBar, "127.0.0.1:6379")
+
+ // It is only after all components have been instantiated that the
+ // command-line arguments can be parsed
+ mcfg.Parse()
+}
+```
+
+While this solves our argument parsing problem, fooRedis and barRedis are not
+usable yet because the actual connections have not been made. This is a classic
+chicken and the egg problem. The func `redis.NewConn` needs to make a connection
+which it cannot do until _after_ `mcfg.Parse` is called, but `mcfg.Parse` cannot
+be called until after `redis.NewConn` has returned. We will solve this problem
+in the next section.
+
+### Instantiation vs Initialization
+
+Let’s break down `redis.NewConn` into two phases: instantiation and
+initialization. Instantiation refers to creating the component on the component
+structure and having it declare what it needs in order to initialize (e.g.,
+configuration parameters). During instantiation, nothing external to the
+program is performed; no IO, no reading of the command-line, no logging, etc.
+All that’s happened is that the empty template of a `redis` component has been
+created.
+
+Initialization is the phase during which the template is filled in.
+Configuration parameters are read, startup actions like the creation of database
+connections are performed, and logging is output for informational and debugging
+purposes.
+
+The key to making effective use of this dichotomy is to allow _all_ components
+to instantiate themselves before they initialize themselves. By doing this we
+can ensure, for example, that all components have had the chance to declare
+their configuration parameters before configuration parsing is done.
+
+So let’s modify `redis.NewConn` so that it follows this dichotomy. It makes
+sense to leave instantiation-related code where it is, but we need a mechanism
+by which we can declare initialization code before actually calling it. For
+this, I will introduce the idea of a “hook.”
+
+#### But First: Augment Component
+
+In order to support hooks, however, `Component` will need to be augmented with
+a few new methods. Right now, it can only carry with it information about the
+component structure, but here we will add the ability to carry arbitrary
+key/value information as well:
+
+```go
+// Package mcmp
+
+// SetValue sets the given key to the given value on the Component, overwriting
+// any previous value for that key.
+func (*Component) SetValue(key, value interface{})
+
+// Value returns the value which has been set for the given key, or nil if the
+// key was never set.
+func (*Component) Value(key interface{}) interface{}
+
+// Children returns the Component's children in the order they were created.
+func (*Component) Children() []*Component
+```
+
+The final method allows us to, starting at the root `Component`, traverse the
+component structure and interact with each `Component`’s key/value store. This
+will be useful for implementing hooks.
+
+#### Hooks
+
+A hook is simply a function that will run later. We will declare a new package,
+calling it `mrun`, and say that it has two new functions:
+
+```go
+// Package mrun
+
+// InitHook registers the given hook to the given Component.
+func InitHook(cmp *mcmp.Component, hook func())
+
+// Init runs all hooks registered using InitHook. Hooks are run in the order
+// they were registered.
+func Init(cmp *mcmp.Component)
+```
+
+With these two functions, we are able to defer the initialization phase of
+startup by using the same `Components` we were passing around for the purpose
+of denoting component structure.
+
+Now, with these few extra pieces of functionality in place, let’s reconsider the
+most recent example, and make a program that creates two redis components which
+exist independently of each other:
+
+```go
+// Package redis
+
+// NOTE that NewConn has been renamed to InstConn, to reflect that the returned
+// *RedisConn is merely instantiated, not initialized.
+
+func InstConn(cmp *mcmp.Component, defaultAddr string) *RedisConn {
+ cmp = cmp.Child("redis")
+
+ // we instantiate an empty RedisConn instance and parameters for it. Neither
+ // has been initialized yet. They will remain empty until initialization has
+ // occurred.
+ redisConn := new(RedisConn)
+ addrParam := mcfg.String(cmp, "addr", defaultAddr, "Address of redis instance to connect to")
+
+ mrun.InitHook(cmp, func() {
+ // This hook will run after parameter initialization has happened, and
+ // so addrParam will be usable. Once this hook as run, redisConn will be
+ // usable as well.
+ *redisConn = makeRedisConnection(*addrParam)
+ })
+
+ // Now that cmp has had configuration parameters and intialization hooks
+ // set into it, return the empty redisConn instance back to the parent.
+ return redisConn
+}
+```
+
+```go
+// Package main
+
+func main() {
+ // Create the root Component, an empty Component.
+ cmp := mcmp.New()
+
+ // Create the Components for two sub-components, foo and bar.
+ cmpFoo := cmp.Child("foo")
+ cmpBar := cmp.Child("bar")
+
+ // Add redis components to each of the foo and bar sub-components.
+ redisFoo := redis.InstConn(cmpFoo, "127.0.0.1:6379")
+ redisBar := redis.InstConn(cmpBar, "127.0.0.1:6379")
+
+ // Parse will descend into the Component and all of its children,
+ // discovering all registered configuration parameters and filling them from
+ // the command-line.
+ mcfg.Parse(cmp)
+
+ // Now that configuration parameters have been initialized, run the Init
+ // hooks for all Components.
+ mrun.Init(cmp)
+
+ // At this point the redis components have been fully initialized and may be
+ // used. For this example we'll copy all keys from one to the other.
+ keys := redisFoo.Command("KEYS", "*")
+ for i := range keys {
+ val := redisFoo.Command("GET", keys[i])
+ redisBar.Command("SET", keys[i], val)
+ }
+}
+```
+
+## Conclusion
+
+While the examples given here are fairly simplistic, the pattern itself is quite
+powerful. Codebases naturally accumulate small, domain-specific behaviors and
+optimizations over time, especially around the IO components of the program.
+Databases are used with specific options that an organization finds useful,
+logging is performed in particular places, metrics are counted around certain
+pieces of code, etc.
+
+By programming with component structure in mind, we are able to keep these
+optimizations while also keeping the clarity and compartmentalization of the
+code intact. We can keep our code flexible and configurable, while also
+re-usable and testable. Also, the simplicity of the tools involved means they
+can be extended and retrofitted for nearly any situation or use-case.
+
+Overall, this is a powerful pattern that I’ve found myself unable to do without
+once I began using it.
+
+### Implementation
+
+As a final note, you can find an example implementation of the packages
+described in this post here:
+
+* [mcmp](https://godoc.org/github.com/mediocregopher/mediocre-go-lib/mcmp)
+* [mcfg](https://godoc.org/github.com/mediocregopher/mediocre-go-lib/mcfg)
+* [mrun](https://godoc.org/github.com/mediocregopher/mediocre-go-lib/mrun)
+
+The packages are not stable and are likely to change frequently. You’ll also
+find that they have been extended quite a bit from the simple descriptions found
+here, based on what I’ve found useful as I’ve implemented programs using
+component structures. With these two points in mind, I would encourage you to
+look and take whatever functionality you find useful for yourself, and not use
+the packages directly. The core pieces are not different from what has been
+described in this post.
diff --git a/src/_posts/2020-04-26-trading-in-the-rain.md b/src/_posts/2020-04-26-trading-in-the-rain.md
new file mode 100644
index 0000000..3a31a95
--- /dev/null
+++ b/src/_posts/2020-04-26-trading-in-the-rain.md
@@ -0,0 +1,55 @@
+---
+title: >-
+ Trading in the Rain
+description: >-
+ All those... gains... will be lost like... tears...
+---
+
+<!-- MIDI.js -->
+<!-- polyfill -->
+<script src="/assets/trading-in-the-rain/MIDI.js/inc/shim/Base64.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/MIDI.js/inc/shim/Base64binary.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/MIDI.js/inc/shim/WebAudioAPI.js" type="text/javascript"></script>
+<!-- MIDI.js package -->
+<script src="/assets/trading-in-the-rain/MIDI.js/js/midi/audioDetect.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/MIDI.js/js/midi/gm.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/MIDI.js/js/midi/loader.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/MIDI.js/js/midi/plugin.audiotag.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/MIDI.js/js/midi/plugin.webaudio.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/MIDI.js/js/midi/plugin.webmidi.js" type="text/javascript"></script>
+<!-- utils -->
+<script src="/assets/trading-in-the-rain/MIDI.js/js/util/dom_request_xhr.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/MIDI.js/js/util/dom_request_script.js" type="text/javascript"></script>
+<!-- / MIDI.js -->
+
+<script src="/assets/trading-in-the-rain/Distributor.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/MusicBox.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/RainCanvas.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/CW.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/SeriesComposer.js" type="text/javascript"></script>
+<script src="/assets/trading-in-the-rain/main.js" type="text/javascript"></script>
+
+
+<div id="tradingInRainModal">
+For each pair listed below, live trade data will be pulled down from the
+<a href="https://docs.cryptowat.ch/websocket-api/">Cryptowat.ch Websocket
+API</a> and used to generate musical rain drops. The price of each trade
+determines both the musical note and position of the rain drop on the screen,
+while the volume of each trade determines how long the note is held and how big
+the rain drop is.
+
+<p id="markets">Pairs to be generated, by color:<br/><br/></p>
+
+<button id="button" onclick="run()">Click Here to Begin</button>
+<p id="progress"></p>
+
+<script type="text/javascript">
+ fillMarketP();
+ if (window.addEventListener) window.addEventListener("load", autorun, false);
+ else if (window.attachEvent) window.attachEvent("onload", autorun);
+ else window.onload = autorun;
+</script>
+</div>
+
+
+<canvas id="rainCanvas" style=""></canvas>
diff --git a/src/_posts/2020-05-30-denver-protests.md b/src/_posts/2020-05-30-denver-protests.md
new file mode 100644
index 0000000..710987f
--- /dev/null
+++ b/src/_posts/2020-05-30-denver-protests.md
@@ -0,0 +1,161 @@
+---
+title: >-
+ Denver Protests
+description: >-
+ Craziness
+---
+
+# Saturday, May 30th
+
+We went to the May 30th protest at Civic Center Park. We were there for a few
+hours during the day, leaving around 4pm. I would describe the character of the
+protest as being energetic, angry, but contained. A huge crowd moved in and
+around civic center, chanting and being rowdy, but clearly was being led.
+
+After a last hurrah at the pavilion it seemed that the organized event was
+"over". We stayed a while longer, and eventually headed back home. I don't feel
+that people really left the park at the same time we did; mostly everyone just
+dispersed around the park and found somewhere to keep hanging out.
+
+Tonight there has been an 8pm curfew. The police lined up on the north side of
+the park, armored and clearly ready for action. We watched all of this on the
+live news stations, gritting our teeth through the comentary of their reporters.
+As the police stood there, the clock counting down to 8, the protesters grew
+more and more irritated. They taunted the police, and formed a line of their
+own. The braver (or more dramatic) protesters walked around in the no-man's land
+between them, occasionally earning themselves some teargas.
+
+The police began pushing forward just before 8 a little, but began pushing in
+earnest just after 8, after the howling. They would advance, wait, advance, wait
+again. An armada of police cars, ambulance, and fire trucks followed the line as
+it advanced.
+
+The police did not give the protesters anywhere to go except into Capital Hill,
+southeast of Civic Center Park. We watched as a huge crowd marched past the
+front of our house, chanting their call and response: "What's his name?" "GEORGE
+FLOYD". The feeling wasn't of violence still, just anger. Indignant at a curfew
+aimed at quelling a movement, the protesters simply kept moving. The police were
+never far behind.
+
+We sat on our front stoop with our neighbors and watched the night unfold. I
+don't think a single person in our building or the buildings to the left and
+right of us hadn't gone to protest today in some capacity. We came back from our
+various outings and sat out front, watching the crowds and patrolling up and
+down the street to keep tabs on things.
+
+Around 9pm the fires started. We saw them on the news, and in person. They were
+generally dumpster fires, generally placed such that they were away from
+buildings, clearly being done more to be annoying than to accomplish anything
+specific. A very large set of fires was started a block south of us, in the
+middle of the street. The fire department was there within a few minutes to put
+those out, before moving on.
+
+From the corner of my eye, sitting back on the stoop, I noticed our neighbors
+running into their backyard. We ran after them, and they told us there was a
+dumpster fire in our alley. They were running with fire extinguishers, and we
+ran inside to grab some of our own. By the time we got to the backyard the fire
+was only smouldering, and the fire department was coming down the alley. We
+scurried back into the backyard. A few minutes later I peeked my head around the
+corner, into the alley, to see what happening. I was greeted by at least two
+police in riot gear, guarding the dumpster as the fire department worked. They
+saw me but didn't move, and I quickly retreated back to the yard.
+
+Talking to our neighbor later we found out she had seen a group of about 10
+people back there, and watched them jump the fence into another backyard in
+order to escape the alley. She thinks they, or some subset of them, started the
+fire. She looked one in the eye, she says, and didn't get the impression they
+were trying to cause damage, just to make a statement.
+
+The fires stopped not long after that, it seems. We're pretty sure the fire
+trucks were just driving up and down the main roads, looking into alleys and
+stopping all fires they could find. In all this time the police didn't do much.
+They would hold a line, but never chase anyone. Even now, as I write this around
+midnight, people are still out, meandering around in small groups, and police
+are present but not really doing anything.
+
+It's hard to get a good view of everything though. All we have is livestreams on
+youtube to go on at this point. There's a couple intrepid amateur reporters out
+there, getting into the crowds and streaming events as they happen. Right now
+we're watching people moving down Lincoln towards Civic Center Park, some of
+them trying to smash windows of buildings as they go.
+
+The violence of these protests is going to be the major story of tonight, I know
+that already. That I know of there's been 3 police injured, some broken
+windows, and quite a bit of graffiti. I do believe the the tactic of pushing
+everyone into Cap Hill had the desired effect of reducing looting (again, as far
+as I can tell so far), but at that expense of those who live here who have to
+endure latent tear gas, dumpster fires, and sirens all through the night.
+
+Even now, at midnight, from what I can see from my porch and from these live
+streams, the protesters are not violent. At worst they are guilty of a lot of
+loitering. The graffiti, the smashed windows, the injured officers, all of these
+things will be held up as examples of the anarchy and violence inherent to the
+protesters. But I don't think that's an honest picture. The vast, vast majority
+of those out right now are civily disobeying an unjust curfew, trying to keep
+the energy of the movement alive.
+
+My thoughts about these things are complicated. When turning a corner on the
+street I'm far more afraid to see the police than to see other protesters. The
+fires have been annoying, and stupid, and unhelpful, but were never threatening.
+The violence is stupid, though I don't shed many tears for a looted Chili's or
+Papa Johns. The police have actually shown more restraint than I expected in all
+of this, though funneling the protest into a residential neighborhood was an
+incredibly stupid move. Could the protesters not have just stayed in the park?
+Yes, the park would likely have been turned into an encampment, but it was
+already heading into that direction due to Covid-19. Overall, this night didn't
+need to be so hard, but Denver handled this well.
+
+But, it's only 1am, and the night has a long way to go. Things could still get
+worse. Even now I'm watching people trying to break into the supreme court
+building. Civic Center Park appears to be very populated again, and the police
+are very present there again. It's possible I may eat my words.
+
+# Monday, June 1st
+
+Yesterday was quite a bit more tame than the craziness Saturday. I woke up
+Sunday morning feeling antsy, and rode my bike around to see the damage. I had a
+long conversation with a homeless man named Gary in Civic Center Park. He was
+pissed, and had a lot to say about the "suburban kids" destroying the park he
+and many others live in, causing it to be shut down and tear gassed. The
+protesters saw it as a game, according to him, but it was life and death for the
+homeless; three of his guys got beat up in the street, and neither police nor
+protesters stopped it.
+
+Many people had shown up to the park early to help clean it up. Apart from the
+graffiti, which was also in the process of being cleaned, it was hard to tell
+anything had actually happened. Gary had some words about them as well, that
+they were only there for the gram and some pats on the back, but once they left
+his life would be back as it was. I could feel that, but I also appreciated that
+people were cognizant that damage was being done and were willing to do
+something about it.
+
+I rode around 16th street mall, down colfax, and back up 13th, looking to see if
+anything had happened. For the most part there was no damage, save the graffiti.
+A mediterranean restaurant got its windows smashed, as well as the Office Depot.
+The restaurant was unfortunate, Office Depot will be ok.
+
+The protest yesterday was much more peaceful. The cops were nowhere to be found
+when curfew hit, but did eventually show up when the protest moved down Colfax.
+They had lined the streets around their precinct building there, but for the
+most part the protesters just kept walking. This is when the "violence" started.
+The cops moved into the street, forming a line across Colfax behind the
+protesters. Police cars and vans started moving. As the protest turned back,
+presumably to head back to the capitol lawn, it ran into the riot line.
+
+Predictably, everyone scattered. The cat-and-mouse game had begun, which meant
+dumpster fires, broken windows, tear gas, and all the rest. Watching the whole
+thing it was extremely clear to us, though not the news casters, unfortunately,
+that if the police hadn't moved out into Colfax nothing would have ever
+happened. Instead, the news casters lamented that people were bringing things
+like helmets, gas masks, traffic cones, shields, etc... and so were clearly not there
+"for the right reasons".
+
+The thing that the news casters couldn't seem to grasp was that the police
+attempting to control these situations are what are catalyzing them in the first
+place. These are protests _against_ the police, they cannot take place under the
+terms the police choose. If the police were not here setting terms, but instead
+working with the peaceful protesters (the vast, vast majority) to quell the
+violence, no one would be here with helmets, gas masks, traffic cones,
+shields... But instead the protesters feel they need to protect themselves in
+order to be heard, and the police feel they have to exercise their power to
+maintain control, and so the situation degrades.
diff --git a/src/_posts/2020-07-07-viz-3.md b/src/_posts/2020-07-07-viz-3.md
new file mode 100644
index 0000000..f56dbb6
--- /dev/null
+++ b/src/_posts/2020-07-07-viz-3.md
@@ -0,0 +1,154 @@
+---
+title: >-
+ Visualization 3
+description: >-
+ All the pixels.
+series: viz
+---
+
+<canvas id="canvas" style="padding-bottom: 2rem;"></canvas>
+
+This visualization is built from the ground up. On every frame a random set of
+pixels is chosen. Each chosen pixel calculates the average of its color and the
+color of a random neighbor. Some random color drift is added in as well. It
+replaces its own color with that calculated color.
+
+Choosing a neighbor is done using the "asteroid rule", ie a pixel at the very
+top row is considered to be the neighbor of the pixel on the bottom row of the
+same column.
+
+Without the asteroid rule the pixels would all eventually converge into a single
+uniform color, generally a light blue, due to the colors at the edge, the reds,
+being quickly averaged away. With the asteroid rule in place the canvas has no
+edges, thus no position on the canvas is favored and balance can be maintained.
+
+<script type="text/javascript">
+let rectSize = 12;
+
+function randn(n) {
+ return Math.floor(Math.random() * n);
+}
+
+let canvas = document.getElementById("canvas");
+canvas.width = window.innerWidth - (window.innerWidth % rectSize);
+canvas.height = window.innerHeight- (window.innerHeight % rectSize);
+let ctx = canvas.getContext("2d");
+
+let w = canvas.width / rectSize;
+let h = canvas.height / rectSize;
+
+let matrices = new Array(2);
+matrices[0] = new Array(w);
+matrices[1] = new Array(w);
+for (let x = 0; x < w; x++) {
+ matrices[0][x] = new Array(h);
+ matrices[1][x] = new Array(h);
+ for (let y = 0; y < h; y++) {
+ let el = {
+ h: 360 * (x / w),
+ s: "100%",
+ l: "50%",
+ };
+ matrices[0][x][y] = el;
+ matrices[1][x][y] = el;
+ }
+}
+
+// draw initial canvas, from here on out only individual rectangles will be
+// filled as they get updated.
+for (let x = 0; x < w; x++) {
+ for (let y = 0; y < h; y++) {
+ let el = matrices[0][x][y];
+ ctx.fillStyle = `hsl(${el.h}, ${el.s}, ${el.l})`;
+ ctx.fillRect(x * rectSize, y * rectSize, rectSize, rectSize);
+ }
+}
+
+
+let requestAnimationFrame =
+ window.requestAnimationFrame ||
+ window.mozRequestAnimationFrame ||
+ window.webkitRequestAnimationFrame ||
+ window.msRequestAnimationFrame;
+
+let neighbors = [
+ [-1, -1], [0, -1], [1, -1],
+ [-1, 0], [1, 0],
+ [-1, 1], [0, 1], [1, 1],
+];
+
+function randNeighborAsteroid(matrix, x, y) {
+ let neighborCoord = neighbors[randn(neighbors.length)];
+ let neighborX = x+neighborCoord[0];
+ let neighborY = y+neighborCoord[1];
+ neighborX = (neighborX + w) % w;
+ neighborY = (neighborY + h) % h;
+ return matrix[neighborX][neighborY];
+}
+
+function randNeighbor(matrix, x, y) {
+ while (true) {
+ let neighborCoord = neighbors[randn(neighbors.length)];
+ let neighborX = x+neighborCoord[0];
+ let neighborY = y+neighborCoord[1];
+ if (neighborX < 0 || neighborX >= w || neighborY < 0 || neighborY >= h) {
+ continue;
+ }
+ return matrix[neighborX][neighborY];
+ }
+}
+
+let drift = 10;
+function genChildH(elA, elB) {
+ // set the two h values, h1 <= h2
+ let h1 = elA.h;
+ let h2 = elB.h;
+ if (h1 > h2) {
+ h1 = elB.h;
+ h2 = elA.h;
+ }
+
+ // diff must be between 0 (inclusive) and 360 (exclusive). If it's greater
+ // than 180 then it's not the shortest path around, that must be the other
+ // way around the circle.
+ let hChild;
+ let diff = h2 - h1;
+ if (diff > 180) {
+ diff = 360 - diff;
+ hChild = h2 + (diff / 2);
+ } else {
+ hChild = h1 + (diff / 2);
+ }
+
+ hChild += (Math.random() * drift * 2) - drift;
+ hChild = (hChild + 360) % 360;
+ return hChild;
+}
+
+let tick = 0;
+function doTick() {
+ tick++;
+ let currI = tick % 2;
+ let curr = matrices[currI];
+ let lastI = (tick - 1) % 2;
+ let last = matrices[lastI];
+
+ for (let i = 0; i < (w * h / 2); i++) {
+ let x = randn(w);
+ let y = randn(h);
+ if (curr[x][y].lastTick == tick) continue;
+
+ let neighbor = randNeighborAsteroid(last, x, y);
+ curr[x][y].h = genChildH(curr[x][y], neighbor);
+ curr[x][y].lastTick = tick;
+ ctx.fillStyle = `hsl(${curr[x][y].h}, ${curr[x][y].s}, ${curr[x][y].l})`;
+ ctx.fillRect(x * rectSize, y * rectSize, rectSize, rectSize);
+ }
+
+ matrices[currI] = curr;
+ requestAnimationFrame(doTick);
+}
+
+requestAnimationFrame(doTick);
+
+</script>
diff --git a/src/_posts/2020-11-16-component-oriented-programming.md b/src/_posts/2020-11-16-component-oriented-programming.md
new file mode 100644
index 0000000..3400090
--- /dev/null
+++ b/src/_posts/2020-11-16-component-oriented-programming.md
@@ -0,0 +1,352 @@
+---
+title: >-
+ Component-Oriented Programming
+description: >-
+ A concise description of.
+---
+
+[A previous post in this
+blog](/2019/08/02/program-structure-and-composability.html) focused on a
+framework developed to make designing component-based programs easier. In
+retrospect, the proposed pattern/framework was over-engineered. This post
+attempts to present the same ideas in a more distilled form, as a simple
+programming pattern and without the unnecessary framework.
+
+## Components
+
+Many languages, libraries, and patterns make use of a concept called a
+"component," but in each case the meaning of "component" might be slightly
+different. Therefore, to begin talking about components, it is necessary to first
+describe what is meant by "component" in this post.
+
+For the purposes of this post, the properties of components include the
+following.
+
+&nbsp;1... **Abstract**: A component is an interface consisting of one or more
+methods.
+
+&nbsp;&nbsp;&nbsp;1a... A function might be considered a single-method component
+_if_ the language supports first-class functions.
+
+&nbsp;&nbsp;&nbsp;1b... A component, being an interface, may have one or more
+implementations. Generally, there will be a primary implementation, which is
+used during a program's runtime, and secondary "mock" implementations, which are
+only used when testing other components.
+
+&nbsp;2... **Instantiatable**: An instance of a component, given some set of
+parameters, can be instantiated as a standalone entity. More than one of the
+same component can be instantiated, as needed.
+
+&nbsp;3... **Composable**: A component may be used as a parameter of another
+component's instantiation. This would make it a child component of the one being
+instantiated (the parent).
+
+&nbsp;4... **Pure**: A component may not use mutable global variables (i.e.,
+singletons) or impure global functions (e.g., system calls). It may only use
+constants and variables/components given to it during instantiation.
+
+&nbsp;5... **Ephemeral**: A component may have a specific method used to clean
+up all resources that it's holding (e.g., network connections, file handles,
+language-specific lightweight threads, etc.).
+
+&nbsp;&nbsp;&nbsp;5a... This cleanup method should _not_ clean up any child
+components given as instantiation parameters.
+
+&nbsp;&nbsp;&nbsp;5b... This cleanup method should not return until the
+component's cleanup is complete.
+
+&nbsp;&nbsp;&nbsp;5c... A component should not be cleaned up until all its
+parent components are cleaned up.
+
+Components are composed together to create component-oriented programs. This is
+done by passing components as parameters to other components during
+instantiation. The `main` procedure of the program is responsible for
+instantiating and composing the components of the program.
+
+## Example
+
+It's easier to show than to tell. This section posits a simple program and then
+describes how it would be implemented in a component-oriented way. The program
+chooses a random number and exposes an HTTP interface that allows users to try
+and guess that number. The following are requirements of the program:
+
+* A guess consists of a name that identifies the user performing the guess and
+ the number that is being guessed;
+
+* A score is kept for each user who has performed a guess;
+
+* Upon an incorrect guess, the user should be informed of whether they guessed
+ too high or too low, and 1 point should be deducted from their score;
+
+* Upon a correct guess, the program should pick a new random number against
+ which to check subsequent guesses, and 1000 points should be added to the
+ user's score;
+
+* The HTTP interface should have two endpoints: one for users to submit guesses,
+ and another that lists out user scores from highest to lowest;
+
+* Scores should be saved to disk so they survive program restarts.
+
+It seems clear that there will be two major areas of functionality for our
+program: score-keeping and user interaction via HTTP. Each of these can be
+encapsulated into components called `scoreboard` and `httpHandlers`,
+respectively.
+
+`scoreboard` will need to interact with a filesystem component to save/restore
+scores (because it can't use system calls directly; see property 4). It would be
+wasteful for `scoreboard` to save the scores to disk on every score update, so
+instead it will do so every 5 seconds. A time component will be required to
+support this.
+
+`httpHandlers` will be choosing the random number which is being guessed, and
+will therefore need a component that produces random numbers. `httpHandlers`
+will also be recording score changes to `scoreboard`, so it will need access to
+`scoreboard`.
+
+The example implementation will be written in go, which makes differentiating
+HTTP handler functionality from the actual HTTP server quite easy; thus, there
+will be an `httpServer` component that uses `httpHandlers`.
+
+Finally, a `logger` component will be used in various places to log useful
+information during runtime.
+
+[The example implementation can be found
+here.](/assets/component-oriented-design/v1/main.html) While most of it can be
+skimmed, it is recommended to at least read through the `main` function to see
+how components are composed together. Note that `main` is where all components
+are instantiated, and that all components' take in their child components as
+part of their instantiation.
+
+## DAG
+
+One way to look at a component-oriented program is as a directed acyclic graph
+(DAG), where each node in the graph represents a component, and each edge
+indicates that one component depends upon another component for instantiation.
+For the previous program, it's quite easy to construct such a DAG just by
+looking at `main`, as in the following:
+
+```
+net.Listener rand.Rand os.File
+ ^ ^ ^
+ | | |
+ httpServer --> httpHandlers --> scoreboard --> time.Ticker
+ | | |
+ +---------------+---------------+--> log.Logger
+```
+
+Note that all the leaves of the DAG (i.e., nodes with no children) describe the
+points where the program meets the operating system via system calls. The leaves
+are, in essence, the program's interface with the outside world.
+
+While it's not necessary to actually draw out the DAG for every program one
+writes, it can be helpful to at least think about the program's structure in
+these terms.
+
+## Benefits
+
+Looking at the previous example implementation, one would be forgiven for having
+the immediate reaction of "This seems like a lot of extra work for little gain.
+Why can't I just make the system calls where I need to, and not bother with
+wrapping them in interfaces and all these other rules?"
+
+The following sections will answer that concern by showing the benefits gained
+by following a component-oriented pattern.
+
+### Testing
+
+Testing is important, that much is being assumed.
+
+A distinction to be made with testing is between unit and non-unit tests. Unit
+tests are those for which there are no requirements for the environment outside
+the test, such as the existence of global variables, running databases,
+filesystems, or network services. Unit tests do not interact with the world
+outside the testing procedure, but instead use mocks in place of the
+functionality that would be expected by that world.
+
+Unit tests are important because they are faster to run and more consistent than
+non-unit tests. Unit tests also force the programmer to consider different
+possible states of a component's dependencies during the mocking process.
+
+Unit tests are often not employed by programmers, because they are difficult to
+implement for code that does not expose any way to swap out dependencies for
+mocks of those dependencies. The primary culprit of this difficulty is the
+direct usage of singletons and impure global functions. For component-oriented
+programs, all components inherently allow for the swapping out of any
+dependencies via their instantiation parameters, so there's no extra effort
+needed to support unit tests.
+
+[Tests for the example implementation can be found
+here.](/assets/component-oriented-design/v1/main_test.html) Note that all
+dependencies of each component being tested are mocked/stubbed next to them.
+
+### Configuration
+
+Practically all programs require some level of runtime configuration. This may
+take the form of command-line arguments, environment variables, configuration
+files, etc.
+
+For a component-oriented program, all components are instantiated in the same
+place, `main`, so it's very easy to expose any arbitrary parameter to the user
+via configuration. For any component that is affected by a configurable
+parameter, that component merely needs to take an instantiation parameter for
+that configurable parameter; `main` can connect the two together. This accounts
+for the unit testing of a component with different configurations, while still
+allowing for the configuration of any arbitrary internal functionality.
+
+For more complex configuration systems, it is also possible to implement a
+`configuration` component that wraps whatever configuration-related
+functionality is needed, which other components use as a sub-component. The
+effect is the same.
+
+To demonstrate how configuration works in a component-oriented program, the
+example program's requirements will be augmented to include the following:
+
+* The point change values for both correct and incorrect guesses (currently
+ hardcoded at 1000 and 1, respectively) should be configurable on the
+ command-line;
+
+* The save file's path, HTTP listen address, and save interval should all be
+ configurable on the command-line.
+
+[The new implementation, with newly configurable parameters, can be found
+here.](/assets/component-oriented-design/v2/main.html) Most of the program has
+remained the same, and all unit tests from before remain valid. The primary
+difference is that `scoreboard` takes in two new parameters for the point change
+values, and configuration is set up inside `main` using the `flags` package.
+
+### Setup/Runtime/Cleanup
+
+A program can be split into three stages: setup, runtime, and cleanup. Setup is
+the stage during which the internal state is assembled to make runtime possible.
+Runtime is the stage during which a program's actual function is being
+performed. Cleanup is the stage during which the runtime stops and internal
+state is disassembled.
+
+A graceful (i.e., reliably correct) setup is quite natural to accomplish for
+most. On the other hand, a graceful cleanup is, unfortunately, not a programmer's
+first concern (if it is a concern at all).
+
+When building reliable and correct programs, a graceful cleanup is as important
+as a graceful setup and runtime. A program is still running while it is being
+cleaned up, and it's possibly still acting on the outside world. Shouldn't
+it behave correctly during that time?
+
+Achieving a graceful setup and cleanup with components is quite simple.
+
+During setup, a single-threaded procedure (`main`) first constructs the leaf
+components, then the components that take those leaves as parameters, then the
+components that take _those_ as parameters, and so on, until the component DAG
+is fully constructed.
+
+At this point, the program's runtime has begun.
+
+Once the runtime is over, signified by a process signal or some other mechanism,
+it's only necessary to call each component's cleanup method (if any; see
+property 5) in the reverse of the order in which the components were
+instantiated. This order is inherently deterministic, as the components were
+instantiated by a single-threaded procedure.
+
+Inherent to this pattern is the fact that each component will certainly be
+cleaned up before any of its child components, as its child components must have
+been instantiated first, and a component will not clean up child components
+given as parameters (properties 5a and 5c). Therefore, the pattern avoids
+use-after-cleanup situations.
+
+To demonstrate a graceful cleanup in a component-oriented program, the example
+program's requirements will be augmented to include the following:
+
+* The program will terminate itself upon an interrupt signal;
+
+* During termination (cleanup), the program will save the latest set of scores
+ to disk one final time.
+
+[The new implementation that accounts for these new requirements can be found
+here.](/assets/component-oriented-design/v3/main.html) For this example, go's
+`defer` feature could have been used instead, which would have been even
+cleaner, but was omitted for the sake of those using other languages.
+
+
+## Conclusion
+
+The component pattern helps make programs more reliable with only a small amount
+of extra effort incurred. In fact, most of the pattern has to do with
+establishing sensible abstractions around global functionality and remembering
+certain idioms for how those abstractions should be composed together, something
+most of us already do to some extent anyway.
+
+While beneficial in many ways, component-oriented programming is merely a tool
+that can be applied in many cases. It is certain that there are cases where it
+is not the right tool for the job, so apply it deliberately and intelligently.
+
+## Criticisms/Questions
+
+In lieu of a FAQ, I will attempt to premeditate questions and criticisms of the
+component-oriented programming pattern laid out in this post.
+
+**This seems like a lot of extra work.**
+
+Building reliable programs is a lot of work, just as building a
+reliable _anything_ is a lot of work. Many of us work in an industry that likes
+to balance reliability (sometimes referred to by the more specious "quality")
+with malleability and deliverability, which naturally leads to skepticism of any
+suggestions requiring more time spent on reliability. This is not necessarily a
+bad thing, it's just how the industry functions.
+
+All that said, a pattern need not be followed perfectly to be worthwhile, and
+the amount of extra work incurred by it can be decided based on practical
+considerations. I merely maintain that code which is (mostly) component-oriented
+is easier to maintain in the long run, even if it might be harder to get off the
+ground initially.
+
+**My language makes this difficult.**
+
+I don't know of any language which makes this pattern particularly easier than
+others, so, unfortunately, we're all in the same boat to some extent (though I
+recognize that some languages, or their ecosystems, make it more difficult than
+others). It seems to me that this pattern shouldn't be unbearably difficult for
+anyone to implement in any language either, however, as the only language
+feature required is abstract typing.
+
+It would be nice to one day see a language that explicitly supports this
+pattern by baking the component properties in as compiler-checked rules.
+
+**My `main` is too big**
+
+There's no law saying all component construction needs to happen in `main`,
+that's just the most sensible place for it. If there are large sections of your
+program that are independent of each other, then they could each have their own
+construction functions that `main` then calls.
+
+Other questions that are worth asking include: Can my program be split up
+into multiple programs? Can the responsibilities of any of my components be
+refactored to reduce the overall complexity of the component DAG? Can the
+instantiation of any components be moved within their parent's
+instantiation function?
+
+(This last suggestion may seem to be disallowed, but is fine as long as the
+parent's instantiation function remains pure.)
+
+**Won't this will result in over-abstraction?**
+
+Abstraction is a necessary tool in a programmer's toolkit, there is simply no
+way around it. The only questions are "how much?" and "where?"
+
+The use of this pattern does not affect how those questions are answered, in my
+opinion, but instead aims to more clearly delineate the relationships and
+interactions between the different abstracted types once they've been
+established using other methods. Over-abstraction is possible and avoidable
+regardless of which language, pattern, or framework is being used.
+
+**Does CoP conflict with object-oriented or functional programming?**
+
+I don't think so. OoP languages will have abstract types as part of their core
+feature-set; most difficulties are going to be with deliberately _not_ using
+other features of an OoP language, and with imported libraries in the language
+perhaps making life inconvenient by not following CoP (specifically regarding
+cleanup and the use of singletons).
+
+For functional programming, it may well be that, depending on the language, CoP
+is technically being used, as functional languages are already generally
+antagonistic toward globals and impure functions, which is most of the battle.
+If anything, the transition from functional to component-oriented programming
+will generally be an organizational task.
diff --git a/src/_posts/2021-01-01-new-year-new-resolution.md b/src/_posts/2021-01-01-new-year-new-resolution.md
new file mode 100644
index 0000000..8e9edc7
--- /dev/null
+++ b/src/_posts/2021-01-01-new-year-new-resolution.md
@@ -0,0 +1,50 @@
+---
+title: >-
+ New Year, New Resolution
+description: >-
+ This blog is about to get some action.
+---
+
+At this point I'm fairly well known amongst friends and family for my new year's
+resolutions, to the point that earlier this month a friend of mine asked me
+"What's it going to be this year?". In the past I've done things like no
+chocoloate, no fast food, no added sugar (see a theme?), and no social media.
+They've all been of the "I won't do this" sort, because it's a lot easier to
+stop doing something than to start doing something new. Doing something new
+inherently means _also_ not doing something else; there's only so many hours in
+the day, afterall.
+
+## This Year
+
+This year I'm going to shake things up, I'm going to do something new. My
+resolution is to have published 52 posts on this blog by Jan 1, 2022, 00:00 UTC.
+Only one post per day can count towards the 52. A post must be "substantial" to
+count towards the 52. A non-substantial post would be something like the 100
+word essay about my weekend that I wrote in first grade, which went something
+like "My weekend was really really really ('really' 96 more times) really really
+boring".
+
+Other than that, it's pretty open-ended.
+
+## Why
+
+My hope is that I'll get more efficient at writing these things. Usually I take
+a lot of time to craft a post, weeks in some cases. I really appreciate those of
+you that have taken the time to read them, but to be frank the time commitment
+just isn't worth it. With practice I can hopefully learn what exactly I have to
+say that others are interested in, and then go back to spending a lot of time
+crafting the things being said.
+
+Another part of this is going to be learning how to market myself properly,
+something I've always been reticent to do. Our world is filled with people
+shouting into the void of the internet, each with their own reasons for wanting
+to be heard. Does it need another? Probably not. But here I am. I guess what I'm
+really going to be doing is learning _why_ I want to do this; I know I want to
+have others read what I write, but is it possible that that desire isn't
+entirely selfish? Is it ok if it is?
+
+Once I'm comfortable with why I'm doing this it will, hopefully, be easier to
+figure out a marketing avenue I feel comfortable with putting a lot of energy
+towards. There must be at least _one_...
+
+So consider this #1, world. Only 51 to go.
diff --git a/src/_posts/2021-01-09-ginger.md b/src/_posts/2021-01-09-ginger.md
new file mode 100644
index 0000000..3a97d7f
--- /dev/null
+++ b/src/_posts/2021-01-09-ginger.md
@@ -0,0 +1,352 @@
+---
+title: >-
+ Ginger
+description: >-
+ Yes, it does exist.
+---
+
+This post is about a programming language that's been bouncing around in my head
+for a _long_ time. I've tried to actually implement the language three or more
+times now, but everytime I get stuck or run out of steam. It doesn't help that
+everytime I try again the form of the language changes significantly. But all
+throughout the name of the language has always been "Ginger". It's a good name.
+
+In the last few years the form of the language has somewhat solidified in my
+head, so in lieu of actually working on it I'm going to talk about what it
+currently looks like.
+
+## Abstract Syntax Lists
+
+_In the beginning_ there was assembly. Well, really in the beginning there were
+punchcards, and probably something even more esoteric before that, but it was
+all effectively the same thing: a list of commands the computer would execute
+sequentially, with the ability to jump to odd places in the sequence depending
+on conditions at runtime. For the purpose of this post, we'll call this class of
+languages "abstract syntax list" (ASL) languages.
+
+Here's a hello world program in my favorite ASL language, brainfuck:
+
+```
+++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.++
++.------.--------.>>+.>++.
+```
+
+(If you've never seen brainfuck, it's deliberately unintelligible. But it _is_
+an ASL, each character representing a single command, executed by the brainfuck
+runtime from left to right.)
+
+ASLs did the job at the time, but luckily we've mostly moved on past them.
+
+## Abstract Syntax Trees
+
+Eventually programmers upgraded to C-like languages. Rather than a sequence of
+commands, these languages were syntactically represented by an "abstract syntax
+tree" (AST). Rather than executing commands in essentially the same order they
+are written, an AST language compiler reads the syntax into a tree of syntax
+nodes. What it then does with the tree is language dependent.
+
+Here's a program which outputs all numbers from 0 to 9 to stdout, written in
+(slightly non-idiomatic) Go:
+
+```go
+i := 0
+for {
+ if i == 10 {
+ break
+ }
+ fmt.Println(i)
+ i++
+}
+```
+
+When the Go compiler sees this, it's going to first parse the syntax into an
+AST. The AST might look something like this:
+
+```
+(root)
+ |-(:=)
+ | |-(i)
+ | |-(0)
+ |
+ |-(for)
+ |-(if)
+ | |-(==)
+ | | |-(i)
+ | | |-(10)
+ | |
+ | |-(break)
+ |
+ |-(fmt.Println)
+ | |-(i)
+ |
+ |-(++)
+ |-(i)
+```
+
+Each of the non-leaf nodes in the tree represents an operation, and the children
+of the node represent the arguments to that operation, if any. From here the
+compiler traverses the tree depth-first in order to turn each operation it finds
+into the appropriate machine code.
+
+There's a sub-class of AST languages called the LISP ("LISt Processor")
+languages. In a LISP language the AST is represented using lists of elements,
+where the first element in each list denotes the operation and the rest of the
+elements in the list (if any) represent the arguments. Traditionally each list
+is represented using parenthesis. For example `(+ 1 1)` represents adding 1 and
+1 together.
+
+As a more complex example, here's how to print numbers 0 through 9 to stdout
+using my favorite (and, honestly, only) LISP, Clojure:
+
+```clj
+(doseq
+ [n (range 10)]
+ (println n))
+```
+
+Much smaller, but the idea is there. In LISPs there is no differentiation
+between the syntax, the AST, and the language's data structures; they are all
+one and the same. For this reason LISPs generally have very powerful macro
+support, wherein one uses code written in the language to transform code written
+in that same language. With macros users can extend a language's functionality
+to support nearly anything they need to, but because macro generation happens
+_before_ compilation they can still reap the benefits of compiler optimizations.
+
+### AST Pitfalls
+
+The ASL (assembly) is essentially just a thin layer of human readability on top
+of raw CPU instructions. It does nothing in the way of representing code in the
+way that humans actually think about it (relationships of types, flow of data,
+encapsulation of behavior). The AST is a step towards expressing code in human
+terms, but it isn't quite there in my opinion. Let me show why by revisiting the
+Go example above:
+
+```go
+i := 0
+for {
+ if i > 9 {
+ break
+ }
+ fmt.Println(i)
+ i++
+}
+```
+
+When I understand this code I don't understand it in terms of its syntax. I
+understand it in terms of what it _does_. And what it does is this:
+
+* with a number starting at 0, start a loop.
+* if the number is greater than 9, stop the loop.
+* otherwise, print the number.
+* add one to the number.
+* go to start of loop.
+
+This behavior could be further abstracted into the original problem statement,
+"it prints numbers 0 through 9 to stdout", but that's too general, as there
+are different ways for that to be accomplished. The Clojure example first
+defines a list of numbers 0 through 9 and then iterates over that, rather than
+looping over a single number. These differences are important when understanding
+what code is doing.
+
+So what's the problem? My problem with ASTs is that the syntax I've written down
+does _not_ reflect the structure of the code or the flow of data which is in my
+head. In the AST representation if you want to follow the flow of data (a single
+number) you _have_ to understand the semantic meaning of `i` and `:=`; the AST
+structure itself does not convey how data is being moved or modified.
+Essentially, there's an extra implicit transformation that must be done to
+understand the code in human terms.
+
+## Ginger: An Abstract Syntax Graph Language
+
+In my view the next step is towards using graphs rather than trees for
+representing our code. A graph has the benefit of being able to reference
+"backwards" into itself, where a tree cannot, and so can represent the flow of
+data much more directly.
+
+I would like Ginger to be an ASG language where the language is the graph,
+similar to a LISP. But what does this look like exactly? Well, I have a good
+idea about what the graph _structure_ will be like and how it will function, but
+the syntax is something I haven't bothered much with yet. Representing graph
+structures in a text file is a problem to be tackled all on its own. For this
+post we'll use a made-up, overly verbose, and probably non-usable syntax, but
+hopefully it will convey the graph structure well enough.
+
+### Nodes, Edges, and Tuples
+
+All graphs have nodes, where each node contains a value. A single unique value
+can only have a single node in a graph. Nodes are connected by edges, where
+edges have a direction and can contain a value themselves.
+
+In the context of Ginger, a node represents a value as expected, and the value
+on an edge represents an operation to take on that value. For example:
+
+```
+5 -incr-> n
+```
+
+`5` and `n` are both nodes in the graph, with an edge going from `5` to `n` that
+has the value `incr`. When it comes time to interpret the graph we say that the
+value of `n` can be calculated by giving `5` as the input to the operation
+`incr` (increment). In other words, the value of `n` is `6`.
+
+What about operations which have more than one input value? For this Ginger
+introduces the tuple to its graph type. A tuple is like a node, except that it's
+anonymous, which allows more than one to exist within the same graph, as they do
+not share the same value. For the purposes of this blog post we'll represent
+tuples like this:
+
+```
+1 -> } -add-> t
+2 -> }
+```
+
+`t`'s value is the result of passing a tuple of two values, `1` and `2`, as
+inputs to the operation `add`. In other words, the value of `t` is `3`.
+
+For the syntax being described in this post we allow that a single contiguous
+graph can be represented as multiple related sections. This can be done because
+each node's value is unique, so when the same value is used in disparate
+sections we can merge the two sections on that value. For example, the following
+two graphs are exactly equivalent (note the parenthesis wrapping the graph which
+has been split):
+
+```
+1 -> } -add-> t -incr-> tt
+2 -> }
+```
+
+```
+(
+ 1 -> } -add-> t
+ 2 -> }
+
+ t -incr-> tt
+)
+```
+
+(`tt` is `4` in both cases.)
+
+A tuple with only one input edge, a 1-tuple, is a no-op, semantically, but can
+be useful structurally to chain multiple operations together without defining
+new value names. In the above example the `t` value can be eliminated using a
+1-tuple.
+
+```
+1 -> } -add-> } -incr-> tt
+2 -> }
+```
+
+When an integer is used as an operation on a tuple value then the effect is to
+output the value in the tuple at that index. For example:
+
+```
+1 -> } -0-> } -incr-> t
+2 -> }
+```
+
+(`t` is `2`.)
+
+### Operations
+
+When a value sits on an edge it is used as an operation on the input of that
+edge. Some operations will no doubt be builtin, like `add`, but users should be
+able to define their own operations. This can be done using the `in` and `out`
+special values. When a graph is used as an operation it is scanned for both `in`
+and `out` values. `in` is set to the input value of the operation, and the value
+of `out` is used as the output of the operation.
+
+Here we will define the `incr` operation and then use it. Note that we set the
+`incr` value to be an entire sub-graph which represents the operation's body.
+
+```
+( in -> } -add-> out
+ 1 -> } ) -> incr
+
+5 -incr-> n
+```
+
+(`n` is `6`.)
+
+The output of an operation may itself be a tuple. Here's an implementation and
+usage of `double-incr`, which increments two values at once.
+
+```
+( in -0-> } -incr-> } -> out
+ }
+ in -1-> } -incr-> } ) -> double-incr
+
+1 -> } -double-incr-> t -add-> tt
+2 -> }
+```
+
+(`t` is a 2-tuple with values `2`, and `3`, `tt` is `5.)
+
+### Conditionals
+
+The conditional is a bit weird, and I'm not totally settled on it yet. For now
+we'll use this. The `if` operation expects as an input a 2-tuple whose first
+value is a boolean and whose second value will be passed along. The `if`
+operation is special in that it has _two_ output edges. The first will be taken
+if the boolean is true, the second if the boolean is false. The second value in
+the input tuple, the one to be passed along, is used as the input to whichever
+branch is taken.
+
+Here is an implementation and usage of `max`, which takes two numbers and
+outputs the greater of the two. Note that the `if` operation has two output
+edges, but our syntax doesn't represent that very cleanly.
+
+```
+( in -gt-> } -if-> } -0-> out
+ in -> } -> } -1-> out ) -> max
+
+1 -> } -max-> t
+2 -> }
+```
+
+(`t` is `2`.)
+
+It would be simple enough to create a `switch` macro on top of `if`, to allow
+for multiple conditionals to be tested at once.
+
+### Loops
+
+Loops are tricky, and I have two thoughts about how they might be accomplished.
+One is to literally draw an edge from the right end of the graph back to the
+left, at the point where the loop should occur, as that's conceptually what's
+happening. But representing that in a text file is difficult. For now I'll
+introduce the special `recur` value, and leave this whole section as TBD.
+
+`recur` is cousin of `in` and `out`, in that it's a special value and not an
+operation. It takes whatever value it's set to and calls the current operation
+with that as input. As an example, here is our now classic 0 through 9 printer
+(assume `println` outputs whatever it was input):
+
+```
+// incr-1 is an operation which takes a 2-tuple and returns the same 2-tuple
+// with the first element incremented.
+( in -0-> } -incr-> } -> out
+ in -1-> } ) -> incr-1
+
+( in -eq-> } -if-> out
+ in -> } -> } -0-> } -println-> } -incr-1-> } -> recur ) -> print-range
+
+0 -> } -print-range-> }
+10 -> }
+```
+
+## Next Steps
+
+This post is long enough, and I think gives at least a basic idea of what I'm
+going for. The syntax presented here is _extremely_ rudimentary, and is almost
+definitely not what any final version of the syntax would look like. But the
+general idea behind the structure is sound, I think.
+
+I have a lot of further ideas for Ginger I haven't presented here. Hopefully as
+time goes on and I work on the language more some of those ideas can start
+taking a more concrete shape and I can write about them.
+
+The next thing I need to do for Ginger is to implement (again) the graph type
+for it, since the last one I implemented didn't include tuples. Maybe I can
+extend it instead of re-writing it. After that it will be time to really buckle
+down and figure out a syntax. Once a syntax is established then it's time to
+start on the compiler!
diff --git a/src/_posts/2021-01-14-the-web.md b/src/_posts/2021-01-14-the-web.md
new file mode 100644
index 0000000..4d47a57
--- /dev/null
+++ b/src/_posts/2021-01-14-the-web.md
@@ -0,0 +1,239 @@
+---
+title: >-
+ The Web
+description: >-
+ What is it good for?
+---
+
+With the recent crisis in the US's democratic process, there's been much abuzz
+in the world about social media's undoubted role in the whole debacle. The
+extent to which the algorithms of Facebook, Twitter, Youtube, TikTok, etc, have
+played a role in the radicalization of large segments of the world's population
+is one popular topic. Another is the tactics those same companies are now
+employing to try and euthanize the monster they made so much ad money in
+creating.
+
+I don't want to talk about any of that; there is more to the web than
+social media. I want to talk about what the web could be, and to do that I want
+to first talk about what it has been.
+
+## Web 1.0
+
+In the 1950's computers were generally owned by large organizations like
+companies, universities, and governments. They were used to compute and manage
+large amounts of data, and each existed independently of the other.
+
+In the 60's protocols began to be developed which would allow them to
+communicate over large distances, and thereby share resources (both
+computational and informational).
+
+The funding of ARPANET by the US DoD led to the initial versions of the TCP/IP
+protocol in the 70's, still used today as the backbone of virtually all internet
+communication. Email also came about from ARPANET around this time.
+
+The 80s saw the growth of the internet across the world, as ARPANET gave way to
+NSFNET. It was during this time that the domain name system we use today was
+developed. At this point the internet use was still mostly for large
+non-commercial organizations; there was little commercial footprint, and little
+private access. The first commercially available ISP, which allowed access to
+the internet from private homes via dialup, wasn't launched until 1989.
+
+And so we find ourselves in the year 1989, when Tim Berners-Lee (TBL) first
+proposed the World-Wide Web (WWW, or "the web"). You can find the original
+proposal, which is surprisingly short and non-technical,
+[here](https://www.w3.org/Proposal.html).
+
+From reading TBL's proposal it's clear that what he was after was some mechanism
+for hosting information on his machine in such a way that others could find and
+view it, without it needing to be explicitly sent to them. He includes the
+following under the "Applications" header:
+
+> The application of a universal hypertext system, once in place, will cover
+> many areas such as document registration, on-line help, project documentation,
+> news schemes and so on.
+
+But out of such a humble scope grew one of the most powerful forces of the 21st
+century. By the end of 1990 TBL had written the first HTML/HTTP browser and
+server. By the end of 1994 sites like IMDB, Yahoo, and Bianca's Smut Shack were
+live and being accessed by consumers. The web grew that fast.
+
+In my view the characteristic of the web which catalyzed its adoption so quickly
+was the place-ness of it. The web is not just a protocol for transferring
+information, like email, but instead is a _place_ where that information lives.
+Any one place could be freely linked to any other place, and so complex and
+interesting relations could be formed between people and ideas. The
+contributions people make on the web can reverberate farther than they would or
+could in any other medium precisely because those contributions aren't tied to
+some one-off event or a deteriorating piece of physical infrastructure, but are
+instead given a home which is both permanent and everywhere.
+
+The other advantage of the web, at the time, was its simplicity. HTML was so
+simple it was basically human-readable. A basic HTTP server could be implemented
+as a hobby project by anyone in any language. Hosting your own website was a
+relatively straightforward task which anyone with a computer and an ISP could
+undertake.
+
+This was the environment early adopters of the web found themselves in.
+
+## Web 2.0
+
+The infamous dot-com boom took place in 2001. I don't believe this was a failure
+inherent in the principles of the web itself, but instead was a product of
+people investing in a technology they didn't fully understand. The web, as it
+was then, wasn't really designed with money-making in mind. It certainly allowed
+for it, but that wasn't the use-case being addressed.
+
+But of course, in this world we live in, if there's money to be made, it will
+certainly be made.
+
+By 2003 the phrase "Web 2.0" started popping up. I remember this. To me "Web
+2.0" meant a new aesthetic on the web, complete with bubble buttons and centered
+fix-width paragraph boxes. But what "Web 2.0" actually signified wasn't related
+to any new technology or aesthetic. It was a new strategy for how companies
+could enable use of the web by non-expert users, i.e. users who don't have the
+inclination or means to host their own website. Web 2.0 was a strategy for
+giving everyone a _place_ of their own on the web.
+
+"Web 2.0" was merely a label given to a movement which had already been in
+motion for years. I think the following Wikipedia excerpt describes this period
+best:
+
+
+> In 2004, the term ["Web 2.0"] began its rise in popularity when O'Reilly Media
+and MediaLive hosted the first Web 2.0 conference. In their opening remarks,
+John Battelle and Tim O'Reilly outlined their definition of the "Web as
+Platform", where software applications are built upon the Web as opposed to upon
+the desktop. The unique aspect of this migration, they argued, is that
+"customers are building your business for you". They argued that the
+activities of users generating content (in the form of ideas, text, videos, or
+pictures) could be "harnessed" to create value.
+
+
+In other words, Web 2.0 turned the place-ness of the web into a commodity.
+Rather than expect everyone to host, or arrange for the hosting, of their own
+corner of the web, the technologists would do it for them for "free"! This
+coincided with the increasing complexity of the underlying technology of the
+web; websites grew to be flashy, interactive, and stateful applications which
+_did_ things rather than be places which _held_ things. The idea of a hyperlink,
+upon which the success of the web had been founded, became merely an
+implementation detail.
+
+And so the walled gardens began to be built. Myspace was founded in 2003,
+Facebook opened to the public in 2006, Digg (the precursor to reddit) was
+launched in 2004, Flickr launched in 2004 (and was bought by Yahoo in 2005),
+Google bought Blogger in 2003, and Twitter launched in 2006. In effect this
+period both opened the web up to everyone and established the way we still use
+it today.
+
+It's upon these foundations that current events unfold. We have platforms whose
+only incentive is towards capturing new users and holding their attention, to
+the exclusion of other platforms, so they can be advertised to. Users are
+enticed in because they are being offered a place on the web, a place of their
+own to express themselves from, in order to find out the worth of their
+expressions to the rest of the world. But they aren't expressing to the world at
+large, they are expressing to a social media platform, a business, and so only
+the most lucrative of voices are heard.
+
+So much for not wanting to talk about social media.
+
+## Web 3.0
+
+The new hot topic in crypto and hacker circles is "Web 3.0", or the
+decentralized web (dweb). The idea is that we can have all the good of the
+current web (the accessibility, utility, permanency, etc) without all the bad
+(the centralized platforms, censorship, advertising, etc). The way forward to
+this utopian dream is by building decentralized applications (dApps).
+
+dApps are constructed in a way where all the users of an application help to
+host all the stateful content of that application. If I, as a user, post an
+image to a dApp, the idea is that other users of that same dApp would lend their
+meager computer resources to ensure my image is never forgotten, and in turn I
+would lend mine for theirs.
+
+In practice building successful dApps is enormously difficult for many reasons,
+and really I'm not sure there _are_ any successful ones (to date). While I
+support the general sentiment behind them, I sometimes wonder about the
+efficacy. What people want from the web is a place they can call their own, a
+place from which they can express themselves and share their contributions with
+others with all the speed and pervasiveness that the internet offers. A dApp is
+just another walled garden with specific capabilities; it offers only free
+hosting, not free expression.
+
+## Web 2.0b
+
+I'm not here solely to complain (just mostly).
+
+Thinking back to Web 1.0, and specifically to the turning point between 1.0 and
+2.0, I'd like to propose that maybe we made a wrong turn. The issue at hand was
+that hosting one's own site was still too much of a technical burden, and the
+direction we went was towards having businesses host them for us. Perhaps there
+was another way.
+
+What are the specific difficulties with hosting one's own site? Here are the
+ones I can think of:
+
+* Bad tooling: basically none of the tools you're required to use (web server,
+ TLS, DNS, your home router) are designed for the average person.
+
+* Aggregiously complex languages: making a site which looks half decent and can
+ do the things you want requires a _lot_ of knowledge about the underlying
+ languages (CSS, HTML, Javascript, and whatever your server is written in).
+
+* Single point-of-failure: if your machine is off, your site is down.
+
+* Security: it's important to stay ahead of the hackers, but it takes time to
+ do so.
+
+* Hostile environment: this is separate from security, and includes difficulties
+ like dynamic home IPs and bad ISP policies (such as asymetric upload/download
+ speeds).
+
+These are each separate avenues of attack.
+
+Bad tooling is a result of the fact that devs generally build technology for
+themselves or their fellow devs, and only build for others when they're being
+paid to do it. This is merely an attitude problem.
+
+Complex languages are really a sub-category of bad tooling. The concesus seems
+to be that the average person isn't interested or capable of working in
+HTML/CSS/JS. This may be true today, but it wasn't always. Most of my friends in
+middle and high school were well within their interest and capability to create
+the most heinous MySpace pages the world has ever seen, using nothing but CSS
+generators and scraps of shitty JS they found lying around. So what changed? The
+tools we use to build those pages did.
+
+A hostile environment is not something any individual can do anything about, but
+in the capitalist system we exist in we can at least hold in faith the idea that
+eventually us customers will get what we want. It may take a long time, but all
+monopolies break eventually, and someone will eventually sell us the internet
+access we're asking for. If all other pieces are in place I think we'll have
+enough people asking to make a difference.
+
+For single point-of-failure we have to grant that more than one person will be
+involved, since the vast majority of people aren't going to be able to keep one
+machine online consistently, let alone two or more machines. But I think we all
+know at least one person who could keep a machine online with some reliability,
+and they probably know a couple of other people who could do so as well. What
+I'm proposing is that, rather than building tools for global decentralization,
+we need tools for local decentralization, aka federation. We can make it
+possible for a group of people to have their presence managed by a subset of
+themselves. Those with the ability could help to host the online presence of
+their family, friends, churches, etc, if given the right tools.
+
+Security is the hard one, but also in many ways isn't. What most people want
+from the web is a place from which to express themselves. Expression doesn't
+take much more than a static page, usually, and there's not much attacking one
+can do against a static page. Additionally, we've already established that
+there's going to be at least a _couple_ of technically minded people involved in
+hosting this thing.
+
+So that's my idea that I'd like to build towards. First among these ideas is
+that we need tools which can help people help each other host their content, and
+on top of that foundation a new web can be built which values honest expression
+rather than the lucrative madness which our current algorithms love so much.
+
+This project was already somewhat started by
+[Cryptorado](https://github.com/Cryptorado-Community/Cryptorado-Node) while I
+was a regular attendee, but since COVID started my attendance has fallen off.
+Hopefully one day it can resume. In the meantime I'm going to be working on
+setting up these tools for myself, and see how far I can get.