snax

I've been reading a bunch of papers about distributed systems recently, in order to help systematize for myself the thing that we built over the last year. Many of them were originally passed to me by Toby DiPasquale. Here is an annotated list so everyone can benefit.

It helps if you have some algorithms literacy, or have built a system at scale, but don't let that stop you.

prelude

The Death of Architecture, Julian Browne, 2007.

First, a reminder of what it means to build a system that solves a business problem. Browne built the space-based billing system at Virgin Mobile, one of the most well-known SBAs outside the financial and research industries.

That lovely diagram showing clean service-oriented interfaces, between unified systems of record, holding clean data, performing well-defined business processes is never going to be....You have to roll up your sleeves, talk to the business analysts, developers, operations and make a contribution that makes those boxes and arrows real.

System failures in the web world are most often due to inflated technical requirements and integration risks, not an incorrect decision to skip two-phase commit...

May 02, 2009

Memcached is generally treated as a black box. But what if you really need to know what's in there? Not for runtime purposes, but for optimization and capacity planning?

demo

$ sudo peep --pretty 2479

time | exptime | nbytes | clsid |                 key | exprd | flushd
8658 |  613458 |    272 |     5 |      "c3RhdH:171:5" | false |  false
8658 |       0 |      6 |     1 |  "current_c3RhdH:3" | false |  false
8658 |  613458 |    281 |     5 |     "c3RhdH:171:26" | false |  false
8678 |   95078 |      6 |     1 |  "User:1:auth:m4Uq" | false |  false
8658 |       0 |      8 |     2 |   "user_dGltZWxp:4" | false |  false
8686 |  613486 |   1278 |     9 |          "User:1:6" | false |  false
8658 |  613458 |   1286 |     9 |          "User:1:4" | false |  false
...
8658 |  613458 |    283 |     5 |     "c3RhdH:171:28" | false |  false
8658 |  613458 |    277 |     5 |     "c3RhdH:171:30" | false |  false

Peep uses ptrace to freeze a running memcached server, dump the internal key metadata, and return the server to a running state. If you have a good host ejection mechanism in your client, such as in the Twitter libmemcached builds, you won't even have to change the production server pool. The instance is not restarted, and no data is lost.

Installation instructions are here. Requires Linux.

a note about heap management

Memcached has two separate memory management strategies...

April 20, 2009

I'd like to call out something important from my QCon slides: the Railsbench GC settings.

quick study

In my experience, a typical production Rails app on Ruby 1.8 can recover 20% to 40% of user CPU by applying Stefan Kaes's Railsbench GC patch to the Ruby binary, and using the following environment variables:

RUBY_HEAP_MIN_SLOTS=500000 
RUBY_HEAP_SLOTS_INCREMENT=250000 
RUBY_HEAP_SLOTS_GROWTH_FACTOR=1 
RUBY_GC_MALLOC_LIMIT=50000000 

In return, you get a minor increase in peak memory consumption. Not too shabby.

gc behavior

Ruby's GC is non-generational, which means that the GC looks at every object every time it runs. ...

April 09, 2009

My QCon presentation is available.

Improving Running Components at Twitter

Some choice Tweets:

• philwills: Evan Weaver on scaling twitter at #qcon was full of interesting stuff and good questions from audience.
• markhneedham: fascinating reading these stats about #twitter from Evan Weaver's talk #qcon
• jurgenonland: sitting at a presentation from Evan Weaver @ #qcon, wow he must be verry unhappy at his work
• szegedi: Listening to Evan Weaver talking about Twitter system architecture & tuning. Getting to learn from these experiences is priceless.
• rbanks54: This could be a great presentation, but Evan is presenting it monotone/bored & skipping important links :-( #qcon
• oudenampsen: Was just by Evan Weaver of twitter. Gave the impression that any time he could commit suicide. However interesting.

My presentation abilities have gone from "bad" to "tolerable", so I'm relatively satisfied with the situation. Clearly I need to be more engaging.

March 13, 2009

Here are some secret codes I am involved with. They are some of the best codes recently coded.

kestrel

A replacement for Starling, the distributed message queue. Written on the JVM (Scala) because of the mature garbage collector. Has a constant performance profile regardless of the size of the queue.

memcached/libmemcached pre-builds

The C library and the Ruby client as a matched pair. Much improved failure handling over the public builds, and Ketama hashing is built-in as always. Switching from ruby-memcache to memcached at Twitter effectively halved our cluster CPU load. (These changes are getting upstreamed, so you can also just wait.)

cache-money

A Rails object cache layer for ActiveRecord. It can do primary key lookups and single index queries solely out of cache. Especially nice if you have replication lag.

peep

A heap inspector for live memcached server instances. They said it couldn't be done.

thinking sphinx

Pat Allan has been working hard on his Sphinx plugin. He's adding the missing enterprisey features from Ultrasphinx, so that it can finish its years sleeping gently in legacy apps. (Facets, and non-invasive delta indexing.)

bybusyness

Apache 2.2.10 got a new fair balancing algorithm. Smooths out latencies introduced by single-threaded backends with high standard deviation, such as Mongrel/Rails.

postscripts

Speaking of, I'm making sporadic progress on Mongrel 2, but don't hold your breath. The main improvement will be Rack support.

A downside of my position at Twitter is meager open-source progress. But hey, we're hiring (must be in SF or willing to move).

My RSI is slowly going away. The only real solution is to type less. And make sure to keep totally blasting your pecs.

January 24, 2009

As some people already know, I'm moving to San Francisco in early September. One thing I had trouble finding was a map of the San Francisco microclimates. But here's one:

It's from this book.

I'll be living on the border of North Beach and Russian Hill (which is in the warm zone, 6). No place in San Francisco is really "hot", despite what that map says. But the patterns of fog and relative coolness are correct.

postscript

Sorry that my blogging and open-source output has been light; I'm recovering from serious RSI problems. The fact that you see this post at all is a sign that things are getting better, though.

August 05, 2008