qcon presentation

My QCon presentation is available.

Improving Running Components at Twitter

Some choice Tweets:

  • philwills: Evan Weaver on scaling twitter at #qcon was full of
    interesting stuff and good questions from audience.
  • markhneedham: fascinating reading these stats about #twitter
    from Evan Weaver’s talk #qcon
  • jurgenonland: sitting at a presentation from Evan Weaver @
    #qcon, wow he must be verry unhappy at his work
  • szegedi: Listening to Evan Weaver talking about Twitter system
    architecture & tuning. Getting to learn from these experiences is priceless.
  • oudenampsen: Was just by Evan Weaver of twitter. Gave the impression that any time he could commit suicide. However interesting.

My presentation abilities have gone from “bad” to “tolerable”, so I’m relatively satisfied with the situation. Clearly I need to be more engaging.

secret codes

Here are some secret codes I am involved with. They are some of the best codes recently coded.


A replacement for Starling, the distributed message queue. Written on the JVM (Scala) because of the mature garbage collector. Has a constant performance profile regardless of the size of the queue.

memcached/libmemcached pre-builds

The C library and the Ruby client as a matched pair. Much improved failure handling over the public builds, and Ketama hashing is built-in as always. Switching from ruby-memcache to memcached at Twitter effectively halved our cluster CPU load. (These changes are getting upstreamed, so you can also just wait.)


A Rails object cache layer for ActiveRecord. It can do primary key lookups and single index queries solely out of cache. Especially nice if you have replication lag.


A heap inspector for live memcached server instances. They said it couldn’t be done.

thinking sphinx

Pat Allan has been working hard on his Sphinx plugin. He’s adding the missing enterprisey features from Ultrasphinx, so that it can finish its years sleeping gently in legacy apps. (Facets, and non-invasive delta indexing.)


Apache 2.2.10 got a new fair balancing algorithm. Smooths out latencies introduced by single-threaded backends with high standard deviation, such as Mongrel/Rails.


Speaking of, I’m making sporadic progress on Mongrel 2, but don’t hold your breath. The main improvement will be Rack support.

A downside of my position at Twitter is meager open-source progress. But hey, we’re hiring (must be in SF or willing to move).

My RSI is slowly going away. The only real solution is to type less. And make sure to keep totally blasting your pecs.

if you’re going to san francisco

As some people already know, I’m moving to San Francisco in early September. One thing I had trouble finding was a map of the San Francisco microclimates. But here’s one:

It’s from this book.

I’ll be living on the border of North Beach and Russian Hill (which is in the warm zone, 6). No place in San Francisco is really “hot”, despite what that map says. But the patterns of fog and relative coolness are correct.


Sorry that my blogging and open-source output has been light; I’m recovering from serious RSI problems. The fact that you see this post at all is a sign that things are getting better, though.

echoe 3

Echoe 3 is out, right on the heels of Rubygems 1.2. It supports the new runtime vs. development dependencies, and works correctly with the Rubyforge 1.0.0 gem.

It still supports all the usual features like certificate chains, RDoc upload, changeset parsing, manifest building, and cross-packaging. Documentation is here.

By the way, Rubygems 1.2 seems pretty great.

xapian search plugin

Francis Irving sent me a note about his work on a new Rails search plugin, acts_as_xapian. It uses the Xapian engine, which is a C++ indexer similar to Lucene. A particularly neat feature is built-in spellcheck.

I still plan to benchmark all these plugins on the Wikipedia dataset…it’s been delayed by the new job. If anyone has a big piece of iron I could use for a couple weeks I would appreciate it (16GB ram, hundreds of GB of free diskspace, no production load).

rubygems memory patch

This patch against RubyGems 1.1.1 improves memory usage by not keeping every unused gemspec permanently in memory. It should have low CPU impact as long as you do your gem requires up-front.

For MacPorts:

cd /opt/local/lib/ruby/site_ruby/1.8
curl https://evanweaver.files.wordpress.com/2010/12/rubygems-memory-1_1_1.diff \
  | sudo patch -p0

Incidentally, I used BleakHouse to track down which references were getting retained.


Right now, Twitter is suffering slowdowns. Earlier today it was down again. :(

There are an excessive number of single points of failure in the current system, and through developer error* and external circumstance we have managed to hit quite a few of them in the last week. I am sorry and embarrassed.

In particular, we mis-estimated the impact of some cache policy changes. If your site runs so hot that it can’t function without memcached, you’d better understand exactly how much buffer capacity you actually have.

We’re working on fixing it all, but it takes a long time…

* I ain’t sayin’ it was me, but I ain’t sayin’ it wasn’t.


Automatically tag your music collection with metadata from Last.fm.

what it is

A while back Last.fm released a command line tool to retrieve metadata for an arbitrary mp3 from their new fingerprint database. I tried it yesterday and it seemed way better than MusicBrainz. So, as a person with a lot of random mp3s, I cooked up a script for retagging entire folders of songs.

Some neat things used in the script:

  • id3lib-ruby for
    handling mp3 tags
  • Text for calculating
    Levenshtein distance to the nearest correct genre name—amatch is a compiled version
    of the same thing, but not Windows-compatible
  • the incredibly comprehensive Last.fm
  • XSD::Mapping for parsing the XML responses (better than
    Hpricot for small, well-formed documents)

A handy feature in the script is the ability to add the top 10 tagged genres to the comment field, so you can use iTunes or Foobar smart playlists for fancier multi-genre sorting. This is similar to lastfmtagger, but not Mac-specific.


Before running sweeper --genre:

$ id3info 1_001.mp3
*** Tag information for 1_001.mp3
*** mp3 info
MPEG1/layer III
Bitrate: 128KBps
Frequency: 44KHz


$ id3info 1_001.mp3
*** Tag information for 1_001.mp3
=== TPE1 (Lead performer(s)/Soloist(s)): Photon Band
=== TIT2 (Title/songname/content description): To Sing For You
=== WORS (Official internet radio station homepage): http://www.last.fm/music/Ph
=== TCON (Content type): Psychadelic
=== COMM (Comments): ()[]: rock, psychedelic, mod, Philly
*** mp3 info
MPEG1/layer III
Bitrate: 128KBps
Frequency: 44KHz


Documentation is here, but for OS X:

sudo port install id3lib
sudo gem install sweeper
sweeper --help

Linux is similar to the above, depending on your distribution.

On Windows, you can just download a zipfile from the Rubyforge page and extract sweeper.exe to somewhere in your path.

I expect this to be eventually replaced by an official Last.fm tool, but for now, patches are welcome. It would be especially nice if someone could write a tutorial to help non-Ruby people install the script.

If you are going to contribute some code, grab the SVN checkout from Fauna, since the gem doesn’t ship with the test mp3s.

SVN, I know—how embarrassing!

bleakhouse 4

BleakHouse 4 came to life this weekend.

new implementation

BleakHouse now tracks the spawn points of every object on the heap, somewhat like Valgrind and somewhat like Dike.

This means there is no framing necessary, and the analysis task runs in seconds instead of hours. On the other hand, the pure-C instrumentation means it’s fast enough to run in production, won’t introduce new leaks in your app, and can track T_NODE and other Ruby internals.


After exactly 2000 requests:

$ bleak /tmp/bleak.13795.0.dump
1334329 total objects
Final heap size 1334329 filled, 1132647 free
Displaying top 100 most common line/class pairs
408149 __null__:__null__:__node__
273858 (eval):3:String
135304 __null__:__null__:String
29998 /opt/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/lib/mongrel.rb:122:String
14000 /rails/activesupport/lib/active_support/core_ext/hash/keys.rb:8:String
11825 /rails/actionpack/lib/action_controller/base.rb:1215:String
7022 /opt/local/lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:Array
5995 /rails/actionpack/lib/action_controller/session/cookie_store.rb:145:String
4524 /opt/local/lib/ruby/gems/1.8/specifications/gettext-1.90.0.gemspec:14:String
4000 /opt/local/lib/ruby/1.8/cgi/session.rb:299:Array
4000 /rails/actionpack/lib/action_controller/response.rb:10:Array

Somebody’s got an eval leak, for sure. And those session.rb counts are pretty suspicious.

The BleakHouse docs are here. The codebase is very solid and I look forward to adding some neat things in 4.1 and 4.2.

credit where it’s due

Part of the development of BleakHouse 4 was sponsored by a Rails company you have definitely heard of.