snax

rails search benchmarks

I put together some benchmarks for the three main Rails fulltext search solutions: Sphinx/Ultrasphinx, Ferret/acts_as_ferret, and Solr/acts_as_solr. The book Advanced Rails Recipes was a big help in getting Ferret and Solr running quickly.

dataset

The dataset is the entire KJV Bible, indexed by verse and also by book. This gives us 31,102 smallish records and 66 large ones. Ferret and Solr both use a Ruby method for loading the per-book contents (since they traverse a Rails association), while Sphinx (with Ultrasphinx) uses :concatenate to generate a GROUP_CONCAT MySQL query.

You can checkout or browse the benchmark app yourself from here. Especially note the model configurations. The app should be runnable; the migrations include the dataset load.

performance results

These results exercise some basic queries of varying sizes. Some things are not covered; I may update the benches in the future for facet, filter, phrase, and operator usage.

I search 300 times each for a common short word in verses, the same word in books, the same word in all classes, then a rare pair of words in all classes, and finally a very rare long phrase in all classes. All engines were configured for no stopwords.

$ INDEX=1 rake benchmark

Sphinx
                               user     system      total        real
reindex                    0.000000   0.010000   2.310000 (  8.323945)
verse:god                  1.090000   0.160000   1.250000 ( 31.020551)
verse:god:no_ar            0.720000   0.080000   0.800000 ( 27.364780)
book:god                   0.980000   0.100000   1.080000 ( 26.839016)
all:god                    0.970000   0.100000   1.070000 ( 20.297412)
all:calves                 1.030000   0.110000   1.140000 ( 22.806805)
all:moreover               0.980000   0.120000   1.100000 ( 27.763920)
result counts: [3595, 64, 3659, 5, 2]
index size: 7.6M        total
memory usage in kb: {:virtual=>35356, :total=>35688, :real=>332}

Solr
                               user     system      total        real
reindex                  403.500000   4.650000 408.150000 (500.704153)
verse:god                  2.530000   0.500000   3.030000 ( 30.330766)
book:god                   1.910000   0.280000   2.190000 ( 30.164732)
all:god                    2.940000   0.360000   3.300000 ( 30.864319)
all:calves                 2.250000   0.330000   2.580000 ( 19.039895)
all:moreover               1.860000   0.300000   2.160000 ( 23.407134)
result counts: [4077, 64, 4141, 5, 2]
index size: 7.8M        total
memory usage in kb: {:virtual=>219376, :total=>298644, :real=>79268}

Ferret
                               user     system      total        real
reindex                    0.830000   2.130000   2.960000 (512.818894)
verse:god                  0.760000   0.210000   0.970000 (  2.557016)
book:god                   0.740000   0.030000   0.770000 (  1.914840)
all:god                  144.460000   4.430000 148.890000 (602.861915)
all:calves                 1.010000   0.050000   1.060000 (  3.033010)
all:moreover               0.710000   0.060000   0.770000 (  4.185469)
result counts: [3893, 64, 3957, 7, 2]
index size: 13M         total
memory usage in kb: {:virtual=>47272, :total=>112060, :real=>64788}

The horrible Ferret performance for “all:god” happened consistently. The log suggests that it does not use any kind of limit in multi-model search in order to ensure the relevance order is correct. This is a big fail.

The “real” column times for Sphinx and Solr are the same, which suggests that the bulk of it is socket overhead. The dataset is too small for the actual query time to have an effect. However, it looks like Ferret is reusing the socket (via DRb) which is a point in its favor. Sphinx currently does not support persistent connections.

It is important to realize this does not mean that Ferret is fastest overall. It means that Ferret is fastest for small datasets where the constant socket overhead dwarfs the logarithmic actual lookup overhead.

Do note that the “total” time spent (e.g., time spent in Ruby, instead of waiting on IO) is much lower for Sphinx than for Solr.

It would help the benchmark validity to run many query containers in parallel on separate machines, and to use a much larger dataset.

Other people’s benchmarks suggest that Sphinx starts to scale really well as query volume increases. Solr is likely to be within the same order of magnitude.

quality results

An extremely crude evaluation of search quality: which result set for the word “God” has the word repeated the most times in the records?

# Sphinx
Ultrasphinx::Search.new(:class_names => 'Verse', :query => "God",
:per_page => 10).run.map(&:content).join(" ").split(" ").
select{|s| s[/god/i]}.size
=> 45

# Solr
Verse.find_by_solr("God", :limit => 10).docs.map(&:content).join(" ").
split(" ").select{|s| s[/god/i]}.size
=> 30

# Ferret
Verse.find_by_contents("God", :limit => 10).map(&:content).join(" ").
split(" ").select{|s| s[/god/i]}.size
=> 26

Not much, but it’s something.

thoughts on usage

It’s interesting how similar the acts_as_ferret and acts_as_solr query interfaces are, and how different Ultrasphinx’s is. Multi-model search is an afterthought in Ferret and Solr, and it shows. (No other Rails Sphinx plugin supports multi-model search.)

The configuration interfaces are pretty similar until you start to get into engine-specific stuff like Ultrasphinx’s :association_sql, or Ferret’s analysis modules. Solr has its scary schema.xml but acts_as_solr hides that from you.

Ultrasphinx has some initialization annoyances which acts_as_solr doesn’t suffer from.

Ferret acted weird alongside Ultrasphinx unless I specifically required some acts_as_ferret files in environment.rb. Ferret also will index your entire table when you first reference the constant, which was a big surprise. In general Ferret is overly coupled. Solr is better and acts_as_solr does an especially nice job of hiding the Java from you.

I didn’t test any faceting ability. Solr probably has the best facet implementation. Ferret doesn’t seem to support facets at all.

on coupling

Ferret is unstable under load, and due to its very tight coupling, takes down your Rails containers along with it. Solr is pretty stable, but suffers from the opposite problem—if something goes wrong in a Rails container, and a record callback doesn’t fire, that record will never get indexed.

Sphinx avoids both these problems because it integrates through the database, not through the application. This is what databases are for. Sphinx is incredibly stable, but even if something happens to it, the loose coupling means that the only thing that fails in your app is search. And since it doesn’t rely on container callbacks, your index is always correct. This is the main reason I wrote Ultrasphinx.

Both Solr and Ferret are too slow to reindex on a regular basis. They could be much, much faster if they didn’t have to roundtrip every record through Ruby, but that’s how they’re designed.

Takeaway lesson—be deliberate about your integration points.