Some people are still unclear about where exactly the flaws in the 1.1.x release series were, how they could be exploited, and what the patches did (or didn’t do). So step by step, I am going to lead you through the two main avenues of attack against 1.1.4, and explain the problems with the withdrawn 1.1.5 patch.

testing environment

I am testing this on OS X 10.4.7, Webrick, and MySQL 5. Rails tags are checked out into /vendor/, and the application is a slightly modified version of the well-known Cookbook example.

denial-of-service

This is what people have been referring to as the routing bug.

I added some logging code into rails/action_pack/lib/action_controller/routing.rb. First let’s see what happens when we request a correct controller and action.

in "traverse_to_controller" method
  Recognized route segment: recipe
  Looking for controller RecipeController in module Object
  in "attempt_load" method
  Loading RecipeController in module Object in path recipe_controller
    found script/../config/../app/controllers/recipe_controller.rb as a possible match
    loaded RecipeController
    found script/../config/../app/controllers/recipe_controller.rb as a possible match
    loaded RecipeController
  No controller found, so looking for module: Recipe
  in "attempt_load" method
  Loading Recipe in module Object in path recipe
    found script/../config/../app/models/recipe.rb as a possible match
    loaded Recipe

in "traverse_to_controller" method
  Recognized route segment: recipe
  Looking for controller RecipeController in module Object
  Found it

The reasons why it executes in the order that it does are not entirely obvious. Singleton classes get tossed around in the source and it can be difficult to follow.

Still, it is easy to see the look-up functions stepping through the module hierarchy and checking for already loaded constants for the relevant controller and model name. If they are not found, it sees if there are any files it can load that might contain them. If they are still not found, it tries to decend into whatever modules are available (or directories, which it dynamically creates as modules for namespacing purposes), and looks again.

Let’s try a malicious URL:

in "traverse_to_controller" method
  Recognized route segment: breakpoint_client
  Looking for controller BreakpointClientController in module Object
  in "attempt_load" method
  Loading BreakpointClientController in module Object in path breakpoint_client_controller
  No controller found, so looking for module: BreakpointClient
  in "attempt_load" method
  Loading BreakpointClient in module Object in path breakpoint_client
    found script/../config/../vendor/rails/railties/lib/breakpoint_client.rb as a possible match

Whoops. Controllers and models aren’t buried deep in /vendor/rails/railties/lib/. Still, it managed to find breakpoint_client.rb, and, being very optimistic about where you might like to store your controllers, tried to load it up to see if it was the right thing. Only after that load does the framework check if it found the class it was actually looking for.

Class definitions in Ruby are executed just like any other method call, so the entire file has to get evaluated during the load in order to really guarantee what is inside it. Of course, now the server is hanging around waiting for a breakpoint from itself, and the app no longer responds.

database corruption

What if we continue with the routing bug, but instead try something really nasty? First, how many recipes are in the database?

roaming-205-133:~/Projects/cookbook eweaver$ script/console
Loading development environment.
>> Recipe.count
=> 1
>> 

Looks like we won’t be eating very well.

in "traverse_to_controller" method
  Recognized route segment: db
  Looking for controller DbController in module Object
  in "attempt_load" method
  Loading DbController in module Object in path db_controller
  No controller found, so looking for module: Db
  in "attempt_load" method
  Loading Db in module Object in path db

in "traverse_to_controller" method
  Recognized route segment: schema
  Looking for controller SchemaController in module Db
  in "attempt_load" method
  Loading SchemaController in module Db in path db/schema_controller
  No controller found, so looking for module: Schema
  in "attempt_load" method
  Loading Schema in module Db in path db/schema
    found ./db/schema.rb as a possible match

...

The application eventually returns a routing error, and continues to function. Except:

[2006-08-12 02:57:05] INFO  WEBrick::HTTPServer#start: pid=14539 port=3000
-- create_table("categories", {:force=>true})
   -> 0.0459s
-- create_table("recipes", {:force=>true})
   -> 0.0059s
-- initialize_schema_information()
   -> 0.0007s
-- columns("schema_info")
   -> 0.0029s

Cue stunned silence.

roaming-205-133:~/Projects/cookbook eweaver$ script/console
Loading development environment.
>> Recipe.count
=> 0
>>

Start the database restore, because otherwise we’re going to starve. You could probably also force the server to execute migrations if you could guess their names, and damage the database that way, too.

remote code execution

This is what people have been referring to as the $LOAD_PATH bug, but it’s really an extension of the same attack as above. What if we could drop an arbitrary .rb file into one of the directories being examined, then load it?

We’ll add some pictures to our recipes using the file_column plugin, so we add a filename field to our recipes table, and put the file_column :filename macro in the Recipe class definition. We also upload a picture to Recipe 1. Let’s look on the filesystem to see where file_column put it.

~/Projects/cookbook/public/recipe/filename/1/yum.jpg

Fine, whatever. But maybe the application author didn’t check file extensions properly. Can we upload an .rb file?

Turns out that we can. Now we have ~/Projects/cookbook/public/recipe/filename/1/hax.rb, too. Can we find a route that will load it?

Routing Error
Recognition failed for "/public/recipe/filename/1/hax/"

No, we can’t, because “1” is not a valid module name. However, if you have text-based primary keys, or use a custom upload process that puts files in a subpath of /public/ such that each subdirectory name begins with a letter, you are vulnerable.

For example, Typo 4, last time I checked, uploads files into /public/files/. On an unpatched system, anyone with a login to the Typo instance could effectively run any Ruby code they wanted on the server—a classic privileges escalation.

(Actually, the file doesn’t always have to be in a subdirectory of public, because ”.” is included in the load path. On some webservers such as Lighttpd, ”.” will usually point to /public/, but on others, it points to RAILS_ROOT, leaving any file within the entire Rails tree potentionally loadable on such servers.)

From where exactly is the framework coming up with all of these “alternative” directory choices? Let’s add a little bit more logging code and request a route that doesn’t exist.

in "traverse_to_controller" method
  Recognized route segment: whatever
  Looking for controller WhateverController in module Object
  in "attempt_load" method
  Loading WhateverController in module Object in path whatever_controller
    entered directory ./script/../config/../vendor/rails/actionwebservice/lib/action_web_service/v
endor/
    entered directory ./script/../config/../vendor/rails/actionmailer/lib/action_mailer/vendor/
    entered directory ./script/../config/../vendor/rails/actionpack/lib/action_view/helpers/../../
action_controller/vendor/html-scanner
    entered directory ./script/../config/../vendor/rails/actionpack/lib/action_view/vendor
    entered directory ./script/../config/../vendor/rails/actionpack/lib
    entered directory ./script/../config/../vendor/rails/activerecord/lib
    entered directory script/../config/../test/mocks/development
    entered directory script/../config/../app/controllers/
    entered directory script/../config/../app
    entered directory script/../config/../app/models
    entered directory script/../config/../app/controllers
    entered directory script/../config/../app/helpers
    entered directory script/../config/../components
    entered directory script/../config/../config
    entered directory script/../config/../lib
    entered directory script/../config/../vendor
    entered directory script/../config/../vendor/rails/railties
    entered directory script/../config/../vendor/rails/railties/lib
    entered directory script/../config/../vendor/rails/actionpack/lib
    entered directory script/../config/../vendor/rails/activesupport/lib
    entered directory script/../config/../vendor/rails/activerecord/lib
    entered directory script/../config/../vendor/rails/actionmailer/lib

Routing Error
Recognition failed for "/whatever"

Crazyness! That’s a lot of places to look for whatever_controller.rb. Remember, too, that any subdirectories under any of these places are vulnerable if they can be properly namespaced in the route.

It turns out that Rails constructs the safe_load_paths array from several different places. Most of them are set in the application’s environment.rb or in the framework itself, and are immutable. But it also adds the contents of the $LOAD_PATH environment variable.

Previously I said that on some webservers, sending a Load-Path: header in the request might add the contents to the Rails $LOAD_PATH. I have tested, and I was wrong. There was no vulnerability related to the header. I don’t know if there are server-side ways to manipulate the $LOAD_PATH, though.

what did the 1.1.5 patch do?

The important change occurred in routing.rb, although 1.1.5 also added an integration testing method relating to the $XHTTP_LOAD_PATH request parameter. This turned out to be not relevant to production environments. Also, a test was added for an SQL injection vulnerability, but there is no known exploit related to that test. It seems to be merely a preventative measure.

You can get a diff of the changes between 1.1.4 and 1.1.5 in routing.rb here. There were some other changes, too, but this is the most relevant. If you want to look yourself, be sure to delete /doc/, /test/, and other irrelevant things like that before you run diff -r or you will get a huge mess of changes.

# removed
base[0, extended_root.length] == extended_root ||
  base =~ %r{rails-[\d.]+/builtin}

# added
base.match(/\A#{Regexp.escape(extended_root)}\/*#{file_kinds(:lib) * '|'}/) ||
  base =~ %r{rails-[\d.]+/builtin}

Let’s see what that does.

    entered directory ./script/../config/../vendor/rails/actionwebservice/lib/action_web_service/v
endor/
    entered directory ./script/../config/../vendor/rails/actionmailer/lib/action_mailer/vendor/
    entered directory ./script/../config/../vendor/rails/actionpack/lib/action_view/helpers/../../
action_controller/vendor/html-scanner
    entered directory /opt/local/lib/ruby/gems/1.8/gems/RedCloth-3.0.4/bin
    entered directory /opt/local/lib/ruby/gems/1.8/gems/RedCloth-3.0.4/lib
    entered directory ./script/../config/../vendor/rails/actionpack/lib/action_view/vendor
    entered directory ./script/../config/../vendor/rails/actionpack/lib
    entered directory ./script/../config/../vendor/rails/activerecord/lib
    entered directory script/../config/../app/controllers/
    entered directory script/../config/../app
    entered directory script/../config/../app/models
    entered directory script/../config/../app/controllers
    entered directory script/../config/../app/helpers
    entered directory script/../config/../lib
    entered directory script/../config/../vendor/rails/railties/lib
    entered directory script/../config/../vendor/rails/actionpack/lib
    entered directory script/../config/../vendor/rails/activesupport/lib
    entered directory script/../config/../vendor/rails/activerecord/lib
    entered directory script/../config/../vendor/rails/actionmailer/lib
    entered directory script/../config/../vendor/rails/actionwebservice/lib
    entered directory ./script/../config/../vendor/rails/activesupport/lib/active_support/vendor
    entered directory ./script/../config/../vendor/rails/activesupport/lib
    entered directory /opt/local/lib/ruby/site_ruby/1.8
    entered directory /opt/local/lib/ruby/site_ruby/1.8/i686-darwin8.6.1
    entered directory /opt/local/lib/ruby/site_ruby
    entered directory /opt/local/lib/ruby/vendor_ruby/1.8
    entered directory /opt/local/lib/ruby/vendor_ruby/1.8/i686-darwin8.6.1
    entered directory /opt/local/lib/ruby/vendor_ruby
    entered directory /opt/local/lib/ruby/1.8
    entered directory /opt/local/lib/ruby/1.8/i686-darwin8.6.1
  No controller found, so looking for module: Whatever
  in "attempt_load" method
  Loading Whatever in module Object in path whatever

@file_kinds:
[:app, :lib]

file_kinds(:lib):
[:app, :lib]

It turns out that the regular expression change merely limits the safe_load_paths to directories prefixed with /lib/ or /app/ relative to some fixed starting points (give or take). This broke Rails engines, which keep their libraries somewhere else. Also, there were still some unsafe files living in some of the directories listed above that could cause denials-of-service if requested via a route.

A stop-gap measure, mainly.

what does the 1.1.6 patch do?

Again, a diff of the changes between 1.1.5 and 1.1.6 in routing.rb is here.

The new method file_kinds, added by 1.1.5, is gone again. The regular expression is now:

base.match(/\A#{Regexp.escape(extended_root)}\/*(app|lib|components)\/[a-z]/) ||
  base =~ %r{rails-[\d.]+/builtin}

Another logger output. I’m getting tired of these.

in "traverse_to_controller" method
  Recognized route segment: whatever
  Looking for controller WhateverController in module Object
  in "attempt_load" method
  Loading WhateverController in module Object in path whatever_controller
    entered directory script/../config/../app/controllers/
    entered directory script/../config/../app/models
    entered directory script/../config/../app/controllers
    entered directory script/../config/../app/helpers
  No controller found, so looking for module: Whatever
  in "attempt_load" method
  Loading Whatever in module Object in path whatever
    entered directory script/../config/../app/controllers/
    entered directory script/../config/../app/models
    entered directory script/../config/../app/controllers
    entered directory script/../config/../app/helpers

Look how short it is! So much nicer. The fix itself, though, still seems kind of shady. The new routing and constantizing code in Rails edge is a big improvement. Personally, I recommend that you update to edge if it won’t break your application, in case there are still vulnerable URLs or routes in 1.1.6.

in the wild?

I have heard very few reports of actual public-facing Rails installations being exploited. Please let me know if you know of (or own) any such sites, and any details about the attack that you have.

conclusion

That’s all, really. I completely understand that after such a marathon patch session, the team was tired and that their patience was wearing thin. They have been good (I think) about letting critical comments stay on the weblog. But still, Hansson, core, and everyone, please don’t bully us poor users, especially right after a previous patch rollback and keeping in mind this ticket from June 16.

Anyway, lessons learned by all, etc., etc. Next time will be better… or it will be Django >:) .

I do know that Camping’s routes are immune to this particular problem. There has also been a little bit of speculation on IRC about back-forking Rails into a security-audited version every six months or so. IronRails, anyone?

Finally, I would like to thank a few people on IRC who gave me clues or encouragement during this ordeal: defunkt, isotopp, bitsweat, icblenke, and joelmichael.

postscript

I will be presenting on Camping at the September Philly-on-Rails meeting.