Send files faster with X-Sendfile

Mon, 16 Apr 2007 20:51:00 GMT

Please don’t let this post scare you away from Mongrel. I have found it much more stable and easier to set up than FCGI. Mongrel + Capistrano saved me from the brink of insanity when deployment came around. The memory issue with send_file is indicative of a bigger issue with Ruby’s garbage collection—not a problem with Mongrel itself.

There’s this slick HTTP header called “X-Sendfile” that Apache and Lighttpd support. It was made to address this kind of problem (isn’t that convenient?). Now, when you send an HTTP response, all you have to do is tack on an X-Sendfile header with the path of the file you need to send—don’t worry about actually reading the file or sending any of those bytes yourself. The web server will load the file you specified and send it downstream.

This is loads easier on your Rails process (it just sets the headers and gets on with life), and you get all the cool file transfer functionality already baked into your web server (like proper caching, resuming, etc). Of course, you can send any file that the web server can read, not just files in your normal public directory. This makes it perfect for sending private or protected content to specific users of your app.

Setting those magical headers

You shouldn’t be too surprised that Rails makes it easy to set this header correctly. You could do something like this:

This code sample has been excerpted from the Ruby on Rails Wiki.

filename = "/var/www/myfile.xyz" 
response.headers['Content-Type'] = "application/force-download" 
response.headers['Content-Disposition'] = "attachment; filename=\"#{File.basename(filename)}\"" 
response.headers["X-Sendfile"] = filename
response.headers['Content-length'] = File.size(filename)
render :nothing => true

Apache users will need mod_xsendfile. Lighttpd users should look into mod_fastcgi and mod_proxy_core.

Now, as long as your server is correctly configured, you file should be zipping along at record speeds. If you need this functionality in more than one action or more than one app it’s not terribly DRY… and you should probably write some functional tests for this… ugh. Someone really ought to write a plugin.

The XSendfile plugin

The advanced or impatient may want to jump straight to the rdoc or source.

I wrapped up all this goodness in a plugin. I’ve had this running on a production server (Apache2 + Mongrel) for a couple of weeks with no issues. For the good of humanity, it comes complete with rdoc and tests.

Installation

You can install the plugin using Rails’ built-in plugin command.

ruby script/plugin install http://john.guen.in/svn/plugins/x_send_file/

Usage

The interface is as close as possible to Rails’ send_file command. If your server requires it, you can override the HTTP header used for X-Sendfile with the new :header option (the default is X-Sendfile).

x_send_file('/path/to/file')
x_send_file('/path/to/image.jpg', :type => 'image/jpeg', :disposition => 'inline')
x_send_file('/path/to/file/', :header => 'X-LIGHTTPD-SEND-FILE')

If you want x_send_file jump in and take over any time send_file is used, add this to your environment.rb:

That ! stands for “danger!” Use this method responsibly.

XSendFile::Plugin.replace_send_file!

More information

If you’re ready to dig further, check out these resources:

The rdoc
The source
RoR wiki’s How To Send Files Fast
Apache’s mod_xsendfile
Lighttpd’s mod_fastcgi and mod_proxy_core

So your subversion server is full...

Sun, 08 Apr 2007 22:47:00 GMT

Once you have your fancy new server up & running, you have to prep your svn repository:

svnadmin create /path/to/new/repos

Once that’s done, you just need to send the data over from your old repository. Run this command on your old server:

svnadmin dump /path/to/old/repos | ssh user@newserver "svnadmin load /path/to/new/repos"

Hey, you’re done! Isn’t the understated elegance of Unix grand?

Hello World

Sun, 08 Apr 2007 22:13:00 GMT

I plan on using this blog to record my musings about Ruby on Rails, humane web interfaces, and other interesting topics regarding the webotubosphere. Hopefully, I’ll even write a tutorial or two.

I’ll do my best to stay on topic and avoid writing about my cats or how my taste in music is better than yours. Aside from that giant, ego-stroking masthead with my name, I’ll try to keep it in the first person whenever practical.

This blog is powered by Simple Log with a little help from Haml & Sass. It’s about 80% done. I’m going to run with it and work out that last annoying 20% (print style sheet, search, actually look at it in IE 6…) as those issues present themselves.

If you’ve got something to share, feel free to hit the “contact” link at the bottom of the page at any time. Better yet, add a comment and let everyone join in the fun.

Happy coding.

John Guenin