cfis

DNC Event Map

Posted by Charlie Wed, 27 Aug 2008 18:55:00 GMT

One of the projects we've been working on for MapBuzz the last few weeks is building an interactive map that shows all the events going on in Denver during the Democratic National Convention.  Users can pick the event type and date they are interested in, and the map refreshes with icons for relevant events. By clicking on a given event, the user can see exactly where and when the event is taking place.  I think the map turned out pretty well - its a good example of mashup pulling data from different sources.   In this case, base maps from Google, event information from Zvents, and all rendering/styling/page from MapBuzz.

It did clarify my thinking on a few points.  First, Rails built-in page caching is really limited - it ignores query parameters and only works for html.   So we had to hack around that, more info coming in a later post.  Second, for building mashups xml really is superior to JSON simply because it supports namespaces (for all their pain points, namespaces really do facilitate merging of data from multiple sources).  Third, when you need it, xslt is invaluable.  Zvents serves its data using RSS, but our client only supports Atom. The simple solution was a quick xsl transformation to convert Zvent's rss feed over to Atom using libxslt (and thus MapBuzz's contribution back to the Ruby community to get the libxml and libxslt bindings back into good shape).

Posted in Design | 2 comments | no trackbacks

Resurrecting libxml-ruby

Posted by Charlie Wed, 16 Jul 2008 16:38:00 GMT

There is general discontent with the state of XML processing in Ruby - see for example here or here. An obvious solution is to use libxml. However that has been a non-starter since the libxml Ruby bindings have historically caused numerous segementation faults, don't run on Windows and recently lost their current maintainer, Dan Janowski. Making it even more frustrating is that Dan had spent the last year rearchitecting the bindings, successfully fixing the segmentation faults.

Since MapBuzz heavily depends on libxml, it seemed time to step in and contribute. Over the last two weeks I've added support for Windows, cleaned out the bug database and patch list, resolved the few remaining segmentation issues, greatly improved the RDocs and refactored large portions of the code base to conform with modern Ruby extension standards.

After iterating through a couple of releases over the last two weeks, the Ruby libxml community is happy to announce the availability of version 0.8.0, which we believe is ready for prime time. It offers a great combination of speed, functionality and conformance (libxml passes all 1800+ tests in the OASIS XML Tests Suite).

So give it a try - its as easy to install as:

gem install libxml-ruby

If you're on Windows there may be an extra step if you haven't already installed libxml2. If not, then the libxml-ruby distribution includes a prebuilt libxml2 dll in the libxml-ruby/mingw directory. Copy the dll to libxml-ruby/lib, your Ruby bin directory, or somewhere on your path (basically put it someplace where Windows can find it).

Undoubtedly there are still some bugs left, so please report anything you find, so we can fix them in future releases.

Blindingly Fast

The major reason people consider using libxml-ruby is performance. Here are the results from running (on my laptop) a few simple benchmarks that have recently been blogged about on the Web (you can find them in the benchmark directory of the libxml distribution).

From Zack Chandler:

              user     system      total        real
libxml    0.032000   0.000000   0.032000 (  0.031000)
Hpricot   0.640000   0.031000   0.671000 (  0.890000)
REXML     1.813000   0.047000   1.860000 (  2.031000)

From Stephen Bannasch:

              user     system      total        real
libxml    0.641000   0.031000   0.672000 (  0.672000)
hpricot   5.359000   0.062000   5.421000 (  5.516000)
rexml    22.859000   0.047000  22.906000 ( 23.203000)

From Andreas Meingast:

LIBXML THROUGHPUT:
	10.2570516817665 MB/s
	10.2570830340359 MB/s
	12.6992253283934 MB/s
  10.2570516817665 MB/s
	8.51116888387252 MB/s
	10.2570830340359 MB/s

HPRICOT THROUGHPUT:
	0.211597647822036 MB/s
	0.202390771964726 MB/s
	0.180272812529665 MB/s
	0.198474511420818 MB/s
	0.198474499681793 MB/s
  0.180925089981179 MB/s

REXML THROUGHPUT:
	0.130301425548982 MB/s
	0.131630590068325 MB/s
	0.128316078417727 MB/s
	0.125203555921636 MB/s
	0.120181872867636 MB/s
	0.115330940074107 MB/s

I can't vouch for the appropriateness of the tests, but they show libxml clocking in at 10x hpricot and 30x to 60x REXML. I'd be happy to accept additional tests or more appropriate tests if you have any.

An Embarrassment of Riches

In addition to performance, the libxml-ruby bindings provide impressive coverage of libxml's functionality. Goodies include:

• SAX
• DOM
• XMLReader (streaming interface)
• XPath
• XPointer
• XML Schema
• DTDs
• XSLT (split into the libxslt-ruby bindings)

Now, your first reaction might be that SAX, DOM and XPath are all you need, but validating parsers make it a whole lot easier to sanitize user contributed content on web sites. And the XMLReader offers a clever way of combining the DOM's ease of use (well, ok, compared to SAX at least) with SAX's memory and speed advantages.

Better yet, most of this functionality is exposed via an easy-to-use, Ruby like API. There are still of course some warts lurking in the code, where libxml's C api leaks through to Ruby, but they are being removed one by one. And for those of you who aren't C hackers, much of this work can be done in good old Ruby.

A Long History

For such a useful, and full-featured library, the libxml-ruby bindings have a star-crossed history. Out of curiosity, I went back and traced their lineage. Sean Chittenden originally wrote them back in 2002. At the start of 2005, Trans Onoma adopted the project after Sean had moved on, and at the end of 2005 the bindings found their current home on Ruby Forge. At that point Ross Bamford took over maintenance and worked on the bindings for roughly a year, until early 2007, when then the bindings again became unmaintained. Dan Janowski picked up the ball in 2007 and completely overhauled the binding's memory model. Sadly, Dan had to give up active support this spring.

But on the bright side, Trans, Dan and Sean are all once again active on the mailing list, providing valuable experience and insight. From my point of view, with the renewed push towards a production quality release, and bringing in new users, the libxml-ruby community is as healthy as it has been in a long while.

Posted in Ruby | 28 comments | 2 trackbacks

A New Take on Transparent PNGs in IE6 - Performance and VML

Posted by Charlie Mon, 23 Jun 2008 01:51:00 GMT

The web is full of articles discussing how to render transparent 24-bit png images in IE6 - so why write another one?

Three reasons. First, although the long hoped for demise of IE6 is finally showing some progress, IE6 still has a 20% to 40% market share, which is still more than Firefox. Second, none of the existing articles talk about the terrible performance degradations caused by the most common solutions proposed to display png files and how to avoid them. Third, I discovered an alternate solution using VML that I've never seen documented.

AlphaImageLoader is Slow

Let's start with the performance issues. Most websites recommend enabling transparent png's in IE6 by running javascript code when the page loads. The javascript finds all png images on a page and applies Microsoft's proprietary AlphaImageLoader filter.

What almost no article mentions (the one I linked to above being an exception) is how badly this degrades page load performance. But don't take my word for it - I've setup an example page that loads sea-level rise data from Peter Black's excellent climate atlas blog.

At the top of the page are several buttons - click the far left one, titled Slow, to see how the typical png solution performs. Go ahead, try it now (using IE6 of course). Notice how the browser freezes for over 10 seconds and doesn't display any images until the very end? Not very good, is it?

But if you view the same files using Google maps, the performance is much better. And you can also see each tile as it loads, instead of waiting for the end. So how did Google achieve this sleight of hand? A bit of digging shows the trick - instead of fixing all the images at once, Google fixes them one at a time. This is done by attaching an onload event handler to each image that needs to be fixed. When the image is loaded, the onload event applies the AlphaImageLoader filter. This avoids IE freezing. And to avoid the annoying flashing caused by applying the filter, images start off "hidden" and are only made visible when their onload events are fired. Clever, isn't it?

Now go back to the example page and click the second button, titled Fast. The difference is amazing - like night and day. As an extra bonus, the onload event for image elements that point to invalid or non-existent images never fires, meaning that they are never made visible. That neat trick avoids IE displaying an annoying red x () for invalid images.

A New Approach - VML

A couple of years ago I discovered an entirely new way of displaying 24-bit transparent png images in IE - use VML. Somehow I've never managed to put it down on paper (wrong metaphor I know, but humor me). I figure I'd better do it now before this tidbit of knowledge becomes irrelevant.

It's little known that VML supports not only vector graphics, but also raster images via its image element. And I've never seen it mentioned that the vml:image element supports transparent 24-bit pngs.

To see VML in action, go back to the example page and click the third button, titled VML. The first time you click the button, the performance won't be very good since all the images are first downloaded. But once the page is loaded, click the Clear button and then click the VML button. From my testing, the performance is noticeably better than using the AlphaImageLoader.

There are several things to note about the VML code that took me hours to figure out:

• The setupVml function enables VML via Javascript - its two lines of code are barely documented on the web and thus took hours of fiddling to get working
• You must set the width and height of the VML images, otherwise nothing is displayed
• You must set the coordsize of the VML images - otherwise they will be randomly one pixel to short or wide

Sadly, the VML solution suffers three problems that make in unsuitable most of the time. First, notice the images are a bit fuzzy when displayed with VML. I haven't a clue as to why.

Second, images that don't load are shown with IE's annoying red x mentioned above. The problem is that VML images don't seem to support the onload event (or onreadystate), so there is now way to start them off as hidden and then make them visible once the image has loaded.

Third, IE6 doesn't cache VML images across page loads. To see this, reload the example page and press the VML button again. Notice the long delay? If you watch IE's http traffic (say using Fiddler), you'll see that IE6 requests each image again. It does not do that for html image elements, which you can verify by running the same experiment but clicking the Fast button instead.

Together, these three issues make the VML solution inferior to the AlphaImageLoader solution, but I thought I'd write it down in case someone ever needs to know about it.

Posted in Design, Web | no comments | no trackbacks

Rails - Simple Asynchronous Processing

Posted by Charlie Tue, 17 Jun 2008 05:55:00 GMT

Sooner or later, for most large websites you have to bite the bullet and implement some form of asynchronous processing to deal with long-running tasks. For example, with MapBuzz we have a several long-running tasks:

• Importing data
• Batch geodcoding
• Emailing event notifications to users

If you're developing a Facebook application, moving long-running tasks to a background process or thread is critical since Facebook times out requests to your server within ten to twelve seconds.

So Many Choices

Having decided you need asynchronous processing, the next question is how to do it. And this is where things get complicated - there are a myriad of approaches, each applicable for certain problem domains. Let's look at some possibilities:

• Spawn/Fork - Create processes on demand to perform background tasks
• Distributed Objects - Use a distributed object protocol (RMI, Corba, DCOM, DrB, etc) to communicate with another process to perform background tasks
• Job Queue - Persist tasks in shared files or databases and execute them using background processes
• Messaging Processing - Send messages to another process via a message bus

In the Ruby world, there are a number of implementations for each approach - a few examples include:

• Spawn/Fork - Spawn
• Distributed Objects - BackgroundDRb
• Job Queue - Delayed Job (DJ), BackgroundJob (BJ), BackgroundFu, Starling, Beanstalkd, Sparrow
• Message Sending - AP4R

Not surprisingly, most of these solutions are designed to work with Rails, since there's no need to speed up processing if its just another machine on the other end instead of an impatient human.

Selecting the best one for your application is totally dependent on your use cases. Having said that, its still possible to reach some broad conclusions. Spawning or forking processes makes it impossible to offload processing to additional machines, so you'll quickly run into scalability limits. Distributed objects solve that problem, but experience has shown distributed object protocols are very brittle because they bind clients and servers so tightly together - thus I would never use them. Job queues are more reliable because tasks are represented in a standard format (usually text based, such as xml) that is persisted to files or database tables. Message queues are similar, but add significantly more functionality such as message routing, transformation, prioritization, etc.

For many websites, a job queue is the best solution. Job queues are relatively light weight and let you distribute processing across multiple machines. However, the ruby based solutions listed above require installing and managing additional software as well as writing the job processing code itself. They also make it more difficult to develop and test software since you know have to debug multiple processes at once.

A Simple HTTP Based Solution

So what's a simpler solution? Reuse what you already have. Most Rails applications are divided into multiple instances, distributed across one or more machines, that embed an http server (mongrel, thin, ebb) for requests. Thus we already have our background processes and an easy way to communicate with them - http (of course!). And if your using mongrel or a proxy server (Pound, Lighttpd, Nginx, Apache, etc.), then you also get a built-in request queue.

In other words:

simple background queue = HTTP + Load Balancer + Rails instances 

Besides simplicity, a big advantage to this approach is that background tasks run within the Rails environment, giving you access to ActiveRecord, your models, etc.

Worker Plugin

Thus enters a new Rails plugin called worker (yeah the name leaves something to be desired). Let's look at an example:

class ImportController < ApplicationController
  # Add support for using workers
  include Worker

  # Incoming requests are handled by this method
  resource :Geodata do
    def post
      read_file(params)
    end
  end
  
  # This method handles requests in a worker process
  resource :process do
    def post
    end
  end

  private 
  
  def read_file(params)
    worker_params = {:file_name => file_name, 
                     :tags => params['tags'],
                     :controller => 'import',
                     :resource => 'process',
                     :map_id => @map.id}

    # Create worker request
    create_worker.post(worker_params)
  end
end

So how does this work? A user POSTs a file to http://myserver/import/geodata. That method does various checks (deleted for brevity) and then sends a request to http://myserver/import/process which runs in a separate Rails instance. Although this controller delegates back to itself (in a separate process) it could call any controller it wishes.

The worker plugin will pass a session key, if available, to the background process. That turns out to be very useful since it allows sharing session information between the foreground process and background process if you're storing session information since in memcached or a background database. That means you can use the same authentication and authorization mechanisms in the background process as you do in the foreground process.

In addition, all worker requests are signed with a MD5 hash to verify that no-one in the middle is spoofing requests.

Environments and Configuration

By default, Rails applications use three environments - testing, development and production. Each environment is quite different, which affects how you want to use worker processes. To deal with these differences, the worker plugin uses a strategy pattern to invoke requests.

In a test environment, there are no background running Rails instances. More importantly, you need to be able to check that worker requests correctly complete. Thus you want worker requests to happen synchronously and within the test process. This is the Worker::Controller strategy, and works similarly to how Rails render_component functionality works. To set this up, add the following lines to your test environment file:

config.after_initialize do
  Worker::Config.strategy = Worker::Controller
end 

In development mode, you have one Rails instance running. In this case, you want worker requests to happen asynchronously but within the single development process. This is the Worker::HttpAsync strategy. To set this up, add the following lines to your development environment file:

config.after_initialize do
  Worker::Config.strategy = Worker::HttpAsync
end 

Note this assumes that your development process is running on the standard port 3000.

Finally, in production mode, you'll have multiple Rails instances running. To be on the safe side, some of these instances should be dedicated to only fulfilling worker requests. The easiest way to do this is put them on an internally accessible IP address, say 8500, that outsides cannot access. Thus the port, and perhaps IP address, of the user-facing Rails instances will be different than worker instances. To set this up, add an additional line to your config file that globally sets the host and port number of workers. Note this assumes that there is either one worker or a pool or workers at the given host and port.

config.after_initialize do
  Worker::Config.strategy = Worker::HttpAsync 
  Worker::HttpAsync.options = {:host => 'some_other_host',
                               :port => '8500'}
end 

The Code

We're releasing the worker plugin under an MIT license. If there is sufficient interest, we're happy to setup a RubyForge project.

Read more...

Posted in Design, Rails | 15 comments | 2 trackbacks

Facebook Follies - Setting the Stage

Posted by Charlie Thu, 10 Apr 2008 23:38:00 GMT

Hearing about the latest and greatest Facebook application reminds me of reading FastCompany or Business 2.0, before it went bust. The typical story goes like this - a couple of ninja developers spend every waking moment for a year coding up the Web's next killer site. Launched with little fanfare, the website quickly goes viral, and starts to generate huge amounts of traffic. In no time, the company is worth millions of dollars (with our without revenue is optional), and everyone is rich and happy.

The rise of Facebook hyper-charges this narrative. Facebook has dramatically compressed the time between an idea and fortune and fame. We've left "web years" in the dust by moving into the parallel universe of "Facebook time." This go-round, the prototypical story goes like this - Stanford student whips up a Facebook app over the weekend and releases it on Monday. By the end of the week it has half a million users and by the end of the month the student flips the application, neatly paying for this year's tuition.

Just as some of the women you see on the covers of women's magazines really do exist, some of these stories are surely true. But for the rest of us, success is not so easy. It usually comes the way it has always come, through insight, perseverance, hard work and a bit of luck.

Which gets us to the point of this blog - telling the technical story behind the development of MapNotes - MapBuzz's first Facebook application. MapNotes is a simple application - it makes it easy to put PostIt notes on a map and share them with friends.

Our naive thinking behind MapNotes was too quickly develop a Facebook application to get familiar with the platform, and then roll out a series of applications quickly thereafter. We soon discovered that a number of technical gremlins lay behind MapNote's apparent simplicity. Above all was our desire to deeply integrate MapBuzz with Facebook, so that what users do on MapBuzz is available on Facebook and what users do on Facebook is available on MapBuzz.

Although there is plenty of information on the Web about developing one-off Facebook application, precious little exists describing the challenges of integrating a destination website with Facebook and how to support multiple facebook applications.

To share what we've learned, I'm kicking off a new series of posts over the coming weeks that will dive deep into the technical details of developing for Facebook. I'll link to the articles from this page, so consider it both an introduction and Table of Contents rolled into one. Some things I'll cover are:

• Login and installation
• Session management
• Account and link management
• Asynchronous processing
• Notifications and newsfeeds
• Using Ajax

If you're more into usability issues and nice looking screenshots, then head over to the MapBuzz blog for more info. Enjoy!

Posted in MapBuzz, Facebook | no comments | no trackbacks

Daylight Savings Time

Posted by Charlie Sun, 30 Mar 2008 19:06:00 GMT

Its a funny thing when things that you wish would happen do happen, but then you realize they aren't so great after all. Take for example the change this year in daylight savings time. Not being a morning person, I love daylight savings time - it means an extra hour of sun every day. And if it was up to me, I would abolish standard time.

But now that daylight savings time has been moved up a month, I've found it disappointing. Why? Because is still winter in Denver. The extra hour of sunlight a day is just a tease - its still chilly outside and its still snowing. At least when its dark and cold outside I don't feel like I'm missing out by being locked up inside working.

Posted in Colorado | no comments | no trackbacks

Reusing Rails Controllers is a Good Thing

Posted by Charlie Mon, 11 Feb 2008 18:11:00 GMT

If there were such a thing as the Ten Commandments of programming, code reuse would surely be included. Now you're probably thinking I've lost my mind, because any good developer knows that code reuse is a pipe dream. But that's because you are thinking on the macro level and not the micro level. At the micro level, code reuse has altogether more pleasant acronym, DRY, or do not repeat yourself.

The tussle in the Rails community over components illustrated this tension well. On one extreme, advocates dreamed of creating plug-and-play components that could be reused across multiple applications. On the other extreme, detractors sneered at code reuse as a hopeless endeavor and vowed to strip out all traces of components from the the Rails 2.0 release. Left out in the cold was the view that reusing code, specifically controllers, within a single application was a good thing.

Reusing Controllers

Rails is built using the Model-View-Controller pattern, which is designed to segment an application into controllers, models and views. Rails also adds in the concept of filters, which are pieces of code that run before and after controllers.

The Rails community encourages reuse of models, filters and views, but seems to actively discourage the reuse of controllers if the Ruby on Rails book is any indication:

When Rails was initially released, it came with a system for creating components. Unfortunately, the implementation of components left a lot to be desired: performance was poor, and there were unanticipated side effects. As a result,
components are being phased out.

Instead, the common wisdom now is to synthesize component-like functionality using a combination of before filters and partials. Use the before filter to set up the context for the partial, and then render the fragment you want using a regular render :partial call.

Like much conventional wisdom, this advice is hogwash.

Filters + Partials != Controllers

The problem with just using filters and partials is that it only applies to a subset of web applications. A good example, and of course the one used in the Ruby on Rails book, is a shopping website. The focal point of most shopping websites is a shopping cart. The point of the application is to make it easy for users to add things to the cart, modify the cart and hopefully buy the contents of the cart. Since the cart plays such a crucial role, it often make sense to have a filter that setups the cart so each controller has easy access to it. A nice side affect of this approach, is that views also have access to the cart, as described in the quote above.

But for many other types of applications, filters and partials can't make up for controllers. For example, take a look at the Boulder community on MapBuzz. The top-left side of the page is rendered by the community controller, the comments on the bottom left by a comment controller and the map listings on the right are by a map browser controller. If you log-in, then a couple of additional tabs are added to the page, each rendered by its own controller.

This type of composition is quite common in Web 2.0 applications. Take most social networking sites - they'll mix together news feeds, discussion boards, friends/friends lists, pictures,etc., in a variety of different ways depending on the current page.

The design problem is that any given controller can be called in two different contexts:

• When the whole page is rendered via a Browser page refresh
• When just the controller is rendered via an Ajax call

Trying to do this with just filters and helpers is a non-starter, because you end up with one big controller that needs to run different filters depending on the context of the call.

The better approach is to divide your controllers into logical units, and then have a separate page controller for the entire page. When the page is rendered, the page controllers should delegate rendering the various sub-parts (for example, tabs) of the page to the appropriate controller. When just one of the sub-parts of the page needs to be rerendered, due to an Ajax call, then you directly call the appropriate controller.

Performance

In Rails, a controller or view can call another controller using the much maligned render_component method. Part of the problem is the method is misnamed. It no longer has anything to do with rendering components - instead its used to invoke another controller. Therefore, it would be more appropriately named render_controller, call_controller, invoke_controller, etc.

Assuming you agree with my so far, reading the Rails documentation for render_component with certainly give you pause:

Components should be used with care. They‘re significantly slower than simply splitting reusable parts into partials and conceptually more complicated. Don‘t use components as a way of separating concerns inside a single application. Instead, reserve components to those rare cases where you truly have reusable view and controller elements that can be employed across many applications at once.

So to repeat: Components are a special-purpose approach that can often be replaced with better use of partials and filters.

Undoubtedly this was true once upon a time. Is it still? There is one way to find out - run a test. I created a new Rails application using the built-in generators and then added the following simple code:

controller/main_controller.rb

class MainController < ApplicationController
  def get_without_controller
    a = 1
  end
  
  def get_with_controller
    a = 1
  end
end

controller/sidebar_controller.rb

class SidebarController < ApplicationController
  def get
    render(:partial => 'sidebar/content')
  end
end

views/main/get_without_controller.html.erb

<p>Some fun content goes here</p>
<div class="sidebar">
  <%= render(:partial => 'sidebar/content') %>
</div>

views/main/get_with_controller.html.erb

<p>Some fun content goes here</p>
<div class="sidebar">
  <%= render_component(:controller => SidebarController,
                       :action => 'get') %>
</div>

views/sidebar/_content.html.erb

<p>Hi there</p>

There are two paths through this application:

• GET '/main/get_without_controller.rb'
• GET '/main/get_with_controller.rb'

In case its not obvious, get_without_controller usesrender(:partial) to include the sidebar content while get_with_controller uses render_component. Using both benchmark and ruby-prof, I ran each method 100 times using a souped up integration test (more about that in a future post). The results, using Rails 2.02 on Ruby 1.8.4 on WindowsXP on a Pentium M laptop (about 3 years old) are:

Method	100 Requests (s)	1 Request (s)
get_without_controller	0.30	0.0030
get_with_controller	0.45	0.0045

 

So using components is 50% slower, but the overhead is a miniscule 0.0015 seconds per request. That overhead is obviously lost in a real application. Of course you have to be careful when using render_component to not try and do to much per HTTP request - but the same is true using filters and partials.

DRY Up Your Controllers

In truth, render_component is the most primitive way imaginable of reusing controllers. But it does let you to DRY up your Rails application by letting you create more cohesive controllers that can be reused within a single website. For most websites you won't need this functionality, but when you do, there isn't a substitute for it and don't let anyone browbeat you into thinking there is.

Posted in Design, Rails | 13 comments | 2 trackbacks

Rails and Postgresql - Eliminate Hundreds of Thousands of Queries a Day

Posted by Charlie Fri, 08 Feb 2008 18:16:00 GMT

Update - It turns out that Rails does cache column data dictionary queries (which is what you would expect), but not for :has_and_belongs_to_many associations (HABTMA). I know those are "old fashioned," but they fit our data model perfectly in a couple of places. So be warned - using just a couple of HABTMA associations will generate a huge number of data dictionary queries.

As part of monitoring the performance of MapBuzz, we run a nifty little program called PgFouine to analyze the postgresql log files every night. PgFouine summarizes the most common queries and slowest queries. Here is our data from yesterday:

Most frequent queries (N)

Rank	Times executed	Total duration	Av. duration (s)	Query
1	140,115	2m42s	0.00	SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum WHERE a.attrelid = ''::regclass AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum;

Besides being the most run query, this was also the seventh most time consuming query.

I was shocked the first time I saw this many months ago - my first guess was that we were not caching classes in our production mode. But it turned out that wasn't the problem. Its just Rails being silly - every time it loops over a model's columns it strikes up a conversation with the database. That happens a fair bit - when you use :include to add additional tables to a query, when you use dynamic finders (find_by_x), when you use relationships setup by :has_many, :has_and_belongs_to, etc. And this isn't the only place Rails is wasteful - it constantly queries the database for table names and indices - it just happens those queries don't run nearly as often.

Rails Plugin

Clearly there is no reason to do this in a production environment, and in truth, I don't see much reason to do it in a development environment either. So yesterday I finally go around to patching Rails and submitting a bug report. The patch caches data dictionary queries for the Postgresql adapter. After loading the patch, Rails still supports the ability to add tables to your database at runtime, but no longer supports adding or removing columns from a table at runtime or recycling table names. If these things are important to you, the patch also provide a flush_dd_cache method that flushes the query cache. An obvious alternative solution is to add a cache_dd_info class variable to ActiveRecord::Base, which would be off by default but on in production. However, I'm skeptical there is a need for such a flag.

On a per request basis, you won't see much performance gain from the patch in your Rails application servers, but it will remove needless load from your database. And while you are waiting for Rails to be patched (if it is patched), feel free to download the Rails plugin we are using to solve the problem. Note the plugin also fixes two other ActiveRecord bugs, which are its incorrect handling of Postgresql schemas and its ignoring of views.

Posted in Rails | 5 comments | no trackbacks

Fun With Here Documents

Posted by Charlie Thu, 07 Feb 2008 21:07:00 GMT

Today I was fixing up some inefficiencies in Rail's Postgresql adapter (more about that in a later post), and came across this strange looking code:

def tables(name = nil)
  schemas = schema_search_path.split(/,/).map { |p| quote(p) }.join(',')
  query(<<-SQL, name).map { |row| row[0] }
        SELECT tablename
        FROM pg_tables
        WHERE schemaname IN (#{schemas})
        SQL
end

The code is passing a here document, which is basically a long string, as a parameter to the query method. The here document is demarked by the string "SQL."

What took me by surprise is the here document is defined after the method call, which is a code construct I've never seen before. Making sure my memory wasn't failing me, I double checked my copy of The Pick Axe book and verified it never mentions this feature.

Hats off to Jamis, who added this neat little trick to Rails in revision 2317, way back in September 2005, and to Anthony who blogged about it last month. And finally, the Ruby Wikibook has a great tutorial about here documents. Who knew that you can have multiple here document parameters per method call?

Posted in Ruby | 3 comments | no trackbacks

ruby-prof 0.6.0 and Memory Profiling

Posted by Charlie Sun, 03 Feb 2008 22:57:00 GMT

Shugo and I are happy to announce the release of ruby-prof 0.6.0. If you haven't used ruby-prof, its a superfast, open-source, profiler for Ruby that shows you where your program is slow.

The big news is that this release, thanks to Shugo, supports Ruby 1.9. And there is talk about merging ruby-prof into Ruby itself, but we'll have to wait and see if that really happens.

This release also includes experimental support for memory profiling, added by Alexander Dymo, which I'll talk about in more detail below. And of course it includes a number of bug fixes, almost all of which were reported and fixed by the community. Special thanks goes out to Sylvain Joyeux, Michael Granger, Makoto Kuwata and Dan Fitch.

The best way to get started with ruby-prof is to look at the readme file which is online at RubyForge.

Memory Profiling

The 0.5.0 release introduced the concept of measurement modes, which offer a way of extending ruby-prof to track all sorts of metrics about a running program, including time, object allocations, memory usage, etc. Developers can implement custom measurement modes by simply implementing a measurement function that tracks the metric of interest, and ruby-prof takes care of all the rest.

The idea was inspired by Sylvain Joyeux, who used it in the 0.5.0 release to track object allocations in a running program. In this release, Alexander Dymo added support for tracking memory allocations as I discussed in my last post.

The only disadvantage of these two measurement modes is that they require patched ruby interpreters, making them unavailable to most ruby developers. Tracking memory usage is particularly important, and there has been some great work done by Evan Weaver with BleakHouse, Erick Hodel with mem_inspect and Alexander Dymo. I'm not partial to any of these approaches, outside the fact they should be implemented in C for performance reasons. But I would like to see a single solution which could be merged into Ruby itself, opening it up to all Ruby developers.

I've asked Alexander to start up a discussion on ruby-core, so we'll see where it leads.

1 comment | no trackbacks