Posted by Charlie
Mon, 11 Feb 2008 18:11:00 GMT
If there were such a thing as the Ten Commandments of programming, code reuse would surely be included. Now you're probably thinking I've lost my mind, because any good developer knows that code reuse is a pipe dream. But that's because you are thinking on the macro level and not the micro level. At the micro level, code reuse has altogether more pleasant acronym, DRY, or do not repeat yourself.
The tussle in the Rails community over components illustrated this tension well. On one extreme, advocates dreamed of creating plug-and-play components that could be reused across multiple applications. On the other extreme, detractors sneered at code reuse as a hopeless endeavor and vowed to strip out all traces of components from the the Rails 2.0 release. Left out in the cold was the view that reusing code, specifically controllers, within a single application was a good thing.
Reusing Controllers
Rails is built using the Model-View-Controller pattern, which is designed to segment an application into controllers, models and views. Rails also adds in the concept of filters, which are pieces of code that run before and after controllers.
The Rails community encourages reuse of models, filters and views, but seems to actively discourage the reuse of controllers if the Ruby on Rails book is any indication:
When Rails was initially released, it came with a system for creating components. Unfortunately, the implementation of components left a lot to be desired: performance
was poor, and there were unanticipated side effects. As a result,
components are being phased out.
Instead, the common wisdom now is to synthesize component-like functionality
using a combination of before filters and partials. Use the before filter to set up the context for the partial, and then render the fragment you want using
a regular render :partial call.
Like much conventional wisdom, this advice is hogwash.
Filters + Partials != Controllers
The problem with just using filters and partials is that it only applies to a subset of web applications. A good example, and of course the one used in the Ruby on Rails book, is a shopping website. The focal point of most shopping websites is a shopping cart. The point of the application is to make it easy for users to add things to the cart, modify the cart and hopefully buy the contents of the cart. Since the cart plays such a crucial role, it often make sense to have a filter that setups the cart so each controller has easy access to it. A nice side affect of this approach, is that views also have access to the cart, as described in the quote above.
But for many other types of applications, filters and partials can't make up for controllers. For example, take a look at the Boulder community on MapBuzz. The top-left side of the page is rendered by the community controller, the comments on the bottom left by a comment controller and the map listings on the right are by a map browser controller. If you log-in, then a couple of additional tabs are added to the page, each rendered by its own controller.
This type of composition is quite common in Web 2.0 applications. Take most social networking sites - they'll mix together news feeds, discussion boards, friends/friends lists, pictures,etc., in a variety of different ways depending on the current page.
The design problem is that any given controller can be called in two different contexts:
- When the whole page is rendered via a Browser page refresh
- When just the controller is rendered via an Ajax call
Trying to do this with just filters and helpers is a non-starter, because you end up with one big controller that needs to run different filters depending on the context of the call.
The better approach is to divide your controllers into logical units, and then have a separate page controller for the entire page. When the page is rendered, the page controllers should delegate rendering the various sub-parts (for example, tabs) of the page to the appropriate controller. When just one of the sub-parts of the page needs to be rerendered, due to an Ajax call, then you directly call the appropriate controller.
Performance
In Rails, a controller or view can call another controller using the much maligned render_component method. Part of the problem is the method is misnamed. It no longer has anything to do with rendering components - instead its used to invoke another controller. Therefore, it would be more appropriately named render_controller, call_controller, invoke_controller, etc.
Assuming you agree with my so far, reading the Rails documentation for render_component with certainly give you pause:
Components should be used with care. They‘re significantly slower than simply splitting reusable parts into partials and conceptually more complicated. Don‘t use components as a way of separating concerns inside a single application. Instead, reserve components to those rare cases where you truly have reusable view and controller elements that can be employed across many applications at once.
So to repeat: Components are a special-purpose approach that can often be replaced with better use of partials and filters.
Undoubtedly this was true once upon a time. Is it still? There is one way to find out - run a test. I created a new Rails application using the built-in generators and then added the following simple code:
controller/main_controller.rb
class MainController < ApplicationController
def get_without_controller
a = 1
end
def get_with_controller
a = 1
end
end
controller/sidebar_controller.rb
class SidebarController < ApplicationController
def get
render(:partial => 'sidebar/content')
end
end
views/main/get_without_controller.html.erb
<p>Some fun content goes here</p>
<div class="sidebar">
<%= render(:partial => 'sidebar/content') %>
</div>
views/main/get_with_controller.html.erb
<p>Some fun content goes here</p>
<div class="sidebar">
<%= render_component(:controller => SidebarController,
:action => 'get') %>
</div>
views/sidebar/_content.html.erb
<p>Hi there</p>
There are two paths through this application:
- GET '/main/get_without_controller.rb'
- GET '/main/get_with_controller.rb'
In case its not obvious, get_without_controller usesrender(:partial) to include the sidebar content while get_with_controller uses render_component. Using both benchmark and ruby-prof, I ran each method 100 times using a souped up integration test (more about that in a future post). The results, using Rails 2.02 on Ruby 1.8.4 on WindowsXP on a Pentium M laptop (about 3 years old) are:
| Method |
100 Requests (s) |
1 Request (s) |
| get_without_controller |
0.30 |
0.0030 |
| get_with_controller |
0.45 |
0.0045 |
So using components is 50% slower, but the overhead is a miniscule 0.0015 seconds per request. That overhead is obviously lost in a real application. Of course you have to be careful when using render_component to not try and do to much per HTTP request - but the same is true using filters and partials.
DRY Up Your Controllers
In truth, render_component is the most primitive way imaginable of reusing controllers. But it does let you to DRY up your Rails application by letting you create more cohesive controllers that can be reused within a single website. For most websites you won't need this functionality, but when you do, there isn't a substitute for it and don't let anyone browbeat you into thinking there is.
Posted in Design, Rails | 13 comments | 2 trackbacks
Posted by Charlie
Fri, 08 Feb 2008 18:16:00 GMT
Update - It turns out that Rails does cache column data dictionary queries (which is what you would expect), but not for :has_and_belongs_to_many associations (HABTMA). I know those are "old fashioned," but they fit our data model perfectly in a couple of places. So be warned - using just a couple of HABTMA associations will generate a huge number of data dictionary queries.
As part of monitoring the performance of MapBuzz, we run a nifty little program called PgFouine to analyze the postgresql log files every night. PgFouine summarizes the most common queries and slowest queries. Here is our data from yesterday:
Most frequent queries (N)
| Rank |
Times executed |
Total duration |
Av. duration (s) |
Query |
| 1 |
140,115 |
2m42s |
0.00 |
SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull
FROM pg_attribute a
LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attrelid = ''::regclass AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum; |
Besides being the most run query, this was also the seventh most time consuming query.
I was shocked the first time I saw this many months ago - my first guess was that we were not caching classes in our production mode. But it turned out that wasn't the problem. Its just Rails being silly - every time it loops over a model's columns it strikes up a conversation with the database. That happens a fair bit - when you use :include to add additional tables to a query, when you use dynamic finders (find_by_x), when you use relationships setup by :has_many, :has_and_belongs_to, etc. And this isn't the only place Rails is wasteful - it constantly queries the database for table names and indices - it just happens those queries don't run nearly as often.
Rails Plugin
Clearly there is no reason to do this in a production environment, and in truth, I don't see much reason to do it in a development environment either. So yesterday I finally go around to patching Rails and submitting a bug report. The patch caches data dictionary queries for the Postgresql adapter. After loading the patch, Rails still supports the ability to add tables to your database at runtime, but no longer supports adding or removing columns from a table at runtime or recycling table names. If these things are important to you, the patch also provide a flush_dd_cache method that flushes the query cache. An obvious alternative solution is to add a cache_dd_info class variable to ActiveRecord::Base, which would be off by default but on in production. However, I'm skeptical there is a need for such a flag.
On a per request basis, you won't see much performance gain from the patch in your Rails application servers, but it will remove needless load from your database. And while you are waiting for Rails to be patched (if it is patched), feel free to download the Rails plugin we are using to solve the problem. Note the plugin also fixes two other ActiveRecord bugs, which are its incorrect handling of Postgresql schemas and its ignoring of views.
Posted in Rails | 5 comments | no trackbacks
Posted by Charlie
Thu, 07 Feb 2008 21:07:00 GMT
Today I was fixing up some inefficiencies in Rail's Postgresql adapter (more about that in a later post), and came across this strange looking code:
def tables(name = nil)
schemas = schema_search_path.split(/,/).map { |p| quote(p) }.join(',')
query(<<-SQL, name).map { |row| row[0] }
SELECT tablename
FROM pg_tables
WHERE schemaname IN (#{schemas})
SQL
end
The code is passing a here document, which is basically a long string, as a parameter to the query method. The here document is demarked by the string "SQL."
What took me by surprise is the here document is defined after the method call, which is a code construct I've never seen before. Making sure my memory wasn't failing me, I double checked my copy of The Pick Axe book and verified it never mentions this feature.
Hats off to Jamis, who added this neat little trick to Rails in revision 2317, way back in September 2005, and to Anthony who blogged about it last month. And finally, the Ruby Wikibook has a great tutorial about here documents. Who knew that you can have multiple here document parameters per method call?
Posted in Ruby | 3 comments | no trackbacks
Posted by Charlie
Sun, 03 Feb 2008 22:57:00 GMT
Shugo and I are happy to announce the release of ruby-prof 0.6.0. If you haven't used ruby-prof, its a superfast, open-source, profiler for Ruby that shows you where your program is slow.
The big news is that this release, thanks to Shugo, supports Ruby 1.9. And there is talk about merging ruby-prof into Ruby itself, but we'll have to wait and see if that really happens.
This release also includes experimental support for memory profiling, added by Alexander Dymo, which I'll talk about in more detail below. And of course it includes a number of bug fixes, almost all of which were reported and fixed by the community. Special thanks goes out to Sylvain Joyeux, Michael Granger, Makoto Kuwata and Dan Fitch.
The best way to get started with ruby-prof is to look at the readme file which is online at RubyForge.
Memory Profiling
The 0.5.0 release introduced the concept of measurement modes, which offer a way of extending ruby-prof to track all sorts of metrics about a running program, including time, object allocations, memory usage, etc. Developers can implement custom measurement modes by simply implementing a measurement function that tracks the metric of interest, and ruby-prof takes care of all the rest.
The idea was inspired by Sylvain Joyeux, who used it in the 0.5.0 release to track object allocations in a running program. In this release, Alexander Dymo added support for tracking memory allocations as I discussed in my last post.
The only disadvantage of these two measurement modes is that they require patched ruby interpreters, making them unavailable to most ruby developers. Tracking memory usage is particularly important, and there has been some great work done by Evan Weaver with BleakHouse, Erick Hodel with mem_inspect and Alexander Dymo. I'm not partial to any of these approaches, outside the fact they should be implemented in C for performance reasons. But I would like to see a single solution which could be merged into Ruby itself, opening it up to all Ruby developers.
I've asked Alexander to start up a discussion on ruby-core, so we'll see where it leads.
1 comment | no trackbacks
Posted by Charlie
Sat, 02 Feb 2008 22:58:00 GMT
A couple of days ago, Alex Dymo from Pluron sent me an email describing some of the great work he has done optimizing the performance of their online project managment software Accunote. His great insight was that their performance problems were caused by allocating too much memory, thus forcing Ruby's Garbage Collector to frequently run ruining performance.
Using a patched version of ruby and ruby-prof, Alex was able to more than double performance (with hints of more to come) and reduced memory consumption by 75%, or 750MB (yes - that is Megabytes). Alex does a wonderful job of documenting his approach with a series of blog posts here and here.
The main culprit was Rail's handling of attributes, which is dreadfully designed (an obvious case of the simplest solution to a problem is the wrong solution - something I've been meaning to blog about for almost a year now ). But he also implicated Ruby's built-in benchmarking module.
Even better, Alex provided patches to ruby core (already accepted), Rails (already accepted) and ruby-prof. We'll also gladly accept his patches, and since its about time for a ruby-prof refresh, we'll spin out a new release as soon as we can. More to follow.
Posted in ruby-prof | 5 comments | no trackbacks
Posted by Charlie
Fri, 01 Feb 2008 08:22:00 GMT
Maybe its just me, but what I want from a JavaScript library seems to be diverging from what Prototype provides. What I want, in order of importance, is:
- A cross-browser API that hides some of the major differences between Internet Explorer and standards compliant browsers
- Unobtrusive
- As small as possible, and if that's not possible, then at least modular
- A selector API
Where Prototype really falls down is on points two and three - its very obtrusive, getting larger by the day and isn't modular.
Accepting Something for What It Is
Prototype's greatest sin is its disdain for JavaScript.You can see this disdain shine through in a number of ways.
First, Prototype originated as part of Rails, which provides helpers that use Ruby code to generate JavaScript. If programs could talk, Rails would be saying "Let me take care of this for you since you certainly don't want to dirty your hands with JavaScript."
Second, Prototype wastes over 200 lines of code (about 5%) duplicating Ruby's Enumerable API in JavaScript, for no obvious reason except the developers prefer Ruby's way of doing things. The problem is that Ruby's Enumerable API is based on one of the core features of Ruby - its elegant use of anonymous functions (called blocks) to apply snippets of code to a sequence of items. JavaScript has first-class anonymous functions, but it doesn't have the language support for using them as iterators. As a result, Prototype's JavaScript code doesn't look natural because it is working outside the design strengths of JavaScript. And more importantly, it forces Prototype into using exceptions as a iteration signaling method, which is a nasty hack.
For example, let's look at the any method. In Ruby, any? returns true if an item in a list matches some criteria. Thus to find if any number in an array is odd you would write this:
[2, 4, 6, 8, 11].any? do |value|
value.even? # even? is from Rails, not Ruby
end
In my view, porting any to JavaScript is of dubious value at best. But let's look at the contortions that Prototype has to go through to do it:
any: function(iterator, context) {
iterator = iterator ? iterator.bind(context) : Prototype.K;
var result = false;
this.each(function(value, index) {
if (result = !!iterator(value, index))
throw $break;
});
return result;
}
each: function(iterator, context) {
var index = 0;
iterator = iterator.bind(context);
try {
this._each(function(value) {
iterator(value, index++);
});
} catch (e) {
if (e != $break) throw e;
}
return this;
}
_each: function(iterator) {
for (var i = 0, length = this.length; i < length; i++)
iterator(this[i]);
}
The any method calls each with calls _each which then calls your method. And since JavaScript doesn't support returning values from an anonymous function used as an iterator (there is no yield keyword like in Ruby), the any method is forced to throw an exception (see $break) to signal that an element has been found. That might seem like a small offense until you are trying debug JavaScript code using Venkman and keep interrupted by meaningless exceptions (which happens if you've asked Venkman to stop at all errors and exceptions).
More examples of trying to make JavaScript more like Ruby abound:
- The addition of a Class object that introduces an initialize function, instead of just accepting JavaScript's combined constructor/initalizer idiom
- A number of useless additions to the String class (methods like succ, times, etc) - 100 plus lines of code
- A number of useless additions to the Array class (methods like succ, times, etc) - a bit less than 100 lines of code
The end result is that over 10% of Prototype is wasted trying to add Ruby like-features to JavaScript that don't fit well, simply because the Prototype designers prefer Ruby's idioms over JavaScript's idioms. The obvious problem is that Prototype is a JavaScript library, not a Ruby library.
Lay Off My Prototypes
Prototype also fails miserably on the unobtrusive test. In its first version, Prototype added methods to JavaScript's Object prototype - which is a big no-no. Not learning from its past mistakes, the latest version of Prototype has this gem:
(function() {
var element = this.Element;
this.Element = function(tagName, attributes) {
attributes = attributes || { };
tagName = tagName.toLowerCase();
var cache = Element.cache;
if (Prototype.Browser.IE && attributes.name) {
tagName = '<' + tagName + ' name="' + attributes.name + '">';
delete attributes.name;
return Element.writeAttribute(document.createElement(tagName), attributes);
}
if (!cache[tagName]) cache[tagName] = Element.extend(document.createElement(tagName));
return Element.writeAttribute(cache[tagName].cloneNode(false), attributes);
};
Object.extend(this.Element, element || { });
}).call(window);
Take a good, long look at this method. It replaces a browser's built in Element object, which is used to represent elements in a DOM tree, with an Element function. Replacing a core browser object is nuts. Especially for the ridiculously small payoff. Instead of writing this:
var element = document.createElement('foo')
element.id = 7
This change lets you write this:
var element = new Element('foo', {id: 7})
And how many times does Prototype use this function? A measly 4 times! And to add insult to injury, the code as written is broken because it breaks the prototype chain. The last line in the function should be:
this.Element.prototype = element.prototype
Without this line, any custom extensions you've made to the Element object are lost. Trust me, it took a good long time to debug why our code no longer worked.
Time for a Diet
Finally, Prototype is getting bigger with every release. Version 1.5 weighs in at 3,396 lines of code while version 1.6 is 4,307 lines, a 27% increase. I'm sure the additional code is useful, but I'm also sure there are great swaths of Prototype that I don't need. Unfortunately, Prototype doesn't provide a mechanism to package up only the parts of it you want. When the library was smaller, that was a reasonable decision. But as Prototype continues to grow, there will come a point where its benefits are outweighed by its weight (and for me I've passed that point).
So What Next
The last few years have been JavaScript's golden years, marked by an amazing outpouring of experimentation and creativity that has led to a number of great JavaScript libraries. A huge benefit of this work is revealing the pain points, beyond cross-browser compatibility issues, of working with JavaScript. These issues include the lack of a Selector API, better iterators, better chaining of DOM methods, wordy method names (getElementById), etc.
Of course each library takes its own approach to solving these problems, and with that comes a downside - lockin. For large JavaScript projects, switching between libraries is a boring, tedious, time-consuming undertaking. Which is the reason we've remained with Prototype for as long as we have and will continue to do so for a bit longer while we plan our migration to a new library.
Posted in JavaScript | 15 comments | 1 trackback
