Release Cycle

At a consumer-facing site, you can push small updates on a daily basis. The users see a gradual evolution over time. Contrast this with the enterprise Rails site, where each version goes through months of development before any customers see it – and its a big jump when they do. Hopefully the big jump is in the right direction – an improvement – but its hard to guess right every time.

Development Cycle

The development process for the two types of sites reflects the developer’s needs. An enterprise app only needs to be perfect once or twice a year, when it is shipped out the door, whereas a consumer-focused site needs to be bug-free every single day for every single user. This leads to radically different development strategies.  For a consumer-facing site, the agile processes of making small fast changes with solid test coverage is about the only way to stay sane – lack of tests means unexpected late-night debugging while user’s stare at a fail-whale. Enterprise developers, while they may talk about TDD and agile, have no need to actually practice it since they have months to fix issues that creep in.

Death by a thousand Demos

The final area where this difference in process manifests is in how the business needs mesh (or don’t) with the development team. As a vendor to an enterprise-level company, we’re expected to periodically provide a “demo” version of the site to display at trade shows. This demo-ware needs to look like the final product and work, at a cursory level, like the final product – but it doesn’t need to actually work in a full and complete way. Smoke and mirrors is the name of the game when creating demos.

A consumer-facing site has no demos. What you see is what you get. It either works or it doesn’t – you can’t hide your failures from the user.

Developer Practices

Given the disparity in business requirements, its no wonder there is a huge difference in developer practices. When end-users are always using your site you have little choice but to use best-practices: test everything, keep all the tests passing, leave little development debt. When the end-users won’t see it for many months its easy to get lazy (I’ve done it): let the development debt pile up, rely on QA testing since you didn’t take the time to write automated tests, hope QA will catch all the bugs – but pay the price in rework and bug fixing long after you thought you were long done with with writing that code.

I’m curious to see whether this dichotomy will ever change. Maybe it already has somewhere. As one of my favorite authors says: “The future is already here, its just not evenly distributed”.

I’ve followed the work of @hcatlin, @defunkt and @justinbmeyer on templating languages (haml, mustache and javascriptMVC) and have looked for something that met my needs. Here is an idea that seems simpler than javascriptMVC but has more power than haml or mustache – hopefully the sweet spot I’ve been looking for.

The initial idea, inspired by haml but not needing a text parser, was to represent markup as JSON:

Which would generate:

However, this provides no support for interaction with the markup. When considering generating a set of dialogs for user interaction I really wanted field validation built in, like this:

Eventually I realized that wrapping all the functionality as a filter, in essence creating a tiny scrap of functionality, was what I was looking for:

I could see wrapping a form scrap in a larger dialog-handling scrap that knew enough to validate its internal scraps before allowing the OK button to save the data. It should work – I just wish it could be as pretty as the others – but I don’t see that as possible, given the semantics of JSON.

The true test will be when I actually implement the code to support such a construct. Look for a johnb/scrappy project on Github. Until then, I’ll just assume it works!

Wednesday night I received my ticket via email from a nice guy in New York who couldn’t go at the last minute. Lucky for me!

Thursday 9am the conference started with a keynote from “Matz” (Yukihiro Matsumoto) the creator of Ruby. It was not the most interesting talk I’d heard, but everyone listened politely. Immediately after Matz’s talk I picked up my free Poken device to track the people I met. The Poken was kinda fun at the conference but not very useful unless you’re near a lot of new people who also have them. But at the conference it was kinda fun to touch someone’s Poken to capture a digital business card.

Thursday 10am I attended the BERT and ERNIE talk from one of the GitHub guys, which was fairly close to what I’d already read of the new GitHub server infrastructure. Its amazing what you can do with smart people and enough servers to effectively distribute your workload.

Thursday 11am to 4pm I listened to talks on Bits and Bytes, DSLs and Flying Robots – the quality of these talks was inconsistent and there was not much material I wasn’t already aware of, but I met some interesting people at each session.

At 4pm Thursday was a talk on MongoDB that cleared up some misconceptions I had. This dovetailed nicely with the Friday 9am “Not Only SQL” talk about other non-SQL database options. Unfortunately, I don’t see any obvious winner over my current use of PostgreSQL – there probably is a better one for our particular needs but the cost, in time, of trying the various options seems prohibitive.

Thursday 5pm was my last session of the day, with a presentation on Experiment Driven Development. The basic idea is that rather than relying on your best guess, or your Marketing guy’s best guess about a new feature, you just test both new ideas on actual users before committing to a definite plan of action. He hasn’t put up any slides for his talk (he has a 1 month old baby at home) so you’ll have to rely on my judgement that it was interesting. For me, it means that I’ll be looking for opportunities to capture metrics to measure user response to system changes. I expect the rest of the ConFreaks’ videos will be online in the next week or two.

Rather than stay for the 8pm Lightning Talks I went with my friend Dan and his friends to a yummy tapas restaurant in the Mission.

Friday I learned more than I wanted to know about ongoing improvements to the RubyGems system, partial evaluation of Ruby (pseudo JIT compilation) and an alpha-level persistence layer called Aqua.

At lunch on Friday (and again on Saturday), I watched a fun programming competition to see which team could refactor a chunk of code better and faster. It was all very subjective, but gave a real-world view into how pair programming actually works and other people’s ideas on what makes for a consistent test-driven process.

Friday 2pm was a talk on the latest in MacRuby, a very promising Ruby VM running on top of Objective-C using native objects and garbage collection. It looks promising but I won’t play with it until the 0.5 release is officially out (any day now).

Friday 3pm was Jim Weirich’s talk on SOLID Ruby that expanded on some ideas about Object Oriented Programming that were originally described by “Uncle Bob”. He is a good speaker who really knows his topic so my high expectations were happily met.

Friday 4pm was a fun talk by the always-interesting creator of GitHub, Chris Wanstrath, on what Rubyists can learn from Python.

Friday evening I attended the Startup Crawl, eating tasty food, drinking beer and meeting a few people. Scribd had the best food; Heroku had the best ambiance (and one of their guys solved a technical problem I had on one of my personal sites – nice bonus – instead of telling me to RTFM).

Saturday I woke a bit late but made it in time for the Clojure talk – it seems like a good language for concurrency, but not nearly as easy to write as Ruby. Tim Bray has a nice ongoing series of blog posts on various languages that focus on concurrency. He recently delved into Clojure and it looks somewhat promising (but there is no free lunch).

Saturday 11am I attended an interesting talk on scaling called  Synchronous Reads, Asynchronous Writes. Paul Dix went on at length on the various strategies his site has used to improve performance by off-loading work to services that allow eventual data consistency. It looks like a fairly standard infrastructure once you get to a large number of servers but it was good to hear the nitty gritty details discussed.

My last session was Gregg Pollack’s discussion of  a number of performance-related gems and plugins. It was essentially what he covers in his screencasts, but since I hadn’t seen them it was well worth hearing it in person. I’ve now subscribed to his Ruby5 podcast which I might finally listen to on the train.

All in all, it was a very good conference. I met, in person, a lot of interesting people that I’d been following in various ways (twitter, blogs, RSS feeds, etc.) and I’d love to go again next year – but I’ll try to get a ticket when they go on sale instead of the night before!

#
# toothpicks.rb
#
def toothpicks str
  puts str
  str.gsub!(' ','')
  raise 'Wrong format' unless str =~ /^(x|\||\+|-)*$/
  count = str.length
  count += str.scan(/(x|\+)/).length
  str.length.downto(1) do |n|
    str.gsub!( '|'* n, n.to_s )
  end
  str.gsub!('x','*')
  puts str
  result = eval(str)
  puts "#{result} from #{count} toothpicks"
end

if ARGV[0]
  toothpicks ARGV[0].dup
else
  toothpicks '||| x || + ||||| - |||||||'
end

Running with no arguments tests the example equation:

>>toothpicks.rb
||| x || + ||||| - |||||||
3*2+5-7
4 from 22 toothpicks

Note: syntax highlighting provided by this nice page.

I was recently discussing the problem of tracing program execution through code that doesn’t exist.  Sometimes, usually to understand and debug a program, you need to trace through it. If the code doesn’t exist, its hard to follow where it goes. Actually, the code exists, its just not in source control or otherwise easily searchable.  Let me explain.

In Ruby, whose source code is a lot like its executable format, its trivial to write code that writes code – so you, the programmer, don’t have to. The Rails web framework uses this a lot, for creating whatever form of ‘find’ strikes your fancy: find_by_firstname, find_by_firstname_and_gender, find_by_pet_species_and_viciousness, etc. – as long as your database table has the appropriately named columns then it will find what you’re looking for with no work on your part!  Actually, some work is required – you need to know that most any routine starting with ‘find’ is probably auto-generated by Rails and thus will be un-findable in the source tree.

A more complex example is the routing helpers – a shorthand way of saying “put a link here on this web page that will take the user to that web page over there.  Until you realize that most any method ending in ‘path’ is a URL helper then you’ll be confused by code that references user_edit_path or formatted_pet_list_path or formatted_pet_species_list_path.  The last one says that you want to get the path to a specially-formatted list of the various pet species represented by the system (e.g. “http://myPetSite.com/pet_species/list.csv”). Fairly clear once you know what it does, but fairly obtuse until you get to that point.  And it is, trust me on this, shorter, more maintainable and more clear than the alternative (after, of course, you learn to read it).

So this difficulty in finding the source to “formatted_pet_species_list_path” (or something like that) started a discussion that eventually got around to the idea that Ruby and Rails uses code to write code.

“Its the 11th commandment”, said one, ” – thou shalt not write self-modifying code!”

Yes and no. Call me irrationally exuberant or say that I drank some sugary flavored colored water – but I don’t think its as cut-and-dried as it used to be.  This commandment, when Moses brought it down from on high, was written for a compiled language, where the source was very very different from the executing code, and it actually did modify the code being executed.  This is bad bad bad – its hard to understand, it leads to intractable bugs, it’ll make you go blind, melt glaciers and other bad stuff.  No argument there.

But is this what Rails is doing with Ruby? I’d say its different.  Instead of modifying existing code it is merely creating code that did not exist before. Its more akin to a pre-processor that could, just before compiling a program, generate all the permutations of finding database rows from columns X, Y and Z (and any number of others).  Looked at this way, it appears to be similar to a C++ template – generic code that a programmer has written to simplify the writing of code that does similar things in similar ways.  The main difference is that with Ruby there is no pre-processor – the routine gets created at the time it is first used.

The “template” in this case is a routine called method_missing that gets handed the name and arguments of any routine that doesn’t exist.  It looks for a function name matching the form ‘find_by_X‘ where X makes sense for the database table in question (of course, if it doesn’t match the correct format then it just passes the name and arguments onward for some other routine to either make sense of or to burp up an error).  Once the routine is created it is available to be called again, with none of the overhead incurred when it was first created. More importantly, it was created with no additional overhead on the part of the programmer – to write it, test it, debug it or modify it. This, in my opinion, is a huge advantage – but I’m a lazy developer who doesn’t want to write or debug any code that I don’t have to.

Kool-aid anyone?

[Update: presentation from the 4/15/2008 Ruby Meetup is now available here.]

I like reading code. Its like a novel and I want to read it cover-to-cover. Some, such as Why’s Camping framework, I struggle to comprehend. But most code that I read comes up slightly short. Like a novel with some mis-spellings, awkward phrasing or repeated analogies, I mentally mark it as “could be better”. And sometimes I really do sit down and write something better – maybe just for my own amusement but often for a useful purpose.

I recently had the experience of reading some code that parsed a variable-length binary data structure. This sort of thing comes up often when parsing a file format or communications protocol. Most of the code looks fairly similar because it does similar stuff: ignore one byte, read the next four as the length of the following junk, read two important bytes, ignore two more, read another four-byte length and skip past the following N bytes – ad nauseum.

I’ve written it in C, and it looks something like this (ignoring error conditions like getting to the end of the buffer):

ptr = &data;                  // start at the beginning of our data
ptr++;                        // skip junk we don't care about
UInt32 len = *(UInt32 *) ptr; // get the 4-byte length
len = ntohl(len);             // convert from network byte ordering
ptr += sizeof(UInt32);        // skip past the length we just read
ptr += len;                   // skip past the data we don't care about
UInt16 cost = *(UInt16 *)ptr; // read our important two bytes
cost = ntohs(cost);           // convert to the correct byte ordering

In Ruby, this tends to be shorter due to the handy String.unpack() routine, which takes a concise format string to define how many bytes to read and what to do with them. “a3″ reads 3 bytes as a string, “N” reads 4 bytes in network order, “n” reads 2 bytes in network order, etc. The code above could be rewritten in Ruby like this:

array = data.unpack( "a1N")        # read the junk and the 4 length bytes
len = array[1]                     # only get the length value we care about
data = data[5..-1]                 # throw away the stuff we just read
array =  data.unpack( "a#{len}n" ) # define the length to read on the fly
cost = array[1]                    # get our data in its correct ordering
data = data[(len+2)..-1]           # again, throw away what we just read

This code works fine, but its not much more readable than the C code. A first step would be do define a string.unpack!() routine, where the ‘!’ exclamation clues us in that it modifies the object we’re working with. In this case, the modification is to eat (discard) the data we just read. This shortens the code to:

array = data.unpack!( "a1N")       # read the junk and the 4 length bytes
len = array[1]                     # only get the length value we care about
array =  data.unpack!("a#{len}n")  # define the length to read on the fly
cost = array[1]                    # get our data in its correct ordering

But again, this isn’t much more readable (in my opinion) than the C code. Additionally, it doesn’t help us understand the code much better in the case where our format string is “a3Nna5″ and we need to remember which item in ‘array’ corresponds to the ‘n’ in the string (in this case, it is array[2]). After a test iteration or two, what I finally hit upon was to encapsulate the behavior we want in a separare Unpacker class, that automatically eats the data it reads and stores the results in an internal Hash object, to map the name ‘len’ or ‘cost’ to the data. I also combined the format string and the resulting variable so we can clearly see the relationships. The result looks like this:

u = Unpacker.new(data)
u.u! "a1        => unused
      N         => len"
u.u! "a#{u.len} => unused
      n         => cost"

Now we can clearly see which values are ignored, which are given meaningful names, and how the format codes relate to the meaning of the data. Changing it to reflect a better understanding of the underlying data will be very easy. Note that the only reason its in two statements is to define a value for u.len before we use it – blocks of fixed-length data can be one statement.

The code to implement the Unpacker class is only about 30 lines of Ruby – including the string.unpack!() routine that can be reused separately.

class String
  def unpack! format
     array = self.unpack(format+"a*")
    self.replace array.pop
     return array
   end
end
class Unpacker < Hash
   attr_reader :data
 def initialize string
     @data = string
    super
  end
  # format string is expected to have whitespace between each
  # "unpackCode=>variableName" pairing (which can have whitespace
  # around the "=>").  u! was picked to be short so it would
  # look nice, and to connote a destructive "unpack!" operation.
  def u! format
    format.gsub(/\s*=>\s*/,'=>').strip.split(/\s+/).each do |segment|
    src,dst = segment.split(/=>/)
    self[dst] = @data.unpack!("#{src}")[0]
 end
end
# Hash_with_Attrs - For the simplicity of using either u.len or u['len'],
# makes a hash appear to have members for each hash entry. Many thanks
# to Why_ for collecting this handy routine on his a href= RedHanded blog.
# Note of Caution: 'len' is fine but 'length' would not be since u.length
# would give the number of entries in the hash, not the just-parsed value.
def method_missing(meth,*args)
  meth = meth.id2name
  if meth =~ /=$/
    self[meth[0..-2]] = (args.length<2 ? args[0] : args)
  else
    self[meth]
  end
end
end

Update: An even cleaner and shorter way would be to implement a DSL as a module so the code above could look like this:

a 1,    :unused
N       :len
a :len, :unused
n       :cost

(and yes, this is valid Ruby code)

# xkcd.rb
# extract all the titles from xkcd comics since they
# tend to be too long to fully show in the browser

# USAGE: ruby -rubygems -rxkcd.rb -e 'Xkcd.new.show_all'

require 'open-uri'
require 'hpricot'

class Xkcd
  DOMAIN = 'http://xkcd.com/'

  def show id = 343  # 343 is the NSA/RSA one
    begin
      @hp = Hpricot.parse( open( "%s/%d/" % [DOMAIN,id.to_i] ) )
      (@hp / :img).each do |el|
        puts "%4d: %s" % [id.to_i, el[:title]] if el[:title]
      end
    rescue
    end
  end

  def show_all
    0.upto(400) do |i|
      show i
    end
  end
end

"a".upto("z") { |c| puts c * (1 + c[0] - "a"[0]) }

And then showed him the same program in C:

#include "stdio.h"
int main( int argc, char **argv )
{
  int loop = 0;
  for( loop = 0; loop < 26; loop++ )
  {
    int innerloop = 0;
    for( innerloop = 0; innerloop <= loop; innerloop++ )
    {
      printf( "%c", 'a' + loop );
    }
    printf("n");
  }
  return 0;
}

Enough said.

a
bb
ccc
dddd
eeeee
ffffff
ggggggg
hhhhhhhh
iiiiiiiii
jjjjjjjjjj
kkkkkkkkkkk
llllllllllll
mmmmmmmmmmmmm
nnnnnnnnnnnnnn
ooooooooooooooo
pppppppppppppppp
qqqqqqqqqqqqqqqqq
rrrrrrrrrrrrrrrrrr
sssssssssssssssssss
tttttttttttttttttttt
uuuuuuuuuuuuuuuuuuuuu
vvvvvvvvvvvvvvvvvvvvvv
wwwwwwwwwwwwwwwwwwwwwww
xxxxxxxxxxxxxxxxxxxxxxxx
yyyyyyyyyyyyyyyyyyyyyyyyy
zzzzzzzzzzzzzzzzzzzzzzzzzz

2/6/2008 Update: it might be shorter and more clear like this

("a".."z").each_with_index { |c,i| puts (c * (i + 1)) }

d efine ruby

to twitter should, if my twitter-bot works as intended, return a direct reply of

"A clear, deep, red, valued as a precious stone."

Which is a fairly accurate definition (even if it does leave out my favorite computer language).

So in this part I’ll describe the definition-grabbing piece, which queries wiktionary.org for the first definition. This first iteration is stupidly simple: read the entire page, parse its contents with the wondrous Hpricot tool, grab the first item from the first ordered list on the page and throw away any links. It sometimes gets odd or partial definitions so it will need improvement – but works great for the five minutes it took to write.

require 'open-uri'
require 'hpricot'
def efine word
  open("http://en.wiktionary.org/wiki/#{word}") do |f|
    (Hpricot(f.read) / "ol" / "li")[0].to_plain_text.gsub(/s*[.*]/,'')
  end
end

That’s all. You’ll have to wait for the twitter-integration piece in my next post. I haven’t written it yet, but given the functionality in twitter4r, I doubt it will be much longer than the efine() method above. In fact, my usual peeve about Ruby is just that: it takes longer to describe the code than to write it!

doh! Rails nearly-seamless simplicity strikes again!

I had just changed my view code (.rhtml file) to include a new div that I wanted to be updated in an AJAXy manner. So I clicked the link in the browser (remember: this is AJAX – no page refresh) and expected my new div to be replaced with the neat new content. Nope. I had to do a decidedly non-AJAX page refresh so my browser would now have the neat new div – only then could the div be replaced.

Its so simple to swap between edit and test, edit and test, edit and test, that the few times you’re required to step out of the cycle seem like a huge hassle. But not when compared to every other development process I’ve used.

I had a similar experience with the routes.rb file. Unlike models and controllers and non-AJAX views, the routes.rb file only gets loaded when the web server starts. Stopping and starting the server fixed the problem – but I think I had to run into it multiple times before I realized what the issue was. A minor pot-hole on the smooth Rails path.

To mis-quote a bumper sticker: the worst day coding Ruby is better than the best day fighting C++.