The Burlap Sack

RBridge 0.2

vosechu@create-on.com (Chuck Vose) — Sat, 24 May 2008 02:12:00 -0500

RBridge has undergone a lot of bug fixes and I think it’s worthy of a 0.2 release at long last.

1 major enhancement

Actually runs headless via the rulang command.

3 minor enhancements

Uses optparse for clearer command-line options.
Option to specify port, compile directory, mnesia directory, sname, and location of server file.
Some error checking, and debug output added. Checks to see if server is already running on specified port.

You can check it out here or just gem install rbridge

RBridge-0.1.2 Announcement

vosechu@create-on.com (Chuck Vose) — Mon, 10 Mar 2008 16:20:00 -0500

Just wanted to poke my head in and let people know that a new version of rbridge has been released. New to this revision is a few little tweaks and bug fixes but the main point is that you can install the gem and just type ‘rulang’ from anywhere to start the erlang server. This enabled us to build a little test suite which should improve quality.

Also, the end of quarter summary has been posted per our class requirements: Review

Concurrent code in Ruby 1.8.6 through inlining

vosechu@create-on.com (Chuck Vose) — Mon, 07 Jan 2008 13:11:00 -0600

Slight typo in the code fixed: 2008-01-08

Toshiyuki and I have released a new gem called rbridge which allows us to execute functional, side-effect free, concurrent code directly in Ruby regardless of the version by using Erlang as a processor. This includes using the Mnesia distributed database and ETS/DETS.

To try it out please follow these steps:

1. Download Erlang for your os. Windows has binaries and OS X can be configured with `./configure—prefix=/opt/local` to make MacParts happy. I haven’t yet tried it with Linux but the default configure options should be okay.

2. Download the rbridge gem. `sudo gem install rbridge`

3. Start the rulang server in Erlang on port 9900. Change dir to the gem directory which is usually /usr/local/lib/ruby/gems/1.8/gems/rbridge-0.1/lib and run sudo erlc rulang.erl. Enter the Erlang shell by typing erl. Finally, start the server with rulang:start_server(9900). (There’s a dot at the end of the command).

4. Require rubygems and rbridge in your code and create a new connection to the rulang server. This is the simplest bit of inline code I can think of but there is a lot more we can do: asynchronous access and ruby-style syntax specifically.

require 'rubygems'
require 'rbridge'

@r = RBridge.new(nil, 'localhost', 9900)

puts @r.erl('10*10.')

To read more check out the documentation on ruby-mnesia.rubyforge.org.

Aside: Toshiyuki Hirooka found me. Thank you to everyone that helped search and offered to translate for us. I’m constantly impressed by the support from the Ruby community.

rbridge 0.1 Released

vosechu@create-on.com (Chuck Vose) — Fri, 04 Jan 2008 19:54:00 -0600

FEATURES:

Allows use of Erlang code within Ruby

Changes:

v0.1 / 2008-01-04 (vosechu)

1 major enhancement

Allows many-lined Erlang commands when using the Erlang class for direct access.

3 minor enhancements

Revised error checking to allow custom resolution
stop_server/0 now shuts down any servers running in an erl shell
created tests to run with RSpec and autotest (erl server must be running on port 9900).

3 other changes

Forked onto rubyforge complete with rdocs, gem, etc.
Translated some comments into english as well as the README
Packaged explicitly with gpl documentation

Complex commands with Rulang

vosechu@create-on.com (Chuck Vose) — Thu, 03 Jan 2008 12:17:00 -0600

Rulang was giving me a lot of trouble with regards to multi-line commands and complex commands in general. After hacking on it for a while I’ve developed a patch that will allow ruby code such as the following:

require 'rulang'

@mnesia = RulangBridge::Erlang.new("localhost", 9900)

def find
  @mnesia.eval(<<-EOF
    QH = qlc:q([X || X <- mnesia:table(shop)]),
    F = fun() -> qlc:eval(QH) end,
    {atomic, Val} = mnesia:transaction(F),
    Val.
  EOF
  )
end

puts find

PATCH

diff -u Desktop/rulangbridge/rulang.erl Current Schoolwork/Project/mnesia/rulang_test/rulang.erl
--- Desktop/rulangbridge/rulang.erl     2007-05-17 20:25:50.000000000 -0700
+++ Current Schoolwork/Project/mnesia/rulang_test/rulang.erl    2008-01-03 10:12:05.000000000 -0800
@@ -30,11 +30,15 @@
 handle_connection(Socket) ->
-       Reason = (catch communication(Socket)),
-       gen_tcp:send(Socket, io_lib:format("Error: ~w~n", [Reason])),
+       try communication(Socket)
+  catch
+    error:Reason ->
+         {gen_tcp:send(Socket, io_lib:format("Error: ~p~n", [Reason]))}
+  end,
        ok = gen_tcp:close(Socket).

+% Try to evaluate the code submitted throwing an exception if the evaluation
+% doesn't work. Return the code submitted.
 communication(Socket) ->
        {ok, Binary} = gen_tcp:recv(Socket, 0),
        {ok, Result} = eval(binary_to_list(Binary)),
@@ -43,9 +47,9 @@

 eval(Expression) ->
-       {ok, Scanned, _} = erl_scan:string(Expression),
-       {ok, [Parsed]} = erl_parse:parse_exprs(Scanned),
-       {value, Result, _} = erl_eval:expr(Parsed, []),
+       {done, {ok, Scanned, _}, _} = erl_scan:tokens([], Expression, 0),
+       {ok, Parsed} = erl_parse:parse_exprs(Scanned),
+       {value, Result, _} = erl_eval:exprs(Parsed, []),
        {ok, Result}.

diff -u Desktop/rulangbridge/rulang.rb Current Schoolwork/Project/mnesia/rulang_test/rulang.rb
--- Desktop/rulangbridge/rulang.rb      2007-05-24 10:42:22.000000000 -0700
+++ Current Schoolwork/Project/mnesia/rulang_test/rulang.rb     2008-01-03 10:02:19.000000000 -0800
@@ -79,7 +79,7 @@
                def eval(command)
                        socket = TCPSocket.new(@host, @port)
                        socket.write(command)
-                       socket.gets # ...?
+                       socket.read # ...?
                end

Mnesia to ruby bridge evaluation

vosechu@create-on.com (Chuck Vose) — Wed, 02 Jan 2008 17:25:00 -0600

Owing to the complexities of building a Rails adapter for Mnesia I’ve been looking into using a Ruby to Erlang bridge and have looked at the various projects that seem to be available and want to share my comments on how each is stacking up so far.

Rebar

Rebar was announced on 2007-04-20 and has since never surfaced to the public as far as his blog and Google are concerned. The code looked to be among the easiest to understand but is thankfully very similar to the Japanese RulangBridge below. The existence of the Japanese project could explain why Tom Werner never finished the project but that’s merely a guess.

Erlectricity

Erlectricity by Scott Fleckenstein was the first project I really tried out and I am impressed by some aspects and disappointed by others. On one hand it was easy to obtain being hosted on RubyForge and Google Code and works well at using Ruby from Erlang. On the other hand it comes with no documentation or rdocs outside of two uncommented examples.

I could never figure out a way to get it to bridge Erlang commands from Ruby but there may be a way that I’m missing; I have to admit I’ve not asked Scott about whether this code can go both ways or not.

So if you need something that Ruby does well but Erlang doesn’t then this may be your project. The second example uses the bridge to generate a gruff graph from Erlang which seems like it’ll come in handy for a lot of people.

Outside of documentation I’m concerned about whether the bridge can handle asynchronous requests from Erlang. From first glance it doesn’t seem like there’s any blocking and each Erlang thread calls the script directly instead of central blocking thread so I’m guessing that there’s no concurrency built in; something that we would have to work with if we were to use this bridge for a many-threaded project.

RulangBridge

RulangBridge by Toshi Hirooka (?) is a Japanese project allowing Ruby to use Erlang functions. Google’s translation allows us to read the usage instructions and browse the code which has allowed me to start using it in earnest to build the Ruby to Mnesia bridge.

My first impressions are that the bridge is that it is a little immature despite the >1.0 release number. The example code is excellent (if in Japanese) but it isn’t packaged in a gem or hosted on RubyForge and doesn’t have a way to auto-start an Erlang server. Furthermore, using only the built in classes we have to make the choice of asynchronicity or complicated/multi-module code.

Of course the raw class (called Erlang) can be made asynchronous by wrapping it in a Thread.new which is essentially what the Rulang wrapper class does, but it would be nice to have this built-in. Putting the asynchronous switch in the wrapper class is fine but the wrapper class suffers from a desire to make the Erlang calls feel like ruby and therefore makes calling complicated code impossible through the wrapper and thus makes the asynchronous aspect moot also.

The last concern I have is that each class must connect to a node explicitly rather than being able to automatically find and balance between nodes. In order to be happy with the adapter we’ll have to figure out how to load-balance or make Erlang do it for us.

Conclusion

For the purposes of the the Ruby to Mnesia Adapter I believe that RulangBridge will be sufficient but Erlectricity definitely has its purposes in the Erlang community. Despite some concerns and drawbacks both are usable code and could easily move forward with a little love from the community or follow-up versions by the owners.

Any project involving Erlang and Ruby will have to deal extensively with concurrency. Ideally it would be nice to see a broker model between the two that supports communication in both directions and deals with concurrency issues and load-balancing transparently. Until that happens we’ll have to work with Erlectricity and Rulang and love the creators for their hard work.

Mnesia + Ruby Announcement

vosechu@create-on.com (Chuck Vose) — Sat, 22 Dec 2007 01:17:00 -0600

My goal this year is to release a Ruby port of the Mnesia distributed database and hopefully start the process of moving to a true slice architecture. The port is an interesting project but I think the importance of the slice architecture is paramount.

For the last couple of years we’ve been working on the n-tier model with ruby. It’s well established and it has been working nicely for us. But the web server industry is starting to move more towards the idea of instances or clouds of ambiguous slices. Amazon is doing it, mongrel is a part of it certainly, Mnesia has always worked this way.

My hope is that my port will help us to create an EC2 instance that is both the master of it’s domain and a part of a cloud at the same time. I would like to see an EC2 instance that can autoconfigure itself and automatically find its neighbors, which contains a complete Mnesia instance, a couple mongrels, and a proxy/load balancer.

I’m not sure if we can do this quite yet but multi-processor theory suggests that it can be done. Whether it’s advantageous to remove all the bottle-necks and have to deal with the scheduling individually is where we’ll have to analyze but I’m confident that we are moving somewhere truly incredible.

In the future I hope to be able to drop in a new EC2 and just have it completely figure things out for me. No more MySQL master, no more apache proxying. Whether we use my new port or SimpleDB is of no concern to me at all.

find_or_create by params (extension to dynamic attribute based finders)

vosechu@create-on.com (Chuck Vose) — Wed, 12 Sep 2007 22:40:00 -0500

Rails has a dynamic method where you can do find_or_create_by_attr_name(attr) but it the method names get incredibly long very quickly. So SJS wrote this article in which he tries to remedy the situation. His solution was pretty elegant but it still only worked for fields that were already defined in the database; if you often redefine setter methods it doesn’t work at all.

Here’s my attempt to remedy the situation.

module ActiveRecordExtensions
  def self.included(base)
    base.extend(ClassMethods)
  end

  module ClassMethods
    def find_or_create(params)
      begin
        return self.find(params[:id])
      rescue ActiveRecord::RecordNotFound => e
        attrs = {}

        # search for valid attributes in params
        self.column_names.map(&:to_sym).each do |attrib|
          # skip unknown columns, and the id field
          next if params[attrib].nil? || attrib == :id

          attrs[attrib] = params[attrib]
        end

        # call the appropriate ActiveRecord finder method
        found = self.send("find_by_#{attrs.keys.join('_and_')}", *attrs.values) if !attrs.empty?

        if found && !found.nil?
          return found
        else
          return self.create(params)
        end
      end
    end
    alias create_or_find find_or_create
  end
end

ActiveRecord::Base.send(:include, ActiveRecordExtensions)

Newb instructions

Create a file called active_record_extensions.rb in your the lib directory. Then add `require ‘active_record_extensions’` to your environment.rb at the bottom (without the ``). Restart your server and see what happens!

Long Bets

vosechu@create-on.com (Chuck Vose) — Tue, 11 Sep 2007 23:42:00 -0500

Sam Ruby occasionally does an article about his long bets and I really respect his ability to predict the future. This time I feel that he was focusing too close and forgot the long part of the ‘long bet’. REST is already important, and there are further developments in edge rails that make it even more important (active resource). MySQL is already a pain for scaling and the plugins are starting to come out but haven’t really hit the fan yet.

Bet One – MySQL becomes the exception

First, I think Sam Ruby is dead on about databases. Right now there are only a few things that are a consistent pain for deployment and scaling and databases are the top of the heap. I know there are things like MaxDB for mysql and there are ways to set up ring replication but it’s hard; something that Ruby’s community is really good at solving.

My first bet is that MySQL dies out for the rails community and something new pops in. It wouldn’t surprise me if it was Mnesia from Erlang/OTP or something involving Hadoop and HBase. It would surprise me if CouchDB popped onto the scene in a big way but it’s the sort of thinking that could lead us somewhere interesting. The other option is that someone comes up with a really concrete stack of abstractions that makes it easy to balance mysql requests and writes.

Bet Two – Rails drops REST completely

Secondly, I think REST is the wrong way to move forward. REST maps very well onto the CRUD principles, but I feel like we very rarely actually use just CRUD. More often than not I want to run custom little things and create crazy associations. And I realize that this is all possible in the REST model, but it makes the controllers obscene sometimes.

What we really want is Query Language for the Internet and what better language to build that in than Ruby. I’ve seen DSL’s for direct database access and it seems like the routes would be just around the corner if this is where we decide to take it.

The question is whether DHH sees the writing on the wall or desperately wants to hang on to REST. Since he’s put so much effort into the REST idea it seems like he would be loath to drop it, but at the same time he’s an incredibly mature developer and would hopefully handle a change like this if it ever happened.

Conclusion

All respect to Sam Ruby, I really do respect his predictions over my own. But I think his predictions are too close to us right now. I would like to know what happens after REST and what happens in the database arena.

Inheritance without STI: method_missing routing

vosechu@create-on.com (Chuck Vose) — Mon, 13 Aug 2007 17:05:00 -0500

Every once in a while I find myself in the situation where I need to override a model or create an inheritance without using single table inheritance (STI).

The method_missing call can help us to create an intelligent router that will route calls to the parent class or the child while maintaining an extremely small code footprint.

Here’s an example of what I’m talking about. We needed a landing page that would be dynamic for each user that wanted one. But we needed a template landing page which just needed the values substituted in at the right spot. Here’s a quick Yaml of what the table might look like.

# Example Page
landing_page_template:
  title: Welcome to [COMPANY-NAME]
  meta_keywords: [COMPANY-NAME], [COMPANY-META-KEYWORDS]
  ...

# Example Landing Page
landing_page:
  company_name: CV, Inc. 
  company_logo: www.chuckvose.com/fake_logo.gif
  company_meta_keywords: happy, rails, method_missing, inheritance
  ...

So here’s the hope, when we load up the LandingPage object and call LandingPage#title it should display “Welcome to CV, Inc.”

Pseudo-code and implementation

def method_missing(method_name, args*)
  if method_name is in the extended database
    call method_name
  if method_name isn't in the extended database
    try calling parent.method_name
      return results directly or mangle them somehow

We chose to mangle the results but if you just wanted to add a couple columns you could. You could run into performance issues doing this instead of STI. However this is an absolutely ideal place to mangle the code too if you want. Probably as good as a controller after_filter.

Two steps needed to happen for this, we need to load up the template on initialization, then use method_missing to switch between calls to the LandingPage class (such as :company_name) and the Page class (such as title).

Step 1: Load the template

The first part is a little weird because of rails performance. If you haven’t used after_find/after_initialize rails-core had to put in a little kludge for performance reasons. The solution is just to define a blank function named after_find or after_initialize.

class LandingPage < ActiveRecord::Base
  after_initialize :find_template

  # Required to activate the after_initialize filter
  def after_initialize; end 

  private

  def find_template
    @template = Page.find_by_url("landing_page_template")
  end
end

Step 2: Create the method_missing switcher

The method_missing switcher had to incorporate two things and one trick. First, it needed to look in the LandingPage table for its own attributes. Then it needed to look in the Page table for anything else before passing it through a regex engine and returning. The last little trick was that the regex engine needed to only be run on Strings, trying to regex a boolean doesn’t work very well. This allows the plethora of query functions such as Page.valid?.

class LandingPage < ActiveRecord::Base

  private

  def method_missing(meth, *args)
    # See if this attribute is in this object already. 
    # If so just call out to super and let 
    # ActiveRecord::Base deal with trying to find the 
    # attribute in the database. 
    if @attributes.include?(meth.to_s) || @attributes.include?(meth.to_s.gsub(/(=|_before_type_cast)/, ""))
      super
    else
      # Call Page#meth and store the response. Usually something like @page.body or @page.title
      resp = @template.send(meth)

      # If the attribute returned from above is a String, toss it into the regex grinder
      # and return the results. 
      if resp.is_a?(String)
        return resp.html_sub(self)

      # If the response is a boolean, symbol, or something else, just return it since we
      # can't (and probably don't need to) regex it.
      else
        return resp
      end
    end
  end
end

Step 3: The regex grinder

Security Warning: Before I begin this I want people to know that it should never be used by people outside your direct trust. It calls ruby code directly off of text found in the database. It can be rewritten to be more secure but this wasn’t necessary for us. If you want slightly more security drop some constraints on the LandingPage.send() method. For even more security do one regex per tag to replace (such as html = html.gsub(/\[MERCHANT-NAME\]/, template.merchant_name || ””). Security Warning

The last step is totally up to you. We needed it to find things like [MERCHANT-NAME] and replace them with LandingPage#merchant_name. This can be done fairly simply monkey-patching String with a blocked gsub.

class String
  # Look for snippets like [MERCHANT-NAME] and replace 
  # them with LandingPage#merchant_name
  def html_sub(landing_page)
    html = self # self cannot and should not be modified directly.

    # Look for [WHATEVER] tags
    html = html.gsub(/\[[^\]]+?\]/) do |match|
      # For each match, call the LandingPage instance method 
      # with the same name and return those results or the empty
      # string.
      landing_page.send(match.gsub(/[\[\]]/, "").gsub(/-/, "_").downcase) || "" 
    end
  end
end

Conclusion

The method_missing call seems to be a rough spot for a lot of discussions. Why_ wrote an article (The Best of Method Missing) that shows this pretty well: he starts by explaining how he has hated every piece of code he’s written with it then proceeds to point out some method_missing code that he loves. Lots of people feel this way about it and about many aspects of Rails in general.

But sometimes it’s a wonderful thing. This example should have taken several hundred lines of code and lots of kludges and bugs, but method missing trimmed it to ~20 LOC and we haven’t been able to find a bug yet.

Hope this helps you with some project, I’ve enjoyed playing with it and writing it up.