Posted by Chuck Vose
Wed, 02 Jan 2008 23:25:00 GMT
Owing to the complexities of building a Rails adapter for Mnesia I’ve been looking into using a Ruby to Erlang bridge and have looked at the various projects that seem to be available and want to share my comments on how each is stacking up so far.
Rebar
Rebar was announced on 2007-04-20 and has since never surfaced to the public as far as his blog and Google are concerned. The code looked to be among the easiest to understand but is thankfully very similar to the Japanese RulangBridge below. The existence of the Japanese project could explain why Tom Werner never finished the project but that’s merely a guess.
Erlectricity
Erlectricity by Scott Fleckenstein was the first project I really tried out and I am impressed by some aspects and disappointed by others. On one hand it was easy to obtain being hosted on RubyForge and Google Code and works well at using Ruby from Erlang. On the other hand it comes with no documentation or rdocs outside of two uncommented examples.
I could never figure out a way to get it to bridge Erlang commands from Ruby but there may be a way that I’m missing; I have to admit I’ve not asked Scott about whether this code can go both ways or not.
So if you need something that Ruby does well but Erlang doesn’t then this may be your project. The second example uses the bridge to generate a gruff graph from Erlang which seems like it’ll come in handy for a lot of people.
Outside of documentation I’m concerned about whether the bridge can handle asynchronous requests from Erlang. From first glance it doesn’t seem like there’s any blocking and each Erlang thread calls the script directly instead of central blocking thread so I’m guessing that there’s no concurrency built in; something that we would have to work with if we were to use this bridge for a many-threaded project.
RulangBridge
RulangBridge by Toshi Hirooka (?) is a Japanese project allowing Ruby to use Erlang functions. Google’s translation allows us to read the usage instructions and browse the code which has allowed me to start using it in earnest to build the Ruby to Mnesia bridge.
My first impressions are that the bridge is that it is a little immature despite the >1.0 release number. The example code is excellent (if in Japanese) but it isn’t packaged in a gem or hosted on RubyForge and doesn’t have a way to auto-start an Erlang server. Furthermore, using only the built in classes we have to make the choice of asynchronicity or complicated/multi-module code.
Of course the raw class (called Erlang) can be made asynchronous by wrapping it in a Thread.new which is essentially what the Rulang wrapper class does, but it would be nice to have this built-in. Putting the asynchronous switch in the wrapper class is fine but the wrapper class suffers from a desire to make the Erlang calls feel like ruby and therefore makes calling complicated code impossible through the wrapper and thus makes the asynchronous aspect moot also.
The last concern I have is that each class must connect to a node explicitly rather than being able to automatically find and balance between nodes. In order to be happy with the adapter we’ll have to figure out how to load-balance or make Erlang do it for us.
Conclusion
For the purposes of the the Ruby to Mnesia Adapter I believe that RulangBridge will be sufficient but Erlectricity definitely has its purposes in the Erlang community. Despite some concerns and drawbacks both are usable code and could easily move forward with a little love from the community or follow-up versions by the owners.
Any project involving Erlang and Ruby will have to deal extensively with concurrency. Ideally it would be nice to see a broker model between the two that supports communication in both directions and deals with concurrency issues and load-balancing transparently. Until that happens we’ll have to work with Erlectricity and Rulang and love the creators for their hard work.
Posted in Ruby, Mnesia, Programming, Erlang | Tags bridge, Erlang, Ruby | no comments | no trackbacks
Posted by Chuck Vose
Sat, 22 Dec 2007 07:17:00 GMT
My goal this year is to release a Ruby port of the Mnesia distributed database and hopefully start the process of moving to a true slice architecture. The port is an interesting project but I think the importance of the slice architecture is paramount.
For the last couple of years we’ve been working on the n-tier model with ruby. It’s well established and it has been working nicely for us. But the web server industry is starting to move more towards the idea of instances or clouds of ambiguous slices. Amazon is doing it, mongrel is a part of it certainly, Mnesia has always worked this way.
My hope is that my port will help us to create an EC2 instance that is both the master of it’s domain and a part of a cloud at the same time. I would like to see an EC2 instance that can autoconfigure itself and automatically find its neighbors, which contains a complete Mnesia instance, a couple mongrels, and a proxy/load balancer.
I’m not sure if we can do this quite yet but multi-processor theory suggests that it can be done. Whether it’s advantageous to remove all the bottle-necks and have to deal with the scheduling individually is where we’ll have to analyze but I’m confident that we are moving somewhere truly incredible.
In the future I hope to be able to drop in a new EC2 and just have it completely figure things out for me. No more MySQL master, no more apache proxying. Whether we use my new port or SimpleDB is of no concern to me at all.
Posted in Ruby, Mnesia, Programming, Erlang | Tags Amazon, DDBMS, Distributed, EC2, Erlang, Mnesia, Ruby, SimpleDB | no comments
Posted by Chuck Vose
Thu, 13 Sep 2007 03:40:00 GMT
Rails has a dynamic method where you can do find_or_create_by_attr_name(attr) but it the method names get incredibly long very quickly. So SJS wrote this article in which he tries to remedy the situation. His solution was pretty elegant but it still only worked for fields that were already defined in the database; if you often redefine setter methods it doesn’t work at all.
Here’s my attempt to remedy the situation.
module ActiveRecordExtensions
def self.included(base)
base.extend(ClassMethods)
end
module ClassMethods
def find_or_create(params)
begin
return self.find(params[:id])
rescue ActiveRecord::RecordNotFound => e
attrs = {}
# search for valid attributes in params
self.column_names.map(&:to_sym).each do |attrib|
# skip unknown columns, and the id field
next if params[attrib].nil? || attrib == :id
attrs[attrib] = params[attrib]
end
# call the appropriate ActiveRecord finder method
found = self.send("find_by_#{attrs.keys.join('_and_')}", *attrs.values) if !attrs.empty?
if found && !found.nil?
return found
else
return self.create(params)
end
end
end
alias create_or_find find_or_create
end
end
ActiveRecord::Base.send(:include, ActiveRecordExtensions)
Newb instructions
Create a file called active_record_extensions.rb in your the lib directory. Then add `require ‘active_record_extensions’` to your environment.rb at the bottom (without the ``). Restart your server and see what happens!
Posted in Rails, Programming | Tags ActiveRecord, attribute, based, create, dynamic, find, find_or_create, finders, Rails
Posted by Chuck Vose
Wed, 12 Sep 2007 04:42:00 GMT
Sam Ruby occasionally does an article about his long bets and I really respect his ability to predict the future. This time I feel that he was focusing too close and forgot the long part of the ‘long bet’. REST is already important, and there are further developments in edge rails that make it even more important (active resource). MySQL is already a pain for scaling and the plugins are starting to come out but haven’t really hit the fan yet.
Bet One – MySQL becomes the exception
First, I think Sam Ruby is dead on about databases. Right now there are only a few things that are a consistent pain for deployment and scaling and databases are the top of the heap. I know there are things like MaxDB for mysql and there are ways to set up ring replication but it’s hard; something that Ruby’s community is really good at solving.
My first bet is that MySQL dies out for the rails community and something new pops in. It wouldn’t surprise me if it was Mnesia from Erlang/OTP or something involving Hadoop and HBase. It would surprise me if CouchDB popped onto the scene in a big way but it’s the sort of thinking that could lead us somewhere interesting. The other option is that someone comes up with a really concrete stack of abstractions that makes it easy to balance mysql requests and writes.
Bet Two – Rails drops REST completely
Secondly, I think REST is the wrong way to move forward. REST maps very well onto the CRUD principles, but I feel like we very rarely actually use just CRUD. More often than not I want to run custom little things and create crazy associations. And I realize that this is all possible in the REST model, but it makes the controllers obscene sometimes.
What we really want is Query Language for the Internet and what better language to build that in than Ruby. I’ve seen DSL’s for direct database access and it seems like the routes would be just around the corner if this is where we decide to take it.
The question is whether DHH sees the writing on the wall or desperately wants to hang on to REST. Since he’s put so much effort into the REST idea it seems like he would be loath to drop it, but at the same time he’s an incredibly mature developer and would hopefully handle a change like this if it ever happened.
Conclusion
All respect to Sam Ruby, I really do respect his predictions over my own. But I think his predictions are too close to us right now. I would like to know what happens after REST and what happens in the database arena.
Posted in Rails, MySQL, Programming | Tags Bets, CouchDB, Erlang, Future, Hadoop, HBase, Mnesia, MySQL, Rails, REST, Ruby, Scaling
Posted by Chuck Vose
Thu, 26 Apr 2007 22:25:00 GMT
Arrow keys doing funny things when you use vi in iTerm 0.9.5? It’s because we’re too used to messing with things. Set your Default bookmark’s keyboard type back to Global.
“When the terminal type of your sessions is set to one of those “xterm” types, iTerm handles the arrow keys, the home key, and the end key in a special way.” says the FAQ while not actually addressing the solution.
But who can complain? Global works well now! Hooray!
Posted in Programming | Tags arrow, iTerm, keyboards, keys, vi, vim
Posted by Chuck Vose
Wed, 25 Apr 2007 21:21:00 GMT
At On & On Creative we have a lot of sites on a common rails backend. It’s incredibly flexible so we can offer a lot of support to customers but because there are so many customers with different needs testing has become a huge pain. Not only does it let us run tests custom to each client, it provides a way to import the data to the test db really easily and it’s way faster than loading the fixtures each time you start a test. Yay!
After googling around I happened upon this caboo.se article about how to run without fixtures at all. It’s a very indepth article with lots of good things in it; far be it for me to try to compete.
Instead, here’s the low down on how we prepare each of our sites for testing.
Add the following to the end of your rails_root Rakefile:
class Rake::Task
def detract(prerequisite)
@prerequisites.delete(prerequisite)
end
end
%w(units functionals integration recent uncommitted).each do |task|
Rake::Task["test:#{task}"].detract('db:test:prepare')
Rake::Task["test:#{task}"].enhance(['environment'])
end
Add the following to lib/tasks/fixtures.rake. Make sure to look at the two commented lines:
# This code courtesy of the "Rails Recipe's book":http://www.pragmaticprogrammer.com/titles/fr_rr/ I believe.
namespace :db do
namespace :fixtures do
desc 'Create YAML test fixtures from data in an existing database. Defaults to development database. Set RAILS_ENV to override.'
task :dump => :environment do
# Remove the limit if you want to. It's a timesaver mostly
sql = "SELECT * FROM %s LIMIT 0,1000"
# Add tables you don't want dumped.
skip_tables = ["schema_info", "plugin_schema_info", "engine_schema_info", "sessions"]
ActiveRecord::Base.establish_connection(:development)
(ActiveRecord::Base.connection.tables - skip_tables).each do |table_name|
i = "000"
File.open("#{RAILS_ROOT}/test/fixtures/#{table_name}.yml", 'w') do |file|
data = ActiveRecord::Base.connection.select_all(sql % table_name)
file.write data.inject({}) { |hash, record|
hash["#{table_name}_#{i.succ!}"] = record
hash
}.to_yaml
end
end
end
end
end
Then the fun easy stuff happens:
rake db:fixtures:dump RAILS_ENV=production
rake db:fixtures:load RAILS_ENV=test
rake test:plugins PLUGIN=your_plugin
If you want to commit the fixtures to the repo you can but it seems a waste to me since you might be regenerating them often.
Also, check the two comments in the fixtures.rake, it could be doing things you don’t expect.
Posted in Rails, Programming | Tags fixtures, plugins, Rails, rake, testing
Posted by Chuck Vose
Mon, 08 May 2006 08:19:00 GMT
One of the biggest fears of all system admins and developers is publishing work to a live server and having it break everything. Naturally we all go to great lengths to make sure this doesn’t happen but even the most well-tested code in the simplest environment still breaks sometimes.
In addition to these fatalities is the less intrusive but equally damning maintenance that sometimes just has to happen in the middle of the day whether there’s users in the system or not. I’m sure you can think of a few scenarios and horror stories.
At any rate I’ve come to realize that it’s extremely important to be able to tell the people in the system that things are going to break for a bit. This is fortunately extremely easy in Rails; a quick three or four lines in the application.rb will do it. In PHP it’s easy if there is even a little planning that went into the project.
Despite the ease I don’t think this is very common in the world of small-to-medium businesses and this seems like a shame.
Letting your users know what’s about to happen or what happened a moment ago will increase their sense of security in many cases and in the very least will keep them from calling you in a panic.
Blissful silence.
Posted in Programming | no trackbacks
Posted by Chuck Vose
Fri, 21 Apr 2006 15:32:28 GMT
I’m finally learning, oh yes.
The reason IDE’s make it so difficult to copy folders around is because it’s an incredibly difficult process. In order to copy in Subversion you have to copy the folders/files, delete any .svn directories in each level of a folder, then those folders/files have to be added to the repo.
Very complicated process which has been killing me lately. I should have known better; how not DRY of me to copy a folder…
Posted in Programming | no trackbacks
Posted by Chuck Vose
Thu, 20 Apr 2006 19:39:00 GMT
Holy crap I finally figured out why unit tests exist! To give me something to think about in the shower. Yep, simple as that.
No, wait, there are two really good uses off the top of my head: packaging and testing filesystem crap.
The first use is if you’re exporting a plugin for later use (or for general distribution). In this scenario there’s a fairly good chance that the user importing your plugin has messed up their install or is using a database that isn’t the same version or even the same type as the one you’re using.
In either case if you have unit tests and none of them fail then you know that the database probably isn’t at fault.
The second use is for filesystem crap. I can still count the number of times I’ve messed up the permissions on a folder only to find it after a half hour of debugging. This should never happen and there are wonderful tools to prevent it. `ri File` and `ri FileUtils` should get you started if you’re curious.
Specifically I’ve been working on a photo upload admin because I don’t understand file_column. In this I occasionally have to make a directory (okay, every time I upload something that isn’t a graphic) and it would be really nice to know that when I post it onto our production server that it’ll work when I release it into the wild.
Under this column is a wonderful addition to test_helper to check that things are correctly set in other places. Since rake is seperate from the dispatchers it could check their permissions, check the permissions of the public and log folders, and a myriad of other wonderful tests that could help newbies get on their way.
Rake is already pretty good at this but it serves as an excellent example of one use for unit testing.
Posted in Rails, Programming | no trackbacks
Posted by Chuck Vose
Thu, 20 Apr 2006 19:33:00 GMT
Background
After a lot of thought and an enlightening article by Joel Spolsky , I’ve come to the following conclusion: Hiring new developers is almost never the answer to the problem of having too many programming projects.
We have that problem right now. Demand is very high and Tynan no longer has much time to code due to business requirements. So we have at least four pending projects, two of them very large, and three on-going projects but only 1.5 developers.
No, we really have much closer to 3/4 of a developer. If that.
I spend roughly 1/3 to 1/2 of each day doing server related projects. Unbreaking the repo is a pretty consistent project (probably because Tynan and I are bad citizens but we don’t know any better), as well as maintaining the email, doing backups, messing with mysql, doing security checks and making sure apache is still running.
I usually consider it a good day if I get in two hours of programming before I’‘m too tired to continue. It happens fairly often that after all the server management I’m just too tired to get in the zone so I pour out crap that I have to re-program later on.
Tynan is in an even more painful place. I’ve only seen him work once but he spent the whole day on the phone with customers. He’s figured out what to do with his problem to some degree but I estimate him at somewhere around 1/4 to 1/3 of a developer.
Solutions
What the hell do you do? Adding more developers is the natural idea; it seems like it would work brilliantly. But anyone that’s worked with another developer knows that this is dead wrong. As Brooks points out in The Mythical Man-Month adding a developer increases the complexity of communication exponentially.
Gamble with time
So that’s out, the only thing left is to reclaim lost developer time and gamble a little bit. By hiring a secretary, a server admin and a tester will the two developers produce enough to offset the three extra salaries?
It’s a business choice, the sort of thing I really suck at. If the developer is twice as productive will it be worth twice the money or will something happen where the developer isn’t twice as productive? How long will it take to train and hire a server admin? How much of this time will be from the developer? Can we afford to lose the main developer for the time it takes to train the server admin?
What about a secretary? How long will it take to train the secretary to find all the information in the office? How long will it take to organize the information so they can find it period?
Time management
Maybe that’s not the only solution. The conflict is that the developer doesn’‘t have time to do all the projects. So you have two approaches, make the developer more effective or give them more time.
The other option is one that requires months of foresite. We have a project starting in May, two weeks from now. It’s not going to happen. I knew it wasn’t going to happen when it was proposed a month and a half ago. I had hoped but I think that in the back of my mind I knew it wasn’t going to happen.
But telling a client you’ll get to their project seven months from now is going to be a hard sell.
Conclusion
Point is, hiring developers is pretty much out. There are two options in our case which are to make the developer better or make the projects farther apart.
There are things blocking both, all we need to do is figure out what the blocks are, which are biggest, and how to deal with them.
Posted in Programming | no trackbacks
