The Vose Way: Tag drupal

Playing Drupal the card game the Drupal way

vosechu@create-on.com (Chuck Vose) — Mon, 20 Sep 2010 17:21:00 -0400

Now that NodeOne is running out of Drupal the Card Game I feel like it’s time to introduce my version of the game rules which I believe are actually significantly more Drupaly than the main rules.

If you don’t already know the rules check out this video:

Now, in the main rules you compete to complete modules and sites, which is all well and good right, but we don’t develop modules in the dark at all, we all know what’s going on with other modules through twitter, blogs, and actually meeting up in person at conventions and meetups. We even talk about our sites but in my experience outside of the exciting modules we rarely go into the details of what modules we’re using.

Modifications to rules

During play: When you put development effort into a module you play your points face up so that all can see when a module is completed -2 cards have to be played and cannot be played on the same module and cannot take a module negative

During scoring: Now put your cards down on the table face up and figure out how many points were scored in your hand. Add this number to everyone else’s number and total. Write this number on a whiteboard and then give each other player a high five. Now try to outdo yourselves in the next version of Drupal (the next game you play).

Bonus points: Bonus points if you’re within 20% of the maximum possible points for all hands. This may have to be adjusted in your games. I usually represent this achievement by giving each player a peach or some similar fruit depending on season.

What's in a framework, experiences from Rails and Drupal

vosechu@create-on.com (Chuck Vose) — Tue, 31 Aug 2010 13:48:00 -0400

I’ve now been working professionally in Drupal for a year and have learned a lot about it; I have some patches into contrib but I’ve not really done much with core other than some simpletests I was too shy to commit at Drupalcon and some comments trying to help people out on d.o. Prior to my stint as Drupal programmer I was a hardcore Ruby on Rails developer for about 4 years. Over the last two years in particular I’ve learned a lot about frameworks and I’d like to share an observation about which framework feels right to me for which situations and why.

Administrators vs End-users

At the heart of my observation is this question: Who is putting content into your database and what percentage of the database are they creating? I think it’s fair to assume that there is a database but for those that are in the NoSQL camp and might be using multiple types of database lets just all agree to talk about some mythical database which content comes from.

At least 80% of my client’s content comes primarily from their staff members and is consumed by the faceless masses on the Internet
At least 80% of my client’s content comes primarily from logged in users via uploads, comments, content creation, etc.

Nobody is going to argue that making the admin interface of a website easy to use is critical for the first case. If you have people spending their entire days working on the back side of your website it had better be something awesome or they’re all going to stalk you and break your flowers just below the point where they can regrow. Unacceptable of course. Similarly, when it comes right down to it, for me of course, creating a slick admin interface to something like stackoverflow.com would be completely pointless. Probably someone uses it, but the interface needs to be sitting in the front for the users.

As it turns out, this seems to be a great dividing line between Ruby on Rails and Drupal in my mind. I believe that Drupal has one of the greatest admin interfaces of any CMS. Failing that, I don’t think that I’ll get many complaints if I say that it is easy to cajole the Drupal admin into a super usable state if you know what you’re doing (or you just used Drupal 7 to begin with). Drupal manages content exceedingly well, but it is extremely hard to build applications like todo lists in the thing. I’m taking todo lists because they’re an easy concept that doesn’t really require an admin at all.

Ruby on Rails on the other hand, does not come with an admin interface because it is not something that every web application needs. There are some marvelous backend builders for Rails, but it’s not the core competency. Same thing for Sinatra and many of the other Ruby frameworks. What they excel at is being versatile, they excel at doing things that just can’t really be done by the web engines we already understand (Content management and blogging in particular).

What’s crucial to understand here is that for the average website for your average client that just wants to get online, we’ve probably already built an engine that will make them happy. Drupal in particular of course, but Wordpress and Joomla are also excellent at doing the things that everyone else is already doing. A CMS can be done very profitably in Rails, I’ve done it, but it probably would have been much more profitable had I learned Wordpress or Drupal back in the day.

Conclusion / Summary

My point, summarized to one sentence is thus: Do your clients want to be on the Internet, or do your clients want to create some chunk of the Internet. If they want to create awesome content and reap the rewards of it, Drupal is probably your framework of choice. If they want to do something that really doesn’t fit into something that every other website is doing, Ruby on Rails is probably your framework of choice.

The hardest question though, is what do you do when it’s not really so clear cut? When your client wants half of the site to be essentially a perfect candidate for a CMS, but the other half to be a crazy interactive, totally innovative thing? Do you try to use both? Do you strap it all together with Ajax? Or do you try to build the CMS in Ruby and lose out on that time? I don’t know that answer, but I hope that I’ve made you think about your clients and their projects in a slightly different way.

Nesting Views like a Mecha-Eagle

vosechu@create-on.com (Chuck Vose) — Wed, 25 Aug 2010 13:47:00 -0400

As you have probably figured out, I love views. I love teasing out little features that are completely buried and obscure. Nesting Views is one of my favorite new travesties; not sure if it’s ready for Drupal for Evil, but it’s certainly not kosher.

If birds can nest, why can’t I?

image by Benjamin Verdonck, reported by apartmenttherapy.com

Ah nesting season, time to buckle down, collect bits of straw, and raise little Views babies that will eventually grow up and save the world. An anecdote from several weeks ago: I was creating a fairly epic view, if I do say so myself, and it just wasn’t panning out. Somewhere between the 4 relationships and the three sorts my taxonomies were getting mismatched and my images were coming back in the wrong order. Now the taxonomies were a pretty easy fix which I’ll talk about in a later snippet post, but the images were killing me.

The situation was thus: I needed to find the most recent image uploaded for a node, but the images were stored in a related child node rather than in a cck field or something. So I had this awkward situation where I’m sorting on the child node by date, but I want the child node’s image sorted by date the other direction. This just doesn’t work in views, no amount of hook_views_query_alter or sort ordering was working out for me.

But I had a similar view in another area, actually on the child node’s page I was displaying those images in the correct order, was it possible that I could use the child nid and nest a view inside my view? Is this anathema? Surely it must be.

Turns out, this is completely painless and extremely performant! How is this possible Chuck? You’re issuing an extra Views display for every single row in a View! Ah, yes, this is true, but Views and MySQL have excellent caching for simple things and complicated Views that have 4 relationships end up not really being cacheable at all. So comparatively, a nested view is much, much faster.

This is in my views-view-field.tpl.php in my themes folder. Make sure when you add it the first time to rescan your views template files. There’s a button in Views under Theme -> Information. After that if you don’t have caching on you shouldn’t need to rescan unless you add/delete more template files.

$views = array('homepage_recently_updated',
  'node_index',
  'node_index_list');
$vname = $view->name;
if (array_search($view->name, $views) !== FALSE) {
  // Print out the correct image for a node. 
  // Finding the field_alias for the correct row is just homework, try running a
  // dpm on $field->field_alias when you save the view and you'll see a list of 
  // possibilities. Mine was absurdly named because of the many relationships,
  // I'm sure that yours will be much nicer because you're smarter than I am.
  if ($field->field_alias == "node_node_data_field_journal_node_data_field_scan_field_scan_fid") {
    // My view relies on the nid of the parent node, but this is your standard
    // list of args that you would expect to have in the query string.
    $view_args = array($row->nid);

    // If you know which display_id you're going to be using drop it here and
    // skip the switch later on. 
    $display_id = '';

    // Pull down the inner view for display later
    $view_inner = views_get_view('journal_node_primary_image');
    if (!empty($view)) {
      // Select the correct display based on which view called us
      switch ($vname) {
        case 'node_index':
          $display_id = 'block_2';
          break;
        case 'node_index_list':
        case 'homepage_recently_updated':
          $display_id = 'block_4';
          break;
      }

      // Don't use display or execute if you're just wanting to output the View. 
      // Those both have extra stuff strapped onto them that may cause problems
      // later on in the Views lifecycle according to the comments in the code.
      print $view_inner->preview($display_id, $view_args);
      return;
    }
  }

So really, all it comes down to is making sure that you’re in the right field_alias, pulling down the view with views_get_view(), figuring out your display id and any arguments, then doing a $view->preview($display_id, $view_args). Not too shabby!

Making your clients love Drupal's menu system as much as I love candy

vosechu@create-on.com (Chuck Vose) — Tue, 24 Aug 2010 13:43:00 -0400

NOTE: Apologies for the gap, I said I would post on Monday but well, I decided to get married instead. So there.

You might be saying to yourself, “So Chuck, D7UX is all well and good right, but I’m running D6 right now. How can your wisdom possibly enhance my life and vitality?” Well, you’re in luck friend, for my limited stints into D7 have left me craving more of that optimized UX goodness and blogging is a side-effect of learning for me. Today, not in fact monday like I had promised, not like my promise really means anything to my adoring audience anyways, I want to talk about getting users around Drupal in the fastest way possible. In addition, I want to talk about some roles and permissions that will result in less training and less support calls.

Making love to admin_menu

There exists in this world a beautiful love-child of admin_menu and Chuck, I call it Ted but most people don’t call it anything because it’s really just an awesome idea and most of the time those don’t get names outside of the patent office. If you aren’t already using admin_menu go out and get the new 3.x version, install it, and give your role permissions to use admin_menu (you don’t really need permissions to ‘display drupal links’, that just adds some links to drupal.org which we’ll turn off in the next section). Sweet, you’ve now made a huge step to productivity by enabling a pretty bar at the top of your screen. But wait, there’s more!

Here’s what the new admin_menu looks like, so pretty and so much more clickable:

Admin_menu is pretty awesome, but it’s a little confusing unless you already know how to use it (something that is patently untrue in 100% of my clients’ cases, bless their little hearts). So we’re going to take advantage of the fact that admin_menu uses Drupal’s menu system to build all those links and structures. Mouse over to Site Building -> Menus on your fancy new admin_menu or go to /admin/build/menu and click on the Administration Menu. You should be presented with a staggering amount of links and structure that we’re going to steal from. Most of what you need will be in this menu but you might find some things in the Navigation menu or the Development menu.

Before we begin create a new menu item by clicking the Add Item button. I usually set the path to admin and name it something clever like Puppies but I’d imagine that you’re a great deal more professional than I so maybe something more along the lines of Common Tasks or Everyday. This should also be Enabled and Expanded for maximum happiness.

Next, identify tasks that you think your client is going to use a lot, not things that they may use, only those tasks that you know that they’ll use a lot. The truth is that clients typically won’t and don’t want to explore your hot little admin interface, they really just want to get things done and get out as quickly as their legs can take them away, it’s scary in here and frankly, swimming through their money like Scrooge McDuck is way more fun.

Here we have a bit of a choice, personally I would like to just copy these links onto our new menu item but this is not an option with stock Drupal 6, so unless you don’t need things to be in the right place this usually isn’t an option. If you’re lucky enough to be building a site for your Mom and you know that there will not be a successor, at least, statistically, then go ahead and move menu entries around by clicking the clicking the crosshair and dragging menu items around. Otherwise, we’ll have to do a little more work or do some database hackery.

Note: There’s somewhat of a furor over whether to include Create Content in the Content menu. Since we’re creating a list of menu entries our clients will actually use now is a great time to move the Create Content menu over from the Navigation menu by hitting the edit button on Create content and changing the Parent Item to Administration Menu -> Administer -> Content. Then we can cherry pick out items or copy them in the next section.

Copying menu items the hard but obvious way

If you’re only doing a few entries or you’re really gungho about things you can click edit on a menu item, remember the info on that screen, and paste it into a new menu item somewhere else. Not really a bad way to do things but obviously time consuming, maybe for one of those braindead afternoons.

Copying menu items the easy but less-obvious way

There are better ways to copy menu items but since Drupal really doesn’t like this concept very much it’s not been ironed out very well. One possibility is to use menu_clone to clone a particular menu and then cherry pick those menu items you actually want in your optimized menu.

Blocks for links

There is one other way of course, probably many others now that I think about it, but only one that I was thinking about in particular: blocks. Dropping a block on /admin can make life a lot easier. This block wouldn’t be available all the time but it can be a great 80/20 thing to get people running without having to work on the menu system. You could also make this available on all pages but restrict it to admin users which would be pretty snazzy. You can find one such list here: Admin Links.

I accept that this list may be controversial, but it’s a block so you can edit to your heart’s content. I tried to keep it to just the items I think my current client is actually going to use (minus a few that were very specific to this project).

Enough of this boring stuff, we want more unicorns!

Yes, I hear you on that; unicorns are wonderful. But there are no unicorns this time. Well, I guess there is this one that appeared on my desk today wrapped in magical gift wrap and sacred plastic. It includes a description of different types of horn attachments and their relative merits. Currently I have the horn equipped that allows for super hearing. Very important in my line of work, you never know when the CEO is going to sneak up on you and ask you to stop writing about unicorns!

Do your clients send you presents? If not, maybe we can figure it out together, lets talk about D7UX.

vosechu@create-on.com (Chuck Vose) — Fri, 23 Jul 2010 13:35:00 -0400

Do your clients love you with such a furious passion that it borders on problematic? Do you find new customers calling you and saying things like, “I must have your services! My friend has been bugging me all week to call you. Seriously, I just want him to stop camping out on my lawn. It’s scaring the children.”

I found this on my desk the other day and I found myself filled with a sense of joy and satisfaction. I have to admit though, it wasn’t from a customer; but what if it were? What if my relationship with our clients was so good that they found themselves thinking about me when they were at the science museum? Can anyone argue that this would be a bad thing?

No, of course your clients don’t do that yet. But we can have dreams can’t we? This post marks the beginning of a long journey to figure out what we need to do, as programmers, managers, and as a community, to make Drupal so filled with rainbows that the concept of using anything other than Drupal is just the domain of masochists.

D7UX, the U stands for Unicorns.

D7UX is one of, I believe, the largest group attempts to improve the user experience of content creators to date. Certainly there are bigger movements in the Drupal community, hell, D7 itself is just absolutely amazing, but I’m not sure that there are other communities that have a centralized structure that’s strong enough to facilitate movements like D7UX, D7AX, D7CX, and the myriad other groups.

But the most wonderful thing about D7UX is that I know, in my artichoke of hearts, that it’s going to bring Drupal out of the dark ages and drag every other CMS with it. UX is singlehandedly the biggest complaint I hear about Drupal, nobody is seeing the absolute beauty of Rubik/Cube or the new D7 default theme!

CMSs are entering a new era, we know how to manage content on the web, we know roughly how to make it scale at the small to medium end, some might say we even understand i18n a little bit, now we get to focus on things like making users deliriously happy all the time. Drupal is one of the first post-modern web application frameworks; it is not a stretch to say that the Drupal community is pushing the boundaries of what people thought was a good user experience.

So while D7UX is working on making core a positive experience how do we apply their principles in our own work? We cannot just idly rely on others to do this for us, we must also keep these things in mind when we program.

Print these principles and laminate them. Then put stickers of ponies on the card to remind of the feelings you got when you saw your first pony: that’s what we want our clients to feel when they work on their site.

D7UX’s Main Principles:

Make the most frequent tasks easy and less frequent tasks achievable
Design for the 80%
Privilege the Content Creator
Make the default settings smart

Continuation

On monday I will elaborate on point one and talk about using Drupal’s excellent menuing system to make the most frequent tasks easy and fast. We’re also going to talk about some default roles and modules that make user management worlds easier. There is no reason for D6 people to wait to improve user experience.

For now I want you to have my Main Principles:

My clients will be so in love with the site that they will personally purchase sausages and send them to me to placate my terrible hunger
My clients’ users will be so overjoyed that they will recommend the site to everyone they know which will allow me to have fun with big problems like scaling and high performance backends
I will do more for my clients than I knew was possible, I will push the boundaries of UX and I will always keep the main principles in mind when I build software

Print that and laminate it too. But put more stickers on my version, because we’re awesome and stickers make the whole affair more serious. Only serious projects get the good stickers.

Drupal snippets #1 (Views exposed filters, F5 tweaks)

vosechu@create-on.com (Chuck Vose) — Fri, 09 Jul 2010 19:50:00 -0400

Cross-posted at the Metaltoad Blog

While I’m certain that I could never work up to the standards of Peter Cooper his “Interesting Ruby Tidbits That Don’t Need Separate Posts” series was a great help to me when I was a ruby developer. You can see the original here: IRTTDNSP #1. I’ve not found similar for the Drupal community but I believe a lot of the things I do during the week are interesting but not at all worthy of a complete post of their own. So this will be my attempt to categorize and commit those very thoughts to memory. I hope that even one of these things sparks something for you.

How to get Views’ exposed filters to display in a block

If you create a block display and have some exposed filters you may have trouble getting them to actually show up. The problem is that blocks don’t seem to like to display their exposed filters unless two prerequisites are met: there must be a full page display somewhere and it must have a path, the block must be set to use ajax. I would have never guessed this either but if you look at the requests that are going across the wire they’re going to that page display and the results are coming back over ajax.

How to reorder taxonomy terms in Views’ exposed filters

On that same page we realized that we needed to order the dropdown in a reasonable way in part just to keep people from using it on the awards day to stave off some of the traffic. Analysis of traffic suggests we actually got the ordering wrong but whatever, you just want to know how to reorder taxonomy terms in an exposed filter.

Despite the ordering that Views seems to be suggesting below, the actual display order of terms is determined by the taxonomy weights page at /admin/content/taxonomy/. So you can feel confident that even though views doesn’t have a means to order your taxonomy selections, they will be ordered for you. Now the real question is though, is it possible to reorder those terms on just one page and not on another? I don’t know, maybe I’ll get to post about that later.

How to get an F5 Load Balancer to stop putting the Vary headers on your responses

When you’re using a CDN like Akamai having a Varies header in your response absolutely kills your caching despite all the hard work drupal puts into getting things right. We were seeing about 50-75% cachability through Akamai because of it. Mostly this was because Akamai couldn’t cache the JS/CSS which was clearly incorrect. After we figured out that it was the Varies header (props to Grendzy as usual :P), it was a hop skip and a jump to figure out which settings could be causing the actual problem.

Turns out the Varies header setting isn’t the culprit though it does in fact fix the problem we were having it also introduces some others. The culprit is in fact having Browser Workarounds on which fixes IE6 and gzip and not having HTTP/1.0 Requests on. Of course the manual doesn’t really describe what’s going on so massive props out to spark on the F5 forums for answering this question before I had the chance to ask it.

Conclusion

I’ve run out of time but I’d be grateful for any feedback. If people don’t find this useful I’ll not post it on planet but if you do please keep reading and encouraging me with your great comments.

Drupal benchmark results using AB and the simple things we did to get here

vosechu@create-on.com (Chuck Vose) — Fri, 04 Jun 2010 00:48:00 -0400

Cross posted at the Metal Toad Blog

I was trolling around the Internet today looking for benchmarks and I actually had a little trouble finding something current. Dries has one comparing D6 on PHP4 vs D6 on PHP5 but that was clearly ages ago. I also realize that this is going to be out of style in about 12 minutes, will probably be fraught with contention, and generally mocked by everyone; but in the interest of those few souls out there that really actually just want to know what is reasonable to expect from production hardware under some load I want to post these stats anyways.

Our setup

We’re using a gaggle of 1U SuperServers by Super Micro. The basic stats are here. We have them loaded with a pair of 4-core opterons and 32GB of RAM. Not entirely unaffordable nowadays for production kit.

These sit behind some firewalls and an F5 load balancer which helps make SSL a little quicker and makes sure that if things go down we can fail over to a different webhead. The truth is that all this redundancy stuff up front actually slows our pages down pretty heavily for small loads but when doing a lot of traffic on those happy spikey days it helps out a lot.

We’ve done a lot of the normal things to speed up the servers:

Turn on APC, this is huge!
Get rid of your .htaccess files and configure apache to do this at restart
Use Boost if you can’t run Varnish, but run Varnish wherever you can
Turn off all the devel modules
Turn on as many of your drupal caching options as you can without breaking the site. Then figure out why the site broke and try to get those breaking caches online
Turn on your MySQL Query Cache, it’s off by default and makes an amazing difference
Put MySQL on its own disk, put logs and even the MySQL binlogs on a different disk if you can
MAKE SURE YOU HAVE ENOUGH BANDWIDTH! Getting capped by your ISP will kill you during spikes since your servers will be resending lost data constantly

Commands Used

Since this article is about ab I’m not going to post our configs from SIEGE or JMeter.

Running from a box with an extremely fast network connection we were doing the following:


ab -n 10000 -c 100 -k -H 'Accept-Encoding: gzip,deflate' www.site-name.tld/

I like to run from a box within the network directly to the webhead as well to find out how much things are getting slowed down in the load balancers and firewalls. Since our webheads often do more than one site we have to use a Host header and the actual IP address as shown in the following snippet.


ab -n 10000 -c 100 -k -H 'Accept-Encoding: gzip,deflate' -H 'Host: www.site-name.tld' 192.168.0.27/

I also like to use relatively high concurrency and number of requests so that momentary spikes even out a little bit. There is a rule out on the net that you should take the average of three attempts for any benchmark and I think that’s totally necessary.

We’ve noticed that our servers start to really crack around 500 concurrent anonymous users. Peak performance seems to be around 100-200 so I stick with that so that the differences in config changes are the most obvious. If I go from 100 #/s to 250 #/s I know it’s a big change (or vice versa in a lot of cases).

Results

Without further ado, these are my results. I realize that there is certainly more that we should be doing to squeeze out performance and I also realize that my methods are pretty unscientific but I hope they give you an idea of what you might be looking for.

Internal – Test 1


Document Path:          /
Document Length:        10716 bytes

Concurrency Level:      100
Time taken for tests:   40.694054 seconds
Complete requests:      10000
Failed requests:        9993
   (Connect: 0, Length: 9993, Exceptions: 0)
Write errors:           0
Keep-Alive requests:    0
Total transferred:      112128752 bytes
HTML transferred:       106842680 bytes
Requests per second:    245.74 [#/sec] (mean)
Time per request:       406.941 [ms] (mean)
Time per request:       4.069 [ms] (mean, across all concurrent requests)
Transfer rate:          2690.81 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2  84.8      0    2999
Processing:    61  379 809.9    147   12255
Waiting:       53  344 799.4    134   12255
Total:         61  381 814.4    148   12259

Percentage of the requests served within a certain time (ms)
  50%    148
  66%    170
  75%    265
  80%    345
  90%    599
  95%   1371
  98%   3159
  99%   4725
 100%  12259 (longest request)

Internal – Test 2

Document Path:          /
Document Length:        10683 bytes

Concurrency Level:      100
Time taken for tests:   29.175060 seconds
Complete requests:      10000
Failed requests:        5389
   (Connect: 0, Length: 5389, Exceptions: 0)
Write errors:           0
Keep-Alive requests:    0
Total transferred:      111006011 bytes
HTML transferred:       105724371 bytes
Requests per second:    342.76 [#/sec] (mean)
Time per request:       291.751 [ms] (mean)
Time per request:       2.918 [ms] (mean, across all concurrent requests)
Transfer rate:          3715.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5 123.8      0    3033
Processing:    41  273 445.8    135   11841
Waiting:       36  245 430.6    124   11840
Total:         41  278 463.5    136   11851

Percentage of the requests served within a certain time (ms)
  50%    136
  66%    150
  75%    176
  80%    287
  90%    491
  95%   1305
  98%   1534
  99%   3118
 100%  11851 (longest request)

External – Test 1

Document Path:          /
Document Length:        10472 bytes

Concurrency Level:      100
Time taken for tests:   31.161653 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      110620000 bytes
HTML transferred:       104720000 bytes
Requests per second:    320.91 [#/sec] (mean)
Time per request:       311.617 [ms] (mean)
Time per request:       3.116 [ms] (mean, across all concurrent requests)
Transfer rate:          3466.66 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   13 199.5      0    3051
Processing:    46  273 414.5    138    4558
Waiting:       41  246 404.3    126    4545
Total:         46  287 457.2    139    4558

Percentage of the requests served within a certain time (ms)
  50%    139
  66%    156
  75%    299
  80%    336
  90%    482
  95%   1119
  98%   1580
  99%   3143
 100%   4558 (longest request)

External – Test 2

Document Path:          /
Document Length:        10472 bytes

Concurrency Level:      100
Time taken for tests:   29.336590 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      110000000 bytes
HTML transferred:       104720000 bytes
Requests per second:    340.87 [#/sec] (mean)
Time per request:       293.366 [ms] (mean)
Time per request:       2.934 [ms] (mean, across all concurrent requests)
Transfer rate:          3661.67 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3  95.1      0    3034
Processing:    50  264 447.7    129    4532
Waiting:       45  240 435.7    119    4528
Total:         50  268 459.6    129    6128

Percentage of the requests served within a certain time (ms)
  50%    129
  66%    143
  75%    162
  80%    295
  90%    423
  95%   1307
  98%   1532
  99%   3125
 100%   6128 (longest request)

Using git-svn to manage standard and non-standard branches

vosechu@create-on.com (Chuck Vose) — Thu, 06 May 2010 00:44:00 -0400

Cross posted at the Metal Toad Blog

When Webchick announced that Drupal was moving to Git at Drupalcon 2010, our office erupted in pleasure at the news. Lots of great Drupalists are already using Git and there’s even an unofficial Github branch of Drupal for your branching and stashing pleasure Github mirror. However, Metal Toad Media has been an SVN shop for a long time we still have a lot of processes that use SVN, so we elected unanimously to do a gradual rollout: new sites get a private repo on Github, old sites just use git-svn. Looking back I wondering why we ever delayed.

Standard Git-svn setup

If you’ve got a standard SVN setup (with trunk under trunk, branches under branches, and tags under a tags directory), then you’re lucky, because the standard Git-SVN setup is pretty much all you’ll ever need. In that case you merely need to use: `git svn clone -s https://repo.tld/repo`. After this step you’ll magically be able to switch branches super fast, commit to stage, etc.

If you have an old svn repo with just stackloads of changes use the -r flag to specify a range of revisions you’re actually interested in. We have one repo that’s well over 10k commits with tons of branches, this literally takes 20 hours to checkout with Git-svn.

Now that you’re here what do you do? There are only a couple commands you need:

git svn rebase | svn up
git svn dcommit | svn ci
git update-index --assume-unchanged path | ignore this file, great for settings.php
git checkout -- path | delete my current changes
git reset --hard | really, really delete my current changes
git merge branch_name | merge in the changes from the branch called branch_name
git stash | temporarily store my changes and check out master
git stash pop | restore my stashed changes and take those changes off of the stash list

When in doubt check out the git-svn cheatsheet and the git cheatsheet or install the cheat gem and get this direct to your command line:

sudo gem install cheat
cheat gitsvn

If you’re going to work with Github or a some other straight Git repo it’s well worth setting your Name and email address so that people know who you are and GitX can pull down your gravatar. You can do so with the following commands:

git config --global user.name 'John Doe'
git config --global user.email johndoe@example.com

I know that’s a lot of commands, it’s okay to bookmark this or the cheatsheets if you don’t want to forget. The beauty of Git is that you get to find the way that works best for you; these and a few others are all I ever need from git-svn.

Non-standard Git-svn setup

Okay, so the standard setup is all well and good but since SVN gives you the flexibility to drop branches and tags anywhere this can cause some problems for Git. Normally this is as simple and manually specifying the directories that it actually uses via the -T (for trunk), -t (for tags), or -b (for branches). However if you’re in the unlucky situation of having branches sitting alongside your trunk directory there is some additional love that you need to give.

First set up the repo as if it were a standard repo, get as much done right as you can by using -s or -T/-t/-b. Once you have the repo checked out edit the .git/config file In the svn-remote section take a peek at the braches/tags/fetch lines. If something doesn’t fall completely into one category add it as another fetch line. For instance, if you had a client branch that wasn’t in the branches folder you could do the following:

[svn-remote "svn"]
        url = https:// svndomain.tld/svn/project
        fetch = trunk:refs/remotes/trunk
        fetch = client:refs/remotes/client
        branches = branches/*:refs/remotes/*
        tags = tags/*:refs/remotes/tags/*

After you’ve edited this you can do a `git svn fetch` to pull down all remote branches. Now when you’ve done that you should see other remote branches when you do `git branch -r` and you should be able to do awesome things like `git merge client && git commit -m “happiness is a fresh merge”`

Merging to client branch

After a while you’re going to get out of date with the client branch unless they’re doing a similar thing with git-svn. But merging into the client branch isn’t quite as easy as a normal git merge, or at least it’s not the same. Do merge from trunk -> client do the following:

git reset --hard remotes/client
git merge trunk
git svn dcommit

I’ve had varying results doing this. On some projects it seems to work great, on others it seems to switch me back to trunk instead of dcommit’ing. Worth a try at least.

Conclusion

Git has been amazing for us so far. Grendzy has been using it for about a year and I’m at about 9 months. Getting everyone else on board has been pretty easy and already I’ve seen more smiles around the office. Hope your transition is as easy and fruitful as ours is so far!

Recursively load nodes from a rich data model

vosechu@create-on.com (Chuck Vose) — Wed, 02 Dec 2009 17:59:00 -0500

Cross posted on the Metal Toad Blog

If you’ve ever worked with a rich data model in drupal you know it can be a pain to load up all the children and parents of a node within the templating engine. One method that could save you a lot of time is to load the data recursively in node_load and save your poor front-end guy some wrist pain (or yourself if you’re that guy!)

Edit: Please look at the get_metadata() definition towards the bottom or none of this is going to make sense.

Here is the initial bit which loads on details about a node. Here of course you could load all manner of things like read/write attributes, cck fields, etc.


/**
 * Implementation of hook_load().
 */
function wrapper_load($node) {
  $metadata = get_metadata($node);

  // Load our attributes without children. We'll load children later. 
  $node = wrapper_load_without_friends($node);

One thing we found was that search would break when trying to index since it was trying to load all the data from associated nodes as well. One way of dealing with this is to use hook_nodeapi(‘update index’) to only load a subset of data instead of the whole shebang. But I didn’t do it like that and I’m not going to put untested code on the blog (well, aside from slightly edited code).

This is what I did instead. It’s interesting for its hackishness. There must be a better way though.


  $backtrace = debug_backtrace();
  if ($backtrace[4]['function'] == 'node_update_index') {
    return $node;
  }

Here we get to the fun part though. This bit loads up the parents, those that this node belongs to. Due to performance constraints when doing this you need to pick a direction to load infinitely. If you choose to load parents recursively you have easier code and it’s actually a lot faster as far as SQL is concerned. If you choose to load children recursively, as I will do in a moment, the SQL is a little slower but on-the-fly SQL is easier to write for a belongs_to relationship. When all the data necessary is already in the db row you load to build yourself it’s easy to include your parents too.

So you’ll note that here we load up a collection of parents being careful to make sure that the recursive function knows who called it by the parent attribute on $obj.


  // Load up any parents
  if ($metadata['belongs_to']) {
    foreach ($metadata['belongs_to'] as $drupal_attr => $legacy_attr) {
      // When we load up a child it shouldn't load its parents
      if ($node->parent != $drupal_attr) {
        $node->{$drupal_attr."_collection"} = array();

        // Get all the node attributes for our new object
        $obj = db_fetch_object(db_query(
          "SELECT * FROM %s as extra WHERE node.nid = %s", 
          $node->$drupal_attr
        ));

        // This ensures that a child doesn't reverse and load its parent in the 
        // next call
        $obj->parent = $node->type;

        // Push our recursively loaded object into the empty collection. Here we 
        // choose to only load one level but you could use hook_load again to load
        // deeper structures. The parent attribute should prevent us from getting 
        // into loops. 
        array_push(
          $node->{$drupal_attr."_collection"}, 
          wrapper_load_without_friends($obj)
        );
      }
    }
  }

Very similarly, here we load up the children. This time we load recursively with no end condition. This is prone to cycles so you may have to go with a non-recursive loader here if you have a cyclic loading cycle or some other way of terminating the recursion.


  // Load any children
  if ($metadata['has_many'] && $node->nid) {
    foreach ($metadata['has_many'] as $drupal_attr => $legacy_attr) {
      // Bail out if this node has a parent at all. We just want to keep it 
      // simple for now. 
      if (!$node->parent) {      
        // Get the list of things this object owns
        $res = db_query(
          "SELECT * FROM $drupal_attr as extra WHERE extra.%s = %s", 
          $node->type, 
          $node->nid
        );

        // node_load all children and drop them in an array
        $node->{$drupal_attr."_collection"} = array();
        while ($obj = db_fetch_object($res)) {

          // Once again, prevent our children from loading us and creating loops.
          $obj->parent = $node->type;

          // Push our recursively loaded children and their children onto the empty 
          // collection. This time we go all the way and create a much deeper data
          // model. These two could be reversed, loading belongs_to indefinitely
          // but I find this way easier. 
          array_push($node->{$drupal_attr."_collection"}, wrapper_load($obj));
        }
      }
    }
  }

  return $node;
}

Here’s where we load extra attributes from the node addon table that you see associated with every custom content-type. These attributes just get added onto the node directly so there is some concern about columns named with php reserved words. Conflicting column names like title just need to be carefully considered, they may not actually be bad a bad idea.


// Hook load that doesn't recursively load children/parents, just the attributes 
// of another table
function wrapper_load_without_friends($node) {
  $metadata = get_metadata($node);

  // Load up extra info from the node addon table
  if ($node->nid) {
    $extra_attributes = db_fetch_object(db_query(
      "SELECT * FROM %s WHERE nid = %s", 
      $node->type, $node->nid)
    );

    // Foreach of the linking attributes which will allow us to find children 
    // load them onto the node directly.
    foreach ($extra_attributes as $key => $value) {
      $node->$key = $value;
    }
  }

  return $node;
}

This is just a sample of our metadata loader. Naturally you could do this some other way but it works pretty well for us. There is room for improvement though, using a different format like YAML could buy some extra win for instance.


// Sample metadata describing has_many and belongs_to relationships as well as
// the read/write attributes each table has.
function get_metadata($node=NULL) {
  $metadata = array(
    'belongs_to' => array(
      'staff' => array(
        'office' => 'office_id',
      )
    ),
    'has_many' => array(
      'office' => array(
        'staff' => 'office_id',
      )
    ),
  );

  $return['belongs_to'] = $metadata['belongs_to'][$node->type];
  $return['has_many'] = $metadata['has_many'][$node->type];

  return $return;
}

The Power of Yes

vosechu@create-on.com (Chuck Vose) — Fri, 30 Oct 2009 16:22:00 -0400

Cross-posted at metaltoad.com: The Power of Yes

Of all the powerful words in any language assertion is in my opinion the absolute most powerful. With affermative language we are able to create trust, enact people’s will, and begin processes. Silly of course, to take one word to mean so much but I believe everyone can agree that when a client asks for something ‘yes’ should be the thing that jumps to your lips immediately, hopefully without qualification but at times you have to add on things like ‘and it will cost x’ or ’, now lets look at the plan and see how we can make this a reality’.

But this post isn’t about clients, it’s about our language and framework choices.

When I was new to programming I said no a lot. This was in part an aspect of my first language PHP, which was perhaps as immature as I was at the time. ‘No’ was probably my biggest buzzword, maybe followed by ‘can’t’ which I justified by saying that PHP couldn’t do it. Turns out I was wrong, PHP really can do almost anything, but at the time I couldn’t. But ‘No’ was certainly an artifact of the culture. The things I do in PHP are often much harder than they need to be, at least they feel unnecessarily difficult to me now, and in this environment it is easy to say that it’s to hard. I’ve noticed since starting to use PHP again I’ve noticed ‘can’t’ creeping into my language more and more.

Ruby was the first language that was strongly oriented towards ‘yes’ both in the language and the culture. It could have been that I learned later in life or in a place that had a very strong Ruby culture (Seattle), but I suspect that the language had more to do with it than anything else. Ruby positively wants to make things faster for you so that the time between your thought and your expression of said thought grows shorter and shorter as you learn more Ruby. As such, when your language is saying ‘yes’ so often you naturally begin to start thinking this way yourself.

Drupal was a bit of a revelation to me in this regard. Drupal, as everyone knows, is a PHP app, yet despite the underlying language constructs the ui says ‘yes’ even more often than Ruby does. As my coworker pointed out, the amount of a website that gets finished before you hit the code is absolutely staggering. Maybe 90% of any given Drupal website could be accomplished by Drupal using only CCK/Views (et al) with the rest of the customizations being fairly quick using the copious APIs available to us. This is assuming that you find the correct project, in all things of course selecting the best tool for the job yields the best results.

So what are your languages saying to you every day? As products of our surroundings it is absolutely critical that we evaluate what our languages and tools say to us no just what they say about us. PHP may have a stigma, and Drupal through association, but I believe that the way it works with us is a much stronger reason to invest in exploring Drupal.