The folks over at CMSWire.com have a great writeup on the 100 hours to build a website challenge that took place at SxSW. A team of developers from Drupal, Joomla, and WordPress got together to build a website based on a specific list of specifications and design. At the end, representatives from each team had the chance to get together to showcase what they created.
The WordPress team headed by Matt Mullenweg created a website on WordPress.com called WPShowdown to provide updates on the project. You can check out the CMSWire article to see an image of the scores given to each project but it looks like WordPress came in second although in the end, the results of the competition were decided to be a DRAW with no clearly defined winner.
One interesting thing to note about this competition is that Marybeth Schroeder who had no preference for any of the CMS’s used in this competition preferred Drupal as her first pick, then Joomla and then WordPress. Apparently, Marybeth had a hard time navigating through the dashboard which strikes me as odd considering all of the UI work that went into the design.
i understand marybeth. there is a lot of things going on inside the wp dashboard. a good thing for seasoned users but might be a nightmare for inexperienced clients.
people who hire developers are only concerned with two things:
1. get the site up; and
2. simple way for them to add content.
In the past we’d spent probably 10-15 percent of the time or budget modifying the back end of whatever CMS (WordPress, Drupal, ExpressionEngine etc) we were using so the client could understand it. Often we’d have to write up documentation on what was where and how to access specific content.
Now it’s simpler since most CMS’s have roles and making changes to the admin UI is usually straightfoward.
I wonder how much time Matt and the others had devoted to making the back end “Marybeth-Friendly®”. I also wonder if Marybeth was interviewed after 2 days of using each CMS, which she’d ultimately prefer.
Setting up a comparative test between 3 CMS and issuing a draw for all 3 is a total non event I say. It really screams “oh shoot, after all we’ve done all this for nothing as we can’t really judge”
This is the most interesting stat to me:
Kind of demonstrates how Drupal and Joomla can do a bit more from the web console than with custom code, but of course WordPress can do anything with the right amount of code. Expect to see that number for WordPress drop over the coming years as more features are added to WP…or maybe not if it is only blogging software? :P
@Brad -
Hah hah.. that’s why Ma.tt is optimizing the “bazillion” lines of code. They should count themselves lucky that only needed 1,808 instead of the full bazillion.
The number of lines of custom code on the WordPress site was largely due to the custom installer that the team wrote to allow someone to install the entire package as quickly and easily as it is for them to do a base WP install. That team also did a lot of work updating various plugins to be WP-MU compatible, which is a big boon for the community.
As far as the user evaluation, I think that WordPress was hurt by the fact that they were third in the random draw and that our tester had already been acclimated to the “edit some things on the front end” paradigm established by Drupal and Joomla!, and she wasn’t used to having to edit everything through the dashboard. On the plus side, she has agreed to perform a second evaluation after having viewed training/orientation material provided by all three teams so that next time she’ll have had at least some basic knowledge of how the site is supposed to work.
We’ll be posting this and more information on cmsshowdown.org in the coming days and weeks, but I want to make sure to give major props to Matt, Beau, the other members of the WP team and everyone else who worked so hard to make this project a success!
I was in the audience at the SxSW panel, and when I saw the video of the testing session (Marybeth, told to test all three in a row with the instruction that she couldn’t eat lunch until she finished), I almost had a heart attack. It’s no wonder that Marybeth was confused. The video of their testing session showed that she was logged in to WordPress as a subscriber, rather than as an author, editor or admin. So, the dashboard was completely non-utilitarian in terms of creating content and almost all navigation was hidden, since subscribers can pretty much only edit their profile. I would very much like to see a comparison of the sites with all of them being tested with the same access levels. The first time I logged into a site as a Contributor, I freaked out, too, since almost the entire application is invisible.
The lines of code for WordPress included a custom installer, b/c the WordPress team thought the test subject would be installing the software on a shared host, not testing a pre-installed version. I think Matt said that accounted for about a thousand lines of code. The representative from Joomla also mentioned that there was code not included in that number for them b/c some code was pre-written. It sounded like how lines of code were counted was not standardized.
I thought the idea of the panel was interesting, but wondered about the impartiality of it all. The spec for the site everyone created a version of was created by someone affiliated with Drupal, and may have unintentionally favored “out of the box” Drupal functionality. I think they’re talking about doing another round of testing, addressing some of the issues with the process that was used for the panel test. I’ll be very interested to see results on a new round of testing.
@iRonnie – That may be the case, but without totally dumbing things down to the point where you lose functionality over simplicity, where do you draw the line? Unless you provide a simple manual to start things off, how do you solve such a problem?
@Ozh – Yeah, I think it sucks that it came to a draw with no declared winner.
@Brad – I think Brad was highlighting the fact that it took a large amount of custom code to make WordPress do something it wasn’t meant to do out of the box while not so much so with Drupal or Joomla.
@George DeMet – George, thank you very much for stopping by and hosting this event. I’m definitely interested in her second evaluation. Do you have any timelines as to when another event like this might take place?
@Jane Wells – Hope you had a great time down in Austin. Judging from all the pictures, looks like everyone had a great time plus some great food.
Logged in as a Subscriber eh? That would explains all of the confusion with Marybeth trying to figure out where to go in order to accomplish a task.
Sounds to me based on your review that the entire process needs to be created and overseen by a committee of people who have no bias or experience with any of these three content management systems. Which is sort of weird as then you would have people with no experience settings up and conducting the text. Not sure how you could strike a balance between the two. However, looks like lessons were learned and I too will be interested in seeing this type of test performed in a second round.
Is there anything you saw Marybeth do in WordPress 2.7 that immediately made you make a mental note to address in 2.8?
I get that the custom installer was the majority of custom code for WordPress, but remember Drupal has this feature (installer profiles) built in so my point is still valid.
I’m not sure about Joomla custom install scripts
@Jane Wells – Marybeth tested the site using all of the logins provided by the WordPress team, including both subscriber and administrator.
While I believe that Marybeth was logged in as a subscriber in the particular clip that I showed in the presentation, she had just as much trouble using the Dashboard when logged in as an administrator. In fact, the plethora of new options available to her at that level was even more confusing to her.
The point of the clip that I showed in the presentation was not to illustrate any lack of functionality at the subscriber level, rather it was to show how easy it was for a novice user to inadvertently leave the dashboard and go to one of the WordPress.org development blogs.
I will soon be releasing all of the video from the Joomla! and WordPress user evaluation sessions as soon as I go through them and make sure that no passwords are revealed in them. Marybeth has also agreed to re-evaluate all three sites after reviewing documentation and training material provided by the teams, and we’ll be releasing the results of that by the end of April.
As far as the criticism that the spec may have been influenced by my experience with Drupal, all I can say is that in my nearly 15 years of experience working with the Web, I’ve worked with a wide variety of CMS platforms, including both Drupal and WordPress, which I used to power my wedding blog. The functionality that we asked for in the spec was intended to be representative of a wide array of common Web social networking tools, and included a number of items that posed significant challenges for the Drupal team.
After having received a lot of criticism early on for including WordPress in the challenge and being told that “it wasn’t really a CMS”, I have to say that the WP team shattered a lot of stereotypes with their fantastic site, and one of the comments I heard several times coming out of the presentation was, “I didn’t realize that WordPress could do all that”. Again, I think the team deserves a lot of credit for proving the skeptics wrong, and establishing once and for all that WP is a system that can compete at the same level as Drupal or Joomla!
@George DeMet – Geroge, Definitely keep me posted on further updates on not only this challenge, but of the results from April.
Marybeth Schroeder should give up her day job if she had problems with the WordPress administration UI, compared to Drupal and Joomla the WordPress UI is the cleanest and most intuitive out of them all.
Joomla i would say has to be the worst overall, personally my copmpany has used it for several projects in the past and due to continuous problems we no longer use it in a professional capacity or accept work that requires Joomla to be used.
Drupal on the other hand is good for certain projects if you can get your head around it’s theme engines and learn how to override functions etc, it was the perfect choice for WPscoop when we wanted to create an advanced Digg.com type website dedicated to WordPress.
Overall though WordPress is our chosen system and we use it in around 90% of our projects and that of our clients, WP has come on leaps and bounds in the past few years and is still sometimes sorely underestimated by professionals as merely a blogging system which it is not.
So wordpress win’s for me every time :)