Back in episode 82 of WordPress Weekly, a round-table discussion took place centered around the topic of the WordPress Beta testing conundrum. In that conversation, we discussed why critical bugs were found only after the so called stable release was offered to the public. The answer to the problem is more beta testers, but getting those testers is hard. Stephen Cronin of Scratch99.com takes a hard look at the problem and his proposed solution I think has merit and is worthy of more discussion.
Live user testing with staged releases. Companies such as Google use this method to roll out features over time to a select group of users for live testing. This gives them a chance to iron out any kinks in more environments and use cases rather than going through a strict beta testing team. Perhaps the same method could be used on WordPress where 100,000 or so installs receive an upgrade notice that explains that a new feature has been added and then asks if the user wants to participate in the trial of that feature. If any problems are discovered, to report them. Now I don’t think it’s a good idea to put people on Trunk versions of WordPress but the way Stephen explains it, the version of WordPress would be kept the same, only when the feature has been coded into WordPress and is ready for testing would the trial upgrades go out to the public.
There are definitely some logistical issues involved with this entire process but I’d like to see the idea discussed in an upcoming developers chat to see if it has legs. What are your thoughts on this method?