Theme Standards System

[Leland]

The difference between stylesheets and theme options pages, is that a stylesheet is pretty much required, while a theme options page tends to be uneccessary bloat.

With that said, I think a theme options page should still be considered a "feature," however placing too much weight on features like having "configurable layout" might be unfair to other quality themes without these sort of features in place. They're not bad or anything, they're just different.

The more I think about it, the more I think themes should be rated on quality of code, design, support, etc. Any "features" like theme options page should be considered strictly a bonus. Any thoughts?

[andrea_r]

I think options pages (which you know I am a fan of) are becoming one of those new standards. I do expect to find them on premium themes, as well as being well-coded and having options that make sense.

[dancole]

Saying a theme has an options page doesn't really mean anything, because it is too broad of a term. While some themes don't have an options page, others have one with a single feature, and others have ever feature under the sun.

There is also a difference in the quality of the features on an options page. Is that feature useful to most people or is it pointless? Is it done in an easy to understand way or is it overly complex? Is it done right or just thrown together without testing every case?

[chipbennett]

I'd like to ask your opinion.

A few days I blogged about the idea of a theme standards system. I'm considering setting this up as an independent site and I want to know if you as a theme user would find it useful when searching for themes or if the theme authors would find it useful to help set them apart. This is in part my way of adding some specifics to claims that themes are premium.

Conceptually, it's a good idea, I think.

The standards system would be a series of ratings (1 to 5 stars), each of which relate to a set of features and functionality. In general the ratings would mean:

• 1-star - A solid, entry-level theme, built and tested to agreed standards, using accepted practices.
• 2-star - A theme where the look and feel can be customised
• 3-star - A theme where the layout can be customised
• 4-star - A highly configurable blog theme
• 5-star - A theme enabling signficant cms capabilities

What do you mean by "can be customized"? All themes can be customized, by editing the PHP/CSS files. Do you mean that the theme includes a wp-admin options page for customization?

The ratings are also comunlative [sic], meaning that a 2-star theme will comply with the requirements set out under both the 1-star and 2-star ratings.

That's probably a given, no?

Each star rating would drill down to a much more specific set of features. To give you an idea a 1-star rating would mean:

• Validated (x)html 4 or html 5 (with IE workarounds)
• validated CSS
• compliance with at least the lowest wcag rating
• All marketed features can be controlled using a user interface included within WordPress or added by the theme that is specifically intended for the provided use.(Note this excludes the use of custom fields for advertised features but they can be used as an optional fallback)
• No manual editing of the theme is required for normal operations, i.e. widgets.
• The theme contains all of the key page template types (this excludes item specific files such as category-x)
• contains all wordpress specific css tricks, photo captions, et al.
• Contains all the standard WordPress theme hooks
• Compliant with nested comments

Personally I would leave the first point at standards-compliance. I don't think theme authors should be required to use non-compliant work-arounds/hacks, or penalized by not doing so.

More than likely, a 1-star functionality-rated theme wouldn't have any custom functionaity, so your fourth point would probably be moot.

Why should a 1-star functionality-rated theme (i.e. "A solid, entry-level theme, built and tested to agreed standards, using accepted practices") be required to use widgets? I would put widget customization probably at the 2-star functionality level.

What's wrong with using custom fields? That is, why should use of custom fields exclude a theme from earning a rating?

WordPress-specific CSS tricks?

Also, one thing you're missing (and, IMHO, it's awfully important): WP version compatibility. A theme that is 1-star compliant under WP 2.0 (or 2.3, or whatever) maybe shouldn't necessarily be 1-star compliant under WP 2.7, especially considering the theme-related functionality that gets added/changed as WP versions change.

A lot of those points would be expanded to contain a full list of the template files, css tricks, etc.

Having such lists would make the rating much easier to evaluate.

2-star would go on to list specific ways of how the look and feel could be amended and the minimum functionality that should be provided, and so on.

Obviously a great deal of discussion would need to happen around the contents of each rating to make sure they were relevant to users.

This would probably be the second-most difficult part: getting "everyone" to agree to the ratings.

It would be up to the theme author to decide that his / her theme met all of the requirements and to choose to include a badge in their marketing, and tags in the theme repository (if the theme is included). They would need to register the theme for inclusion in a directory where users of the theme could comment on whether they felt something was missing. This wouldn't be something that is awarded.

And this will be the most difficult part. Without some oversight (and enforcement), the system will just get abused, and wouldn't prove useful for users.

So, if you are a theme user, would you find it useful for themes to have a star rating to indicate that it provided a level of functionality?

Can't comment, as I only use my own theme.

If you are a theme author, would you be prepared to use a rating as a benchmark of the type of functionality to include in your themes and to include a badge in your marketing? Do you think it would help you to set your themes apart from those that do not want to?

I'm unsure. As has been said already in the comments, a "star" rating implies quality, rather than level of functionality.

Perhaps you could use a Level-X Functionality rating system (i.e. Level 1 - Level 5 Functionality), or a descriptive rating system (e.g. No Custom Functionality, Basic Custom Functionality, Standard Custom Functionality, Advanced Custom Functionality, Maximum Custom Functionality).

Such a system properly describes the rating system, and doesn't imply level of quality.

I like the basic idea, although I'm not sure about the "star" system. In my opinion, there's absolutely nothing wrong with a theme which you describe as having 1 star: a solid, entry-level theme, built and tested to agreed standards, using accepted practices. Being a "1 star" might imply to some that the theme may be dysfunctional, one that should be avoided.

I can't imagine a theme author marketing their theme who would say something like "yeah, this is a certified 1 star theme." Even though it may a perfectly functional, solid theme, if I was just a beginner, I would imagine being turned off by this "1 star" label.

Instead of stars, maybe there could be word labels such as: solid, flexible, advanced, things like that. Just a thought.

In a certain regard, I would liken such a system somewhat to the Michelin Star award system, in which a restaurant that is awarded even one (out of a possible three) stars is highly, highly regarded.

That said, I don't think a functionality rating system should in any way imply quality or prestige, but rather should differentiate themes so that users can identify the themes that provide the level of custom functionality they require.

One thing that just popped into my mind ....

What if there were categories like:

1. Code
2. Design
3. Features
4. Usability
4. Support

And each one had a 10 point scale. A theme could be rated like so:

1. Code: 9/10
2. Design: 8/10
3. Features: 5/10
4. Usability: 8/10
4. Support: 10/10

for a total of 40 out of a possible 50.

The theme would be rated a 40, but the users could see each category of ratings clearly as well. They could see that the code is solidly built, but it may not be feature-packed. It might have crappy code, but the support is superb ... tons of features, but sloppy code.

Make sense? Thoughts?

Again, this is more of a theme quality rating - which is perfectly fine in its own right, but I don't think it is the intent of the OP.

The lack of sidebars is a non-issue because normal operations for that theme assume none. On the other hand if it did provide customisable areas but did not widgetise them then it would be deficient.

This doesn't make a whole lot of sense to me. Why should sidebars require widgets, in order not to be "deficient"? Just as normal operations for one theme may assume no sidebars, why can't normal operations for another theme assume non-widgetized sidebars - especially for a theme that claims no/minimal custom functionality?

That doesn't mean it is better than Manifest which has none, it just says that if the theme says you can customise the appearance the user knows what that means.

This makes more sense; just make sure that the rating system is consistent. The lowest-level rating really should be focused on standards compliance.

[greenshady]

The biggest problem here is just being able to overcome the predefined criteria on which we base our reviews/ratings. The idea of a rating system is flawed because it will always imply some themes are less than others because they don't include a feature set of another theme. A large problem is that we're comparing themes. When doing so, we've already made up our minds about what differientiates a quality theme from a mediocre theme before ever looking at the theme we're rating.

Granted, there are some things which we can decide beforehand that make a good theme: solid coding, up to date with WordPress, great typography, and maybe a few other things.

Rate a theme based on what the theme accomplishes not on what it accomplishes compared to others or what we think that users think would make for a great theme.

The best theme rating you can do: Good / Bad.

I do believe I said this in my last post but not in so many words.

[Cristi]

Theme rating in general - not a good idea. As Justin Tadlock said it's either good or bad. There is even room in the middle.

Objective rating like php, html and css coding, standards and browser support is something that we should look forward and encourage.

Everything else in my opinion should relate to whether the theme brings value for the potential user or not!

As far as the future of wpthemecritic: I would like to see subjective reviews of themes, trash crappy themes and bring to the surface the real deal.

[andrew]

The thing is that a set of standards is simply a set of standards. They are useful to the user if the user wants the things that the standards define. It isn't an objective point for comparison to say what is better or not as the standards refer to different things.

For example, Chip doesn't see a lack of widgetisation, assuming the theme has areas equivalent to the traditional sidebar, as a deficiency. I think most users would strongly disagree with that statement. Further, I think most users would assume that it would be widgetised. Defining that as part of a core quality standard for themes seems to me to be reasonable, and useful.

Similarly the issue for me isn't whether a theme has an options page but if it has one and the author decides that it should contain functionality to allow the appearance to be customised wouldn't it be nice, as a user, to know that you can change the typography in specific ways, colours in specific ways, override specific graphics (assuming they are present)?

As a theme author wouldn't it be useful to have a defined baseline and know that if you go further and look to truly innovate beyond that baseline then you will be exceeding expectations?

I don't agree that good/bad is an appropriate theme rating at all. What is a bad theme? In my opinion 90% of themes are garbage, 5% are on the boundaries of acceptable, and 5% have put some actual thought into things. That is being charitable. I would quite like to weed out the 95% before bothering to make any other assessment (assuming I was going to rate themes after they are build, which I am not) because the 90% are not even themes, they are decoration (if that) and nothing more, and I though that adopting a set of standards would help to do that as well as indicating to those theme authors where the baseline is.

If we want to drag up the standards of themes somebody, at some point, is going to need to define what acceptable looks like in a way that is visible in every area of theming.

[andrew]

What's wrong with using custom fields? That is, why should use of custom fields exclude a theme from earning a rating?

Producing a theme that is intended for release and providing only custom fields as a means to carry out customisations that are advertised as features is lazy.

If an author feels very strongly about bloating the UI with unused features then he / she should built an appropriate solution to allow features to be turned on or off, or offer complimentary plugins.

[greenshady]

Here's a thought:

Most of us are developers and/or designers around here debating this stuff. This is great and all, but we can all debate it until we ultimately end up in the same place we started.

Aren't we doing these reviews/ratings for the end users?

Maybe someone should do a large user poll on what the most important stuff is to them. Then, take those results and see how well they fit in with our ideas. It could possibly help mold what our criteria should be.

[chipbennett]

The biggest problem here is just being able to overcome the predefined criteria on which we base our reviews/ratings. The idea of a rating system is flawed because it will always imply some themes are less than others because they don't include a feature set of another theme. A large problem is that we're comparing themes. When doing so, we've already made up our minds about what differientiates a quality theme from a mediocre theme before ever looking at the theme we're rating.

Granted, there are some things which we can decide beforehand that make a good theme: solid coding, up to date with WordPress, great typography, and maybe a few other things.

Rate a theme based on what the theme accomplishes not on what it accomplishes compared to others or what we think that users think would make for a great theme.

The best theme rating you can do: Good / Bad.

I do believe I said this in my last post but not in so many words.

I think the issue for the OP is not one of rating the quality of themes, but rather of rating the degree of custom functionality of themes.

While I agree with you regarding trying to standardize the former, I think that there actually could be some value/benefit in trying to standardize the latter. The former is often highly subjective; however, the latter, if standardized, would be much more objective.

Users generally will have a pretty good idea of the level of custom functionality they desire in the themes they choose. Some users want a fully turn-key theme, that they activate and ignore. Other users want a highly configurable theme with lots of UI options, etc. The remaining users fall somewhere between the two extremes.

Thus, such a custom functionality level rating would be very useful for users who are looking for themes with a certain level of custom functionality, whether none or maximum.

Quality, of course, can still be debated and left up to more subjective determination.