Blog archive


Tom Nixon

Design for the Wisdom of Crowds

More notes from SXSW. This was a great presentation by Derek Powazek. It was fast and covered a lot so I hope my notes make sense! Fire away in the comments if you want me to clarify anything.

If you put a group of people together, they’re not necessarily smart. Our job is to enable and facilitate people so that the magic happens.

Talking about the Francis Galton experiment to guess the weight of the cow. Everyone guessed wrong, but the average of the guesses was very close.

Same is true for guess the value of the coins in the jar.

Newspaper sites with ‘most emailed stories’ facility. People make their own choices, but we get a useful insight into what are the most popular stories of the day.

P2P filesharing: A group of people create a libray of things to share. The more people that share something, you can see what’s popular and get a Top 10 list of what the best tracks are by a particular artist.

Stock market: Lots of people making small decisions that together dictate the value of companies.

So why is so much community stuff on the web dumb? See YouTube comments!

Elements of wise crowds:

Diversity – wide variey of inputs
Independent – people must be able to contribute in their own way for the own reasons (not groupthink)
Decentralisation – nobody in charge
Aggregation – all of the diverse data needs to be brought together.

For the web:

1. Small, simple tasks: i.e. not having an open free text field for people to contribute, but a more defined way to contribute. Wisdom of crowds works well when there is a specific answer that we’re looking for, not an open ended question. Examples: HotOrNot (voting whether someone’s attractive) or Threadless (voting for T-Shirt designs.) Bad example: Assignment Zero – crowdsourcing experiment to get people to contribute to creating stories, but it was too open ended. Then changed tack to creating a list of people they would like to interview and then people volunteered to do the interviews. The collective task became more manageable.

2. Given to a large diverse group: Opposite of GroupThink where decisions are made by a small clique where the priorities of the group are put before the selfish interests of the individuals. Design systems to encourage lots of people to participate. Bad example: Chevvy Tahoe experiment to allow people to create straplines for ads but people just took the piss “This asshole’s SUV”.

3. Design for selfishness: Large groups don’t participate for the good of the commons. Create systems where people can participate for their own selfish reasons. Threadless: “Submit an idea for a chance of fame and $25,000.” Google algorithm is based on links that humans have created for their own reasons. Nobody created links so that Google could have a great search engine. Tagging photos on Flickr – you do it for your own reasons, but it creates value for Flickr e.g. helping it to differentiate between ‘Apple’ as computer; fruit or New York.

4. Aggregating the results: We’re often talking about taking votes or inputs to score things and generate lists of outputs. The problem is that it becomes a game. Example: – takes the Favouriting behaviour in Twitter and aggregates the most popular tweets. There’s no public function to rank a tweet on the site, it just harvests an existing behaviour. So if we do these 4 things and it generates some outputs like a leaderboard, people might want to game the results. Example: Flickr ‘most interesting leaderboard’ which is an algorithm to find the most interesting photos on Flickr. Created an incentive for ‘bad behaviour’ on the site so people could game the system to get their photo onto the list. Same as SEO spam. Flickr changed how it displayed interestingness – instead of it being a ranked list (which people want to beat) and made it a random display of photos which it thinks are interesting. Makes is less of a game and so less likely to be gamed by people. Other examples: Threadless: on voting for T-Shirts – votes aren’t displayed until voting finishes to avoid groupthink. With online polls – you submit your answer before you see the results from others.

Popularity does not have to rule. The most popular thing is not always the BEST thing – the most votes deosn’t always win. Example: amazon reviews – doesn’t always rank the most popular reviews first – sometimes will find a popular positive and negative review.

Implicit vs Explicit Feedback.

Explicit: voting and rating mechanisms. Asking for an immediate decision from the user e.g. voting, rating etc. Tip: Use the minimum number of options necessary: do you really need to vote on scale of 1-10 or even 1-5. Might just need thumbs up or thumbs down.

Implicit: Monitoring pageviews; searches; velocity (how much is something changing over time) Interestingness (algorithms to make sense of all the data.)

Design matters. How you design the interface affects the results you will get. Even in subtle ways like colour. Changed a design from black and red version to white and votes changed.

Experiment with the crowd: red slide, audience shouts angry, warning, stop etc. blue slide: calm, ocean, relax etc.

Experiments with adverts, changing border colour: blue border did best for ads conjouring invention, imagination (because blue colour was calming so good for being creative). Red: best for ads where you want recall and attention to detail (because red invokes fear and not wanting to make mistakes)

If you are a visual designer, you need to learn colour theory!

Putting it all together:

Brooklyn museum Click! Exhibition. Users submitted photos which were rated by the community. First the community rated themselves as how much of an expert they were and how serious they were about art, then for the photos themselves. So you could view the ‘best’ photos based on how the critics rated themselves e.g. professional or amateurs. Lots of good examples of wisdom of crowds: Users submitting and voting for ideas for companies and products. Also contains some implicit feedback about how many people are participating, and the ‘mood’ – what the collective sentiment of the community is.

This post was filed under Working culture and tagged , Comments are currently closed.


  1. kenneth WedMore

    Super! Very interesting!

    Makes me think about why we’re having a tough time getting our users to review / rate our products, once they buy them……

    Posted 15th March 2009 at 9:13 pm | Permalink
  2. Absolutely right. Your rules are spot on.

    So much misguided advice is given on the potential of using the Wisdom of the Crowds approach. In many instances they seek to use the crowd’s intelligence in ways which do not fit the model. Most common is the notion that you can use WOC techniques to discover a new marketing message or a new key customer insight. While there are ways of enlisting UGC to help in these sorts of activities the WOC is the wrong approach and when applied in these areas is bound to fail.. Few take notice of the small task and diverse group rules.

    Great stuff.

    Posted 16th March 2009 at 11:51 am | Permalink
  3. Joshua

    surely EVERY designer knows colour theory right?

    Posted 16th March 2009 at 4:36 pm | Permalink
  4. @Josh – you’d hope so wouldn’t you!

    @Tunde – agreed.

    @kenneth – is this for an ecommerce website? What’s the URL?

    Posted 16th March 2009 at 9:00 pm | Permalink
  5. Tom – thanks for these notes and the others, really interesting reading. Becks

    Posted 16th March 2009 at 9:41 pm | Permalink
  6. @kenneth some interesting research on reviews just published by the Economist:

    and good commentary suggesting that getting the behaviour you want (in your case, for your customers to post reviews) is a function of the user interface design:
    “behavior and culture are a function of UI”

    Posted 17th March 2009 at 1:18 am | Permalink