Analysing the analysts

As the retail market continues to slide and new gaming categories burst onto the scene, it’s getting increasingly difficult to find out how big or fast growing the overall market or its components are.

At the same time, finding this out is becoming more important for companies wrestling with which horse to back or how to convince VCs and other external investors to fund their growth in these newer markets.

This month, I want to turn our spotlight on research providers and explain what to look for, what to avoid, and why market estimates can vary so widely between analysts.

A WORLD OF DIFFERENCE

The best example of massive differences in market projections can be found in mobile gaming. Whilst 2010 estimates for mobile gaming varied between researchers by 100 per cent (from around $3bn worldwide to over $6bn), 2013 forecasts were even more divergent, ranging from $4bn to as high as $12bn. There are several reasons why analysts can come up with such different interpretations of the same market.

The first – and most cynical – reason is that some can err on the side of bullishness over caution when it comes to market sizing because, put bluntly, this sells more reports. We frequently hear senior execs say they have selected a particular report over another simply because it has the most attractive market size figures for them to use in their investor documents.

In practice, market size overestimates benefit nobody except the research company; no investor will invest in a company based on market growth forecasts alone and the better ones will quickly learn what the more realistic figures are during their due diligence process.

The other reason is that it is extremely difficult to estimate the sizes of these new games markets or agree upon common metrics – such as a unit sale – for very different markets. Unlike retail, where publishers agree to share comparable sales data, most new markets have no such solid data sources, despite the fact that, as digitally distributed media, such tracking should be relatively straight forward.

The problem is that online publishers, developers and their distribution and payment partners are reluctant to share actual sales data and both their greater control of the network games supply chain and the lack of peer pressure have given them the power to say no. Whilst some markets in which traditional publishers operate, such as premium downloads, may eventually get sales charts and data, this is only a small fraction of the total online market. With around a dozen distinct network games markets (mobile, MMOGs, social and so on), there is zero prospect of the industry being able to produce a single reliable digital and retail chart. Faced with that dead end, what about separate studies of these market categories?

SIZING UP

There are several methods analysts use for sizing markets, none of which are perfect, but all of which rely heavily on researchers using deep knowledge of the market category they are analysing.

One way is ground-up, estimating individual companies’ – or games’ – sales based on public or confidential acquired data. Another is top-down, using a solid piece of data – for example the number of PS3 owners – and work back from that to a market value (e.g. PSN sales), using other data sources for guidance.

In both cases, the margin for error is entirely based on the quality of the guiding data and the experience of the researchers. We believe that a mix of both yields the most reliable market figures but it does not stop analysts with only a superficial understanding of gaming producing hopelessly inaccurate market perspectives.

The most suspect and, sadly, common way to size markets is using questionnaire-based consumer surveys. In isolation, this process is almost guaranteed to produce incorrect results because it relies on finding a representative cross-section of consumers (it’s very easy for this to skew or self-select in a particular direction) who answer questions about their exact spending accurately – which few do. Then the results must be grossed up in the correct ratio to reflect the market as a whole.

Unfortunately the lack of concrete data has driven some companies to use this type of consumer research to size network games markets without any bottom-up research to act as common sense guidance. When questioned, one such MMOG study’s authors tried to justify hugely overestimated figures by pointing to several companies without realising that these companies’ actual figures were public knowledge and represented a fraction of their estimates.

Needless to say, there are good analysts out there but you will need to exercise scrutiny and scepticism to identify them. You should ask: what is their motivation? Do they have sufficient experience of the games markets they are talking about? Is their research methodology sound and can it be justified? Do their market size figures stack up next to public company data? Does it pass a common sense test? In other words, make sure you do your research before buying into (or from) them.

[GIC provides strategy and research consulting services for games, media and finance companies, plus commercial check-ups and online game optimisation for studios]