Industry panel of QA and localisation experts discuss the impact of open testing
The development sector’s growing reliance on public betas has made a significant – and often troublesome – impact on the development process, according to a panel of industry experts.
Publishers are more commonly offering the public a chance to play early builds of titles such as Halo 3 and LittleBigPlanet, which in theory provide developers with vast sums of feedback.
However, taking part in a Develop panel discussion, a number of leading QA and localisation firms say open betas can often disrupt the development process if not handled with care.
Babel Media’s Keith Russell said that “using Joe Public for cheap QA is a myth”.
“What you get are hundreds of random messages, none in a useful format for the dev team, many without the steps to reproduce,” he added.
Advertisement
“So what you need is a good QA service that filters all that noise and turns them into correctly written, reproducible bugs for the dev team that can be regressed once fixed.”
Russell concluded that public betas hasn’t made the design, QA and localisation testing process more or less complicated per se.
“Though for high value IP,” he added, “I still wonder if people realise what they are putting out there.”
His reservations were echoed by professionals based at a number of QA and localisation groups.
Localize Direct’s Christoffer Nilsson said open betas have complicated the design process as there now exists an additional layer that must be included in the development process.
He said, “Creating an open beta currently requires an interruption to the development cycle and creates additional work for stakeholders in the process”
Nilsson also urged developers to begin QA and localisation as early as possible.
Meanwhile, Stephanie Deming, of XLOC acknowledged the “incredibly valuable” feedback from open betas, though warned of their significant impact.
“If the full direction of a game is changed, new assets written, new levels produced and new VO recorded pretty late in the development process that can have a very significant impact on both the localisation and the QA process. Text is one thing, but re-recording voiceover can be incredibly expensive, when doing so for many languages.”
This article, and apparently that panel, completely misses the entire point of public betas. Nearly all public betas have absolutely no intentions of collecting voluntary customer feedback on the beta product itself. Public betas are generally intended to achieve two highly valuable objectives: (1) Raising marketing awareness for a product; and (2) Large scale load testing and other usage/performance related data.
Both betas specifically mentioned in this article (LBP and Halo 3) benefited hugely from these two goals (not to mention the Halo 3 Beta sold a LOT of copies of Crackdown). I'd be willing to bet that both of these companies would argue fiercely in favor of the technical value and sales revenue that both of these open betas generated.
If done well these two things can be VERY effective, and obviously like all processes and programs - if done badly can be extremely detrimental (see Richard Garriott's comments on Tabula Rasa).
Long story short - Public beta isn't, and never was "using Joe Public for cheap QA". Anyone under that impression is way off base. That said, like all marketing programs (E3 Demos, Press Previews) - anything that requires extra work for development obviously complicates their life. So be it - without these things, nobody would know about (or buy) their games in the first place.
