A really big playtest failure

"Playtesting is sovereign", I like to say to students. But it has taken a long time for videogame companies to recognize this, and there're still examples of complete failures to do playtesting right even when they play test.

An example from last year is Final Fantasy XIV. Final Fantasy games are normally praised but this one has been trashed by reviewers. It's score on Metacritic (an aggregator of reviews) is 49, which is really bad for a videogame. http://www.metacritic.com/game/pc/final-fantasy-xiv-online/critic-reviews

Here are some excerpts:

"It may be hard to believe, and painful to accept, but the last 'official' chapter of Final Fantasy is a huge disappointment, and it not only becomes the worst chapter of the saga, but also one of the worst MMORPG we have played in a long time. It's obvious that Square-Enix is no longer what it used to be."

"Final Fantasy XIV is in no way, shape, or form ready for commercial release. We suppose there is potential down the line for the experience to improve with patches and tweaks, but that doesn't justify why the game is in such shambles at launch. "

GameSpy (gave a 40). "Extended stretches of confused incredulity, punctuated now and then by bursts of intense anger. And yeah -- I just equated FF XIV with a filthy bathroom."

PC Gamer (gave a 30). "A shallow, slow, grind-heavy MMO crippled by a horrible interface and nonsensical player limitations."

So didn't they playtest it? Yes, they did! I found evidence of an open beta: (http://www.ffxivvault.com/ffxiv-beta-information.php and http://www.wired.com/gamelife/2009/12/final-fantasy-xiv-beta-test-now-recruiting/). "Anyone that wishes to be among the first to play the new MMORPG, which will launch on PlayStation 3 and PC in 2010, can sign up at the official beta site. Square Enix says that no prior MMO experience is required, and players selected to be in the beta will test the Windows version of the game."

So what happened? Did they test only with fanboys and girls? Did they ignore the results? It seems unlikely that they ran out of time and put the game out before was ready, because Square Enix is one of the more respected and consistently good videogame publishers.
So what can happen to give a "false positive" from playtesting?

First, you could be playtesting with people who know the game so well that the problems just don't register. But this would be likely only if the people within the studio were the playtesters, and in this case it was an open beta.

Second, you can have open playtesting but somehow you only get fanboys and fangirls, people who will say good things about your product no matter what. The have lost their critical faculties where you are concerned. Perhaps this is what actually happened. This is why you want to try to get many playtesters who don't know your products and you aren't prone to this kind of thing.

Third, you could just ignore the results, saying "oh well we know it's good". I'm convinced that Microsoft has done this quite frequently, adding smugly "we know what's best". There can be many examples but in the video game category the most famous one is the huge controller that Microsoft included with the original Xbox. Apparently it was sized for people my height (2 meters) with correspondingly large hands. According to what I've read Microsoft continued to maintain that it was fine but finally changed to a much more normal sized controller.

Fourth, perhaps they asked the wrong questions and yet didn't pay attention to the actual play. Frequently if you ask people what they think you'll get a different story than if you simply watch them and listen to them without interfering. Jakob Nielsen, the guru of web usability, always cautions you to watch people, not asked them what they will do or what they have done, because you get better data.