Does Reviewing A Game Before Its Release Date Really Matter?
Updated: Jan 23, 2018
The publisher Bethesda announced this past week that it was no longer going to share advance copies to reviewers far before release, as many publishers have for big titles in the past. Instead, the company will share review copies only one day before release, because, says Bethesda, it wants “everyone, including those in the media, to experience our games at the same time.”
Across the Internet, there have been a number of reactions to this news, from press sites attacking the move as anti-consumer, to other publishers also keeping their review copies to themselves until release day. 2K Games declined to send out Mafia 3copies in advance, and other publishers have elected to keep their review copies out of reviewers’ hands until release day as well.
EEDAR has lots of access to relevant data, and we’re very interested in the best ways to publish and develop games (we help lots of publishers across the industry in all kinds of different ways). So we decided to pull the numbers and ask the question: Does reviewing a game before its release date really matter?
From GamePulse, we pulled the review scores and release dates of all physically released, eighth generation titles in 2015 with a Metascore (including those on PS4, Xbox One, and the Wii U). We wanted to see a full year’s worth of data, and we focused on physical HD releases because that market is the one where a critical Metascore seems to matter most. The goal was to look at a broad set of titles during a specific time period to see if any trends emerged.
We then compared the release dates of those games to when their earliest reviews appeared on Metacritic. In many cases, reviews appear on the same day on many different sites, because publishers agree with the press to “embargo” their reviews for a specific time (to ensure that all reviews hit at the same time, and create the biggest reaction possible). Some reviews on Metacritic can appear as far as a month or two after release (as some reviewers take a long time to experience the game and write up their verdicts), but we recorded the likely embargo date, or at least the clearest set of early reviews, for each title in the set.
When we compared the average review scores on each day with the days relative to the release date, a surprisingly clear trend emerges: The earlier reviewers get to see a game, the higher the average review score that they give to it.
Within the range of review scores from 15 days pre-release to 15 days post-release, and from the earliest reviews to the latest, there’s a very clear trend downward among average review scores. According to these statistics, the earlier reviewers get to see and review a title, the higher score they tend to give it.
Of course, correlation doesn’t imply causation, and these numbers certainly don’t mean that any game given early to reviewers will automatically get a high score. EEDAR studies and tracks many factors that go into a review score, and we routinely predict and make suggestions to improve review scores for publishers and developers based on those factors.
There are, however, a number of reasons why this trend might be evident:
Game developers that are able to get their games finished and ready to review before release day may just be better at being organized and hitting production deadlines, and thus better at coordinating their resources and creating more polished and entertaining titles.
Reviewers who get to see a game early may be excited to do so, and thus give it a more positive score regardless of the actual content.
Publishers may choose to show off titles early that they know are high quality, in order to increase player interest and pre-orders based on good early reviews.
Some reviewers may be incentivized to go through content more quickly to finish reviews early, giving them less time to find problems and uncover issues with the title.
Any problems caused by server issues or multiplayer gameplay may not be experienced pre-release (before the servers are full of players), which may trend the scores higher overall.
It should also be noted that these “review dates” are when the review appeared online, not necessarily when the reviewer received their copy from the publisher. Reviews that appear on release day, in this case, may have been given to the press days or weeks before they showed up online, or not. In general, however, the earlier a review is posted, the earlier an outlet has received its copy of the game. Reviewers who are able to post reviews on release day or earlier usually do so, because that’s when public interest is usually highest for the review content.
While the reasons for this data aren’t clear, there is definitely a clear trend that games reviewed earlier compared to release day tend to score more highly. EEDAR also examined some preliminary information for a select group of 2016 games, and while that group of data was much smaller (and didn’t include any holiday releases yet), the trend has continued this year as well.
Bethesda certainly has the right to give its advance copies to anyone, whenever it chooses to, whether the press has problems with the lack of early access or not. The data shows, however, that the earlier that reviews for titles are posted online, the higher the scores that they’ve historically earned.
Manager, Game Evaluation