‘DeepMind unfairly compares AlphaStar and pro player StarCraft II’ performance
In graphs showing the match between a human pro-StarCraft II player and his AlphaStar-ai, Google DeepMind has apparently not fairly compared the two sides. The numbers of actions per minute of the two are compared, but not the effective actions per minute.
That is the conclusion of an extensive blog post by Aleksi Pietikäinen. He has been following StarCraft II and developments in artificial intelligence for years, but still calls himself a layman. First, he argues that a distinction must be made between apm, actions per minute, and epm, effective actions per minute; the first consists of clicks and keystrokes and the second only of clicks and keystrokes that pass a command that was not yet given. That is, the spam, the wild clicking, has been filtered out. In the heat of battle, a lot of people do, maybe everyone.
DeepMind charts the pro player’s apm, MaNa, and AlphaStar’s. It did so during the contest as well as in a blog post. This shows that the average apm of MaNa over the entire match is higher than that of AlphaStar. Pietikäinen notes, however, that the ‘tail’ of this apm chart presents a disingenuous picture; the times when the human player enters so much apm are the times when he spams. However, AlphaStar spams little or not and its tail is said to consist for a much larger part of effective commands.
These effective commands are the moments when the two players face off on the battlefield and the artificial intelligence wins using its superhuman micro, the ability to enter commands superhumanly fast to win a battle where a human wouldn’t. can. In the StarCraft II community, people generally agree that the AI didn’t win thanks to its strategy. Pietikäinen labels the chart on the blog post as “dangerously close to lying through statistics.”
AlphaStar, the AI of Google subsidiary DeepMind, defeated Polish professional player MaNa in five out of five matches on Thursday. However, the human player did win a live match.
The apm of MaNa, AlphaStar and AI trainer TLO, from DeepMind’s blog post