Last night, Microsoft trumpeted the announcement of Muse, a new “generative AI breakthrough” designed to aid “gameplay ideation”.
The company also published some grainy-looking gifs of AI-generated gameplay footage, based on Xbox studio Ninja Theory’s multiplayer game Bleeding Edge. (The images were miniature in size, presumably to avoid highlighting some of the wonkiness AI is known for.)
Finally, Microsoft made the claim that Muse would “radically change how we preserve and experience classic games in the future”, and that the algorithm could be used to make older games compatible with “any device”.
Reaction to Microsoft’s announcement was swift, with social media awash with posts pointing out that, yes, Microsoft was absolutely jumping on the tech industry’s AI buzzword bandwagon – but also the suggesting that Xbox was about to start using Muse to pump out AI slop.
Thankfully, someone has done a better job than Microsoft itself of explaining what Muse is actually doing – and that person is AI researcher and game designer Dr Michael Cook.
For background, Cook is an expert on the subject of AI in games – he’s the guy who built an artificial intelligence to see if it could win a game jam a full decade ago, whose work Eurogamer has covered on several occasions. He’s also a senior lecturer at King’s College London, and has published and spoken extensively on the subject of AI.
As Cook lays out in an extended blog post on Muse, the AI model is not generating gameplay or creating its own original ideas.
In short, Muse was fed seven years of video footage of people playing a singular game – in this case, Bleeding Edge – to see if it could then generate further gameplay footage of it.
(If this all sounds familiar, it’s similar to the process Google used to generate footage of classic first-person shooter Doom last year.)
So, what’s the point of all this? Well, as Cook writes, it’s so Microsoft researcher could ask Muse to predict what might come next if changes to a game were made.
“They made a tool that let game developers edit a game level using existing game concepts like adding in a jump pad to a place where there wasn’t one before,” Cook explains.
“They then gave this new level to their model, and asked it to show what it thought the footage of a player playing from this new position would look like.”
In other words, the idea is Muse could be used as a shortcut tool for predicting and visualising how gameplay might adapt to a particular input by a developer. And, crucially, that developer is still a human.
Microsoft’s research paper on Muse says the AI model is required to understand persistency, consistency, and diversity in order to succeed. In other words, if a human input is provided, the AI needs to ensure the effects of that input remain, whatever else is going on, and that the effects remain similar while adapting to a range of player behaviours.
For example, say a human adds a jump pad to a game’s level. The paper says Muse must predict gameplay footage to reflect this which also ensures the jump pad persists (no deleting it!), that it reacts the same each time (it should always make players jump), regardless of what else is happening (this should apply even if a different player activates it).
“The paper is not really about ‘generating gameplay’ or ‘ideas’,” Cook continued. “It’s about these researchers thinking about the implications of how people will work with these tools.”
Notably, there’s nothing in the paper that discusses the implications of Muse being added to the development processes of teams with a strong aversion to using AI, Cook notes.
That said, in Xbox’s blog post on Muse yesterday, Microsoft gaming AI exec Fatima Kardar stated that the company had simply “empowered creative leaders here at Xbox to decide on the use of generative AI” – suggesting there was no specific mandate from Microsoft to use AI, for now.
But even with Muse being just used for a limited purpose, Cook has further questions around whether it will ever be fully viable.
“This is not a practical process,” Cook writes. “It’s impressive that it can do this using visual information because things like lighting, camera angles, user interface and so on are a lot for an AI model to handle.
“But ultimately, even with all of this data, all the time spent annotating datasets, and so on, it was still only just about able to generate footage predicting player behaviour.”
Such a system currently lacks any practical use, is enormously expensive, and requires any developer already have a vast vault of gameplay footage lying around for Muse to ingest in the first place.
“The research team behind this probably believe it will get more efficient over time, which might make it more affordable or tractable for small developers,” Cook continues. “However, it still raises the question of how we get video footage of people playing our game in the first place.
“If you’ve been in development for a couple of months then you won’t have enough footage, and even if we make the systems able to run on less input data there must be a minimum level required to understand the full game logic. So I think there is a question here not just of whether it makes sense as a tool now, but whether it can ever make sense.”
Lastly, on game preservation.
“You could imagine a world where, from gameplay data and video, a model could learn old games and really make them portable to any platform where these models could run,” Xbox chief Phil Spencer said yesterday in a video on Muse. “I feel that’s really exciting.”
Bluntly, Cook calls Spencer’s comments “idiotic”.
“I mean, in a sense anything is a preservation tool,” Cook writes. “I could ask my friend’s five-year-old son to draw a crayon picture of what he thinks the ending cutscene of Final Fantasy 8 looks like and that would still count as game preservation of a certain sort.”
Despite a decade of AI growth, Cook says, there’s no method yet to measure what exactly an AI model has captured and what it has not. Muse is able to provide grainy gifs of one fairly simple video game based on seven years of footage, but it is not a solution for holding everything about a game or every possible outcome of what players could do.
“This is absolutely not a solution for game preservation,” Cook concludes, citing a report by gaming archeologist Florence Smith Nicholls about the archiving of digital games. “What does it mean to preserve a gameplay experience? Even if this model was a perfect replication of the original executable software, this is not the be-all and end-all of game preservation. A generative model of what game footage maybe looked like once might be a nice curio on the side of a real preservation process, but it is always going to be inferior to other ways we approach the problem.”
Last month, Take-Two Interactive boss Strauss Zelnick weighed in with his opinion of AI, saying that “artificial intelligence is an oxymoron, there’s no such thing”.
fbq('init', '560747571485047');
fbq('track', 'PageView'); window.facebookPixelsDone = true;
window.dispatchEvent(new Event('BrockmanFacebookPixelsEnabled')); }
window.addEventListener('BrockmanTargetingCookiesAllowed', appendFacebookPixels);