How SPYSCAPE Added Magic Into Their AR Game Using Nyckel

mugshot Oscar Beijbom
Aug 2022
Spyscape hero

SPYSCAPE is a destination spy museum and experience in New York City, with interactive exhibits about fictional and real spies. The museum’s permanent exhibits include interactive features and immersive games through which visitors can figure out what kind of spy they would be. Are you a handler or a technician? Special ops or cryptographer?

“Normally you have a problem and then you go for a solution... What I’ve now got is a really nice solution, and we’re actually looking for other problems we can apply it to!” – Gwyn, SPYSCAPE Tech Lead
HIGHLIGHTS
- SPYSCAPE wanted to add something that felt `magic` to their AR game.
- Decided on an camera based game interaction.
- Trained and deployed a Nyckel custom image classifier in 10 minutes.
- Function was "spooky-accurate" right away.
- Nyckel-powered magic camera a game highlight.

The HQ also features a rolling exhibit. 007 x SPYSCAPE has recently moved out, and a physical component of a new Batman x SPYSCAPE mobile augmented reality game has moved in. Batman x Skyscape can be played anywhere in the world, but, if you happen to be in NYC, the game will also take you through the exhibition. It gives you the immersive experience of actually working with Batman and the characters from that universe.

You are playing an important part of that mission. We’re trying to put you in the story.” – Gwyn, SPYSCAPE Tech Lead

What you do in the game moves the story forward. Although you’re in fact communicating with software, SPYSCAPE wants it to feel like you’re communicating with real people. But the question was how to do this at scale? The game already used a large branching script about the length of a typical novel. This allows the characters to react to the choices that players make in the game, which begins to establish the feeling of reality.

Spyscape UX
“But we really wanted to have that magic: making it seem like, how could they have done that? It must be a person!” – Gwyn, SPYSCAPE Tech Lead

The physical exhibits in New York use RF wristbands and NFC to maintain a personalised sense of reality throughout the game, producing an immersive magic that resonates with customers. But another powerful element of the HQ’s immersiveness comes from its use of picture recognition.

“What we were looking to do was basically Things That Are Magic!” – Gwyn, SPYSCAPE Tech Lead

At one point in the exhibit, visitors have to go into a laboratory and take pictures with the Batcam. SPYSCAPE needed an AI to classify the photographs that visitors were taking within the game in order for the game to react to them in a realistic way.

“People love the magic camera!” – Gwyn, SPYSCAPE Tech Lead

SPYSCAPE already had a lot of infrastructure in AWS, so that was their first thought. But there were lots of competing priorities within the company at the time, and it didn’t make sense to take on a whole raft of machine learning researchers just to solve this problem. Outsourcing made sense, and that’s when Tech Lead Gwyn saw an advertisement for Nyckel on Hacker News.

“Literally over the space of ten minutes, I created an account, threw some random images at it… and it blew my mind! It was spooky-accurate!” – Gwyn, SPYSCAPE Tech Lead

There was very little to do in terms of API integration: a dozen or so example images per class of training data, uploaded to the Nyckel web interface; and a server written in Node, deployed in AWS, to do a little bit of munging of the output.

That’s spooky magic! That feels real!” – Gwyn, SPYSCAPE Tech Lead

Gwyn and his team are so pleased with the magic of the ML-powered Batcam that they are now looking for other use cases within Skyscape.

“Thanks for building this thing – it’s super-cool!” – Gwyn, SPYSCAPE Tech Lead