AI-enabled user interface for cloud gaming

The tripartite collaboration represents the first time that an AI and machine learning-driven user interface is utilized, wearable-free, with a live cloud gaming solution. The technology demonstrates the incredible potential of integrating localized and far-Edge enabled AI capabilities into home gaming experiences. 

The AI and machine learning-enabled user interface is connected to a cloud gaming solution that operates without joysticks or wearable accessories. The demonstration leverages unique technologies, including real-time video analysis on home and local edge devices, dynamic adaptation to available compute resources, and shared AI models managed through an in-home AI hub, to implement a cutting-edge gaming experience.

In the demonstration, users play a first-person view snowboarding game streamed by Blacknut and displayed on a commercial television. Users do not require a joystick or handheld controller to play the game; instead, their movements and interactions are tracked by AI processing of the live video capture of the users’ movements. The users’ presence is detected using an AI model and his or her body movements are matched with the snowboarder in the game, in real time, using InterDigital’s low latency Edge AI running on a local AI accelerator. The groundbreaking demo addresses the challenges of ensuring the lowest possible end-to-end latency from gesture capture to game action, while accelerating inference of concurrent AI models serving multiple applications to deliver an interactive and more seamless gaming experience. This demonstration enables AI and machine learning tasks to be completed locally, revolutionizing our current implementation of cloud gaming solutions.

“We are so proud of the work of this demonstration, as it displays the real potential of AI and edge computing, highlights the power of industry collaboration, and helps blaze a trail for new cloud gaming capabilities. Of course, such a success would not have been possible without the utmost implication of all the teams from Interdigital, Blacknut, and Nvidia, and I would like to take the opportunity to credit and thank their outstanding work,” said Laurent Depersin, Director of the Home Experience Lab at InterDigital.

The far-Edge AI and machine learning technologies put forth by InterDigital bring a plethora of new capabilities to the cloud gaming experience. Far-Edge AI enables low-latency analysis to deliver an interactive and entertaining experience, reduces cloud computing costs by leveraging available computing resources, and saves significant bandwidth by prioritizing up-linking. In addition, far-Edge AI in edge cloud architecture offers an important solution for privacy concerns by localizing computing and supports a variety of new and emerging vertical applications beyond gaming, including smart home and security, remote healthcare, and robotics.

Cloud gaming with far-Edge AI leverages artificial intelligence and localized Edge computing to showcase the ways an interactive television or gaming experience can be enhanced by the localized AI analysis of a camera’s video stream. Ongoing research in the real-time processing of user generated data will drive new innovations and vertical applications in the home, from cloud gaming to remote medical care, and those innovations will be enhanced by the ability to execute artificial intelligence models under low latency conditions.

“Blacknut’s mission is to bring to our customers unlimited hours of gaming fun in the simplest manner,” said Pascal Manchon, CTO at Blacknut. “Our unique cloud gaming solution allows to free games from dedicated consoles or hardware. Using AI and machine learning to transform the human body itself in a full-fledge game controller was challenging but Blacknut’s close collaboration with Interdigital and NVidia led to outstanding performances. And yes, it is addictive and fun to play this way!”