Apple Reveals New Game Controls UI and SDKs

Apple is working on introducing a virtual game controller UI for iOS 15, iPadOS 15, macOS, and tvOS. Apple is also extending support for the physical gaming controllers from the likes of major gaming developers. The controller will make it easier to capture images and videos from iOS devices while gaming. This announcement was made in Apple's Platforms State of the Union held after the World Wide Developers Conference 2021. Apple also announced that it will be easier to customise the controls as well.

Apple announced the new UI in its Platform State of the Union. An engineer discussed the following while unveiling the Apple's game controller UI, "These new on-screen controls for iPhone and iPad look amazing, and they're carefully tuned for grip locations across hand sizes and for a great responsiveness and feel. They're easy to add to your application and act just like a game controller. Instead of writing your own on-screen overlay UI and translating touch inputs into your game input system, these on-screen controls appear to your code just like game controller framework GCController objects, so your input logic can remain much more consistent."

Apple is also utilising ReplayKit to allow users to quickly capture the last 15 seconds of gameplay by long-pressing the share button. The engineer also mentioned, "As I showed you earlier, the system gestures for media capture are a double press to capture a screenshot to your Camera Roll, and a long press to start and stop a ReplayKit recording. This year, we're adding a super cool new type of media capture, 15-second highlights. So now, instead of having to remember to start and stop your recording, players can just turn on automatic background buffering, which will let them save the last 15 seconds of gameplay any time they long press."

The engineer also mentioned that it will be possible to customise the controller with your own symbols as integration has become easier with the colour and style of the developer's game. "They adjust to a variety of layouts depending on whether you want just one button, three buttons, or a D-pad and multiple buttons," said the Apple engineer. "The core principle of the on-screen controller is that the left and the right regions are independently configured and that the layout is determined based on the configuration."