Apple Vision Pro: Will it feature an Apple Pencil-like controller?

Apple positions the Vision Pro as a mixed-reality headset that operates entirely without traditional controllers. Gestures, gaze tracking, and voice commands are intended to play the central role. However, new patent documents reveal that an additional input device is being developed internally. This device is a hand controller that is visually and functionally reminiscent of the Apple Pencil, but is intended to be significantly more than a simple stylus.

Apple patents are deliberately broadly worded and not synonymous with concrete product announcements. Nevertheless, they offer valuable insights into strategic considerations. In the case of the Apple Vision Pro, it becomes clear that Apple hasn't yet exhausted its understanding of physical input devices. Especially with precise movements, creative applications, and complex interactions, pure gesture control reaches its limits. This is precisely where the new patent proposal comes in.

A hand controller instead of a classic pen

Although Apple emphasizes that the Apple Vision Pro does not require a hand controller, the new patent describes a device that outwardly resembles an oversized Apple Pencil. It is less a writing instrument and more a fully-fledged hand controller. The casing is elongated and marker-shaped, designed for a secure grip.

Image: US Patent and Trademark Office / Apple

The patent describes the device as a hand controller with an elongated, marker-like housing. It is designed to extend across the width of the hand and can be held like a pen, pencil, marker, wand, or tool. Its shape appears bulkier than a classic Apple Pencil, emphasizing its focus on control and stability.

Use beyond Apple Vision Pro

Apple isn't solely focused on headsets with this controller. According to the patent, the device should be compatible with a wide range of electronic products. These include head-worn devices like the Apple Vision Pro, as well as smartphones, tablets, laptops, smartwatches, speakers, and other electronic systems.

The controller is thus conceived as a universal input device, not tied to a single product category. This openness aligns with the long-term strategy of ensuring consistent input methods across different devices.

Virtual tips and visible tools

The controller becomes particularly interesting when used with the Apple Vision Pro. In this scenario, the headset can overlay virtual content onto real-world environments. The hand controller itself has a physical tip area onto which a computer-generated tip is projected. This virtual brush tip or tool head is only visible on the display.

Apple explicitly describes a computer-generated brush head that is placed on the controller. They also mention a more general computer-generated tool head. This opens up many application possibilities, such as drawing in the air, precisely placing objects, or manipulating complex user interfaces in space.

Technology inside the controller

The device is said to contain extensive sensors. Among these is an inertial measurement unit with an accelerometer. This can detect movements such as swiping, rippling, writing and drawing motions, shaking, and rotations.

Wireless communication capabilities are also included. Tracking technologies also play a role. Apple mentions active or passive visual markers that can be detected by optical sensors on an external device. This would allow the controller's position and movement in space to be tracked very precisely.

Patent scope and participating developers

As is typical with patents, the proposal is formulated as broadly as possible. It generally describes a handheld controller with a housing that has an elongated shaft extending between a first and a second tip section. However, the example images shown mostly depict a kind of solid Apple Pencil with a conspicuously large physical tip.

The patent is attributed to five inventors. One of them is Yuhao Pan, who previously worked on a concept using ultrasonic sensors to authenticate user voices. This shows that Apple is working internally on very different, but interconnected, input technologies.

Apple is thinking beyond gestures with Vision Pro

The Apple Vision Pro is officially marketed as a device without a controller. However, the recently granted patent shows that Apple continues to work intensively on physical input solutions. An Apple Pencil-like hand controller with virtual tool tips could be a useful addition, especially for precise and creative applications.

Whether this concept will ever become a product remains to be seen. What is clear, however, is that Apple realistically assesses the limitations of gesture control alone and is keeping all options open. This controller could play a crucial role in the long-term development of the Apple Vision Pro. The best products for you: Our Amazon storefront offers a wide selection of accessories, including those for HomeKit. (Image: Shutterstock / Hadrian)