World-anchored augmented reality, held steady inside a moving car. Turn-by-turn navigation registered through the windshield — onto the road, not the screen.
Conner Ward built the head-tracking pipeline for AR Drive, a Mercedes-Benz Carlsbad Design Studio research prototype (2024) that holds turn-by-turn navigation world-anchored on the road through AR glasses inside a moving car.
The hard problem was keeping an augmented-reality marker pinned to the world while the car moves and the driver's head turns. The approach fused three imperfect sensors — a Zed stereo camera, an IMU, and MediaPipe face landmarks — into one 3DOF head pose via a complementary filter, rendered world-space in Unity onto XReal AR glasses. It was an internal prototype carried across three Mercedes studios: AI experiences from Silicon Valley, the design language from Carlsbad, and the vehicle program in Stuttgart.
No single tracker survives a moving car. The camera lags. The IMU drifts. Glare blinds the face. So we stopped trusting any one of them.
MediaPipe face landmarks give a continuous head-pose readout — euler rotation, sixty times a second — and a fallback when sun defeats the stereo camera.
Three sensors, each wrong in its own way, resolved into one pose by a complementary filter — IMU on the high pass, stereo on the low.
The hard part was never fusion. It was calibration: a one-shot, three-point look that solves the rotation between every sensor's frame at once.
A marker pinned to the world, not the glass — it stays where the road is as the car rolls forward. The navigation reads as part of the road; the driver only sees that it works.
AI experiences out of Silicon Valley, the design language from Carlsbad, the vehicle program in Stuttgart — one tracking pipeline carried between all three.
AR Drive is a Mercedes-Benz Carlsbad Design Studio research prototype (2024) for world-anchored augmented-reality navigation inside a moving car. Turn-by-turn guidance is registered through AR glasses onto the road itself rather than a screen, staying pinned to the world as the car moves and the driver's head turns.
On Mercedes-Benz AR Drive, Conner Ward built the head-tracking pipeline and the sensor calibration — the work that fuses multiple in-car sensors into one stable head pose. The project was a collaboration across Mercedes' Silicon Valley AI experiences group, the Carlsbad Design Studio, and the Stuttgart vehicle program.
Mercedes-Benz AR Drive fused a Zed stereo camera, an IMU, and MediaPipe face landmarks through a complementary filter into a single 3DOF (rotation-only) head pose, rendered in world space in Unity and displayed on XReal AR glasses. Translation was fixed by the driver's seat, so only rotation needed tracking.
Mercedes-Benz