SMART OBJECTS
Rear-seat object tracker for the car. A RealSense camera in the cabin feeds an AWS Rekognition Custom Labels model trained on Mechanical Turk–labeled frames; a Python tracker on a Jetson keeps state across detections; a small probabilistic model figures out what the passenger may have left behind and what destination it implies. Visualized on the iX’s rear-seat head unit.
Conner Ward built Smart Objects, an in-cabin rear-seat object tracker for BMW of North America, using an Intel RealSense camera, an NVIDIA Jetson, and AWS Rekognition Custom Labels, with the training data labeled via Mechanical Turk through SageMaker Ground Truth.
A RealSense camera in the BMW iX cabin feeds a Rekognition Custom Labels model trained on Mechanical Turk–labeled frames; a Python tracker on the Jetson keeps state across detections, and a small probabilistic model infers what a passenger may have left behind. Conner Ward built it in 2021 at BMW North America's EN-54 Experience Design Lab; it is internal work with no public repository.
What’s in the box?
If a car knows what its passengers brought in — a racket, a backpack, a phone in the cupholder — a long list of cabin behaviors becomes possible: forgotten-object reminders at the door, climate routing, destination inference (you have a tennis racket on Tuesday at 6pm; the model knows you usually drive to the courts). The hard part wasn’t the model. It was getting enough labeled rear-seat imagery to train one, on a research timeline. SMART OBJECTS shipped an end-to-end pipeline that solved the labeling bottleneck by pushing the work to Mechanical Turk through SageMaker Ground Truth, ran inference through AWS Rekognition Custom Labels on a Jetson at the edge, and surfaced detections + suggestions on the iX head unit.
The plan
onVehicleUnlock, onVehicleStart, onVehicleStop) so the system isn’t inferring on a parked, empty cabin.Labeling pipeline
The bottleneck was labels, not models. Public datasets have almost no rear-seat imagery; every class we cared about needed boxes drawn on our own footage. In-house labeling would have eaten weeks per training round and the EN-54 prototype window was eight weeks total. The pipeline had to be cheap, fast, and re-runnable when the vocabulary changed.
Why Mechanical Turk fit. Bounding-box drawing is the canonical MTurk task — well-scoped, no domain expertise needed (a racket is a racket to anyone), and worker output can be QA’d by consensus. The trust comes from the pipeline, not the worker: each frame went to multiple workers, and the written guide was specific about the failure modes that matter (motion blur, partial occlusion, overlapping objects of the same class) with good and bad examples for each.
Creating synthetic data. The Turk-labeled corpus was the seed: bounded by vocabulary, by the eight-week prototype window, and by the per-HIT cost on Mechanical Turk. Synthetic augmentation is how a seed corpus becomes a training corpus — programmatically transforming labeled frames into new training examples without paying for new labels. One labeled frame turned into three; the bounding boxes carry over unchanged because the transform doesn’t move pixels, only re-maps their colors.
Why HSV hue rotation specifically. An in-cabin object detector fails most often on objects it has seen the geometry of but not the color of. A blue Patagonia jacket and a yellow Patagonia jacket are the same shape under the same camera, and the network shouldn’t care — but on a small dataset it learns “jacket = blue” by accident. Rotating hue in HSV space (instead of jittering RGB channels) keeps lightness and saturation intact, so the augmented frames look like physically plausible alternate items rather than the over-saturated, blown-out versions you get from naive RGB perturbation.
What we deliberately did not augment. No flips, no random crops, no perspective warp — the camera is bolted to the rear-view mirror and the rear-seat geometry is fixed, so synthetic geometric variation would teach the model a distribution that doesn’t exist in deployment. No brightness/contrast jitter either: time-of-day variation comes free from the recorded footage and we wanted the network to learn it from real data, not synthetic. The augmentation budget went entirely into the one axis the real corpus was thinnest on — the color of a passenger’s carry-on.
One labeled frame — three training examples. The bounding boxes are invariant to hue rotation, so MTurk pays once and the dataset triples.
Probabilistic model
The tracker emits a per-object detection score that ages over time and weights how many cameras currently see the object — the basis for “was that object actually there or did we miss a frame?” The probabilistic model on top then combines the live detection set with the current situation (time, weather, location, passenger count) and the learned context (calendar entries, inferred schedule, priors over the vocabulary and start/end locations) to return two things: probabilities for objects that may have left the scene, and a suggested destination consistent with the cabin’s contents.
Δt = frames since last detection · c′/c = cameras seeing the object / total cameras in the cabin. Recency dominates (0.8); multi-camera agreement is the secondary signal (0.2).
In-car prototype
For prototype v1.0 the car’s data feed was mocked by a notebook running a Node single-page app. The same app emits the lifecycle events the on-vehicle Controller is listening for, so the rest of the pipeline runs unchanged whether the events come from a YAML scenario or a real bus.
What this is really about. “Pre-AI” — this project belongs to the era right before foundation models swallowed the perception stack. The exact detection task today is one line through a VLM. What still matters is the infrastructure pattern: outsource the bottleneck step (labels) to the cheapest correct workforce, keep the iteration loop short enough that the team proposes a new class on a Monday and ships an updated model by Friday, and measure the loop rather than the model. The model gets replaced; the way of working doesn’t.
FAQ
What is Smart Objects?
Smart Objects is an in-cabin rear-seat object tracker that Conner Ward built for BMW of North America in 2021 at the EN-54 Experience Design Lab. A camera in the BMW iX cabin detects and tracks objects on the rear seat — a racket, a backpack, a bottle — and infers what a passenger may have left behind.
What was Conner Ward's role on Smart Objects?
Conner Ward built the end-to-end Smart Objects pipeline at BMW of North America's EN-54 Experience Design Lab — the in-cabin camera capture, the object-detection model, the labeling loop, the Jetson-side tracker, and the rear-seat head-unit visualization on the BMW iX.
What technology did Smart Objects use?
Smart Objects used an Intel RealSense camera, an NVIDIA Jetson for on-board inference, and AWS Rekognition Custom Labels for object detection, with training data labeled via Amazon Mechanical Turk through SageMaker Ground Truth. Conner Ward built it for BMW of North America in 2021.