SMART OBJECTS

SPEC-03.02 · BMWNA · EN-54 EXPERIENCE DESIGN LAB · INTERNAL · 2021 Pre-AI

Rear-seat object tracker for the car. A RealSense camera in the cabin feeds an AWS Rekognition Custom Labels model trained on Mechanical Turk – labeled frames; a Python tracker on a Jetson keeps state across detections; a small probabilistic model figures out what the passenger may have left behind and what destination it implies. Visualized on the iX’s rear-seat head unit.

Running prototype on the BMW iX rear-seat head unit — Smart Objects screen showing Live View with bounding boxes around detected items in two camera tiles, a Detected Objects list (rocket, wallet, dingo), and a Suggestions panel below.

BMWNA · EN-54 Experience Design Lab · 2021 · internal · no public repo

If a car knows what its passengers brought in — a racket, a backpack, a phone in the cupholder — a long list of cabin behaviors becomes possible: forgotten-object reminders at the door, climate routing, destination inference (you have a tennis racket on Tuesday at 6pm; the model knows you usually drive to the courts). The hard part wasn’t the model. It was getting enough labeled rear-seat imagery to train one, on a research timeline. SMART OBJECTS shipped an end-to-end pipeline that solved the labeling bottleneck by pushing the work to Mechanical Turk through SageMaker Ground Truth, ran inference through AWS Rekognition Custom Labels on a Jetson at the edge, and surfaced detections + suggestions on the iX head unit.

Stack

Intel RealSense D435: RGB stream from a camera mounted to the roof or rear-view mirror, framed on the rear seats. 640×360 at the frame rate the Jetson’s pipeline could keep up with.
NVIDIA Jetson (edge) · DeepStream: The on-vehicle compute. DeepStream wrapped the RealSense stream into the inference pipeline; the Jetson handled tracking + the probabilistic model locally so the head unit got results without a round-trip.
AWS Rekognition Custom Labels: The detection model itself — image bytes go up, labels + bounding boxes + confidence scores come back. Higher-level than a raw SageMaker training job; the abstraction was right for the timeline (mid-Sep 2021 prototype goal). Labels were a closed vocabulary: racket, ball, ball-tube, helmet, umbrella, backpack, bottle, phone, wallet.
SageMaker Ground Truth + Mechanical Turk: The labeling pipeline. Frames extracted from the RealSense camera were uploaded as a Ground Truth job; Mechanical Turk workers drew tight bounding boxes around the vocabulary objects using a written guide that handled motion blur and occlusion. Pricing landed around $0.036 per HIT with an ~8–10 second median.
Python tracker (self-written, no frameworks): Runs on the Jetson. Maintains state of objects across frames (identity + last-seen position + a detection score that ages out over time), triggers warnings when an object’s position changes significantly, and emits the state into the probabilistic model. No DeepSORT or off-the-shelf tracker — a small, legible state machine that knew exactly the few classes it had to handle.
Probabilistic model (self-written): Given the current detected objects, a current-situation context (time, weather, mocked passenger count, mocked location), and a learned context (calendar entries, inferred schedule, prior probabilities over the object vocabulary and start/end locations), it returns probabilities for objects that may have left the scene and proposes likely destinations.
Controller module: The on/off switch — ties detection to vehicle lifecycle events (onVehicleUnlock, onVehicleStart, onVehicleStop) so the system isn’t inferring on a parked, empty cabin.
Node web GUI (control panel): A single-page Node app on a notebook that mocks the car’s data feed for the prototype. Reads/writes YAML scenario files (time, weather, start/end locations, passenger count) and emits the lifecycle events the Controller listens for. For prototype v1.0, the car connection itself was mocked; the same surface could be wired to a real vehicle bus later.
Unity (head-unit visualization): The rear-seat screen UI. Live camera tiles with bounding boxes drawn around detected items, a Detected Objects list, and a Suggestions panel surfacing the probabilistic model’s output.

Labeling pipeline

The bottleneck was labels, not models. Public datasets have almost no rear-seat imagery; every class we cared about needed boxes drawn on our own footage. In-house labeling would have eaten weeks per training round and the EN-54 prototype window was eight weeks total. The pipeline had to be cheap, fast, and re-runnable when the vocabulary changed.

Why Mechanical Turk fit. Bounding-box drawing is the canonical MTurk task — well-scoped, no domain expertise needed (a racket is a racket to anyone), and worker output can be QA’d by consensus. The trust comes from the pipeline, not the worker: each frame went to multiple workers, and the written guide was specific about the failure modes that matter (motion blur, partial occlusion, overlapping objects of the same class) with good and bad examples for each.

SageMaker Ground Truth labeling job configuration — Amazon Mechanical Turk workforce selected, task timeout 1 hour, task expiration 6 hours, price per task $0.036, 8–10 second time estimate. — Ground Truth job configuration — Mechanical Turk workforce, $0.036 per HIT, ~8–10 seconds per box.

Label vocabulary panel in the SageMaker Ground Truth job: helmet, umbrella, wallet, racket, backpack, phone — six of the closed-vocabulary classes. — The closed label vocabulary configured in Ground Truth. Six of nine shown.

A reference image from the worker guide: rear-seat scene with multiple bounding boxes drawn around each object — racket, ball-tube, umbrella, helmet — including partially occluded ones. — Worker guide reference frame — tight boxes per object, edge-to-edge for objects that run off-screen.

SageMaker Ground Truth dataset view — two thumbnails of rear-seat car interiors with bounding boxes drawn around objects (a person, an item on the far seat). File names start with 2021-08-27 13_02_. — Returned dataset — rear-seat frames with worker-drawn boxes, ready for Rekognition Custom Labels training.

Synthetic augmentation. The Turk-labeled corpus was the seed; an HSV color-space augmentation expanded it without re-shooting. Same scene, same bounding boxes, hue-shifted within bounded ranges — cheap class robustness against the long tail of jacket / backpack / bottle colors a passenger might bring into the car.

Original frame — blue jacket on the rear seat of a BMW iX cabin. — HSV color-space augmentation — one labeled frame becomes a family. The jacket cycles through hues without re-drawing a single box.

HSV-rotated frame — the same jacket re-tinted yellow. — HSV color-space augmentation — one labeled frame becomes a family. The jacket cycles through hues without re-drawing a single box.

Architecture

An actual RealSense D435 frame captured inside the test vehicle — a person in the rear seat reaching across to the next seat. — A real D435 frame from inside the test vehicle — the input that drove training and live inference.

Probabilistic model

The tracker emits a per-object detection score that ages over time and weights the current confidence — the basis for “was that object actually there or did we miss a frame?” The probabilistic model on top then combines the live detection set with the current situation (time, weather, location, passenger count) and the learned context (calendar entries, inferred schedule, priors over the vocabulary and start/end locations) to return two things: probabilities for objects that may have left the scene, and a suggested destination consistent with the cabin’s contents.

detectionScore equation: detectionScore = ((5 / (1 + delta-t)) * 0.8) + ((c-prime over c) * 0.2) — Per-object score — recency dominates (0.8 weight on the time-decay term), with the current/prior confidence ratio as the secondary signal (0.2 weight).

Prototype

For prototype v1.0 the car’s data feed was mocked by a notebook running a Node single-page app. The same app emits the lifecycle events the on-vehicle Controller is listening for, so the rest of the pipeline runs unchanged whether the events come from a YAML scenario or a real bus.

SmartObjects 'Current Scenario' web GUI: Time and day (Tue 9/07/21 9:30 am), weather (cloudy 68 F start, sunny 72 F end), car events (Unlock/Lock/Start/Stop buttons), passengers (1), and locations (San Francisco to Mountain View). — Control panel — a populated scenario (SF to Mountain View, cloudy to sunny, one passenger) driving the YAML config the probabilistic model reads.

BMW concept render of a phone resting on the rear armrest with a glowing visual outline detecting it. — Early concept render of the rear-seat detection moment — the visual target the head-unit UI was working back from.

What this is really about. “Pre-AI” — this project belongs to the era right before foundation models swallowed the perception stack. The exact detection task today is one line through a VLM. What still matters is the infrastructure pattern: outsource the bottleneck step (labels) to the cheapest correct workforce, keep the iteration loop short enough that the team proposes a new class on a Monday and ships an updated model by Friday, and measure the loop rather than the model. The model gets replaced; the way of working doesn’t.

Built at BMWNA EN-54 Experience Design Lab, 2021, with Anderson and Francesco.