Dataset & benchmark · human–automation transitions · arXiv 2026
Control transitions observed in both directions on the same synchronized streams — and takeover scored twice: onset detection (T3-D), where the driver's inputs are visible, and pre-override anticipation (T3-A), where they are withheld.
781 real-world driving-automation routes from 173 drivers across 108 vehicle models, 204.9 hours. Front video, in-cabin video, decoded CAN, radar lead interaction and route context form one record around every handover and takeover. The frozen benchmark subset holds 565 routes, 162.1 h and 3,593 events.
1University of South Florida · MOTIF-Lab 2Tongji University 3University of Arizona
Frozen benchmark on Hugging Face: 565 routes · 162.1 h · 150 drivers · 3,593 events (1,800 ↑ handovers / 1,793 ↓ takeovers). The arXiv listing abstract still shows the v1 figures (136.6 h, 127 drivers, 380 routes); the v2 paper body (27 Jul 2026) and the Hugging Face card describe the current release.
The v2 abstract, verbatim, beside what you need to know at a glance.
Existing Level-2 driving-automation (DA) systems on production vehicles still rely on human drivers to decide when to engage automation, and ask for drivers' continuous attention and readiness to intervene in case of emergency. This human–machine-interface (HMI) design demands good situational judgment and imposes high cognitive loads, producing a steep learning curve for new drivers, poor DA user experience, and possibly increased safety risks. Improving DA HMIs hinges on accurately predicting when drivers hand control to automation and when they take it back, but no existing resource jointly captures both directions of driver–automation transitions with synchronized road, cabin, vehicle-control, and route observations at this scale. To fill this gap, we introduce BATON, a large-scale multimodal dataset of 781 real-world DA routes from 173 unique drivers across 108 vehicle models, spanning 204.9 hours of driving. BATON synchronizes front-view video, in-cabin video, decoded CAN (Controller Area Network) signals, radar-based lead-vehicle interaction, and GPS-derived route context into one record around each control transition.
The benchmark separates transition detection from anticipation: alongside handover prediction and an auxiliary action-recognition task, takeover is evaluated under two official protocols, onset detection (T3-D), where driver inputs are observable, and pre-override anticipation (T3-A), where every driver-override channel is withheld and windows end before the override begins. We evaluate baselines spanning gradient-boosted trees, sequence models, cross-modal and hierarchical Transformers, and a frozen V-JEPA2 video encoder; zero-shot vision–language models are also benchmarked, all under a leakage-audited protocol. Results show that i) multimodal context helps most on handover, where video world-model features raise AUPRC by 42% over the strongest tabular baseline; ii) takeover onset detection is driven largely by observable driver-input cues; and iii) under the anticipation protocol all baselines score close to the base rate, indicating that the remaining signals carry little anticipatory information, establishing BATON as a rigorous benchmark for multimodal driver–automation transition modeling.
A public sample route: the driver engages the assistance system (↑ handover), drives with it, then takes control back (↓ takeover). Front camera and 10 Hz signals on one clock; the shaded stretch is the assistance system engaged; the two events are the benchmark's official ones.
A production system already carries a disengagement predictor. The upper trace is openpilot's own probability of a disengagement within two seconds, recovered from the route logs, with its green and yellow alert levels; the step series is BATON-WM's window score with its decision threshold. Zero is the moment the driver overrides.
The same XGBoost readout scores 0.479 sample AUPRC on T3-D and 0.056 on T3-A; BATON-WM 0.514 and 0.070 — against base rates of 0.120 and 0.035. Observing the driver's action is most of the detection signal; anticipating it from the remaining streams is largely unsolved.
At 1 false alarm per hour, the best T3-D model recalls only 28% of takeover events (median earliest warning 3.0 s, the horizon cap); the best T2 model recalls 7% of handovers.
Takeover onset detection: 4.3× the base rate, carried by observable driver inputs.
Cross-driver, h = 3 s, 3-seed mean; bar = sample AUPRC, second value = event AUPRC. The CAN-statistics XGBoost (.479) beats every neural multimodal model; adding driver pose makes BATON-WM (.514).
Handover prediction is where multimodal context helps most.
Video world-model features raise the tabular baseline from .236 to .335 (+42 %); base rate .148. Handovers depend on immediate context — the road ahead — more than on the driver's hands.
Anticipation is 2.0× the base rate — honest, and small.
With every driver-override channel withheld and windows ending before the override, all baselines stay within .033–.070. Distillation from a detection teacher that sees the override during training does not lift it meaningfully. The remaining signals carry little anticipatory information.
Base rates: T2 .148 / .037 · T3-D .120 · T3-A .035 (sample / event). Operating point: at 1 false alarm per hour, the best T3-D model recalls only 28% of takeover events (paper v2).
A deployed predictor leaves most of the task unsolved.
openpilot's own disengagement predictor, scored on the 26 benchmark routes where its outputs exist (5 in the test split): .293 / .101 sample / event AUPRC on T3-D, above the .122 base rate but far below the tabular baseline on the same rows (.771 / .732) (paper v2).
Scope, stated plainly.
The corpus: 781 routes, 173 drivers, 108 models, 204.9 h; the benchmark is defined on a frozen, quality-filtered subset of 565 route bundles (162.1 h, 150 drivers, 99 models) with 3,593 events — 1,800 handovers and 1,793 takeovers. Two scope facts from the paper: a single configuration accounts for 96% of all transitions (openpilot lateral + stock adaptive cruise), and because each driver drives their own vehicle, the cross-driver split measures driver and platform generalization jointly.
A dash-mounted recorder on the driver's own car logs road and cabin video, decoded CAN, radar, the driver monitor and the planner; every control transition becomes one time-aligned record. The benchmark freezes the subset, the splits and a leak-safe field list.
Main transition results · cross-driver · h = 3 s · sample / event AUPRC · 3-seed mean · arXiv v2
| Method | Input | T2 handover | T3-D takeover detection | T3-A anticipation |
|---|---|---|---|---|
| Base rate | — | .148 / .037 | .120 / — | .035 / — |
| GRU | struct | .202 / .072 | .280 / .119 | — |
| TCN | struct | .190 / .058 | .332 / .167 | — |
| Cross-Modal Transformer | struct | .242 / .103 | .316 / .135 | — |
| V-JEPA2 fusion | video + struct | .235 / .089 | .329 / .161 | — |
| RG-HBT-Q | struct + video | .254 / .111 | .398 / .230 | .058 / .026 |
| DI-RG-HBT-Q | struct + video | .219 / .116 | .366 / .215 | — |
| XGBoost | CAN statistics | .236 / .103 | .479 / .380 | .056 / .026 |
| BATON-WM | multimodal | .335 / .171 | .514 / .413 | .070 / .040 |
Operating points and the deployed predictor · arXiv v2
| Measure | Value |
|---|---|
| Event recall at 1 false alarm / hour · best T3-D model | 28 % (median earliest warning 3.0 s = horizon cap) |
| Event recall at 1 FA/h · best T2 model | 7 % |
| Event recall at 1 FA/h · T3-A | ≤ 2 % |
| openpilot's deployed predictor on T3-D (26 covered routes) | .293 / .101 sample / event AUPRC (base rate .122) |
| Tabular baseline on the same rows | .771 / .732 |
| Protocol ladder (XGBoost): full input → leak-safe → anticipation-safe, T3 | .520 / .428 → .478 / .385 → .056 / .026 |
Zero-shot vision–language models · T1 macro-F1, T2 and T3-D AUPRC · leakage-inflated diagnostic · arXiv v2
| Model | T1 F1M | T2 | T3-D |
|---|---|---|---|
| Gemini-2.5-Flash | .362 | .184 | .202 |
| GPT-4o | .303 | .165 | .196 |
| Qwen3-VL-8B | .280 | .166 | .177 |
| Qwen3-VL-4B | .212 | .173 | .167 |
| MiniCPM-V-4.5 | .170 | .174 | .182 |
Honest limits. 96 % of transitions come from one assistance configuration; drivers drive their own vehicles, so driver and platform generalization are measured jointly; anticipation (T3-A) stays near the base rate for every baseline; the four case studies are hand-picked and openpilot predictions exist for only 26 routes.
Two drivers, 45 routes, all nine modalities — including the in-cabin video, under the same license. The clip on this page is one of these routes (front camera only).
as of 2026-10-02 · gated: auto · CC BY-NC 4.0
565 routes · 162.1 h · 150 drivers · 3,593 events; benchmark protocol, splits and leak-safe field list included.
as of 2026-10-02 · gated: manual · CC BY-NC 4.0
“This dataset is released for academic research use only under CC BY-NC 4.0.”Hugging Face dataset card · fetched 2026-10-02
Terms. Non-commercial research use; no re-identification; the in-cabin video stays under the gated license and is never redistributed in the open; cite the paper.
@article{wang2026baton,
title = {BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving},
author = {Wang, Yuhang and Xu, Yiyao and Yang, Chaoyun and Li, Lingyao and Sun, Jingran and Zhou, Hao},
journal = {arXiv preprint arXiv:2604.07263},
year = {2026},
note = {arXiv:2604.07263v2}
}
