Skip to content

Driver monitoring · world model · real time

TriDriveJoint Driver, Vehicle, and Road Modeling for Forecasting and Driver Monitoring

To our knowledge the first unified framework that jointly forecasts driver kinematics, vehicle dynamics and road demands through one automation-conditioned transition model — and it runs in the car at 5 Hz.

Three streams enter one latent state: the driver's anchored kinematics, causal CAN-bus dynamics, and frozen V-JEPA 2 road latents with structured road margins. Directed residual connections let the driver and the road refine the vehicle forecast; a lightweight probe turns the current state into a warning.

arXiv 2609.33000 · 2026 · under reviewcomma four + 8 GB GPU · 5 Hz probeAIDE · BATON · on-road study
Explore the forecast arXiv Paper Code & model

Yuhang Wang1·Jingxin Yang2·Chuheng Wei3·Yuechen Guo1·Jinghan Xu4·Zhao Han1·Hao Zhou1

1University of South Florida · MOTIF-Lab   2NVIDIA   3Purdue University   4Hunan University

● REC
Cabin and road view from the lab recorder at the moment the driver picks up a phone
live · lab recorder, consented · 10 Hzrisk + forecast fan
48.05All-MPJPE on AIDE (paper) 197.2 hnaturalistic BATON driving (paper) 177 msp95 probe latency · comma four (paper) 14drivers in the paired on-road study (paper)
01 · OVERVIEW

A world model for the cockpit

The abstract, verbatim, beside what you need to know at a glance.

Predicting how drivers, vehicles, and road scenes interact and evolve together is central to human-centered driving. Prior work models in-cabin activity or traffic-conditioned driver motion in isolation; a framework that models all three jointly and runs in real time on vehicle hardware had not been shown. TriDrive jointly models the driver, the vehicle, and the road: to our knowledge it is the first to forecast driver kinematics, vehicle dynamics, and road demands together from an automation-conditioned transition model, with forecasting and warning as its two applications. Modality-specific encoders — an anchored kinematic representation of the driver, causal CAN-bus dynamics, and frozen V-JEPA 2 road latents with structured road margins — are connected by directed residual connections through which driver and road context refine vehicle forecasts.

On the public AIDE benchmark the kinematic encoder recipe sets a new full-set state of the art among published baselines (48.05 vs. 71.47 All-MPJPE). On 197.2 hours of naturalistic BATON driving, directed connections and road margins raise assistance-engaged PR-AUC by 0.084 for steering onset and 0.286 for time-to-collision drops. For real-time use we distill the road encoders and run TriDrive on a comma four with an external 8 GB GPU, where a lightweight current-state warning probe updates at 5 Hz with 177 ms p95 latency while the forecasting model runs concurrently. The probe is above an openpilot-based baseline on human-labeled manual-driving warnings (AUROC 0.725 vs. 0.563), and in a paired on-road study 14 drivers rate its warnings as more appropriate (+1.79) and timely (+2.67) than those of openpilot's driver-monitoring system.

02 · THREE STREAMS, ONE FORECAST

Watch the state and the forecast move together

Cabin and road from the lab recorder, time-aligned with the vehicle and the driver-monitoring signals at 10 Hz. The onset of the annotated distraction sits at 5.0 s in every clip; wedges show how an earlier forecasting model's probability of a warranted warning grows over the next five seconds from that anchor.

● REC
Cabin and road view at the moment of a phone pickup
Static view. The live explorer needs JavaScript; the clip set and signals are listed under “Data as text”.

What you are looking at. Seven clips: five phone pickups with forecast fans, and two control clips (hands off the wheel, attention off the road) with vehicle and DMS signals only. Risk and TTC come from the recorder's logged openpilot state; TTC is capped at 30 s and drawn hatched where capped. Shading marks the stretches where the driving-assistance system was engaged.

03 · EVIDENCE

What the numbers rest on

Each card is one claim, its number, and the data behind it — recomputed from the logs where we have them.

82.7 → 152.6 ms p50 → p95, drive 1

Real time, measured on the car — and re-derived from the device logs.

The paper's three drives are matched in the logs by time-to-first-ready and aggregated over the paper's windows: log vs paper p50/p95 are 82.7/152.6 vs 83/153, 138.5/239.2 vs 138/239 and 135.3/176.5 vs 135/177 ms. Triangles mark the paper's values; the dashed line is the 300 ms p95 target.

Runtime table in the paper: 5 Hz probe, 135 / 177 ms median / p95 on comma four with an external 8 GB GPU.

Tick rule: every probe tick with an end-to-end latency sample, as in the paper's analysis script. Device logs; the fourth row pools every other session we logged, including warm-ups.

+1.79 appropriateness · 14 drivers

On the road, drivers rated TriDrive's warnings as more appropriate and more timely than openpilot's.

Each line is one participant, openpilot DMS → TriDrive on a seven-point scale; coincident pairs merge with a ×n tag, unpaired cells are hollow. Annoyance is a cost and is drawn in coral. Preference overall: 7 TriDrive · 4 openpilot · 3 equal — the paper calls it inconclusive, and so do we.

Bar chart of paired on-road ratings for TriDrive and openpilot
Paired on-road ratings (paper figure).

Δ, 95 % participant-bootstrap intervals and p from the paper: +1.79 [0.86, 2.75] p = .008 · +2.67 [1.33, 4.08] p = .008 · annoyance +1.17 [0.33, 2.08] p = .031.

0.725 AUROC vs 0.563

The 5 Hz probe beats an openpilot-based baseline on human-labeled manual-driving warnings — and loses when assistance is engaged.

Dumbbells run baseline → probe. The engaged row goes the other way (0.746 → 0.664); the probe's calibration error is 0.028 against 0.408 for the baseline.

AUROC manual 0.725 vs 0.563 · pooled 0.716 vs 0.637 · engaged 0.664 vs 0.746.
48.05 All-MPJPE · AIDE

The kinematic encoder recipe sets the published full-set state of the art on AIDE.

Baseline first: zero-velocity 52.89 → Driver-WM 71.47 (published) → TriDrive 48.05 ± 0.06, PCK@0.05 87.13. Lower MPJPE is better.

MethodAll-MPJPE ↓PCK@0.05 ↑
Zero-velocity52.89—
Driver-WM (published)71.47—
TriDrive (ours, 3 seeds)48.05 ± 0.0687.13

Controls in the appendix (3 seeds) include MotionBERT with released weights at 48.71; the zero-velocity row shows how much of the metric a static pose already explains.

+0.286 PR-AUC · TTC drops, engaged

Directed connections and road margins lift forecasting when the assistance system is driving.

Steering onset PR-AUC 0.774 manual (+0.030) / 0.463 engaged (+0.084); time-to-collision drops +0.206 manual / +0.286 engaged.

TargetManualEngaged
Steering onset PR-AUC0.774 (+0.030)0.463 (+0.084)
TTC-drop PR-AUC gain+0.206+0.286
197.2 h · 162 drivers

Trained and evaluated on naturalistic driving, not a simulator.

The BATON subset used for forecasting: 197.2 hours from 162 drivers, plus 60 further drivers (11.71 h) held out; pose pretraining on 3,410,103 frames from 440 routes. Cabin footage here is from the lab's own consented recorder only.

BATON benchmark → · dataset on Hugging Face

04 · METHOD

Encode three streams, connect them with intent

An anchored kinematic representation of the driver, causal CAN-bus dynamics and frozen V-JEPA 2 road latents with structured road margins meet in one latent state. Directed residual connections carry driver and road context into the vehicle branch; an autonomy flag conditions the transition on who is driving.

TriDrive architecture: driver, vehicle and road encoders, joint latent state with directed residual connections, forecasting and warning heads
Architecture. Modality-specific encoders → joint latent state → future-state forecasts (driver kinematics, vehicle dynamics, road demands) and a current-state warning head.

Driver

Frozen RTMW whole-body keypoints, re-anchored to the seat and the wheel so the representation is invariant to the camera; a GRU over four seconds of kinematics with a frozen keypoint forecaster.

Vehicle

A causal TCN over four seconds of CAN signals — speed, acceleration, steering, pedals — conditioned on whether the assistance system is engaged.

Road

Frozen V-JEPA 2 latents over eight seconds of road video, with radar and lane-line margins as structured inputs; distilled into a road student for the car.

Overview figure: inputs, state representations, joint modeling and the forecasting and warning heads
From inputs to warnings. Driver, vehicle and road inputs; their state representations; the joint model; and the forecast and warning heads. Cabin regions are blurred on this page.

In the car

Recorder and external GPU box mounted in the test vehicle
comma four + external 8 GB GPU in the test car
The device showing a 'pay attention — driver distracted' warning
“Pay attention” — the warning head on the device
The device showing a 'disengage immediately' warning next to the GPU enclosure
“Disengage immediately” — escalated warning
Two deployment cases: driver, vehicle, road and probe signals ten seconds around an anchor
Two on-road cases. Driver (hands off, head turned), vehicle (response and steering onset), road (TTC drop, forward risk) and the probe score around a warning anchor; the probe threshold is 0.49. Cabin panels blurred.
05 · RESULTS

Tables, baseline first

Runtime on comma four + RX 9060 8 GB · arXiv v1

ComponentRateLatency (median / p95)
Warning probe (current state)5 Hz135 / 177 ms
Joint forecasting backbone2.35 Hz—
Pose (RTMW)9.38 Hz—
Road student (distilled V-JEPA 2)1.89 Hz—

Honest limits. Forecasts did not feed the on-road warning decision — the warning came from the current-state probe. On assistance-engaged driving the openpilot-based baseline scores higher AUROC than the probe. Overall preference in the study is inconclusive.

Sources & notes
    06 · DATA & ACCESS

    Model, code and the data behind it

    On request

    TriDrive model & code

    Checkpoints, tables and documents under CC BY-SA 4.0; access is gated with manual approval on Hugging Face.

    as of 2026-10-02 · gated: manual · CC BY-SA 4.0

    Request access
    On request

    BATON (naturalistic driving)

    The 197.2 h forecasting subset comes from BATON; the full dataset is released separately under CC BY-NC 4.0 with a frozen benchmark.

    as of 2026-10-02 · gated: manual

    BATON pageHugging Face
    Not released

    Cabin recordings & study videos

    Participant recordings from the on-road study and raw cabin video are not released (consent). The clips on this page come from the lab's own consented recorder only.

    AIDE is a public third-party benchmark; obtain it from its authors.

    Reproduce. Model and training code on Hugging Face · AIDE protocol as published · BATON forecasting splits with the dataset · runtime figures from the device logs (tick rule above).

    07 · CITE

    Read the paper, cite the work

    BibTeX
    @article{wang2026tridrive,
  title   = {TriDrive: Joint Driver, Vehicle, and Road Modeling for Forecasting and Driver Monitoring},
  author  = {Wang, Yuhang and Yang, Jingxin and Wei, Chuheng and Guo, Yuechen and Xu, Jinghan and Han, Zhao and Zhou, Hao},
  journal = {arXiv preprint arXiv:2609.33000},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.33000}
}

    Related work from the lab

    Benchmark · arXiv 2026

    BATON

    Bidirectional human–automation control transitions on synchronized streams.
    Benchmark · WACV 2027

    DriveMotion

    Whole-body driver motion forecasting from 133 keypoints.
    CoRL 2026

    VLAlert

    Observe before you alert: adaptive driver alerting with vision-language models.