Matrix New York

Overview

We drive pretrained interactive video world models through New York City starting from street-level imagery. For a query location we retrieve the nearest Google Street View panorama, remove near-field vehicles, and insert a 3D vehicle rendered under a chase camera. A kinematic bicycle model converts throttle and steering into the action space of each model: keyboard and camera-yaw actions for Matrix-Game-3.0, a camera-to-world trajectory for Matrix-Game-3.5, and a translation trajectory over the full 360° panorama for PanoWorld. The ego vehicle is composited from renders indexed by the controller's yaw relative to the camera. No model is fine-tuned, and each rollout runs on a single GPU.

Models

The three models differ in what they condition on and how they are controlled. Each receives the same Street View observation and the same controller trajectory; only the action adapter differs.

Interactive viewer

Select a location on the map or in the list to play its rollout. The panel below the video shows the controller input at the displayed frame.

Throttle

Brake

Steering input over the clip

Comparison

Each panel starts from the same Street View observation and receives the same controller trajectory, mapped to that model's action space. Playback is synchronized.

0.0 / 0.0 s

Panoramic output

PanoWorld conditions on the full Street View panorama and generates the whole sphere. The perspective views elsewhere on this page are crops of this output along the chase-camera heading. The last two clips restart every 81 frames from the real panorama nearest the vehicle's position.

Drag to look around

The same output as an equirectangular video.

Method

Initial observation

(a) Perspective view along the capture heading.

(b) Near-field vehicles selected for removal (red).

Panoramas are retrieved through Street View coverage tiles, which list only captures from Google's own vehicles, and downloaded at zoom 4 (zoom 3 if tiles are missing). The view is an equirectangular-to-perspective remap with yaw equal to the capture heading, pitch −3°, and 90° horizontal field of view. Near-field vehicles are segmented with Mask2Former (Swin-L, Cityscapes classes car, truck, bus, train, motorcycle, bicycle). A component is removed if it lies in the ego lane ahead, touches the bottom of the frame, or overlaps the inserted vehicle; parked vehicles at the sides are kept. Masks are dilated by 18 px, black no-data regions of the panorama are added, and LaMa fills the holes. The inserted vehicle is a CC-BY MeshFleet mesh rendered with pyrender from about two car lengths behind, composited at 31% of the image width with a contact shadow and a global gain and tint match.

Street View perspective view — **Figure 1.** Times Square, panorama 2024-08, (40.7580, -73.9856). Drag to compare the reprojected panorama with the prepared frame.

Prepared initial observation — **Figure 1.** Times Square, panorama 2024-08, (40.7580, -73.9856). Drag to compare the reprojected panorama with the prepared frame.

Vehicle model

Inputs are throttle and brake \(u\in[-1,1]\) and steering \(s\in[-1,1]\), low-pass filtered with \(\tau=0.15\) s. A kinematic bicycle model with speed-dependent steering authority and a first-order chase camera integrates them:

\[\dot v = a^{+}u^{+} - a^{-}u^{-} - c\,v,\quad \delta=\frac{s\,\delta_{\max}}{1+v/v_{\mathrm{ref}}},\quad \dot\psi=\frac{v}{L}\tan\delta,\quad \dot\psi_c=\frac{\psi-\psi_c}{\tau_c}\]

Lane changes are steering doublets, so the vehicle's heading changes and it is never translated sideways. In all three backends the ego vehicle is drawn from 17 renders (−24° to +24°, pivot at the rear axle) at the relative yaw \(\psi-\psi_c\).

Parameter	Value
Wheelbase \(L\)	2.7 m
Maximum steering \(\delta_{\max}\), \(v_{\mathrm{ref}}\)	32°, 6 m/s
Acceleration \(a^{+}\), braking \(a^{-}\)	3 m/s², 7 m/s²
Drag \(c\), top speed	0.12 s⁻¹, 25 m/s
Camera lag \(\tau_c\)	0.35 s
Lane change	s = ±0.7 for 0.5 s each

Action mapping

	Matrix-Game-3.0	Matrix-Game-3.5	PanoWorld
Conditioning	vehicle-free perspective frame and text prompt	vehicle-free perspective frame and text prompt	360° panorama with nearby vehicles inpainted, and text prompt
Forward motion	W with duty cycle min(v / 10 m/s, 1)	camera translation along the Street View capture path	camera translation, 0.10 units per frame at top speed
Steering	camera yaw Δψ_c / 15° per frame	camera yaw ψ_c in the trajectory	yaw of the perspective crop; generated orientation is fixed
Ego vehicle	composited at relative yaw ψ − ψ_c	composited at relative yaw ψ − ψ_c	composited at relative yaw ψ − ψ_c
Frames shown	every second (controller at 34 Hz)	every frame (16 fps)	every frame (16 fps)
Generation time per clip	4.8 min for 817 frames	8 min for 337 frames	2 min for 321 frames

Results

All rollouts start from a single Street View observation. Matrix-Game-3.0 clips show every second generated frame; Matrix-Game-3.5 and PanoWorld clips show every frame. Select a clip to open it in the viewer above.

Analysis

Ego vehicle in the conditioning frame

Matrix-Game-3.0 has no ego-vehicle prior. A vehicle composited into the conditioning frame is treated as static scene content: the camera drives into it and it smears across the lower image within about a second. Conditioning on the inpainted frame and compositing the vehicle afterwards removes the artifact.

Vehicle in frame, t=0.0s — **Figure 2.** Same scene, actions and seed. Top: the conditioning frame contains the vehicle and no overlay is drawn. Bottom: vehicle-free conditioning frame with the vehicle composited from the render bank. Times are controller time; every second generated frame is shown.

Vehicle in frame, t=0.5s — **Figure 2.** Same scene, actions and seed. Top: the conditioning frame contains the vehicle and no overlay is drawn. Bottom: vehicle-free conditioning frame with the vehicle composited from the render bank. Times are controller time; every second generated frame is shown.

Forward speed

Matrix-Game-3.0 advances a roughly constant distance per frame while W is held. Its action interface can slow it down through the W duty cycle but not speed it up: scaling the camera translation used for its Plücker-ray conditioning left the output essentially unchanged. We therefore set apparent speed through temporal subsampling. The controller runs at 34 Hz and every second generated frame is displayed.

Setting (FDR Drive, highway route)	Mean flow (px/frame)
W duty cycle 12% (first 2 s)	0.04
W held, translation ×1	0.66
W held, translation ×3	0.65
W held, frame stride 2 (shown)	1.35

Farnebäck optical flow on a mid-image band, frames resized to 320 px width.

Road following

Matrix-Game-3.5 reproduces the commanded camera path closely but has no notion of the road, so an open-loop trajectory leaves the roadway wherever the real road bends or the initial heading is slightly off the road axis. Driving the route relative to the Street View capture path, obtained by chaining panorama links ahead of the start, and removing small corrective steering inputs reduced the drift but did not remove it. In all six scenes the camera reaches the sidewalk, a barrier or the river after 6 to 15 s, with a consistent bias to the right. Without Patch Memory a straight path reached the barrier within 5 s. Matrix-Game-3.0, which is driven by keyboard and yaw actions rather than poses, stays on the roadway on the straight and lane-change routes. For its first 5 to 10 s Matrix-Game-3.5 is the sharpest of the three models, with no streaking and plausible pedestrians and traffic.

Panoramic conditioning

PanoWorld conditions on the full 360° capture and keeps the scene layout recognisable for the whole 20 s clip, for example the crane and river on FDR Drive and the billboards in Times Square. With the released 480p model a 90° perspective crop covers about 240 of the 960 panorama columns and is upsampled about 5×, so the driving view is soft. Apparent forward speed saturates at a slow roll: steps above roughly 0.2 units per frame streak, and above 0.4 the scene dissolves within 2 s. Orientation is fixed during generation, so steering rotates the crop but the generated road does not bend with the trajectory.

Re-anchoring to real panoramas

Street View captures are spaced roughly 10 m apart along a street. Restarting each 81-frame PanoWorld segment from the real panorama nearest the vehicle's position keeps the rollout tied to real imagery, at the cost of cuts at segment boundaries where traffic and, if capture dates differ, lighting change. Neither Matrix-Game model accepts additional observations: the Matrix-Game-3.5 memory is indexed by rollout frame and cannot be seeded with external views without changes to its rollout code.

Limitations

Only the first frame or panorama is observed, except in the re-anchored PanoWorld clips. Content entering the view later is sampled from the model's prior and does not match the real street.
Matrix-Game-3.0 quality is highest in roughly the first 15 s of generated time; afterwards facades simplify, road texture streaks, and color shifts. Matrix-Game-3.5 leaves the roadway after 6 to 15 s, and the Fifth Avenue clip washes out toward white. PanoWorld stays recognisable but is soft and slow.
The composited ego vehicle is a 2D layer. It is not occluded by, does not shadow, and does not interact with generated content.
Large heading changes away from intersections steer the camera into facades. Turn routes are only meaningful when the first frame is at an intersection.
The Matrix-Game-3.5 memory depends on Depth Anything 3 weights released under CC-BY-NC-4.0. The PanoWorld weights on Hugging Face carry no license tag; its code is Apache-2.0.

Citation

@misc{turkcan2026matrixnewyork,
  title        = {Matrix New York: Street View Initial Observations and Vehicle Control
                  for Interactive Video World Models},
  author       = {Turkcan, Mehmet Kerem},
  year         = {2026},
  howpublished = {\url{https://github.com/mkturkcan/matrix-new-york}}
}