WildCity

Abstract

Humans can navigate an unfamiliar city and gradually form a coherent spatial mental map spanning tens of square kilometers. Can AI build spatial representations at a comparable scale? Although recent foundation models have advanced scene reconstruction and embodied intelligence, scaling to entire cities remains an open challenge, primarily due to the lack of city-scale data. To bridge the gap, we introduce WildCity, a real-world multimodal dataset collected by autonomous fleets traversing complex urban environments. Our dataset includes 18 trajectories, each averaging 83.7 kilometers in length, and preserves the core challenges of in-the-wild perception, e.g., dynamic objects, lighting variations, and imperfect camera poses. We further establish an urban-tailored reconstruction baseline and convert the reconstructed environments into a closed-loop simulator. Beyond the dataset and baseline, we systematically analyze the key challenges on the path to simulation-ready urban digital twins: scalability, extrapolation, and uncertainty. Ultimately, WildCity aims to catalyze progress not only in city-scale rendering, but more broadly in the pursuit of AI that can perceive, remember, and reason across space at a scale comparable to human cognition.

Dataset

WildCity is collected from real-world urban driving across six U.S. cities — Atlanta, Arlington, Ann Arbor, Eden Prairie, Grand Rapids, and Miami — with a multimodal sensor suite: a roof-mounted LiDAR, six surround-view RGB cameras (three narrow-angle forward, three wide-angle lateral/rear), an IMU, and a GPS receiver, all rigorously calibrated to the ego frame. Logs are captured on different days during regular daytime operations, naturally introducing variation in illumination, weather, appearance, and traffic dynamics.

Dataset & reconstruction statistics. Sensor configuration, total data scale, and the compute footprint required for reconstruction at increasing trajectory lengths.

Geographic coverage across six U.S. cities. Trajectories collected in each city; colors mark the 5-km segments used for standardized data organization and reconstruction.

Reconstruction Baseline

We establish an urban-tailored 3D Gaussian Splatting baseline for large, noisy, and unbounded street scenes:

• Rig pose optimization jointly refines per-keyframe ego poses and camera extrinsics, respecting the rigid multi-camera setup for globally consistent geometry.
• Sky model — a lightweight view-dependent MLP composites an infinite background, decoupling sky appearance from scene geometry and reducing far-depth floaters.
• Ground regularization stabilizes weakly constrained road surfaces via height, alignment, and opacity priors on ground Gaussians.
• Extrapolated-view post-repair progressively renders, repairs (Difix3D+), and re-optimizes to mitigate artifacts under off-trajectory viewpoints.
• Multi-GPU training shards Gaussian parameters across devices to scale to billion-level primitives with bounded per-GPU memory.

To support region-aware reconstruction, we generate ground, sky, and dynamic-object masks (via SAM3 with text prompts) and use 3D tracking cuboids to separate truly dynamic objects from stationary instances.

Quantitative validation of semantic masks. Automatically generated dynamic, sky, and ground masks evaluated against 100 manually annotated images across four cities (mIoU, %); the high overall score confirms reliable region-level supervision.

Closed-loop Simulation

WildCity turns reconstructed environments into a closed-loop simulator: a Vision-Language-Action model iteratively predicts the next target pose from the current observation and task, then receives the rendered view at that pose. Using Alpamayo 1 as a proof of concept, we observe smooth navigation and plausible multi-step reasoning in the reconstructed urban environment.

Closed-loop simulation demo: an Alpamayo agent perceives surround-view renderings, reasons over the task (chain-of-thought), and acts by planning its next trajectory.

Results

We evaluate 3D reconstruction across spatial scales using two sequences captured with the same sensor setup: Ann Arbor (short, 0.25 km) and Atlanta (long, 2.5 km). We report PSNR, SSIM, and LPIPS on static regions, and Depth-L1 (meters) on pixels with valid LiDAR depth. Best per column in bold.

Quantitative comparison — Ann Arbor (0.25 km, short; 500 frames). Reconstruction quality versus baselines; best per column in bold.

Quantitative comparison — Atlanta (2.5 km, long; 5,000 frames). Reconstruction quality versus baselines; best per column in bold.

Partitioning and hierarchy help scale to large scenes, but alone they do not resolve the geometric ambiguity of narrow, long street-view trajectories. Reliable city-scale reconstruction requires structural constraints and pose consistency — not just memory-efficient scene decomposition.

Qualitative comparison

In-trajectory Extrapolation

For closed-loop simulation, in-trajectory quality is not enough: agents must query views beyond the recorded path. Simulation-ready urban digital twins therefore require stable geometry and coherent appearance under off-trajectory viewpoints.

We analyze three coupled bottlenecks on the path to simulation-ready digital twins — scalability, extrapolation, and uncertainty.

Component ablation

Notably, 2D image metrics can misalign with geometric quality: dropping the sky model yields a small PSNR change but collapses Depth-L1 (148.54 m), and dropping ground regularization slightly improves 2D metrics while hurting structural consistency.

Component ablation. Effect of ground regularization, sky modeling, and rig pose optimization on Arlington (0.1 km) and Atlanta (0.25 km); the full model (final row, highlighted) balances 2D fidelity and geometric consistency.

Scalability

Release Scope & Privacy

• We release anonymized sensor data only.
• Faces and license plates are blurred.
• Raw, unblurred images are not distributed.
• Released modalities: RGB, LiDAR, calibration, poses, masks, train/val/test splits, code, and configs.
• Sensitive locations are handled with care and excluded where appropriate.
• Distributed under a documented license / terms of use.
• Access is granted via a dataset request form (link coming soon).

Code, Data & Benchmark

• Code: coming soon (GitHub).
• Data: request access (download link coming soon).
• Benchmark config: train/val/test splits + evaluation script.
• Leaderboard: we will maintain a public benchmark page with submitted results after the initial release.

BibTeX

@inproceedings{wildcity2026,
  title     = {WildCity: A Real-World City-Scale Testbed for Rendering, Simulation, and Spatial Intelligence},
  author    = {Han, Xiangyu and Yang, Mengyu and Li, Jiaqi and Chang, Bowen and Chen, Ziyu and Zhao, Hexu and Agrawal, Rahul Kumar and Rodriguez, Anthony and Acharya, Rajani and Hua, Fiona and Pavone, Marco and Feng, Chen and Li, Yiming},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026},
}