Sheng WANG

GenSight unifies future video generation and dense 4D perception in a shared video diffusion transformer. It jointly models appearance, geometry, semantics, and motion through video, depth, embodiment segmentation, and first-frame-relative 2D flow. The same model supports both multimodal future generation from an initial frame and a text prompt, and single-step dense perception from observed video.

WorldFlow jointly predicts future video and query-based 3D trajectories, coupling visual evolution with explicit geometric dynamics in a unified Mixture-of-Transformers backbone. It is pretrained on 1.44 million object-level samples from simulated and real-world domains. The work also introduces WorldFlowBench, with 1,678 evaluation samples spanning 11 data sources, to evaluate motion prediction across diverse interactions and spatial reasoning tasks.

Decision-making and planning in autonomous driving critically reflect the safety of the system, making effective planning imperative. Current imitation learning-based planning algorithms often merge historical trajectories with present observations to predict future candidate paths. However, these algorithms typically assess the current and historical plans independently, leading to discontinuities in driving intentions and an accumulation of errors with each step in a discontinuous plan. To tackle this challenge, this paper introduces LHPF, an imitation learning planner that integrates historical planning information. Our approach employs a historical intention aggregation module that pools historical planning intentions, which are then combined with a spatial query vector to decode the final planning trajectory. Furthermore, we incorporate a comfort auxiliary task to enhance the human-like quality of the driving behavior. Extensive experiments using both real-world and synthetic data demonstrate that LHPF not only surpasses existing advanced learning-based planners in planning performance but also marks the first instance of a purely learning-based planner outperforming the expert. Additionally, the application of the historical intention aggregation module across various backbones highlights the considerable potential of the proposed method. The code will be made publicly available.

Evaluating and training autonomous driving systems require diverse and scalable corner cases. However, most existing scene generation methods lack controllability, accuracy, and versatility, resulting in unsatisfactory generation results. Inspired by DragGAN in image generation, we propose DragTraffic, a generalized, interactive, and controllable traffic scene generation framework based on conditional diffusion. DragTraffic enables non-experts to generate a variety of realistic driving scenarios for different types of traffic agents through an adaptive mixture expert architecture. We employ a regression model to provide a general initial solution and a refinement process based on the conditional diffusion model to ensure diversity. User-customized context is introduced through cross-attention to ensure high controllability. Experiments on a real-world driving dataset show that DragTraffic outperforms existing methods in terms of authenticity, diversity, and freedom.

Accurate trajectory prediction is crucial for safe and efficient autonomous driving, but handling partial observations presents significant challenges. To address this, we propose a novel trajectory prediction framework called Partial Observations Prediction (POP) for congested urban road scenarios. The framework consists of two key stages: self-supervised learning (SSL) and feature distillation. POP first employs SLL to help the model learn to reconstruct history representations, and then utilizes feature distillation as the fine-tuning task to transfer knowledge from the teacher model, which has been pre-trained with complete observations, to the student model, which has only few observations. POP achieves comparable results to topperforming methods in open-loop experiments and outperforms the baseline method in closed-loop simulations, including safety metrics. Qualitative results illustrate the superiority of POP in providing reasonable and safe trajectory predictions.

In recent years, the rapid evolution of autonomous vehicles (AVs) has reshaped global transportation systems, leading to an increase in autonomous shuttle applications in people’s daily lives. Leveraging the accomplishments of our earlier endeavor, particularly Hercules [1], an autonomous logistics vehicle for transporting goods, we introduce Snow Lion, an autonomous shuttle vehicle specifically designed to transform on-campus transportation, providing a safe and efficient mobility solution for students, faculty, and visitors.

Our goal is to train a neural planner that can capture diverse driving behaviors in complex urban scenarios. We observe that even state-of-the-art neural planners are struggling to perform common maneuvers such as lane change, which is rather natural for human drivers. We propose to explore the multi-modalities in the planning problem and force the neural planner to explicitly consider different policies. This is achieved by generating the future trajectories conditioned on every possible reference line, which could simply be the centerline of the surrounding lanes. We find this simple strategy yet enables the planner to perform rich and complex behaviors. We train our model using real-world driving data and demonstrate the effectiveness of our method through both open-loop and closed-loop evaluations.

With the development of autonomous driving, more autonomous vehicles are equipped with multiple sensors to perform better. For multi-sensor systems, accurate extrinsic calibration is a prerequisite for perception and localization systems. Extrinsic calibration aims to transform two or more sensors’ coordinates into a unified spatial coordinate system. Sensors usually need to be calibrated after installation to ensure accurate measurements. Most existing methods require special calibration markers or rely heavily on additional sensors to solve this problem. Recently, researchers have proposed a novel method that can calibrate extrinsic parameters using the common corners in surroundings to solve the multi-LiDARs extrinsic calibration problem, three linearly independent planar surfaces of the corner are utilized to compute the extrinsic parameters. However, this method is relatively inefficient and requires manual operation, especially in some hard scenes. To overcome this problem, we propose using the learning-based method to segment corners, improving efficiency and convenience without losing accuracy. By introducing learning-based corner segmentation method, we can achieve “one-click” extrinsic calibration, that is, there is no need to prepare auxiliary markers, there is no need to collect a large amount of data and data preprocessing, and the external parameters can be automatically calibrated.