Multimodal Foundation Models
Unified models that understand, reason, and generate across visual, linguistic, acoustic, and embodied signals.
Explore selected work →Research Scientist · Physical AI
Jianhua Han
2030 Lab, Yinwang Intelligent Technology Co., Ltd.
I study how multimodal foundation models can move beyond recognition toward grounded reasoning and reliable action, with a focus on vision-language-action models, robotics, and open-world perception.
Research statement. Intelligence can be defined as self-motivated skill-acquisition efficiency in the real world.
Research
From perception to physical intelligence
My work connects four complementary research directions, moving from broad visual understanding to agents that can acquire skills and act in complex environments.
Vision-Language-Action
Connecting visual grounding and language-level reasoning with precise, generalizable actions in the physical world.
Explore selected work →Robotics
Building adaptable embodied agents that compose reusable skills and improve through interaction and experience.
Explore selected work →Open-World Perception
Scalable visual recognition and spatial grounding beyond fixed vocabularies, distributions, and predefined scenarios.
Explore selected work →Selected work
Publications
Complete list on Google Scholar ↗A curated path through my research. Each entry summarizes both the technical contribution and the broader idea it advances.
13 selected papers
Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving ↗
Unifies 2D perspective-view and 3D bird's-eye-view perception as world tokens inside one VLM, then connects those representations directly to trajectory prediction.
Why it matters. Spatial grounding and action planning can share a foundation-model backbone without reducing perception to text-only reasoning.
AtomicVLA: Unlocking the Potential of Atomic Skill Learning in Robots ↗
Jointly generates plans, atomic skill abstractions, and low-level actions, using a skill-guided mixture of experts to compose specialized capabilities.
Why it matters. Modular skill routing offers a practical path from monolithic policies toward continual and long-horizon robot learning.
4D-WAM: 4D Consistent World Modeling for Autonomous Driving ↗
Uses geometric foundation models as training-time supervisors so predicted driving futures preserve 4D structure, with no additional inference cost.
Why it matters. Visually plausible futures are not enough; physically grounded supervision matters where decisions begin to form.
SUV: Future Scene Understanding as Video Generation for End-to-End Driving ↗
A shared video expert predicts future RGB, semantics, relative depth, and instance dynamics while an action expert reads their latent tokens to plan trajectories.
Why it matters. Generative future prediction can become a common interface between structured scene understanding and control.
WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA ↗
Transfers an autoregressive driving generalist into a parallel discrete-diffusion model through hierarchical distillation and block-causal attention.
Why it matters. Generation-paradigm conversion can reduce VLA latency while retaining perception, reasoning, and planning capabilities.
ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance ↗
Builds a unified multimodal model around visual understanding and generation, then uses generated visual data to iteratively improve its own capabilities.
Why it matters. Understanding and generation are complementary learning signals rather than separate endpoints.
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions ↗
Aligns vision, text, and speech end to end with a semantic-acoustic disentangled tokenizer and lightweight controls for expressive speech styles.
Why it matters. Careful tokenization can align additional modalities without requiring scarce fully paired multimodal data.
Reason2Drive: Towards Interpretable and Chain-Based Reasoning for Autonomous Driving ↗
Introduces a large video-text benchmark that decomposes driving decisions into perception, prediction, and reasoning chains, plus an evaluation metric for those chains.
Why it matters. Interpretable driving requires evaluating the decision process, not only the final answer or trajectory.
HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Driving ↗
Combines low-resolution temporal reasoning with a high-resolution spatial stream and a lightweight adapter for fine-grained risk-object understanding.
Why it matters. Efficient multimodal systems should allocate visual resolution according to temporal and spatial information needs.
NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning ↗
Structures navigation reasoning into future imagination, visual observation matching, and action prediction, then learns the process with parameter-efficient tuning.
Why it matters. An embodied agent can use an LLM as both a lightweight predictive model and an interpretable decision module.
DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-Training for Open-World Detection ↗
Reformulates heterogeneous detection data into parallel visual-concept learning and enriches category supervision with a concept dictionary.
Why it matters. Language knowledge can expand a detector beyond the closed label set while keeping pre-training efficient.
DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-Training via Word-Region Alignment ↗
Learns direct word-region alignment from large-scale image-text pairs under unified detection, grounding, and caption supervision.
Why it matters. Fine-grained alignment makes web-scale image-text data useful for localization without expensive pseudo-label pipelines.
DetCLIPv3: Towards Versatile Generative Open-Vocabulary Object Detection ↗
Adds object-level caption generation and hierarchical concept labels to open-vocabulary detection, bridging localization and dense description.
Why it matters. A detector becomes more versatile when recognition and object-centric generation reinforce each other.
Experience
Research journey
Building multimodal systems across foundation-model research and real-world physical intelligence.
— Present
Research Scientist
2030 Lab · Yinwang Intelligent Technology Co., Ltd.
Researching multimodal foundation models, vision-language-action systems, robotics, and open-world decision-making for Physical AI.
—
Research Scientist
Huawei Noah's Ark Lab
Worked on large-scale multimodal foundation models and open-vocabulary perception, including model architecture, pre-training, data optimization, and the DetCLIP series.
Community
Research resources
I maintain practical resources that help researchers navigate fast-moving work in embodied and physical intelligence.
Awesome VLA Papers
A curated and continuously updated map of Vision-Language-Action research across robotics, autonomous systems, world models, and self-improving embodied agents.
Explore the repository →Google Scholar
Browse the complete publication record, citation history, and latest research updates.
View all publications →Collaborate
Interested in working together?
We welcome self-motivated interns and full-time researchers or engineers interested in multimodal foundation models, VLA, robotics, and Physical AI.