Efficient 3D Generative AI: From Reconstruction to Generation

Generative AI (GenAI) tools like ChatGPT and GPT-4o have revolutionized how we generate text and images — but creating fast, controllable, and high-quality 3D content remains a major challenge. Unlike text or 2D images, 3D scenes are inherently more complex and data-hungry, and suitable training data is often scarce. Recent breakthroughs like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have made it possible to turn ordinary 2D photos into richly detailed 3D scenes. These methods work well when many input images are available, aiming to reconstruct scenes with high fidelity. However, when only a few images are provided, the AI must not only reconstruct what's visible, but also generate unseen content — drawing from prior knowledge, all while remaining efficient. In this talk, I’ll start with describing the current trends of GenAI and briefly introducing some frontline 2D GenAI research done by my group at Monash. Then, I will focus on sharing how my team is advancing the 3D GenAI area through models like MVSplat, MVSplat360, and PanSplat, which aim to generate realistic 3D scenes more efficiently and with less input — bringing us closer to fast, flexible, and accessible 3D creation.

About the Speaker

Professor Jianfei Cai

Monash University

Jianfei Cai is a Professor at Faculty of IT, Monash University, where he had served as the inaugural Head for the Data Science & AI Department. Before that, he was Head of Visual and Interactive Computing Division and Head of Computer Communications Division in Nanyang Technological University (NTU). His major research interests include computer vision, deep learning and multimedia. He has successfully trained 40+ PhD students with three getting NTU SCSE Outstanding PhD thesis award and one getting Monash FIT Graduate Research Student Excellence Award. Many of his PhD students joined leading IT companies or become faculty members in reputable universities. He is a co-recipient of paper awards in ACCV, ICCM, IEEE ICIP and MMSP, and a winner of Monash FIT’s Dean's Researcher of the Year Award. He serves or has served as an Associate Editor for TPAMI, IJCV, IEEE T-IP, T-MM, and T-CSVT as well as serving as Senior/Area Chair for CVPR, ICCV, ECCV, IJCAI, ACM Multimedia, ICME, ICIP and ISCAS. He was the Chair of IEEE CAS VSPC-TC during 2016-2018. He had served as the leading TPC Chair for IEEE ICME 2012, the best paper award committee chair & co-chair for IEEE T-MM 2020 & 2019, and the leading General Chair for ACM Multimedia 2024. He is a Fellow of IEEE.