A Review of 3D Surface Modeling Techniques Based on Images

3D surface modeling is a core topic in computer vision and graphics, with wide-ranging applications in virtual reality, medical imaging, cultural heritage preservation, game development, and autonomous driving. This paper reviews image-based 3D surface modeling techniques, including methods such as single-image reconstruction, stereo vision, multi-view stereo (MVS), structured light, and deep learning. The development history and main contributors of each technique are detailed.

1. 3D Reconstruction from a Single Image

Reconstructing 3D shapes from a single image is a challenging task because a 2D image contains only partial information about a 3D scene. Early works in this area often relied on geometric reasoning and shape priors.

Shape from Shading (SFS): This method uses the intensity of light reflected from a surface to estimate its 3D shape. Horn first proposed the theoretical framework for this approach in 1989【Horn, B. K. P. (1989). “Shape from Shading: A Method for Obtaining the Shape of a Smooth Opaque Object from One View,” MIT Press】.
Shape from Texture (SFT): This method reconstructs 3D shapes by analyzing the variations in surface texture. Maliska and Sclaroff proposed the mathematical model for this method in 1997【Maliska, D., & Sclaroff, S. (1997). “Deformable Models for Shape from Texture,” Computer Vision and Pattern Recognition】.

2. Stereo Vision

Stereo vision uses two or more images taken from different viewpoints to recover 3D information about a scene. This method typically involves feature matching, disparity calculation, and triangulation.

Stereo Matching: Marr and Poggio proposed an early stereo matching algorithm in 1979, which relied on matching local features in images【Marr, D., & Poggio, T. (1979). “A Computational Theory of Human Stereo Vision,” Proceedings of the Royal Society B】.
Disparity Map Generation: Disparity map generation is a key step in stereo vision. Scharstein and Szeliski systematically analyzed different disparity algorithms in 2002 and created a widely used disparity benchmark【Scharstein, D., & Szeliski, R. (2002). “A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms,” International Journal of Computer Vision】.

3. Multi-View Stereo (MVS)

Multi-view stereo (MVS) refers to reconstructing 3D shapes from multiple images taken from different viewpoints. This technique is central to photogrammetry and is widely used in cultural heritage preservation and environmental modeling.

Photometric Stereo: Woodham’s photometric stereo method, proposed in 1980, is a significant development in MVS. It recovers surface normals by analyzing images under different lighting conditions【Woodham, R. J. (1980). “Photometric Method for Determining Surface Orientation from Multiple Images,” Optical Engineering】.
Patch-based MVS: Furukawa and Ponce’s patch-based MVS method, proposed in 2010, significantly improved MVS accuracy and has become one of the most widely used MVS algorithms【Furukawa, Y., & Ponce, J. (2010). “Accurate, Dense, and Robust Multi-View Stereopsis,” IEEE Transactions on Pattern Analysis and Machine Intelligence】.

4. Structured Light and TOF Sensors

Structured light and Time-of-Flight (TOF) sensors directly measure the depth information of objects by actively projecting light or emitting laser pulses. These techniques offer significant advantages in terms of accuracy and speed, particularly in industrial and autonomous driving applications.

Kinect and Structured Light Technology: Microsoft’s Kinect is a famous application of structured light technology. It calculates depth using a point cloud pattern projected by infrared light【Zhang, Z. (2012). “Microsoft Kinect Sensor and Its Effect,” IEEE MultiMedia】.
Time-of-Flight Technology: Rapp and his team conducted detailed research on TOF sensors in 1995, developing high-speed and accurate 3D scanning methods【Rapp, H., Abert, O., & Seitz, P. (1995). “Time-of-Flight 3D Imaging for Machine Vision,” IEEE International Conference on Image Processing】.

5. Deep Learning and 3D Modeling

With the advancement of deep learning, neural network-based 3D modeling methods have rapidly developed. Deep learning models can automatically extract high-level features from images, enabling more accurate 3D reconstruction.

3D Convolutional Neural Networks (3D CNNs): Maturana and Scherer proposed 3D CNNs in 2015 for extracting 3D information from voxel grids【Maturana, D., & Scherer, S. (2015). “VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition,” IEEE/RSJ International Conference on Intelligent Robots and Systems】.
NeRF (Neural Radiance Fields): NeRF is a significant breakthrough in 3D reconstruction, proposed by Mildenhall et al. in 2020. This method uses a neural network to implicitly represent the radiance field of a scene, enabling high-quality 3D rendering from a few viewpoints【Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,” European Conference on Computer Vision】.

Conclusion

Image-based 3D surface modeling techniques have evolved from traditional geometric methods to modern deep learning models. Each technology has its unique application scenarios and advantages. Looking ahead, the integration of multimodal data (such as RGB images and depth maps) and more efficient neural network models will continue to drive the advancement of 3D modeling techniques.