Quality Enhancement of 3D Volumetric Contents Based on 6DoF for 5G Telepresence Service
DOI:
https://doi.org/10.13052/jwe1540-9589.2138Keywords:
5G telepresence, web-based graphics, point cloud, 3d reconstruction, rgb-d, illumination compensation, color correctionAbstract
In general, the importance of 6DoF (degree of freedom) 3D (dimension) volumetric contents technology is emerging in 5G (generation) telepresence service, Web-based (WebGL) graphics, computer vision, robotics, and next-generation augmented reality. Since it is possible to acquire RGB images and depth images in real-time through depth sensors that use various depth acquisition methods such as time of flight (ToF) and lidar, many changes have been made in object detection, tracking, and recognition research. In this paper, we propose a method to improve the quality of 3D models for 5G telepresence by processing images acquired through depth and RGB cameras on a multi-view camera system. In this paper, the quality is improved in two major ways. The first concerns the shape of the 3D model. A method of removing noise outside the object by applying a mask obtained from a color image and a combined filtering operation to obtain the difference in depth information between pixels inside the object were proposed. Second, we propose an illumination compensation method for images acquired through a multi-view camera system for photo-realistic 3D model generation. It is assumed that the three-dimensional volumetric shooting is done indoors, and the location and intensity of illumination according to time are constant. Since the multi-view camera uses a total of 8 pairs and converges toward the center of space, the intensity and angle of light incident on each camera are different even if the illumination is constant. Therefore, all cameras take a color correction chart and use a color optimization function to obtain a color conversion matrix that defines the relationship between the eight acquired images. Using this, the image input from all cameras is corrected based on the color correction chart. It was confirmed that the quality of the 3D model could be improved by effectively removing noise due to the proposed method when acquiring images of a 3D volumetric object using eight cameras. It has been experimentally proven that the color difference between images is reduced.
Downloads
References
Ralf Schäfer, Peter Kauff, Robert Skupin, Yago Sánchez, and Christian Weißig. Interactive steaming of panoramas and vr worlds. SMPTE Motion Imaging Journal, 126(1):35–42, 2017.
T.H.D. Nguyen, T.C.T. Qui, K. Xu, A.D. Cheok, S.L. Teo, Z.Y. Zhou, A. Mallawaarachchi, S.P. Lee, W. Liu, H.S. Teo, L.N. Thang, Y. Li, and H. Kato. Real-time 3d human capture system for mixed-reality art and entertainment. IEEE Transactions on Visualization and Computer Graphics, 11(6):706–721, 2005.
Zongqian Zhan, Gaofeng Zhou, and Xue Yang. A method of hierarchical image retrieval for real-time photogrammetry based on multiple features. IEEE Access, 8:21524–21533, 2020.
Photogrammetry, July 2021.
Soon-Yong Park and Sung-In Choi. Convenient view calibration of multiple rgb-d cameras using a spherical object. KIPS Transactions on Software and Data Engineering, 3:309–314, 08 2014.
Shahram Izadi, Richard A. Newcombe, David Kim, Otmar Hilliges, David Molyneaux, Steve Hodges, Pushmeet Kohli, Jamie Shotton, Andrew J. Davison, and Andrew Fitzgibbon. Kinectfusion: Real-time dynamic 3d surface reconstruction and interaction. In ACM SIGGRAPH 2011 Talks, SIGGRAPH ’11, New York, NY, USA, 2011. Association for Computing Machinery.
Kyung-Jin Kim, byung-Seo Park, Dong-Wook Kim, and Young-Ho Seo. Point cloud registration algorithm based on rgb-d camera for shooting volumetric objects. Journal of Broadcast Engineering, 5(5), Sep 2019.
Alessandro Rizzi, Carlo Gatta, and Daniele Marini. A new algorithm for unsupervised global and local color correction. Pattern Recognition Letters, 24(11):1663–1677, 2003. Colour Image Processing and Analysis. First European Conference on Colour in Graphics, Imaging, and Vision (CGIV 2002).
Edoardo Provenzi, Carlo Gatta, Massimo Fierro, and Alessandro Rizzi. A spatially variant white-patch and gray-world method for color image enhancement driven by local contrast. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(10):1757–1770, 2008.
V. Vonikakis. Fast centre–surround contrast modification. IET Image Processing, 2:19–34(15), February 2008.
H.-S. Le. Fused logarithmic transform for contrast enhancement. Electronics Letters, 44:19–20(1), January 2008.
Christophe Schlick. Quantization techniques for visualization of high dynamic range pictures. Photorealistic Render Techn, 03 1998.
Wanpeng Cao, Rensheng Che, and Dong Ye. An illumination-independent edge detection and fuzzy enhancement algorithm based on wavelet transform for non-uniform weak illumination images. Pattern Recognition Letters, 29(3):192–199, 2008.
Chung-Ming Kuo, Nai-Chung Yang, Chih-Shan Liu, Pi-Yun Tseng, and Chi-Kao Chang. An effective and flexible image enhancement algorithm in compressed domain. Multimedia Tools and Applications, 75(2):1177–1200, Jan 2016.
Teck Long Kong and Nor Ashidi Mat Isa. Enhancer-based contrast enhancement technique for non-uniform illumination and low-contrast images. Multimedia Tools and Applications, 76(12):14305–14326, Jun 2017.
Yu-Ren Lai, Ping-Chuan Tsai, Chih-Yuan Yao, and Shanq-Jang Ruan. Improved local histogram equalization with gradient-based weighting process for edge preservation. Multimedia Tools and Applications, 76(1):1585–1613, Jan 2017.
Shiv Ram Dubey, Satish Singh, and Rajat Singh. A multi-channel based illumination compensation mechanism for brightness invariant image retrieval. Multimedia Tools and Applications, 74:11223–11253, Dec 2015.
Yunbo Rao, Lei Hou, Zhihui Wang, and Leiting Chen. Illumination-based nighttime video contrast enhancement using genetic algorithm. Multimedia Tools and Applications, 70(3):2235–2254, Jun 2014.
Jianbing Shen, Xiaoshan Yang, Yunde Jia, and Xuelong Li. Intrinsic images using optimization. In CVPR 2011, pages 3481–3487, 2011.
Yanxiang Han and Zhisheng Zhang. An efficient estimation method for intensity factor of illumination changes. Multimedia Tools Appl., 72(3):2619–2632, October 2014.
Alok Kumar Singh Kushwaha and Rajeev Srivastava. Automatic moving object segmentation methods under varying illumination conditions for video data: comparative study, and an improved method. Multimedia Tools and Applications, 75(23):16209–16264, Dec 2016.
Sebastian Ruder. An overview of gradient descent optimization algorithms, 2017.
Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’96, page 303–312, New York, NY, USA, 1996. Association for Computing Machinery.
Kyung-Jin Kim, Byung-Seo Park, Jin-Kyum Kim, Dong-Wook Kim, and Young-Ho Seo. Holographic augmented reality based on three-dimensional volumetric imaging for a photorealistic scene. Opt. Express, 28(24):35972–35985, Nov 2020.
Kyung-Jin Kim, Byung-Seo Park, Dong-Wook Kim, Soon-Chul Kwon, and Young-Ho Seo. Real-time 3d volumetric model generation using multiview rgb-d camera. Journal of Broadcast Engineering, 3(3), May 2020.
