ADVISOR: Frédéric Devernay
TEL: 04 76 61 52 58
EMAIL: frederic.devernay at inria.fr
TEAM AND LAB: PRIMA team, Inria Rhône-Alpes & LIG
THEMES: Image, Vision
DURATION: 3 to 6 months
Above: Examples of panoramic video cameras by Immersive Media (Dodeca 2360), PtGrey (Ladybug) and Totavision
Panoramic stitching from photographs taken by a standard camera has been around for more than a decade, and many commercial or free stitching tools ar now available which implement state-of-the-art stitching methods (for example Autopano, Panorama Tools, or Hugin, Autodesk Stitcher).
For panoramic video, all images must be taken simultaneously, so that the hardware system usually consists of several high-resolution cameras, and each frame of the resulting video has to be stitched from all videos. Sample applications for panoramic video include movies for planetariums or Omnimax (Totavision), or interactive movies (Loop'in).
Image stitching consists of several phases: Image alignment, image warping, and blending. For panoramic video, the image alignment and warping phases can be treated using the same methods used for photographs, since the relative position of cameras does not change over time. However, the blending phase is very different, because a panoramic video cannot tolerate temporal artifacts, where an objects suddenly jumps from one place to another. Most artifacts present in panoramas are either due to images that were not taken simultaneously (which does not occur in panoramic video), or residual parallax due to the fact that the cameras' optical centers are not at the exact same position in 3D (which unfortunately happens in panoramic video because several cameras cannot be physically at the same place). We are thus essentially interested in methods that could reduce artifacts due to residual parallax (see example below).
Traditional panorama blending techniques include feathering or alpha blending [1], optimal seam [2], laplacian pyramid blending [3] or its simplification, two-band blending [4], and gradient domain blending [5,6]. Feathering and laplacian pyramid blending are linear operations, so that when the images change slowly the output will change slowly. On the opposite, optimal seam and gradient dimain blending are the result of an optimization, and the optimum may jump abruptly from one local minimum to the other when images change slowly. This second category should thus not be considered for panoramic video blending. However, even with the first category, a detail may slowly disappear from one place to reappear at another place in the images, so that the parallax effects have to be dealt with separately.
Several methods propose to compensate the effects of parallax [7,8,9,10], and within this project we propose to compensate the effects of parallax before the blending stage. Using the stereo disparity or the optical flow between cameras in the overlap regions, we will apply view-interpolation techniques, combined with ideas borrowed from the laplacian pyramid blending, to produce panoramic videos that exhibit no temporal or spatial artifacts.
Above: the effect of parallax error on a stitched panoramic image.
Please send a motivition letter and a resumé (CV) by email to frederic.devernay at inria.fr
[1] Richard Szeliski, Image Alignment and Stitching: A Tutorial, Microsoft Research Technical Report MSR-TR-2004-92, October 2004. http://research.microsoft.com/pubs/70092/tr-2004-92.pdf
[2] Efros, A., Freeman, W.: Image quilting for texture synthesis and transfer. Proceedings of SIGGRAPH 2001 (2001) 341-346
[3] C. H. Anderson , J. R. Bergen , P. J. Burt , J. M. Ogden: Pyramid Methods in Image Processing, Radio Corporation of America, RCA Engineer, 29(6):33-41 http://persci.mit.edu/pub_pdfs/RCA84.pdf
[4] Matthew Brown and David G. Lowe, "Recognising panoramas," International Conference on Computer Vision (ICCV 2003), Nice, France (October 2003), pp. 1218-25. http://www.cs.ubc.ca/~lowe/papers/brown03.pdf
[5] Perez ,Gangnet and Blake ,Poisson Image Editing ,Proc. Siggraph, 2003
[6] A. Levin, A. Zomet, S. Peleg and Y. Weiss. Seamless Image Stitching in the Gradient Domain. Proc. of the European Conference on Computer Vision (ECCV), Prague, May 2004. http://www.wisdom.weizmann.ac.il/~levina/papers/eccv04-blending.pdf
[7] Heung-Yeung Shum; Szeliski, R.; , "Construction and refinement of panoramic mosaics with global and local alignment," Computer Vision, 1998. Sixth International Conference on , vol., no., pp.953-956, 4-7 Jan 1998 http://research.microsoft.com/pubs/75611/ShumSzeliski-ICCV99.pdf
[8] Kang S. B., Szeliski R., Uyttendaele M.: Seamless Stitching Using Multi-Perspective Plane Sweep. Microsoft Research Technical Report MSR-TR-2004-48, June 2004. ftp://ftp.research.microsoft.com/pub/tr/TR-2004-48.pdf
[9] Michael Adam, Christoph Jung, Stefan Roth, and Guido Brunnett. Real-time stereo-image stitching using GPU-based belief propagation. In Proc. of the Workshop on Vision, Modeling and Visualization (VMV), Braunschweig, Germany, Nov. 2009 http://www.gris.tu-darmstadt.de/%7Esroth/pubs/vmv2009.pdf
[10] Alan Brunton, Chang Shu, "Belief Propagation for Panorama Generation," 3D Data Processing Visualization and Transmission, International Symposium on, pp. 885-892, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06), 2006 http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=5765151
[11] Helmut Dersch, Panorama Tools, Open source software for Immersive Imaging, 2007