liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Efficient Video Rectification and Stabilisation for Cell-Phones
Linköping University, Department of Electrical Engineering. Linköping University, The Institute of Technology.
Linköping University, Department of Electrical Engineering, Computer Vision. Linköping University, The Institute of Technology.ORCID iD: 0000-0002-5698-5983
2012 (English)In: International Journal of Computer Vision, ISSN 0920-5691, E-ISSN 1573-1405, Vol. 96, no 3, 335-352 p.Article in journal (Refereed) Published
Abstract [en]

This article presents a method for rectifying and stabilising video from cell-phones with rolling shutter (RS) cameras. Due to size constraints, cell-phone cameras have constant, or near constant focal length, making them an ideal application for calibrated projective geometry. In contrast to previous RS rectification attempts that model distortions in the image plane, we model the 3D rotation of the camera. We parameterise the camera rotation as a continuous curve, with knots distributed across a short frame interval. Curve parameters are found using non-linear least squares over inter-frame correspondences from a KLT tracker. By smoothing a sequence of reference rotations from the estimated curve, we can at a small extra cost, obtain a high-quality image stabilisation. Using synthetic RS sequences with associated ground-truth, we demonstrate that our rectification improves over two other methods. We also compare our video stabilisation with the methods in iMovie and Deshaker.

Place, publisher, year, edition, pages
Springer Verlag (Germany) , 2012. Vol. 96, no 3, 335-352 p.
Keyword [en]
Cell-phone, Rolling shutter, CMOS, Video stabilisation
National Category
Engineering and Technology
Identifiers
URN: urn:nbn:se:liu:diva-75277DOI: 10.1007/s11263-011-0465-8ISI: 000299769400005OAI: oai:DiVA.org:liu-75277DiVA: diva2:505943
Note
Funding Agencies|CENIIT organisation at Linkoping Institute of Technology||Swedish Research Council||Available from: 2012-02-27 Created: 2012-02-24 Last updated: 2017-12-07
In thesis
1. Geometric Computer Vision for Rolling-shutter and Push-broom Sensors
Open this publication in new window or tab >>Geometric Computer Vision for Rolling-shutter and Push-broom Sensors
2012 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Almost all cell-phones and camcorders sold today are equipped with a CMOS (Complementary Metal Oxide Semiconductor) image sensor and there is also a general trend to incorporate CMOS sensors in other types of cameras. The sensor has many advantages over the more conventional CCD (Charge-Coupled Device) sensor such as lower power consumption, cheaper manufacturing and the potential for on-chip processing. Almost all CMOS sensors make use of what is called a rolling shutter. Compared to a global shutter, which images all the pixels at the same time, a rolling-shutter camera exposes the image row-by-row. This leads to geometric distortions in the image when either the camera or the objects in the scene are moving. The recorded videos and images will look wobbly (jello effect), skewed or otherwise strange and this is often not desirable. In addition, many computer vision algorithms assume that the camera used has a global shutter, and will break down if the distortions are too severe.

In airborne remote sensing it is common to use push-broom sensors. These sensors exhibit a similar kind of distortion as a rolling-shutter camera, due to the motion of the aircraft. If the acquired images are to be matched with maps or other images, then the distortions need to be suppressed.

The main contributions in this thesis are the development of the three dimensional models for rolling-shutter distortion correction. Previous attempts modelled the distortions as taking place in the image plane, and we have shown that our techniques give better results for hand-held camera motions.

The basic idea is to estimate the camera motion, not only between frames, but also the motion during frame capture. The motion can be estimated using inter-frame image correspondences and with these a non-linear optimisation problem can be formulated and solved. All rows in the rolling-shutter image are imaged at different times, and when the motion is known, each row can be transformed to the rectified position.

In addition to rolling-shutter distortions, hand-held footage often has shaky camera motion. It has been shown how to do efficient video stabilisation, in combination with the rectification, using rotation smoothing.

In the thesis it has been explored how to use similar techniques as for the rolling-shutter case in order to correct push-broom images, and also how to rectify 3D point clouds from e.g. the Kinect depth sensor.

Place, publisher, year, edition, pages
Linköping: Linköping University Electronic Press, 2012. 85 p.
Series
Linköping Studies in Science and Technology. Thesis, ISSN 0280-7971 ; 1535
Keyword
rolling shutter cmos video rectification stabilisation push-broom kinect
National Category
Engineering and Technology Computer Vision and Robotics (Autonomous Systems) Signal Processing
Identifiers
urn:nbn:se:liu:diva-77391 (URN)978-91-7519-872-9 (ISBN)
Presentation
2012-06-08, Visionen, Hus B, Campus Valla, Linköpings universitet, Linköping, 13:00 (English)
Opponent
Supervisors
Projects
VGS
Available from: 2012-05-28 Created: 2012-05-14 Last updated: 2015-12-10Bibliographically approved
2. Geometric Models for Rolling-shutter and Push-broom Sensors
Open this publication in new window or tab >>Geometric Models for Rolling-shutter and Push-broom Sensors
2014 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Almost all cell-phones and camcorders sold today are equipped with a  CMOS (Complementary Metal Oxide Semiconductor) image sensor and there is also a general trend to incorporate CMOS sensors in other types of cameras. The CMOS sensor has many advantages over the more conventional CCD (Charge-Coupled Device) sensor such as lower power consumption, cheaper manufacturing and the potential for onchip processing. Nearly all CMOS sensors make use of what is called a rolling shutter readout. Unlike a global shutter readout, which images all the pixels at the same time, a rolling-shutter exposes the image row-by-row. If a mechanical shutter is not used this will lead to geometric distortions in the image when either the camera or the objects in the scene are moving. Smaller cameras, like those in cell-phones, do not have mechanical shutters and systems that do have them will not use them when recording video. The result will look wobbly (jello eect), skewed or otherwise strange and this is often not desirable. In addition, many computer vision algorithms assume that the camera used has a global shutter and will break down if the distortions are too severe.

In airborne remote sensing it is common to use push-broom sensors. These sensors exhibit a similar kind of distortion as that of a rolling-shutter camera, due to the motion of the aircraft. If the acquired images are to be registered to maps or other images, the distortions need to be suppressed.

The main contributions in this thesis are the development of the three-dimensional models for rolling-shutter distortion correction. Previous attempts modelled the distortions as taking place in the image plane, and we have shown that our techniques give better results for hand-held camera motions. The basic idea is to estimate the camera motion, not only between frames, but also the motion during frame capture. The motion is estimated using image correspondences and with these a non-linear optimisation problem is formulated and solved. All rows in the rollingshutter image are imaged at dierent times, and when the motion is known, each row can be transformed to its rectied position. The same is true when using depth sensors such as the Microsoft Kinect, and the thesis describes how to estimate its 3D motion and how to rectify 3D point clouds.

In the thesis it has also been explored how to use similar techniques as for the rolling-shutter case, to correct push-broom images. When a transformation has been found, the images need to be resampled to a regular grid in order to be visualised. This can be done in many ways and dierent methods have been tested and adapted to the push-broom setup.

In addition to rolling-shutter distortions, hand-held footage often has shaky camera motion. It is possible to do ecient video stabilisation in combination with the rectication using rotation smoothing. Apart from these distortions, motion blur is a big problem for hand-held photography. The images will be blurry due to the camera motion and also noisy if taken in low light conditions. One of the contributions in the thesis is a method which uses gyroscope measurements and feature tracking to combine several images, taken with a smartphone, into one resulting image with less blur and noise. This enables the user to take photos which would have otherwise required a tripod.

Place, publisher, year, edition, pages
Linköping: Linköping University Electronic Press, 2014. 41 p.
Series
Linköping Studies in Science and Technology. Dissertations, ISSN 0345-7524 ; 1615
National Category
Computer Vision and Robotics (Autonomous Systems) Computer Engineering
Identifiers
urn:nbn:se:liu:diva-110085 (URN)10.3384/diss.diva-110085 (DOI)978-91-7519-255-0 (ISBN)
Public defence
2014-09-19, Visionen, hus B, Campus Valla, Linköpings universitet, Linköping, 10:15 (English)
Opponent
Supervisors
Note

The research leading to this thesis has received funding from CENIIT through the Virtual Global Shutters for CMOS Cameras project.

Available from: 2014-09-02 Created: 2014-09-02 Last updated: 2015-12-10Bibliographically approved

Open Access in DiVA

fulltext(12749 kB)11166 downloads
File information
File name FULLTEXT01.pdfFile size 12749 kBChecksum SHA-512
63f23366a55709f410efb0b412742e52af0c8db03be698d1a1951aca5ab263d217e6aa5d3f4360bb1a3379405ca530a509982fbeed9c0814535965b133d407b9
Type fulltextMimetype application/pdf
Supplementary material(55550 kB)185 downloads
File information
File name ATTACHMENT01.zipFile size 55550 kBChecksum SHA-512
2fcb6b8890558fd9f3b5b948753be81f2b64a668a8637f420711f1873b61da477314b2f330dcbc54cb85181c4f3b70d0261bbdfeba69296b842031e7cf4174e4
Type attachmentMimetype application/zip

Other links

Publisher's full text

Authority records BETA

Ringaby, ErikForssén, Per-Erik

Search in DiVA

By author/editor
Ringaby, ErikForssén, Per-Erik
By organisation
Department of Electrical EngineeringThe Institute of TechnologyComputer Vision
In the same journal
International Journal of Computer Vision
Engineering and Technology

Search outside of DiVA

GoogleGoogle Scholar
Total: 11166 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 6709 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf