Using the Wii Remote to correct perspective

Introduction

Soon after I started shooting with a fisheye lens I realized the image projected by this lens can be remapped into a "nicer" projection. Of course this is nothing new: people has been "defishing" these lenses from their native (azimuthal equal area) to a rectilinear for a long time. This is a rather automatic process because the rectilinear is also an azimuthal projection: image in, reprojected image out. Another azimuthal projection that is often overlooked by people who defish their images is the stereographic. One interesting property of the stereographic projection is that it is conformal (it preserves angles in any location, which in practical terms means that small areas maintain their shape). Extreme wide-angle images using the stereographic projection look more natural than rectilinear because they don't unnaturally widen objects at the edges of the frame. Unfortunately it also means that straight lines get curved (the closer to the edges, the more curved). But its curvature is less accentuated than the azimuthal equal area. You lose something, you gain something.

Like many, I started to experiment with remapping fisheye images to cylindrical projections and discovered that I liked the Mercator projection (which is also conformal). The Mercator has one advantage over the stereographic: it keeps vertical lines straight. When I compared the stereographic and the Mercator projections of the same images I tended to like more the Mercator (particularly in images that had clear straight lines, such as corners of rooms, door frames, buildings, etc).

The major problem of converting fisheyes to the Mercator projection is that one needs to set the roll and pitch by hand, or use software (such as Hugin) to set interactively set 2 control lines (usually two vertical lines), then use PToptimizer to find the pitch and roll. In other words it is a time consuming process.

In Christmas 2006 I was asked to photograph an event and chose to do it using the Peleng 8mm. I felt the lens let me get close to the action, and include more people on the photos than with any other lens I owned. I liked the look of the remapped images using Mercator.

I did not mind to do it for a few images, but it was tedious for a bunch of them. For these reasons the Christmas images never reached their intended audience.

I thought: all I need is a way to automatically record pitch and roll as I take the photo. Easier said than done. I needed an inclinometer with the ability to record its readings some time of memory.

I approached Mantis Cheng, an electronics expert in our department, who suggested the perfect solution: use a Wii Remote (WiiR). It can measure angles, and it has bluetooth, so you can record its readings using a computer.

Finding a WiiR wasn't easy. It took me around 20 visits to the local computer games shop. Every day the same litany: come tomorrow, early; sorry, we can reserve one for you.

Using a WiiR to record pitch and roll

At last I got one. I modified an open source DarwiinRemote http://sourceforge.net/projects/darwiin-remote/ to record inclination. It happens that if you keep the WiiR in a stationary state, its three accelerometers can be used as two inclinometers. Fortunately most of the time, when one takes a handheld photo, one keeps the camera steady. Steady enough to read the inclination of the WiiR.



I also modified the driver to record the reading from the accelometers, the estimated inclinations, and the timestamp. I would then use the timestamp to match the images from the camera to the readings from the WiiR.

The process would be:

• Start the driver, with output to the stdout
  
• Initialize the WiiR
  
• Take a photo of the screen to match time of the computer and that of the camera
  
• Take photographs
  
• End driver
  
• Save log.
  

Later the timestamp of the first photo would provide a shift in seconds between the camera and the WiiR logs. This provides the basis to find the proper reading for a given photo.

In order to make this process easier it is recommended to keep the camera stationary for at least a couple of seconds.

Results

What do we get from the WiiR: two angles: pitch and roll. Unfortunately it is not very precise. My experiments indicate that the values returned by the accelerometers vary from -20 to +20 (once we have normalized the value 0 to the horizon), corresponding to -45 to 45 degrees; the values are evenly spaced), giving a maximum theoretical precision of approximately 2.25 +/- 1.12 degrees.

Such angles are then fed to the remapping software, and there is no need for manual intervention.

This diagram shows the process followed:



There are three practical uses for this approach:

1. Correcting tilted horizons: once the roll is known, it is trivial to ``straighten'' a photo;
   
2. Correcting perspective errors: using panoramic software (such as panotools) it is possible to automatically remap the photo taking into account the angle in which the camera was tilted-up (the pitch).
   
   

   Photograph taken with a rectilinear lens (Canon EF-S 10-22)
   

   Remapped image. Output projection rectilinear.
   
   
   
3. Remapping to another projection: once the pitch and roll is known a photo can be mapped to any other normal projection.
   
   

   Taken with a Nikkor 10.5 Fisheye lens (on a Canon 20d).
   

   Remapped image using Lambert Equal Area.
   
   

The experiments I have performed show that the precision of cheap accelerometers (as found in the WiiR) is enough to effective perform all three approaches if the photo is not taken with a large telephoto (the longer the lens, the more the error is magnified).

With a fast enough processor these corrections could happen in the camera, in real time, and the user could preview them using the camera's LCD display. These might be features that one day become common in our digital cameras.


These results have been published in: German, Daniel; d'Angelo, Pablo ; Gross, Michael and Postle, Bruno (June 2007). "New Methods to Project Panoramas for Practical and Aesthetic Purposes". "Proceedings of Computational Aesthetics 2007": 15--22, Banff: Eurographics.