Inverted FTIR: easy multitouch sensing for flatscreens

Inverted FTIR: Easy Multitouch Sensing for Flatscreens Florian Echtler, Andreas Dippon, Marcus T¨ nnis, Gudrun Klinker o {echtler|dippona|toennis|klinker}@in.tum.de Fachgebiet Augmented Reality Technische Universit¨ t M¨ nchen a u Institut f¨ r Informatik I16 u Boltzmannstr. 3, 85747 Garching, Germany ABSTRACT The increased attention which multitouch interfaces have received in recent years is partly due to the availability of cheap sensing hardware such as FTIR-based screens. However, this method has so far required a bulky projector-camera setup behind the screen. In this paper, we present a new approach to FTIR sensing by ”inverting” the setup and placing the camera in front of the screen. This allows the use of unmodified flat screens as display, thereby dramatically shrinking the space required behind the screen and enabling the easy construction of new types of interactive surfaces. INTRODUCTION In the last few years, research in the area of multitouch interfaces has received considerable attention. At least partly, this is due to various new optical touch-detection methods which have made the necessary sensing hardware available to a wide range of researchers. However, as these setups are usually based on a rear-projection screen, they require a significant amount of space behind the display. This limits the options which are available for designing such a system. For example, a wall-mounted display would probably require major structural alterations to the wall in order to achieve the necessary minimum distance between camera, projector and screen. The popular ”interactive table” format is also constrained by these space requirements, as there is usually no room beneath the table where users can put their feet. While one possible solution to this problem is using short-throw projectors, these are usually expensive and limited in their resolution. A different approach which solves the space problem and to a certain extent - also the resolution problem is to use a high-definition flatscreen. These are available in sizes which rival even large projection screens and mostly offer a considerably higher resolution at a lower price. Unfortunately, due to the opaque nature of a flatscreen display, the previously Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ITS ’09, November 23-25 2009, Banff, Alberta, Canada. Copyright 2009 ACM 978-1-60558-733-2/09/11... $10.00. Figure 1. Using an LCD screen with inverted FTIR mentioned sensing methods can not be employed directly as they require the camera to capture light shining through the screen. While some attempts have been made at removing the camera entirely and embedding the sensor directly behind the display, they require major re-engineering of the entire display and deep knowledge of electronic design. In this paper, we therefore present a novel approach. Similar to the back-projection method, an FTIR sensing surface is put in front of a regular, unmodified flatscreen display. An infrared camera which views the screen from the same side as the user is attached to the display in an off-center position to keep interference with the users’ actions to a minimum. By synchronizing the FTIR light source with the camera, a sufficiently high contrast can be achieved to capture infrared light shining through the user’s finger upon surface contact. A setup based on this concept is shown in figure 1. RELATED WORK As previously mentioned, a variety of sensing techniques has been published in recent years which have contributed to the popularity of multitouch research. Probably the most widely used one is frustrated total internal reflection (FTIR) which has been presented by J. Han in 2005 [5]. As our approach is based on this method, it shall be detailed in the following section. Another approach for rear-projected touch sensing is the socalled diffuse illumination method. Infrared light sources illuminate the screen from behind. Objects such as fingers approaching the surface reflect additional light back towards the camera which can detect the increased brightness in those surface regions. To counter the significant space requirements of these solutions, other approaches get rid of the camera altogether and embed the entire sensor directly behind the flatscreen. This method was first presented as ThinSight by Izadi et al. [7]. Small holes have been cut into the backlight diffusor and allow an array of commercial infrared-based distance sensors to operate through the LCD panel, resulting in a similar sensor response to the diffuse illumination method. A second, hybrid method called FlatIR has been presented by Hofer et al. [6]. Here, an FTIR overlay has been added on top of an LCD screen. An array of photodiodes behind the panel detects the decoupled infrared light from the contact points. Other, non-optical sensing techniques also exist. For example, the DiamondTouch surface [2] is based on capacitive coupling and can therefore remain quite thin. It is usually combined with a front-mounted projector. However, although it is commercially available, the considerable price has somewhat limited its distribution. Another option are transparent capacitive sensor overlays similar to that used in the iPhone [8]. For example, projectedcapacitive touchscreen overlays are available from EloTouch [4]. Unfortunately, as these screens require the deposition of thin layers of indium tin oxide (ITO) on the glass substrate, they are also very expensive to manufacture. In summary, it is apparent that most existing sensor solutions still are either expensive, difficult to build or just bulky. We shall now present our approach to solve some of these problems. LED bright spot finger acrylic glass projection screen projector/ camera Figure 2. FTIR principle of operation When changing to a more abstract point of view for a moment, it becomes apparent that a finger touching the surface is simply lit from below. However, a human finger is far from being totally opaque, as a short experiment with a flashlight can easily show. In particular, it allows partial transmission of the red and infrared parts of the light spectrum. This property is sometimes being used in pulse sensors which are clipped to the finger tip and measure the transmission intensity of infrared light. Due to this effect, some of the infrared light hitting the finger through the FTIR surface will also radiate outwards. Moreover, a fraction of the light reflected downwards will be reflected a second time on the display surface, thereby creating a halo of infrared light around the contact point. These effects are illustrated in figure 3. camera finger LED acrylic glass LCD panel Figure 3. Inverted FTIR THEORY OF OPERATION As the presented approach is based on the FTIR sensing method, we will describe it briefly. A thin acrylic plate is placed in front of the screen. On the rim of this plate, infrared LEDs are placed which shine IR light into the material. As a large difference in refractive index exists between the acrylic and the surrounding air, most of the light is reflected internally, similar to an optic fiber. However, when a dense material such as human skin touches the surface, then the total internal reflection is interrupted at this point and some of the infrared light radiates out of the plate. This process is illustrated in figure 2. These various paths emit light towards the user which can now be sensed by an infrared camera placed on the front side of the display. The total intensity of the light which is reflected outwards is lower than that reflected towards the surface. Therefore, special care must be taken to achieve a high contrast with respect to stray light such as the infrared emissions from the LCD backlight. One way to improve contrast is to pulse the LEDs while synchronizing these pulses with the camera’s exposure time. While this method slightly increases the complexity of the setup by requiring a control circuit, it is able to increase the contrast by up to a factor of 8 as described in [3]. The implementation details of the camera setup in front of the screen will be given in the next section. HARDWARE SETUP For evaluation of our approach, we have chosen an 42 inch LCD television screen from LG. This display supports full HD resolution with 1920 x 1080 pixels and offers a maximum brightness of 500cd/m2 , which is appropriate for most indoor conditions. Between the front bezel and the LCD panel, a 5 mm acrylic plate was inserted. The plate is held in place by the reattached front cover and also provides additional protection for the LCD panel. On the rim of this acrylic plate, 150 high-powered infrared LEDs (Osram SFH 4650) with a emission wavelength of 850nm are attached with instant glue. These LEDs are organized in 30 groups of 5 diodes each. Within each group, the LEDs are connected in series, whereas the groups themselves are wired in parallel to a 12V power supply through the switching circuit. This circuit receives trigger signals from the camera at the start of each exposure interval and activates the LEDs for about 300µs, causing them to emit a single intense flash. As the exposure interval is configured to the same duration, the contrast during that short period is much higher than possible with continuous operation. The camera used in this setup is a PointGrey Dragonfly2 with a resolution of 1024 x 768 pixels running at a frame rate of 30 Hz. The camera is equipped with an infrared low-pass filter to block all interference from visible light. A diagram of the entire setup is shown in figure 4. age 5(a) was taken using standard camera settings and illustrates the view area of the camera. However, this mode of operation is unsuitable for touch detection due interference by ambient light. Therefore, synchronized illumination [3] is used during operation. Image 5(b) illustrates the significant contrast gain afforded by this technique, as it was taken under the same external lighting conditions. Five touching fingers are clearly visible. (a) standard settings camera mounting bracket infrared LEDs acrylic glass LCD panel electronics rear case front bezel (b) synchronized illumination Figure 5. Raw camera images IMAGE PROCESSING Figure 4. Schematic view (not to scale) The correct placement of the camera is of particular importance. By using a fish-eye lens with a focal length of 2.5 mm, the camera can be moved close to the screen while still being able to view the entire surface. However, care must be taken to position the camera in such a way that the user is never impeded. For a vertical display, attaching the camera to a short beam above the user’s heads is the most suitable solution. In case of a horizontal display, the best position for the camera would likely be centered above the surface to avoid blocking one side of the display. However, attaching the camera to the ceiling should be considered in this case, as the mounting beam might otherwise still interfere with the user’s movement. In figure 5, two images taken by the camera are shown. Im- A closeup of a raw image from the camera is shown in figure 6. The halos around the three fingers touching the surface are clearly visible, also the increased brightness of the fingertips themselves. Although this image is similar to that taken from the backside of an FTIR surface, there are some important differences. In FTIR setups, the bright blobs resulting from surface contacts usually reach maximum intensity near their center while gradually getting darker towards the border of the contact area. With this setup, the brightness distribution is less clearly defined. In most cases, a contact results in two bright stripes to the left and right of the finger with a slightly darker area (the finger itself) in-between. It is possible to directly process this image with widely used touch detection software such as touchlib [10] or reacTIVision [9]. When applying the usual image processing steps such as background subtraction, thresholding, denoising and connected-component analysis to such an image, this will raw camera image. One aspect which we have not yet been able to study are long-term effects of the modification to the hardware. The acrylic front plate acts as heat insulator, thereby increasing the temperature of the panel assembly during operation. This may reduce the lifetime of the backlight or of the LCD panel itself. While this sensing method is highly suitable for a table-based setup where users are able to place their feet beneath the table, some reports suggest that running LCD screens horizontally for longer periods of time may also lead to premature failure of the device. We will continue to monitor these effects in our setup and try to mitigate them, e.g. by adding active cooling. Figure 6. Closeup: touching the surface with 3 fingers REFERENCES usually result in the two bright stripes besides the finger being detected as blobs which move and rotate in parallel. This data can directly be delivered to multitouch-aware software and will produce the expected results in most cases despite generating two blobs per contact point. As turning the finger rotates the blobs with respect to each other, even singlefingered rotation is possible to a certain degree. Depending on the camera location and angle, however, this may at some point lead to occlusion of one blob by the finger itself. An important observation in this context was that the backlight of the LCD panel may interfere with the touch detection under some circumstances, as the fluorescent tubes which are usually employed emit considerable amounts of infrared light. While the backlight does reduce the contrast between background image and touching fingers, this does not pose a problem as the LED pulsing method still produces sufficiently high contrast. However, some LCDs offer the ability to regulate the brightness of the backlight. This is achieved through pulse-width modulation of the backlight voltage. In the case of the LG television which we are using, this modulation is done at a frequency of 200 Hz. With the camera running at a frame rate of 30 Hz, this causes intense pulsing and flickering of the LCD background in the camera image. An easy solution to this problem is to turn the LCD brightness up to maximum intensity, thereby effectively turning the modulation off. This results in a constant, even background brightness. Note that LCD panels are mostly transparent to infrared light regardless of the displayed image. Therefore, changing content of the on-screen image does not influence the background brightness. DISCUSSION AND OUTLOOK 1. Dietz, P. and Leigh, D. DiamondTouch: a multi-user touch technology. In UIST ’01: Proceedings of the 14th annual ACM symposium on User interface software and technology (2001), 219–226. 2. Echtler, F., Sielhorst, T., Huber, M., and Klinker, G. A Short Guide to Modulated Light. In TEI ’09: Proceedings of the conference on tangible and embedded interaction (2009). 3. Elo TouchSystems. Projected capacitive touchscreens. http://www.elotouch.com/Technologies/ ProjectedCapacitive/, accessed 2009-06-06. 4. Han, J. Low-cost multi-touch sensing through frustrated total internal reflection. In UIST ’05: Proceedings of the 18th annual ACM symposium on User interface software and technology (2005), 115–118. 5. Hofer, R., Naeff, D., and Kunz, A. FLATIR: FTIR multi-touch detection on a discrete distributed sensor array. In TEI ’09: Proceedings of the 3rd International Conference on Tangible and Embedded Interaction, ACM (2009), 317–322. 6. Izadi, S., Hodges, S., Butler, A., Rrustemi, A., and Buxton, B. ThinSight: integrated optical multi-touch sensing through thin form-factor displays. In EDT ’07: Proceedings of the 2007 workshop on Emerging displays technologies (2007), 6. 7. Jobs, S. et al. Touch screen device, method, and graphical user interface for determining commands by applying heuristics. U.S. Patent 7,479,949, 2008. 8. Kaltenbrunner, M. and Bencina, R. reacTIVision: a computer-vision framework for table-based tangible interaction. In TEI ’07: Proceedings of the 1st international conference on Tangible and embedded interaction (2007), 69–74. 9. NUI Group Community. Touchlib. http://www.nuigroup.com/touchlib/, accessed 2009-06-28. We have presented an easily implementable method to enable FTIR-based multitouch sensing on an unmodified flatpanel display by placing the camera on the same side as the user. While construction of the touch-sensitive surface itself requires some basic engineering and electronics skills, the significantly more complex disassembly of the display panel itself is not necessary. Existing image processing software can be used without modifications to detect touches in the