Multi-Touch for Stereoscopic Displays

Multi-Touch for Stereoscopic Displays
Dr.NaveenBansal Profile Pic
Published Date:25-10-2017
Your Website URL(Optional)
9 Multi-Touch for Stereoscopic Displays Dimitar Valkov Always think of what is useful and not what is beauti- ful. Beauty will come on its own accord. —Nikolai Gogol 9.1 Understanding 3D Touch Notwithstanding the initial excitement around multi-touch interfaces it has quickly become apparent that using touch as the primary input modality poses (even in 2D contexts) some fundamental limitations for traditional interface design Benko and Wigdor 10, Müller-Tomfelde et al. 10. Some of the most important problems are the missing hover, occlusion and precision problems and – depending on the implementation – missing or non-adequate visual feedback. In particular, the size of the human fingers and the lack of sensing precision make precise touch screen interactions difficult Benko et al. 06, Holz and Baudisch 10. The approaches to handle this can be roughly separated into two groups. Approaches from the first group try to shift the problem into the interface design space. Therefore, precise selection is distinguished as a new interface requirement, which demands additional functionality and thus an extended set of interaction metaphors or techniques. Characteristic for the second group of solutions is that they try to overcome or reduce the problem by modeling the user perception and action during the touch. Thus, these approaches try to identify a set of traceable features, which may help to better recognize the intended touch position. Examples of such features are the orientation of the user’s finger Holz and Baudisch 10 or visual features on 205206 9. Multi-Touch for Stereoscopic Displays the upper finger surface Holz and Baudisch 11. The primary benefit of these approaches over the pure “brute-force” interface solutions is that they help to understand the mechanics of a touch gesture, when used for input, and provide indications which help to identify the sources of the inaccuracy in traditional touch devices. Recent work has also identified the hand pre-shaping as a valuable source of information in this regard Daiber et al. 12b. Indeed, as the investigations of many neuro-psychological and robotic research groups have shown, there is a strong correlation between the course of hand shaping and the object, which is subject to interaction Daiber et al. 12b, Santello et al. 02. Extending the interaction environment to the third dimension usually intensifies the impact of these issues on the user experience and satisfaction Schöning et al. 09 and introduces new problems which are negligible in monoscopic contexts. In this chapter we examine these problems in more detail and consider several high-level approaches to address them. Furthermore, we investigate the effect of stereoscopic parallax on the touch precision and discuss some of the design implications for designing stereoscopic 3D touch interfaces. 9.1.1 Problems with Stereoscopic Touch Interfaces Recently many approaches for extending multi-touch interaction techniques to 3D applications with monoscopic rendering have been proposed Hancock et al. 07, Martinet et al. 10, Reisman et al. 09, Wilson et al. 08. For instance, Hilliges et al. Hilliges et al. 09 have tested two depth sensing approaches to enrich the multi-touch interaction space beyond the touch surface in a tabletop setup with monoscopic projection. Hancock et al. Hancock et al. 07 have introduced the concept of shallow-depth 3D, i.e., 3D with limited depth, in order to extend the interaction with digital 2D surfaces and have developed one, two, and three fingers interaction techniques for object selection and manipulation in this context. Martinet et al. Martinet et al. 10 have designed a multi-view direct and a single- view indirect technique for 3D object positioning, and Reisman et al. Reisman et al. 09 propose an energy-minimization technique for adapting 2D interaction to 3D transformation. The benefits of using physics engines for multi-touch input specification are discussed by Wilson et al. Wilson et al. 08, and the interaction with objects with negative parallax on a multi-touch tabletop setup is further addressed by Benko et al. Benko and Feiner 07, who have proposed the balloon selection metaphor to support precise object selection and manipulation in augmented reality setups. In 2007 Grossman and Wigdor Grossman and Wigdor 07 provided an exten- sive review of the existing work on interactive surfaces and developed a taxonomy to classify the current work and to point out new directions. This framework takes into account the perceived and the actual display space, the input space and the physical properties of an interactive surface. As shown in their work, 3D volu- metric visualizations are rarely being considered in combination with 2D direct9.1. Understanding 3D Touch 207 surface input. More recent surveys, e.g., Argelaguet and Andujar Argelaguet and Andujar 13, still identify 3D direct touch interaction as a promising research direction, which is still not sufficiently investigated. Direct touch interaction with stereoscopically rendered scenes introduces new challenges, as described by Schöning et al. Schöning et al. 09. In their work an anaglyph- or passive polarization-based stereo visualization was combined with FTIR-based touch detection on a multi-touch enabled wall, and approaches based on mobile devices for addressing the formulated parallax problems were discussed. A similar option for direct touch interaction with stereoscopically rendered 3D objects is to separate the interactive surface from the projection screen, as proposed by Schmalstieg et al. Schmalstieg et al. 99. In their approach, the user is provided with a physical transparent prop, which can be moved on top of the object of interest. This object can then be manipulated via single- or multi-touch gestures, since it has almost zero parallax with respect to the prop. Nevertheless, this requires instrumentation again, which may defeat some of the benefits of touch interaction. 9.1.2 Parallax Problem Stereoscopic perception requires each eye to see a slightly different perspective of the same scene, which results in two distinct projections on the display. Depending on the disparity between the two projections, virtual objects can be presented with positive, negative, or zero parallax, resulting in different visual impressions. • If objects are rendered with zero parallax they appear aligned with the plane of the display surface and are therefore perfectly suited for touch-based interaction Schöning et al. 09. Unlike the positive and negative parallax half-spaces, the zero parallax plane poses considerable constraints on the placement, dimensions, and form of the objects, and therefore contradicts the benefits of using stereoscopy, or 3D in general. In this context the question arises how sensitive humans are with respect to misalignment between visually perceived and tactually felt contact with virtual objects. For example, if an object is rendered at some small distance in front of the display surface, is the user going to move her finger through the object until she receives tactile feedback due to the contact with the display, and how small may this distance be? In particular, this may allow touch (and possibly multi-touch) interaction within stereoscopic environments without losing the advantages of common 2D techniques. While it is reasonable to assume that users tolerate a certain amount of misalignment between the perceived visual depth and the exact point at which haptic feedback is received Valkov et al. 12, similar effects may lead to misalignment between perceived and actual object depth depending on object size, form, texture, etc. This may then infer the perceived alignment between two objects or between an object and the plane of the display surface. Nevertheless, if 2D interaction is intended or the displayed virtual objects have no associated208 9. Multi-Touch for Stereoscopic Displays depth information (e.g. UI widgets), the zero parallax plane may provide superior user experience compared with alternative depth distributions. • Objects displayed with positive parallax are perceived to be behind the screen surface. These objects cannot be accessed directly, since the user’s reach is limited by the display. Since the display surface usually has no visual representation in a stereoscopically rendered scene, trying to reach an object with strong positive parallax may become unnatural and in some cases even harmful. Nevertheless, if the object is close to the surface – rendered with shallow depth – the only effect is that the user receives haptic feedback shortly before its visual representation is reached, i.e., the points of receiving haptic and visual feedbacks are spatially misaligned. In Section 9.5 we discuss the problem in more detail and make the first steps toward determining within what range this misalignment is still unnoticeable for the user. For objects rendered with strong positive parallax, indirect techniques might be more adequate. For instance, one could cast a virtual ray from the camera’s origin through the on-surface touch point and determine the first object hit by that ray or use some abstract interface widget Daiber et al. 12a to virtually move the user’s touches in the 3D space below the surface. Even though such techniques are indirect, it is often claimed that users experience them to be “natural” and “obvious” Bowman et al. 04, Daiber et al. 12a. • Objects that appear in front of the projection screen, i.e., objects with nega- tive parallax, introduce the major challenge in this context. When the user wants to interact with such an object by touching, she is limited to touching the area behind the object, since most touch-sensitive screens capture only direct contacts, or hover gestures close to the screen. Therefore the user has to penetrate the visual objects to reach the touch surface with her finger. In addition to the fact that users commonly consider this as unnatural (or not in- tuitive), the stereoscopic perception may be disturbed, since the user’s visual system is fed with contradicting information. If the user penetrates an object while focusing on her finger, the stereoscopic effect for the object would be disturbed, since the user’s eyes are not accommodated and converged on the projection screen’s surface. Thus the left and right stereoscopic images of the object’s projection would appear blurred and could not be merged anymore (Figure 9.1 (a)). However, focusing on the virtual object would lead to a disturbance of the stereoscopic perception of the user’s finger, since her eyes are converged to the object’s 3D position (Figure 9.1 (b)). In both cases the stereoscopic impression may be lost due to these artifacts. Another significant problem with stereoscopic touch interfaces is the dis- crepancy between the disparity and occlusion cues. Indeed, as illustrated in Figure 9.2 (b) if the user’s finger penetrates the object in the last phase of the touch gesture, binocular disparity cues are suggesting that her finger is already behind9.1. Understanding 3D Touch 209 Side Note 9.1: Accommodation-Convergence Accommodation-Convergence or simply accommodation reflex is the reflex of the eye to focus on an object. It consists of 2 simultaneous actions: (a) vergence is the simultaneous movement of both eyes, such that their optical axes cross on the object and thus allow stereoscopic vision, (b) accommodation of each eye’s lens shape and pupil size, such that the object is in focus. With real objects both actions are coherent, i.e., the vergence distance and lenses’ focal length are highly correlated. Nevertheless, stereo- scopic displays only simulate the effect of the vergence reflex by providing each eye with different (slightly shifted) projection. In contrast, the eyes’ lenses are always accommodated to the display surface. This is a common problem with virtually any stereoscopic display technol- ogy. Nevertheless, with touch interfaces the problem is sharply intensified, since at some point the user’s finger and the aimed object are at the same depth (i.e. same vergence distance), but the eye lenses have to be focused either on the finger or on the display. (a) (b) Figure 9.1: Illustration of the accommodation-convergence problem; The user is either focused on the finger (a), which makes the selection ambiguous, or on the object (b), which disturbs the visual perception of the finger. the object. Nevertheless, the stereoscopic projection on the display surface cannot occlude the finger (or any object for that matter) in front of it. Thus, the finger is occluding parts of the object, and occlusion cues are confirming that the object is in front of the screen (Figure 9.2 (a)). Since occlusion cues usually dominate over disparity, disparity cues may be ignored and the images for the left and the210 9. Multi-Touch for Stereoscopic Displays (a) (b) Figure 9.2: Illustration of the occlusion problem; while the occlusion cues (a) indicate that the user’s finger is in front of an object, binocular disparity cues (b) are suggesting that the user’s finger is behind the object. right eye may not be merged any more, which results in loss of the stereoscopic impression. In both cases touching an object may become ambiguous. However, as discussed in detail in Section 9.5, users have difficulties precisely estimating the depth of an object, which is displayed close enough to the surface, when they try to touch it. 9.1.3 Design Paradigms for Stereoscopic Touch Interaction While one could simply use a 3D tracking technique to capture the user’s finger or hand motions in front of a display surface, it has been shown that touching an intangible surface (i.e., touching the void) leads to confusion and a significant number of overshooting errors Chan et al. 10, and passive haptic has the potential to considerably enhance the user experience with touch- or grasp-based interfaces. Furthermore, touch interaction is nowadays becoming a standard for most mo- bile devices or tabletop setups, thus a change to other technology is usually not desirable, and sometimes not possible. Existing approaches to deal with the problems of touch interaction in stereo- scopic contexts could be roughly separated into three distinct paradigms (cf. Fig- ure 9.3): “Move the surface,” “Move the touches,” and “Perceptual illusions,” which are briefly discussed in the following. Move the Surface Paradigm The main point in the move the surface concept is that one can decouple the interactive surface from the display and move it freely in the 3D volume above the display. One possibility to achieve this is to use a multi-touch enabled transparent prop Schmalstieg et al. 99, Valkov et al. 10, which can be aligned with a floating9.1. Understanding 3D Touch 211 Figure 9.3: Illustration of the three design paradigms for touch interaction with stereoscopic content: (a) “move the surface” paradigm; (b) “move the touch” paradigm, and (c) “perceptual illusions” paradigm Side Note 9.2: Transparent Props One of the main problems with a transparent prop is that the user is usually holding it at arm’s length and the display surface is far behind the prop. While the stereoscopic projection could be tuned such that an object appears on the transparent prop’s surface the discrepancy between vergence and accommodation depths is usually so strong that the objects cannot be merged any more. Another significant problem with both transparent props and tangible views is that one has to precisely track both the prop and user’s fingers on top of the prop, which is a challenging task and current solutions are usually very limited and prone to errors. object and used as input to interact with this object in place. Thus, the user interacts “directly” with the object through the prop and receives haptic feedback. Nevertheless, since the objects aligned with the prop are projected with very large disparity, the users often have considerable problems maintaining the fusion of the images for the left and the right eyes. This is further impaired by even very small scratches on the surface of the prop, which may distract the eye accommodation on the top of the prop instead of on the display surface. Another recently published alternative is to use opaque props and a top projection exactly on the surface of these props, i.e., to use tangible views Spindler et al. 10. Nevertheless, to our best knowledge the “tangible views” have not been considered with stereoscopic projections. Move the Touches Paradigm With the second paradigm the touches are “moved” into the 3D space above or below the display surface by using the on-surface 2D positions of multiple touch points to calculate a 3D position of a distant cursor Benko and Feiner 07, Strothoff212 9. Multi-Touch for Stereoscopic Displays et al. 11. As with the touch precision, the approach shifts the problem into the interface design space by defining the stereo touch as distinct input modality. Examples of interface techniques based on this approach are the balloon selection metaphor Benko and Feiner 07, the triangle cursor Strothoff et al. 11, the fishnet metaphor Daiber et al. 12a, and many more Hachet et al. 11, Cohé et al. 11, Song et al. 12. The main drawback of these techniques is that 2D interaction on the surface of the display is either not supported or realized with a different set of techniques, which leads to frequent switching between different interaction modes. Nevertheless, interaction techniques based on this paradigm could be both faster and more precise than in-air free hand gestures. Two examples of such techniques are presented in Sections 9.3 and 9.4. Perceptual Illusions Use of perceptual illusions to manipulate the properties of the rendered scene or parts of it in such a way that the user’s finger is redirected onto the display surface while reaching to touch a floating object, is the core idea of the last paradigm. The essential part of this approach is that such manipulations have to be imperceptible for the user, i.e., the visual effects of their application must remain below her perceptual detection threshold. Indeed, as shown by Dvorkin et al. Dvorkin et al. 07, there is only a (small) finite number of parametric functions for ballistic arm motions which are selected and parameterized according to the arm and object positions prior to the execution. Thus, if the user detects a change in the scene she would abort the entire gesture and “reprogram” a new gesture rather than adjust the current one. This usually takes more than 200ms Dvorkin et al. 07 and may thus significantly impair performance. Perceptual illusions for 3D touch interaction are discussed in detail in Sections 9.5 and 9.5.9. While the next chapters describe particular incarnations of the presented design paradigms we first concentrate on the effect of parallax shifts on the touch precision. 9.2 Touching Parallaxes In the monoscopic case the mapping between an on-surface touch point and the intended object point in the virtual scene is straightforward, but with stereoscopic scenes this mapping introduces problems. In particular, since there are different projections for each eye, the question arises where users touch the surface when they try to “touch” a stereoscopic object. In principle, the user may touch anywhere on the surface to select a stereoscopically displayed object. However, according to observations we have made, it appears most reasonable that users try to select a stereoscopically rendered object by touching: • the midpoint between the projections for both eyes – the so called middle eye projection9.2. Touching Parallaxes 213 Side Note 9.3: Eye Dominance As with handedness, eye dominance is formed in early childhood and manifests itself in the fact that humans preferably use one of the eyes when the task is reduced to 2D. Common examples are pointing, aiming before rolling a bowling ball, looking through a telescope, etc. Eye dominance can easily be determined with the so-called hole-in-the-card test (also known as the Dolman method). For this test the subject is given a card with a small hole in the middle, and is instructed to hold it with both hands and to view a distant object through the hole with both eyes open. The observer then slowly draws the card back to the head while keeping the object in view. While doing this he unintentionally converges to the dominant eye. • the projection for thedominant eye • the projection for thenon-dominant eye • the orthogonal projection of the object onto the touch surface – i.e., the object’s shadow A precise mapping between a touch and an object is important to ensure correct selections, in particular in a densely populated virtual scene, where a great number of objects are distributed over a relatively small space. In order to allow the user to select arbitrary objects, a certain area of the touch surface, which we refer to as on-surface target, must be assigned to each object. Therefore, it is important to know where the user will touch the surface for a given object. Recent research has indicated that neither of the targets listed above is fully correct, and users tend to touch between the projections for the two eyes with an offset toward the projection for the dominant eye. Indeed, as can be seen in Figure 9.4 users with left eye dominance and with right eye dominance tend to choose the same strategy to select a stereoscopic object on a two-dimensional touch surface. Interestingly the touch precision is also affected by the parallax. As illustrated in Figure 9.4 the touch points for negative and positive parallax are more scattered than the touch points on the zero parallax plane, although the size of the projected images for objects behind the screen are smaller than the size of the projections for objects on the surface. Furthermore, the touch points on the planes in front of and behind the display surface are comparably scattered, although the projected images for objects in front of the screen are greater than those of the object behind. This indicates that touching objects displayed with positive or negative stereoscopic parallax on a 2D surface induces more imprecision than touching objects with zero parallax. Furthermore, as one can see in the figure, imprecision increases with stereoscopic parallax, in particular for objects displayed with negative parallax.214 9. Multi-Touch for Stereoscopic Displays Figure 9.4: Typical touch results from a 3D touch precision experiment: (top left) shows the touch locations for strong negative parallax, (top right) for negative parallax, (bottom left) for condition zero, and (bottom right) for condition positive parallax. In addition, users tend to perform more slowly for objects with strong negative parallax. This is in particular due to the fact that for objects in front of the screen most users perform a “usual” point gesture until they reach the visual representation of the object and then move the finger slowly through it until it reaches the interactive surface. Thus the balance between ballistic and correction phases is different, leading to degradation in performance. In contrast, some users are “surprised by the surface” while performing some of the touch gestures in9.2. Touching Parallaxes 215 order to select objects behind the surface. This may lead to decreased performance times, but also degrades the precision of the touch gesture, since in these cases, the gesture ended prematurely, without users fully executing the slower and more precise correction phase. Furthermore, since the motion of a user’s arm during a touch gesture may differ very much among users and for different object positions, the prematurely ended gestures may lead to “random touches.” In order to quantify the discussed issues, we have conducted some experiments (e.g. Valkov et al. 11). Not surprisingly the results have confirmed that the middle eye and the dominant eye targets are best guesses for the location of the on-surface touch targets for stereoscopic objects, but the calculated mean distances to the actual touch points are still rather large. For instance, the mean distance between the dominant eye targets and the corresponding touch points was in all cases more than 2 cm. Furthermore, as discussed above, observations of the subjects during the experiment reveal that during most of the trials they neither touched the dominant nor the middle eye target, but rather a point “in-between” both touch targets, which raises the question of whether we can do better than this in predicting the actual on-surface touch point. Therefore, one can express the position P of the actual touch point, which IMD we call intermediate target point (IMD), as a linear blend between the positions of the dominant eye target P and the middle eye target P : DE ME P = P +a(P P ) IMD ME DE ME The parametera20; 1 determines the position of the point P according to the IMD segment (P P ). For instance, fora = 1 the IMD coincides with dominant DE ME eye target, whereas fora = 0 it coincides with the middle eye target. With this parameterization we can finally express the optimal position of the on-surface target point. Let P be the camera position,n its normalized right CAM vector and d the user’s inter-pupilar distance (usually hard-coded to 6:5 cm). The on-surface touch point P for an object at position P is then: TOUCH d P =(1 k)(P na )+ k P TOUCH CAM 2 with h CAM k= (h h) CAM where h and h are the depths of the object and the camera, i.e., their distance CAM to the display surface. The sign in the equation above captures the user’s eye dominance. In a right-handed coordinate system (e.g., OpenGL) the on-surface touch point has to be calculated with a positive camera offset for users with right- eye dominance and with negative camera offset for users with left-eye dominance. Apparently, the optimala may be influenced by several parameters such as the parallax, the user’s handedness, performance speeds and preferences, eye216 9. Multi-Touch for Stereoscopic Displays (a) (b) (c) Figure 9.5: Three typical tasks with the triangle cursor metaphor: (a) following a predefined path; (b, c) completing a small 3D model. dominance, etc. Nevertheless, values between 0:45 and 0:55 are a reasonable initial guess and could decrease the overall error rates significantly, especially if the user’s eye dominance is known. 9.3 Multi-Touch above the Tabletop In the previous sections we have discussed the main problems when interacting with stereoscopically rendered 3D objects and how the on-surface touch point depends on the object’s position when the user directly touches through the object to the display surface. Nevertheless, directly touching a floating object usually only works in a small vicinity (about5 cm) in front of and behind the display 1 surface. For objects further away above or below the interactive surface, indirect techniques are usually more appropriate. In this and in the following sections we present two techniques, specially designed for these scenarios. 9.3.1 Triangle Cursor Triangle cursor (Figure 9.5) is an indirect selection and manipulation metaphor for objects rendered stereoscopically above an interactive tabletop’s surface Strothoff et al. 11. The metaphor is designed to overcome the occlusion artifacts and fat finger problem Holz and Baudisch 10 usually accompanying similar interaction techniques Benko and Feiner 07, Coffey et al. 12c. In its basic implementation 4-DOF are supported, i.e., 3D position and yaw rotation. Since the technique is usually controlled only with the dominant hand it could easily be extended to support further actions. For instance, pitch and roll rotations could be mapped to a trackball metaphor controlled with the non-dominant hand. We came up with the idea for our triangle cursor technique when we examined how well existing 1 How to extend this range is discussed in more detail in Section Multi-Touch above the Tabletop 217 selection techniques for multi-touch surfaces would work combined with a display using a stereoscopic projection. Triangle cursor is a particular incarnation of the “move the touches” paradigm and thus uses an indirect approach to specify a 3D position above the tabletop display. Instead of specifying the 3D position directly (with, e.g., a free-hand technique) the user only interacts with the zero parallax plane, i.e., the table surface. This essentially splits the 3-DOF positioning task into a 2-DOF positioning task on the table surface and a 1-DOF task to select the desired height above the table. Even though triangle cursor uses this indirect approach, it allows the user to combine the position specification with the height specification into a single 3-DOF task . A spherical cursor is used to represent the currently selected 3D position. An isosceles triangle is displayed between two touch points and perpendicular to the table surface with the 3D cursor attached to the apex (Figure 9.5). When the user touches the surface at two points an isosceles triangle is dis- played with the two base vertices at the touch positions. The triangle’s altitude is displayed to provide additional visual feedback to the user. The altitude’s base point represents the 2D position on the surface and is located at the midpoint between the user’s fingers. The altitude’s length is equal to the height above the surface. When the user touches the surface at two points an isosceles triangle is displayed with the two base vertices at the touch positions. The triangle’s altitude is displayed to provide additional visual feedback to the user. The altitude’s base point represents the 2D position on the surface and is located at the midpoint between the user’s fingers. The altitude’s length is equal to the height above the surface. The use of the midpoint between the two fingers has two benefits for accurately specifying the position on the surface – first, the point of interest is not occluded by the user’s fingers and second, the movement speed of the midpoint can be reduced by a factor of up to 2 by moving a single finger Benko et al. 06. The triangle’s position can be controlled by moving the fingers on the surface (2-DOF). The height above the surface (1-DOF) is controlled by the distance between the two fingers and independent of their absolute positions. When the fingers are moved the triangle is scaled according to the distance between the fingers.This behavior resembles the usual scaling gesture that is used in many multi- touch applications, and the user can effectively scale the triangle and accordingly the height above the surface. It is possible to use triangle cursor with two hands or with two fingers of the same hand, in most cases the index finger and the thumb of the dominant hand. When using a single hand the height of the interaction space is limited by the amount the user’s fingers can be spread apart. As shown by Hancock et. al. Hancock et al. 07, similar to real tables rich interactions with digital tables can be implemented by limiting the interaction to a shallow area above the table. The use of a stereoscopic projection already limits the maximum height at which objects above the table can be visualized, depending on the user’s point of view. Initial tests have shown that mapping the distance between the fingers to the altitude of the triangle using a quadratic function allows users to218 9. Multi-Touch for Stereoscopic Displays cover the interaction space required by most applications. Close to the table surface the users have fine control over the height changes, while they are still able to access the upper boundary of the interaction space. To accommodate differences in hand size or applications that require a fine level of control in a deeper interaction space the metaphor could be extended to allow a change of the base-to-altitude mapping or adding a height offset while using it. Moscovich et al. Moscovich and Hughes 08a have shown that positioning and rotating the hand and adjusting the span of the fingers are compatible and can be combined into a uni-manual manipulation task. A yaw rotation around an axis perpendicular to the table surface can be applied using the relative position of the touch points and midpoint. A rotation of the fingers around the midpoint is mapped to a rotation around the axis defined by the triangle’s altitude. Thus the technique is particularly well suited for tasks which require fluent 3/4 DOF control, such as 3D path following or construction (cf. Figure 9.5). To select an object the spherical cursor has to intersect the object as illustrated in Figures 9.5a and 9.5b. When the user triggers the selection the object is attached to the spherical cursor and is moved and rotated with it until it is deselected. To trigger the selection a technique like SimPress clicking Benko et al. 06 or a simple tap with a finger of either the dominant or non-dominant hand can be used. Extension to 6 DOF With the possibility to control 3-DOF position and yaw orientation with only two fingers of a single hand, we explored an extension of the technique to simultane- ously control the other 2-DOF of the orientation. To control the pitch and roll rotation a trackball metaphor could be used. When the user touches the surface with the free hand a trackball is displayed at the touch point. The movement of the touch point is mapped to the rotation of the trackball and accordingly to the orientation of the selected object. The combination of triangle cursor with orientation and a trackball results in a bi-manual 6-DOF interaction technique (cf. Figure 9.6b). 9.3.2 Balloon Selection Balloon selection Benko and Feiner 07, illustrated in Figure 9.6a, offers a similar selection mechanism that decomposes the 3-DOF positioning task into a 2-DOF positioning and a 1-DOF height adjustment task. A spherical cursor (balloon) is displayed to show the currently selected 3D position. The user can specify the position on the surface with the first touch. Using a second touch point the user can change the cursor’s height by varying the distance between the two touch points. As the original version of balloon selection does not offer support for an additional specification of the orientation, one can expand the technique. If the user rotates the second touch point around the primary touch point9.3. Multi-Touch above the Tabletop 219 (a) Balloon Selection (b)6 DOF Triangle Cursor Figure 9.6: The two “move the touches” techniques: (a) balloon selection and (b) the extended version of the triangle cursor. the rotation is applied to the currently selected object. Nevertheless, this extension, although straightforward, may be suboptimal for the technique, since it may change the underlying mental model of the user. Furthermore, it also introduces some interference between setting the height of the balloon and changing its orientation. Indeed, since users are usually not able to move their finger in a perfect circle around a fixed point, changing the orientation will inevitably lead to altering the height of the balloon. 9.3.3 Triangle Cursor vs. Balloon Selection Our studies have indicated that most of the users have the subjective feeling of being faster with triangle cursor than with balloon selection, when performing a path-following or construction tasks, and have rated the positioning precision as equal. All users have appreciated the fact that triangle cursor provides a smooth single motion to control all degrees of freedom with a single hand, and they have rated it as more appropriate for the path-following task. Nevertheless, some users have described the balloon selection as having a simpler mental model, clearly separating surface positioning from changing the height. Indeed the subjective user comments have indicated that the extension of balloon selection to support yaw rotation leads to difficulties separating the yaw rotation from height changes. Nevertheless, none of our test users rated this as a problem, and most of them intuitively overcame this by first adjusting the object’s orientation and then its height. Quantitative evaluations (cf. Figure 9.7) have largely confirmed these subjective estimations for a synthetic docking task and a more real world-like construction task. As one can see in this figure users are able to perform the220 9. Multi-Touch for Stereoscopic Displays 40 2.0 200 4 10 0.6 8 30 1.5 150 3 6 0.4 20 1.0 100 2 4 0.2 10 .5 50 1 2 0 0.0 .0 0 0 0 Mean Mean Mean Mean Mean Mean Completion Position Orientation Completion Position Orientation Time s Error cm Error ° Time s Error cm Error ° (a) Docking Task (b) Construction Tesk BALLOON TRIANGLE Figure 9.7: Comparison of balloon selection and triangle cursor for: (a) docking task (moving a die) and (b) construction task (build a 9-piece puzzle). Error bars =  SE. synthetic die moving task about 20% faster with our triangle cursor than with the similar balloon selection technique. The increase in speed did not come at the expense of precision, as the positioning and orientation errors were either nearly identical for both techniques or slightly in favor of triangle cursor. In the more complex puzzle task, which more closely resembles what a real-world task could be like, we were able to achieve nearly identical results. Triangle cursor is again about 20% faster while maintaining a similar precision. As triangle cursor is more complex and multiple degrees of freedom are controlled at the same time using a single hand one could expect it to be less precise than balloon selection. The results show that this is not the case. On average, triangle cursor even outperformed balloon selection by a small margin. We believe that this is the result of using the midpoint of two touches that allow a more stable control of the cursor position on the surface—one of triangle cursor’s initial design ideas. One of the advantages of triangle cursor is the ability to directly approximate the desired height by spreading the two fingers the right amount apart before touching the surface. Most users were able to approximately guess the right height to perform a selection after they had completed several trials. This is one of the main reasons why triangle cursor outperformed balloon selection. After the initial guess only a small correction phase is required for triangle cursor, whereas a larger height change for the cursor of balloon selection was required. Furthermore, the users utilized the time while they were moving the selected object from the starting position to the target, i.e., the users adjusted the spread of their fingers so that the cursor approximately matched the target’s height at the end of the gesture. Thus, only a small additional adjustment of the cursorÃTs ¸ position and orientation was9.3. Multi-Touch above the Tabletop 221 necessary once the target position was reached. In contrast, with balloon selection most users had more difficulty in adjusting the height and orientation while moving, so that either a larger correction at the target position was necessary or the users performed the task with discrete phases for moving and adjusting the height and orientation, leading to higher times to complete the tasks. Although triangle cursor is an indirect interaction technique, it is usually not perceive as such. The combination of orientation, position on and height above the surface in a single-handed technique results in fluid motions. This is particularly observable during the more complex tasks. Nevertheless it should be noted that it is sometimes awkward to use triangle cursor for rotations with large angles. Indeed, the user has to stop during the rotation, reposition his hand, and select the object again to resume the rotation. We observed that users who seemed more comfortable with touch interaction quickly learned to plan ahead and oriented their hand to grab the object so that no or only one re-selection was necessary. Other users always grabbed the objects in the same way and were forced to re-select the object more often. While some users reported the re-selection step as slightly disturbing, their results show that they still performed faster than with balloon selection, which usually does not need the additional re-selection step. 9.3.4 Design Considerations In this section we discuss several design considerations for using triangle cursor within more complex interfaces and briefly consider possible variations suitable for different application requirements. Widget vs. Tool In an application where the user manipulates a single or just a few objects over a longer period of time or a continuing sequence of manipulations is necessary it might be beneficial to modify triangle cursor and use it as a manipulation widget. When the user removes his fingers from the surface while an object is selected, the widget remains visible. Thus it acts as a handle, and the user can instantly grab it to continue manipulating the object, without needing to re-select the object first. A separate gesture, like tapping on the widget, could be used to deselect the object and dismiss the widget, when it is no longer used. However, in a dense visualization environment, where a large number of selectable objects or two objects that are very close to each other are manipulated, the widgets might occlude other widgets or could be hit accidentally by the user while trying to perform a new selection. Supporting Multiple Users The techniques could easily be extended to support multiple users. Using an existing technique that is able to identify to which user each touch point belongs, multiple tools can be active at the same time – one for each pair of touches of the same user that are in close proximity.222 9. Multi-Touch for Stereoscopic Displays The fact that triangle cursor is typically operated using a single hand can aid in the assignment of touches to the users. For example, the touch detection for a tabletop based on the rear diffuse illumination principle can be extended to also detect the users’ hands above the surface. While this information does not provide a complete user identification, it is sufficient to decide which touches belong together, so that multiple users can use multiple triangle cursors at the same time. The widget variants described above could be especially useful for tasks that require multiple users to collaborate. For instance, a selected object could be moved and then passed on to another user. While the techniques could be extended to multiple users, special considera- tions to extend stereoscopic displays to multiple users have to be made. A possible solution using a combination of shuttering and polarization has been proposed by Fröhlich et. al. Fröhlich et al. 05. Conflicts with Other Touch Gestures While not a considerable problem for balloon selection, triangle cursor might conflict with other touch gestures, for instance the most common camera navigation gestures pan, pinch, and rotate. This is especially apparent for the pinch gesture that is commonly used to adjust the camera zoom or scale objects, as triangle cursor was designed to resemble the scale gesture for height changes. In applications where there is a clear separation between objects that can be manipulated and a static environment this could provide a context to decide whether the user wants to select an object or perform a camera navigation action. For instance, if the shadows of selectable objects are displayed below them they provide a visual cue for the user where to initiate a triangle cursor gesture. When two touches are registered by the system and the midpoint between the touches lies inside an object’s shadow, triangle cursor is initiated. When the user touches an empty spot, a camera navigation action can be performed. A good example of this is a geographic information system (GIS) application with a predefined landscape and movable building models. When the user touches next to a building triangle cursor is used, and when the user touches on the landscape a pinch gesture can be used to scale the landscape, respectively zoom in or out. Different Height Mappings In the setup used in our experiments we used a quadratic function to map the distance of the user’s fingers on the surface to the height of the cursor above the surface. While the use of a quadratic function to map the 1D finger distance to height is adequate for the most common application tasks, as it had the highest level of precision close to the surface and it was still possible to precisely reach the highest points necessary, there are, applications where there is another reference height. Imagine an application in which the user controls a group of aerial vehicles9.4. Interaction with Virtual Shadows 223 that are displayed above a map. It would make sense to place the map in the zero parallax plane and show the aerial vehicles actually above the surface. If all vehicles operate at a common flying altitude the range with the most precise height control should be around that altitude and the height mapping function should be adapted accordingly. In some applications there might be more than one reference height. In this case a switching mechanism to select different height mappings could be added to the application. Another possibility is to extend the interface by another gesture to change the height of an object while it is selected. A sliding gesture might be used to move the selected object up or down without changing the distance between the two fingers which define the height. Changing the distance of the fingers would then result in a manipulation relative to the new height. Nevertheless, one has to consider that a changing height mapping might cancel out the benefit of the triangle cursor technique, being able to guess the right finger placement to get close to the desired selection height. 9.4 Interaction with Virtual Shadows While the two techniques presented in the previous section map the touches directly to a 3D position and orientation above the tabletop, here we present a conceptually different approach suitable for interaction with objects stereoscopically rendered below the interactive surface, e.g., in the so-called fish tank virtual reality setup. Both techniques make use of virtual shadows to access the otherwise unreach- (a)The VoidShadows Technique (b)The Shadow HandTechnique Figure 9.8: Illustration of the two shadows interaction techniques: (a) the void shadows technique and (b) the shadow hand technique.224 9. Multi-Touch for Stereoscopic Displays able objects, but make use of an orthogonal mental concept. The void shadows technique, illustrated in Figure 9.8a, uses the object’s shadow as proxy for in- teraction on the display surface, while the shadow hand (Figure 9.8b) technique casts a shadow from the user’s hand into the scene that acts as a distant cursor for interaction. Void Shadows In the void shadows (cf. Figure 9.8a) interaction, each interactive object casts a shadow on the touch-enabled display Giesler et al. 14. An object’s geometry, orientation, and spatial location in the scene controls its shadow’s shape and size and helps the user to identify which object is represented. In order to interact with an object, the user manipulates the associated shadow. Therefore, void shadows uses the objects’ fake shadows as mental model, similar to the shadow metaphor of Herndon et al. Herndon et al. 92. Furthermore, the display surface is typically invisible in 3D stereo applications. However, void shadows visualizes the interac- tive shadows on the zero parallax plane, which is aligned to the display surface. Thus, the metaphor implicitly associates some semantical meaning with the inter- active surface and facilitates planning and execution of the touch gesture Valkov et al. 14. With this technique touch and multi-touch gestures performed on a virtual shadow are transferred to the object casting this shadow. For instance, if the user moves a void shadow on the surface with one finger, the object will be translated in the X/Y-plane, while the shadow always remains under the user’s finger. If she scales the shadow up or down with two fingers by a stretch or pinch gesture, the associated object will be lifted or lowered, respectively, in the Z-coordinate. Furthermore, rotating the shadow on the display surface with two fingers will rotate the object around the Z-axis. As with the triangle cursor and balloon selection techniques, void shadows separates the degrees-of-freedom between different modes in order to reduce unwanted operations Martinet et al. 10, Nacenta et al. 09. Obviously, users can move both fingers simultaneously, combining position, rotation, and height manipulation at the same time. The void shadows metaphor uses a non-realistic shadow rendering technique, i.e., the shadows are always projected orthogonally onto the touch surface. As Figure 9.9a shows, this projection is controlled by an imaginary plane above the touch surface, called the shadow plane, in which each object is represented by a single point. By construction, the shadow volume is generated by this point and the object’s silhouette. The void shadow is then the intersection of this shadow volume and the zero parallax plane on the touch surface. To simplify the understanding of the shadow–object link even further, the part of the shadow volume under the display surface is rendered with a semi-transparent light gray color. With decreasing distance to the display surface, objects generate a more obtuse angled shadow volume, which results in a larger shadow. In contrast, objects that are more

Advise: Why You Wasting Money in Costly SEO Tools, Use World's Best Free SEO Tool Ubersuggest.