This article describes a novel user interface concept called “haptable”. The idea concerns a multi-touch sensitive table top display that has been extended by a 3D haptic user interface.
The following video and image shows the result of a first feasibility study at Chair for Computer Aided Medical Procedures & Augmented Reality, TU Munich. A haptic device (1) of the company Sensable, the Phantom Omni, has been placed on top of a multi-touch display (2).
A virtual extension of the handheld part, the so-called stylus, of the haptic device is projected onto the display in addition to a virtual blue 3D cube (3). This virtual extension moves with the stylus and interacts with the cube. In case of collision of both objects, i.e. the extension of the stylus touches the surface of the cube, force feedback is generated by the haptic device. Also acoustic feedback and visual feedback can be generated upon collision using OpenAL and OpenGL features.
The haptic device can be moved on the display surface to be positioned at a suitable location for interaction with the virtual scene. This movement is tracked by the feed of the haptic device touching the touch-sensitive surface. Additional 2D graphics pop up as soon as the new location of the haptic device has been registered and accurate alignment of the virtual extension of the stylus with the real stylus is guaranteed.
A second version of the haptable includes a virtual mirror (4) that is attached to the haptic device. Such a mirror allows for additional, physically restricted views of the virtual objects being projected onto the table top display.
The combination of a haptic force feedback device with the touchscreen allows for a broad variety of applications. The first one, I had in mind, was surgical simulation to train cognitive, technical, as well as communicative skills of surgical teams. Such an interface would be capable of e.g. simulating many critical situations around and on top of the operating table to better train upcoming generations of surgical teams.
The system proposed by Gu Ye, Andrei State and Henry Fuchs in the article „A Practical Multi-viewer Tabletop Autostereoscopic Display“ allows for multiple perspectives on renderings.
Standard tabletop display with simple projection of virtual content are already commercially available e.g. from the companies Evoluce or Microsoft. The company Sectra uses this kind of user interface for rendering 3D medical imaging data.
A combination of haptable with these systems might result in a powerful 3D user interface having the following benefits:
- Intuitive human-computer interface using natural interaction: hand-guided tools, touch-based gestures, and mirror reflection.
- Supports team-oriented workflows at the “round table”.
- Imitates bench-based working environments using handheld 3D tools: Operating table, workbench, plotting boards …
- No need for extra interface hardware: haptic input device or table top multi-touch display can also be used independently as standalone devices.
Surgical simulation is only one potential application domain for haptable. Other interesting applications might be:
- Surgical Planning: Multimedia desk to support team-oriented planning/discussion for collaborative decision making, e.g. planning neurosurgeries.
- Design: E.g. Working environment for architecture/industrial design. Shape modeling tools and pencils are attached to the haptic input device (1) and the building, room, car, furniture, landscape are pre-sented on table top multi-touch display (2).
- Gaming: For instance, a billiard game could use the haptic input device (1) as the billiard cue and balls, holes and playing surface remain virtual.
The haptable apparatus has been filed as two patents by the Technical University of Munich. As agreed by the patent office, TU Munich, the concept is published here. Please find the two official technology offers (pdf) here:
- Human-Computer Interface Combining a Haptic Input Device and a Multi-Touch Table Top Display
- Human -Computer Interface Combining a Haptic Input Device, a Multi-Touch Table Top Display and a Virtual Mirror
Regarding the patents, please contact:
Forschungsförderung & Technologietransfer
Patent- und Lizenzbüro
z. Hd. von Frau Anita Widmann
If you are interested in implementing the user interface concept haptable for your own solutions, please contact:
I’m offering my consulting service to technically support you integrating the concept haptable into your application scenario. This includes implementation, integration into your software, planning the adaptation of the concept haptable to your application needs and hardware selection.