Virtual Worlds / Games With Haptics Utilizing Microsoft Robotic Studio Toolbox
The framework of this research problem is the synthesis between a human operator physically exchanging mechanical energy through force feedback with a robot using it as haptic device while interacting with virtual objects in a virtual reality (VR) environment. This scenario offers a fruitful ground for scientific enquiries that may focus on different aspects and levels of human machine interfaces.
The ultimate goal of this research effort was to explore the synthesis between Microsoft Robotic Studio (MSRS) and two wearable upper limb exoskeletons that are utilized as haptic devices. An application in MSRS was developed. The application is easily configurable in terms of the VR objects geometry and physical characteristics. The application was used for haptic exploration of an immersed, engaging, and interactive VR environment with two exoskeletons.
While using the exoskeleton as a haptic device, the operator wears the two exoskeleton arms as well as a VR goggles that immerses him or her into an interactive and hapticaly enabled VR environment. Due to an implemented of gravity compensation algorithm the two exoskeleton arms are “weightless” as the subject moves them in space. Once any part of the arm comes into contact with a virtual object in the VR environment the VR presents this interaction visually while the exoskeleton arm utilizing its actuators generates the appropriate force feedback on selected human arm joints given the material properties of the virtual objects and the physics of the simulated interaction. For example if a virtual static ball or a wall is represented in the VR environment the exoskeleton stops the user at the surface of the object allowing him or her to explore the object freely but not to penetrate it. If the object has a certain compliance the the VR environment renders a force that is be applied by the exoskeleton on the subject, based on the amount of the penetration into the object and in that way simulate the object’s compliance.
Figure - A subject using a two arms exoskeleton while wearing a VR head-mounted display creating an immersed environment
The system is controlled by two PC (servo PC and VR PC) and a unique application that is developed with MSRS (Figure 2). The servo PC is responsible for maintaining low-level servo control. The VR PC runs the MSRS VR application and project a visual view of the virtual environment into the VR goggles. The VR PC maintains the current state of the VR environment and calculate via the inherent MSRS physics engine the force feedback that needs to be rendered by and applied by the exoskeleton arm. The UDP protocol is be used as the communication protocol between the servo PC and the VR PC. The servo PC acquires the joint positions of the exoskeleton which is physically coupled with the human arm as well as the interaction forces and torques between the operator and the exoskeleton device. Based on the required force feedback that is calculated by the VR PC, the servo PC provides a servo command to the actuators to generate the appropriate joint torques that simulate physical interaction with the virtual object. The current joint position is transmitted by the servo PC to the VR PC for keeping the VR environment up to date.
Figure - The Exoskeleton system architecture (only one out of two exoskeleton arms is shown for simplicity)
Device & Software
Upper-Limb Exoskeleton Prototype 3
EXO-UL7
The third upper-limb exoskeleton prototype composed of two arms with seven DOF each (3 shoulder joint; 1 elbow joint; 3 wrist) aimed to study fully functional multiple joint movements of the human arms.
| Status: Active Research |
Microsoft® Robotics Studio is a Windows-based environment for a variety of hardware platforms. The Microsoft Robotics Studio includes a lightweight REST-style, service-oriented runtime, a set of visual authoring and simulation tools, as well as tutorials and sample code to help you get started.
Microsoft Robotics Downloads
Virtual Worlds / Games With Haptics Utilizing Microsoft Robotic Studio Toolbox is a research that was funded Microsoft Research
Jacob Rosen (PI)
Multimedia
This clip demonstrates the potential application of the exoskeleton as a haptic device. A user utilizes the upper limb exoskeleton (EXO-UL7) to interact with a virtual ball . The interaction in the virtual environment generates a force between the virtual human figure and the virtual ball. This force is rendered, generated by the physical exoskeleton arm, and applied on the user arm. As a result the user feels the haptic interaction as if he would have felt by interacting with a real ball. The virtual environment was developed with Microsoft Robotic Studio Toolbox.
PUblications
(*) Note: Most of the Bionics Lab publications are available on-line in a PDF format.
Research