Late last year news of Microsoft's futuristic HoloDesk system began to surface and earlier in this quarter, Microsoft's patent pending invention finally came to light. While the project is definitely well ahead of the curve, aspects of it could make its way to market sooner rather than later via a Head Mounted System which we covered earlier this year. In all honesty, while Microsoft's HoloDesk is an ambitious project that may never get off the ground we must recognize the great strides that Microsoft has made via their Kinect and Surface projects over the years. They have clearly demonstrated that they're making incredible headway into ground breaking territory that will eventually spill over to the mainstream markets. Today's report presents you with just one of the many pieces in Microsoft's future computing user experience puzzle.
Microsoft's HoloDesk Patent Background
Modern computing hardware and software enables the creation of rich, realistic 3D virtual environments. Such 3D virtual environments are widely used for gaming, education/training, prototyping, and any other application where a realistic virtual representation of the real world is useful. To enhance the realism of these 3D virtual environments, physics simulations are used to control the behavior of virtual objects in a way that resembles how such objects would behave in the real world under the influence of Newtonian forces. This enables their behavior to be predictable and familiar to a user.
It is, however, difficult to enable a user to interact with these 3D virtual environments. Most interactions with 3D virtual environments happen via indirect input devices such as mice, keyboards or joysticks. Other, more direct input paradigms have been explored as means to manipulate virtual objects in such virtual environments. Among them is pen-based input control, and also input from vision-based multi-touch interactive surfaces. However, in such instances there is the mismatch of input and output. Pen-based and multi-touch input data is inherently 2D which makes many interactions with the 3D virtual environments difficult if not impossible. For example, the grasping of objects to lift them or to put objects into containers etc. cannot be readily performed using 2D inputs.
An improved form of 3D interaction is to track the pose and posture of the user's hand entirely in 3D and then insert a deformable 3D mesh representation of the users hand into the virtual environment. However, this technique is computationally very demanding, and inserting a mesh representation of the user's hand into the 3D virtual environment and updating it in real-time exceeds current computational limits. Furthermore, tracking of the user's hand using imaging techniques suffers from issues with occlusion (often self-occlusion) of the hand, due to limited visibility of large parts of the hand in certain postures, which leads to unreliable and unpredictable interaction results in the 3D virtual environment.
Microsoft's HoloDesk Solution
Microsoft's invention relates to a three-dimensional user interaction. A virtual environment having virtual objects and a virtual representation of a user's hand with digits formed from jointed portions is generated, a point on each digit of the user's hand is tracked, and the virtual representation's digits controlled to correspond to those of the user. An algorithm is used to calculate positions for the jointed portions, and the physical forces acting between the virtual representation and objects are simulated.
In another example, an interactive computer graphics system comprises a processor that generates the virtual environment, a display device that displays the virtual objects, and a camera that capture images of the user's hand. The processor uses the images to track the user's digits, computes the algorithm, and controls the display device to update the virtual objects on the display device by simulating the physical forces.
Microsoft's invention details a technique for enabling 3D interaction between a user and a 3D virtual environment in a manner that is computationally efficient, yet still allows for natural and realistic interaction. The user could use their hand in a natural way to interact with virtual objects by grasping, scooping, lifting, pushing, and pulling objects. This is much more intuitive than the use of a pen, mouse, or joystick. This is achieved by inserting a virtual model or representation of the user's hand into the virtual environment, which mirrors the actions of the user's real hand.
To reduce the computational complexity, only a small number of points on the user's real hand are tracked, and the behavior of the rest of the virtual model or representation are interpolated from this small number of tracked points using an inverse kinematics algorithm. A simulation of physical forces acting between the virtual hand representation and the virtual objects ensures rich, predictable, and realistic interaction.
Head Mounted Display
In other examples, the display device could take on a different form, such as head-mounted display (for use with either augmented or virtual reality), a projector, or as part of a dedicated augmented/virtual reality system shown in patent figure 8 below.
Video One: HoloDesk Augmented/Virtual Reality System
The following video presentation is supported by Microsoft's current patent application as you'll come to recognize.
The Heart of the HoloDesk System
Microsoft's patent FIG. 4 illustrates a 3D virtual environment; FIG. 8 illustrates an example augmented reality system using the 3D user interaction technique.
Microsoft's patent FIG. 8, illustrates an example augmented reality system in which 3D user interaction techniques can be utilized. FIG. 8 shows the user interacting with an augmented reality system which comprises of a display device (104) which is arranged to display the 3D virtual environment; it also comprises a user-interaction region (802), into which the user has placed his hand. The augmented reality system further comprises an optical beam-splitter (804). The optical beam-splitter reflects a portion of incident light, and also transmits (i.e. passes through) a portion of incident light. This enables the user when viewing the surface of the optical beam-splitter to see through the optical beam-splitter and also see a reflection on the optical beam-splitter concurrently. In one example, the optical beam-splitter could be in the form of a half-silvered mirror.
The optical beam-splitter is positioned in the augmented reality system so that, when viewed by the user it reflects light from the display device and transmits light from the user-interaction region. Therefore, the user looking at the surface of the optical beam-splitter could see the reflection of the 3D virtual environment displayed on the display device and also their hand in the user-interaction region at the same time.
View-controlling materials, such as privacy film, could be used on the display device to prevent the user from seeing the original image directly on-screen. Hence, the relative arrangement of the user-interaction region optical beam-splitter and the display device enables the user to simultaneously view both a reflection of a computer generated image (the virtual environment) from the display device and the hand located in the user-interaction region. Therefore, by controlling the graphics displayed in the reflected virtual environment, the user's view of their own hand in the user-interaction region could be augmented, thereby creating an augmented reality environment.
Using Transparent Displays
Microsoft notes that in other examples, different types of displays could be used. For example, a transparent OLED panel could be used, which could display the augmented reality environment, but is also transparent. Such an OLED panel enables the augmented reality system to be implemented without the use of an optical beam splitter.
The augmented reality system also comprises of a the camera (106) which captures images of the user's hand in the user interaction region so as to allow the tracking of the set of point locations, as described above. In order to further improve the spatial registration of the virtual environment with the user's hand, a further camera (806) could be used to track the face, head or eye position of the user. Using head or face tracking enables perspective correction to be performed, so that the graphics are accurately aligned with the real object. The camera shown in FIG. 8 is positioned between the display device and the optical beam-splitter. However, in other examples, the camera could be positioned anywhere where the user's face could be viewed, including within the user-interaction region so that the camera views the user through the optical beam-splitter.
Video Two: A HoloDesk Styled Workstation
Microsoft's patent application was originally filed in Q4 2010 and published by the US Patent and Trademark Office this month.
At the End of the Day
At the end of the day, a number of companies are trying to imagine the desktop of the future. Apple has shown us a number of visions (one, two and three), while others reach out to science fiction styled hand gesturing. Microsoft has shown us over the years that their Surface technology is taking shape and moving into many markets. And just recently, they've shown us how they're seriously eyeing the use of hand gesturing for even future tablets.
At the end of the day, I think it'll take years before the current experimentation taking place today in labs around the world even begins to slow down. Just yesterday we pointed to a new hand gesturing tool coming to both OS X and Windows called Leap in the next several months.
Yet so much experimentation is yet to follow in the coming years involving Siri like voice interfaces mixed with other advanced technologies such as hand-hovering, acoustics, head-tracking and much more.
Microsoft's vision of the HoloDesk presented in today's patent application may seem be a little funny to most of us today, but it'll evolve over time just as all technologies do. At the moment however, it seems to be a project that's a little too out-there for most of us to appreciate. Yet the research spin-offs described in the patent application could lead to advancing video glasses and/or head-mounted computer accessories for video gaming and/or wearable computers in the coming years.
The future of the user interface experience is far from being settled and with no end in sight. That'll keep the buzz and the search for the next great thing alive for many years to come.
Note to Referring Sites: We ask that referring sites limit the use of our graphics to a maximum of one per report. Thank you for your cooperation.
Notice
The Patent Bolt blog presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. About Comments: Patent Bolt reserves the right to post, dismiss or edit comments.
