We began our multipart series on Monday covering new capabilities coming to Microsoft's new "Surface" tablet with a report titled "Microsoft Invents New Cooperative Touch + Pen Input Mechanism." In that report we covered how Surface would take on new combination multitouch and pen gestures. In today's report we cover Microsoft's future tablet-gesturing. Users will soon be able to simply twist their devices a certain way to initiate a command or be able to send or share a document with a colleague in a very simple way without even having to tap devices. In this latest Microsoft patent they advance their new Interpretation and Behavior Selection Module (IBSM) with capabilities covering device input events for orientation, motion, imaging and more. While multitouch may have kick started the next phase of the digital revolution in terms of user gesturing, it only scratched the Surface of what's to come.
Microsoft's Patent Background
A developer may wish to provide an expressive contact-type input mechanism that accommodates a rich set of input gestures. However, increasing the number of gestures may introduce a number of challenges. For instance, assume that a computing device accommodates two or more predefined intentional gestures that are nonetheless similar. In this case, the user may intend to enter a particular gesture, but the computing device may mistakenly interpret that gesture as another, but similar, gesture. In another case, the user may seek to perform a task using the computing device that does not involve intended interaction with a contact-type input mechanism. Yet the user may handle the computing device in a manner which causes inadvertent contact with the contact-type input mechanism. Or the user may accidently brush or bump against the display surface of the contact-type input mechanism while inputting information, which causes accidental contact with the contact-type input mechanism. The contact-type input mechanism may incorrectly interpret these accidental contacts as legitimate input events. These problems may understandably frustrate the user if they become a frequent occurrence, or, even if uncommon, if they cause significant disruption in the task that the user is performing.
A developer can address some of these concerns by developing compound idiosyncratic gestures. However, this is not a fully satisfactory solution because the user may have difficulty remembering and executing these gestures, particularly when these gestures are complex and "unnatural." Furthermore, complex gestures often take longer for the user to articulate. For this reason, adding these types of gestures to a set of possible gestures yields diminishing returns.
Microsoft's Proposed Solution
A computing device is described in Microsoft's invention which receives one or more contact input events from one or more contact-type input mechanisms, such as a touch input mechanism and/or a pen input mechanism. The computing device also receives one or more movement input events from one or more movement-type input mechanisms, such as an accelerometer and/or gyro device.
The movement-type input mechanisms indicate the orientation or dynamic motion (or both) of the computing device during operation of the contact-type input mechanisms. (More specifically, as used herein, the term movement broadly encompasses either orientation of the computing device or the motion of the computing device, or both.) Based on these input events, the computing device can recognize gestures that incorporate movement of the computing device as an intentional and integral part of the gestures.
In any of the examples which follow, the computing device can present any type of feedback to the user which indicates that a gesture has been recognized and a corresponding behavior is about to be applied or is in the process of being applied. For example, the computing device can present any combination of a visual feedback indicator, an audible feedback indicator, a haptic (e.g., vibratory) feedback indicator, and so on. The use of non-visual feedback indicators may be useful in some cases because it may be difficult for the user to notice a visual indicator while moving the computing device. According to another general feature, the computing device 100 can present an undo command to allow a user to remove the effects of any unintended gesture.
Tilt a Device to Control a Zoom Function & More
Microsoft's patent FIG. 6 shown below illustrates one gesture that incorporates a rotational movement as shown in FIG. 4. In this case, assume that the user's intent is to perform some function with respect to object 602 which is displayed on a display surface of a computing device. As part of the gesture, the user touches the object (or icon) to perform a zooming action. The zooming action will be produced at a point directly above the thumb. In this scenario, the angle at which the user holds the computing device controls the rate at which the action is performed, e.g., the rate at which zooming occurs.
Simultaneously, or in interleaved fashion, the user can use his or her other hand to pan the content in a lateral direction, e.g., by touching a finger to the content and moving the finger in a desired direction. The embodiment of FIG. 6 accommodates this type of complex control because it leaves one hand of the user free to make panning-type gestures (or any other type of meaningful gesture or input command). If performed in an interleaved fashion, this example is also a demonstration of how one gesture (zooming by rotating the device) can be seamlessly integrated with another gesture (such as panning by moving a finger across the display surface).
In Microsoft's patent FIG. 7 above we see how tilt commands could involve scrolling. While the user is able to scroll through a document with traditional touch gesturing, the user could also just touch a scroll button and allow tilt to control the speed of the scroll. For instance, by tilting the portable device slightly backwards, the scroll will quicken. Tilting the device slightly forward will slow it down. .
In the case of scrolling through a massive document, the tilt function will have no equal. In this third scrolling control option, Microsoft uses the fishing term of "line casting" to describe the action. For example, the user can repeatedly snap the device downward to make successive large jumps through a long document (or alternatively, to make discrete jumps to the beginning or end of a document). In this mode of illustrative behavior, the device's IBSM (The Interpretation and Behavior Selection Module) can be configured to ignore the motion of the computing device when the user returns the computing device back to an initial position after completing a downward snap. The IBSM was first described in our June twenty-fifth report and is discussed later in our report with added functionality.
Sending Information to Targets: People or Otherwise
Microsoft's patent FIG. 10 shows a scenario in which the user holds their portable device and simply points it in the direction of a target entity. The user then may use their free hand to choose an icon, document, file, picture or other object to send to the target entity. The device's IBSM interprets this gesture as a request to send a copy of the object 1008 to the target entity, or to sync up the content and/or status of the object with the target entity, or to achieve any other application-specific objective vis-a-vis the object and the target entity. The IBSM 114 can determine the direction at which the portable device is pointed based on any type of movement-type input mechanism. The IBSM can determine the relative location of target entities in different ways. Microsoft also states that the shared target object may also correspond to a shared peripheral such as a printer, display and other.
Quick Bookmarking
Microsoft's patent FIG. 8 shows a scenario that incorporates the rotational movement. If a user is reading a document, book or long article and wants to return to the first page, a quick rotation of the user's device will do just that. Alternatively, or in addition, the computing device can use a visual, audio, and/or haptic feedback indicator, etc. that is dynamically invoked when the user places a finger on the display surface in a manner that is interpreted as a bookmark as shown at patent point # 808 of FIG. 8. The computing device can also provide any type of appropriate visual experience which conveys the page-flipping action to the user, e.g., by displaying a visual simulation of pages being flipped.
Microsoft's patent FIG. 9 shows the same basic page-flipping gesture explained above with respect to FIG. 8, but in this case in the context of an e-book reader 902. Here, the user may stick his or her thumb 904 on the lower left margin of a display surface of the computing device. This bookmarking operation marks an original page.
Future Microsoft Surface Devices will Recognize Multiple Types of Motion Gesturing
Microsoft's has many other twists in store for future Surface devices in terms of tablet-gesturing. In patent figure 11, Microsoft describes the user setting up the IBSM to recognize the difference between light and very hard touches on the display to mean different commands. The user will be able to adjust various touch commands to mean various things in varying applications. One example is a user applying a gentle touch to answer an incoming call and a harder tap to ignore it or another to mute the ringer.
In the case of patent figure 16, sensors in future Surface tablets will know when you've but the phone in your pocket by the lack of light and automatically shut down or go into sleep mode depending on how the user sets up the IBSM.
In respect to patent FIG. 13, the user can touch an object 1310 on the display surface of a computing device and then vigorously shake the computing device. The IBSM can interpret this action in any predetermined manner, such as a request to delete the designated object, undo the last action that was taken on the designated object, move the designated object to a particular folder, and shuffle the object so that it is displayed in the foreground, and so on.
Microsoft also briefly touches on image sensing input mechanism(s), such as a video capture input mechanism, a depth sensing input mechanism, a stereo image capture mechanism, and so on. The depth sensing input mechanism measures the distance of objects from some part of the computing device. For example, in one case, the depth sensing input mechanism measures the distance of objects from the display surface of the computing device. The depth sensing input mechanism can be implemented using any type of capture technology (e.g., time-of-flight technology) in conjunction with any type of electromagnetic radiation (e.g., visible spectrum radiation, infrared spectrum radiation, etc.).
Part Two: Overview of the Interpretation and Behavior Selection Module
Microsoft's patent FIG. 1 noted above provides us with an overview of an example computing device that takes into account movement of the computing device when analyzing contact input events. Microsoft's patent FIG. 2 shows another depiction of the Interpretation and Behavior Selection Module (IBSM) 114 introduced in FIG. 1. As shown there, the IBSM receives various input events. For example, the IBSM can receive touch input events, pen input events, orientation input events, motion input events and image sensing input events. In response to these events, the IBSM provides various output behaviors.
When contrasted to the IBSM presented in our June twenty-fifth report, you can see how this module has expanded from mere touch and pen inputs to now include orientation, motion and image sensing input events. The Interpretation and behavior selection module has vastly increased its interpretational behavior to now included intentional gestures, input rejection, state restoration, input refinement and configuration adjustments.
The advancements that are coming to future versions of Microsoft's Surface tablet family from handhelds to tables through to large wall displays are extensive. The Multitouch capabilities brought to such devices as the iPhone in 2007 changed the way we use tablets forever. Yet with dramatic new input methodologies and behavioral controls coming to the hardware itself, the tablet is going to take on a series of leap frog technologies that will open the market to all kinds of new form factors and exciting new applications.
Microsoft's patent application was originally filed in Q4 2010 and published by the US Patent and Trademark Office in Q2 2012.
Note to Referring Sites: We ask that referring sites limit the use of our graphics to a maximum of two per report. Thank you for your cooperation.
Notice
The Patent Bolt blog presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. About Comments: Patent Bolt reserves the right to post, dismiss or edit comments.
