Early last summer MIT was showing off their "Deep Shot" technology to the press and it was stated at the time that Google owned the rights to it. A new patent published by the US Patent and Trademark Office confirms that Google not only owns the new technology but also has one of the original inventors working on this project at Google Research. Basically the technology works like this: A user takes a picture of a website like Google Maps on their desktop with a smartphone which then enables the user to open the site in the same state on their phone. Not as a static photo, but as a fully interactive map like the one on their desktop. When you're an on-the-go person, this is an excellent feature to have on your smartphone. Google's patent also indicates that they're already working on applying this technology to video games. The bottom line is that with Google's marketing and technical muscle behind this technology – there's an excellent chance of this becoming a killer app. The caveat however, is that Google will have to work with major industry players like Microsoft and Apple to first make their Dee Shot technology an industry standard.
Google's Patent Background
Many people now own a number of different computers, from a desktop or laptop computer at work or home, to a tablet or slate for reviewing digital content, to a smartphone for mobile computing and communications. Computer users transition frequently between these devices, for example, leaving their personal computer in their office during lunch and carrying their smartphone with them.
At times, a computer user may want to move information from one device onto another device. For example, a computer user may employ the power and convenience of their desktop computer to type in the address of an upcoming travel destination and to get directions to the destination. To make the information mobile, the user can print the directions and carry them along on the trip. The user may also take a picture of the desktop computer's screen, and look at the image of the map captured in the picture as need be on their mobile device display. Alternatively, someone could email themselves a copy of the URL so that they could later find the map. But there could be a much better way of handling such situations.
Google's Proposed Solution
Google's patent FIG. 2A is a schematic flow diagram of a system and process for sharing information between computers.
Google's patent application describes an invention about systems and techniques that provide mechanisms for moving information conveniently between computing devices. The information may be the current state of an application on a first computer, so that the state can be duplicated automatically on a second computer.
In one such example, a user who wants to duplicate or approximate a part of a state of another computer may capture an image of the display of the other computer, such as by using a camera on their smartphone. For example, the user may have employed their desktop computer to generate driving directions from their own office to an office of a colleague, and may want a corresponding mapping application to open on their smartphone so that they could carry the directions with them during their drive to the colleague's office. The techniques described in the patent application may cause the captured image to be sent to each of the other active (logged in) devices that are assigned to the user, and those devices may compare the image to their respective current displays (because one of them will be displaying the original display that is in the captured image) in order to determine if they are the intended target of the information sharing request.
The device that determines it is the target that may then converse with the relevant application to obtain information about the state of the application, such as a Uniform Resource Identifier (URI) from a web browser or other forms of information that may be defined by an application programming interface (API) for the system. Such information may then be sent back to the smartphone, which may then cause a corresponding application to be made active and to match the state of the application on the computer. In one implementation, this process is known as deep shooting, or "Deep Shot," because it allows the initiating or initial device to obtain, not only the image, but also the underlying state data so that it can open an actual functioning application with which the user can interact with.
In other instances, image analysis can be used to identify the state of a device whose display has been captured in a digital image, such as by identifying text in an image using optical character recognition. For example, if a user captures an image of a map, the names of the towns that are displayed on the map may be identified, and those names may be used to identify the state of the application, such as by providing the names as search query terms for returning an area on the map. Subsequent steps may be taken to make a tighter match for the state of the application, such as by comparing a generated map to the image to make the zoom level proper. Image features in the image may also be compared with image features stored for a polarity of displays of computer applications.
Google Thinks this Could Work with Games
The patent discusses a few future twists to this technology like working with games. They titled scenario 5 "Continued Videogame Game Play," and state the following:
"The techniques discussed here may be used to provide continuous game play of a video game by a single player, such as through a long-running game campaign. When the player is at home, he can play on a full-featured gaming console. When he goes to work, he can pause the gameplay and take a picture of his television screen with a smartphone. The console may then receive the image, determine that it is the relevant recipient, and send game status information to the smartphone. The user may then pick up the game on the smartphone where he left off, though at lower resolution and perhaps with certain additional limits on the gameplay functionality. The user can perform the inverse process when he returns home."
Will it Work with Apple Devices Too?
It should be noted that in the patent application's "Experiment 2," we see that the invention originally worked on a Mac as well. That particular experiment worked with a 15" MacBook Pro in concert with an HTC Nexus One smartphone. Now that Google owns the patent, it's unknown whether they'll want to continue to work with Apple. On the other hand, they may have to if they want this technology to be an industry standard.
Jeffrey Nichols, a researcher at IBM's Almaden research center who specializes in mobile devices weighed in on this technology by telling MIT News that he found "it a really compelling use case" and that he really hoped that "companies like Microsoft would really consider adding it." Yet he pointed out that getting Deep Shot to work with desktop software from a lot of different manufacturers would require broad agreement on interoperability standards, which is hard to come by. "I see it being much more likely to happen with websites than with desktop applications," Nichols said. "On the other hand, to some extent, we're moving away from desktop applications and moving more and more to the Web, so it's not clear to me how important it is that we really bring all the native application developers into the fold."
Deep Shot May be Integrated into Google Assistant over Time
The patent states that it will eventually work with input other than the smartphone's camera. The patent states that interaction with a user may extend to "sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input." This could mean that Google will integrate their Deep Shot applications with Google Assistant in the future.
We began our report with Google's Patent Background scenario with someone trying to take a picture of their desktop with a smartphone camera but working with a static photo. The answer to that scenario is clearly presented in the video below.
Deep Shot: The Video
Patent Credits
Google is the assignee of patent 20120069199 which was filed in Q3 2011. The patent's inventors are noted as being Tsung-Hsiang Chang and Yang Li. Tsung-Hsiang Chang is pursuing a Ph.D. in Electrical Engineering and Computer Science at MIT. Yang Li is now working at Google in their Google Research Lab. The full MIT news article that we quoted from could be found here.
Notice: Patent Bolt presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. Revelations found in patent applications shouldn't be interpreted as rumor or fast-tracked according to rumor timetables. About Comments: Patent Bolt reserves the right to post, dismiss or edit comments.
Here are a Few Sites covering our Original Report: MacSurfer, Google Reader France, Google Reader US, Real Clear Technology,Twitter, Facebook, GottaBeMobile, ZDNet, WebProNews, and more.
