The world @ your fingertips

Tags: Films

An IIT alumnus and MIT researcher triggers excitement in the tech world by inventing a device to integrate web information with the real world

The world @ your fingertips
Remember Who Framed Roger Rabbit — the 1988 Steven Spieldberg produced Robert Zemeckis comedy that animated the world by combining animation and live action? It went on to become a hit, raising some $325 million at the box office, a fortune at that time Another kind of a merger of concepts is happening in the world of technology. It’s a marriage of digital information with the real world. Pranav Mistry, a PhD student and researcher at the MIT Media Lab in the US, has developed a wearable interface that makes interactive information from the internet available on the go, anywhere.

Part of Prof Patti Maes’ team at Media Lab, Mistry has developed the interface called SixthSense which created a big buzz at the Technology, Entertainment and Design (TED) conference at Long Beach, California in February this year. “The idea of how well we can connect the real tangible world with the intangible digital information world seamlessly has always been there with me for several years,” says Mistry. The MIT has taken out a patent on the invention.

The SixthSense prototype comprises a pocket projector, a mirror, a camera and a mobile computing device. The hardware components are coupled in a pendant like mobile wearable device. Both the projector and the camera are connected to the mobile computing device in the user’s pocket.

The projector shows information on surfaces, walls or any physical objects around us. The camera recognises and tracks a user’s hand gestures and the physical objects, using computer-vision based techniques. The software processes the video stream data captured by the camera and tracks the locations of the colored markers (called visual-tracking fiducials) at the tip of the user’s fingers. The movement and arrangement of these markers act as instructions for the application.

In simple words, the device helps you process information on the move, using the whole internet as your right-hand man, and the entire world as your multi-touch surface. “There is a lot of information on the internet, but humans do not have access to it always. SixthSense gives you the ability to receive information about anything and anyone you encounter anywhere, and at all times,” says Mistry. The SixthSense also supports multi-touch and multi-user interaction.

Mistry, who goes by the nickname of Zombie because of his inclination to work at night, grew up in Palanpur, a small town in Gujarat. He got his engineering degree from Gujarat University before going on to graduate from IIT Bombay.

But in practical terms, what can Mistry’s invention do for you? Well, let’s say you walk into a bookstore and pick up a book. Before you make a purchase, you may want to know what the reader ratings and reviews of the book are. All you need to do is to hold the book in front of you and SixthSense will glean the information from the internet and project it on to the book’s cover. You can also check its price on Amazon and find out if any

discounts are on offers from online stores.

On way to the airport you can hold up the boarding pass to the device and the message—delayed by 20 minutes, or on time—beams on to the boarding pass. You run into a guy at a party and the gizmo projects his name and Facebook keywords onto his T-shirt.

Hand gestures are key to the information world of SixthSense. You hold your fingers out at arm’s length forming a square, the way a cliché movie director does to frame a scene. The system snaps a photo of what’s enclosed by your fingers. Later, you can sort, resize and fiddle with these photos by projecting them onto any wall, expanding or reducing the image by just dragging it with your fingertips. Draw a magnifying glass symbol and you will be guided to the map application. Outlining the @ symbol lets you check your mail. The system also interacts with many physical objects. Visuals projected on a newspaper can show relevant videos from the internet.

Mistry says many companies have approached him for commercial adaptation of the device. “I am working with Samsung to integrate everything inside a mobile phone, including the projector, so that the phone itself becomes the sixth sense,” he said. Besides, he says, LG and Microsoft are in talks with him. But how much will a device like this cost? His prototype cost him $350. But being from a middle class family, he says he wants to keep the price low. It would never go beyond Rs 10,000, he emphasises.

Mistry, who plans to join the research division of Microsoft in Redmond for three months “to work on some ideas,” says he plans to return to India after completing his doctoral studies. “I know I can do a lot in India,” says Mistry. His professor at IIT Bombay and head of department at the institutes’s Industry Design Centre, Ravi Poovaiah says Mistry is a well-grounded person, and that’s why he’s planning to come back. “India is a goldmine for a designer. There are so many problems that you can solve here at any level—urban and rural,” he adds.

Even if he’s not able to find solutions to the problems, Mistry’s presence is likely to spark interest among the youth in the fascinating world of technology. No mean feat in itself.


  • Annual reports make sense only if accountable governance is in place

    It’s a sign of a lack of imagination to expect an annual report by a party in power to pull out some impressive performance given the complex nature


Stay informed on our latest news!


Amita Sharma

Political rhetoric makes for counter poetry

Poetic flourishes flavour politics. Ghalib and Hafez flowed profusely to ...

Zehra Naqvi

Watch your words, for they can kill

You must’ve heard the ph­rase ‘if looks could kill’. Ever ...

Dharmendra Khandal

Biodiversity day has come and gone. Yet again

Every year on May 22, world celebrates international biodiversity day. ...


William D. Green

Chairman & CEO, Accenture