New smartphone AR app lets users hold and place virtual objects

Researchers have developed a new software system that turns smartphones into augmented reality (AR) portals with which users can place virtual building blocks, furniture, and other objects in real-world backdrops using their hands as if they were really there.

The researchers from Brown University in the US said that the new software system, called Portal-ble, could be a tool for artists, designers, game developers, and others to experiment with AR. "We wanted to make something that made AR portable so that people could use anywhere without any bulky headsets. We also wanted people to be able to interact with the virtual world in a natural way using their hands," said Jeff Huang, an assistant professor of computer science at Brown University, who developed the system with his students.

Unlike existing AR apps like Pokemon-GO, where users play with virtual objects and characters by swiping on their smartphone screens, Huang said that the interaction with objects in Portal-ble was more "hands-on." "Swiping just wasn't a satisfying way of interacting. In the real world, we interact with objects with our hands. We turn doorknobs, pick things up and throw things. So we thought manipulating virtual objects by hand would be much more powerful than swiping," Huang said.

The researchers said that the app requires a small infrared sensor to be mounted on the back of smartphones which tracks the position of people's hands in relation to virtual objects. They added that this function enables users to pick objects up, turn them, stack them, or drop them. The application also lets people use their hands to virtually "paint" onto real-world backdrops, the researchers said. According to Huang, the main technical contribution to the app was developing the right accommodations and feedback tools that lets people interact intuitively with virtual objects.

"It turns out that picking up a virtual object is really hard if you try to apply real-world physics," Huang said. "People try to grab in the wrong place, or they put their fingers through the objects. So we had to observe how people tried to interact with these objects and then make our system able accommodate those tendencies." The researchers also added sensory feedback -- visual highlights on objects and phone vibrations -- to make the interactions easier.

The users, Huang said, feel the vibrations in the hand they're using to hold the phone, not in the hand that's actually grabbing for the virtual object. Still the vibration feedback helped users to be more successful in their interaction with the virtual objects, Huang said. The researchers plan to expand Portal-ble's object library, refining interactions and developing new activities. They also hope to streamline the system to make it run entirely on a phone. Currently the app requires an infrared sensor and an external compute stick for extra processing power. The team is slated to present the work at the ACM Symposium on User Interface Software and Technology (UIST 2019), held from October 20-23, in New Orleans in the US.