Imagine spilling a box full of Lego bricks over a table. Now—take a leap with me—don your imaginary augmented reality glasses. The camera in the AR glasses will immediately start cataloging all the different types of bricks in front of you, from different shapes to colors, offering up suggestions on models you can build with the pieces you have. But wait, someone is at the door. You go to check it and come back. Thankfully, your glasses don't need to rescan all of those pieces. The AR knows they're sitting on the table where you left them.
That ability to continuously remember real-life objects that have been scanned is the main pitch of a new AR software platform called Perceptus from Singulos Research. Perceptus can hold those objects in memory even if the camera is not directly looking at the scene anymore. As you walked over to answer the door, the Perceptus platform kept thinking about what else you could build with the pieces on the table. It didn't stop working just because you were no longer looking at the pieces.
“When we are in an AR space, we don't look at the whole room all at once, we only look at a part of it,” says Brad Quinton, Singulos Research's CEO. “As humans, we have no trouble with the idea that there are things that exist that we can't see at the moment because we saw them before and we remember them. Once you have AR that can understand what's around you, it can go off and proactively do things for you.”
At least, that's the idea. Perceptus acts as a layer above existing AR technologies like Apple's ARKit or Google's ARCore, which developers use today to create AR apps. But a lot needs to happen behind the scenes before this can work on your smartphone or tablet.
The app developer provides Singulos Research with 3D models of the Lego bricks—or any object—it wants Perceptus to detect. The platform then uses a type of machine learning process in which it studies all the different ways it can expect to see the object in the real world, with different lighting conditions, on various surfaces, and so on. Perceptus is then layered over the developer's app, allowing it to utilize this new object comprehension. It's the developer's job to make sure the app actually gives you things to do with the objects, like the way our imaginary Lego app might suggest stuff you can build using the bricks it identifies.
Object scanning and identification are still very much manual processes. To start, app developers who license the Perceptus platform will need to provide computer-aided design models of the objects they want it to memorize. But those CAD models will be added to Singulos' library, and future developers will be able to hunt through the digital stacks to more quickly find the objects they need. Soon, Quinton expects Perceptus to be able to identify a swath of common items—especially since there are already “large numbers of very accurate 3D models available" from video game makers.
Because the platform is trained to identify certain objects well before you launch an AR app that might utilize them, there's no need for image data to be sent to a cloud server for analysis. Perceptus runs locally on the device, and it can run just fine on existing mobile processors. Seeing it in action is impressive. Quinton moved an iPad closer to a table filled with Lego bricks, and I watched as the camera began identifying all the shapes and their colors in real time. It wasn't perfect—it missed a few pieces—but it was very close.
More impressive was the chess demo the company has built, which I used to virtually play chess against Quinton. He pointed the iPad's camera at a chessboard with only white pieces on it. As he moved a physical piece on his board, I saw the piece move on the illustrated board running in a browser tab on my computer screen. As I made a move, a virtual black piece (seen on the iPad's display) moved where I directed it on his board. It's awkward when it's viewed through the screen of an iPad, but it makes so much more sense when you envision this game playing out while you're wearing AR glasses.
That's the long-term goal for Perceptus, Quinton says, pointing out how the platform already works on Apple devices, mobile devices powered by Qualcomm's Snapdragon chips, and even Google's Tensor processor—chips with neural accelerators that will likely power the upcoming wave of augmented reality devices from these companies. It should easily translate to other AR hardware.
“The thing that I find the neatest about this is the interaction between virtual and physical worlds,” Quinton says. “We kind of have this metaverse-y thing that's not real—there aren't any [chess] pieces here, but we've created this new reality. It's not hard to imagine you could have a chessboard on your side and you could have this app. Then we've created an overlapped, physical reality that we're both in but doesn't actually exist anywhere.”
Matthew Turk, a computer vision researcher and the president of the Toyota Technological Institute at Chicago, says there's an advantage to this approach. You don't need to take a bunch of pictures of an object or have people find thousands of photos on the internet to feed into a machine learning algorithm. Turk says it's a nice solution for AR apps that require a physical component, but its applicability for general-purpose AR might be limited.
“You don't have a CAD model of every object you come in contact with,” Turk says. “If they're really kind of focusing on just things you have CAD models for, then that's a fairly limited set—even though that set can grow over time through libraries that people provide. In the long run, that's not enough for everybody, but it is enough for a lot of interesting applications.”
Working with 3D models in this way is a starting point, but we're still a few steps away from a world in which you just aim your AR glasses at something and they know exactly what they're looking at.
- 📩 The latest on tech, science, and more: Get our newsletters!
- How Telegram became the anti-Facebook
- Where to stream the 2022 Oscar nominees
- Health sites let ads track visitors without telling them
- The best Meta Quest 2 games to play right now
- It's not your fault you're a jerk on Twitter
- 👁️ Explore AI like never before with our new database
- ✨ Optimize your home life with our Gear team’s best picks, from robot vacuums to affordable mattresses to smart speakers
