Want To Learn A New Language? With This AR App, Just Point And Tap

There’s a new app gold rush. After Facebook and Apple both released augmented reality development kits in recent months, developers are demonstrating just what they can do with these new technologies. It’s a race to invent the future first.

To get a taste of how quickly and dramatically our smartphone apps are about to change, just take a look at this little demo by front end engineerFrances Ng, featured on Prosthetic Knowledge. Just by aiming her iPhone at various objects and tapping, she can both identify items like lamps and laptops, and translate their names to a number of different languages. Bye bye, multilingual dictionaries and Google translate. Hello, “what the heck is the Korean word for that?”

My first attempt on #ARKit and #CoreML using the pre-built model Inception v3! ARKit is awesome!!! pic.twitter.com/62d7vZcgAt

— Frances Ng (@franxyzxyz) August 14, 2017

One could imagine how quickly you could learn foreign words for the items around your house, with each object annotated much like it would be in a beginner’s foreign language textbook. But it’s also the sort of interface that’s perfect for the real-time panic that can happen abroad, when you find yourself completely unable to communicate something important.

On the technical end, the UX magic is possible through a combination of Apple’s ARKit and an existing database of about 1,000 learned objects that Ng ported into the app. While Ng tells Co.Design that her app is just a demo and she has no immediate plans to take it to market, what’s so remarkable is that while companies like Microsoft spent many years and dollars on mastering object recognition, Ng was able to build her app in a weekend, simply because it’s building off so much past work that’s now freely available and baked into platforms like Apple’s. “Personally, I find it useful as I’m learning Thai haha!” she writes over email. “It would be great way to learn language.”

Of course the coming wave of AR apps will usher in as much silliness as utility. (Remember all the old fart apps? Just wait until they incorporate cartoon gas clouds in AR!) Still, we’re really on the precipice of digital experiences that go beyond the app as it’s been defined since Apple launched the iPhone a decade ago. Now, the iPhone is capable of seeing the world and understanding it. And developers can finally apply all of the mind-bending research that got us here to create the next wave of irresistible apps, pieced together in a weekend.