How WayGo can help ARKit and ARCore developers

We spoke to Ryan Rogowski of WayGo to understand how machine learning and computer vision can help ARKit & ARCore developers.

What is your background?

It’s a long ways back, but I studied Electrical Engineering and Linguistics at the University of Illinois Urbana-Champaign. From there I moved to China and worked at an iPhone game development startup called ThinkNao. We built educational iPhone games for both the US and China markets.

My work in iPhone game development and background in Computer Vision/Linguistics led to starting Waygo. Waygo was initially just an augmented reality translator application helping travelers to Asia. Then after success with the App, we decided to evolve the technology into a platform for recognizing any language text in real-time to create Augmented Reality experiences.

How does Waygo help ARKit & ARCore developers?

Waygo has an AR-enabled text recognition SDK that allows ARKit / ARCore developers to recognize and interact with Text in the real world. We enable experiences where the phone can interact with product labels, signboards, books, menus, and more.

Imagine you are hovering your phone over a food menu and a 3D model of the food appears before your eyes. Or imagine panning your phone over various product labels and surfacing reviews and prices as you scan across the environment. Waygo’s text recognition can help developers achieve these types of experiences.

What work have you folks done at Waygo?

Waygo has built its own flagship Augmented Reality translation application, Waygo Translator. It is available in both the Apple App store and Google Play store now. Waygo has also partnered with a number of education companies and industrial AR companies to integrate text recognition into AR experiences.

Education companies have used Waygo technology to let students interact with language text instantly while in a real world experience. In one use case, children were able to take their paper language workbooks and see digital content overlaid after completing a lesson. This gave children incentive to finish the lesson and check their own answers to their work.

Waygo has also built experiences on Augmented Reality headsets such as the Microsoft Hololens. One application allows a user to look at foreign text, instantly recognize it, then see it pronounced and translated into his or her native tongue. Another tool Waygo built was for automatically recognizing industrial equipment labels to surface Augmented Reality style repair and maintenance instructions.

What's WayGo's tech and IP?

Waygo has 5 patents issued from the US Patent Office. The patented technology includes real-time language recognition of 16 languages. Waygo’s general patents cover a mobile foreign language translator that uses a video camera. Other patented technologies include producing real-time translations corrected over frames of video camera movement.

Waygo also has design patents on recognizing language text vertically and horizontally using smart phone cameras. This is unique to Asian languages that can be written horizontally, forwards or backwards, as well as vertically.

What can AR developers do better / do more of?

Augmented Reality presents an entirely new way to see and interact with the world. Given that most developers have spent years writing software for screen-based systems on laptops or smartphones, we all need to take a step back and re-imagine how digital information can come to life. Information no longer needs to be limited to the bounding box of a screen, but can now take it’s own unique form. I believe the most powerful Augmented Reality experiences will come from understanding this truth.

The second Holy Grail will be to create social experiences in this AR world. AR experiences will be novel and can create new ways of interpreting information, but it will be akin to surfing the Internet alone in 1999 until those experiences can be shared across users. There are already a number of companies working on this problem of meshing 3D spatial maps across devices for multi-user experiences, but ARKit and ARCore are a necessary first step in making this possible.

Why is OCR important to AR?

OCR is the gateway to allowing Augmented Reality experiences interact with written language. Just as an Augmented Reality device should be able to understand an object such as a chair or pedestrian, it should also understand written text.

Which platform is better, ARKit or ARCore?

This is a very longwinded questioned, but essentially both platforms perform on par with one another for all intensive purposes. Only highly experienced computer vision developers will notice the differences. The obvious advantages of Apple’s ARKit have to do with its longstanding belief in controlling both hardware and software. One core component in AR spatial mapping is to have highly reliable predictions of where the device is moving in space. Since Apple controls its hardware, it is able to tune the accelerometers and others sensors to have a high fidelity Inertial Measurement Unit. This can smoothly track where the device is moving in 3D space.

ARCore has advantages around its evolution from the Google Tango project. Another important component of spatial mapping is to actually create the map itself and remember it using some sparse point cloud. Google’s technology is at least a year ahead of Apple in terms of ability to create large point clouds, figure out where the device sits in this point cloud, and recover the device’s location within these large maps when it loses tracking.

Where do you see the future of AR heading?

Again a challenging question, but I think we are already witnessing the beginning of a long awaited revolution in technology. With the release of ARKit and ARCore, Augmented Reality will hit mainstream and create a basis for understanding that is unprecedented. You can think back to when Google Glass launched and there was a lot of buzz around Augmented Reality, but it was mainly within the technology community. Now the average smart phone users, which are basically 1/3 of the world, will be seeing for themselves what Augmented Reality can do.

Then as the world begins to adopt and understand Augmented Reality experiences, technology for Augmented Reality glasses will catch up and we will all be seeing AR experiences through our every day glasses.

Disclaimer: Sriram is an advisor and investor in WayGo