New AR Project from VFX Studio Known for Avatar is Made WIth Unity
Case Study for AR Horse By Weta Digital
The studio
Weta Digital, a studio founded in 1993 by Peter Jackson, Richard Taylor and Jamie Selkirk to create effects for their film Heavenly Creatures has gone on to create some of the most stunning effects in films including the Lord of the Rings trilogy, Avatar, and The Jungle Book. While the majority of the Weta team remains dedicated to delighting our eyes with stunning VFX in films, a section of their company is experimenting with augmented reality.
Through a collaboration with graphic technology provider Graphine and Unity, they’ve managed to bring a high-detail film-quality animated asset into mobile AR, demonstrating the incredible potential of AR.
The project
While the technology is improving on a consistent trajectory, mixed and augmented reality have some very interesting challenges to overcome. For end users to be able to set aside reality and buy into the fantasy, assets will need to be close to indistinguishable from reality.
Weta Digital has created some truly special assets throughout the years, but some of their most impressive are also among the least fantastical of their designs, in this case a horse ridden by an animated film character. Most people would understand that character isn’t real but many of those same people might not know that the horse underneath it has also been brought to life through artistry and CG.
Bringing film assets to mobile
You’d probably be forgiven for thinking that bringing a singular horse into AR wouldn’t be that complicated, though reality is quite the opposite. Evolving the a film grade asset into something that could be displayed on a mobile device without losing too much detail brings issues that mobile game creators generally don’t have to consider. There are two major components that needed to be adjusted: animation and texturing.
Realistic Animation
A horse’s motion is something filmmakers have been studying since 1878, when Eadweard Muybridge revealed a horse is completely aloft for a brief moment during a stride, using stop motion photography.
Weta’s solution to creating realistic animation is with their Tissue system, which won a technical oscar in 2013. The actual movement of the horse’s skin surface is driven by the structures within its body such as bones, muscles, skin, tendons, and fat. Tissue calculates how this anatomy would move in the real world – and drives the skin deformation based on that.
Big Textures for Small Devices
Film assets typically contain very large texture files which are not created to run on mobile devices.
In this case, the AR Horse consists of 58 separate 4096×4096 patches (textures) on each of three layers (diffuse, normal, and specular) of the anisotropic material, for a grand total of 174 patches for a singular model. To decrease download times, some of the textures were reduced from 4K to 2K. Additionally, some lower resolution textures were applied via a more standard UV layout to act as controls for anisotropy and ambient occlusion. Lastly, the horse also includes four wrinkle maps to create the write effect of skin and hair folding as the horse moves.
To achieve a high quality texture result, without resorting to a single ultra high res image, UDIM textures were utilized. Given the relatively low memory budget of mobile devices (128 MB GPU Memory), Graphine’s texture streaming technology was implemented to send the UDIM data to the application.
In the end through the mixture of innovative techniques and collaboration, Weta Digital’s AR Horse runs at a solid 30 fps on a first generation pixel phone.
Download
If you have a Google Pixel or Pixel 2, install ARCore Preview, then grab the app here and see it for yourself!
