IDEA

Future deluxe approached Inition to help construct an application that would be shown by Google at a Shanghai WAIC exhibition. The application was to demonstrate the ability of Googles AI and machine learning platforms to use a camera to track a users pose. This pose would then be read by the application and a Vase would morph to a specific shape specified by that pose. The vase would also be textured with an AI generated texture using the persons clothes as a starting point.

CHALLENGES

I would be working with a further agency based in Shanghai who were working on the AI part of this application to be built in python. The front end morphing vase vase part of the application would have to be built independently of the AI pose detection and texture generation to meet the tight deadline. These parts would then need to be continually merged as the project was developed by both the our London agency and the Shanghai agency. The final display would need to dynamically morph the vase and distort it as the users pose changed as well as tracking hand positions to distort it further. The processing power needed for the AI part of the application meant it would need to run on its own machine, so all pose data and tracking data would need to be sent over a network UDP port to a separate machine running the Unity front end application.

SOLUTION

I used Unity's latest HDRP rendering to generate the very high resolutions needed to make the vase as realistic as possible. 3D studio max was used to generate the vase shapes using morph targets that could be controlled in Unity. I integrated a third party plugin to distort the vase further and taper different parts as the user moved there hands. A bespoke shader was built in Unity to control the texture on the vases which were all built using substance designer. The final build also integrated a full configuration setup which allowed the installers in Shanghai to tailor the experience as required. It allowed them to change the timings of vase transitions, total interaction time ect. all of which was saved to a local configuration file.

TECHNOLOGIES USED

Unity, Substance designer, Python, 3D Studio Max

RESULT

The final exhibition was live for 3 days and is due to travel to other locations around the world.
A video of the development of the app and final installation can be viewed here.