Approach

I’ve been experimenting with mixed reality at-home listening experiences and have pulled together a few early prototypes.

The audio was pre-processed with Essentia to generate additional data streams that could be leveraged at runtime. Using the Meta Mixed Reality Toolkit, I anchored and scaled effects to existing room elements and applied additive lighting to the floor, ceiling, and walls. I had hoped to use the global mesh/depth texture for this, but the results were ultimately too coarse.

Audio credit: Kulakovka