Freakfort: Art, Music, and Augmented Reality

There are two words I’d use to describe late March in Downtown Boise—energy and discovery. And there are two reasons for those two words: one, the long-awaited sunshine bringing Boiseans out of hibernation and two, Treefort Music Festival. Treefort is the annual convergence of music, art, food, and technology—a Pacific North-by-Northwest of sorts. Over a five-day period over 400 bands and thousands of indie music lovers descend on the City of Trees.

In the spirit of Treefort, we wanted to build something that would inspire curiosity and discovery, and something that would literally bring an iconic staple of Downtown Boise to life. That iconic staple is Freak Alley, an open-air multi-artist mural gallery spanning a full city block in the heart of Downtown. 

New Possibilities with Web-Based Augmented Reality

Augmented reality and spatial computing has been around for some time now, but until recently it’s been mostly limited to app-based experiences. This is great for things like social filters or spatial design tools, but the need to download an app presents a barrier to reaching wider audiences in-the-moment. Advances in web-based mobile technology have opened up new possibilities for creating quick and accessible AR experiences, no app required. Now, we can build for wider audiences and more ad hoc purposes, making AR available as a storytelling medium for more brands and organizations.

For this project, we explored three web-based AR platforms: AR.js, MindAR, and 8th Wall. These platforms all have a few things in common. First, they all give you the option to use the popular web VR framework A-Frame or the also-popular 3D framework Three.js to create the digital elements of the experience. They’re all web-based. And they all offer different options for how the AR functions, for example image tracking using a key visual, face tracking, and location-based tracking.

Ultimately, we chose 8th Wall with Three.js as our AR platform. We needed highly accurate tracking of complex images for our mural animation overlay effect to work, and through testing we discovered that 8th Wall seemed to be the strongest in that area. It also offers a solid cloud-based editing suite and emulator. Unlike AR.js and MindAR—which are both open source—8th Wall is a paid platform, which may be a barrier to consider as commercial licensing can get quite expensive. That said, the non-commercial pricing is reasonable and the platform provides a ton of added value in terms of guides and tools.

How it Works

The AR experience has three main components. First, the animation itself, which is simply an MP4 video cropped to match the surrounding environment. The animated video is positioned directly over a target image, allowing the AR to be as seamless as possible against the real-world mural background. Aside from being the most optimal way to serve an animation of this type, using videos also gave us the benefit of automatically serving the audio, synced up perfectly.

As the camera position moves, the position of the elements in 3D space are recalculated in relation to the target image, keeping everything locked in place. 

To create the animations, we captured high-resolution images of the artwork and broke the images out into layers. We then conceptualized rhythmic, looping animation effects that pay homage to the original artwork while syncing up to the beat of the featured song.

Alongside the animated video overlays in 3D space are the band and artist info. For the bands, we tie in Treefort using the iconic banners that hang on behind the stages.

About The Artwork

For this initial proof of concept, we started with three murals and three local bands. Big thanks to the artists and bands that let us use their work for this project.

Dana Wagner + Amoeba Arena’s “Anthill”

Animated by Gilberto Sauceda

Ashley Dreyfus + Sove the Second’s “Answer the Phonuuuh”

Animated by Preston Valles

Julie Pegan + Moon Owl’s Mages “Pressure”

Animated by Kelsey Rath

Try It For Yourself

The Freakfort AR Experience is free and open for anyone to use. If you’re not physically at Freak Alley, you can try it out by opening the website on your mobile device and pointing your camera at the mural images below:

Let’s Build Something!

Do you have a story you’d like to tell through augmented reality? If so, we’d love to chat with you about it.