I am currently the recipient of an ACNI grant. Under this grant, I have endeavoured to upskill and explore the possibilities of using Virtual Production techniques and technology in the field of interactive and immersive art.

One of the areas of my research has been focused on building small, scalable prototype artworks that coordinate audience position with camera position in a virtual environment. So, like the parallaxing you can get in VP, VR, or in a video game, but based on your positional data and on a screen that exists in a room, rather than a headset. Or, from a VP standpoint, instead of tracking a physical camera to adjust what is displayed on the LEDs, you would be tracking a person. More on the potentials of this in another post.

To begin with, I am building this artwork for an audience of one, with the goal to expand to a multi-user system.

I have learned a lot in terms of how virtual productions track cameras, how they configure Unreal Engine to handle that tracking, and what technologies and processes are used. I have also been comparing that pipeline and workflow to what would be necessary for an immersive installation.

There are many ways to accomplish what I am looking to do, and these differences in VP and immersive art and the way it is presented do make a difference. The VP workflow requires trained technicians during each session to setup, monitor, calibrate, and contribute to the experience. But, this workflow allows greater flexibility with what is happening on-screen. This approach may not work for all immersive arts experiences, however.

One major roadblock for digital artists getting their work into traditional arts spaces is the technology gap. The artist may not be able to provide technical support every day, but the employees of the gallery, exhibition space, etc. may not have the technical skills needed to support the work. Additionally, brining on a technician may be out of the budget.

It may be that the solution is actually situation dependent. If the artwork is almost more of a live performance, where you would want artist control and flexibility throughout, then using VP technologies such as nDisplay, can and even should be used. But if it is a still-flexible, but more controlled piece that will be in a space where various people may need to set it up each day, a packaged game may make more sense. I am still in the process of understanding where the differences will lie in these two approaches.

VP Options

I am not going to explain all the technology behind virtual production here, but there are some facets worth mentioning. Virtual Productions run environments through Unreal Engine, not through a packaged game made with Unreal Engine. This usually involves a computer with an open active editor session and another display running in “game mode.” This is made possible through technologies like nDisplay. While the game mode display looks polished and finished, the editor node is able to make changes to it in realtime – the huge advantage of VP. Tracking comes into Unreal, from one of a variety of devices, and often through the LiveLink plugin. This translates real-world positioning to virtual-world positioning – the core of what I am trying to accomplish.

Alternative Options

The VP setup requires powerful devices, sometimes many, and as I mentioned before, a technician to operate them. The benefits are huge, but not always realistic for a single artist. A packaged game may provide some similar benefits, but a divergence in the technology may be needed, and there would be some drawbacks. In a packaged game, there are a few ways that data could be fed through. With some configuring, it could still ingest data via UDP/LiveLink, but it would have to be fed to something other than an ICVFX Camera/nDisplay would not be used. There are also possibilities of using custom data handling solutions. In these scenarios, it is possible that the piece could be started up each day by opening two pieces of software, potentially run off of one machine. There is room for technical failures, but the process would be simpler. Any variation to the scene/artwork would have to be accounted for and programmed in ahead of time, though. But, it could still be manipulated in realtime. There is also the possibility of feeding the data in as a controller, the way you would with a joystick or mouse, but I need to look into this further.

Where I Am At

I have looked into a lot of options, and eventually just decided to jump in and start experimenting. I’ve attached a video here, where you can see a successful implementation of my face being tracked and it being mapped to the camera data in Unreal. I’ll go a bit further into this implementation, my takeaways from it, and where I think I’ll go next.

Because the end goal of this is to have an immersive experience, I did not want to implement solutions that would detract from the user’s immersion. So, while using technologies like a HTC Vive puck or an OptiTrack TrackIR would have pre-exisiting integrations with Unreal, they would require the user to hold something, or wear some sort of marker for the IR camera to pick up. I wanted to create something more seamless. I had a few ideas which included:

• Webcam utilizing computer vision
• Exisiting System like the Kinect V2
• Some sort of physical sensor mapped to keyboard inputs

In addition to wanting to preserve the immersive nature, I also wanted to find a system that would be low-cost and also replicatable for different scenarios and artists. This influenced my choices. Another key difference between immersive art and virtual production can often be the resources and funding.

I did initially explore the physical sensors mapped to keyboard inputs with the use of an Arduino Micro. While I am still interesed in pursuing this as an option, I became stuck on which type of sensor would make the most sense, and this option was also the hardest to reproduce. I also have started exploring the kinect, but have found that because it is no longer in production, adapters and the like can be difficult to source. The webcam option is definitely the most cost-efficient and accessible option of the three, but it does require a bit more upfront investment of time in order to get it working.

In the video you can see Unreal Engine on the left. On the right monitor you can see my face being tracked. This is software that I created myself. It is being tracked using OpenCV facial detection. The positioning data is then encoded as an OSC Message and sent to Unreal using a custom blueprint where I then map that data to positional data for a pawn actor. This should work with using two devices on a local network to split the processing, but could also work on one local devices. The data is being sent via UDP. This was a major step in my progress, but I have a lot of changes and takeaways that I am considering.

Open CV worked great, and I am definitely going to continue down this route, given the accessibility of it.

OSC was a great way to test out that this was a possibility, but I don’t know that I will continue with it. LiveLink generally uses other protocols or APIs such as FreeD or OpenXR. While these are a bit more inaccessible to learn on the backend, once I get that sorted, it will have broader capabilities. It should open up the possibility for both types of workflows mentioned before.

Operating via UDP I think is working ok for now. In any scenario it will require an additional piece of software, but even if I established the camera as a controller type, there would still be additional drivers needed that may interfere, etc. So this will work fine for now.

Moving forward, I am going to look into encoding the data in a different way, establish a firmer understanding of best practices for implementing that data in Unreal, and I will also be looking at what is possible with the Kinect V2, as an alternative. A first prototype should be ready to display at the XR Lab at Digital Arts Studios on April 25th.

If you have any suggestions on this work, or would like to provide consultation please contact me!