Rokoko Smartgloves Interview with Jakob Balslev, Founder & CEO of Rokoko

Rokoko Smartgloves Interview with Jakob Balslev, Founder & CEO of Rokoko

September 26, 2020 – Today, Jakob Balslev talks to VFX Online about his experience of Rokoko Introduces ‘Smartgloves,’ the Only Studio-Quality Hand and Finger Tracking Solution for Under $1K and VFX/Animation Industry.

Award-winning motion capture creator Rokoko, introduces its Smartgloves, a hand and finger tracking solution that captures every nuance of a hand in the most affordable way possible. Durable enough for fight scenes, but precise enough for individual fingers, the Smartgloves can capture the full spectrum of an actor’s hand performance, giving VFX, VR, game dev and digital artists a faster way to create better characters, and one less thing to animate.

Headquartered in Copenhagen and with a core team located in San Francisco, Rokoko was founded with the goal of creating an intuitive and accessible mobile body interface, capable of enabling a physical presence in the virtual world. After debuting its flagship Smartsuit Pro, Rokoko continued working toward its promise of offering a complete body solution, including both facial capture and finger tracking, all of which can be processed and edited within the Rokoko Motion Studio. With the introduction of its Motion Library, a marketplace filled with ready-to-use content, Rokoko has become a hub for all motion capture users, as its pioneering research and development teams continue to create new techniques, technologies and systems that will benefit all motion capture users, regardless of the hardware they use.

// From Jakob Balslev, Founder & CEO of Rokoko

Tell us a bit about Rokoko.

Rokoko was founded in 2014 with the mission of making high-end motion capture and 3D character animation tools accessible to all creators. Being a film producer myself, I often found that I was unable to fully realize my ideas, purely because of the lack of access to tools. The hardware and software were all way too expensive and hard to use for someone without a huge budget and a big team.

As a company, we began by focusing on the hardware side. We created the Smartsuit Pro, an inertial motion capture suit that solved the three main issues of motion capture: the price of hardware; the technical expertise required; the need for a dedicated studio. The Smartsuit Pro is an affordable one-piece suit that is completely intuitive to use and set up, regardless of technical expertise. It just needs WiFi, meaning you can use it anywhere, and the price is $2,495 for a full system, a fraction of any comparable system on the market.

The next step for Rokoko was to create a finger and hand tracking device that solved the same issues as the Smartsuit Pro. That’s where the Smartgloves come in. Later this year, we will also release a new face capture solution, and with that, we will have completed the full performance capture hardware offering we set out to do. After that, it’s on to the software side of things!

What made you create Rokoko Smartgloves? How did it begin?

Creating the Smartloves was a bigger challenge than creating the Smartsuit. Finger tracking is extremely complex. There are so many nuanced movements to record in such a confined area, and the hands are often connecting with each other or an object, making the tracking even more unpredictable. On top of that, while body movements are natural and intuitive for most people (we all have a good sense of how a body looks when walking, running, etc.), hand and finger movements are very unpredictable and unique to each person.

It is very hard to describe or imagine how fingers will move in any given situation, yet there is such immense meaning and importance to how we use them. Watching finger movements is key in understanding peoples’ intentions. We gesture and articulate with fingers often without even realizing that we are doing it. Sometimes we even reveal things we were trying to hide (like being nervous, anxious etc.). That’s why it makes sense to capture the movements instead of trying to keyframe animate them.

However, capturing fingers with an optical system is nearly impossible. How will you film your fingers when they are in your pockets? How will you avoid losing details when one hand is blocking the other when they are crossing, or when hands are stacked on top of each other? So it had to be a sensor-based solution like the suit — which is, of course, what we initially wanted to do. The challenge here was not having an absolute position like you have with a camera-based system. The accuracy relies on the strength of the algorithms and different mechanisms keeping the data reliable over time. That’s what we’ve spent the last years perfecting, and we are at a point where we’ve really cracked the code. It just works.

The gloves are light and non-intrusive to wear (it even makes you feel cool), and with the fingertips free you can touch and feel the world around you. At the same time, you get a constant stream of data through the sensors, never missing a beat. There are no occlusion issues, and the hands and fingers are tracked at all times, even when your hands are in your pockets. And best of all, we managed to keep the price under $1,000 per pair, which essentially opens an entirely new space for a professional finger tracking product like this. When it comes to data and usability, no other product is comparable at that price.

We’ve also managed to remove one of the main obstacles with inertial systems – magnetic interference. Usually, inertial systems rely heavily on a magnetometer (compass) to steer the accuracy. The problem is that any metal (in walls, floors, etc.) or magnetic interference from things like loud speakers, for instance, will significantly affect the data. With fingers, that quickly gets problematic, because there are so many movements in a confined space, and any inaccuracies will quickly be very visible. We’ve removed the magnetometer completely, meaning that you can grab hold of a metal stick or rest your hand on a speaker, and still get the same accurate data.

Rokoko Smartgloves is a hand and finger tracking solution. Could you explain how it’s helpful for VFX, VR, game dev and digital artists?

Creators from all industries have never really had a good choice when it comes to finger and hand animation. Keyframe animation of fingers is extremely time-consuming and tricky, and generally not an option if you aren’t on a big production (with lots of time and resources). The motion tracking systems on the market have traditionally been either too expensive or too fragile, and the data quality was too poor to be usable. And regardless of price, they were all difficult and time consuming to use.

The Smartgloves are super quick to set up, and easy to use even without prior training. They can be used anywhere, and at less than $1,000 they are within reach of the budgets of most creators. This the dawn of a new era in finger and hand tracking, and a powerboost to all those creators that have struggled with this key aspect of their pipeline. They can put their time and focus into creating!

Share something about that Rokoko Smartsuit Pro?

We have shipped several thousand Smartsuit Pro suits to over 70 countries, and we are seeing our dream come true. When people that have never had access to motion capture before suddenly share their creations, and they credit our tools for making it possible, we can’t tell you how gratifying that is. In that sense, we have solved the issues we were facing when we started out, and made exactly the system that we ourselves used to wish was available back then.

While we will keep working to improve the Smartsuit Pro, it is now time to focus on improving the workflow around the suit. Along with our hardware, we are also focused on the handling and organizing of all the data our tools generate, along with things like retargeting, IK solving, livestreaming and keyframe adjustments — things we know that many creators are struggling with. Solving these issues will bring as much value as the hardware itself.

Share something about that Rokoko Studio?

Our core focus over the next few years will be on Rokoko Studio, which is available now. The Rokoko Studio started out with the sole purpose of visualizing the data you get from the Smartsuit Pro, but over time, we’ve added more and more animation features. We’ll soon be adding a range of software animation tools so that you can start doing almost all of your character animation work directly inside of Studio, eliminating the need for more expensive software tools. We will also be adding asset management tools to make Studio the backend for all your work with character animation files. You will be able to store and organize files from all your previous projects, cloud collaborate with your colleagues live and control your entire history.

Studio will be “input/output agnostic,” and open to any hardware device you choose to create your assets. It will also have plugins for any software platform you’d like to use your assets in, after you’ve worked with them in Studio.

How does Inertial Measurement Unit (IMU) technology work?

IMUs have a gyrometer, an accelerometer and a magnetometer (a compass). We use those three components to calculate your motions and stream them through WiFi from the hub of the suit and gloves, directly to your computer.

While super useful for getting an accurate capture, the magnetometer is sensitive to magnetic interference that comes mostly from metals. Therefore, we’ve decided to remove that component from the Smartgloves, and we found a way to get the full capture we wanted without it.

Share something about that Face Capture?

We currently have a face capture solution for iPhone X series devices based on Apple’s ARKit. We integrated it deeply into Rokoko Studio, so you can sync your data with your other devices, and record and export in FBX, BVH or CSV. You can also use our native plugins for Unreal, Unity, Maya, Blender and MotionBuilder to stream your data directly onto your custom character there.

Share something about the world’s largest marketplace, the Rokoko Motion Library?

With our suit, gloves and face capture devices, we’ve made it possible for any creator to create motion assets within minutes. But what if you’re looking for something that you can’t perform yourself? Although they have a great understanding of human motion and can often act out many of their shots themselves, animators are not actors. It’s one thing to envision a backflip or an extremely physical action, but it’s another to do it yourself without any training. This is part of what led us to create the Motion Library.

If you can’t create a specific motion asset yourself, you can search through thousands of files — all created by the world’s top mocap actors and studios — and find what you need. The Motion Library is a true marketplace, with the world’s leading mocap studios on one side creating assets, and users across all platforms buying those assets for $3 or $6 each.

It’s a way for the studios to monetize assets they have already created, or set up a shoot specifically for the Motion Library. You can access the Motion Library natively in Maya and Unity (with more plugins on the way), or through Rokoko Studio.

What are the pricing options for the Rokoko Smartgloves?

The Smartgloves are currently available for pre-order for $895. On October 21, the pre-order period will end and the price going forward will be $995.

You can also purchase the Smartgloves in a bundle, along with the Smartsuit Pro and a yearly Rokoko Studio Plus subscription for $3395 (saving $223)

What challenges did you face when creating Rokoko Smartgloves? Any fun stories from behind the scenes?

The biggest challenge was getting the accuracy we needed with only the gyro and the accelerometer, and then filtering the data in a way so the fingers would not collide with each other but still looked natural despite the constraints we applied.

As for fun stories from behind the scenes, just imagine testing the gloves by coming up with all the finger poses, hand signs and gestures you can think of to see if they work. You quickly end up in some awkwardly funny situations.

What does the future hold for Rokoko?

With the Smartgloves and the upcoming release of our face capture solution (more info soon), we’ve reached a point where we are very happy with our hardware offerings. We will, of course, continue to improve on our hardware, but in order to create the most value for our users, we will start focusing more on the software side of things — especially Rokoko Studio. That means adding new animation tools, motion editing tools, collaboration features, asset management tools and more. We will also allow all files to be imported — not just those created with our own hardware tools. Studio will soon become a vital tool for 3D character animation software.

We would like to thank Jakob Balslev for the great interview. If you would like to know more about Rokoko, go to www.rokoko.com