Avengers: Endgame VFX Interview with Enrik Pavdeja, Compositing Supervisor – Framestore

Avengers: Endgame VFX Interview with Enrik Pavdeja, Compositing Supervisor – Framestore

March 25, 2020 – Today, Enrik Pavdeja, Compositing Supervisor, Framestore talks to VFX Online about working on Avengers: Endgame. Framestore, is a Bafta and Oscar-awarding winning creative studio based in London, Montreal, New York, Chicago, Los Angeles and Mumbai.

Avengers: Endgame​ is a 2019 American superhero film based on the Marvel Comics superhero team the Avengers, produced by Marvel Studios and distributed by Walt Disney Studios Motion Pictures. It is the sequel to 2012’s The Avengers, 2015’s Avengers: Age of Ultron, and 2018’s Avengers: Infinity War, and the twenty-second film in the Marvel Cinematic Universe (MCU). It was directed by Anthony and Joe Russo. It grossed nearly $2.8 billion worldwide, surpassing Infinity War’s entire theatrical run in just eleven days and breaking numerous box office records, including becoming the highest-grossing film of all time.

Enrik Pavdeja​, Compositing Supervisor at Framestore London. His credits include Avengers: Endgame, Spectre, Star Wars Episode VIII and Jurassic World: Fallen Kingdom, for which he was nominated for the VES award in Outstanding Composting in a Photoreal Feature.

// From Enrik Pavdeja, Compositing Supervisor, Framestore

How did you and Framestore get involved on this show?

Over the last decade we’ve worked closely with Marvel on many of their shows. It was a natural progression that Framestore would get involved in the biggest film of all time, the culmination of everything Avengers. We were involved from the pre production phase of the show running tests for Smart Hulk knowing that he would be the centerpiece of our VFX work.

How did you feel working in the MCU Films?

It was exciting to be involved in the climax of 10 years of film making from Marvel. It was going to be a challenge as these films always are, but everyone from the roto and tracking team to the supervisors and production couldn’t wait to get going on this show. We had a hard time selecting the team as it was everyone’s first choice film to work on once we got it through the door.

How was this collaboration with Directors Russo Brothers and VFX supervisor Dan DeLeeuw? What was their approaches and expectations about the visual effects?

On a show of this scale there are always many different teams operating all at the same time and therefore direct interaction with the Russo Brothers and Dan were at times limited. Regardless we had a great and open relationship with the client and our internal teams worked incredibly hard to answer all creative briefs in the most collaborative manner. We have an internal Framestore bar that we always strive for in terms of creativity and content delivery, which I’m sure surpassed all expectations, and we ended up with a body of work that both us and the clients are very proud to have on the big screen.

What are the sequences made by Framestore?

We had a varied body of work on Avengers – everything from extensive character animation, digital suit replacement, fully digital environments, quantum time travel FX and holograms – all delivered through complex, oftentimes invisible photoreal compositing, while staying true to filmed photography.

How did you work with the art department for Avengers: Endgame?

We work closely with the art department at the early stage of concept work for shots, general look and effects. We used the art department for initial suit designs, quantum time travel concept ideas, as well as environment paint overs. We tend to go back and forth with the art department to quickly asses and conclude creative challenges at the concept phase before other departments develop these concepts further.

How did you create the Quantum Effects?

The quantum time travel effects started life out in the van. This was the scene we see post credits in the Antman film, where Scott uses the van. Having to match the overall look, we started approaching this in our own way, jazzing up the overall effect along the way.

We replaced the housing of the quantum van with a newly modelled and rendered version. We then received a bunch of utility passes from FX and got to work in comp land. We rebalanced the utilities, added interactive lighting. Distortions. Aberrations. Light glows. Flares and optical effects to complete the look.

Similar to the quantum gate van, as a pivotal part of the story our heroes develop the quantum gate – which in many ways is just a bigger quantum van gate with some more fancy parts added on. We went through a variety of concepts. Our compositing team worked closely with FX to quickly turn around different looks using Nuke tools and a host of utility passes. We threw everything at it at this concept phase – auroras, plasma, distortions, energy, electricity etc…. In the end, the clients started to lean more towards the original quantum van gate look. They wanted it to feel like a progression of the same antman tech, but upgraded further by Bruce, Tony and Rocket.

For the time travel effect we start with the environment which is run through FX to create the cone stretching. This is then passed to lighting together with a wide range of utility passes. Once rendered these passes are then carefully balanced and treated in Nuke by the comp department to create the optical effect. Similar to the quantum van gate, comp applied interactive lighting, distortions, aberrations, light glows, flares and optical effects to achieve the final look.

Can you explain in detail about the design and creation of the Avengers suits?

We built very high detailed suits with a range of real life and imagined materials, and a design that appeared both practical and of course Avengers worthy cool. We had to develop a bespoke costume for each character, mainly because they have different proportions, but also, some are just different. Take Rocket and War machine for example – a tiny Racoon and War Machine – a hero already in a suit. Or the difference between male and female proportions between Cap and Widow. Or Smart hulk and everyone else.

The suits started out life as the hero’s original costumes as it wasn’t known during shooting what the quantum suits would look like. The process would begin with a very tight body track. As we were replacing the suits form the neck down, having a very tight neck track was paramount. A lot of this was frame by frame tracking.

As the suits didn’t line up exactly to the hero costumes, and because they moved slightly differently due to the clothing on set, our animators had to adjust the tracks and do an animation pass to get the suits to fit better and animated in a more natural way. We would then run a cloth sim with lighter areas of the suit that were stiffer, similar to the ant man suit and the darker regions a more flexible carbon fibre material.

Our paint and roto team would go in and digitally remove the plate costumes, and rebuild the heroes necks. This was more often than not necessary as some of the costumes have higher collars than our quantum suit. We also had a rendered neck from lighting to help comp with the plate neck integration.

Originally there were no helmets to the suits as the concept was to have a shimmery sheath as seen in Guardians 2. Eventually the clients were keen on the design of the Antman helmet, which was the film that had released prior to this one which in turn informed the general look of the helmet, similar in materials to the rest of the suit, inheriting the Antman look.

The helmet manifestation was based on the bleeding edge nano tech that we developed for the iron man suit in Infinity War. Our FX teams run simulations of the helmet manifesting from the collar of the suit. The CG suit with manifestation is then comped in with a 2d visor effect which we developed in Nuke through utility passes from FX.

What was the main challenge about New Tech Holograms?

We developed new tech holograms for Endgame. Clients wanted these to feel like familiar tech, but a progression of what’s been seen in the previous films. Holograms were a fully comp solution. Heroes were completely removed by paint, meticulously rebuilding the background. We had renders for Rocket and Captain Marvel’s suit which we integrated into the scans. The artists would then reveal back the characters through transparency in comp. We use 3d cylinders as the basis for the hologram screens. We then run the elements through numerous noise passes while emitting Nuke particles from the rooted edges of each character. We then add localised interactive light, scan lines, flickering, flaring, aberrations, and optical effects to achieve the final hologram look – a one stop shop Nuke solution for driving the look of these holograms. .

What was the main challenge about Smart Hulk?

A big part of our work centred around Smart Hulk. At Framestore we have the experience of doing many creatures in the past – and although we’d already done work on Hulk on Thor Ragnarock, Smart Hulk came with the need for us to innovate and develop new tech for our pipeline and push the envelope further.

While familiar in appearance, with Smart Hulk we now have a character that needs to be able to convey emotion – emotion that our audience can relate to, feel empathy for, and understand – and no differently to how Ruffolo would as a stand in actor – performing as Bruce Banner. He is now a lot more human and the clients wanted us to capture the essence of Mark Rufolo’s performance.

With that in mind we run a lot of Medusa test footage through our machine learning system – which is essentially a camera rig capturing the actor performing extreme complex expressions. Based on this footage; our AI learning results; and our keyframe animation tests – we started developing more animation blend shapes. We started with around 100 but in the end ended up with about 400.

Our very skilled animation team then run a pass of key framed animation to tighten up and match Smart Hulk closer to the Ruffolo scan reference. Through this process we noticed that the solve provided a good starting base for animation, and in many ways got us quite far with the more intricate secondary animation…such as muscle twitches and vibrations, skin sliding, and soft tissue micro movement. To get to the final performance, we still required a finessed layer of keyframe animation for the final delivery.

As we developed the character further, our animators inherited the role of actors, as we needed to push the performance of Rufolo to be more Hulk like, something the clients were keen to easily differentiate between.

In the end, Smart Hulk was incredibly detailed – with hairs, peach fuzz, pores on the skin and muscles under, wrinkles and eye/skin micro-motion. We were doing everything we could to get Hulk to appear as real as the tech will let us.

What was the main challenge about Rocket Character?

Rocket is a character that we designed and built for Guardians of the Galaxy. Although we’d originally designed and built him from the ground up, this being a new movie, we had to do it all over again as he was in a new costume.

In the scans we had plate reference from stand in – Sean Gunn which our paint & roto team had to painfully paint out in every shot. Thankfully we had a lot of clean plates shot as reference to help with this. We then had voice acting reference form Bradley Cooper, Our animation team will then use Sean Gunn and the voice acting reference from Bradley Cooper to deliver Rocket’s final key framed animation performance, with no help from AI this time.

Can you explain in detail about Creating CG Environments?

AVENGERS HANGER

This is the same hanger that we’ve seen many times before in the Avengers franchise, but during shooting there was too much equipment to clear out and re-dress the set, so they had to put up a lot of greenscreens.

Having to replace and fill in a lot of the hanger with CG, we had no option but to build a photo real CG hanger. Everything was based on carefully captured reference, and replicated through high resolution modelling, texturing and shading. We also built the environment outside to cater to various weather requirements over multiple sequences.

In a lot of the hanger sequences, the only thing we’re keeping from the original scans is the hero’s faces, as a lot of the BG, their suits, and oftentimes their whole heads are full CG replications.

ASGARD

A lot of our work in Asgard was Rocket animation, and Thor eye treatments. A part of it was also environment extensions of the palace interiors, and asgard exterior. We had one particular shot which was an establisher of Asgard. We reused the environment that we’d built for Ragnarock, but had to redo our a lot of our layout and general lookdev for the purpose of this shot alone.

This shot was driven mainly by our environment department and once the renders finally made their way through, it was quite a nice creative task for comp. All we had to do was make a full CG city that couldn’t possibly exist, appear familiar, photographic and full of live life. We achieved this by adding layers and layers of atmos, careful grading, additional elements such as ships, asgardian birds, mist and final optical treatments.

WAKANDA

Another one off environment that we built specifically for one shot, was Wakanda. It’s towards the end of the movie where the city is celebrating Thanos’ fall. We’d not seen Wakanda at night, and it the original scan it was just the Royal family on a balcony against a green screen. We built a fully digital Wakanda, threw in celebrating crowds, moving ships and a lot of

This was again a perfect marriage between environments and comp. We added layers of atmosphere, dressed in 2d elements and a lot of depth cueing. We run optical treatments on light sources, added flares and of course a lot of overall integration with the scan.

TOKYO

One last environment we worked on was Tokyo. Actually for this shot we did in fact have a plate, so the work consisted of mostly plate augmentation. The clients had filmed a fly over Tokyo, but again as with all these shots, there is a story point, and in this case it was that buildings in half the city were left abandoned after the snap. Between paint, roto and comp we turned off half the lights in the scan, removed half of moving cars, boats, people and just life in general. We then added CG clouds delivered to comp by the environment team, a lot of atmosphere and rain using our extensive 2D elements library, and finally a CG quintet. All very carefully graded and balanced to give the right mood and feel for the sequence.

What was the most challenging shot or sequence that you did and why?

The biggest challenge in some ways was the suit replacements. Throughout the hanger sequences we had a lot of suit integration that we needed to get right. These were almost fully cg shots a lot of the time. With so much environment work replacing the greenscreens and such tight integration between hero plate heads and fully CG quantum suits, it’s always a huge ask to get the integration so seamless that the viewers have no idea that 90% of the shot is fully digital.

What is your favorite shot or sequence?

I’m a big fan of photoreal environments so my favourite would have to be all the environment work we did. Whether it was the hanger or asgard, or wakanda or even the tokyo shot. These are often times seamlessly integrated into the shots, photoreal, even though a lot of the time they’re environments that couldn’t possibly exist, they appear invisible in the context of the film as the viewers are so engaged in the the creativity of our directors. Hidden VFX are always the hardest to achieve. If we’ve done it right, you’ll have no idea we did anything at all. The best of all the thankless tasks.

What type of softwares did you use for Avengers: Endgame?

We use a lot of Foundry, Autodesk, and Sidefx software integrated well within our pipeline. Our paint and Comp teams use Nuke for compositing while Maya is used by the animation team. Houdini is what we tend to use for FX and a range of other software is used for building assets. We also have an extensive R&D department that develops lots of proprietary tools to keep everything running like a well oiled machine.

How long did you work on the show, what was the overall shot count, and what was the size of the team?

We had a crew of 310 people that worked throughout the show, with around 210 working simultaneously at the height of the show. We delivered around 300 shots over a period of nine months.

What is your next project?

I’m currently one of the supervisors on “A Boy Called Christmas” – A film based on a famous children’s book, about the tale of our hero who journeys North and becomes Father Christmas.

We would like to thank Enrik Pavdeja for the great interview, and if you like to know more about him, check out his .

If you would like to know more about Framestore, go to www.framestore.com