Motion Capture to UE4 Pipeline (Joker Project)

Lucio Salt

11/02/2022

I got to listen to my professor, and he clarified the importance of PreVis. I understood what he meant because I've always tended to do ReVis's due to the blueprints of the scene already being there. However; doing that would mean me being limited by those blueprints. Due to more creative liberties, I decided to make a PreVis for the Joker sequel called "Joker: Folie à Deux". This was based on rumors from IGN, and it may be in the style of a musical, so I'll be doing something similar to fit that theme.

The scene will comprise of a skyscraper setting at night. Joker stands at the top of the skyscraper, and he is slowly walking to the ledge giving the impression of suicidality. When he reaches the edge, he pauses and clutches his chest with both hands suddenly with strong emotion and weeps. He then extends his arms wide open and smiles to his life with that strong joker grin. Explosions along the city skyline in other buildings are set off at the same time. 

When the explosions are continuing, he gets off the ledge and starts winding up and dancing along the fiery landscape lighting in the background. He dances for some time and then goes to a natural pause in the music and dance. At the same time of the pause, a cone light moves to his position, and he resumes his dancing. Camera changes to the helicopter aid's view looking at the Joker with their light beaming on him. Person near the opening of the helicopter says, "we've found the sick sonovabitch, send reinforcements to the top of Gotham Tower." etc. 

Camera then changes to one across from the skyscraper to an adjacent one. Two goons are armed with rocket launchers, and one directs the other to take aim at that helicopter. He fires on his mark and the rocket flies. Camera changes back to the helicopter's POV and the operator says, "what's that coming toward us, shi......!" Screen goes black and static/explosion sounds from the helicopter when they crash. Joker's manic laugh is heard in the back. End scene.

Lucio Salt
Motion Capture Presentation




Lucio Salt
5/7/2020
CAGD 493

Final post for this project.

I definitely learned a lot about using the Unreal Engine to create better renders for the environments and animations I was making. I better utilized what I learned in a prior semester to create a pretty decent looking environment for the Joker scene. On top of all that, I learned how to blend-in the animations via Mixamo and Motion Builder as I was unfamiliar with the animation process using UE.





Using what I had on-hand, I used my time and learned how to mess with the editors of their respective models, meshes, animations, materials, etc. A bit bummed out though, I couldn't complete the animation scene showcasing my staircase as that took most of the time via trial/error.

Below are the 4k renders and the end results of the environment I planned on using.







My goals this semester were to better learn Motion Capture through Re-Visualizations and Pre-Visualizations through the Unreal Engine and MotionBuilder from Autodesk.

I planned for myself to follow the plan I had set up with the Re-Vis' of the Joker and God of War.

I didn't completely finish one of my goals this semester due to the lack of proper equipment such as access to the Motion Capture Room/Programs and the outbreak of the Coronavirus. Thus I stayed with working on environment creation of these scenes so that I can simply drag in the actors with their respective animations/models.

Below are the remaining motion-captured data files showing what is cleaned/uncleaned along with the actor. These were the last pieces that would've made it completed.







Again, everything I did, worked out perfectly fine thanks to the criticisms of my peers and professors. The only issue was the limitations at the time and other priorities. As for what I'd do differently, I'd take some more time learning about animations to export into Unreal or create in Unreal. The lessons I got from my professor helped out immensely utilizing MotionBuilder, but the movements look not smoothed/blended yet still the motions get the point across.

From the provided images, one can track the progress from the planning, to the blocking out and to the finale shown at the top. Its been fun experimenting with ideas in early developments to seeing the end product and to fantasize what could've been. But I won't scrap away this project and I'll finish it in due time.







Lucio Salt
5/7/2020
CAGD 493




For this update, I am done with the video itself. I was trying to understand the MotionBuilder workflow and gather the necessary assets which took a bit of my time. Prior to this, I had difficulties blending animations in Unreal, but thanks to the advice from my professor and classmates this helped immensely.

I'm satisfied with it, but the movements could be smoothed out in the last animation.



Here is the outside scene of the trailer I'm referencing, but the COVID staggered my plans so I focused on the inside scene. I cleaned up the outside atmosphere and light as I had filler lights scattered to fill the scene which looked unnatural. Here's a 4K render of the outside of  1165 Shakespeare Ave, Bronx, NY.









I added in a spotlight to act as the sun since the atmosphere function was being difficult as it never cast the shadows the way I was looking for. Thank you to my friend for telling me there's keyable visibility for animations, which saved me a huge hassle.

Lucio Salt
4/30/2020
CAGD 493

Good Evening,

There were a few hiccups this run-through. I was at first deciding to import from Megascans for the weeds in the street scene by bringing in multiples to test them out. However; doing so caused my laptop to freeze hard and I lost all progress in that session. So I went back in the file and only imported one of the Megascans for a weed and duplicated along where to test/was needed.

Now comes the tricky bit. I'm very fresh when it comes to animations, but even more so when it comes to animations in Unreal. So I started out with exporting my Joker Model from Maya and exported him out to Mixamo to attach a Swagger Walk and this turned out fantastic after slapping on some basic textures.





Now I tried to animated a gesture for a button-pushing which I cheated with the Mixamo exports. But when I rendered the scene I can see from the first shot, there were two actors. I tried to find a way to hide the actor until they were needed but couldn't find anything definite/simple. I thought to myself why not export out the first scene and cut together the shots after. I tried to do some blend animations, but the animations would not blend and I couldn't find out why.





Animations are a new skill for me in UE4 sadly.



Lucio Salt
4/23/2020
CAGD 493

Things did with my allotted time.

I've messed up using a marquee selection on my scene, but that destroyed my level sequencer, so I had to rebuild progress to get back to the current level of completion. The downside is that I cannot increase the size of the NY street from a one-lane to a two-lane. The plus side is that I don't have to add more assets.

From the image provided below, I've added assets of a new storefront, two cars, rubble, trash cans/trash bags, and trees. I had some issues importing the trees I was going to use initially but the OBJ file froze my UE and Maya. So I scrapped that idea and found a new one I think fits the place (poplar tree).









I created multiple materials and instances to slap onto the OBJs imported. I've also added a new actor for the tall lamppost as I thought I already did that but apparently never did. The actor includes two lights as there were some issues with the lighting effects.


Lucio Salt
4/16/2020
CAGD 493

Hey there, so for this update I tinkered with the lighting to match better as there were too many cool color tones in the hallway scene.



I've spent my time animating with the level sequencer the 2 of the 3 camera shots. The first shot focuses on going down the hallway, the second focusing on the elevator buttons, and the last one focusing on the opening and closing of the elevator door.





I've also spent time making a simple Joker model from an existing rigged model.

Everything's coming together now. I have to animate the character walking down the hallway, pushing the button, and entering then turning in the elevator.

One issue I'm having a hard time figuring out is to have camera 1 blur the edges of the shot while all in focus for the center.


Lucio Salt
4/9/2020
CAGD 493

Hello there! For this entry and update, I've done quite a bit in designing/texturing. For the elevator/hallway scene, I've changed up the walls, carpet, and doors. Looking into the reference, the camera shot solely has the Joker in the shot with the back/mid-ground being blurred out so detail to those components is not too important.




I've tinkered with the blueprint actor for my self created ceiling lights to not only have a strong emissive, but a light function to match the reference shot of being blue/gloomy. Then created a lightbox to prevent light bleeding.

I'm still trying to understand the camera functions so I can get the desired effect.








On the stairway scene, I've added more assets to it to match its reference. I added side railing and concrete slab to the stairs as well as imported some made assets from Maya that were not in UE4 to match, doesn't look perfect but its something. To combat this, I've been messing with Material Instances to tinker with the UV tile scalings to break up the repetition.





I also added in some light boxes around the backsides of the buildings as the rendered lights would bake to the scene, simply looking awful. I'm in the process of making the street in front of the buildings. My main mistake was constructing the scenes to close to each other so I have to figure out how to marquee select in the live view to correctly space out the scenes.







Lucio Salt
4/1/2020
CAGD 493

Hello there, for this update its been a bit hectic getting properly set up. Nonetheless, I was able to work on a few projects, but not as much as I would've liked too.

For the following shots, I was able to use a bit of TwinMotion materials added to my folder, but I'm tinkering with the Material Editors to have them work properly as I'm experimenting with the best fits.

I mainly worked on the first scene with the hallway. I have to clean up the presentation materials to match like the door frames do not match despite them not being visible in the shots as they are blurred out.

As you can see some textures look off like the one spot of carpet on stone and the bland lights. Those are going to be touched up in a timely manner.
 

I know it's not much but a majority of my time has been dedicated to game design projects were I'm creating a gun and texturing it.

Lucio Salt
3/13/2020
CAGD 493

Good day, for this week I dedicated a good amount of time to this project. In short, I've built out both buildings for my scene of Joker dancing and I have cleaned up all the motive data. So far, I need to export the data to MotionBuilder to slap on a rig to a model that I have to make look like the Joker.

As you can see from the shots provided, I created a roughed out scene of the dancing steps located at 1170 Shakespeare Avenue Bronx, NY. I only made the sides that were going to be in the scene and compensated for a few feet on the perpendicular sides in case the camera catches the planes.







For the buildings, you can see I've tried to add in a few objects to break apart the surface, but it still does look a bit plane, but I'll worry about the finer details after I have everything laid out.



I have the videos of the motion-captured data cleaned, but I've not composed them to be side by side to the Uncleaned / Cleaned Versions yet. But I'll provide a link to view these on Youtube for your viewing.

I've Broken up the dance into separate parts and will crop the data to reflect the scene. I've also tossed in two more in the last video to lighten up with some humor and dance to Missy Elliot's Work It song.




Lucio Salt
3/6/2020
CAGD 493

Good day, not really much to update on this project for the current week. I've been busy with other projects that seemed to take my time more than previously thought.

For this project, I've spent a majority of the time allotted to cleaning up the MoCap data. I've cleaned all the elevator scene and the hallway walk. I've started cleaning the dancing scenes, but still haven't had the time to clean it up.

I was able to build out one of the buildings in the dance scene. The image of one of them is provided below. It still looks a bit plain, but I'll be adding the final touches after I built out the whole scene I'm revising.



I couldn't find a good angle that faced the inside of the stairway, so I built out the front of the left building. In the meantime, feel free to check my 495 page as that shows what I've spent my time doing during this period.

Lucio Salt
2/28/2020
CAGD 493

The progress has still been the same from last week and there has been a bit to add on for progress. I've captured some new motions for the scene as some were in need of better capturing and better movement.

The only thing I recaptured was the outdoor scenes dancing movements as I've extended the dance captures.

More information and updates to be followed up on in the coming posts.

Lucio Salt
2/21/2020
CAGD 493

Hey there, for this update, I was assembling the scenes for the Joker's Dancing scene. I was informed that Unreal Engine had a modular building set for New York City types of buildings. I'm still sorting through the assets to make the scene as close as I can.

So far I'm still roughing the entirety of these.

Below are some of the pictures I've submitted. One is a simple block of the elevator. This image below is the outline of the hallway in the introduction from the provided clip. The camera shot is also included to make sure this is in the proper angle as the clip.





Below is the image of the stair scene, I have assets for the left and the right buildings as they are not similar as one is a beige brick color and the other is a red brick building. I've been sorting through the assets in the pack and picking what can fit in the scene to the closet possible with little to no repetition.





The clip itself did show a decent amount of detail but I needed a bit more detail to see what I'll be working towards. So, I searched around for the locations of both the shots. However; I have so far been able to locate the stair's location.

Address of the famed Joker Dance Stairs is 1170 Shakespeare Ave, Bronx NY.




Lucio Salt
2/6/2020
CAGD 493

Good day, for this week all I was able to do was to practice the shot-list movement due to a miscommunication in the scene. I was able to reconstruct the scene with a rough block out.

A little bit of recap, I'll be focusing on the MoCap to re-Vis the Joker Scene. Below this post is my actor/experience pop locker Josh to do the motions.



I'll be focusing on creating the environments for the entire scene as posted originally in the past posts. The shots below are of the hallway/elevator scene I've created in Maya.





Given I have the motions captured, I'll be reshooting the scene this weekend to get better data as the scene was not rehearsed at the time of recording due to technical difficulties with the embed links. I have all the needed props, and the environment needed to recreate everything, all of which are in the prior provided hyperlink.



Lucio Salt
2/6/2020
CAGD 493

Good day, this week has been a full one and I have been gathering assets for my portfolio pieces.

I've set aside looking for other scenes to potentially do a re-vis or a pre-vis.

I've decided to take it slow in this process to build up what I can do and develop the experience and workflow. I'll be doing a Re-Vis of the dancing on the stairs scene from the Joker to the song of Gary Glitter's Rock and Roll track. I've broken down the scenes to 8 movements.

My decision for why I chose this is to work on an easier project to finish it and to refresh my experience. After that Re-Vis, I'll take what I learn to start a new project of a Re-Vis of the first fight scene in the recent God of War where Kratos fought against Baldur.

(Timestamps for both scenes are in the previous post.)

I've spent quite some time locating assets for the Joker scene and compiling a movement list for both scenes to mocap.

Movement Sheet for Joker


Lucio Salt
1/30/2020
CAGD 493

Hello there! For this semester, I plan on doing a Re/Pre-Viz using motion capture data and piece together the data into a scene using the Unreal Engine.

There are some scenes I'm interested in doing a scene for.

There's, of course, the Joker dancing scene from the recent Joker Movie with Joaquin Phoenix.



The clip I'd use would be between the times of 0:01 to 0:50 when he hits the slow-mo.

Or a scene in the recent God of War game where Kratos first encounters Baldur. My reasoning for this scene is that the entire game is motion-captured. I've watched the behind the scene from the game's development and everything is done with the same actor in a pre-built environment with a camera rig attached to a person in real space following the MoCap actors.

The clip I'd be using would be between the times of 3:23 to 3:57.



I'm still looking to get approval for other scenes from such advisors before proceeding.