The paper craft artwork was shot using a Canon 5D Mark III recording raw files to a tethered Mac laptop. The style was meant to be analog and papercrafty, so no specialized stop motion animation software was used, just the Canon EOS Utility.
Production was fairly straightforward with the team shooting the planned scenes generally in the order they were going to be included in the final piece. Shoot, nudge, shoot, nudge, etc. No attempt was made at attempting lip sync with the audio recording, the mouths were just cycled through and would be augmented in post.
When I received the CR2 files from production my first step was to import them into Lightroom 5. This gave me a chance to find obviously-bogus frames that wouldn’t ever be used, then I exported everything out as full-size 5760×3840 JPEG files.
The JPGs were imported as a sequence into After Effects CC and interpreted as 10fps. Then the image sequence was placed into a 1920×1080 comp running at 23.976, the same size and frame rate we would finish in. These HD comps, as well as nearly-full-size 5760×3240 23.976 comps were rendered out to ProRes 422.
With the prep work of turning the raw frames into QuickTime movies completed, I proceeded to edit just like I had any other footage. Because After Effects would be integral later on and because it was flexible in dealing with multiple frame sizes, Premiere Pro CC was selected as the editing tool. I created subclips for different scenes and starting building the structure of my sequence. Some clips were sped up for action, some slightly slowed down. On set, shots were not meticulously timed out, so I had a lot of freedom to just make it work.
I also had the ability to choose when we would cut to a closer shot, and I had plenty of resolution as well. At first I just scaled up the 1920×1080 “wide shot” then later the full-frame 5760×3240 ProRes clips were edited in and scaled down. Overkill? Yes, and it caused problems later.
As the edit became more refined I also turned my attention to refining the mouth movements– at least somewhat. The mouth opening and closing randomly while the character spoke was fine, but during pauses of voice the moving mouth was distracting. In Premiere Pro I would composite a freeze frame of the “resting mouth”, which was his smile. I used the crop effect to make a rough box around the mouth so that any animation in the rest of the frame could still be seen. It was rough, indeed, but it helped me quickly place the mouth rests in the realtime NLE environment.
This is about the time the crashing started.
Over the course of the afternoon I crashed Premiere several times. With some help from Twitter I discovered my use of the very large 5760×3240 frames in Premiere was causing it to crash, and my cropping of one 5760 frame over another probably wasn’t helping.
I didn’t really need that many pixels, so I replaced the 5760×3240 renders with 4K versions and then I had no crashing problems at all.
After the timing was finalized in Premiere Pro I moved into After Effects to do for-real the adding of the mouth pauses. The freeze frame clips from Premiere helped me find where the pauses were to go, but I also ran the audio file containing the spoken answers through Prelude and along with a text transcript performed Speech to Text analysis which placed markers for each word that I was able to see in After Effects.
When the character’s mouth pauses, it is actually his whole head that I replaced. It was easier to mask the head and place it on top than to try to replace just the mouth. In a few shots there was some flicker in the moving main shot and so the frozen head would not completely match, so I used the GenArts Sapphire FlickerMatch plug-in to make the head have the same flicker as the rest of the shot.
When I was done I rendered a QuickTime movie and edited it into my sequence on top of the dynamically linked comp/clip. If a change was needed to the After Effects comp I could easily choose Edit Original to reopen the comp. Then when done in AE I would rerender the comp and Premiere would immediately see the new render file.
The last thing to do was deal with the flicker that I mentioned before. I recently sent my friend Pete Litwinowicz from RE:Vision Effects a timelapse shot I took at Machu Picchu in Peru, he wanted to test their DE:Flicker plug-in with it. Because of that interaction I had it in my head that maybe DE:Flicker could Deflicker my stop motion movie. With the default settings I wasn’t getting the result I was after so based on a recommendation from the excellent AE-List listserve I tried Digital Anarchy’s Flicker Free plug-in. With the defaults I was able to have the flickering gone immediately. However, I ended up using the DE:Noise plug-in from RE:Vision with some settings Pete provided, because it was a little smoother. It is worth noting, however, that DA’s Flicker Free got about 90% of the way there with just the defaults.
Here’s a screenshot of the final timeline in Premiere Pro.
It was a fun project and everyone involved was happy. I learned a lot too!