over the last couple of years one of my goals as a business owner has been to
deepen our pool of talent and intellectual properties so that if Linus
the guy gets hit by a bus Linus media
group the company can survive but i know
that for many of you lioness tech tips just wouldn't be the same without the
Linus which is why we formulated a plan
we posted this video on our sister channel ShortCircuit where i appeared
to be the host but in actual fact i had no involvement whatsoever
here's how my team made it and how monday.com who generously sponsored this
time-consuming but super fun and very educational journey made the process a
whole lot easier if you haven't heard of monday.com it's a productivity tool to
help you manage you and your team's work from start to finish
most deep fakes you see online are made by taking an existing video clip and
then using software to replace the face
and only the face unfortunately our team's task was a
little bit more complicated because we wanted our final video to not only look
like me but also sound like me Linus's
voice is proof that his balls are the only two things he hasn't dropped yet
this goal meant that we would have to write a script
hire a Linus impersonator to read the script on camera record and edit that
video and then deep fake it we used monday.com's recruitment management
template with a few tweaks to keep track of everything one small problem there
are no Linus impersonators well none that are usable anyway that we could
find and even if we did find someone who could do Linus's voice what are the odds
that that guy or gal also looks like Linus and i'm not
even talking about facial features although that helps no matter how
convincing the face swap the final video simply wouldn't work if the impersonator
was say built like a brick or had curly hair
okay then so we go the other way then get someone who looks like me and then
deep fake the face and voice right not quite
you see deep fake for voice or high fidelity voice conversion mostly exists
in research papers right now so we couldn't record like a look-alikes take
and then morph that audio clip to sound
like Linus instead we sent high-quality audio and
script reference pairs to a company called resemble ai who use them to train
a neural network to generate the voice from scratch
i give consent to have my voice cloned by resemble
meanwhile james wrote a fresh script for our eventual fake Linus actor to read a
keyboard review for the cleave from truly ergonomic the machine's first pass
was a little rough just when you think you've reviewed every ergonomic keyboard
on the planet a new company with a name like truly ergonomic launches something
like this the cleave keyboard apparently the world's most comfortable keyboard
but with some patience and a ton of help from resemble by the way shout out to
sakib muhammad you are awesome we were able to get several options for each
sentence arrange our selections in premiere along with some room tone et
voila an imperfect but definitely
recognizable voice clone just when you think you've reviewed every ergonomic
keyboard on the planet a new company with a name like truly ergonomic
launches something like this the cleave keyboard apparently the world's most
comfortable keyboard that is a pretty sexy sounding robot if i do say so
myself now for the visuals we made a casting
call on vancouver actor's guide looking for someone with my hair type face shape
skin tone and well stature and boy did
some people ever not read the ad but some people did and we ended up hiring a
lovely gentleman named dylan tebow lttstore.com i surrendered my favorite
clothes to help him get into character and well the kovit 19 pandemic had made
his hair a little longer than the picture but we decided to run with it
it's up here now that's way too far how often i use it now instead i have two
control keys right next to each other how is that helpful hey dylan can you
act more unreasonably upset about a small detail okay
losing so many keys that your productivity suffers and it's priced at
300 us dollars which cut dylan i don't want you to break any
company property but do you think you could drop it
oh my god armed with the footage from dylan's shoot we opted to use deep face
lab a free tool that is surprisingly easy to use you just supply a video of
your source so that's me and your destination so that's dylan and then use
the various batch files to split the videos into frames extract faces from
the frames train the ai model and then convert the final video
let's look at each of those steps in a bit more detail most deep fix are made
by taking a famous video that already exists and then swapping a new face onto
it but we're actually making a brand new video so that gives us a huge advantage
not only are we able to use a set that Linus has shot on before
but this one has a giant softbox that's always there that makes matching the
lighting between our new video and our existing Linus footage really easy but
just in case we also took brightness sweeps of dylan from multiple angles
which helped the ai model become more generalizable
we then grabbed the footage from Linus's ShortCircuit videos plus a couple LTT's
with different lighting and face angles and put them all on a new timeline in
premiere it's a good idea to remove any extra b-roll sponsor spots or non-Linus
faces at this point to save time later next we exported the video and dropped
it into our deep fake folder with the correct name so the batch files can
point to it the first batch file splits the video into frames which you could
have actually done in your editing software which also means less compression while the next one extracts
faces from those frames now for the all-important step of massaging our data
set for both size and quality monday.com's shareable boards helped us
work through all the things we needed you need at least a couple thousand
images for your source and your destination but the thing is you don't
want too much so somewhere between two and ten thousand pictures of each face
will do nicely depending on how much time you have and for quality you want
to remove any faces that are blurry are of other people or are not actually
faces at all it can be a bit tedious to go through so many images even if they
are someone so handsome but fortunately deepface Labs has some
built-in sorting tools to help it go faster with that done it's finally time
to train the model there are a number of knobs to adjust in the training step to
ensure that our hardware is being used to its maximum potential and a few that
have to be tweaked over the course of training so that our faces have detail
in the eyes teeth and skin tone for example it's basically a matter of
turning up the key parameters until you get an out of memory error and then
backing off to the point where you're confident it won't crash to get the best
possible results we needed to build a pretty badass deep fake rig we started
with the unlimited budget pc from a previous video it's equipped with a 9900
ks at 5.1 gigahertz along with 64 gigabytes of RAM but since video memory
is everything when it comes to defects we upgraded the gaming video cards to an
rtx quadro 8000 with a tier inducing 48
gigabytes of video memory and while we do have two of these cards we were
actually worried that sli would hurt our performance rather than help because
deep face lab really isn't optimized for it so i guess rip sli
fast forward a couple of days or weeks rather actually and our model previews
were starting to look pretty detailed that means it's time to merge
this can be done automatically or you can use the interactive converter to go
through your video frame by frame and tweak settings like super resolution the
blur of the mask edge or the face scale after that our settings were used to
output a final video and
just when you think you've reviewed every ergonomic keyboard on the planet a
new company with the name like truly ergonomic launches something like this
the cleave keyboard apparently the world's most comfortable keyboard
well the face swap is good but the lip sync
pretty far off the body language doesn't look very lineasy my hair is curly and
the voice sounds frankly just kind of tacked on at this
point i thought we were pretty screwed but fortunately while this video was
being made a new version of deep face lab came out that supports whole head
swaps meaning we no longer needed an actor with similar hair so i took it
upon myself to don the costco jeans slather deputy do in my hair to make it
as small and shiny as possible and talk
with exaggerated facial expressions ltdstore.com and we also use this
opportunity to collect some foley this is the ambient noises you'd expect to
hear when the host handles packaging for example the new model showed promise but
it had its own new issues number one see
how the hair derps out when he turns his head that's because our source data set
contained images from multiple LTT videos meaning multiple slightly
different gelled hairstyles that limits the machine to low detail approximations
of my hair to make it crispier we removed most of the images from our data
set keeping only stills from the original ShortCircuit video but that
didn't end up solving it the hair looks great when i'm facing forward but since
i actually never turned my head in that video we had no data for my right side
so i did some detective work and i learned that Linus shot that short
circuit video on november 29 2019 so i
looked through our internal email and chat history to try to find out if he
shot anything else that same day as luck would have it he did including multiple
takes where he turns the correct way over and over again issue number two you
can still see some remnants of james in there
handsome but not necessary to fix that we ran the
merge three times to create three different masks then brought those into
premiere and created a composite that completely removed james while
preserving the background and the new generated Linus
not new and improved just new and generated i see you there
and i see a result that i'm honestly
really proud of i can't believe what the team accomplished here like it's not
perfect james could have spent a little bit more time mastering my cat-like
grace but given that this was our first attempt
and we learned so much doing it not to mention had a ton of fun i am glad that
we completely blew our time budget for this project
because hey at least we had monday.com to help us keep that under control
monday.com made it way easier to keep track of tasks recruit actors and
delegate assignments across departments monday.com is built for the way you want
to use it with tons of customization you can import any data you need track
progress among team members and automate tasks so you can focus on getting work
done don't take my word for it though give it a try with a 30-day free trial
at monday.com when you sign up today at the link below which is monday.com it's
really easy when your service is the same as your url hyundai.com right it's
easy to find guys if you haven't seen it already the finished fully deep faked
video is up on our ShortCircuit channel so go have a look thanks for watching
is this really me you'll never know