i'm here today at peacemaker filmworks who have worked on small projects you
might have heard of like deadpool 2 and the new ghostbusters
they're known for their crazy camera cars but today their space is being used
for possibly the craziest 3d capture setup in the world created by volumetric
camera systems we've got over 200 CPU
cores 100 terabytes of storage and if
you follow me in here 239
cameras capturing a staggering 920k
image 30 times a second which begs the
question who needs to capture that much detail oh
yeah it's neil freaking blomkamp the director of district 9 elysium and the
film we're looking at today demonic it dropped yesterday and the way they use
3d capture in it is just mind-bendingly cool
and terrifying and you know what's also terrifying this segue to our sponsor
thanks to origin pc for sponsoring this video origin pcs can be customized with
11th gen Intel core processors and NVIDIA GeForce rtx 30 series graphics
with max p design backed by a 24 7 support team check out origin pc at the
link below
this whole setup starts with the yi camera uh 4k action camera that's well
inexpensive volumetric camera systems or vcs would have obviously preferred to
have cinema cameras all around here but when you have to buy 200 of something
plus spares it gets expensive fast and
it's not like these are bad little cameras each one has the sony imx 377
sensor the same one as in the google pixel and when you have over 200 of them
pointed at a subject you can get quite a bit of detail out of it each camera has
custom firmware and an extra pin installed in the usb cable that allows
for sync across all of the cameras this allows for the clocks to be synced
between every single camera so if i hit play on the master camera right here
which is conveniently marked by not having a front plate
all 239 cameras start recording and then
click it again and they all turn off the special thing
about this rig though is how all of that is turned into 3d data this is
accomplished using feature detection so all the tape in the background creates
these sharp edges so ai can go in and figure out where each camera is compared
to all of the others in the scene also every two cameras so like these two
right here are used to create a stereo pair and using some complicated
trigonometry they're able to calculate the depth of anything in front of them
this is one of the keys to why the vcs camera system is so sick by using this
depth data in post they can just tell the software to discard everything
behind the subject removing the need for green screens and allowing for a
complete 360 capture of the subject
again using a bunch of complicated math and ai
video is actually a sponsor vcs and contributed a gv100 to help crunch those
huge equations they're able to create a point cloud of the subject with all of
the depth data on its own this is really cool it removes the need for a bunch of
post-processing time because the animators don't have to worry about
things like cloth or hair physics but
the huge advantage over motion capture is the ability to shoot full color
textures of the actors in full makeup and costume this does have its own set
of problems though the large five meter ring of cameras surrounding me is great
for filming larger action sequences but when you consider each one of these
cameras has a wide angle 155 degree lens
if you're standing even a meter or two away there won't be nearly as much
texture data which is where the high resolution array comes in this was
originally made using a trampoline that was cut in half and held together with
zip ties but they upgraded to this steel frame for shooting with neil anyway
sequences will be planned out so that the actor can do their larger movements
in the large array and then land in the high resolution array for parts that
require more detail obviously the camera that is just a foot away from my face is
going to be getting a lot more detail than the ones further out even that
though isn't quite enough detail for vcs like they are going up against cinema
cameras so they're upscaling all of the footage from 4k to 8k every single
camera they custom-made an algorithm by capturing 4k footage on these cameras
downscaled to 1080p and then used the 4k footage to train the ai for their best
chance of upscaling it to 8k and with all that in place
it still isn't quite the level of detail that is needed for a full blownsies
hollywood film unfortunately there isn't really any way
around the detail at this point but that isn't really a problem for demonic they
did a lot less post-processing than i expected and left in a lot of the
artifacts that happened naturally and it really adds to the horror of the film
it's super cool how neil and the crew even use stuff like the point cloud data
to act as a transition between real life and the simulation parts of the movie
as you've probably guessed by now though running over 200 cameras at once uses a
crapload of storage like this thing is creating
9500 gigapixels per second and 7 to 14
million polygons per frame which translates into about 10 terabytes of
data for every 20 minutes of shooting the first day of shooting on neil's
movie poor tobias and scott of vcs were here until 6 am grabbing the data with
the two computers that they had so clearly an upgrade was needed
surrounding the camera away we have 24 computers from oat studio each one is
equipped with a 10 core i7 69 50k nice
64 gigabytes of RAM and a gtx 1080 ti
they don't actually need the gpus but they certainly need the CPU RAM and io
to copy the data from each stand of cameras sorting all 200 plus video files was a
huge problem for vcs so they created their own custom program called
clippycopy named after everyone's favorite paperclip that is based off of
the robocopy API that automatically pulls and sorts the insane number of
files that are made every time that they press play once it gets to the post
processing stage they create a global alignment of all the cameras this is
basically looking at the features on the wall to figure out where each camera is
in 3d space after that they create a depth map of everything in the room
using the stereo pairs and from there it's very simple to create a point cloud
of the entire scene every point in the cloud will give you not just the
position in 3d space but also the RGB data of every point every three points
in the point cloud is then poisson mapped to create polygons and using
extra ai goodness using the camera data the textures are then improved again
after all of that we get what basically looks like the best video game character
model ever so like right here we have a capture of tobias's nan just chilling an
unreal engine and i'm able to just sort of float around her in oh geez unreal
engine using wasd
it's just so cool and she's just playing back in real time
in 3d it's crazy this processing obviously requires some
balling computers that were provided by main gear and hp so down here we
actually have two tiny backpack rigs rocking an rtx 2080.
with all of that the director has an incredible amount of flexibility in
posts so they're able to pick camera angles import the character into any
environment they want and relight the scene to their liking in order to
re-light the character though there can't be any shadows which means they
need a lot of light
they've got a lot of light in our office we actually use these same re sky panels
we've got three of them and each one will run you about 20 grand so
naturally they're using 24 of them here but even that was not enough light while we
were here today they actually installed an additional 11 rem40s which
are an additional 40 grand each so unsurprisingly they rented the lights
for this shoot it's also insanely hot in here right now you probably can't tell
because my shirt's black ltd store.com but i'm drenched you don't necessarily
need a balling studio setup like this though to do volumetric capture since
they're effectively just action cameras on a stand vcs can record anywhere and
just you know footage goes locally on the cameras and they have batteries
built in this way they can set up and say like the jungle at a farm or at a
trade show and capture jensen wong and give us the data allowing us to have a
little 3d jensen wherever we want for an extra portable setup they created
this little box that has an image sensor in each one of these holes then you can
set up say five of them around the subject and boom you're good to go there
are additional applications outside of cinema 3d capture one of their biggest
funders is actually epic games who hope to use this technology to do things like
the travis scott concert but without all the time and money required to 3d model
the whole thing like he could just do a set surrounded by these cameras and
bippity bop he's straight into fortnite they're also looking into medical and
military applications so say there's a world-renowned surgeon who knows how to
do a really obscure knee surgery that surgery can then be recorded in 3d and
played back from any angle for training purposes future improvements are planned
for both hardware and software so hopefully one day cinemagrade 3d models
can be captured volumetrically obviously getting better higher
resolution cameras would allow for a better result but as more ground truth
data is captured and imported they can train ai algorithms to better interpret
the footage so hopefully you'll be able to get a similar level of fidelity to
these 200 cameras with just 70 or so so
huge shout out to tobias and scott for showing us their awesome tech and neil
blomkamp and chris harvey for sharing their incredible expertise demonic is
available now so check it out at the link below and as you'd expect from one
of neil's films it's a fantastic technological showcase and proper
frightening and also huge thanks to segways for bringing us to our sponsor
ting ting mobile has new rates that make it easier to see how much you can save
by switching you can get unlimited talk and text for ten dollars data plans to
start at 15 and their new set 12 plan with 12 gigs of data for 35 dollars and
unlimited data for 45 dollars a month if you like their previous pay as you use
plans it's still there with tang mobile's flex plans starting at just
five dollars per Gigabyte data can also be shared if you have a family plan
connect more phones and save more you'll still get national and award-winning
coverage and pretty much any phone will work with ting mobile t-mobile now does
have the perfect plan for everyone no matter what your needs are so check them
out at Linus.ting.com and receive a 25 credit
if you're looking for something else to watch while it isn't really similar to
this video but the porsche tycan review was loads of fun to shoot so go watch
that