WEBVTT

00:00:00.080 --> 00:00:05.839
i'm here today at peacemaker filmworks who have worked on small projects you

00:00:04.319 --> 00:00:09.840
might have heard of like deadpool 2 and the new ghostbusters

00:00:08.080 --> 00:00:14.320
they're known for their crazy camera cars but today their space is being used

00:00:12.080 --> 00:00:19.920
for possibly the craziest 3d capture setup in the world created by volumetric

00:00:16.640 --> 00:00:23.519
camera systems we've got over 200 CPU

00:00:19.920 --> 00:00:25.359
cores 100 terabytes of storage and if

00:00:23.519 --> 00:00:31.039
you follow me in here 239

00:00:27.439 --> 00:00:34.480
cameras capturing a staggering 920k

00:00:31.039 --> 00:00:35.520
image 30 times a second which begs the

00:00:34.480 --> 00:00:39.840
question who needs to capture that much detail oh

00:00:37.920 --> 00:00:44.399
yeah it's neil freaking blomkamp the director of district 9 elysium and the

00:00:42.239 --> 00:00:48.879
film we're looking at today demonic it dropped yesterday and the way they use

00:00:46.239 --> 00:00:52.399
3d capture in it is just mind-bendingly cool

00:00:49.840 --> 00:00:56.879
and terrifying and you know what's also terrifying this segue to our sponsor

00:00:54.879 --> 00:01:01.359
thanks to origin pc for sponsoring this video origin pcs can be customized with

00:00:59.199 --> 00:01:06.479
11th gen Intel core processors and NVIDIA GeForce rtx 30 series graphics

00:01:04.080 --> 00:01:11.880
with max p design backed by a 24 7 support team check out origin pc at the

00:01:08.640 --> 00:01:11.880
link below

00:01:18.560 --> 00:01:26.560
this whole setup starts with the yi camera uh 4k action camera that's well

00:01:24.000 --> 00:01:31.119
inexpensive volumetric camera systems or vcs would have obviously preferred to

00:01:28.560 --> 00:01:36.720
have cinema cameras all around here but when you have to buy 200 of something

00:01:33.200 --> 00:01:38.000
plus spares it gets expensive fast and

00:01:36.720 --> 00:01:43.200
it's not like these are bad little cameras each one has the sony imx 377

00:01:41.600 --> 00:01:48.000
sensor the same one as in the google pixel and when you have over 200 of them

00:01:45.759 --> 00:01:52.399
pointed at a subject you can get quite a bit of detail out of it each camera has

00:01:50.479 --> 00:01:56.960
custom firmware and an extra pin installed in the usb cable that allows

00:01:54.399 --> 00:02:01.119
for sync across all of the cameras this allows for the clocks to be synced

00:01:58.320 --> 00:02:06.000
between every single camera so if i hit play on the master camera right here

00:02:03.759 --> 00:02:08.360
which is conveniently marked by not having a front plate

00:02:07.759 --> 00:02:13.680
all 239 cameras start recording and then

00:02:11.360 --> 00:02:17.520
click it again and they all turn off the special thing

00:02:15.680 --> 00:02:21.680
about this rig though is how all of that is turned into 3d data this is

00:02:19.360 --> 00:02:26.640
accomplished using feature detection so all the tape in the background creates

00:02:23.760 --> 00:02:31.599
these sharp edges so ai can go in and figure out where each camera is compared

00:02:29.040 --> 00:02:36.400
to all of the others in the scene also every two cameras so like these two

00:02:33.920 --> 00:02:40.480
right here are used to create a stereo pair and using some complicated

00:02:38.640 --> 00:02:44.640
trigonometry they're able to calculate the depth of anything in front of them

00:02:42.720 --> 00:02:49.519
this is one of the keys to why the vcs camera system is so sick by using this

00:02:47.200 --> 00:02:53.599
depth data in post they can just tell the software to discard everything

00:02:51.280 --> 00:02:58.160
behind the subject removing the need for green screens and allowing for a

00:02:55.040 --> 00:03:00.160
complete 360 capture of the subject

00:02:58.160 --> 00:03:03.120
again using a bunch of complicated math and ai

00:03:01.280 --> 00:03:07.440
video is actually a sponsor vcs and contributed a gv100 to help crunch those

00:03:05.519 --> 00:03:12.640
huge equations they're able to create a point cloud of the subject with all of

00:03:09.760 --> 00:03:16.560
the depth data on its own this is really cool it removes the need for a bunch of

00:03:14.720 --> 00:03:21.360
post-processing time because the animators don't have to worry about

00:03:17.760 --> 00:03:23.840
things like cloth or hair physics but

00:03:21.360 --> 00:03:28.239
the huge advantage over motion capture is the ability to shoot full color

00:03:25.840 --> 00:03:32.879
textures of the actors in full makeup and costume this does have its own set

00:03:30.560 --> 00:03:37.599
of problems though the large five meter ring of cameras surrounding me is great

00:03:34.720 --> 00:03:42.640
for filming larger action sequences but when you consider each one of these

00:03:39.040 --> 00:03:44.480
cameras has a wide angle 155 degree lens

00:03:42.640 --> 00:03:48.159
if you're standing even a meter or two away there won't be nearly as much

00:03:46.239 --> 00:03:52.879
texture data which is where the high resolution array comes in this was

00:03:50.720 --> 00:03:57.360
originally made using a trampoline that was cut in half and held together with

00:03:54.879 --> 00:04:02.000
zip ties but they upgraded to this steel frame for shooting with neil anyway

00:03:59.920 --> 00:04:06.159
sequences will be planned out so that the actor can do their larger movements

00:04:04.159 --> 00:04:10.480
in the large array and then land in the high resolution array for parts that

00:04:07.920 --> 00:04:14.239
require more detail obviously the camera that is just a foot away from my face is

00:04:12.319 --> 00:04:18.799
going to be getting a lot more detail than the ones further out even that

00:04:16.320 --> 00:04:23.120
though isn't quite enough detail for vcs like they are going up against cinema

00:04:20.799 --> 00:04:28.160
cameras so they're upscaling all of the footage from 4k to 8k every single

00:04:26.000 --> 00:04:33.360
camera they custom-made an algorithm by capturing 4k footage on these cameras

00:04:30.639 --> 00:04:38.080
downscaled to 1080p and then used the 4k footage to train the ai for their best

00:04:35.840 --> 00:04:42.400
chance of upscaling it to 8k and with all that in place

00:04:39.840 --> 00:04:45.280
it still isn't quite the level of detail that is needed for a full blownsies

00:04:44.000 --> 00:04:49.520
hollywood film unfortunately there isn't really any way

00:04:47.199 --> 00:04:54.080
around the detail at this point but that isn't really a problem for demonic they

00:04:51.919 --> 00:04:58.720
did a lot less post-processing than i expected and left in a lot of the

00:04:55.840 --> 00:05:02.639
artifacts that happened naturally and it really adds to the horror of the film

00:05:00.560 --> 00:05:07.120
it's super cool how neil and the crew even use stuff like the point cloud data

00:05:04.880 --> 00:05:11.360
to act as a transition between real life and the simulation parts of the movie

00:05:09.680 --> 00:05:16.639
as you've probably guessed by now though running over 200 cameras at once uses a

00:05:14.479 --> 00:05:22.400
crapload of storage like this thing is creating

00:05:18.039 --> 00:05:25.360
9500 gigapixels per second and 7 to 14

00:05:22.400 --> 00:05:30.080
million polygons per frame which translates into about 10 terabytes of

00:05:27.520 --> 00:05:33.919
data for every 20 minutes of shooting the first day of shooting on neil's

00:05:31.600 --> 00:05:39.199
movie poor tobias and scott of vcs were here until 6 am grabbing the data with

00:05:36.960 --> 00:05:43.440
the two computers that they had so clearly an upgrade was needed

00:05:41.039 --> 00:05:50.000
surrounding the camera away we have 24 computers from oat studio each one is

00:05:45.680 --> 00:05:53.440
equipped with a 10 core i7 69 50k nice

00:05:50.000 --> 00:05:55.280
64 gigabytes of RAM and a gtx 1080 ti

00:05:53.440 --> 00:05:59.440
they don't actually need the gpus but they certainly need the CPU RAM and io

00:05:57.680 --> 00:06:05.120
to copy the data from each stand of cameras sorting all 200 plus video files was a

00:06:02.960 --> 00:06:08.720
huge problem for vcs so they created their own custom program called

00:06:06.720 --> 00:06:13.039
clippycopy named after everyone's favorite paperclip that is based off of

00:06:10.800 --> 00:06:16.960
the robocopy API that automatically pulls and sorts the insane number of

00:06:15.120 --> 00:06:21.199
files that are made every time that they press play once it gets to the post

00:06:19.120 --> 00:06:24.880
processing stage they create a global alignment of all the cameras this is

00:06:23.360 --> 00:06:29.199
basically looking at the features on the wall to figure out where each camera is

00:06:27.039 --> 00:06:33.280
in 3d space after that they create a depth map of everything in the room

00:06:31.199 --> 00:06:37.199
using the stereo pairs and from there it's very simple to create a point cloud

00:06:35.199 --> 00:06:41.360
of the entire scene every point in the cloud will give you not just the

00:06:38.720 --> 00:06:45.360
position in 3d space but also the RGB data of every point every three points

00:06:43.840 --> 00:06:50.080
in the point cloud is then poisson mapped to create polygons and using

00:06:47.759 --> 00:06:53.759
extra ai goodness using the camera data the textures are then improved again

00:06:52.080 --> 00:06:57.680
after all of that we get what basically looks like the best video game character

00:06:55.600 --> 00:07:02.080
model ever so like right here we have a capture of tobias's nan just chilling an

00:07:00.000 --> 00:07:08.240
unreal engine and i'm able to just sort of float around her in oh geez unreal

00:07:05.120 --> 00:07:09.840
engine using wasd

00:07:08.240 --> 00:07:14.880
it's just so cool and she's just playing back in real time

00:07:12.319 --> 00:07:18.560
in 3d it's crazy this processing obviously requires some

00:07:16.880 --> 00:07:22.720
balling computers that were provided by main gear and hp so down here we

00:07:20.800 --> 00:07:26.479
actually have two tiny backpack rigs rocking an rtx 2080.

00:07:24.960 --> 00:07:30.160
with all of that the director has an incredible amount of flexibility in

00:07:28.240 --> 00:07:33.680
posts so they're able to pick camera angles import the character into any

00:07:31.759 --> 00:07:37.120
environment they want and relight the scene to their liking in order to

00:07:35.759 --> 00:07:43.280
re-light the character though there can't be any shadows which means they

00:07:39.599 --> 00:07:43.280
need a lot of light

00:07:43.520 --> 00:07:50.080
they've got a lot of light in our office we actually use these same re sky panels

00:07:48.160 --> 00:07:54.960
we've got three of them and each one will run you about 20 grand so

00:07:52.800 --> 00:07:59.599
naturally they're using 24 of them here but even that was not enough light while we

00:07:58.000 --> 00:08:05.280
were here today they actually installed an additional 11 rem40s which

00:08:02.800 --> 00:08:09.759
are an additional 40 grand each so unsurprisingly they rented the lights

00:08:06.960 --> 00:08:13.840
for this shoot it's also insanely hot in here right now you probably can't tell

00:08:11.599 --> 00:08:18.479
because my shirt's black ltd store.com but i'm drenched you don't necessarily

00:08:16.720 --> 00:08:22.720
need a balling studio setup like this though to do volumetric capture since

00:08:20.560 --> 00:08:27.840
they're effectively just action cameras on a stand vcs can record anywhere and

00:08:25.520 --> 00:08:31.360
just you know footage goes locally on the cameras and they have batteries

00:08:29.520 --> 00:08:36.479
built in this way they can set up and say like the jungle at a farm or at a

00:08:34.080 --> 00:08:40.959
trade show and capture jensen wong and give us the data allowing us to have a

00:08:38.320 --> 00:08:46.080
little 3d jensen wherever we want for an extra portable setup they created

00:08:43.599 --> 00:08:50.080
this little box that has an image sensor in each one of these holes then you can

00:08:47.920 --> 00:08:54.480
set up say five of them around the subject and boom you're good to go there

00:08:52.560 --> 00:08:59.279
are additional applications outside of cinema 3d capture one of their biggest

00:08:56.880 --> 00:09:03.360
funders is actually epic games who hope to use this technology to do things like

00:09:00.959 --> 00:09:08.240
the travis scott concert but without all the time and money required to 3d model

00:09:05.839 --> 00:09:12.560
the whole thing like he could just do a set surrounded by these cameras and

00:09:10.320 --> 00:09:16.320
bippity bop he's straight into fortnite they're also looking into medical and

00:09:14.080 --> 00:09:20.720
military applications so say there's a world-renowned surgeon who knows how to

00:09:18.399 --> 00:09:25.040
do a really obscure knee surgery that surgery can then be recorded in 3d and

00:09:22.880 --> 00:09:28.720
played back from any angle for training purposes future improvements are planned

00:09:27.040 --> 00:09:32.880
for both hardware and software so hopefully one day cinemagrade 3d models

00:09:30.880 --> 00:09:36.320
can be captured volumetrically obviously getting better higher

00:09:34.720 --> 00:09:40.560
resolution cameras would allow for a better result but as more ground truth

00:09:38.560 --> 00:09:45.040
data is captured and imported they can train ai algorithms to better interpret

00:09:43.279 --> 00:09:50.560
the footage so hopefully you'll be able to get a similar level of fidelity to

00:09:46.880 --> 00:09:52.480
these 200 cameras with just 70 or so so

00:09:50.560 --> 00:09:56.000
huge shout out to tobias and scott for showing us their awesome tech and neil

00:09:54.240 --> 00:10:00.080
blomkamp and chris harvey for sharing their incredible expertise demonic is

00:09:58.320 --> 00:10:03.760
available now so check it out at the link below and as you'd expect from one

00:10:02.000 --> 00:10:08.480
of neil's films it's a fantastic technological showcase and proper

00:10:06.160 --> 00:10:12.800
frightening and also huge thanks to segways for bringing us to our sponsor

00:10:10.320 --> 00:10:16.720
ting ting mobile has new rates that make it easier to see how much you can save

00:10:14.640 --> 00:10:21.440
by switching you can get unlimited talk and text for ten dollars data plans to

00:10:18.880 --> 00:10:26.959
start at 15 and their new set 12 plan with 12 gigs of data for 35 dollars and

00:10:24.240 --> 00:10:30.640
unlimited data for 45 dollars a month if you like their previous pay as you use

00:10:28.720 --> 00:10:34.880
plans it's still there with tang mobile's flex plans starting at just

00:10:32.480 --> 00:10:38.640
five dollars per Gigabyte data can also be shared if you have a family plan

00:10:36.399 --> 00:10:42.079
connect more phones and save more you'll still get national and award-winning

00:10:40.320 --> 00:10:45.839
coverage and pretty much any phone will work with ting mobile t-mobile now does

00:10:44.160 --> 00:10:50.399
have the perfect plan for everyone no matter what your needs are so check them

00:10:47.600 --> 00:10:52.560
out at Linus.ting.com and receive a 25 credit

00:10:51.279 --> 00:10:56.640
if you're looking for something else to watch while it isn't really similar to

00:10:54.399 --> 00:11:01.440
this video but the porsche tycan review was loads of fun to shoot so go watch

00:10:59.040 --> 00:11:01.440
that