1
00:00:00,080 --> 00:00:05,839
i'm here today at peacemaker filmworks who have worked on small projects you

2
00:00:04,319 --> 00:00:09,840
might have heard of like deadpool 2 and the new ghostbusters

3
00:00:08,080 --> 00:00:14,320
they're known for their crazy camera cars but today their space is being used

4
00:00:12,080 --> 00:00:19,920
for possibly the craziest 3d capture setup in the world created by volumetric

5
00:00:16,640 --> 00:00:23,519
camera systems we've got over 200 CPU

6
00:00:19,920 --> 00:00:25,359
cores 100 terabytes of storage and if

7
00:00:23,519 --> 00:00:31,039
you follow me in here 239

8
00:00:27,439 --> 00:00:34,480
cameras capturing a staggering 920k

9
00:00:31,039 --> 00:00:35,520
image 30 times a second which begs the

10
00:00:34,480 --> 00:00:39,840
question who needs to capture that much detail oh

11
00:00:37,920 --> 00:00:44,399
yeah it's neil freaking blomkamp the director of district 9 elysium and the

12
00:00:42,239 --> 00:00:48,879
film we're looking at today demonic it dropped yesterday and the way they use

13
00:00:46,239 --> 00:00:52,399
3d capture in it is just mind-bendingly cool

14
00:00:49,840 --> 00:00:56,879
and terrifying and you know what's also terrifying this segue to our sponsor

15
00:00:54,879 --> 00:01:01,359
thanks to origin pc for sponsoring this video origin pcs can be customized with

16
00:00:59,199 --> 00:01:06,479
11th gen Intel core processors and NVIDIA GeForce rtx 30 series graphics

17
00:01:04,080 --> 00:01:11,880
with max p design backed by a 24 7 support team check out origin pc at the

18
00:01:08,640 --> 00:01:11,880
link below

19
00:01:18,560 --> 00:01:26,560
this whole setup starts with the yi camera uh 4k action camera that's well

20
00:01:24,000 --> 00:01:31,119
inexpensive volumetric camera systems or vcs would have obviously preferred to

21
00:01:28,560 --> 00:01:36,720
have cinema cameras all around here but when you have to buy 200 of something

22
00:01:33,200 --> 00:01:38,000
plus spares it gets expensive fast and

23
00:01:36,720 --> 00:01:43,200
it's not like these are bad little cameras each one has the sony imx 377

24
00:01:41,600 --> 00:01:48,000
sensor the same one as in the google pixel and when you have over 200 of them

25
00:01:45,759 --> 00:01:52,399
pointed at a subject you can get quite a bit of detail out of it each camera has

26
00:01:50,479 --> 00:01:56,960
custom firmware and an extra pin installed in the usb cable that allows

27
00:01:54,399 --> 00:02:01,119
for sync across all of the cameras this allows for the clocks to be synced

28
00:01:58,320 --> 00:02:06,000
between every single camera so if i hit play on the master camera right here

29
00:02:03,759 --> 00:02:08,360
which is conveniently marked by not having a front plate

30
00:02:07,759 --> 00:02:13,680
all 239 cameras start recording and then

31
00:02:11,360 --> 00:02:17,520
click it again and they all turn off the special thing

32
00:02:15,680 --> 00:02:21,680
about this rig though is how all of that is turned into 3d data this is

33
00:02:19,360 --> 00:02:26,640
accomplished using feature detection so all the tape in the background creates

34
00:02:23,760 --> 00:02:31,599
these sharp edges so ai can go in and figure out where each camera is compared

35
00:02:29,040 --> 00:02:36,400
to all of the others in the scene also every two cameras so like these two

36
00:02:33,920 --> 00:02:40,480
right here are used to create a stereo pair and using some complicated

37
00:02:38,640 --> 00:02:44,640
trigonometry they're able to calculate the depth of anything in front of them

38
00:02:42,720 --> 00:02:49,519
this is one of the keys to why the vcs camera system is so sick by using this

39
00:02:47,200 --> 00:02:53,599
depth data in post they can just tell the software to discard everything

40
00:02:51,280 --> 00:02:58,160
behind the subject removing the need for green screens and allowing for a

41
00:02:55,040 --> 00:03:00,160
complete 360 capture of the subject

42
00:02:58,160 --> 00:03:03,120
again using a bunch of complicated math and ai

43
00:03:01,280 --> 00:03:07,440
video is actually a sponsor vcs and contributed a gv100 to help crunch those

44
00:03:05,519 --> 00:03:12,640
huge equations they're able to create a point cloud of the subject with all of

45
00:03:09,760 --> 00:03:16,560
the depth data on its own this is really cool it removes the need for a bunch of

46
00:03:14,720 --> 00:03:21,360
post-processing time because the animators don't have to worry about

47
00:03:17,760 --> 00:03:23,840
things like cloth or hair physics but

48
00:03:21,360 --> 00:03:28,239
the huge advantage over motion capture is the ability to shoot full color

49
00:03:25,840 --> 00:03:32,879
textures of the actors in full makeup and costume this does have its own set

50
00:03:30,560 --> 00:03:37,599
of problems though the large five meter ring of cameras surrounding me is great

51
00:03:34,720 --> 00:03:42,640
for filming larger action sequences but when you consider each one of these

52
00:03:39,040 --> 00:03:44,480
cameras has a wide angle 155 degree lens

53
00:03:42,640 --> 00:03:48,159
if you're standing even a meter or two away there won't be nearly as much

54
00:03:46,239 --> 00:03:52,879
texture data which is where the high resolution array comes in this was

55
00:03:50,720 --> 00:03:57,360
originally made using a trampoline that was cut in half and held together with

56
00:03:54,879 --> 00:04:02,000
zip ties but they upgraded to this steel frame for shooting with neil anyway

57
00:03:59,920 --> 00:04:06,159
sequences will be planned out so that the actor can do their larger movements

58
00:04:04,159 --> 00:04:10,480
in the large array and then land in the high resolution array for parts that

59
00:04:07,920 --> 00:04:14,239
require more detail obviously the camera that is just a foot away from my face is

60
00:04:12,319 --> 00:04:18,799
going to be getting a lot more detail than the ones further out even that

61
00:04:16,320 --> 00:04:23,120
though isn't quite enough detail for vcs like they are going up against cinema

62
00:04:20,799 --> 00:04:28,160
cameras so they're upscaling all of the footage from 4k to 8k every single

63
00:04:26,000 --> 00:04:33,360
camera they custom-made an algorithm by capturing 4k footage on these cameras

64
00:04:30,639 --> 00:04:38,080
downscaled to 1080p and then used the 4k footage to train the ai for their best

65
00:04:35,840 --> 00:04:42,400
chance of upscaling it to 8k and with all that in place

66
00:04:39,840 --> 00:04:45,280
it still isn't quite the level of detail that is needed for a full blownsies

67
00:04:44,000 --> 00:04:49,520
hollywood film unfortunately there isn't really any way

68
00:04:47,199 --> 00:04:54,080
around the detail at this point but that isn't really a problem for demonic they

69
00:04:51,919 --> 00:04:58,720
did a lot less post-processing than i expected and left in a lot of the

70
00:04:55,840 --> 00:05:02,639
artifacts that happened naturally and it really adds to the horror of the film

71
00:05:00,560 --> 00:05:07,120
it's super cool how neil and the crew even use stuff like the point cloud data

72
00:05:04,880 --> 00:05:11,360
to act as a transition between real life and the simulation parts of the movie

73
00:05:09,680 --> 00:05:16,639
as you've probably guessed by now though running over 200 cameras at once uses a

74
00:05:14,479 --> 00:05:22,400
crapload of storage like this thing is creating

75
00:05:18,039 --> 00:05:25,360
9500 gigapixels per second and 7 to 14

76
00:05:22,400 --> 00:05:30,080
million polygons per frame which translates into about 10 terabytes of

77
00:05:27,520 --> 00:05:33,919
data for every 20 minutes of shooting the first day of shooting on neil's

78
00:05:31,600 --> 00:05:39,199
movie poor tobias and scott of vcs were here until 6 am grabbing the data with

79
00:05:36,960 --> 00:05:43,440
the two computers that they had so clearly an upgrade was needed

80
00:05:41,039 --> 00:05:50,000
surrounding the camera away we have 24 computers from oat studio each one is

81
00:05:45,680 --> 00:05:53,440
equipped with a 10 core i7 69 50k nice

82
00:05:50,000 --> 00:05:55,280
64 gigabytes of RAM and a gtx 1080 ti

83
00:05:53,440 --> 00:05:59,440
they don't actually need the gpus but they certainly need the CPU RAM and io

84
00:05:57,680 --> 00:06:05,120
to copy the data from each stand of cameras sorting all 200 plus video files was a

85
00:06:02,960 --> 00:06:08,720
huge problem for vcs so they created their own custom program called

86
00:06:06,720 --> 00:06:13,039
clippycopy named after everyone's favorite paperclip that is based off of

87
00:06:10,800 --> 00:06:16,960
the robocopy API that automatically pulls and sorts the insane number of

88
00:06:15,120 --> 00:06:21,199
files that are made every time that they press play once it gets to the post

89
00:06:19,120 --> 00:06:24,880
processing stage they create a global alignment of all the cameras this is

90
00:06:23,360 --> 00:06:29,199
basically looking at the features on the wall to figure out where each camera is

91
00:06:27,039 --> 00:06:33,280
in 3d space after that they create a depth map of everything in the room

92
00:06:31,199 --> 00:06:37,199
using the stereo pairs and from there it's very simple to create a point cloud

93
00:06:35,199 --> 00:06:41,360
of the entire scene every point in the cloud will give you not just the

94
00:06:38,720 --> 00:06:45,360
position in 3d space but also the RGB data of every point every three points

95
00:06:43,840 --> 00:06:50,080
in the point cloud is then poisson mapped to create polygons and using

96
00:06:47,759 --> 00:06:53,759
extra ai goodness using the camera data the textures are then improved again

97
00:06:52,080 --> 00:06:57,680
after all of that we get what basically looks like the best video game character

98
00:06:55,600 --> 00:07:02,080
model ever so like right here we have a capture of tobias's nan just chilling an

99
00:07:00,000 --> 00:07:08,240
unreal engine and i'm able to just sort of float around her in oh geez unreal

100
00:07:05,120 --> 00:07:09,840
engine using wasd

101
00:07:08,240 --> 00:07:14,880
it's just so cool and she's just playing back in real time

102
00:07:12,319 --> 00:07:18,560
in 3d it's crazy this processing obviously requires some

103
00:07:16,880 --> 00:07:22,720
balling computers that were provided by main gear and hp so down here we

104
00:07:20,800 --> 00:07:26,479
actually have two tiny backpack rigs rocking an rtx 2080.

105
00:07:24,960 --> 00:07:30,160
with all of that the director has an incredible amount of flexibility in

106
00:07:28,240 --> 00:07:33,680
posts so they're able to pick camera angles import the character into any

107
00:07:31,759 --> 00:07:37,120
environment they want and relight the scene to their liking in order to

108
00:07:35,759 --> 00:07:43,280
re-light the character though there can't be any shadows which means they

109
00:07:39,599 --> 00:07:43,280
need a lot of light

110
00:07:43,520 --> 00:07:50,080
they've got a lot of light in our office we actually use these same re sky panels

111
00:07:48,160 --> 00:07:54,960
we've got three of them and each one will run you about 20 grand so

112
00:07:52,800 --> 00:07:59,599
naturally they're using 24 of them here but even that was not enough light while we

113
00:07:58,000 --> 00:08:05,280
were here today they actually installed an additional 11 rem40s which

114
00:08:02,800 --> 00:08:09,759
are an additional 40 grand each so unsurprisingly they rented the lights

115
00:08:06,960 --> 00:08:13,840
for this shoot it's also insanely hot in here right now you probably can't tell

116
00:08:11,599 --> 00:08:18,479
because my shirt's black ltd store.com but i'm drenched you don't necessarily

117
00:08:16,720 --> 00:08:22,720
need a balling studio setup like this though to do volumetric capture since

118
00:08:20,560 --> 00:08:27,840
they're effectively just action cameras on a stand vcs can record anywhere and

119
00:08:25,520 --> 00:08:31,360
just you know footage goes locally on the cameras and they have batteries

120
00:08:29,520 --> 00:08:36,479
built in this way they can set up and say like the jungle at a farm or at a

121
00:08:34,080 --> 00:08:40,959
trade show and capture jensen wong and give us the data allowing us to have a

122
00:08:38,320 --> 00:08:46,080
little 3d jensen wherever we want for an extra portable setup they created

123
00:08:43,599 --> 00:08:50,080
this little box that has an image sensor in each one of these holes then you can

124
00:08:47,920 --> 00:08:54,480
set up say five of them around the subject and boom you're good to go there

125
00:08:52,560 --> 00:08:59,279
are additional applications outside of cinema 3d capture one of their biggest

126
00:08:56,880 --> 00:09:03,360
funders is actually epic games who hope to use this technology to do things like

127
00:09:00,959 --> 00:09:08,240
the travis scott concert but without all the time and money required to 3d model

128
00:09:05,839 --> 00:09:12,560
the whole thing like he could just do a set surrounded by these cameras and

129
00:09:10,320 --> 00:09:16,320
bippity bop he's straight into fortnite they're also looking into medical and

130
00:09:14,080 --> 00:09:20,720
military applications so say there's a world-renowned surgeon who knows how to

131
00:09:18,399 --> 00:09:25,040
do a really obscure knee surgery that surgery can then be recorded in 3d and

132
00:09:22,880 --> 00:09:28,720
played back from any angle for training purposes future improvements are planned

133
00:09:27,040 --> 00:09:32,880
for both hardware and software so hopefully one day cinemagrade 3d models

134
00:09:30,880 --> 00:09:36,320
can be captured volumetrically obviously getting better higher

135
00:09:34,720 --> 00:09:40,560
resolution cameras would allow for a better result but as more ground truth

136
00:09:38,560 --> 00:09:45,040
data is captured and imported they can train ai algorithms to better interpret

137
00:09:43,279 --> 00:09:50,560
the footage so hopefully you'll be able to get a similar level of fidelity to

138
00:09:46,880 --> 00:09:52,480
these 200 cameras with just 70 or so so

139
00:09:50,560 --> 00:09:56,000
huge shout out to tobias and scott for showing us their awesome tech and neil

140
00:09:54,240 --> 00:10:00,080
blomkamp and chris harvey for sharing their incredible expertise demonic is

141
00:09:58,320 --> 00:10:03,760
available now so check it out at the link below and as you'd expect from one

142
00:10:02,000 --> 00:10:08,480
of neil's films it's a fantastic technological showcase and proper

143
00:10:06,160 --> 00:10:12,800
frightening and also huge thanks to segways for bringing us to our sponsor

144
00:10:10,320 --> 00:10:16,720
ting ting mobile has new rates that make it easier to see how much you can save

145
00:10:14,640 --> 00:10:21,440
by switching you can get unlimited talk and text for ten dollars data plans to

146
00:10:18,880 --> 00:10:26,959
start at 15 and their new set 12 plan with 12 gigs of data for 35 dollars and

147
00:10:24,240 --> 00:10:30,640
unlimited data for 45 dollars a month if you like their previous pay as you use

148
00:10:28,720 --> 00:10:34,880
plans it's still there with tang mobile's flex plans starting at just

149
00:10:32,480 --> 00:10:38,640
five dollars per Gigabyte data can also be shared if you have a family plan

150
00:10:36,399 --> 00:10:42,079
connect more phones and save more you'll still get national and award-winning

151
00:10:40,320 --> 00:10:45,839
coverage and pretty much any phone will work with ting mobile t-mobile now does

152
00:10:44,160 --> 00:10:50,399
have the perfect plan for everyone no matter what your needs are so check them

153
00:10:47,600 --> 00:10:52,560
out at Linus.ting.com and receive a 25 credit

154
00:10:51,279 --> 00:10:56,640
if you're looking for something else to watch while it isn't really similar to

155
00:10:54,399 --> 00:11:01,440
this video but the porsche tycan review was loads of fun to shoot so go watch

156
00:10:59,040 --> 00:11:01,440
that