Why do we need SO MANY SERVERS??
Linus Tech Tips
·Linus Tech Tips
·2019-05-06
·
2,444 words · ~12 min read
0:00
why the heck do you guys need a whole room full of servers
0:03
just to make youtube videos we get asked this
0:07
and actually a lot of other questions about our data management and our
0:10
workflow all the time so for the last
0:14
year actually ever since we got our red cameras i've been meaning to do an
0:17
update for you guys on how we handle such heavy footage
0:21
how we share resources with such a large team and how we do all of that while
0:27
maintaining our rigorous daily release schedule so
0:32
when jiwin reached out to us asking us to do a
0:36
feature on their smooth 4 camera gimbal i thought well hey there's an
0:41
opportunity why don't we grab this thing
0:44
and take it with us while i show you guys the way that data flows through
0:48
Linus media group to become a video on your screen
1:02
so while i let you all appreciate the irony of a video about our overly
1:06
complicated workflow shot on a cell phone with a simple handheld gimbal i'll
1:11
tell you guys a little bit more about the zoom four so it's got a highly
1:15
integrated control panel down here with apps for both iOS and Android that allow
1:20
you to control your camera it's got focus pull and zoom capabilities it's
1:24
got the ability to do vertigo shots with one button through the app or manually
1:30
it's got what they call phone go mode for instant scene transitions and it's
1:34
got both a quick standby mode and a one-click mode switch
1:38
everything that we shot outside of the intro and this outro right now was
1:41
filmed on this it's got a note 8 on it right now but it was shot on an iphone
1:45
10 and i'll let you guys be the judge of the stabilized footage that comes off of
1:49
it so to demonstrate some of the challenges with working with red footage we're
1:54
actually going to take that intro that we just shot just now and we're going to
1:58
ingest the whole thing to show you guys how we do it now red cameras
2:02
are not great for recording audio in fact even with an external preamp it is
2:08
really difficult to get anything resembling usable audio out of the
2:11
camera directly so recently we finally gave in and resorted to an
2:16
external audio recorder it really sucks to synchronize audio
2:20
especially if you're stuck using the scratch audio that gets recorded
2:23
directly to the camera and then the sd card out of your external recorder and
2:27
aligning it manually but we recently got a little doodad called a tentacle sink
2:32
you get two pieces here a master that sits on our mix pre-6 and a slave that
2:37
sits on our camera you synchronize them at the beginning of the day and what
2:41
they do is they inject what's called time code into the mag and into the sd
2:45
card so that you can easily synchronize them later on down the line
2:50
but that is far from the only problem so the
2:54
thing about the weapon 8k so that's our camera recently renamed the
2:59
dsmc2 helium or something like that
3:02
is that it records footage they're both the same camera at up to 8k
3:07
60 frames per second and with the 480 or 960 gig mags that's
3:13
up to 300 megabytes per second so to put
3:16
that in perspective if we were filming a simple TechLinked episode even though
3:22
we will usually use like a 20 to 1 compression ratio so that's about four
3:28
times the compression that they would probably use on something like a feature
3:31
film that means that a 10 gig clip could
3:34
still end up being in the neighborhood of about did i say 10 gig clip
3:39
that means that a 10 minute clip could still end up being in the neighborhood
3:43
of around 50 gigs and while it's less of
3:47
an issue for something like Linus tech tips or Techquickie when you're
3:50
recording dailies and you're expected to release the video on the same day that
3:55
you film it like with something like a new show
3:58
size of files ends up becoming a potential problem so that's why our two
4:02
ingestations which you might recognize from one of our tech showdown episodes
4:07
are both equipped with 10 gigabit network cards so one of them has this
4:11
ASUS one which is basically the last generation version of this action one
4:17
they're both using aquantia chips that allow them to be somehow under a hundred
4:22
dollars for 10 gig networking there are a number of different ways that we
4:25
ingest footage depending on the type of camera we're using so for red you can
4:29
see each of our video clips is actually a folder made up of smaller broken up
4:34
video clips so we're going to go ahead and grab that and copy the whole thing into our
4:39
project folder for this video so that goes in a roll
4:43
one what we're looking at right here is
4:46
probably some kind of like weird buffering thing or something but this is
4:50
about what we'd expect to see in terms of our ingest speeds
4:54
it's not the full 10 gigabit because that would be in the neighborhood of one
4:59
to 1.1 gigabytes per second but
5:02
the reality of it is that you're going to run into bottlenecks elsewhere and in
5:06
fact these red megs even though we're using an esata interface for them are
5:10
just not that fast and they only read at about 230 megs a second
5:16
then what we got to do is grab the audio clip off of our sd card that goes into
5:20
our audio folder right here and that's pretty much the whole process
5:25
for red footage but for that we wouldn't need powerful
5:30
machines and there is a reason that each of these is running 64 gigs of RAM and a
5:35
10 core extreme edition processor and that's because for our other cameras
5:40
like the a7s for example we actually use
5:43
adobe prelude which is an imperfect
5:46
piece of software but has its uses
5:50
to transcode the footage when we're bringing it in so what we do is we take
5:54
whatever project it is and this is generally fast as possible we still shoot that on the a7s we create a
5:59
subfolder and we set it to transcode to cineform 4k this improves our timeline
6:05
performance when we're scrubbing through the clips in adobe premiere the other
6:09
thing that we do is set a second destination for the original files that
6:13
we're not transcoding just in case something goes wrong with the transcode
6:17
which does happen from time to time and we need to go back and grab the original
6:22
files now just to give you some idea even if
6:25
we're not using the full potential of this network connection all the time of
6:29
what it's capable of let's go ahead and just grab a file
6:33
off the desktop and show you just what this puppy can do
6:37
so that is saturating the read speeds of
6:40
the raid 1 ssds that each of these machines are booted off of so that was a
6:45
10 Gigabyte file let's go have a look on the other end of this Ethernet cable at
6:50
how exactly that whole thing comes together now i've shown you guys our
6:53
server room a fair number of times but i
6:57
haven't given you guys an update in quite a while
7:00
on how exactly it's working in here
7:04
so the main server that everyone is editing off of at the same time
7:09
is this one right here this is wanik server and it's running uh 24 plus one
7:15
two three four five six seven so it's running 31
7:19
Intel 750 series 1.1 terabyte NVMe ssds
7:24
and this guy is an absolute monster
7:28
so if we fire up performance monitor you can see that even though we've got
7:32
four of our editors in office right now and we're seeing each of them doing 150
7:38
120 50 megabytes a second of reads this
7:42
thing is barely suffering and our
7:45
there we go our z disc q depth is only about 0.5 that's the benefit of NVMe is
7:52
that nice fast responsiveness and in fact when we were still using our SATA
7:57
SSD server for everyone we were starting to run into issues once we had the room
8:02
fully staffed over there where adobe premiere would crash but not just crash
8:07
everyone's would crash at the same time and we narrowed that down to slow
8:11
response times on our nas so this guy's
8:15
connected at 40 gigabits per second
8:18
this one right here has 24 SATA ssds and
8:22
this one only gets used this is called qq server this one only gets used for
8:28
large projects that only one person needs to work on at a time it's still
8:34
SSD based so we still don't run into any crashes or any other weird issues like
8:38
that but it doesn't have quite the snap that
8:41
an NVMe machine does now something that we learned
8:45
a really valuable lesson about i guess it must have been about two
8:50
years ago was real time data replication and
8:54
off-site backup and that is where this guy comes in so i want to show you guys
8:59
a little trick here i don't know if i've ever actually done
9:02
this demo in a video before i've shown people that have come into our office
9:06
so i'm going to use my test folder here and i'm going to create a document
9:10
called uh test for video right here this is a text file
9:15
on our main drive then i'm going to jump over to this hard drive based server
9:20
under it so this is running eight eight terabyte drives for a total of 64
9:25
terabytes and i'm just going to pull up the
9:30
wanik sink folder
9:34
get that test folder open test video but
9:38
just like that so within about five to ten seconds whether it's a text file or
9:44
whether it's a large video file the synchronization software that we're
9:48
running here we'll automatically dump it over to here so if this crashes we can
9:53
actually continue editing videos within about 10 to 15 minutes of switching over
9:58
everyone's map drives and another benefit that it gives us is that
10:02
normally on a network drive when you delete a file it's just gone
10:07
but instead check this out if i delete test for video i'm actually
10:12
going to have to do it from a separate server but if i delete test for video
10:20
boop you're gone you're done
10:25
and i go into here that file is going to disappear but
10:30
what will happen is we can go into the deletions folder and we can rescue it
10:36
this has actually saved our butts
10:40
more times than i would like to admit because
10:43
stuff does get accidentally deleted now
10:47
all of this is only for active projects that we are
10:51
working on right now once we're done with something it goes
10:55
on to petabyte project and we're actually running both phases of it right
10:59
now when we originally deployed it we were only using one of them so we could
11:03
just save power on hours for all the drives that were in the other one
11:07
not the case anymore so petabyte project is now up to
11:12
777 terabytes of total space of which
11:16
we've consumed 432. this holds all of
11:19
our archived projects so that in the event that we want to make a video that
11:23
refers back to one of our other videos
11:26
we can grab the original quality files rather than downloading off youtube like
11:31
a lot of other youtubers do bringing us into this room one of the biggest
11:34
challenges is not just working with red footage because the files are so heavy
11:39
one of the biggest problems that we have is that we've got multiple people
11:43
working off of that same shared nas all at the same time
11:47
so this is where the 10 gigabit connections that
11:51
our editors are also using comes in when we're scrubbing through footage we can
11:56
actually see data rates in excess of two and a half gigabit per second so we
12:00
peaked at just shy of four gigabit per
12:04
second that my friends is why we've got the bang and
12:09
nas with the high speed networking
12:12
so once the edit is done taran or one of our other editors will export the entire
12:16
project as an mov but those are extremely inefficient
12:22
files so they're they're great quality but they're absolutely massive so he's
12:27
going to copy that goes into the transcode flow plane yep and then you
12:31
paste it here now what's going to happen there is that
12:35
water cooled server that i built a while ago the one that's at the very bottom of
12:38
the rack is going to see that that file got transferred and then it's going to
12:43
spit out the correct formats for all the different platforms that we upload to
12:47
whether it be youtube or Floatplane or facebook or whatever the case may be
12:53
that way the editor's machine doesn't get tied up with this so this is cool
12:57
once that file's done copying it does take a second and media encoder is not
13:03
always perfect before it'll pick it up but we use vnc
13:07
in order to remote into that machine so everyone can go in at the same time
13:10
unlike remote desktop connection and make sure that it has actually picked up
13:14
the project and that it's transcoding to the correct format okay cool the system
13:19
works so thanks for watching guys if you disliked this video you can hit that
13:24
button but if you liked it hit like get subscribed or maybe consider checking out where to buy the stuff we featured
13:28
at the link in the video description also down there is our merch store which
13:32
has cool shirts like this one and our community forum which you should totally
13:35
join