Can We Fix It? - Server Upgrade Vlog 2019 Pt 2
Linus Tech Tips
·Linus Tech Tips
·2019-05-06
·
4,197 words · ~20 min read
0:00
wait what it works check it out google
0:06
track trips what do you think he changed can you
0:10
hear anything from him speed test is good okay let's pretend
0:13
like it's not working it's still not working
0:17
okay just kidding it works
0:21
all i did was unplug it and replug it that's weird so
0:24
what would not work right now well wi-fi doesn't work what else does
0:29
not work guest Wi-Fi i mean guest wi-fi should work so here's a question for you
0:34
guys do you want to go for like another two hours and possibly be done that's
0:38
what i would say we can get dinner
0:49
dinner okay game plan time it's game plan time i got
0:53
my lovely assistant here's a pencil i would use a pencil no okay so stuff to
0:58
do what is okay what's not working do we know wi-fi is working no idea guest
1:02
wi-fi guest wi-fi fix vlans set up
1:06
monitoring for network devices and servers um right isn't this episode
1:11
brought to you by pulseway pulseway is a real-time remote monitoring and
1:14
management software that allows you to manage and monitor from one app
1:18
compatible machines that run Windows macOS or Linux you can get access to
1:23
real-time status reports system resources logged in users network
1:28
performance Windows updates and more and you can fix problems on the go by
1:32
sending commands from any mobile device we gotta install that replacement part
1:36
in the ups where is that uh yeah about that what they didn't respond to me i
1:41
thought you said it was out here yesterday when i asked you about it what
1:44
no you didn't you didn't ask me at all i think the footage we'll find that i in
1:48
fact did i contacted eaton yeah um it was the u.s
1:54
department so they were like oh well you need to talk to the canadian department
1:58
contacted them radio silence okay well we need to just get it from the u.s guys
2:02
then just provide them a u.s address u.s card whatever it takes we just need to
2:06
whatever well then call him back again don't tell
2:10
him you're in canada that's like what we're doing this weekend well i mean the module's easy to replace
2:15
pass through 10 gigabit steam cache we need to get the dumping ground set up to
2:19
delete files older than 100 days is the mac mini done it's going up today
2:24
okay yeah we're using a mac mini for wiping sd cards we're not just wiping
2:28
the sd cards we also need the 10 gig connectivity in order to okay all right
2:32
i'm going to start by installing servers because i think we're kind of at that stage now okay here i'm going to get a
2:37
colored sharpie not to write on Linus so uh i can just put this on your desk
2:41
right no is that not a thing no brandon
2:45
already said this is fine we just have another monitor here so that can be a
2:48
problem if i just
2:54
that wasn't me david why do you drop that so from time to time we have issues
2:58
with the ingest softwares that are available on pc or there are human
3:03
errors that come in and interrupt our backups so a lot of more reputable i
3:08
don't know if they're better we haven't tried them yet but a lot of ingest solutions are available for mac because
3:13
a lot of productions use mac we're also going to experiment with using a
3:18
dedicated station as our wiping station
3:21
for cards in addition to transferring the data wherever it's supposed to go
3:25
you will also take your card out and anytime it's time to wipe a card you
3:30
must do it in the dedicated wiping station including a full copy that way
3:36
in the event that the initial ingest didn't work theoretically there should
3:39
be at least two copies of the file somewhere
3:44
so that's why we're doing this here's the thing i've plugged in all the
3:48
Ethernet cables from the patch panel so these are all the ones where like i know
3:52
what they are but a lot of this stuff that's just hanging into the room is not
3:57
labeled this is not even terminated oh god
4:01
my boy mickey mouse he's a good guy really weird cable don't know why people
4:04
use it i guess it's for things that need to be small wow this is actually way too
4:08
long hey Linus yeah do you have a do you have
4:12
an eta on uh when we're gonna have the servers back up why
4:17
oh my god we have to upload today's video don't we yeah oh my god
4:23
oh damn it you guys
4:27
kind of got other stuff to do one thing i'm pleased about though is
4:32
that i get to move wanik to be oriented correctly
4:36
above ruskin
4:42
that's fine i got this no i got this
4:45
lifter it's a common task in many workplaces
4:50
yet many employers may not be aware of how to reduce the risk associated with
4:55
lifting bending
4:58
while lifting can result in back pain
5:02
back injuries are painful and can be debilitating
5:10
not gonna lie i grabbed this monitor looked at the back of it and was like
5:13
that's displayport it wasn't displayport it was HDMI so i went and grabbed a
5:17
type-c to HDMI adapter but there's just HDMI on the back of the mac mini you
5:21
know why am i doing all this work i don't know i should just give up i'm
5:24
gonna go get a keyboard now so far i am not getting a signal out of it
5:30
it should at least know that it's plugged in unless it just came oh my god
5:35
okay and that is why you do screw in vga
5:39
shut up Anthony i want to see z drive
5:42
show me z drive show me there it is oh god it's full you can't even ingest to
5:47
this so i needed a keyboard right i'm like yeah let's get some nice logitech
5:51
goodness you know when you're clicky clacking this is not a logitech keyboard what is
5:56
that doing in there this is a cooler master keyboard so connected to the
5:59
guest wi-fi we've got no no ip no internet no nothing so right now it's
6:03
not working let's see if we can figure out the easier way to do guest wi-fi
6:10
handling heavy loads repeating the same motions can also
6:15
increase the risk of injury i made a screen protector well it's not
6:19
really a screen protector it's more like a case for the mac mini with painters tape this
6:24
is very nice artisanal painters tape definitely not from home
6:29
depot so the goal of this is that i can
6:33
put that on top of there well bam she's done
6:36
i got a little excited okay maybe you are a macOS guy
6:41
wow hey look we got a pulse away notification
6:45
says that wanik server is back online yeah neat huh yeah so now we just need
6:50
to install it on all the servers yes all right here is very fine yeah yeah
6:56
so much hair for me i need a haircut that's why
7:00
Jake can you find something useful to do you
7:03
made a list i'm waiting for you you made a list go check it twice i can put this
7:06
in on my own oh my gosh um change all default passwords i don't know how what
7:11
passwords need to be changed i don't know any of the ips and stuff
7:14
you could ask hey what's the eyepiece just stop
7:18
yanking it i like yanking things okay here just
7:22
he david is uncomfortable did you hear that
7:26
that's a shame this is awful yeah you are pretty
7:29
awesome who installed this a company you probably paid thousands of dollars to
7:34
can you imagine like the way they must have to have those kinds of
7:38
conversations on like a porn set like hey if you could just like you know jam
7:42
your knee over there then we'll be able to you know see the you know the action
7:46
better uh so today's video apparently it
7:49
needs to be reposted to youtube but it is not currently on the server because
7:54
it's been pushed back a lot so it's on the vault oh god so when does this have
7:58
to go up noon what time is it uh 11 42
8:02
oh my god okay well then we need to get the vault back up don't we since when
8:05
are they are you sure it's on the vault i don't know why don't we look at wanak
8:09
look i don't know can i just have the transcode server oh where's the nail i'm
8:13
gonna put this hold on i need the nail the what nailed it
8:17
that looks dangerous as heck it should be fine
8:22
just get it in it's not going here stick it in the hole oh and then we just push
8:27
it in and it does this thing maybe no no it's stuck on that okay
8:32
okay you're pushing me back yeah can you
8:35
change the angle a little bit like yeah that way yeah there we go okay now try
8:39
just RAM it in there until it goes until she shoots i love it hey Linus
8:45
you uh disconnected the wi-fi like the whole like now yeah like i
8:49
don't have access to the network let alone the internet like and it just went
8:52
down now like around the time you got this back up what the hell
8:57
here i'll plug it into 21. okay so this
9:00
is working all right so there's a little trick to booting the vault
9:04
when booting press escape then after 90 second timeout control d immediately you
9:08
have to do that because there's kind of a glitch
9:12
if you have 60 drives in it it like times out while it's trying to detect
9:15
them all and then the os fails to boot are you excited to install pulse way on
9:20
this they have a center west client yeah oh cool regular Linux
9:24
is pretty bare bones just in case you wanted all your packages to be out of
9:27
date hit up centos i need to get access to that neck air switch it's not coming
9:31
up i've removed the vlans
9:35
that shouldn't have caused any problems but it did
9:39
so you have to plug into it directly yeah i think so okay
9:42
okay so here's the thing the discs are showing up but the the screen on the
9:47
front says okay 0 out of 32 0 out of 28. isn't it
9:52
supposed to say 32 out of 32 all we did was reboot it and redo our entire
9:57
network that's more than it showed before yeah it's like 4 out of 10
10:01
working wait hold up hold up just give it a sec why is this empty
10:05
what the you're the man brad okay guys
10:09
you man steve help you guys you man steve you're the man so steve
10:14
you're the man we need a PoE injector which i think is
10:18
past Jake here just actually where do we keep the PoE
10:22
injectors i don't know
10:25
there's probably a like a box of them somewhere ha ha what you found one yeah
10:31
oh nice no that thing's not gonna have one you ain't the man Linus so remember
10:35
that stupid network stupid video monitoring proprietary stupid box it's
10:41
not a standard one you i can't even screw it into the rack so it's just
10:44
gonna like sit here i guess but now this switch is not working we
10:49
factory reset it we can't manage it with the cloud where is the cloud key oh the
10:53
cloud key's down here this isn't even lit
10:56
no wonder he can't access the cloud key the good news though the vault
11:00
and the transcode machine are 10 out of 10 working right now small victories
11:07
so we have the factory reset again
11:11
this time we reset the cloud key as well uh so
11:15
what is the default path oh what okay so installing pulseway is actually
11:19
really really damn easy you go to their website you click downloads we're on
11:23
Windows so we're gonna take the Windows 64-bit it's gonna download we're gonna
11:27
click on it and bam it's gonna install look at that bam i needed like five more
11:32
clicks than i said but there you go well bam
11:36
did it work all those those camera movements i go crazy because she look
11:41
like a flower and she sting like a bee
11:45
shaban one hour and 45 minutes that up time
11:49
minty bud well i mean that was on purpose that water cooled server yeah
11:53
that's pretty dank temperatures heck yeah brother wait we can chat with
11:57
the person hold up what
12:01
sup bro so we can run it in vms as well then
12:05
yeah we could know if a particular vm goes down okay so we need to get then we
12:09
need to get albion in here and get all of our vm's pulse weight up oh we need
12:12
to do the 10 gigabit so do we have the cards already theoretically there's
12:16
okay let's figure that out okay so here's what we're gonna do we're gonna
12:20
just lift it up a bit and then just kick the card out rather than trying to like like bring it down okay
12:27
i don't think that was necessarily any easier but sure that's how did i pick
12:31
this side i don't know so this has so many hard drives
12:35
i was like shoddy shoddy that yeah go for it can you imagine we dropped this
12:39
out oh oh yeah oh yeah put it in there
12:42
oh stick it in it's so tight oh my gosh hold on hold on
12:46
this dongle is in my way here we're gonna yeah we're gonna wait just
12:50
wait uh that seems to be in go for it oh yeah you thick girl boy whatever she is
12:56
he is i don't know whatever it's a big server
13:02
it was hard and then it was easy this one's special this one's deeper than all
13:06
the other servers was i wrong when i said she's a thick
13:09
girl no we didn't cheap out on the thickness
13:13
it's so slow why do i pay you am i missing one of the cables for this one i
13:17
don't know okay so we're back on the bare metal
13:20
export i'm going to set up a couple notifications here so we know what's
13:24
going on with this thing yeah it'd be good to know if adobe media encoder is
13:27
closed can we do that if it's closed whoa oh that's all priority elevated but
13:33
it might go not responding while it's encoding yeah all right sure let's do
13:36
critical why not yeah can we here maybe is this done can i just close this
13:42
it's not doing anything right you would know right
13:46
david paul's way critical adobe media encoder stopped on computer bm export it
13:50
works oh really that's awesome yeah let's see what other servers we can put
13:55
this on if there's any oh we can put it on clover uh choose your keyboard layout
14:00
it says reset your pc or see advanced options uh
14:04
uh uh i'm just gonna click turn off your pc it
14:08
was in startup recovery mode oh really here i'm gonna try to turn it on again
14:12
this is the backup one right yeah okay so silver has nothing important on it
14:17
except everything pulse way monitors our servers
14:23
so we know if they die because that's unfortunate when that
14:26
happens okay so obviously the install of pulse
14:30
away on Linux is a little bit different so that's gonna vary between you know
14:34
ubuntu or centos or whatever you're gonna have to change how you go with the
14:37
initial install but that's on their website that's easy to do once we do have it installed though we just wanna
14:42
go into etc pulse way cd etc pulse way
14:46
now that we're in there we can see that there's a sample config file so all you have to do is copy the sample config
14:51
file remove the sample bit and then now you can edit the config and enter all
14:54
your information so once we enter our username and password it's obviously
14:57
going to encrypt the password and then inside this config file you can go ahead
15:01
and enable all the things like we did on Windows just in a text file instead of a
15:07
gui this isn't a great example of how easy pulse wasn't to install because
15:12
slackware which is what unraid runs on and what our server here runs on is a
15:17
little more complicated than most Linux distros most of the time on Linux is
15:20
really easy you go that config file you start the service and you're done so
15:23
this is a bad example anyways it's working now you can see start pulse way damon
15:28
so i figured out why clover's not booting it looks like the cmos battery
15:31
is dead to give you some idea how pantone head ridiculous super micro's
15:35
BIOS crap can be sometimes i had to take
15:39
pictures of all the configuration pages on this stupid thing because like s SATA
15:44
versus SATA and which one is is which is actually different from one of our super
15:49
micro machines to another so i just need to make sure that these all
15:53
match so stupid
15:56
because we factor reset the cloud key yeah we no longer have ownership of uh
16:02
the aps so we need to adopt them do you know the
16:06
password for those no as far as i know
16:09
we never had one let's just let's just
16:12
blow up everything yeah blow it up
16:16
is there even space in there yeah i don't what does s say don't think like
16:21
google s SATA it's just gonna be like did you mean SATA no it's just a super
16:25
micro thing it just means the non chipset controlled say the controller
16:29
yeah except that sometimes it doesn't do you like pina coladas
16:36
you don't drink do you like pina coladas they're all right mojitos
16:41
uh better than pina coladas i just really really want to go home whoa hold
16:46
on a second there why are those configured to be non-raid disks did we
16:50
lose our raid setting what the
16:54
okay so now that we've added a 10 gig card to the steam cache and it actually
16:57
shows up you can see in here we got wired connection one which is our
17:01
regular virtualized network adapter and then we have the pass through a quantity
17:05
a 10 gigabit which is awesome now we actually have to set the ips
17:09
apparently i can't have ipv4 disabled
17:12
that doesn't make sense hopefully potentially
17:16
theoretically it should just start back up
17:20
and we'll be using our 10 gig nick okay it looks like this might have not
17:24
worked so that's cool i never got a different ip maybe
17:28
what is the problem why can't i connect
17:33
i blame you david yeah
17:37
so i've hold the last of the extra cables out of this room
17:40
and i am pretty much ready to just oh i guess
17:44
there's never mind hold on Jake is still working on the steam cache
17:49
over there maybe he has an update for us maybe we can do a test on it i have no
17:53
update everything's i'm going to my desktop
18:01
well i mean we're doing it stupid but like at the same time
18:05
there's nothing preventing anything from working at this point like it's all just
18:09
one big network nothing is segregated everything should be communicating but
18:13
it's not
18:20
i don't know what to tell you that switch only has a single connection to
18:24
the dell switch that's it what about for the netgear it doesn't have a connection
18:27
to the netgear switch everything goes through the dell did we set the subnet
18:31
on this properly i don't think we did possible that we didn't i bet you that's
18:34
what it is that's if anything i've learned in the last three years it's
18:37
almost always user error so was it user error that you came to be
18:42
wow jesus christ
18:46
yourself hell yeah it's not set oh my god if this
18:50
works again dude do you know how many points i am above you right now like
18:54
holy sh so it was the subnet thing yeah so how many other devices on our
18:58
network are going to be worked like that like what about the did we change the
19:02
access points are all the switches updated with this new subnet
19:06
yes are there any um vms that aren't updated
19:10
oh yeah like all of them well yeah so we should probably get on that
19:13
other than this one okay so what we have left to do we got to fix the cameras
19:18
that's in progress they're going to do that fix the key cards they're going to
19:21
do that but i'm going to go fix some subnets on things
19:24
which is super boring so you probably won't see me doing that
19:28
what what what is it i'm saying i'm gonna go
19:32
do this but it's gonna be super boring so we're probably not gonna film it oh oh sh did you just PoE your computer i
19:37
think so that's hilarious you know that's not
19:42
like actually funny right it's like kind of fun it's like actually being not
19:45
funny like if we just destroyed a laptop that's worth like over a thousand
19:49
dollars it's not funny why are you laughing it's kind of funny no it's actually not i remember these being kind
19:54
of unintuitive to figure out but i've never heard of them frying something can
19:57
we plug the amazi injector can we plug it into the switch like in a non-PoE
20:02
port like in the nectar switch or something sure okay
20:06
so plugging into the camera directly okay okay we're in so okay so now what
20:13
okay so what i can do is make sure that it's on the right network now we can try
20:16
plugging it back in and see if that works fine hold on router gateway that's
20:20
not right well i mean that's what the other ones were set too too okay so move it over
20:26
so here's our ghetto solution to PoE inject connect to the camera
20:32
anyway we shouldn't have to do that it's plugged into a PoE switch we just i
20:36
don't okay so you replugged it into the uh
20:40
yeah okay so let's hit it again yeah and it's
20:44
working it's up okay so what we just need to do that one at a time for all
20:48
the remaining ones what a pain in the ass
20:52
okay well it's not perfect
20:56
but all the servers are back up and running
21:00
performance looks fine the steam cache is working which it
21:04
wasn't before we have enough 10 gig ports for everyone
21:08
upstairs and we finally have monitoring running
21:12
on our servers so it's not a total loss it's just
21:16
really not as much as we wanted to accomplish
21:19
but at 6 30 on sunday like we can't be tearing things apart
21:24
now and have things not be at least sort of functional and people come in in the
21:28
morning so we just got to clean up and uh
21:32
try to go to sleep tonight without just like having visions of ip addresses
21:37
hammering our brains