computer problems are a fact of life and
sometimes the fix is as simple as just turning it off and turning it back on
again but other times it's not and when the system you're talking about is
running an air traffic control system controlling a bunch of ATMs or say
routing 911 calls keeping them up and running can be a matter of life and
death now the stakes aren't nearly as
high for us but this server here runs multiple apps that we rely on every day
accelerates our game downloads with steam caching and it runs our DNS if
that service goes down it breaks literally everyone in the company's
internet which my boss informs me isn't great so how do we make it more reliable
it's already a server we build more
servers and what's really cool about this is everything we're about to show
you courtesy of Intel who sponsored this video and sent over their new Emerald
rapid Zeon CPUs can be done on nearly any computer
even your dad's Old Dell that is as long
as you have more than one so if one leaves for cigarettes we can still play
catch more than one dell not more than
one dad oh well anyways I'm done you
want to check this out yeah let's have a look you got your lovely cat picture
your crab rave on that computer M watch this like I can yeah I can interact with
this just just give it a second okay it's it's going hey now it's on this
computer and like no bamboozle here look watch whoa buddy watch watch watch boom
unplugged I can just completely interact with this as I normally would so what's
going on here what you guys just saw was the programs the lovely drawing the
entire operating system just teleporting from the computer over here to the one
over here no trickery this is possible
thanks to the magic of virtualization we've talked about it before but if
you're not familiar virtualization allows you to slice up a single machine
into multiple less powerful virtual machines and this setup leverages that
technology to allow us to move these virtual machines between multiple
physical computers that way if one breaks another one can immediately take
its place and the best part is that while this all sounds super fancy all
the software we're using is both open- source and free and we're going to show
you guys how the setup Works in a little bit first I want to take a look at the
servers we're going to be using for our setup Gigabyte sent over four of their
r163 sg2 aac1 servers these are Bare
Bones so we're going to have to add a few of Our Own Parts but we should be
able to build this in what like 5 minutes I'd like to see you try this guy
we're going to add some of our own parts starting with a pair of patriot 480 gig
SATA ssds that will function as a mirrored boot drive this kind of per
machine redundancy isn't strictly speaking necessary because we could lose
an entire ire machine in our configuration without having any issues
but having them in pairs makes our lives easier in the future potentially since
if one of them fails we can just replace it and then rebuild it from the other
one then on the other side of the machine we're installing two of these
kokia cd6 7 tbte drives for fast bulk
storage that leaves us six more SATA Bays to do nothing with and two more
NVMe Bays for potential future
expansion moving back let's get our CPU installed we're using a Zeon Platinum
8562 y+ in each node these were graciously provided by Intel and with 32
cores 64 threads and 4.1 GHz max turbo
clock speeds these are going to give us a ton of compute to share between our
virtual machines all at a modest 300 wat TTP we're going to have it and the rest
of the parts Linked In the video description now I've never installed in
this socket before so good luck me step
one is to install the carrier on the CPU and you can tell which one of the three
you're supposed to use by the little marking right there on the CPU IHS line
up our little Golden Triangle with our gigantic gargantuan hole in the whole
thing triangle oh this is adorable it's got a cute little ARM so you can break
the thermal pce seal with the cooler so you can get the cooler and the CPU separated more easily love to see it
speaking of thermal paste we're going to be using a Honeywell PTM 7950 pad
available at LTT Store.com this stuff is absolutely perfect for a server install
because it lasts not forever but for a
very very very long time without maintenance now you might think okay go
ahead put it onto the CPU socket you'd be wrong instead I'm going to install it
onto the cooler I'm going to know how to do that in a sec so arrow and arrow so
maybe ah ah
ah hey there we go damn look at that
Vapor chamber love me a vapor chamber okay
we're going to make sure all these are clicked into place look for our little
arrow here line that up with the arrow on the
socket and make sure that the locks are
in their unlocked position then we should be able to just that's it's
locked oh that's it okay next comes something you don't see me do very often
and that is use a screwdriver other than the LTT screwdriver and that's because
these need to be torqued to a specific value that is 6.9 inch PBS
nice it's so cool to think that if I was doing this you know performing
maintenance on the server or upgrading a bad RAM stick our entire operation could
be chugging along as if nothing happened speaking of RAM we've gone with four 96
gig Micron 5600 megat transfer per second registered ECC dims that's a
somewhat unconventional choice because especially in a server giving up half of
the memory channels means that we will be giving up some performance but we
don't really need all of the performance for now and 3 84 gigs is a ton of
capacity for our needs at the moment and of course if anything changes we can
always add more without any downtime to our services the only thing that's
really important here then is making sure that we install our sticks in the
correct slots which is not always super intuitive so make sure to consult your
manual we don't need a GPU for now though we could add one in the future so
that means all that's really left is these NVIDIA connect X6 cards now 100
Gig networking might seem a bit Overkill but because our setup uses high-speed
drives in four servers and we want to be
able to withstand two server failures
anytime we're writing data it has to be simultaneously written to the drives on
at least three machines that ensures we
have three upto-date copies in the event of an unexpected failure now if you were
doing this at home you obviously wouldn't want to spend this kind of
money but the good news is that you can do this with as few as two machines and
if you're not trying to run a highspeed caching server for a 100 people 10 or 25
gig cards are available for a fraction of the price and you can connect them
directly to each other without an expensive switch I mean even one gig
could work for light applications like ensuring that your home automation
system never goes down enough chitchat though let's get on with the demo and
show you what happens if one of these things goes to heaven in a live
environment but not before we get them in the rack and set up specific
specifically here in the lab server room because if you didn't notice earlier the
studio server room is kind of running out of space at least until these
machines are up and running and we can take the machine they're replacing out
let's go grab the service un fortunately the rest of the machines are now magically built off of camera and we can
just slide them in what the hell is going on oh there we go beautiful these
Gigabyte chassis come with nice toolless rails so installing these in our nice
ginormous hammonded rack should be pretty easy yeah look at
that it's getting close I can taste it we just need networking like we
mentioned before 100 Gig but what we didn't mention before is that each is
getting two of them specifically one to each of the network switches in the rack
that way if one of those switches has a problem the servers will stay up and we
even get an added bonus because some fancy Dell magic called VT we get the
throughput of both of these cables so 200 gig to each surface pretty
sick all that's left then is power and like any other good server ipmi which is
a management interface and allows us to control the machines even if they're not
working they have like a hardware problem we can still access them we can
turn them on turn them off it's kind of magic if you have a server that doesn't
have ipmi I don't know I don't even know if that's a server
really there are two main elements to making this setup work clustering the
hypervisor which controls our virtual machines and clustering the storage
which you can skip if you have existing network storage you want to use instead
if you're not not interested in how to set this up you can skip ahead to here
to see what it's like when it's up and running this isn't going to be a perfect step-by-step guide but with the
documentation you can find down in the description you should be able to replicate this setup pretty easily
starting with networking we added both of our 100 Gig ports to a bond created a
bridge and then added a VLAN for three different networks one for our VMS to
use one for cluster communication and one for the storage they can all
technically run on the same network but the cluster needs low latency and the
storage ideally uses jumbo frames so splitting it up like this is best
practice you'll also need to add each node's cluster Network IP address to the
host file on each node with the networking up and running enable the no
subscription repo and disable the Enterprise repo it's not recommended by
the proxmox team for production they want you to pay for the Enterprise repo
which is a bit more stable but the free one is totally fine for a home setup run
any pending updates before proceeding then make sure you have a reliable and
ideally local time server configured on on each of your individual servers as
the clustering software wants the time very closely in sync to stay happy with
that out of the way we can set up our cluster which handles syncing the configuration and management of any
virtual machines between our physical machines and it also orchestrates
migrating or restoring them when a machine goes down creating the cluster
just takes actually a few clicks but you might want to consider the size of your
setup before you continue that's because in order to make sure everything stays
in sync in case of an issue with a machine you need the majority of servers
online and available to be able to say hey I see that one's offline but you
know we're we're still good they call this Quorum if you have an even number
of machines let's say four like we do and each server gets the default single
say or vote the minimum possible majority is then three servers so that
means we can only withstand one going down which is the same amount of
redundancy you'd get if you had three machines cuz you can only lose one to
have two if you only have two computers then you only ever have a majority when
both are online which obviously doesn't work that's not safe but you can screw
it around this by adding a third machine like say a Raspberry Pi to be a
tiebreaker but that's kind of beyond the scope of this video once you're ready
select the cluster Network in the creation menu and then join the other
machines to the cluster once they're in you should be able to see them in the
web goey of any of the machines now on to clustering our storage by default
proxmox is very heavily integrated with SEF and opens Source distributed storage
system that's pretty easy to set up and maintain with that in mind newbie should
start with Seth and you can follow the great tutorial on their Wiki but it
isn't the most performant in a small cluster like this so we're going to be
using something called linore with drbd or distributed replicated block devic
another open source storage system it requires a bit more manual configuration
but they do have a purpose-built tutorial for proxmox and host the files
for free with an optional paid Enterprise version that operates on a
similar model as proxmox itself unlike seph it doesn't handle its own storage
devices so we mirrored our two kokia ssds with ZFS first and then pointed Lin
store to that once it's installed and configured then you can add the
clustered storage to proxmox create a virtual machine with that storage and
it'll automatically be replicated in real time to the number of other nodes
you specify and if you happen to migrate a VM to a server that doesn't have a
copy on it it'll automatically stream the data over the network from one of
those nodes in what they call dis list mode but let's just try it
hey pretty nice right looking good it's
like even cable managed I know right so 200 gig on each of them nice who are you
people and what have you done with our infer team I made one small adjustment
just for you look at the drives they're in the same spot no they're not the top
one's different I hate you so much why would you do that but more importantly
does it work uh yeah obviously okay well here's your Windows desktop obviously he
says what editor a super cut of things
not working here please Jake we have a leak oh God one failure you just
downgraded my Wi-Fi four drives aren't working did you actually break it
anyways you see our Windows right yeah our Windows is running right now on
number four which is the bottom server yes now obviously remoting into the
machine over Wi-Fi okay the video playbacks a little
bit choppy that's not going to affect the type of workload we would normally
be running on something like this like a DNS server or like are we finally doing
active directory we will not not today not today but we can now but this is the
kind of setup that you want for something like ad live playing the video
Let's migrate to number one which is the top one the process will be a little bit
faster but basically what it's doing is
copying the memory like the RAM what's actually in memory and then once it's
done most of it it pauses the operating system for a split second copies the
last tiny little bit and boom that is so cool you're exactly
where you were before because the storage is already there right so in
terms of actual downtime like Interruption to that experience 17
seconds no 270 milliseconds Oh I thought you were pointing the other no 17
seconds is that whole process but oh yeah yeah well that's kind of downtime I
guess no cuz that if there was somebody using this like as a virtual desktop for
instance they would see like a quarter of a second blink and otherwise like
nothing changed I wanted to show a more realistic to us demo sure come hither
here's a Plex server we've got some videos on it and this is on server
number one okay let's play a video now we go and move our Plex server to a
different machine so it's copying the RAM at 2.5 gig gtes a second so it's
like 2.8 gigabytes a second that's pretty good we haven't done any actual
oh it's already done and No Interruption
because video playback like many other applications uses buffers to hide small
interruptions in the service in this case downloading the video in small
chunks a little bit at a time yeah roughly 10c chunks it looks like here
which is plenty to cover that 146
milliseconds of downtime wow you want to try steam download with landan cach I
mean we should yeah why not yep we're CPU bottleneck for sure using you know
80 to 90% of a 24 core Threadripper but
I realized I made a little bit of an oopsy here look you can see the CPU usage we're using % of our eight CPUs
that I assigned to this steam cache we can see our Network traffic's going up
sick except I made this as a container not a VM and the thing with containers
they're great they're a little bit lighter weight better performance but they run within the kernel of the main
system it'll shut down that container and then just reboot on the other
machine right which means it's fine but there will be a longer downtime delay
but way less than hey is that thing
working oh I think the internet is not working somebody should go look at that
yeah trying to figure out what's going on fixing the machine getting the machine back going so cool you're
talking about the matter of a couple minutes maybe yeah now for the most
impressive demo yet the unexpected
migration which one am I yanking okay so number one has three VMS on it they're
all in the high availability Jake's chain ah what means oh I get it okay
sorry which one I wasn't even listening to number one number one and we'll see
how how fast it does we're looking at server one from server two so go for
it from my understanding this process
takes a minute or two okay to go oh it's
already detected the no is offline sure is if you're doing scheduled maintenance
you can actually go and just shut off a machine and then it will just be like oh
crap I need to move all those things before I shut off which is a little bit
nicer in this case it has to be like sure that the server is down so all
three of those are yelling at what was this say hello what happened are you
alive what's going on I can hear them
hello what happened are you alive what's going on oh hey hey oh it did something
so in theory it should distribute them
evenly because that's the option that's set right now right in terms of its
workload you mean yeah there is also a mode that does like resour first
checking sure but right now it's going how many VMS are in each one and just
like filling that number so it's even that is so cool okay so what service was
running on that one was that the steam cache so we should go download a game
you could do go do Plex right now too let's go do it let's go do it come on let's go wait we can take this door and
no movie Magic but but also magic virtualization
magic this is flipping awesome and it's going to be an absolute game changer for
the way that we manage our infrastructure and like I said at the beginning I think the the coolest thing
about it is that this type of architecture doesn't even have to run on
the kind of emerald Rapids latest server
technology that Intel and Gigabyte and Micron and NVIDIA all sent over here so
the takeaway for you guys is whether it's for work or whether it's just for
your home automation or your Plex server at home something like this is
absolutely attainable with potentially very little Financial outlay like go buy
some used like eighth gen Intel core processors those are pretty cheap some
cheap ddr4 and you're Off to the Races or if you're doing this more properly
for your business check out Intel Emerald Rapids and their whole line of
Zeon and GPU products down below where
were you pointing down below that's the
description get your mind out of the description