Computer problems are a fact of life. And sometimes the fix is as simple as just turning it off
and turning it back on again, but other times it's not.
And when the system you're talking about is running an air traffic control system,
controlling a bunch of ATMs, or say routing 911 calls,
keeping them up and running can be a matter of life and death.
Now, the stakes aren't nearly as high for us,
but this server here runs multiple apps that we rely on every day,
accelerates our game downloads with Steam caching, and it runs our DNS.
If that service goes down, it breaks literally everyone in the company's internet,
which my boss informs me, isn't great. So how do we make it more reliable?
It's already a server. We build more servers.
And what's really cool about this is everything we're about to show you,
courtesy of Intel, who sponsored this video and sent over their new Emerald Rapid Xeon CPUs
can be done on nearly any computer, even your dad's old Dell.
That is, as long as you have more than one. So if one leaves for cigarettes,
we can still play catch.
More than one Dell, not more than one dad.
Oh. Well, anyways, I'm done. Do you want to check this out?
Yeah, let's have a look. You got your lovely cat picture, your crab rave on that computer.
Watch this. Like I can, yeah, I can interact with this.
Let's just give it a second. Okay. It's going.
Now it's on this computer and like no bamboozle. Here, look, watch.
Whoa, buddy. Watch, watch, watch. Boom, unplugged.
I can just completely interact with this as I normally would.
So what's going on here? What you guys just saw was the programs,
the lovely drawing, the entire operating system,
just teleporting from the computer over here to the one over here.
No trickery. This is possible thanks to the magic of virtualization.
We've talked about it before, but if you're not familiar, virtualization allows you to slice up a single machine
into multiple less powerful virtual machines.
And this setup leverages that technology to allow us to move these virtual machines
between multiple physical computers.
That way if one breaks, another one can immediately take its place.
And the best part is that, well, this all sounds super fancy.
All the software we're using is both open source and free.
And we're going to show you guys how the setup works in a little bit. First, I want to take a look at the servers
we're going to be using for our setup. Gigabyte sent over four of their R163 SG2 AAC1 servers.
These are bare bones. So we're going to have to add a few of our own parts,
but we should be able to build this in what? Like five minutes?
I'd like to see you try. This guy, we're going to add some of our own parts,
starting with a pair of Patriot 480 gig SATA SSDs
that will function as a mirrored boot drive. This kind of per machine redundancy
isn't strictly speaking necessary because we could lose an entire machine
in our configuration without having any issues. But having them in pairs
makes our lives easier in the future potentially. Since if one of them fails, we can just replace it
and then rebuild it from the other one. Then on the other side of the machine,
we're installing two of these Kyoksia CD6 7TB drives
for fast bulk storage. That leaves us six more SATA bays to do nothing with
and two more NVMe bays for potential future expansion.
Moving back, let's get our CPU installed. We're using a Xeon Platinum 8562Y Plus in each node.
These were graciously provided by Intel and with 32 cores, 64 threads
and 4.1 gigahertz max turbo clock speeds. These are going to give us a ton of compute
to share between our virtual machines, all at a modest 300 watt TDP.
We're going to have it and the rest of the parts linked in the video description. Now, I've never installed in this socket before,
so good luck me. Step one is to install the carrier on the CPU
and you can tell which one of the three you're supposed to use by the little marking right there on the CPU IHS.
Line up our little golden triangle with our gigantic gargantuan hole in the whole thing triangle.
Oh, this is adorable. It's got a cute little ARM so you can break the thermal paste seal with the cooler
so you can get the cooler and the CPU separated more easily. Love to see it.
Speaking of thermal paste, we're going to be using a Honeywell PTM7950 pad,
available at lttstore.com. This stuff is absolutely perfect for a server install
because it lasts not forever, but for a very, very long time without maintenance.
Now, you might think, okay, go ahead, put it onto the CPU socket, you'd be wrong.
Instead, I'm going to install it onto the cooler.
I'm going to know how to do that in a sec. So arrow and arrow.
So maybe, ah, ah, ah, hey, there we go.
Damn, look at that vapor chamber.
Love me a vapor chamber. Okay, we're going to make sure all these are clicked into place.
Look for our little arrow here. Line that up with the arrow on the socket
and make sure that the locks are in their unlocked position
then you should be able to just... That's it's locked.
Oh, that's it. Okay, next comes something you don't see me do very often
and that is use a screwdriver other than the LTT screwdriver.
And that's because these need to be torqued to a specific value.
That is 6.9 inch pounds.
Nice. It's so cool to think that if I was doing this, you know, performing maintenance on the server,
upgrading a bad RAM stick, our entire operation could be chugging along
as if nothing happened. Speaking of RAM, we've gone with four 96 gig micron,
5,600 megatransfer per second registered ECC dims.
That's a somewhat unconventional choice because especially in a server,
giving up half of the memory channels means that we will be giving up some performance,
but we don't really need all of the performance for now
and 384 gigs is a ton of capacity for our needs at the moment.
And of course, if anything changes, we can always add more without any downtime to our services.
The only thing that's really important here then is making sure that we install our sticks
in the correct slots, which is not always super intuitive, so make sure to consult the manual.
We don't need a GPU for now, though we could add one in the future.
So that means all that's really left is these NVIDIA ConnectX 6 cards.
Now, 100 gig networking might seem a bit overkill,
but because our setup uses high speed drives in four servers and we want to be able
to withstand two server failures, anytime we're writing data,
it has to be simultaneously written to the drives
on at least three machines. That ensures we have three up-to-date copies
in the event of an unexpected failure. Now, if you were doing this at home,
you obviously wouldn't want to spend this kind of money, but the good news is that you can do this
with as few as two machines. And if you're not trying to run a high speed
caching server for 100 people, 10 or 25 gig cards are available for a fraction of the price
and you can connect them directly to each other without an expensive switch.
I mean, even one gig could work for light applications
like ensuring that your home automation system never goes down.
Enough chitchat though. Let's get on with the demo and show you what happens
if one of these things goes to heaven in a live environment.
But not before we get them in the rack and set up, specifically here in the lab server room,
because if you didn't notice earlier, the studio server room is kind of running out of space,
at least until these machines are up and running and we can take the machine they're replacing out.
Let's go grab the servers. Unfortunately, the rest of the machines are now magically built off of camera
and we can just slide them in. What the hell is going on?
Oh, there we go. Beautiful. These Gigabyte chassis come with nice tool-less rails.
So installing these in our nice ginormous Hammond rack
should be pretty easy. Yeah, look at that. Ooh, it's getting close.
I can taste it. We just need networking. Like we mentioned before, a hundred gig,
but what we didn't mention before is that each is getting two of them,
specifically one to each of the network switches
in the rack, that way if one of those switches has a problem, the servers will stay up
and we even get an added bonus because it's some fancy Dell magic called VLT.
We get the throughput of both of these cables. So 200 gig to each server.
Pretty sick. All that's left then is power
and like any other good server, IPMI, which is a management interface
and allows us to control the machines. Even if they're not working, they have like a hardware problem.
We can still access them. We can turn them on, turn them off. It's kind of magic.
If you have a server that doesn't have IPMI, I don't know.
I don't even know if that's a server really. There are two main elements to making this setup work.
Clustering the hypervisor, which controls our virtual machines
and clustering the storage, which you can skip if you have existing network storage
you wanna use instead. If you're not interested in how to set this up,
you can skip ahead to here to see what it's like when it's up and running.
This isn't gonna be a perfect step-by-step guide, but with the documentation you could find
down in the description, you should be able to replicate this setup pretty easily.
Starting with networking, we added both of our 100 gate ports to a bond,
created a bridge, and then added a VLAN for three different networks.
One for our VMs to use, one for cluster communication,
and one for the storage. They can all technically run on the same network,
but the cluster needs low latency and the storage ideally uses jumbo frames.
So splitting it up like this is best practice. You'll also need to add each node's cluster network IP address
to the host file on each node. With the networking up and running,
enabled in no subscription repo and disable the enterprise repo,
it's not recommended by the Proxmox team for production.
They want you to pay for the enterprise repo, which is a bit more stable,
but the free one is totally fine for a home setup. Run any pending updates before proceeding,
then make sure you have a reliable and ideally local time server configured
on each of your individual servers as the clustering software wants the time
very closely in sync to stay happy. With that out of the way,
we can set up our cluster, which handles syncing the configuration and management of any virtual machines
between our physical machines, and it also orchestrates migrating
or restoring them when a machine goes down. Creating the cluster just takes actually a few clicks,
but you might want to consider the size of your setup before you continue.
That's because in order to make sure everything stays in sync in case of an issue with a machine,
you need the majority of servers online and available to be able to say, hey, I see that one's offline,
but we're still good. They call this quorum.
If you have an even number of machines, let's say four, like we do,
and each server gets the default single say or vote,
the minimum possible majority is then three servers. So that means we can only withstand one going down,
which is the same amount of redundancy you'd get if you had three machines,
because you can only lose one to have two. If you only have two computers,
then you only ever have a majority when both are online,
which obviously doesn't work, that's not safe.
But you can screw it around this by adding a third machine,
like say a Raspberry Pi to be a tiebreaker,
but that's kind of beyond the scope of this video. Once you're ready, select the cluster network
in the creation menu, and then join the other machines to the cluster.
Once they're in, you should be able to see them in the web GUI of any of the machines.
Now on to clustering our storage. By default, Proxmox is very heavily integrated with Cef,
an open source distributed storage system that's pretty easy to set up and maintain.
With that in mind, newbies should start with Cef, and you can follow the great tutorial on their wiki,
but it isn't the most performant in a small cluster like this. So we're gonna be using something called LinStore with DRBD,
or Distributed Replicated Block Devices, another open source storage system.
It requires a bit more manual configuration, but they do have a purpose-built tutorial for Proxmox
and host the files for free with an optional paid enterprise version
that operates on a similar model as Proxmox itself. Unlike Cef, it doesn't handle its own storage devices,
so we mirrored our two Keoxia SSDs with ZFs first,
and then pointed LinStore to that. Once it's installed and configured,
then you can add the clustered storage to Proxmox, create a virtual machine with that storage,
and it'll automatically be replicated in real time to the number of other nodes you specify.
And if you happen to migrate a VM to a server that doesn't have a copy on it,
it'll automatically stream the data over the network from one of those nodes in what they call diskless mode.
But let's just try it. Hey.
Pretty nice, right? Looking good. It's like even cable-managed.
I know, right? So 200 gig on each of them? Nice.
Who are you people, and what have you done with our infrateam? I made one small adjustment just for you.
Look at the drives. They're in the same spot. No, they're not.
The top one's different. I hate you so much. Why would you do that?
But more importantly, does it work? Yeah, obviously.
Okay, well, here's your Windows desktop. Obviously, he says. Well, what?
Editor, a super cut of things not working here, please.
Jake, we have a leak. Oh, God. One failure.
You just downgraded my Wi-Fi. Four drives aren't working?
Did you actually break it? Anyways, you see our Windows, right?
Yeah. Our Windows is running right now on number four,
which is the bottom server. Yes. Now, obviously, remoting into the machine over Wi-Fi.
Okay, the video playback's a little bit choppy. That's not gonna affect the type of workload
you would normally be running on something like this, like a DNS server, or like are we finally
doing Active Directory? We will, not today.
Not today, but we can now. But this is the kind of setup that you want
for something like AD. Live playing the video, let's migrate to number one,
which is the top one. The process will be a little bit faster,
but basically what it's doing is copying the memory,
like the RAM, what's actually in memory. And then once it's done most of it,
it pauses the operating system for a split second, copies the last tiny little bit, and boom.
That is so cool. You're exactly where you were before
because the storage is already there. Right.
So in terms of actual downtime, like interruption to that experience.
17 seconds. No, 270 milliseconds.
Oh, I thought you were pointing at the other thing. No, 17 seconds is that whole process.
Oh yeah, yeah, well that's kind of downtime, I guess.
No, because if there was somebody using this like as a virtual desktop, for instance.
They would see like a quarter of a second blink,
and otherwise like nothing changed. I wanted to show a more realistic to us demo.
Sure. Come hither. Here's a Plex server.
We've got some videos on it, and this is on server number one.
Okay. Let's play a video. Now we go and move our Plex server to a different machine.
So it's copying the RAM at 2.5 gigabytes a second.
So that's like 2.8 gigabytes a second, that's pretty good. We haven't done any actual,
oh, it's already done. And no interruption, because video playback,
like many other applications, uses buffers to hide small interruptions in the service.
In this case, downloading the video in small chunks a little bit at a time.
Yeah, roughly 10 second chunks it looks like here, which is plenty to cover that 146 milliseconds of downtime.
Wow. You want to try steam download with Lancash?
I mean, we should? Yeah, why not? Yep, we're CPU bottlenecks for sure.
Using 80 to 90% of a 24-core threadripper.
But I realized I made a little bit of an oopsie here. Like you can see the CPU usage,
we're using 4% of our eight CPUs that I assigned to this steam cache.
We can see our network traffic's going up. Sick.
Except I made this as a container, not a VM.
And the thing with containers, they're great. They're a little bit lighter weight, better performance,
but they run within the kernel of the main system.
It'll shut down that container and then just reboot on the other machine. Right, which means it's fine,
but there'll be a longer downtime delay. But way less than, hey, is that thing working?
Oh, I think the internet's not working. Somebody should go look at that. Yeah.
Trying to figure out what's going on, fixing the machine, getting the machine back going. So cool.
You're talking about the matter of a couple minutes maybe. Yeah. Now for the most impressive demo yet,
the unexpected migration. Which one am I yanking?
Okay, so number one has three VMs on it. They're all in the high availability.
Jake's chain. Ah. What?
It means teasing. Oh, I get it. Okay, sorry.
Which one? I wasn't even listening to you. Number one. Number one.
And we'll see how fast it does. We're looking at server one from server two.
So go for it. Ah.
From my understanding, this process takes a minute or two.
Okay. Let's go, let's already detected the notice offline.
Sure is. If you're doing scheduled maintenance, you can actually go and just shut off a machine.
And then it will just be like, oh crap, I need to move all those things before I shut off.
Which is a little bit nicer. In this case, it has to be like, sure,
the server is down. So all three of those are yelling at what was this.
Say, hello? What happened? Are you alive? What's going on?
I can hear them. Hello? What happened? Are you alive?
What's going on? Oh, it did something. So in theory, it should distribute them evenly
because that's the option that's set right now. Right.
In terms of its workload, you mean? Yeah. There is also a mode that does like resource checking.
Sure. But right now it's going, how many VMs are in each one and just like filling that number so it's even.
That is so cool. Okay, so what service was running on that one?
Was that the steam cache? So we should go download a game. You could go do Plex right now too.
Let's go do it. Let's go do it. Come on, let's go. And no movie magic, but also magic, virtualization magic.
This is flipping awesome. And it's going to be an absolute game changer
for the way that we manage our infrastructure. And like I said at the beginning,
I think the coolest thing about it is that this type of architecture doesn't even have to run
on the kind of Emerald Rapids latest server technology
that Intel and Gigabyte and Micron and NVIDIA
all sent over here. So the takeaway for you guys is whether it's for work
or whether it's just for your home automation or your Plex server at home,
something like this is absolutely attainable with potentially very little financial outlay.
Go buy some used like eighth gen Intel core processors.
Those are pretty cheap. Some cheap DDR4 and you're off to the races.
Or if you're doing this more properly for your business, check out Intel Emerald Rapids
and their whole line of Xeon and GPU products down below.
Where were you pointing? Down below, that's the description.
Get your mind out of the description.