June the 14th of this year was probably the worst day of my life so far. The
previous evening, I had found out that my sister passed away. I had spent the
day dealing with the family drama and grief that comes with an unexpected loss
like that. I made my way through one of the hardest WAN shows that I've ever
done. Um, just focusing on the routine.
And when I came home, there was an
unexplained wine that I couldn't really nail down the source of. I made my way
downstairs and I found out that the pump
that runs the cooling system for my server rack here was running dry. That
means one of two things. The reservoir is leaking or the systems are leaking.
And I'll let this water mark on the floor answer the question for you. So
instead of eating, I began the painstaking process of tearing
everything out of the rack as safely as possible. I made my way all the way
through both storage and utility servers, my main and my backup, mine and
Yvon's office machines, and all three of my kids gaming rigs until I found this,
the culprit, a completely corroded off
fitting that dumped gallons of water on
everything underneath it. How could this happen? I mean, that's galvanic
corrosion, plain as the nose on my face. But this is a loop that has nothing but
copper, brass, plastic. What the is
going on here? I've had to sit and wonder from then until now so that I
could get someone over here to point a camera at this because I'm sure that you
guys want to know the answer as badly as I do. And if I'm going to lose a bunch
of server equipment, well, I might as well at least help pay for it with this
message from our sponsor. If I had time to untangle cables, would my hair look
like this? Why not switch to magnetic cables from our sponsor Mag Tame
instead? They pull apart effortlessly, roll up perfectly, and stick to any
magnetic surface. No stress, no mess, no worries.
We'll have them linked in the video description. Now, I know what you guys
are thinking. Ah, it's Lionus. He's always doing janky projects, and I'm
sure you overlook something. I mean, galvanic corrosion can't happen by
accident. That's science. But listen to this. You want to talk
science? How about the scientific method
for a control system? I grabbed one of the other ones in the loop and look at
those fittings. Those are the same two fittings and they're immaculate. So, if
it was a configuration error in the loop, these should be eaten. And yet,
they're pristine. Our goal today is twofold. Then, my son obviously wants
his gaming rig back up and running, which is why he's helping. But before we
do that, we need to know what the heck happened here so that we can ensure that
this won't happen again. The first step of disassembling these blocks is popping
the stainless steel cover off. It's important to note that this is purely
cosmetic since steel is not a compatible metal with the ones that are typically
used for water cooling like copper, brass, and nickel.
Now, let's go a layer deeper and remove the delin top. Okay, not the cleanest
loop I've ever seen. Why is there so much dust on it?
Well, I can tell what some of this is. These look like little plastic filings
from working with the tubing that we installed. Oh, there's a little metal
shaving. Uh, that could be left over from manufacturing of something.
Radiator, heat exchanger. There's a surprising amount of this that looks
like hair or other organic matter, but
there's iodine in the loop. And what
kind of contradicts that theory is that this tube over here, which is running
water to my system right now, is crystal
clear. Well, maybe one of the other systems will give us some hints as to
what that is. Actually, we don't need to go that far. Around the edges of the
reservoir, there is a little bit of something. Not much, but I think I might
need to up my iodine dosage and maybe throw a filter in there. Either way,
none of that would explain the leak. Let's take apart the other block. Here
we go. What
the crap? What is this?
Ew. Why does it look burned? Kind of does look burned. It's
definitely not. It was underwater. But the kind of oxidation that could occur
when burning, I guess, could also happen when it's being corroded. But the funny
thing is that this is where the metal
and you can actually see metallic elements here. This is where the metal
was being deposited, but there's no metal for it to deposit
onto. Normally in galvanic corrosion,
you have a sacrificial anode and then
you have a a cathode that the metal deposits onto, but it's depositing onto
the delin or the space above it. You can
see that's what was being corroded away, right? Which is why the fitting broke
off and liquid gushed out into the system. But the deposit was so close. It
was right here. the mid plate that fitting crumbled, I
think. Whoa.
Okay. Well, that's something that happened.
Now, EK had a theory when I contacted them, and that was that this stainless
steel cover was somehow participating in the galvanic corrosion process because
it was bridged to the fluid somehow or
something. But I just don't see how it
could connect. None of these screw holes externally
pass through to the inside of the block. And you can clearly see with their
decades of experience designing these things, there's no connection here
either, even if I screw this in all the way. Not going to lie, boy. I was hoping
it would be a little bit more obvious at this point. Can we just fix it and then not care
what happens? We got really lucky this time. We only
lost the big UPS at the bottom. And I
lost one of the rows of NVMe backplane
slots on one of the servers. The other row fully works and I don't need the
second row. Um, also your computer is dead. We won't necessarily get that
lucky next time. There were pools of water on the motherboards of some of the
systems under this one. So, we need to figure out what happened is what I'm
trying to say. Which means we need to start dissecting some more victims.
Actually, wait. Before we look at a different system, we should look at a
second block within this system to see if there's something unique about this
environment. These fuel line hoses are a bit of a nightmare to get off. You can
see the reinforcing string that makes even cutting them a bit of a chore.
Okay, what do you think we're going to find? A lot more corrosion and metal.
Oh, crud. I called it. Cuz this means your computer's not
getting fixed today. Oh, still happy? No. That might do it for EK's theory
about that top stainless steel thing. Maybe this is a more widespread issue.
Now we really need to start cracking open more systems and seeing what's up.
One rail came out. Hold on. You got to come straight out.
There you go. Oh my goodness. The plot thickens. The second fitting on this GPU
is pristine. What is going on here? Could it be the
worm drive clamps? They're not touching though. And this one hasn't been moved.
I unscrewed it by just twisting up the tube. This is exactly as it was.
Bro, nasty. What this means to me is every one of
these systems has to have every fitting popped to check if it's corroding. A
couple of the screws going into the top of this GPU block are rusted. This is
not a damp environment. So, the only moisture they should have access to is
from in here. I didn't know which holes they came out
of, though, unfortunately. I don't think your computer's going to be working for
a while. Hey, thank goodness it's summer and there's lots else to do, right?
Yeah. I wouldn't let my dad get away with that, but hey, thanks.
What do we have to do to get it working again? It relies on a cooling system that it
shares with six other systems.
All six systems need to be checked to ensure that they don't have corrosion in
them because if one of them corrods, it takes down the whole thing. So all the
systems need to be taken right down to the disassembly of the bare parts.
Okay, so like 2 months.
I'm the kind of dad who believes in honoring my promises. So I am not going
to promise anything. Okay, this is shaping up to look like it is
going to be an absolutely terrifying amount of work. like better off starting
over amount of work here. So then shouldn't we just start over?
Well, I mean it'd be kind of wasteful, don't you think? And this is your gaming
rig. This has seen you through many battles. Well, you just going to leave
it behind on the field? Yeah. Now, it's clear there are issues with
the loop, not caused by anything to do with block design. Uh there is some
organic matter in here. You can really tell now that it's still wet. But I
reiterate that would not cause a leak. So then it is EK's fault.
Well, we can't say that definitively. I think what we're going to do is we're
going to investigate one or two of the other systems, send them our findings,
and then we'll go from there. The worst corroded one in both cases was nearest
the acceleration or impingement plate. But on the GPU, the deposition didn't
take place on the plate itself. It took place here before it. You're probably
wondering what to do next. What we need is a corrosion check on this, this,
this, and this. You're going to start by
blowing all the water out of the system. How do I do that?
I'm glad you asked. When they aren't coupled, these quick connects stop the
flow of water and air. So, there's no
way for us to blow out the system unless
we put a female fitting on them. Now
she should just kind of drain.
Oi. Hey, there we go. But that's slow. And we can accelerate the process using
Under normal circumstances, I'd be giving you guys a bunch of lectures about air compressors and computers.
There are concerns with moisture being blown onto the components, uh, static
buildup, but in this case, we're just blowing out tubing, so we don't need to
worry about any of that. What you're going to do is you're going to take this. You're going to hold it as tight
up against the in one as you can. And all the water is going to blow out the
other side when you pull the trigger.
Now, we don't need to worry about water leaking all over our system as we
disconnect our hoses. Realistically,
the system is not alive anymore, but there could still be components that are
good. Remember that I said that your SSD
might be okay. Well, that's because it's
over here on the other side from where the leak was. So, as long as the water
didn't come up that high, or as long as there wasn't some kind of malfunction of
the motherboard that caused a cascading malfunction to your SSD, you might still
have all your games and data and all your saves and all that kind of stuff. Get this hex head and pull the block
off. Clean the thermal goop off the bottom of the block while you're handling it so you don't make a mess.
Okay. Next thing you want to do is get these fittings off. Yeah, you might need
to take off these worm drives first. So, you need that flathead. There's certain
things that I really hope are not dead. I mean, a motherboard's a relatively
simple replacement. I even have a cold spare of these just in case, you know,
we experienced a failure, but like these PCIe Gen 4 extensions and stuff like
that, it'll just probably take a little while
to get new ones. Then again, I've made no commitments about when this will be
done. So, now we're just going to play
Twisty Twist until this comes out. Not the world's most fun game.
Twisty twist is very effective. How'd you get to be so sarcastic?
Bad parenting, I guess. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Speaking of bad
parenting, I think I'm just going to have to give you a really sharp knife to
cut the tubing off. Thank you.
Perfect. Okay. So, now we just put it back.
Well, I don't know, man. Like this is raising more questions than it's
answering. I think what we need to do is have a
look at the fittings on the GPU now. So this one's perfect on both.
Yeah, maybe this is a system by system thing
then the fittings are perfect on the GPU on the second system. Remember our first
control system as well. Got two more to
go though. Let's see what's shaking in those trays. Interesting.
So yours is the bad one. Oh man. I guess this is it's kind of good news, but it
also just deepens the mystery. And there you have it. Looking at the last two
systems, the ones above the leak. Flawless. Flawless. Flawless. Flawless.
Only one system experienced corrosion out of the five. And it was on two
different blocks. For real. Chemistry majors, get out of
your armchairs. Or rather, get in your armchairs. What happened here? So,
you're probably wondering, "Well, what now then?" Well, now I'm going to test
the two systems that were under number three and see if they still work, which
means maybe. Ooh, capsule.
White light. Good. Hey, desktop. Better. As unbelievable as
it might seem, the only system casualty
is my son's gaming rig. Almost
everything else survived.
Is it possible that this system still functions? I kind
of doubt it. It gushed water over the motherboard like right over the VRM. But
everything else survived. What do you think, Andrew? Is she just going to like post?
I'm off. I'm not. I think this one's dead.
Now, modern electronics got a lot of
protections. No ShortCircuit, over current, over voltage, but dump gallons
of flipping water over it protection. Obviously, I can't deploy it the way it
is. It's got its blocks and fittings all corroded to heck and back. So, there's
definitely going to be a follow-up where I hear from some of you folks. I go back
to the manufacturer. We figure out exactly what's going on. We rebuild this
mess. But I'd say this is about the best outcome I could have possibly hoped for.
Just like I couldn't hope for a better sponsor than our sponsor.
Next aiggo. Have you ever hopped on a meeting and thought your screen was
frozen because the person you're talking to who's picture is moving at 5 FPS? How
embarrassing? Enhance your work from home setup with the Nexo Hello Cam Pro.
It features Windows Hello facial recognition for an effortless login
experience. And it also has an automated privacy cover that closes when the
camera's not active. Plus, you'll look great at your next meeting thanks to the
1080p 60fps resolution and refresh rate.
Learn more using the link in the description. If you enjoyed this video
and you like this kind of raw troubleshooting slashbuilding with the
kids, then maybe check out uh fixing the gaming minivan. They were helpful.