1
00:00:00,160 --> 00:00:06,080
June the 14th of this year was probably the worst day of my life so far. The

2
00:00:04,560 --> 00:00:10,240
previous evening, I had found out that my sister passed away. I had spent the

3
00:00:08,080 --> 00:00:15,280
day dealing with the family drama and grief that comes with an unexpected loss

4
00:00:12,559 --> 00:00:20,880
like that. I made my way through one of the hardest WAN shows that I've ever

5
00:00:17,520 --> 00:00:24,240
done. Um, just focusing on the routine.

6
00:00:20,880 --> 00:00:26,800
And when I came home, there was an

7
00:00:24,240 --> 00:00:32,800
unexplained wine that I couldn't really nail down the source of. I made my way

8
00:00:29,519 --> 00:00:35,280
downstairs and I found out that the pump

9
00:00:32,800 --> 00:00:41,040
that runs the cooling system for my server rack here was running dry. That

10
00:00:38,160 --> 00:00:46,800
means one of two things. The reservoir is leaking or the systems are leaking.

11
00:00:44,160 --> 00:00:51,360
And I'll let this water mark on the floor answer the question for you. So

12
00:00:49,680 --> 00:00:55,039
instead of eating, I began the painstaking process of tearing

13
00:00:53,120 --> 00:00:58,960
everything out of the rack as safely as possible. I made my way all the way

14
00:00:57,120 --> 00:01:03,920
through both storage and utility servers, my main and my backup, mine and

15
00:01:01,600 --> 00:01:10,320
Yvon's office machines, and all three of my kids gaming rigs until I found this,

16
00:01:07,119 --> 00:01:13,760
the culprit, a completely corroded off

17
00:01:10,320 --> 00:01:16,400
fitting that dumped gallons of water on

18
00:01:13,760 --> 00:01:21,200
everything underneath it. How could this happen? I mean, that's galvanic

19
00:01:18,960 --> 00:01:27,680
corrosion, plain as the nose on my face. But this is a loop that has nothing but

20
00:01:23,439 --> 00:01:30,159
copper, brass, plastic. What the is

21
00:01:27,680 --> 00:01:34,640
going on here? I've had to sit and wonder from then until now so that I

22
00:01:32,720 --> 00:01:39,040
could get someone over here to point a camera at this because I'm sure that you

23
00:01:36,880 --> 00:01:43,360
guys want to know the answer as badly as I do. And if I'm going to lose a bunch

24
00:01:41,040 --> 00:01:46,799
of server equipment, well, I might as well at least help pay for it with this

25
00:01:44,799 --> 00:01:51,280
message from our sponsor. If I had time to untangle cables, would my hair look

26
00:01:48,960 --> 00:01:55,920
like this? Why not switch to magnetic cables from our sponsor Mag Tame

27
00:01:52,960 --> 00:02:00,640
instead? They pull apart effortlessly, roll up perfectly, and stick to any

28
00:01:57,920 --> 00:02:04,079
magnetic surface. No stress, no mess, no worries.

29
00:02:02,799 --> 00:02:07,600
We'll have them linked in the video description. Now, I know what you guys

30
00:02:06,000 --> 00:02:11,280
are thinking. Ah, it's Lionus. He's always doing janky projects, and I'm

31
00:02:09,520 --> 00:02:15,680
sure you overlook something. I mean, galvanic corrosion can't happen by

32
00:02:13,120 --> 00:02:20,560
accident. That's science. But listen to this. You want to talk

33
00:02:17,360 --> 00:02:23,040
science? How about the scientific method

34
00:02:20,560 --> 00:02:27,599
for a control system? I grabbed one of the other ones in the loop and look at

35
00:02:25,280 --> 00:02:32,560
those fittings. Those are the same two fittings and they're immaculate. So, if

36
00:02:30,720 --> 00:02:37,200
it was a configuration error in the loop, these should be eaten. And yet,

37
00:02:35,280 --> 00:02:40,480
they're pristine. Our goal today is twofold. Then, my son obviously wants

38
00:02:39,120 --> 00:02:46,000
his gaming rig back up and running, which is why he's helping. But before we

39
00:02:43,280 --> 00:02:49,280
do that, we need to know what the heck happened here so that we can ensure that

40
00:02:47,599 --> 00:02:52,879
this won't happen again. The first step of disassembling these blocks is popping

41
00:02:50,959 --> 00:02:57,840
the stainless steel cover off. It's important to note that this is purely

42
00:02:54,959 --> 00:03:01,599
cosmetic since steel is not a compatible metal with the ones that are typically

43
00:02:59,519 --> 00:03:06,560
used for water cooling like copper, brass, and nickel.

44
00:03:04,480 --> 00:03:10,959
Now, let's go a layer deeper and remove the delin top. Okay, not the cleanest

45
00:03:09,120 --> 00:03:14,879
loop I've ever seen. Why is there so much dust on it?

46
00:03:12,400 --> 00:03:20,080
Well, I can tell what some of this is. These look like little plastic filings

47
00:03:17,440 --> 00:03:24,239
from working with the tubing that we installed. Oh, there's a little metal

48
00:03:22,080 --> 00:03:29,840
shaving. Uh, that could be left over from manufacturing of something.

49
00:03:27,440 --> 00:03:35,440
Radiator, heat exchanger. There's a surprising amount of this that looks

50
00:03:31,280 --> 00:03:38,720
like hair or other organic matter, but

51
00:03:35,440 --> 00:03:41,680
there's iodine in the loop. And what

52
00:03:38,720 --> 00:03:46,560
kind of contradicts that theory is that this tube over here, which is running

53
00:03:43,519 --> 00:03:48,480
water to my system right now, is crystal

54
00:03:46,560 --> 00:03:52,400
clear. Well, maybe one of the other systems will give us some hints as to

55
00:03:50,159 --> 00:03:55,920
what that is. Actually, we don't need to go that far. Around the edges of the

56
00:03:54,239 --> 00:04:00,239
reservoir, there is a little bit of something. Not much, but I think I might

57
00:03:57,680 --> 00:04:05,280
need to up my iodine dosage and maybe throw a filter in there. Either way,

58
00:04:02,879 --> 00:04:10,879
none of that would explain the leak. Let's take apart the other block. Here

59
00:04:07,439 --> 00:04:15,280
we go. What

60
00:04:10,879 --> 00:04:17,680
the crap? What is this?

61
00:04:15,280 --> 00:04:21,759
Ew. Why does it look burned? Kind of does look burned. It's

62
00:04:19,359 --> 00:04:26,479
definitely not. It was underwater. But the kind of oxidation that could occur

63
00:04:23,919 --> 00:04:32,479
when burning, I guess, could also happen when it's being corroded. But the funny

64
00:04:29,040 --> 00:04:34,960
thing is that this is where the metal

65
00:04:32,479 --> 00:04:40,000
and you can actually see metallic elements here. This is where the metal

66
00:04:37,600 --> 00:04:46,080
was being deposited, but there's no metal for it to deposit

67
00:04:42,400 --> 00:04:49,280
onto. Normally in galvanic corrosion,

68
00:04:46,080 --> 00:04:51,759
you have a sacrificial anode and then

69
00:04:49,280 --> 00:04:58,720
you have a a cathode that the metal deposits onto, but it's depositing onto

70
00:04:54,400 --> 00:05:01,040
the delin or the space above it. You can

71
00:04:58,720 --> 00:05:05,280
see that's what was being corroded away, right? Which is why the fitting broke

72
00:05:03,280 --> 00:05:11,199
off and liquid gushed out into the system. But the deposit was so close. It

73
00:05:08,400 --> 00:05:17,560
was right here. the mid plate that fitting crumbled, I

74
00:05:14,240 --> 00:05:17,560
think. Whoa.

75
00:05:18,479 --> 00:05:24,560
Okay. Well, that's something that happened.

76
00:05:22,639 --> 00:05:28,560
Now, EK had a theory when I contacted them, and that was that this stainless

77
00:05:26,400 --> 00:05:34,320
steel cover was somehow participating in the galvanic corrosion process because

78
00:05:31,120 --> 00:05:37,680
it was bridged to the fluid somehow or

79
00:05:34,320 --> 00:05:40,400
something. But I just don't see how it

80
00:05:37,680 --> 00:05:44,479
could connect. None of these screw holes externally

81
00:05:42,240 --> 00:05:47,919
pass through to the inside of the block. And you can clearly see with their

82
00:05:46,160 --> 00:05:51,840
decades of experience designing these things, there's no connection here

83
00:05:50,000 --> 00:05:55,280
either, even if I screw this in all the way. Not going to lie, boy. I was hoping

84
00:05:53,759 --> 00:06:00,080
it would be a little bit more obvious at this point. Can we just fix it and then not care

85
00:05:58,160 --> 00:06:06,479
what happens? We got really lucky this time. We only

86
00:06:02,720 --> 00:06:09,600
lost the big UPS at the bottom. And I

87
00:06:06,479 --> 00:06:11,840
lost one of the rows of NVMe backplane

88
00:06:09,600 --> 00:06:16,240
slots on one of the servers. The other row fully works and I don't need the

89
00:06:13,280 --> 00:06:21,280
second row. Um, also your computer is dead. We won't necessarily get that

90
00:06:18,800 --> 00:06:25,120
lucky next time. There were pools of water on the motherboards of some of the

91
00:06:23,120 --> 00:06:28,560
systems under this one. So, we need to figure out what happened is what I'm

92
00:06:26,400 --> 00:06:32,240
trying to say. Which means we need to start dissecting some more victims.

93
00:06:30,319 --> 00:06:36,479
Actually, wait. Before we look at a different system, we should look at a

94
00:06:34,160 --> 00:06:41,360
second block within this system to see if there's something unique about this

95
00:06:38,880 --> 00:06:44,479
environment. These fuel line hoses are a bit of a nightmare to get off. You can

96
00:06:42,720 --> 00:06:47,039
see the reinforcing string that makes even cutting them a bit of a chore.

97
00:06:46,000 --> 00:06:52,319
Okay, what do you think we're going to find? A lot more corrosion and metal.

98
00:06:50,720 --> 00:06:55,680
Oh, crud. I called it. Cuz this means your computer's not

99
00:06:54,639 --> 00:07:01,919
getting fixed today. Oh, still happy? No. That might do it for EK's theory

100
00:07:00,000 --> 00:07:06,479
about that top stainless steel thing. Maybe this is a more widespread issue.

101
00:07:04,319 --> 00:07:09,919
Now we really need to start cracking open more systems and seeing what's up.

102
00:07:08,400 --> 00:07:15,440
One rail came out. Hold on. You got to come straight out.

103
00:07:12,720 --> 00:07:22,880
There you go. Oh my goodness. The plot thickens. The second fitting on this GPU

104
00:07:20,240 --> 00:07:28,479
is pristine. What is going on here? Could it be the

105
00:07:25,680 --> 00:07:34,080
worm drive clamps? They're not touching though. And this one hasn't been moved.

106
00:07:31,280 --> 00:07:40,160
I unscrewed it by just twisting up the tube. This is exactly as it was.

107
00:07:37,840 --> 00:07:46,080
Bro, nasty. What this means to me is every one of

108
00:07:43,440 --> 00:07:50,880
these systems has to have every fitting popped to check if it's corroding. A

109
00:07:48,960 --> 00:07:56,720
couple of the screws going into the top of this GPU block are rusted. This is

110
00:07:53,919 --> 00:08:01,440
not a damp environment. So, the only moisture they should have access to is

111
00:07:59,440 --> 00:08:05,440
from in here. I didn't know which holes they came out

112
00:08:03,520 --> 00:08:09,280
of, though, unfortunately. I don't think your computer's going to be working for

113
00:08:07,120 --> 00:08:11,599
a while. Hey, thank goodness it's summer and there's lots else to do, right?

114
00:08:11,120 --> 00:08:16,319
Yeah. I wouldn't let my dad get away with that, but hey, thanks.

115
00:08:14,639 --> 00:08:22,319
What do we have to do to get it working again? It relies on a cooling system that it

116
00:08:19,280 --> 00:08:24,879
shares with six other systems.

117
00:08:22,319 --> 00:08:28,960
All six systems need to be checked to ensure that they don't have corrosion in

118
00:08:26,639 --> 00:08:34,560
them because if one of them corrods, it takes down the whole thing. So all the

119
00:08:31,919 --> 00:08:39,919
systems need to be taken right down to the disassembly of the bare parts.

120
00:08:36,719 --> 00:08:41,680
Okay, so like 2 months.

121
00:08:39,919 --> 00:08:45,839
I'm the kind of dad who believes in honoring my promises. So I am not going

122
00:08:44,320 --> 00:08:50,880
to promise anything. Okay, this is shaping up to look like it is

123
00:08:48,560 --> 00:08:55,920
going to be an absolutely terrifying amount of work. like better off starting

124
00:08:54,000 --> 00:08:59,920
over amount of work here. So then shouldn't we just start over?

125
00:08:58,080 --> 00:09:04,080
Well, I mean it'd be kind of wasteful, don't you think? And this is your gaming

126
00:09:01,839 --> 00:09:07,120
rig. This has seen you through many battles. Well, you just going to leave

127
00:09:05,680 --> 00:09:12,000
it behind on the field? Yeah. Now, it's clear there are issues with

128
00:09:10,160 --> 00:09:15,760
the loop, not caused by anything to do with block design. Uh there is some

129
00:09:14,240 --> 00:09:21,680
organic matter in here. You can really tell now that it's still wet. But I

130
00:09:18,720 --> 00:09:26,160
reiterate that would not cause a leak. So then it is EK's fault.

131
00:09:23,680 --> 00:09:29,440
Well, we can't say that definitively. I think what we're going to do is we're

132
00:09:27,760 --> 00:09:35,040
going to investigate one or two of the other systems, send them our findings,

133
00:09:33,040 --> 00:09:40,880
and then we'll go from there. The worst corroded one in both cases was nearest

134
00:09:38,240 --> 00:09:46,240
the acceleration or impingement plate. But on the GPU, the deposition didn't

135
00:09:43,360 --> 00:09:52,080
take place on the plate itself. It took place here before it. You're probably

136
00:09:49,760 --> 00:09:58,080
wondering what to do next. What we need is a corrosion check on this, this,

137
00:09:54,959 --> 00:10:00,000
this, and this. You're going to start by

138
00:09:58,080 --> 00:10:02,959
blowing all the water out of the system. How do I do that?

139
00:10:01,120 --> 00:10:08,160
I'm glad you asked. When they aren't coupled, these quick connects stop the

140
00:10:04,880 --> 00:10:11,920
flow of water and air. So, there's no

141
00:10:08,160 --> 00:10:16,590
way for us to blow out the system unless

142
00:10:11,920 --> 00:10:18,800
we put a female fitting on them. Now

143
00:10:18,800 --> 00:10:24,880
she should just kind of drain.

144
00:10:22,000 --> 00:10:29,279
Oi. Hey, there we go. But that's slow. And we can accelerate the process using

145
00:10:27,760 --> 00:10:34,079
Under normal circumstances, I'd be giving you guys a bunch of lectures about air compressors and computers.

146
00:10:32,079 --> 00:10:38,800
There are concerns with moisture being blown onto the components, uh, static

147
00:10:36,399 --> 00:10:41,600
buildup, but in this case, we're just blowing out tubing, so we don't need to

148
00:10:40,240 --> 00:10:46,640
worry about any of that. What you're going to do is you're going to take this. You're going to hold it as tight

149
00:10:44,000 --> 00:10:50,870
up against the in one as you can. And all the water is going to blow out the

150
00:10:48,160 --> 00:10:53,440
other side when you pull the trigger.

151
00:10:53,440 --> 00:11:00,640
Now, we don't need to worry about water leaking all over our system as we

152
00:10:57,120 --> 00:11:03,040
disconnect our hoses. Realistically,

153
00:11:00,640 --> 00:11:07,760
the system is not alive anymore, but there could still be components that are

154
00:11:04,640 --> 00:11:11,200
good. Remember that I said that your SSD

155
00:11:07,760 --> 00:11:12,959
might be okay. Well, that's because it's

156
00:11:11,200 --> 00:11:17,519
over here on the other side from where the leak was. So, as long as the water

157
00:11:15,120 --> 00:11:22,079
didn't come up that high, or as long as there wasn't some kind of malfunction of

158
00:11:19,200 --> 00:11:26,079
the motherboard that caused a cascading malfunction to your SSD, you might still

159
00:11:24,560 --> 00:11:30,959
have all your games and data and all your saves and all that kind of stuff. Get this hex head and pull the block

160
00:11:29,440 --> 00:11:36,320
off. Clean the thermal goop off the bottom of the block while you're handling it so you don't make a mess.

161
00:11:34,320 --> 00:11:40,880
Okay. Next thing you want to do is get these fittings off. Yeah, you might need

162
00:11:38,880 --> 00:11:44,399
to take off these worm drives first. So, you need that flathead. There's certain

163
00:11:42,480 --> 00:11:48,240
things that I really hope are not dead. I mean, a motherboard's a relatively

164
00:11:45,920 --> 00:11:52,480
simple replacement. I even have a cold spare of these just in case, you know,

165
00:11:49,839 --> 00:11:57,440
we experienced a failure, but like these PCIe Gen 4 extensions and stuff like

166
00:11:54,959 --> 00:12:00,800
that, it'll just probably take a little while

167
00:11:58,800 --> 00:12:05,279
to get new ones. Then again, I've made no commitments about when this will be

168
00:12:02,240 --> 00:12:08,240
done. So, now we're just going to play

169
00:12:05,279 --> 00:12:12,240
Twisty Twist until this comes out. Not the world's most fun game.

170
00:12:10,160 --> 00:12:15,200
Twisty twist is very effective. How'd you get to be so sarcastic?

171
00:12:14,079 --> 00:12:19,839
Bad parenting, I guess. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Speaking of bad

172
00:12:17,839 --> 00:12:25,720
parenting, I think I'm just going to have to give you a really sharp knife to

173
00:12:21,519 --> 00:12:25,720
cut the tubing off. Thank you.

174
00:12:27,760 --> 00:12:35,440
Perfect. Okay. So, now we just put it back.

175
00:12:32,880 --> 00:12:38,560
Well, I don't know, man. Like this is raising more questions than it's

176
00:12:36,959 --> 00:12:42,800
answering. I think what we need to do is have a

177
00:12:40,160 --> 00:12:46,240
look at the fittings on the GPU now. So this one's perfect on both.

178
00:12:44,399 --> 00:12:51,040
Yeah, maybe this is a system by system thing

179
00:12:48,320 --> 00:12:56,720
then the fittings are perfect on the GPU on the second system. Remember our first

180
00:12:53,519 --> 00:12:58,560
control system as well. Got two more to

181
00:12:56,720 --> 00:13:04,399
go though. Let's see what's shaking in those trays. Interesting.

182
00:13:01,519 --> 00:13:09,200
So yours is the bad one. Oh man. I guess this is it's kind of good news, but it

183
00:13:06,240 --> 00:13:13,760
also just deepens the mystery. And there you have it. Looking at the last two

184
00:13:11,040 --> 00:13:19,680
systems, the ones above the leak. Flawless. Flawless. Flawless. Flawless.

185
00:13:16,959 --> 00:13:24,560
Only one system experienced corrosion out of the five. And it was on two

186
00:13:22,399 --> 00:13:29,040
different blocks. For real. Chemistry majors, get out of

187
00:13:27,040 --> 00:13:34,320
your armchairs. Or rather, get in your armchairs. What happened here? So,

188
00:13:32,639 --> 00:13:39,200
you're probably wondering, "Well, what now then?" Well, now I'm going to test

189
00:13:37,440 --> 00:13:44,639
the two systems that were under number three and see if they still work, which

190
00:13:41,440 --> 00:13:47,440
means maybe. Ooh, capsule.

191
00:13:44,639 --> 00:13:54,720
White light. Good. Hey, desktop. Better. As unbelievable as

192
00:13:50,959 --> 00:13:57,760
it might seem, the only system casualty

193
00:13:54,720 --> 00:14:01,120
is my son's gaming rig. Almost

194
00:13:57,760 --> 00:14:03,360
everything else survived.

195
00:14:01,120 --> 00:14:09,440
Is it possible that this system still functions? I kind

196
00:14:07,040 --> 00:14:14,560
of doubt it. It gushed water over the motherboard like right over the VRM. But

197
00:14:13,040 --> 00:14:18,000
everything else survived. What do you think, Andrew? Is she just going to like post?

198
00:14:17,040 --> 00:14:22,360
I'm off. I'm not. I think this one's dead.

199
00:14:27,360 --> 00:14:33,120
Now, modern electronics got a lot of

200
00:14:30,399 --> 00:14:39,760
protections. No ShortCircuit, over current, over voltage, but dump gallons

201
00:14:36,959 --> 00:14:44,079
of flipping water over it protection. Obviously, I can't deploy it the way it

202
00:14:41,839 --> 00:14:48,000
is. It's got its blocks and fittings all corroded to heck and back. So, there's

203
00:14:46,320 --> 00:14:52,240
definitely going to be a follow-up where I hear from some of you folks. I go back

204
00:14:50,240 --> 00:14:55,680
to the manufacturer. We figure out exactly what's going on. We rebuild this

205
00:14:53,839 --> 00:14:59,519
mess. But I'd say this is about the best outcome I could have possibly hoped for.

206
00:14:58,000 --> 00:15:02,800
Just like I couldn't hope for a better sponsor than our sponsor.

207
00:15:01,360 --> 00:15:05,920
Next aiggo. Have you ever hopped on a meeting and thought your screen was

208
00:15:04,480 --> 00:15:10,959
frozen because the person you're talking to who's picture is moving at 5 FPS? How

209
00:15:08,959 --> 00:15:15,360
embarrassing? Enhance your work from home setup with the Nexo Hello Cam Pro.

210
00:15:13,680 --> 00:15:19,519
It features Windows Hello facial recognition for an effortless login

211
00:15:17,360 --> 00:15:23,199
experience. And it also has an automated privacy cover that closes when the

212
00:15:21,279 --> 00:15:28,000
camera's not active. Plus, you'll look great at your next meeting thanks to the

213
00:15:24,480 --> 00:15:29,680
1080p 60fps resolution and refresh rate.

214
00:15:28,000 --> 00:15:32,880
Learn more using the link in the description. If you enjoyed this video

215
00:15:31,519 --> 00:15:36,959
and you like this kind of raw troubleshooting slashbuilding with the

216
00:15:34,800 --> 00:15:41,839
kids, then maybe check out uh fixing the gaming minivan. They were helpful.