WEBVTT 00:00:00.160 --> 00:00:06.080 June the 14th of this year was probably the worst day of my life so far. The 00:00:04.560 --> 00:00:10.240 previous evening, I had found out that my sister passed away. I had spent the 00:00:08.080 --> 00:00:15.280 day dealing with the family drama and grief that comes with an unexpected loss 00:00:12.559 --> 00:00:20.880 like that. I made my way through one of the hardest WAN shows that I've ever 00:00:17.520 --> 00:00:24.240 done. Um, just focusing on the routine. 00:00:20.880 --> 00:00:26.800 And when I came home, there was an 00:00:24.240 --> 00:00:32.800 unexplained wine that I couldn't really nail down the source of. I made my way 00:00:29.519 --> 00:00:35.280 downstairs and I found out that the pump 00:00:32.800 --> 00:00:41.040 that runs the cooling system for my server rack here was running dry. That 00:00:38.160 --> 00:00:46.800 means one of two things. The reservoir is leaking or the systems are leaking. 00:00:44.160 --> 00:00:51.360 And I'll let this water mark on the floor answer the question for you. So 00:00:49.680 --> 00:00:55.039 instead of eating, I began the painstaking process of tearing 00:00:53.120 --> 00:00:58.960 everything out of the rack as safely as possible. I made my way all the way 00:00:57.120 --> 00:01:03.920 through both storage and utility servers, my main and my backup, mine and 00:01:01.600 --> 00:01:10.320 Yvon's office machines, and all three of my kids gaming rigs until I found this, 00:01:07.119 --> 00:01:13.760 the culprit, a completely corroded off 00:01:10.320 --> 00:01:16.400 fitting that dumped gallons of water on 00:01:13.760 --> 00:01:21.200 everything underneath it. How could this happen? I mean, that's galvanic 00:01:18.960 --> 00:01:27.680 corrosion, plain as the nose on my face. But this is a loop that has nothing but 00:01:23.439 --> 00:01:30.159 copper, brass, plastic. What the is 00:01:27.680 --> 00:01:34.640 going on here? I've had to sit and wonder from then until now so that I 00:01:32.720 --> 00:01:39.040 could get someone over here to point a camera at this because I'm sure that you 00:01:36.880 --> 00:01:43.360 guys want to know the answer as badly as I do. And if I'm going to lose a bunch 00:01:41.040 --> 00:01:46.799 of server equipment, well, I might as well at least help pay for it with this 00:01:44.799 --> 00:01:51.280 message from our sponsor. If I had time to untangle cables, would my hair look 00:01:48.960 --> 00:01:55.920 like this? Why not switch to magnetic cables from our sponsor Mag Tame 00:01:52.960 --> 00:02:00.640 instead? They pull apart effortlessly, roll up perfectly, and stick to any 00:01:57.920 --> 00:02:04.079 magnetic surface. No stress, no mess, no worries. 00:02:02.799 --> 00:02:07.600 We'll have them linked in the video description. Now, I know what you guys 00:02:06.000 --> 00:02:11.280 are thinking. Ah, it's Lionus. He's always doing janky projects, and I'm 00:02:09.520 --> 00:02:15.680 sure you overlook something. I mean, galvanic corrosion can't happen by 00:02:13.120 --> 00:02:20.560 accident. That's science. But listen to this. You want to talk 00:02:17.360 --> 00:02:23.040 science? How about the scientific method 00:02:20.560 --> 00:02:27.599 for a control system? I grabbed one of the other ones in the loop and look at 00:02:25.280 --> 00:02:32.560 those fittings. Those are the same two fittings and they're immaculate. So, if 00:02:30.720 --> 00:02:37.200 it was a configuration error in the loop, these should be eaten. And yet, 00:02:35.280 --> 00:02:40.480 they're pristine. Our goal today is twofold. Then, my son obviously wants 00:02:39.120 --> 00:02:46.000 his gaming rig back up and running, which is why he's helping. But before we 00:02:43.280 --> 00:02:49.280 do that, we need to know what the heck happened here so that we can ensure that 00:02:47.599 --> 00:02:52.879 this won't happen again. The first step of disassembling these blocks is popping 00:02:50.959 --> 00:02:57.840 the stainless steel cover off. It's important to note that this is purely 00:02:54.959 --> 00:03:01.599 cosmetic since steel is not a compatible metal with the ones that are typically 00:02:59.519 --> 00:03:06.560 used for water cooling like copper, brass, and nickel. 00:03:04.480 --> 00:03:10.959 Now, let's go a layer deeper and remove the delin top. Okay, not the cleanest 00:03:09.120 --> 00:03:14.879 loop I've ever seen. Why is there so much dust on it? 00:03:12.400 --> 00:03:20.080 Well, I can tell what some of this is. These look like little plastic filings 00:03:17.440 --> 00:03:24.239 from working with the tubing that we installed. Oh, there's a little metal 00:03:22.080 --> 00:03:29.840 shaving. Uh, that could be left over from manufacturing of something. 00:03:27.440 --> 00:03:35.440 Radiator, heat exchanger. There's a surprising amount of this that looks 00:03:31.280 --> 00:03:38.720 like hair or other organic matter, but 00:03:35.440 --> 00:03:41.680 there's iodine in the loop. And what 00:03:38.720 --> 00:03:46.560 kind of contradicts that theory is that this tube over here, which is running 00:03:43.519 --> 00:03:48.480 water to my system right now, is crystal 00:03:46.560 --> 00:03:52.400 clear. Well, maybe one of the other systems will give us some hints as to 00:03:50.159 --> 00:03:55.920 what that is. Actually, we don't need to go that far. Around the edges of the 00:03:54.239 --> 00:04:00.239 reservoir, there is a little bit of something. Not much, but I think I might 00:03:57.680 --> 00:04:05.280 need to up my iodine dosage and maybe throw a filter in there. Either way, 00:04:02.879 --> 00:04:10.879 none of that would explain the leak. Let's take apart the other block. Here 00:04:07.439 --> 00:04:15.280 we go. What 00:04:10.879 --> 00:04:17.680 the crap? What is this? 00:04:15.280 --> 00:04:21.759 Ew. Why does it look burned? Kind of does look burned. It's 00:04:19.359 --> 00:04:26.479 definitely not. It was underwater. But the kind of oxidation that could occur 00:04:23.919 --> 00:04:32.479 when burning, I guess, could also happen when it's being corroded. But the funny 00:04:29.040 --> 00:04:34.960 thing is that this is where the metal 00:04:32.479 --> 00:04:40.000 and you can actually see metallic elements here. This is where the metal 00:04:37.600 --> 00:04:46.080 was being deposited, but there's no metal for it to deposit 00:04:42.400 --> 00:04:49.280 onto. Normally in galvanic corrosion, 00:04:46.080 --> 00:04:51.759 you have a sacrificial anode and then 00:04:49.280 --> 00:04:58.720 you have a a cathode that the metal deposits onto, but it's depositing onto 00:04:54.400 --> 00:05:01.040 the delin or the space above it. You can 00:04:58.720 --> 00:05:05.280 see that's what was being corroded away, right? Which is why the fitting broke 00:05:03.280 --> 00:05:11.199 off and liquid gushed out into the system. But the deposit was so close. It 00:05:08.400 --> 00:05:17.560 was right here. the mid plate that fitting crumbled, I 00:05:14.240 --> 00:05:17.560 think. Whoa. 00:05:18.479 --> 00:05:24.560 Okay. Well, that's something that happened. 00:05:22.639 --> 00:05:28.560 Now, EK had a theory when I contacted them, and that was that this stainless 00:05:26.400 --> 00:05:34.320 steel cover was somehow participating in the galvanic corrosion process because 00:05:31.120 --> 00:05:37.680 it was bridged to the fluid somehow or 00:05:34.320 --> 00:05:40.400 something. But I just don't see how it 00:05:37.680 --> 00:05:44.479 could connect. None of these screw holes externally 00:05:42.240 --> 00:05:47.919 pass through to the inside of the block. And you can clearly see with their 00:05:46.160 --> 00:05:51.840 decades of experience designing these things, there's no connection here 00:05:50.000 --> 00:05:55.280 either, even if I screw this in all the way. Not going to lie, boy. I was hoping 00:05:53.759 --> 00:06:00.080 it would be a little bit more obvious at this point. Can we just fix it and then not care 00:05:58.160 --> 00:06:06.479 what happens? We got really lucky this time. We only 00:06:02.720 --> 00:06:09.600 lost the big UPS at the bottom. And I 00:06:06.479 --> 00:06:11.840 lost one of the rows of NVMe backplane 00:06:09.600 --> 00:06:16.240 slots on one of the servers. The other row fully works and I don't need the 00:06:13.280 --> 00:06:21.280 second row. Um, also your computer is dead. We won't necessarily get that 00:06:18.800 --> 00:06:25.120 lucky next time. There were pools of water on the motherboards of some of the 00:06:23.120 --> 00:06:28.560 systems under this one. So, we need to figure out what happened is what I'm 00:06:26.400 --> 00:06:32.240 trying to say. Which means we need to start dissecting some more victims. 00:06:30.319 --> 00:06:36.479 Actually, wait. Before we look at a different system, we should look at a 00:06:34.160 --> 00:06:41.360 second block within this system to see if there's something unique about this 00:06:38.880 --> 00:06:44.479 environment. These fuel line hoses are a bit of a nightmare to get off. You can 00:06:42.720 --> 00:06:47.039 see the reinforcing string that makes even cutting them a bit of a chore. 00:06:46.000 --> 00:06:52.319 Okay, what do you think we're going to find? A lot more corrosion and metal. 00:06:50.720 --> 00:06:55.680 Oh, crud. I called it. Cuz this means your computer's not 00:06:54.639 --> 00:07:01.919 getting fixed today. Oh, still happy? No. That might do it for EK's theory 00:07:00.000 --> 00:07:06.479 about that top stainless steel thing. Maybe this is a more widespread issue. 00:07:04.319 --> 00:07:09.919 Now we really need to start cracking open more systems and seeing what's up. 00:07:08.400 --> 00:07:15.440 One rail came out. Hold on. You got to come straight out. 00:07:12.720 --> 00:07:22.880 There you go. Oh my goodness. The plot thickens. The second fitting on this GPU 00:07:20.240 --> 00:07:28.479 is pristine. What is going on here? Could it be the 00:07:25.680 --> 00:07:34.080 worm drive clamps? They're not touching though. And this one hasn't been moved. 00:07:31.280 --> 00:07:40.160 I unscrewed it by just twisting up the tube. This is exactly as it was. 00:07:37.840 --> 00:07:46.080 Bro, nasty. What this means to me is every one of 00:07:43.440 --> 00:07:50.880 these systems has to have every fitting popped to check if it's corroding. A 00:07:48.960 --> 00:07:56.720 couple of the screws going into the top of this GPU block are rusted. This is 00:07:53.919 --> 00:08:01.440 not a damp environment. So, the only moisture they should have access to is 00:07:59.440 --> 00:08:05.440 from in here. I didn't know which holes they came out 00:08:03.520 --> 00:08:09.280 of, though, unfortunately. I don't think your computer's going to be working for 00:08:07.120 --> 00:08:11.599 a while. Hey, thank goodness it's summer and there's lots else to do, right? 00:08:11.120 --> 00:08:16.319 Yeah. I wouldn't let my dad get away with that, but hey, thanks. 00:08:14.639 --> 00:08:22.319 What do we have to do to get it working again? It relies on a cooling system that it 00:08:19.280 --> 00:08:24.879 shares with six other systems. 00:08:22.319 --> 00:08:28.960 All six systems need to be checked to ensure that they don't have corrosion in 00:08:26.639 --> 00:08:34.560 them because if one of them corrods, it takes down the whole thing. So all the 00:08:31.919 --> 00:08:39.919 systems need to be taken right down to the disassembly of the bare parts. 00:08:36.719 --> 00:08:41.680 Okay, so like 2 months. 00:08:39.919 --> 00:08:45.839 I'm the kind of dad who believes in honoring my promises. So I am not going 00:08:44.320 --> 00:08:50.880 to promise anything. Okay, this is shaping up to look like it is 00:08:48.560 --> 00:08:55.920 going to be an absolutely terrifying amount of work. like better off starting 00:08:54.000 --> 00:08:59.920 over amount of work here. So then shouldn't we just start over? 00:08:58.080 --> 00:09:04.080 Well, I mean it'd be kind of wasteful, don't you think? And this is your gaming 00:09:01.839 --> 00:09:07.120 rig. This has seen you through many battles. Well, you just going to leave 00:09:05.680 --> 00:09:12.000 it behind on the field? Yeah. Now, it's clear there are issues with 00:09:10.160 --> 00:09:15.760 the loop, not caused by anything to do with block design. Uh there is some 00:09:14.240 --> 00:09:21.680 organic matter in here. You can really tell now that it's still wet. But I 00:09:18.720 --> 00:09:26.160 reiterate that would not cause a leak. So then it is EK's fault. 00:09:23.680 --> 00:09:29.440 Well, we can't say that definitively. I think what we're going to do is we're 00:09:27.760 --> 00:09:35.040 going to investigate one or two of the other systems, send them our findings, 00:09:33.040 --> 00:09:40.880 and then we'll go from there. The worst corroded one in both cases was nearest 00:09:38.240 --> 00:09:46.240 the acceleration or impingement plate. But on the GPU, the deposition didn't 00:09:43.360 --> 00:09:52.080 take place on the plate itself. It took place here before it. You're probably 00:09:49.760 --> 00:09:58.080 wondering what to do next. What we need is a corrosion check on this, this, 00:09:54.959 --> 00:10:00.000 this, and this. You're going to start by 00:09:58.080 --> 00:10:02.959 blowing all the water out of the system. How do I do that? 00:10:01.120 --> 00:10:08.160 I'm glad you asked. When they aren't coupled, these quick connects stop the 00:10:04.880 --> 00:10:11.920 flow of water and air. So, there's no 00:10:08.160 --> 00:10:16.590 way for us to blow out the system unless 00:10:11.920 --> 00:10:18.800 we put a female fitting on them. Now 00:10:18.800 --> 00:10:24.880 she should just kind of drain. 00:10:22.000 --> 00:10:29.279 Oi. Hey, there we go. But that's slow. And we can accelerate the process using 00:10:27.760 --> 00:10:34.079 Under normal circumstances, I'd be giving you guys a bunch of lectures about air compressors and computers. 00:10:32.079 --> 00:10:38.800 There are concerns with moisture being blown onto the components, uh, static 00:10:36.399 --> 00:10:41.600 buildup, but in this case, we're just blowing out tubing, so we don't need to 00:10:40.240 --> 00:10:46.640 worry about any of that. What you're going to do is you're going to take this. You're going to hold it as tight 00:10:44.000 --> 00:10:50.870 up against the in one as you can. And all the water is going to blow out the 00:10:48.160 --> 00:10:53.440 other side when you pull the trigger. 00:10:53.440 --> 00:11:00.640 Now, we don't need to worry about water leaking all over our system as we 00:10:57.120 --> 00:11:03.040 disconnect our hoses. Realistically, 00:11:00.640 --> 00:11:07.760 the system is not alive anymore, but there could still be components that are 00:11:04.640 --> 00:11:11.200 good. Remember that I said that your SSD 00:11:07.760 --> 00:11:12.959 might be okay. Well, that's because it's 00:11:11.200 --> 00:11:17.519 over here on the other side from where the leak was. So, as long as the water 00:11:15.120 --> 00:11:22.079 didn't come up that high, or as long as there wasn't some kind of malfunction of 00:11:19.200 --> 00:11:26.079 the motherboard that caused a cascading malfunction to your SSD, you might still 00:11:24.560 --> 00:11:30.959 have all your games and data and all your saves and all that kind of stuff. Get this hex head and pull the block 00:11:29.440 --> 00:11:36.320 off. Clean the thermal goop off the bottom of the block while you're handling it so you don't make a mess. 00:11:34.320 --> 00:11:40.880 Okay. Next thing you want to do is get these fittings off. Yeah, you might need 00:11:38.880 --> 00:11:44.399 to take off these worm drives first. So, you need that flathead. There's certain 00:11:42.480 --> 00:11:48.240 things that I really hope are not dead. I mean, a motherboard's a relatively 00:11:45.920 --> 00:11:52.480 simple replacement. I even have a cold spare of these just in case, you know, 00:11:49.839 --> 00:11:57.440 we experienced a failure, but like these PCIe Gen 4 extensions and stuff like 00:11:54.959 --> 00:12:00.800 that, it'll just probably take a little while 00:11:58.800 --> 00:12:05.279 to get new ones. Then again, I've made no commitments about when this will be 00:12:02.240 --> 00:12:08.240 done. So, now we're just going to play 00:12:05.279 --> 00:12:12.240 Twisty Twist until this comes out. Not the world's most fun game. 00:12:10.160 --> 00:12:15.200 Twisty twist is very effective. How'd you get to be so sarcastic? 00:12:14.079 --> 00:12:19.839 Bad parenting, I guess. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Speaking of bad 00:12:17.839 --> 00:12:25.720 parenting, I think I'm just going to have to give you a really sharp knife to 00:12:21.519 --> 00:12:25.720 cut the tubing off. Thank you. 00:12:27.760 --> 00:12:35.440 Perfect. Okay. So, now we just put it back. 00:12:32.880 --> 00:12:38.560 Well, I don't know, man. Like this is raising more questions than it's 00:12:36.959 --> 00:12:42.800 answering. I think what we need to do is have a 00:12:40.160 --> 00:12:46.240 look at the fittings on the GPU now. So this one's perfect on both. 00:12:44.399 --> 00:12:51.040 Yeah, maybe this is a system by system thing 00:12:48.320 --> 00:12:56.720 then the fittings are perfect on the GPU on the second system. Remember our first 00:12:53.519 --> 00:12:58.560 control system as well. Got two more to 00:12:56.720 --> 00:13:04.399 go though. Let's see what's shaking in those trays. Interesting. 00:13:01.519 --> 00:13:09.200 So yours is the bad one. Oh man. I guess this is it's kind of good news, but it 00:13:06.240 --> 00:13:13.760 also just deepens the mystery. And there you have it. Looking at the last two 00:13:11.040 --> 00:13:19.680 systems, the ones above the leak. Flawless. Flawless. Flawless. Flawless. 00:13:16.959 --> 00:13:24.560 Only one system experienced corrosion out of the five. And it was on two 00:13:22.399 --> 00:13:29.040 different blocks. For real. Chemistry majors, get out of 00:13:27.040 --> 00:13:34.320 your armchairs. Or rather, get in your armchairs. What happened here? So, 00:13:32.639 --> 00:13:39.200 you're probably wondering, "Well, what now then?" Well, now I'm going to test 00:13:37.440 --> 00:13:44.639 the two systems that were under number three and see if they still work, which 00:13:41.440 --> 00:13:47.440 means maybe. Ooh, capsule. 00:13:44.639 --> 00:13:54.720 White light. Good. Hey, desktop. Better. As unbelievable as 00:13:50.959 --> 00:13:57.760 it might seem, the only system casualty 00:13:54.720 --> 00:14:01.120 is my son's gaming rig. Almost 00:13:57.760 --> 00:14:03.360 everything else survived. 00:14:01.120 --> 00:14:09.440 Is it possible that this system still functions? I kind 00:14:07.040 --> 00:14:14.560 of doubt it. It gushed water over the motherboard like right over the VRM. But 00:14:13.040 --> 00:14:18.000 everything else survived. What do you think, Andrew? Is she just going to like post? 00:14:17.040 --> 00:14:22.360 I'm off. I'm not. I think this one's dead. 00:14:27.360 --> 00:14:33.120 Now, modern electronics got a lot of 00:14:30.399 --> 00:14:39.760 protections. No ShortCircuit, over current, over voltage, but dump gallons 00:14:36.959 --> 00:14:44.079 of flipping water over it protection. Obviously, I can't deploy it the way it 00:14:41.839 --> 00:14:48.000 is. It's got its blocks and fittings all corroded to heck and back. So, there's 00:14:46.320 --> 00:14:52.240 definitely going to be a follow-up where I hear from some of you folks. I go back 00:14:50.240 --> 00:14:55.680 to the manufacturer. We figure out exactly what's going on. We rebuild this 00:14:53.839 --> 00:14:59.519 mess. But I'd say this is about the best outcome I could have possibly hoped for. 00:14:58.000 --> 00:15:02.800 Just like I couldn't hope for a better sponsor than our sponsor. 00:15:01.360 --> 00:15:05.920 Next aiggo. Have you ever hopped on a meeting and thought your screen was 00:15:04.480 --> 00:15:10.959 frozen because the person you're talking to who's picture is moving at 5 FPS? How 00:15:08.959 --> 00:15:15.360 embarrassing? Enhance your work from home setup with the Nexo Hello Cam Pro. 00:15:13.680 --> 00:15:19.519 It features Windows Hello facial recognition for an effortless login 00:15:17.360 --> 00:15:23.199 experience. And it also has an automated privacy cover that closes when the 00:15:21.279 --> 00:15:28.000 camera's not active. Plus, you'll look great at your next meeting thanks to the 00:15:24.480 --> 00:15:29.680 1080p 60fps resolution and refresh rate. 00:15:28.000 --> 00:15:32.880 Learn more using the link in the description. If you enjoyed this video 00:15:31.519 --> 00:15:36.959 and you like this kind of raw troubleshooting slashbuilding with the 00:15:34.800 --> 00:15:41.839 kids, then maybe check out uh fixing the gaming minivan. They were helpful.