On our last episode of Linus Tech Disasters,
one of the custom liquid-cooled gaming machines in my home rack sprung a leak, pouring gallons of water
over every system beneath it. And my initial thought was,
oh, I forgot to screw in a fitting. But as it turned out,
despite taking precautions to prevent it, it wasn't a fitting not being screwed in.
It was a fitting completely corroding off of the block.
Maybe some of the metals in our loop are incompatible.
Nope, after spending hours inspecting them,
every other system's fittings were spotless,
not a hint of corrosion to be seen. I was so dumbfounded that I turned to the YouTube
comments for help and it actually worked.
There was a strong consensus on what the problem was. So today, with the help of Pulseway,
who sponsored this video, we're gonna hopefully rectify this silent killer
and add a remote monitoring system so that if there ever is another leak,
hey, at least I'll know about it before I come home at the end of the day
and find that my computer peed itself.
The first really good community theory was that the fancy fuel hose we were using as our tubing
might use steel webbing to reinforce it, which could introduce galvanic corrosion,
but this was actually due to a miscommunication on my part.
It's not fuel hose, it's a general purpose hose, and it uses a textile reinforcement rather than steel,
so we could scratch that one.
Another interesting theory was that a microcrack
in the delrin plastic of the GPU block might have allowed the steel screws that hold it down
to come in contact with the water, but a close visual inspection reveals no cracks,
and the threading for all of those screws is completely clean,
which suggests that no corrosion occurred there. A third theory was that a flaw in the plating
of the affected fittings could allow galvanic corrosion to occur with the underlying metal,
which would explain why some of them were eaten while others remained perfect.
But the thing is, when you or EK in this case
is designing anything that involves metal and water, galvanic corrosion is something
that they carefully consider. So, while the most obvious way to avoid corrosion
is to never mix metals, there are many metals that are safe to mix,
especially when they're not directly connected, and the underlying material of these fittings is brass,
which, if you consult this anodic index, should be fine with the copper and other metals in our loop.
Should be fine. The story changes when you apply a voltage,
and well, we obviously weren't intentionally doing that
when you have a large power draw in one spot, like a couple thousand watts of computers in a rack,
and a low power draw in another spot, like a couple hundred watts from a pump,
you're gonna end up with a difference in voltage between them, which accelerates corrosion dramatically,
and can make compatible metals far less compatible.
But the good news is for this solution, oh boy, we don't have to worry about the details.
This is an isolation transformer, and you might be familiar with transformers,
you would use one to take a low voltage and make it a higher voltage, or vice versa.
Sure, like to run your sewing machine from Asia. Yeah, if you're in a different country, that's a big one.
In this case, it's 120 volt on both sides.
You see that? Made in Canada, thanks Hammond Manufacturing,
but it decouples the circuit giving us galvanic isolation,
or preventing a loop. So in a nutshell, we're taking anything
that's plugged into this guy, and we're giving it absolutely no path
back to the main panel. It's almost like Wi-Fi for your electricity.
I feel like our analogies are making this worse. The point is, no voltage difference.
Wrap it before you tap it.
Where do you want to put this thing? How about up here? On top of the tank?
The tank full of water? I mean, it should be fine, right? Yeah, I mean. Realistically, while we're here,
I should probably have the pumps running off the UPS. You'd have to run an extension cord
all the way over there. Conveniently. Is that an extension cord that's all the way right there?
Yeah, I really don't think that's longer than that. I don't think it's worse than that. Yeah, I don't think it's longer than that.
Look how long that is. That's a 25-footer. It'll work. You want to bet everyone's lunch on it?
If I'm right, you pay for everyone's lunch. If I'm wrong, nothing happens.
Fine. It has to go all the way down to plug into the bottom UPS.
It has to be managed. Sure. He's literally trying to cable manage it
as much as possible to prevent it from reaching.
Dude, these are so satisfying. I will admit, they are quite good.
LTTstore.com, let's go. That might be the best thing you've ever done.
My kid's right there. What? No, he's not. Probably somewhere.
It's gonna be so funny when you lose. Wow, it reaches.
No way, no way. You're buying everyone's lunch. Get wrecked.
Fine. Even though I owe him lunch, technically? Yeah.
One of the big things that you guys said was, hey, no matter what caused this problem,
you should really have monitoring in place so that if it were to ever happen again, you would know.
And on top of that, we can make automation. So if there's a leak, shut off the pumps.
Yeah, it's going to be good. Theoretically, everything is fine now.
I mean, but. Maybe. But that doesn't mean we're going to stop here.
OK, what's next? Grounding. Grounding.
All right. You're not going out for a week. Says who?
You going to stop me? I have your stepdad's number. Keyword being step.
Are you going to take that? He might actually see this.
To take things to another level, we're going to follow another recommendation that
came up a lot in the comments. Some of you noticed that I never grounded my rack.
While the systems themselves are theoretically grounded
using the third prong of the plug that we plug into them, the rack itself is not necessarily connected to ground.
So we've got a whole bunch of these grounding leads.
And then what? We just connect these from the back of every chassis
to the rack? You could. But we also got this. Here, catch.
Like a grounding bus bar. So you just stick that in the rack
and then screw all the little thingies onto it. Keyword management. No, no, we got this. We got this.
OK. Oh, that just barely fits. Yeah, so that could go right there.
It's almost like it's made for a rack. Naturally, we have to ground our grounding bar
to something for it to work as a grounding bar. Unfortunately, the rack is a bit of a ways from the panel.
And our electrician who is going to do this properly for us is unavailable today.
So instead, we're going to have to open up the panel and tie the wires directly into the grounding bar.
It also means I didn't have the correct wire. So Linus happened to have these two pretty thick gauge wires.
So we're going to use those instead. I also didn't have the correct connectors.
So I cut these Hammond ones, butt connected them onto the long wire, and now we have the right connector.
Is this how you should do this? Probably not. But it's just a ground.
It's probably right, Rhys. I don't know what I'm doing.
Got it? Hey, it worked. There we go, grounded.
I'm going to let Linus be the one to open the panel and stick his fingers in there because it's his house.
So we're just going to leave this for now. Hammond sent us these sick little grounding kits,
and they have little just blade connectors. So you hook one set up to the server,
one set up to the grounding bar, and then if you need to take the server out for some reason,
you can easily disconnect it. Let's try it with one of Linus's Nasty units here.
Let's screw that.
Look at that. Wow. Now we just repeat like 10 times.
While Rhys works on getting all the systems grounded, let's talk sensors.
We have three different types. Starting with our leak sensor, although calling it a sensor might be a bit generous.
This is basically just like two wires. You apply voltage to one side,
and as your leak bridges the gap with water, it creates an electrical connection
between the other side, and on that side, you're just sensing for any voltage.
So when there's no water, it should be zero volts-ish, and when there is water, say if we did five volts
on this side, or probably should like one, or two volts on this side. So that works.
And then we've got a flow sensor. This one is really just to tell us if, say,
the water runs out, or the pump stops working, which would obviously be bad,
because then the water inside the computers would boil, which might cause a leak.
This way we can notify ourselves if the water is not flowing, and then we've also got a little float switch.
Very simple. This one, if it's at the bottom current passes, if it's at the top current doesn't pass.
So we stick it in the water tank here, at the top,
and we can tell if it's full or not.
I should do not, this is how we're gonna test it later.
But before we can install it, we have to drain the loop, which is going to be interesting.
This valve is open, and nothing is coming out.
I saw a comment. Somebody said they had experience with hose bibs like that,
and they said that some of the manufacturers started using really crappy cheap materials in them
that are not necessarily compatible with plumbing, and I just looked up in there,
and you know what I saw? Corrosion. I read the comment, and I was like,
this is too stupid to be true.
Linus! Guess what doesn't happen when you open this hose bib?
Oh, no.
You're ballsy. Let's take your phone light up there. Uh-oh. Can you imagine if that's literally
what caused all of this? You've gotta be kidding me.
You know, whole room water cooling was exactly the same thing.
It was a vendor cheeping out on the parts. That reservoir was supposed to be stainless steel.
I mean, you should've just used a plastic one. But it wasn't made of stainless steel.
I used a plastic one. Yeah, you also were the one who specced this.
Supposedly brass hose bibs, smart guy. I bought it from a plumbing store.
So it looks like we were both bamboozled. You don't get to be all...
Yeah, but at least my idea was like a good one. It's also good that I got this particulate filter.
Yeah. It's plastic, actually. Oh, it's a polypropylene, I think.
Nice. Oh, no, it's polyester. It's like a t-shirt, basically. What are these made of?
Ah, well, those are metal, but that's the same type that's already in the loop. Because I don't feel like going to the plumbing store
and because I don't trust those things ever again, I'm going to make my own valve.
I've got a little guy and a little valve.
Oh, look, I fixed the leak.
Just gonna get water everywhere.
Ooh. We have to make sure that we put it the right way.
There's these very helpful arrows that I should follow.
Oh, look, I even drew arrows on the tubes. So I just matched the arrow with the arrow
and then I can't screw it up.
Hey, Reese, you leak tested this filter, right?
Filter installed. Is that closed? Beautiful.
Water in the filter, yay. The sensors are great and all, but they don't exactly work without something to read them.
In the past, we've bought little like relay or input output boards for Z-Wave or Wi-Fi on Amazon.
But if you've seen anything about how Linus's wireless has been working, eh, I wanted something hardwired.
And then I stumbled upon a company called Sequent Micro Systems, which sounds super nerdy
and to be honest, it kind of is. All they sell are these custom home
and industrial automation boards. And this one, which is a Raspberry Pi hat,
just plugs into a Raspberry Pi. Weep, this one has 16 universal sensors on it.
That means we could handle all seven of the gaming computers in here for leak sensors.
We could put a couple on the floor, our two flow sensors and still have room to spare.
It was a little complicated to get set up, but they actually had really good customer service.
The guy was emailing me back within the hour, responding. They have a bunch of different boards.
You can get ones that have some sensors, some power outputs, you can get ones that have MOSFETs,
you can get ones that have just cereal. Like, they make a lot of different products
and it's actually pretty awesome. This one in particular requires separate power.
So you plug in a five volt power lead here
and then it powers the Raspberry Pi through it directly.
There's also a battery for a real time clock that's on the board that they use for hardware monitoring.
So if the Raspberry Pi becomes unresponsive, this board can reboot the Raspberry Pi.
It's one of those things where you wouldn't really think about it for a house setup, but if you were stuffing this in a wall somewhere
for industrial controls, you want to make sure this thing is operational
if there's an issue, it fixes itself. Imagine if you had like 200 of them, right?
In a perfect world, I just hooked that sensor pin right up to this board and be done with it.
But unfortunately, electronics can sometimes be a bit more complicated.
In our case, we need to add something called a pull-up resistor. And that just makes sure that when there's no pulse
being sent, the sensor reads a high signal, like close to our voltage around five volts.
If we didn't have that, the sensor wire can actually sometimes float between ground and five volts,
which looks a lot like a pulse, which obviously we don't want.
The pull-up resistor makes sure it's only ever reading the correct value, so it's very important to add.
If you had an Arduino that was reading this sensor instead, they actually have a pull-up pin mode,
which does it on the board. But in our case, we're just going to add one
on this breadboard, and in the future, we'll just solder it into a piece of wire
and be done with it.
Does the flow sensor work? Yeah, we literally just finished wiring it.
Is it in like a home assistant, or like, what is even going to control this?
No, it can be, if you want. Right now, I just wrote a little script.
Obviously, we have to interpret the data, because all we're getting is pulses.
Luckily, they just give you a formula. You take the pulses per second, or hertz,
and divide by 5.5, that's your leader per minute. So I made a little script to do that.
Hey, nine liters per minute. So we're getting roughly, almost exactly, 50 pulses
per second, 50 hertz. The heartbeat of the whole setup.
And then if we go to the other sensor, which I did flow to, it's off right now.
Right, no flow. Oh, no flow to the pool right now. So what we'll do is adjust this script
to send that data to the PulseWay API, and then we can make whatever workflows we want
in PulseWay, and then on top of that, we can just make a little script
that sends it to Home Assistant, if you want. Super cool. Because Home Assistant also allows, like,
custom components, so you can have it on your phone. Now that the pool loop is refilled and running,
we should be able to take a look at our flow from both of the pumps.
I found this cute little terminal app called TTYplot, and you can plumb data from a script to it,
and it makes a little graph in your terminal. It's very cute. This top one is the local loop,
so if I adjust the speed on the pump here, right now it's on low.
If we go to high, hey, look at that.
This is the outdoor loop, only pushing around five liters per minute,
which is pretty low. And then the indoor loop seems to hang out around 10.
Five liters per minute? Yeah, that's pretty bad. Oh, hey, look, everything's grounded.
Yeah, look at that. It's all grounded. We won't be out for weeks. We were overthinking it.
Each of them we just found a screw and put it underneath the screw.
The only thing is we didn't ground it into the panel yet, so it's not technically grounded anything.
Right. I did run the wire over there. Oh, okay. Mostly.
Is this you? Yeah, I grounded it. Okay. We should probably shut the panel off before we open it.
Yeah, where's the one for this? Oh, this one.
Oh my gosh, okay. This is not an insulated screwdriver
and is not rated for electrical work. The only reason I'm doing this
is because this panel is completely off, de-energized.
I do not recommend this. Don't try it at home, et cetera.
Oh my goodness, who put this in here? The screw is shallow.
It's just camming out every time. Can I try to do it? Hold on, hold on.
We might not even need it. My God.
Oh my God. Zero. Zero. Okay, so everything in here is safe.
I mean, well, we hope. The rack is hooked up now. See, you got a little ground cable going in there.
Definitely not spliced inside the panel. That one.
Now that the flow sensors are working, let's wire up a leak sensor. Like I mentioned before,
we have to apply power to one side. So we'll plug this into five volt here.
And then the other side, instead of getting a pull up resistor, gets a pull down resistor.
It functions in a very similar way, but instead of pulling the voltage up to your kind of voltage supply level,
it's pulling it down to zero because the path goes to ground. The last thing we've got to do is adjust the sensor
on the board to be a zero to 10 volt sensor.
We can see that it's reading about 0.17 volts
when, in theory, nothing is connected.
Okay, if I bridge this now with a screwdriver, which is metal,
it goes from like 0.1 volts right up to five,
which makes sense. When I let go, there's no path and it reads 0.14.
Now we just need to try it with some water.
Hey, it immediately registered 0.5 volts.
Now, instead of printing this the terminal, let's send this data to Pulseway.
Fortunately for us, Pulseway has a cool feature called custom fields and it allows you to attach custom bits of data
to systems and entire site. You can do it globally if you want.
Typically you would use this with their script system, say if you wanted to store the bit locker keys
for all the systems on your network or the Windows licenses or check battery data,
things that you would only need to read like once a day or something like that.
But in our case, we want to read these sensors like every five seconds. Fortunately, they also have an API.
So we can just edit our little script instead of printing it to the console,
send it to the Pulseway API and then that'll set the data. While I don't technically need the Pulseway agent
running on our sensor Raspberry Pi, I'm gonna install it anyways because then we can have monitoring.
Obviously if the sensor Raspberry Pi dies, that would be not great and we'd want to know about that.
Once it is installed, it's really easy to connect to the account. They actually have a command called Pulseway registration.
Just go register, no proxy, custom server, yes.
It just pops out this big QR code. And bam, it updates the config file
and starts running Pulseway. We should be able to see it in the Pulseway window now.
Hey, look at that, dream house sensor Pi for, now let's send some data.
I went ahead and created a couple of custom fields in Pulseway, one for the flow rate
of the loop inside the house and one for the flow rate of the loop that goes to the pool.
And then when we run this, output updated account, 9.45.
And then if we go to the site, if we go to Sebastian, dream house, custom fields,
we can see that it's 9.45. Let's see if it stays that way.
If I go ahead and turn the pump down,
boom, 8.72, perfect.
Now we just have to install Pulseway on some machines so we can make them turn off when bad things happen.
We're first gonna make a workflow in Pulseway. I made one here called shut off water cooled computers.
Ideally you would make a system group and then put all the systems in it, but for now we're just gonna use this one,
which is LAN 5 and just set that manually. I'm creating this as an ad hoc workflow,
which means you have to manually trigger it, but you can also set them up to run on a schedule,
say like every day at 2 a.m. Or you can set it to run based off of a notification
like CPU usage too high or disk full. Then we just say shut down.
And there isn't an option to just shut off computer because this is like an IT management solution, right?
Generally you don't want to like fully shut off on computer because now you can't manage it anymore.
They do have an option for reboot, but we're gonna say execute a PowerShell command
of shut down. I'm also gonna add a send email task if the condition check fails
because it would be a little weird for this workflow to get triggered and then the custom field read a value that's fine.
Like there's something wrong at that point. So we should get notified if that's the case.
If the flow goes below one liter per minute, it should shut off LAN 5.
It should have done it. Did it do it? Uh, I wouldn't say that.
It's definitely run the workflow. You're about to be signed out. Windows will shut down in less than a minute.
What, no way. Wait, it's gonna shut down in a minute, it says?
It says less than a minute. She's shutting down, boys.
That's great. Maybe I need to adjust that script a little bit to make it a little more urgent
in terms of the shutting down, but not bad. Yeah, and when we combine that with the leak sensors,
basically we can set it so that in the event that something goes bad, everything shuts off.
The last part of our little electronics project here is being able to turn the pumps on and off.
So if we detect a leak from one of the leak sensors,
we can shut off the pump and limit the amount of water that gets out and the amount of damage.
How do you do that? With relays. Each of these is a 40 amp, 250 volt relay,
which is way overkill for what we need.
I can just plug a serial cable into this, which is, in this case, an Ethernet cable,
and plug the other side into our sensor board and then control this remotely
from the Raspberry Pi and the sensor board. That means we can put this right next to the pumps,
run an Ethernet cable back over here, and Bob's rungo, we can control the pumps remotely.
Long term, I think we could mount it in the big isolation transformer box.
That actually has a switch already, so we could probably just wire this up to the switch,
but for now, we're just gonna do this to try it. This is not my finest work,
but I've spliced the relay in between the load wire
on this power bar. And these are all connected with Wego nuts, so they should be relatively safe.
And if we plug this in, this power bar should not be powered.
Dun dun, dun dun.
Power bar, not powered, that's good. And now, let's plug in the relay board.
It does need its own source of power, specifically 12 volt instead of five volt this time.
So we'll wire that up here. There's also a few dip switches you wanna toggle.
On this board, we enabled TX and RX, and then the termination switch,
and then we just did the termination switch on this side. That's because each of these is the end device.
If you had a bunch in between, daisy chains, you would just turn the termination one on,
on each one on the end. If the board was mounted directly on the Raspberry Pi,
we could use their command line library directly. But instead, since it's going over serial,
I had to turn off the serial functionality on there, so it's just passing through,
and then enable serial on the Raspberry Pi. And in theory, hey, cool.
You see him clicking? Now, we can plug this in.
You ready? On, see that's lit up, and then we go off.
Now I've just gotta make a little script to toggle it based on the leak sensor,
and hope it works. So we've done the demo, where if the pump fails, the systems turn off.
But what we haven't demoed, is if a leak sensor gets triggered, the pump turns off.
Well, let's do it. Okay. I'll go lick a leak sensor.
Five volts not gonna hurt. I've licked a battery before us. But how many amps is it?
Five amps, it's fine. Five amps? Yeah. I wouldn't lick that.
I'm gonna lick it. I wouldn't lick that. I'm gonna do it. Well, make sure you get it on camera.
Fine, fine, I won't lick it. I'll just dip it in the water on the floor. Go for it.
And dip.
Oh my God. Right, we've got a light on that same power bar.
Oh, well did the flow stop? I didn't, yeah. Okay, now let me un-lick.
Okay, obviously we have to fix up the wiring here a little bit.
Is it going? And she's going. Hey!
So theoretically, well there's a lot more protections
than there were before. Most of them are just sitting on the concrete floor.
So if it leaks, I'm sure they'll work great.
I have a box, we'll put it all in a box. We'll make it nice. You know, it also worked great.
Thank you so much for sponsoring this video. Pulseway is your solution for remote management
of all manner of systems, not just janky basement water cooled ones.
Yeah, we built our own monitoring system and then used Pulseway to do all the system stuff.
And they have a fully featured mobile app so you can diagnose and even fix problems on the go.
Yeah, like if your pump stops working, you'll know, even when you're in France.
There's updates, temperature monitoring. You won't be able to do anything about it.
Disc usage monitoring. But you'll know. Yeah, you can call your team.
Get them to do it. You think I'm gonna come to your house and fix your water cooling while you're in France?
It's the Pulseway pitch. Oh, that would be helpful.