Well, hello YouTube
We have a bit of a problem today on this section of the moving, but we've been moved in for months moving vlog
Something is going on with the server
and we haven't been able to work on a project for more than like 30 minutes at a time before the server crashes and
Our computers all freeze and then we're all sad because we can't work on them. Most of the sh** we do is on the server
Well, I'm doing I'm actually doing some writing right now because I can do that
I'd love to be working on my next video. Anything that doesn't need the server we can do now
No, that's not much, but it always seems to go down
Yeah, at the worst when there's nothing else to do. No kidding, but you know YOLO you only
Liao once
See let's see the stress that Linus is going through right now
I'm only getting about 25 minutes per second, 30 minutes per second
It's probably gonna crash like while you're doing this
Yes
How do we fix servers? How do you fix it?
Tell Linus he will fix it
That's your go-to. Yeah
Whether you're running a small business or a
One billion dollar or even multi-billion dollar enterprise Rackspace has your dedicated storage environments covered
Check out the link in the video description to learn more
All right
So here's the situation
Wonix server over the course of the last couple of days has been spontaneously going offline
And turning off in spite of my best efforts to turn it back on and
Complete the backup that I'm in the middle of trying to do to our new unraid vault
While I was standing in front of it. I
observed
One of my raid controllers giving up the ghost
So as a reminder this server is running three raid fives
Striped together in Windows of a total of 24 SSDs
So if one of the raid fives drops out in time
Entirely all the data is gone only about 10% of the backup that I was just in the middle of
remaking is
saved
so
It's time to
Investigate. All right. So here's wonix server
So far troubleshooting steps that I've tried you never want to you never want to rebuild something like this if there's a possibility
You're gonna have to go to a data recovery service because that can make it much much worse
so
In terms of troubleshooting steps, I tried transplanting it into another case so that I could use a different
SATA backplane that didn't work. I tried a different power supply a better power supply
That didn't work
None of that fixed it and I even went as far as to put in
Another LSI raid card that I had and try to import the array
But while it did detect my drives all of them as unconfigured good and a foreign array
It was not able to import it
So there is some good news there though while I'm getting firmware errors kicked out by this card right here when I try to boot
All the drives are detected by another card
So hopefully it didn't write too much garbled data to the array
Okay, so this is basically what I've been doing for the last
14 hours is sending emails contacting different
Data recovery services
There's some that are local
there's some that are not local, but I'm not ruling those ones out. In fact, right
now the most likely solution looks like WeRecoverData.com. They've got some
custom tools that they've created that will allow them to SSH into the system
and actually potentially recover the missing RAID 5 and then the
other two RAID 5s and assemble them all together and export them as one
gigantic pile of data to one of our servers, all without me actually having
to send the drives away, which would be pretty ideal. Now obviously an approach
like that would not work in the case of physical damage to the drives, but
because it is only a RAID controller issue, they're saying, hey, there's a shot
at this, so let's give it a try.
So I just
downloaded their remote recovery client thing here. So right now I'm waiting for
their custom Linux-based tool to load on a bootable external SSD so that they can
SSH into the server. I'm not having a good couple days. Right now I can't get
the utility to detect my USB drive. It's called tech support. Oh bloody hell, I just
figured it out.
This is Linus calling, by the way. I just realized I have to run the WRD disk
image as administrator. You guys should put that in the documentation. It never
fails. This is why it's a good idea to call tech support, because the second you
call them, you'll solve the problem on your own.
Alright, so in theory we now have a bootable drive. Okay, so step one is
getting this unnecessary extraneous server out of the way. I'm grabbing a UPS to use.
Now, in a perfect world, I would have a motherboard or an HBA card, a host bus
adapter card, that could allow me to plug in all 24 of the drives for them to do
their data recovery magic on. But what we're gonna have to settle for is having
eight of them plugged into the motherboard at a time as each RAID 5 is
rebuilt. RAID cards do all kinds of funny, funny nonsense and they don't give other
software
direct access to the drives at a bit level in some cases. So we need to remove
all these drives and plug them in directly to the motherboard. Okay, now it
should boot to that, in theory. Let's just make sure all the drives are detected in
the BIOS and everything. Oh, that's a problem. I need to pull out the other
RAID controllers as well, otherwise they're gonna be looking for their
drives and I don't have them powered up right now. Okay. Oh, actually that doesn't
look very good.
DCI-E port errors. What? So let's make sure all of our drives are even showing
up. Uh-oh. That is extremely worrying. Oh, that is extremely worrying. Oh, that's not
good. That means potentially three of the drives in a RAID 5 are dead, which means
this data is not coming back. None of it.
Not this particular RAID?
No, none of it.
None of the...
But I thought we have three RAID 5s.
None of it will be coming back if we lost three drives from one of the RAID 5s.
So let's try moving things around. I've never tested the SATA ports on this board.
It's possible three of them are dead, however extraordinarily unlikely that
would be. Five, six. Okay, we've got six now. Okay, that gives some hope. I'm just
moving SATA ports around. I was just gonna say right now I'm even having difficulty
in this system to post. So I'm at a whole new level of concern. Yeah, I'll let you
know. I'll send an email. Okay. So I'm at the point now where I don't want to put the
drives on this system. Let's go with this one. I didn't like all those PCI Express errors
that the bootable USB was kicking out. I don't like that some of these drives aren't showing
up. I'm starting to wonder if that's what corrupted the LSI card. This royal pain in
the ass.
A thousand things I can think of that I would rather be doing right now. Okay, I got seven
right now. Really? Yeah. Wow, maybe it was bad PCIe. Linus speaking. Well, I can tell
you what's happening so far is I put it on the system that I was running the RAID card
and the drives in and that baby started spitting out PCIe errors all over the place. So I got
everything pulled out of that system and I have plugged all the drives now into a different
disk bench and I'm trying to see because I wasn't even getting all the drives showing
up. Two, three, four, five, six, seven, eight. Okay. Hey, you guys are on the phone for the
magic moment where all eight drives are detected by a new motherboard. So I'm beginning to
think there may be a motherboard issue that caused the firmware on that LSI controller
to corrupt in the first place. So I'm actually pressing enter and I am booting up to your
USB and it is loading correctly on this. I think that board is dead. Okay. Well, that's
I don't know if that makes it worse.
Probably does, doesn't it? You can tell me. I can take it. It says waiting up to 60 more
seconds for network configuration. Good timing with the follow-up call. If you called one
minute earlier, I've been cursing you for dialing my number. So I have to answer my
phone because I was in the middle of thinking only five of my drives were being detected.
You guys know what you're doing, right? I had a local shop that I was going to physically
take it to and they basically, they were like, they kind of freaked out when I said I was
having someone use their remote tool. They were like, well, unless these guys really
know what they're doing.
That's pretty dangerous. And I was like, okay, well I'll, I'll check with them. I'll make
sure they, they know what they're doing. So, so you guys know what you're doing. So basically
they're explaining that they're going to virtually Mount the volumes. So we're not doing any
writing to my discs. Um, so even in the event that we cannot recover it by this means, uh,
that still leaves other options open to us. Okay. They're in. Okay. Well, that's, that's
very wonderful to hear, but I'm trying not to get too excited right now. They sound pretty
confident. This is like the ultimate roller coaster right now. So to put this in the appropriate
context for the viewers right now, one X server has several in progress projects stored on
it that do not have a backup anywhere else, including a fully shot channel, super fun
that involved a thousand dollars in equipment rental, um, including, uh, lots of footage
from Linus tech tips videos. Like it is going to be a huge, huge, devastating loss for us.
To lose this. I mean, it's got a lot of our, our templates for editing, uh, editing videos
for, for scripting templates for employee reviews, like all kinds of stuff. Like this
is our main working server. No, the offsite backup server has not ever been built yet.
The vault was the backup server. If I ever have to like act in a, in a video where I
have to look for Lauren, I'm going to think back to this moment. Okay. So I do need to
reboot it once in order to add that eight terabyte drive. So anyway, right. Okay. Thank
you very much, gentlemen.
I hope to speak to you, uh, with good news very soon, 12 SATA connectors. Ah, ah, ah.
Okay. So let's make sure everything is showing up 1, 2, 3, 4, 5, 6, 7, 8 of those are boot
device and our eight terabyte recovery media. We are ready to rock. Okay. So within about
10 minutes, they contacted me back. They said, okay, yeah, we're ready to have all the drives
in there. So that means we are going to have to do some janky ass.
Right now. So basically what you're looking at here is these drives are going to be powered
off this power supply with a jumper. These cards and all these drives are going to be
powered off of this rig. So now we can power all those drives without powering up this
motherboard or any of that nonsense.
Okay? So we have a grand total of 25 drives connected to this test bench. 16 of which
are running in Bay's.
of this enclosure and eight of which are spread out here.
So there we go.
There's one of my LSI controllers.
There's its eight drives.
There's my other one.
Whoa, where are your virtual drives?
I think I remember a troubleshooting step
where I tried swapping all the drives to different bays
where I was trying to see if it was the back plane
or the cables.
Wow, panic moment there, but it's okay.
So now we should see two virtual drives here on the screen.
That's one set.
Show me the second set.
Oh, I thought we lost another RAID 5 there.
Okay, it's good.
Feeling good.
Reality TV ain't got nothing on IT work.
Show me all the devices.
Two RAID adapters, one eight terabyte drive,
one, two, three, four, five, six, seven,
eight Kingston drives.
Boom.
Okay.
I will let...
the data recovery specialists know.
All right.
So all that's left now then I think,
I got in touch with them and I asked if we can put
a 10 gigabit network card in
so we can accelerate the offloading of all of our files.
So all we're waiting for now
is for them to reboot the machine
and set me up with a network share that I can access
and fingers crossed,
just start pulling off all the footage
and well, everything else.
All right.
So check this out here, around here.
So I've got the instructions
for how to access our data.
So I go into the Tritium share
and there it is.
Bunch of EULAs, a bunch of stuff.
We've got what looks like the file structure.
Let's test it.
How about one that I,
that we haven't finished releasing yet?
Like something from the, the melee battle here.
Let's just, let's watch it.
Hmm.
Okay. XA footage for sure.
Oh, okay.
Here we can copy one of these.
Let's copy one of these clips to my local machine
and see if that helps.
No.
Oh boy.
Let's have a look.
I wonder if everything's even here.
Like I don't even see the Linus tech tips folder.
One and two are missing.
Useful applications is empty.
That should be full of things.
Okay. Well, I guess we'd better get in touch with them.
So thanks for watching.
Subscribe, follow all that good stuff and see you next time.
I just got an email back from them.
Please stand by addressing your emails with the engineer.
It's possible the RAID definitions were changed
from one reboot to the next
because of drive letter assignments in Linux.
It's 26, so it's 26 drive limit.
We have, I think we have more than 26 devices in there.
I got another email back.
Linus, we recreated the block devices.
Please go ahead and check the videos in context.
That you were looking at again.
No way.
Oh, the suspense.
I'm not enjoying this.
Is nothing there?
I'll let them know the chair's empty again.
This is just like the wildest roller coaster ever.
I don't even, yeah.
I don't even have anything else to say actually.
It's almost funny.
I hope to some of you it's funny.
Here's the email I got 17 minutes ago.
I was stuck on a call.
We got everything remounted.
So you can now go ahead and view the data.
Once more.
This is the moment.
Do we not have a two?
I don't think we ever had a two.
Oh, okay.
Okay, so let's go to that Channel Super Fun video
that we were looking at.
Yes, yes, Swagway jousting.
Okay, and it was the delivery.
I'm getting like tingling.
Okay, okay, you gotta check the source files though
because that was what we know was corrupted.
Okay, okay, okay, okay.
So untranscoded footage.
XA, they did it.
It's back.
My God.
It's back, here's another file.
It's back.
Do we have everything?
In theory, yes.
Really?
Holy .
Oh.
I think it's finally over.
Woo hoo.
That is fantastic.
Hey guys, guess what happened with the recovery?
We got everything back.
Oh, thank goodness.
Woo hoo.
Like it's just been such a stressful few weeks
and that was just such a huge, huge thing.
All of our pitches and proposals and stuff are on that server.
Okay, Wanik's back.
Bully?
Really?
Everything is back as far as I can tell.
I have played back video files.
They are not corrupted.
Wanik is officially back.
Swagway jousting footage for the melee battle should be fine.
And we'll have a better backup scheme in the future, I promise.
That is all.
Yay.
Video shot.
We should physically print all of our videos, every frame of every video on that paper.
I don't think so.
Did you hear the good news?
That Wanik's back?
Wanik is back.
Yay for hiring other people to do our jobs.
No, I don't think it's back, but it is.
It's back.
What are you doing?
I'm just letting everyone know that Taren, it's back.
The server is back.
It's back.
The server is back.
Okay?
So it's not always that our sponsors tie so perfectly into our content, but today it happened.
Today's sponsor is Rackspace, the top tier managed cloud computing company.
They pride themselves on best in class service across all platforms.
They've got over 300,000 customers in 120 countries with 10 worldwide data centers.
They've got Red Hat, Cisco, Microsoft, VMware certifications, and all of that amounts to,
whether you're running a small business or a billion dollar enterprise, whatever your
needs are in terms of capacity storage or flash-based high performance storage, they
can take care of it without you going gray or bald because you did something wrong and
screwed up a backup and went and lost a RAID array or whatever else.
They've got dedicated storage to meet your performance, security, network capacity, and
compliance needs.
Everything from direct attached storage, so that gives you the flexibility and scalability
of just a simple, okay, let's attach some drives to this thing, it's redundant, cool,
off you go to the races, to SAN, so that's high availability and reliability, fully redundant
for business, to even NAS with support for demanding workloads like virtualization, file
sharing, and rich media.
In fact, their NAS stuff can scale up to 20 petabytes of capacity.
They've got public cloud and private cloud options.
As well, you can get your own server, a scalable private cloud in the data center, in your
data center, in their data center, and it's all supported by Rackspace and VMware.
They call their support fanatical because these people are available 24 hours a day,
seven days a week, 365 days a year, and they've got industry leading service level agreements,
both managed and intensive.
So if you need dedicated storage, they got you covered.
Go to rackspace.com.
de to learn more, and not go through what I did.
I just, yeah, I don't really have anything to say
Thanks for watching guys.
If you just like this video, then hit the dislike button, I guess.
But come on.
This is about as real as it gets around here.
If you liked the video, hit that like button.
Get subscribed, maybe even consider checking out our Amazon affiliate, instructions for
how to use it.
Whenever you buy stuff there, instructions are up there.
Buy and share with worksheets.
That's it.
cool shirt like this one or even giving us a monthly contribution through our
forum which gives you a little contributor badge now that you're done
doing all that stuff hey maybe you want to check out that Channel Super Fun
video that we just discovered is back and that'll definitely be worth your
while we joust with each other on swagways it's pretty awesome so see you
next time