1
00:00:00,320 --> 00:00:06,080
it's all here look at this it's been

2
00:00:03,439 --> 00:00:12,240
almost a year since we discovered the major issue that effectively put our

3
00:00:08,639 --> 00:00:14,480
entire video archive server offline

4
00:00:12,240 --> 00:00:17,600
that's right all the full quality footage since i was still working at

5
00:00:16,000 --> 00:00:22,800
ncix gone temporarily after the painstaking

6
00:00:20,720 --> 00:00:28,640
process of recovering nearly all that data love you wendell from level1tx we

7
00:00:25,680 --> 00:00:33,600
finally have all the hardware in place so today's the day we get our vault back

8
00:00:31,519 --> 00:00:40,000
up and running we're talking sprinting we're talking better than before we're

9
00:00:35,600 --> 00:00:41,680
talking an entire 3.6 petabyte array

10
00:00:40,000 --> 00:00:47,520
managed by one single head server instead of the five

11
00:00:44,640 --> 00:00:52,480
servers that we have now and i promise it's gonna be awesome

12
00:00:49,520 --> 00:00:56,480
like the segway to our sponsor i fix it looking to expand your selection of

13
00:00:54,000 --> 00:01:03,039
tools ifixit has you covered learn more about the manta toolkit and its 12 what

14
00:00:59,760 --> 00:01:04,960
112 steel bits what with two premium

15
00:01:03,039 --> 00:01:08,920
drivers learn all about it at the end of this video manta

16
00:01:16,320 --> 00:01:23,759
this is the one computer that is going to run the mother vault and the astute

17
00:01:21,439 --> 00:01:30,560
among you have probably already realized that we are not going to be able to put

18
00:01:26,560 --> 00:01:31,759
3.6 petabytes worth of storage inside it

19
00:01:30,560 --> 00:01:37,920
unless it's from hogwarts it's magic

20
00:01:35,360 --> 00:01:42,240
wingardium leviosa in here no it's one guardian leviosa oh sorry excuse me no

21
00:01:41,200 --> 00:01:46,640
in fact there are no hard drives in here this is

22
00:01:44,720 --> 00:01:51,840
where the magic happens the machine you saw just now that's only the brain so

23
00:01:49,520 --> 00:01:56,799
instead of trying to put 3.6 petabytes of hard drives into there they're gonna

24
00:01:54,000 --> 00:02:02,960
go in separate enclosures like this one called j buds or just a bunch of disks

25
00:02:00,399 --> 00:02:08,239
and you can think of an external jbot enclosure kind of like an external hard

26
00:02:05,759 --> 00:02:14,319
drive dock except instead of one hard drive they hold up to a whopping 90 hard

27
00:02:12,160 --> 00:02:19,040
drives each we'll talk more about these a little bit later

28
00:02:15,760 --> 00:02:21,120
just because the 2024 us-trt from

29
00:02:19,040 --> 00:02:26,959
supermicro is shirking its storage responsibilities doesn't mean that it's

30
00:02:23,360 --> 00:02:29,920
not packing inside here we find a dual

31
00:02:26,959 --> 00:02:34,239
socket AMD epic milan system that honestly has a bit of an unusual

32
00:02:31,760 --> 00:02:37,680
configuration typically a system with this much CPU horsepower so those are

33
00:02:36,800 --> 00:02:44,720
two 24 core AMD epic 74 f3s and the ability

34
00:02:42,319 --> 00:02:51,599
to run up to dual 64 cores would be decked out with a ton of NVMe slots or

35
00:02:48,319 --> 00:02:54,480
some enterprise gpus or something like

36
00:02:51,599 --> 00:02:57,920
that but this guy only has four

37
00:02:55,280 --> 00:03:02,159
NVMe slots and these are an optional accessory the rest of the storage up

38
00:03:00,080 --> 00:03:06,239
here and in fact these four slots by default are i kid you not

39
00:03:04,959 --> 00:03:11,840
six gigabit SATA three and a half inch slots

40
00:03:09,360 --> 00:03:15,440
like that you would use for a hard drive that's because this system is actually

41
00:03:13,599 --> 00:03:19,519
designed with a different purpose in mind say for example uh as a

42
00:03:17,680 --> 00:03:23,599
virtualization server where you might need a ton of CPU that's going to be

43
00:03:21,360 --> 00:03:28,080
split up to different clients or if you're like us a ton of CPU that you're

44
00:03:26,400 --> 00:03:32,959
going to use to drive a boatload of connected storage that

45
00:03:30,000 --> 00:03:38,480
isn't inside the chassis so instead of using up all those express lanes from

46
00:03:35,360 --> 00:03:40,599
those epic cpus which is

47
00:03:38,480 --> 00:03:47,680
buddy 128 lanes each with like NVMe drives in

48
00:03:44,720 --> 00:03:53,360
the front we instead want them allocated to PCIe slots in the back of the server

49
00:03:50,480 --> 00:03:58,640
back there we could install high-end networking adapters what is this dual

50
00:03:55,200 --> 00:03:58,640
100 gigabit this guy right here

51
00:03:58,799 --> 00:04:06,879
or the other piece of our puzzle host bus adapters just like with an external

52
00:04:04,000 --> 00:04:11,760
hard drive our jbods back there which the keynight among you will have noticed

53
00:04:08,480 --> 00:04:14,799
that there are two require some kind of

54
00:04:11,760 --> 00:04:17,120
connection to the actual server with the

55
00:04:14,799 --> 00:04:20,959
actual CPU in it and the actual operating system running on it in a

56
00:04:18,720 --> 00:04:25,360
modern jbod this is usually handled by mini sas hd cables like these ones from

57
00:04:23,919 --> 00:04:29,680
infinite cables we're going to have these linked down below

58
00:04:27,120 --> 00:04:36,000
so each of these ports on the back of our three cards can handle four lanes of

59
00:04:32,960 --> 00:04:38,560
12 gigabit sas that's the enterprise big

60
00:04:36,000 --> 00:04:43,680
brother of SATA for a total of 48 gigabits per second per cable your

61
00:04:41,280 --> 00:04:48,880
average computer even server probably doesn't have any factory mini sas hd

62
00:04:46,880 --> 00:04:54,960
ports so we're going to be adding them to this machine with three of broadcom's

63
00:04:51,600 --> 00:04:57,360
9500 16e host bus adapters that means

64
00:04:54,960 --> 00:05:01,520
we'll have a full dedicated card for each of our j-bonds oh yeah there are

65
00:04:59,840 --> 00:05:04,800
three of these right there's one more j-bud that's actually already in the

66
00:05:03,120 --> 00:05:09,680
server room we will show you guys that one later now each of our jbods allows

67
00:05:07,280 --> 00:05:13,680
us to connect up to three cables for increased throughput so that's exactly

68
00:05:11,600 --> 00:05:18,320
what we're going to do on the subject of throughput to push this thing to the

69
00:05:15,680 --> 00:05:23,280
absolute limit we want to pack this machine with as much RAM as possible

70
00:05:21,600 --> 00:05:28,479
since the file system that we're using called zfs can use RAM as a read caching

71
00:05:26,880 --> 00:05:32,639
layer to significantly improve the responsiveness of the array because

72
00:05:30,560 --> 00:05:38,000
mechanical drives on their own yeah even with this many of them not

73
00:05:35,440 --> 00:05:42,560
that great in terms of access times luckily the folks over at micron just

74
00:05:40,960 --> 00:05:48,880
sent over one terabyte of 3200 megatransfer per second

75
00:05:45,520 --> 00:05:51,199
registered ecc memory which last time i

76
00:05:48,880 --> 00:05:55,600
checked smells like performance but the

77
00:05:53,199 --> 00:06:01,680
performance improvements don't stop there we had some

78
00:05:57,759 --> 00:06:04,160
leftover keoxia 30 terabyte NVMe drives

79
00:06:01,680 --> 00:06:08,479
from the dream server build sorry and we figured they would be the perfect

80
00:06:06,400 --> 00:06:14,319
choice to configure as a special metadata device in our zfs pool this is

81
00:06:12,080 --> 00:06:20,479
a game changer for performance because it allows metadata like where the files

82
00:06:17,680 --> 00:06:26,400
are physically stored on those hundreds of hard drives the allocation tables and

83
00:06:23,520 --> 00:06:30,400
optionally even small complete files to be stored on a discrete higher

84
00:06:28,720 --> 00:06:35,280
performance device like these NVMe drives this can make

85
00:06:33,120 --> 00:06:40,800
navigating through large data structures like the thousands of video projects

86
00:06:37,600 --> 00:06:43,360
that we're archiving much much faster

87
00:06:40,800 --> 00:06:47,199
and that directly results in workflow improvements for our staff i'd say it's

88
00:06:45,680 --> 00:06:53,280
about as much of a quality of life improvement as our flow plane subscribers get with no baked in ads

89
00:06:51,120 --> 00:06:57,600
higher quality streaming and exclusive behind the scenes content we're going to

90
00:06:54,720 --> 00:07:03,199
be setting these up in a mirrored stripe leaving us with a total of 60 terabytes

91
00:07:01,280 --> 00:07:07,919
of capacity which is hilarious given that it's a raid 10 of just four drives

92
00:07:05,120 --> 00:07:13,919
which we think should be an ample amount of fast metadata storage for our 3.6

93
00:07:11,199 --> 00:07:17,520
petabyte array unfortunately because we only have these four NVMe slots we're

94
00:07:16,160 --> 00:07:20,960
not going to be able to expand that storage unless higher capacity drives

95
00:07:19,280 --> 00:07:25,840
come along in the future and the performance compared to some of our

96
00:07:22,639 --> 00:07:28,720
other all NVMe arrays is not going to be

97
00:07:25,840 --> 00:07:33,280
that amazing another problem is that this doesn't leave us any additional

98
00:07:30,639 --> 00:07:38,400
space for NVMe storage drives for a level 2 arc reed cache or an intent log

99
00:07:36,639 --> 00:07:44,080
at least at the front but there's a solution to that huh yes

100
00:07:41,440 --> 00:07:50,479
this cute little half height PCIe card from liquid allows us to install four

101
00:07:47,280 --> 00:07:52,639
m.2 drives in a single slot

102
00:07:50,479 --> 00:07:57,599
inside the server and this will be absolutely perfect for that roll what's

103
00:07:55,599 --> 00:08:00,960
the total capacity of this bad boy i don't actually know what ssds are in

104
00:07:59,120 --> 00:08:04,960
there we don't know what's in it there is one we have that is four or four

105
00:08:02,639 --> 00:08:08,319
terabyte ssds okay so it might be this one it could be this one it could also

106
00:08:06,800 --> 00:08:11,759
not be this one great the problem for later Jake this is

107
00:08:10,319 --> 00:08:16,639
pretty cool Jake did you design this three and a half to two and a half inch adapter thing no we found it on

108
00:08:14,319 --> 00:08:20,560
thingiverse oh cool printed it shut out thingiverse whoa

109
00:08:18,560 --> 00:08:23,440
what well that came out which slot did you want the liquid card in uh you gotta

110
00:08:22,240 --> 00:08:27,599
change this guy and then put it right here oh man can we

111
00:08:25,440 --> 00:08:30,240
find out if it's the right one before we do this because we're going to have to

112
00:08:29,120 --> 00:08:36,479
pull the whole thing off what do you mean what's the right one the the high capacity one oh

113
00:08:33,919 --> 00:08:40,360
no oh come on there's got to be a test bench not far

114
00:08:41,680 --> 00:08:47,200
no there's none over there there's none over there Jake says there is none well

115
00:08:45,680 --> 00:08:51,839
let's just go to my bench upstairs logistics

116
00:08:50,080 --> 00:08:55,839
i think it's important for you guys to know that Jake thinks you guys are

117
00:08:53,680 --> 00:09:00,640
utterly incompetent that's what he said whoa those are his words not mine he

118
00:08:58,480 --> 00:09:05,760
said those guys don't even have a test bench over there at all not one

119
00:09:03,120 --> 00:09:11,120
yeah that's what i thought yeah yeah Jake mere minutes later all right hey we

120
00:09:08,720 --> 00:09:14,640
got a foreign dynamic disc here these are what three terabytes

121
00:09:15,120 --> 00:09:19,680
okay right click on that one let's see if it's

122
00:09:18,240 --> 00:09:23,120
toshiba three terabytes 84. yeah perfect these

123
00:09:21,680 --> 00:09:26,959
are the four terabyte ones thank you fast and competent logistics team i just

124
00:09:25,279 --> 00:09:31,360
want you guys to know that i defended you i was the first to say no i would

125
00:09:29,279 --> 00:09:35,600
never use those words about our logistics team

126
00:09:32,800 --> 00:09:35,600
not like Jake

127
00:09:37,279 --> 00:09:42,320
goodbye get out of here

128
00:09:40,399 --> 00:09:46,480
ah here's what we've got next now i mentioned that this was 100 gigabit but

129
00:09:44,320 --> 00:09:53,120
what i didn't say i don't think is that it has dual 100 gigabit ports it can do

130
00:09:49,200 --> 00:09:56,399
a total of 200 gigabit it's PCIe gen 4

131
00:09:53,120 --> 00:10:00,399
by 16 that is a flipping ton of

132
00:09:56,399 --> 00:10:02,480
bandwidth and it uses these sick qsfp28

133
00:10:00,399 --> 00:10:06,399
connectors while Jake orders his lunch i'm going to go ahead and get our other

134
00:10:04,000 --> 00:10:10,800
two hbas installed here i'm getting mucho burrito editor you better put this

135
00:10:08,480 --> 00:10:13,600
in the goddamn cut okay i'm just gonna put this in because

136
00:10:12,000 --> 00:10:17,120
you're taking too long what do you mean i'm taking too long i'm not the one

137
00:10:15,200 --> 00:10:24,079
sitting here ordering lunch touche this is amazing so we've got our

138
00:10:20,480 --> 00:10:26,800
three hbas we've got our 100 gig network

139
00:10:24,079 --> 00:10:31,279
card we've got our additional SSD which is somewhere in the neighborhood of

140
00:10:28,079 --> 00:10:34,240
around 15 terabytes of storage for our

141
00:10:31,279 --> 00:10:38,160
slog and level 2 arc well they're 3.84 each oh i see so like somewhere in that

142
00:10:36,160 --> 00:10:40,800
15 point something oh

143
00:10:39,200 --> 00:10:44,720
and how do you forget about that i forgot about these what i forget 120

144
00:10:42,720 --> 00:10:48,880
terabytes of NVMe in the front yeah yeah are we ready to turn it on i'm

145
00:10:46,959 --> 00:10:52,880
turned on okay i could be more turned on if i was

146
00:10:50,560 --> 00:10:57,519
chilling for lttstore.com hey check out the RGB diode shirt it's like based on

147
00:10:55,760 --> 00:11:01,760
what the individual sub pixels look like but then it's also just a cool design so

148
00:10:59,440 --> 00:11:06,240
do you want the top yeah do a peel very graceful Jake you're doing a fan oh

149
00:11:04,959 --> 00:11:12,399
oh that's right i'm a monster

150
00:11:09,760 --> 00:11:16,880
this thing is sick do we get to power it on now got a laptop a laptop

151
00:11:15,120 --> 00:11:20,880
uh high this one is easy just wait it's actually

152
00:11:18,880 --> 00:11:25,519
really quiet give it a little bit of time are you

153
00:11:22,000 --> 00:11:28,519
gaslighting me sir no i would never

154
00:11:25,519 --> 00:11:28,519
bmw

155
00:11:31,920 --> 00:11:37,680
oh that is so much better and can we turn off that alarm as well i have a

156
00:11:36,160 --> 00:11:42,000
solution i don't think that's going to work it does pulling the power supply

157
00:11:39,519 --> 00:11:46,959
out right makes the alarm for oh wow it didn't work it worked this is a 1600

158
00:11:44,880 --> 00:11:52,000
watt power supply what i mean to be clear that's only a 200 volt input it's

159
00:11:49,440 --> 00:11:57,320
a thousand watts if you're from 100 227 volts

160
00:11:57,680 --> 00:12:03,600
that's what you wanted to do i want to go fast

161
00:12:01,200 --> 00:12:09,839
you said it was quiet so i thought full fan speed must not be that bad right i

162
00:12:06,560 --> 00:12:12,480
had him log me separately into the ipmi

163
00:12:09,839 --> 00:12:17,200
just so i could do that in other news it's booted we got our terabyte of RAM

164
00:12:15,360 --> 00:12:20,560
we got our oh yeah we got our melanox card in there that's good did you put

165
00:12:18,639 --> 00:12:24,610
the nvmes in the right slot i mean one of them is wrong what do you

166
00:12:22,079 --> 00:12:27,839
mean wrong like oh it's wrong

167
00:12:27,839 --> 00:12:33,200
try the bodies one of those is a boot drive might it

168
00:12:32,079 --> 00:12:37,440
might be good there was an activity flash yeah right

169
00:12:35,279 --> 00:12:40,880
okay it's cool and look because the boot device

170
00:12:38,480 --> 00:12:44,000
is a pool just immediately fine i mean it's

171
00:12:42,560 --> 00:12:49,440
probably mad actually this is going to be kind of complicated because some of the drives we like need to put in the

172
00:12:47,360 --> 00:12:53,760
right spot and not mix up because some of the arrays we're keeping and some of

173
00:12:51,600 --> 00:12:57,839
the drives are in another jbod that wendell's working on right now so we

174
00:12:55,120 --> 00:13:01,040
can't really touch those ones uh-huh it's gonna be fun let's go start

175
00:12:59,279 --> 00:13:05,600
carrying how am i supposed to move this this is a four-man carry it literally

176
00:13:03,519 --> 00:13:09,279
has a label on it that says four people need to keep it actually yeah dude these

177
00:13:07,760 --> 00:13:13,680
are like the best earmuffs put them on oh wow now with 80 percent

178
00:13:11,839 --> 00:13:16,720
less Jake i can endorse these

179
00:13:15,120 --> 00:13:20,959
we don't actually have to take all of these out we could just take this one

180
00:13:19,200 --> 00:13:24,320
and that one and then just move this so the head server will just go here

181
00:13:23,120 --> 00:13:29,440
and then we'll go one so we don't even have to move this one so we just have to take the temp vault out should i let the

182
00:13:27,519 --> 00:13:32,639
editors know temp fault's going away uh i already told them it was going to

183
00:13:30,639 --> 00:13:35,680
happen today how do you tell them it's happening now i'll tell them what's

184
00:13:33,760 --> 00:13:40,000
happening now hello hardworking and truly valued

185
00:13:37,839 --> 00:13:47,639
editing team the vault that you have come to know and love temp vault

186
00:13:42,639 --> 00:13:47,639
is now deadvault press f to pay respects

187
00:13:48,560 --> 00:13:54,720
they don't care at all all right bye kind of raises the question do they

188
00:13:53,519 --> 00:13:57,920
need a vault at all then why don't we just have no vault then forget it

189
00:13:56,639 --> 00:14:00,079
if you guys aren't even going to miss it then

190
00:14:00,399 --> 00:14:07,920
what do you mean you copied the files you guys aren't supposed to keep things on your local machines we copied it to

191
00:14:04,720 --> 00:14:09,760
the active server new new new new monik

192
00:14:07,920 --> 00:14:13,680
hopefully this clears everything up and is not confusing at all i think they're

193
00:14:11,279 --> 00:14:18,639
more confused look somebody is reading files from it

194
00:14:16,160 --> 00:14:23,440
right now like ed was lying look oh somebody's is reading from it or

195
00:14:21,600 --> 00:14:28,320
they're writing to it what are they doing oh my god

196
00:14:25,360 --> 00:14:33,279
at 350 megabytes a second this is a not insignificant data transfer

197
00:14:31,279 --> 00:14:36,560
the chart's like nothing and then somebody's like

198
00:14:35,040 --> 00:14:41,440
you want to shut it off now okay who's messing with the server are you messing

199
00:14:38,959 --> 00:14:44,240
with us ed somebody is vaulting right now

200
00:14:42,320 --> 00:14:46,480
did you copy things to the vault right now

201
00:14:47,440 --> 00:14:55,120
probationary employee number 17. you cut and paste it

202
00:14:52,639 --> 00:14:58,160
i should not i use the tara terra copy

203
00:14:56,639 --> 00:15:03,600
but it's still going it's still kind of taste with territory

204
00:15:00,959 --> 00:15:08,160
dang it guys it's such a good you're done you're good

205
00:15:05,600 --> 00:15:14,079
god damn it ignore the the discs with errors uh what there's um we've lost a

206
00:15:11,680 --> 00:15:19,040
few the original vaults 10 terabyte drives are still like in production and

207
00:15:16,720 --> 00:15:23,360
they're like six years old now and we've had problems too our cooling in that

208
00:15:21,760 --> 00:15:27,040
room didn't used to be super stable there were times when you would walk in

209
00:15:24,959 --> 00:15:31,839
there and it was like a wall of heat yeah because we had this

210
00:15:29,760 --> 00:15:36,639
air conditioning unit that would just whatever when we didn't have an air conditioner

211
00:15:34,560 --> 00:15:39,440
yeah yeah at one point i remember when i started here atlantis was like no we

212
00:15:38,000 --> 00:15:44,160
don't need it we don't need it and then all the ups exploded because it was too

213
00:15:41,920 --> 00:15:46,880
hot and he was like okay fine how are we going to move these

214
00:15:45,279 --> 00:15:50,399
fortunately these aren't really rails they're actually like shelves so all we

215
00:15:48,880 --> 00:15:54,800
really need to do is kind of put it on the ground and then just

216
00:15:52,800 --> 00:15:58,399
tilt one end and then slide it in and tilt the other end that's how we put the

217
00:15:56,560 --> 00:16:04,800
other one in there installation by two persons is recommended if these need two

218
00:16:01,279 --> 00:16:06,800
people that server needs 25.

219
00:16:04,800 --> 00:16:10,000
i think i can handle it i wonder when my burrito's going to arrive all the drives

220
00:16:08,800 --> 00:16:14,399
are going to have to come out of this thing before we lift it out no no it's

221
00:16:12,000 --> 00:16:19,600
fine how are we supposed to lift this we can lift it we've done it okay well how

222
00:16:17,199 --> 00:16:22,720
it's drunk no really though it's not off the ground i just use my

223
00:16:20,959 --> 00:16:26,560
feet like this usually your feet yeah and then you just walk it up walk it out

224
00:16:24,800 --> 00:16:30,800
yeah three two i'm holding the heavy side

225
00:16:27,839 --> 00:16:34,399
don't drop it okay i didn't drop it wow you just put all the weight on my what

226
00:16:32,320 --> 00:16:38,320
was on your foot cool well that's all the hard work yeah now

227
00:16:36,079 --> 00:16:42,959
that we've got lab two over there we might actually have a purpose for all

228
00:16:40,000 --> 00:16:48,079
these additional storinators oh man i haven't gone spelunking in the uh server

229
00:16:46,000 --> 00:16:51,759
rack in ages Jake there's so much treasure in here you're

230
00:16:49,759 --> 00:16:55,519
giving everyone quite the view right now whatever usually you got to pay extra

231
00:16:53,519 --> 00:16:59,680
for that onlyplanes.com how do you like this setup i'm taking a

232
00:16:57,440 --> 00:17:04,720
little server cabinet nap here got my little uh pad on the back of my head

233
00:17:01,920 --> 00:17:07,839
okay bottom of the fifth not bad

234
00:17:05,839 --> 00:17:12,880
this was definitely the best way to do this the front of this shelf is out

235
00:17:10,640 --> 00:17:17,280
yeah that's what it appears uh that's why we screw

236
00:17:14,799 --> 00:17:22,000
how full is this uh it's got 60 drives in it okay what seems to have happened

237
00:17:20,000 --> 00:17:26,160
with this server because we didn't screw the shelf in from the front is this

238
00:17:24,400 --> 00:17:31,360
little tooth here that holds it into place seems to have slipped so it's

239
00:17:29,600 --> 00:17:37,679
sagging down like that which means that a ton of this weight is

240
00:17:34,720 --> 00:17:41,760
being borne by a part that's really not designed for that complicates our

241
00:17:40,080 --> 00:17:45,600
operation here a little bit because we can't just pop it back into place from

242
00:17:43,840 --> 00:17:49,039
here we can't really get at it looks like we need to shut this down

243
00:17:47,600 --> 00:17:53,320
pull it off put it back into place screw it in and

244
00:17:50,799 --> 00:17:58,960
then put it back in which sucks if i take the weight and you slide

245
00:17:57,200 --> 00:18:02,880
it forward or the other way around here i'll use my leg up there we go it's up

246
00:18:01,200 --> 00:18:08,480
just put the put the rail in don't move wait put what rail in

247
00:18:05,760 --> 00:18:11,600
well i can't see it oh my god like this oh i think we got it i think we got it

248
00:18:09,760 --> 00:18:15,440
let go let go yeah this actually went pretty okay

249
00:18:13,280 --> 00:18:19,440
situations like this are when i'm so glad we developed this screwdriver i got

250
00:18:17,760 --> 00:18:25,520
the nice strong magnet holding it in place almost no

251
00:18:22,080 --> 00:18:26,559
back force required to spin the ratchet

252
00:18:25,520 --> 00:18:30,880
like is it in there easy yeah it's in oh my

253
00:18:28,799 --> 00:18:35,200
god wait what how are we supposed to get it from here down to there we got it up

254
00:18:32,880 --> 00:18:38,320
there somehow so it's just like a really heavy storenator why don't we just put

255
00:18:37,039 --> 00:18:43,679
the other one on top that says this cart can take 600 pounds no we're not doing that okay

256
00:18:41,120 --> 00:18:48,320
that's right what are the odds uh that it just goes right in

257
00:18:46,160 --> 00:18:52,160
no the angle is still too much okay i'm getting under it boys put the right side

258
00:18:50,640 --> 00:18:58,000
in a bit more okay good good here oh my god it's it yeah it's on it's on the top

259
00:18:54,559 --> 00:18:58,720
okay go ahead you can push it in

260
00:18:58,000 --> 00:19:03,600
hi all right now for the hard one yeah

261
00:19:01,360 --> 00:19:07,919
this is more of a Linus sized server yolo

262
00:19:06,000 --> 00:19:11,840
well it's easy to yolo when somebody else already did all the work yeah good

263
00:19:10,480 --> 00:19:17,520
work Jake there goes the brain and what's really

264
00:19:14,320 --> 00:19:22,080
wild is that if we want more capacity

265
00:19:17,520 --> 00:19:24,000
more than 90 180 270 drives with the

266
00:19:22,080 --> 00:19:26,799
cards we have in now we could do another one and then i think there's room for

267
00:19:25,360 --> 00:19:30,320
one more card right i would say right now with no hbas we

268
00:19:28,799 --> 00:19:34,559
could probably add two and it would still be pretty good performance

269
00:19:32,080 --> 00:19:42,240
um all right let's uh let's put in the hard one ready one two three no yes

270
00:19:40,320 --> 00:19:46,320
what weak all right still weak now that they're in

271
00:19:44,480 --> 00:19:52,640
there working on these is glorious you just press the buttons

272
00:19:49,840 --> 00:19:56,160
pull them out boom funny story when we first put this in i was like ah it's so

273
00:19:54,880 --> 00:20:00,799
heavy it doesn't really need to be bolted down does it i was told yes in

274
00:19:58,559 --> 00:20:04,720
fact it does need to be bolted down and this

275
00:20:01,840 --> 00:20:10,240
is why because if we had a handful of these hanging off the front of it and it

276
00:20:07,360 --> 00:20:11,919
wasn't bolted into the concrete

277
00:20:11,919 --> 00:20:16,960
we'd have a bit of a problem bit of an update here we shucked all those drives

278
00:20:15,360 --> 00:20:20,799
put them in the servers i plugged a bunch of stuff in some power and we have

279
00:20:19,120 --> 00:20:23,760
blinky lights now do i get to press the button yeah it's on the right side give

280
00:20:22,160 --> 00:20:28,000
them a sec everybody in the next square mile is going to hear this they're just

281
00:20:25,600 --> 00:20:31,120
they're they're spinning up boys wow it's actually not as bad in here

282
00:20:29,840 --> 00:20:35,039
come outside let's uh let's shut the door yeah sure

283
00:20:33,360 --> 00:20:37,600
you can hear how awful it is still through the door

284
00:20:38,799 --> 00:20:44,640
oh yeah we wouldn't be able to shoot with those things going

285
00:20:42,240 --> 00:20:47,440
but they don't go okay that's fine oh you just got to add another layer of

286
00:20:46,240 --> 00:20:51,760
insulation there's no space Jake well this is the

287
00:20:50,080 --> 00:20:57,120
backup plan right because there's a thermostat in here that will start up a

288
00:20:54,799 --> 00:21:00,960
fan that exhausts out the top and will pull air passively through this grill in

289
00:20:59,200 --> 00:21:05,039
the event that the air conditioning completely fails do you remember that

290
00:21:02,960 --> 00:21:08,799
the shroud for this thing i made that in metal shop when i was still in high

291
00:21:06,400 --> 00:21:12,080
school that's hilarious okay that was it it's up so far only 20 drives are

292
00:21:10,640 --> 00:21:16,880
showing up but Jake tried a thing which is a what a

293
00:21:14,159 --> 00:21:20,640
domain change we changed some cables and i changed the jbods from three zone to

294
00:21:19,520 --> 00:21:27,840
one zone are you going to click it yes

295
00:21:23,679 --> 00:21:27,840
okay thank god okay one of these

296
00:21:28,280 --> 00:21:34,480
145 drives god damn brother i mean 45 drives is

297
00:21:33,520 --> 00:21:39,520
nice but 145 drives

298
00:21:36,720 --> 00:21:43,520
now that's walking around hard dress i'm just going to try this it's probably

299
00:21:41,039 --> 00:21:46,480
a bad idea oh i'm just going gonna do it i'm gonna try to import the pools it

300
00:21:44,960 --> 00:21:49,760
should find both of them there's two discreet pools on those existing drives

301
00:21:48,400 --> 00:21:54,960
yeah and there's a bunch of extra drives so let's see what happens what are the odds this is just gonna be like yeah

302
00:21:52,640 --> 00:21:58,480
your ancient ass pool and just well it's not they're not that ancient i think

303
00:21:56,240 --> 00:22:04,159
they've all been recreated now okay but actually the crazy thing is i took like

304
00:22:01,280 --> 00:22:08,559
2015 pools from a version of zfs that was not recommended for production and

305
00:22:06,400 --> 00:22:13,919
had not been updated and i put it in like 2022 ass trunas

306
00:22:11,760 --> 00:22:18,320
it just imported it just upgraded for some reason it was super slow like

307
00:22:15,360 --> 00:22:21,280
scrubs would take like a week which is really weird yeah but there could be

308
00:22:19,760 --> 00:22:25,440
kind of flaky drives it wasn't the drives the drives were fine i moved all

309
00:22:23,120 --> 00:22:28,799
the data off deleted the pool made it and then brought it back and it was fine

310
00:22:26,799 --> 00:22:33,440
it might just be like a fragmentation thing or

311
00:22:30,080 --> 00:22:36,320
oh let's see did something and

312
00:22:33,440 --> 00:22:41,840
oh look at that go fast is the 12 drives i think got it i wish i saved these ids

313
00:22:38,640 --> 00:22:44,000
before i did this let's see what happens

314
00:22:41,840 --> 00:22:48,640
does this mean vault is just going to be like well i'm back baby temp vault will

315
00:22:46,480 --> 00:22:51,679
be back the read-only new vault because we're still recovering a few things

316
00:22:49,840 --> 00:22:55,440
that's why one of the jbods is still mia we'll be back just like that and i can

317
00:22:53,679 --> 00:23:01,520
just recreate new shares i mean the ip will be different but i can just recreate the shares

318
00:22:58,880 --> 00:23:05,120
zfs is cool man don't put that there this is a really nice shirt though oh is

319
00:23:02,880 --> 00:23:08,559
that from ltdstore.com it sure is it will be oh is it not out yet i don't

320
00:23:07,039 --> 00:23:14,000
know i don't think so have you announced it on rancho i don't think so 30 hours

321
00:23:11,039 --> 00:23:17,520
later wow the pool's finally imported holy crap

322
00:23:16,000 --> 00:23:22,080
super micro massive shout out thank you for sending

323
00:23:19,360 --> 00:23:26,960
over the gear it's amazing we did have a little bit of trouble with setting it up

324
00:23:24,480 --> 00:23:31,120
with three domains yeah it's for some reason they're only working in single

325
00:23:29,120 --> 00:23:34,159
zone mode that means all 90 drives are going over one cable i don't know why

326
00:23:32,960 --> 00:23:38,400
because we had it set up on the bench like a couple weeks ago with three zones

327
00:23:36,240 --> 00:23:41,360
anyway it's accessible we're ready to copy some files and hopefully that's

328
00:23:40,000 --> 00:23:46,080
something they can sort out either through helping us set up the configuration or through firmware yeah

329
00:23:44,080 --> 00:23:49,200
it's probably just we're missing something this is one of those solutions

330
00:23:47,919 --> 00:23:54,559
where it's kind of like you just know what you're doing if you're doing it you're an integrator who

331
00:23:51,840 --> 00:24:00,159
sets these things up every day week after week and that's not us but i

332
00:23:57,520 --> 00:24:04,320
copied here i copy the file let's do it look i'm copying a file from it to this

333
00:24:02,400 --> 00:24:08,159
like hundreds of megabytes a second it's not the most impressive scenes ever this

334
00:24:06,400 --> 00:24:12,159
is a SATA SSD and we're screen recording to it now so

335
00:24:09,760 --> 00:24:17,919
it's like ah but hey remember guys this is with

336
00:24:15,200 --> 00:24:24,559
ancient like degraded degraded-ass fragmented v-devs once we strip all the

337
00:24:22,159 --> 00:24:28,559
data off of them rearrange them reshuffle them set them up fresh and

338
00:24:26,640 --> 00:24:31,440
dump all the data back onto them we should be able to be in the neighborhood

339
00:24:30,000 --> 00:24:35,279
of a Gigabyte a second which is all we're going to have for for sure well yeah but that's all we're going to have

340
00:24:34,000 --> 00:24:39,679
yeah that's all we're going to do network interface it's 100 oh that's

341
00:24:37,279 --> 00:24:42,799
right it is but here look at that 147 drives never in actual production that's

342
00:24:41,279 --> 00:24:47,120
amazing and then we'll add 60 more and it'll be

343
00:24:44,400 --> 00:24:51,360
over 200. and then we'll add 50 more and it'll be 250. and then we'll add this

344
00:24:49,039 --> 00:24:55,120
segway to our sponsor i fixed its manta kit has a 112 steel bits and comes

345
00:24:53,679 --> 00:24:59,760
complete with standard bits like phillips and flathead as well as more

346
00:24:57,120 --> 00:25:04,559
niche options like pentalobe hex and tri-point bits it also includes two

347
00:25:02,080 --> 00:25:08,400
aluminum screwdriver handles a weighty quarter inch drive for full size

348
00:25:06,400 --> 00:25:12,080
applications and a smaller four millimeter for precision work both

349
00:25:10,240 --> 00:25:16,240
drivers feature magnetic bit sockets knurled grips and silky smooth spinning

350
00:25:14,320 --> 00:25:20,559
tops you can even use the toolkit's lid as a built-in storage tray best of all

351
00:25:18,480 --> 00:25:24,159
every ifixit tool is backed by a lifetime warranty so if something breaks

352
00:25:22,080 --> 00:25:30,240
they will replace it for free so visit ifixit.com forward slash LTT and pick up

353
00:25:26,799 --> 00:25:32,320
your manta kit today monta if you guys

354
00:25:30,240 --> 00:25:37,679
enjoyed this video maybe you'll enjoy the one where we tried to deploy wanik

355
00:25:34,559 --> 00:25:38,960
no no that was awful okay sure yeah sure

356
00:25:37,679 --> 00:25:43,840
to watch that or the the previous one where we just looked at jbots because they're cool