1
00:00:00,080 --> 00:00:07,200
yeehaw this is one bucking bronco of a server okay that didn't work but it

2
00:00:04,880 --> 00:00:12,080
doesn't matter because we have a serious problem and this

3
00:00:08,880 --> 00:00:13,920
this is one serious solution

4
00:00:12,080 --> 00:00:18,400
in spite of all the issues we encountered with our archival storage

5
00:00:15,920 --> 00:00:24,800
system dubbed the vault we've actually been able to recover nearly all of the

6
00:00:21,119 --> 00:00:27,920
data spread across the 2.4 petabytes of

7
00:00:24,800 --> 00:00:30,400
raw storage that made it up except that

8
00:00:27,920 --> 00:00:33,760
because of the recovery process and the sheer amount of capacity required to

9
00:00:32,640 --> 00:00:39,920
hold it all that data is now scattered to the high

10
00:00:36,960 --> 00:00:44,719
winds across the five discrete servers that are required to hold it some of

11
00:00:42,320 --> 00:00:49,200
which are getting close to a decade old we had no desire to return to the

12
00:00:46,960 --> 00:00:52,079
admittedly imperfect setup we had before which means

13
00:00:50,320 --> 00:00:58,079
it is time for oh oh mother vault

14
00:00:54,960 --> 00:00:59,039
so called because this is one giant

15
00:00:58,079 --> 00:01:05,519
mother the final deployment is gonna be not one

16
00:01:02,079 --> 00:01:09,200
not two but three of these each capable

17
00:01:05,519 --> 00:01:11,600
of housing nearly two petabytes of hard

18
00:01:09,200 --> 00:01:17,280
drive face storage is it time to bring back holy did it ever go away no it

19
00:01:14,479 --> 00:01:21,200
didn't just like our sponsor crucial don't just work faster work better when

20
00:01:19,439 --> 00:01:25,119
you need to open transfer or download files faster get some great RAM with

21
00:01:23,360 --> 00:01:30,880
less lag time and more efficiency with crucial get your crucial ddr5 RAM today

22
00:01:27,759 --> 00:01:30,880
at the link down below

23
00:01:39,840 --> 00:01:46,399
before we get to the new stuff we're going to have to get you guys up to

24
00:01:43,280 --> 00:01:49,439
speed back in 2015 we built the first

25
00:01:46,399 --> 00:01:52,240
petabyte project or vault cluster it was

26
00:01:49,439 --> 00:01:56,799
composed of two 45-drive 60-base storinator servers holding 120

27
00:01:55,200 --> 00:02:02,240
10 terabyte hard drives giving us a total capacity of 1.2 petabytes raw

28
00:02:00,640 --> 00:02:06,079
these were combined into a single network share using a file system called

29
00:02:04,479 --> 00:02:12,879
glosterfest a few years later we built new vault or

30
00:02:09,039 --> 00:02:16,160
petabyte project 2 which used 75

31
00:02:12,879 --> 00:02:18,480
16 terabyte drives across two more 45

32
00:02:16,160 --> 00:02:22,879
drive storinators again they were joined together with bluster however since

33
00:02:21,040 --> 00:02:27,840
those two clusters were built several years apart we ended up setting them up

34
00:02:25,040 --> 00:02:32,800
as two discrete clusters that meant we had two separate network storage shares

35
00:02:30,560 --> 00:02:36,480
which you can kind of think of as having two separate drives in your system so if

36
00:02:35,120 --> 00:02:39,440
you want to find a file and you don't know which drive it's on chances are

37
00:02:38,319 --> 00:02:44,319
you're going to have to search through them separately this was fine at the

38
00:02:42,480 --> 00:02:48,640
time because we could usually guess which server a project was on based on

39
00:02:46,640 --> 00:02:52,720
how old it was and honestly searching just two sources is not that difficult

40
00:02:50,879 --> 00:02:57,040
however if we went to scale that up again and build a third eventually a

41
00:02:54,959 --> 00:03:00,080
fourth cluster well would start to get pretty annoying not to mention

42
00:02:58,480 --> 00:03:04,800
inefficient for the staff who have to use the freaking thing the other issue

43
00:03:02,080 --> 00:03:09,920
is that managing even a single system takes a good amount of time and every

44
00:03:07,680 --> 00:03:15,680
time we scale it up in this manner it's another set of components CPU

45
00:03:12,239 --> 00:03:17,360
motherboard RAM power supplies ah oh my

46
00:03:15,680 --> 00:03:22,239
god this is really heavy that's the heaviest it's 130 kilograms i think

47
00:03:19,840 --> 00:03:27,599
that's when it's full holy crap you ready

48
00:03:24,800 --> 00:03:33,000
oh my god i told you you had the heavy side

49
00:03:28,560 --> 00:03:33,000
yeah but what the hell

50
00:03:33,040 --> 00:03:40,080
anyway more hardware means more potential issues more potential service

51
00:03:38,080 --> 00:03:43,760
so with that in mind we've been looking for an alternative solution for some

52
00:03:42,159 --> 00:03:47,280
time and while we could have continued to use gloucester fs and expand the

53
00:03:45,440 --> 00:03:52,480
existing clusters just don't really have anyone on staff

54
00:03:49,680 --> 00:03:56,879
who is familiar with it and it's not that user friendly i mean heck even

55
00:03:54,720 --> 00:04:00,400
45 drives the company who provided us with the original petabyte project have

56
00:03:58,640 --> 00:04:04,560
actually switched primarily to a different cluster solution called seth

57
00:04:03,439 --> 00:04:09,360
now ceph is cool really cool we use it for

58
00:04:06,959 --> 00:04:12,959
Floatplane for example and it's a bit better for scaling up but it shares a

59
00:04:11,439 --> 00:04:16,160
lot of the same problems of our old setup we don't have a ton of people on

60
00:04:14,560 --> 00:04:22,000
staff who are experienced with it and it would have the same or similar maintenance requirements so instead

61
00:04:19,919 --> 00:04:25,520
we've opted to switch to a configuration that many home Labs and data centers

62
00:04:24,639 --> 00:04:28,960
adore one i personally have actually never

63
00:04:27,680 --> 00:04:32,000
used the jbod

64
00:04:33,040 --> 00:04:38,000
you can fit so much hard drive in here

65
00:04:38,479 --> 00:04:51,040
this magnificent beast is the super micro 947 he1c dash 42k 05 jb od it's a

66
00:04:48,160 --> 00:04:57,919
90 bay chassis that as the name jbot implies is quite literally just a bunch

67
00:04:54,320 --> 00:05:00,240
of disks there's actually

68
00:04:57,919 --> 00:05:03,280
no computer in there which raises the question

69
00:05:01,360 --> 00:05:06,880
what is here in the back where the compute module or

70
00:05:04,960 --> 00:05:10,880
multiple compute modules would normally be this is it

71
00:05:08,639 --> 00:05:15,199
this is what passes for the computer in this chassis and there there

72
00:05:13,440 --> 00:05:19,440
is some computing going on here we've got some kind of asic

73
00:05:17,440 --> 00:05:24,160
that is i mean is this an fpga even i'm not sure it would be for sas though yeah

74
00:05:22,000 --> 00:05:28,320
it could be could are they doing enough oh oh gosh uh let's see you don't want

75
00:05:27,360 --> 00:05:33,759
to break it no i don't want to break it i just want to see it there's a button cell battery

76
00:05:31,360 --> 00:05:36,720
in there this chassis does have ipmi so that's what the battery would be for

77
00:05:34,720 --> 00:05:40,479
keeping track of time so there's going to be some compute for that probably on

78
00:05:38,400 --> 00:05:44,560
the bottom board but the rest of it should just be sas stuff there's that

79
00:05:42,400 --> 00:05:50,240
ipmi management port and then the only other i o on this machine is these six

80
00:05:47,680 --> 00:05:55,840
mini sas external ports what are these 88 so each of these is four lanes of 12

81
00:05:53,039 --> 00:06:00,400
gigabit sas so it's 48 gig per port and you can use up to three at one time for

82
00:05:58,160 --> 00:06:06,639
connection to the head server which is what is that 144 gigabit okay

83
00:06:04,000 --> 00:06:11,440
casual perfect then oh you can see the a-speed uh chip for

84
00:06:09,120 --> 00:06:15,919
the ipmi right there sit in there there'll be some uh non-volatile memory

85
00:06:13,600 --> 00:06:18,880
on there too somewhere it might actually be this

86
00:06:17,039 --> 00:06:23,600
that looks like a nand package these are some chungus cooling fans look

87
00:06:22,000 --> 00:06:28,639
how thick they are there's two in there right 80 millimeter fans i actually

88
00:06:26,000 --> 00:06:32,639
don't think so it's dual blade but no i think these are individual fans Jake oh

89
00:06:31,120 --> 00:06:37,919
no it's got to be two it's got to be two there's two discrete connectors either

90
00:06:34,560 --> 00:06:39,680
way i guarantee you they absolutely rip

91
00:06:37,919 --> 00:06:44,160
and then aside from i guess these are probably the sas and that would be your

92
00:06:41,199 --> 00:06:48,479
power that's pretty much it very very simple and that's with intention it's

93
00:06:46,240 --> 00:06:52,560
designed to be reliable right our other module here is

94
00:06:50,319 --> 00:06:56,319
well doesn't have a whole lot going on to be perfectly honest with you what on

95
00:06:54,400 --> 00:07:03,039
earth does this do you've got a small controller board that has a couple SATA

96
00:06:58,240 --> 00:07:05,680
ports a couple NVMe ports and

97
00:07:03,039 --> 00:07:09,919
more power and whatever the crap this is this is probably PCIe i think when you

98
00:07:07,840 --> 00:07:15,039
buy the computer version of this these two nvmes would come over here somewhere

99
00:07:12,479 --> 00:07:19,919
if you just had a single compute node but yeah i'm not really sure

100
00:07:17,440 --> 00:07:23,680
okay you could have a jbod that's something called dual path now this is a

101
00:07:21,599 --> 00:07:26,800
single path which means there's one path for the data to go from the drives to

102
00:07:25,360 --> 00:07:29,599
whatever computer it's plugged into because remember there's no computer in

103
00:07:28,400 --> 00:07:33,840
here it's got to plug into a different computer yeah but in a dual path there's

104
00:07:32,319 --> 00:07:38,160
two connections and you'll actually have redundant sas expanders which is the

105
00:07:35,599 --> 00:07:41,199
part that splits into multiple drives you'll have redundant connections so

106
00:07:39,840 --> 00:07:44,080
there'll be six more back here and you'll have another ipmi and the reason

107
00:07:42,960 --> 00:07:48,720
for that is you can have a high availability controller so you could have controller 1 and controller 2. and

108
00:07:47,120 --> 00:07:52,639
if controller 1 had problems it would just fail over to controller 2 and you'd

109
00:07:50,479 --> 00:07:56,800
have better availability the problem is that in order to take advantage of that

110
00:07:54,319 --> 00:08:01,599
we need to use sas drives and that's a hassle well it's not only

111
00:07:59,520 --> 00:08:08,319
is it a hassle but we already have 3.6 petabytes of seda drives that's a hassle

112
00:08:04,000 --> 00:08:08,319
that's a that's like 150 000 hassle

113
00:08:08,560 --> 00:08:13,520
not doing it now let's take a look at the sas

114
00:08:11,680 --> 00:08:18,000
expanders that make this magic happen naturally they're hot swappable because

115
00:08:15,280 --> 00:08:23,199
servers and in a nutshell each of these takes in four of those sas 12 gigabit

116
00:08:21,039 --> 00:08:30,160
per second lanes and then is able to split that bandwidth to up to 30 flippin

117
00:08:26,800 --> 00:08:33,760
drives so we only actually need three of

118
00:08:30,160 --> 00:08:36,640
these to populate the entire 90 bay

119
00:08:33,760 --> 00:08:40,320
capacity of this j-bond by the way Jake was talking about how

120
00:08:38,080 --> 00:08:42,800
you could have a dual path well you'd need

121
00:08:41,360 --> 00:08:48,880
three more expanders then but we don't need them Jake do you have any idea what these connectors are called i have no

122
00:08:45,839 --> 00:08:51,200
idea shout out them because these things

123
00:08:48,880 --> 00:08:57,920
are flipping cool look at the way these are joined back onto the board

124
00:08:54,000 --> 00:09:00,880
that is dense with two s's that's oh my

125
00:08:57,920 --> 00:09:05,360
god the pins on the bottom right holy crap the next thing i need Jake is for

126
00:09:03,360 --> 00:09:09,839
you to get me the head it's it's right there right here you can

127
00:09:07,360 --> 00:09:14,480
think of your jbot kind of like an external hard drive or in this case

128
00:09:12,320 --> 00:09:19,360
many many external hard drives which means that as Jake alluded to before we

129
00:09:16,959 --> 00:09:24,560
need to have a computer somewhere well that is where the head or the controller

130
00:09:22,240 --> 00:09:28,800
server comes into play this particular machine is a placeholder supermicro is

131
00:09:26,399 --> 00:09:34,000
actually sending us over a dual AMD epic milan server that's a little more

132
00:09:31,600 --> 00:09:39,360
tailored for this kind of application so instead of being full of NVMe drive bays

133
00:09:36,880 --> 00:09:45,680
in the front it's got its PCIe lanes allocated to allow for lots of hbas and

134
00:09:42,399 --> 00:09:47,839
network connectivity in the back because

135
00:09:45,680 --> 00:09:52,560
hard drives may be slow but when you hook up enough of them you can be

136
00:09:49,440 --> 00:09:55,360
pushing some serious freaking data and

137
00:09:52,560 --> 00:10:00,240
this is going to be connected to up to 270 drives

138
00:09:58,320 --> 00:10:04,240
not 255 i think that's what it works out to 90 times three is 200 and we don't

139
00:10:02,399 --> 00:10:07,839
have 98. oh but yeah i mean you could you get you could say you could add more

140
00:10:05,760 --> 00:10:11,440
drives it's just jbod um so cool right

141
00:10:09,760 --> 00:10:15,279
in the meantime for the purposes of our demo

142
00:10:13,120 --> 00:10:20,079
we've loaded this machine up with three of these broadcom hbas these things

143
00:10:18,000 --> 00:10:24,079
actually contain very little logic compared to raid cards which had to

144
00:10:22,079 --> 00:10:27,760
perform parity calculations on the data running through them

145
00:10:25,440 --> 00:10:32,959
these do almost nothing they just take your bandwidth from your PCIe slot in

146
00:10:29,920 --> 00:10:34,720
this case PCIe 8x gen 4 slot run it

147
00:10:32,959 --> 00:10:40,399
through a sas controller and then that breaks out into for a 12

148
00:10:37,680 --> 00:10:43,519
16 sas connections do you wanna see my wonderful cardboard yeah do you wanna

149
00:10:41,920 --> 00:10:47,680
see my wonderful car do you wanna see my wonderful cardboard yeah

150
00:10:45,600 --> 00:10:51,120
this math actually makes a lot of sense if you have 90 drives and we'll say

151
00:10:49,279 --> 00:10:55,760
optimistically they're running at 250 megabytes a second that's 22.5 gigabytes

152
00:10:53,920 --> 00:11:00,240
a second it's pretty fast conveniently three sets of 48 gigabit sas cables is

153
00:10:58,160 --> 00:11:04,720
around 18 gigabytes a second and even more conveniently PCIe gen four by eight

154
00:11:03,040 --> 00:11:07,760
it's roughly 16 gigabytes a second in either direction so we're going to be

155
00:11:06,160 --> 00:11:11,839
giving up some of the theoretical maximum performance of our drives but

156
00:11:09,440 --> 00:11:15,360
given that this is only under perfect conditions with brand new

157
00:11:13,519 --> 00:11:18,959
high-performance drives with reading and writing sequential data with nothing on

158
00:11:17,760 --> 00:11:22,880
them this will be lots

159
00:11:21,279 --> 00:11:27,040
and that's times three because we have three j bonds

160
00:11:24,560 --> 00:11:30,320
we're never gonna get anywhere close to this maybe one Gigabyte a second i'd

161
00:11:29,040 --> 00:11:34,880
like to point out something else that's convenient what check this out

162
00:11:32,640 --> 00:11:40,160
see that ratchet back pressure wow i can start a screw with it and in spite of

163
00:11:37,760 --> 00:11:44,240
the silver shaft look at that i used it and my dick didn't fall off

164
00:11:42,720 --> 00:11:49,680
i actually think the silver shaft looks better wow all right

165
00:11:48,000 --> 00:11:52,800
he's actually been one of the bigger skeptics internally about this whole

166
00:11:51,440 --> 00:11:56,959
project i'm just a little bit critical

167
00:11:54,959 --> 00:12:01,680
but that ratchet i want it to be good okay you can sign up for a notification when

168
00:11:59,600 --> 00:12:05,839
it comes in stock lttstore.com try to sound less stressed in the meantime we

169
00:12:03,760 --> 00:12:09,519
also have shirts and stuff cash flow's a little tight now you might look at this

170
00:12:07,760 --> 00:12:13,040
machine and think that's ludicrous overkill surely the one that super micro

171
00:12:11,600 --> 00:12:17,920
is going to send is a little more pedestrian but that's not actually the case even

172
00:12:16,320 --> 00:12:21,519
though we're not running super high speed NVMe drives when you're

173
00:12:20,480 --> 00:12:26,000
hooking up hundreds of hard drives with the

174
00:12:23,760 --> 00:12:30,720
potential to expand to hundreds of more you need CPU and RAM galore more than

175
00:12:29,680 --> 00:12:36,880
you'd think in fact we actually have two 32 core

176
00:12:33,519 --> 00:12:40,800
epic 75 f3 processors those are the

177
00:12:36,880 --> 00:12:42,320
frequency oriented 280 watt per cpus and

178
00:12:40,800 --> 00:12:46,000
we're going to have around a terabyte of RAM with the potential to expand that

179
00:12:44,160 --> 00:12:50,399
down the road because we will be using zfs and zfs greatly benefits from RAM

180
00:12:48,880 --> 00:12:53,040
for recaching we'll also have NVMe drives and stuff

181
00:12:51,920 --> 00:12:58,959
but that's tbd you'll have to get subscribed for the next video when we actually deploy the whole are we doing

182
00:12:56,000 --> 00:13:03,680
tiering no oh well maybe one day it's possible let you know a secret

183
00:13:00,959 --> 00:13:07,200
we might build a highly available wanik and set this up as a tiered

184
00:13:05,680 --> 00:13:13,519
archival layer entirely transparent to the user

185
00:13:09,760 --> 00:13:15,519
so just one drive yeah letter one drive

186
00:13:13,519 --> 00:13:19,920
oh my god these are just these are just drive sleds i thought

187
00:13:17,760 --> 00:13:23,279
these were boxes of hard drives Jake no those are the traits i've been deceived

188
00:13:21,680 --> 00:13:26,480
do you want to see what they trade like we were talking about how we wanted to

189
00:13:24,880 --> 00:13:30,399
discuss yeah we also need the high performance CPU for zfs compression i

190
00:13:28,959 --> 00:13:33,680
was like oh well we'll just dump a bunch of drives in here and then we'll talk

191
00:13:32,160 --> 00:13:38,560
about okay we've got this much capacity but we're going to use compression and then Jake's like oh we've only got 13

192
00:13:36,480 --> 00:13:43,360
drives to put in it for the demo come on Jake it's almost like all our hard

193
00:13:40,800 --> 00:13:49,839
drives are deployed holding data temporarily from the old vault yeah

194
00:13:45,760 --> 00:13:51,440
almost okay well i guess it's a 13 drive

195
00:13:49,839 --> 00:13:56,560
demo if you want ed to yell at you we could take apart delta 5. that's got new vault

196
00:13:55,120 --> 00:13:59,519
on it i think and some of old vault when ed yells at

197
00:13:58,720 --> 00:14:04,880
me wendell's still working on delta 3 so that's not available

198
00:14:03,120 --> 00:14:08,240
temp vault we could do temp vault but then ed would also yell at you probably

199
00:14:06,639 --> 00:14:12,079
it's not fit 13 drive seems fine

200
00:14:10,399 --> 00:14:18,480
all right we found another solution to some test hard drives this here

201
00:14:13,760 --> 00:14:20,560
storinator was full of 60 spinnerino's

202
00:14:18,480 --> 00:14:25,920
and now they're in here it has made the balance of this system

203
00:14:22,320 --> 00:14:27,199
super sketchy like

204
00:14:25,920 --> 00:14:30,680
oh it gets worse

205
00:14:31,040 --> 00:14:36,160
look look at that look it's it

206
00:14:36,160 --> 00:14:41,519
oh god oh i can't reach okay there we go but

207
00:14:40,079 --> 00:14:45,440
there you go there's 60 hard drives they're kind of in a weird orientation

208
00:14:43,839 --> 00:14:49,519
because there's a kind of a poop mix of drives in here there's some tens there's

209
00:14:47,279 --> 00:14:54,720
some 12s there's some 16s i think there's even 120 so i just wanted to

210
00:14:51,680 --> 00:14:56,399
kind of separate those but this

211
00:14:54,720 --> 00:15:00,079
is our test now these 60 drives do already

212
00:14:58,800 --> 00:15:03,680
have an array on them and i think it's about 70 full this is some of our

213
00:15:02,079 --> 00:15:08,720
existing archival data but they're going to be set up as four v devs that are

214
00:15:05,600 --> 00:15:10,240
raid z2 15 drives wide not great for

215
00:15:08,720 --> 00:15:14,480
performance i don't think we're gonna see anything crazy uh in terms of

216
00:15:12,320 --> 00:15:18,320
numbers out of this machine but it is kind of indicative of what we're

217
00:15:16,240 --> 00:15:23,760
gonna have it set up as we're probably gonna switch to 10 drive raid z2s

218
00:15:21,120 --> 00:15:27,760
just to make it a little bit faster but this should still be pretty dang fast

219
00:15:25,839 --> 00:15:33,199
now in order to get a connection between our server and our jbod we have these

220
00:15:30,160 --> 00:15:34,959
these cool mini sas hd external cables

221
00:15:33,199 --> 00:15:39,600
from infinite cables i think supermicro also sent us some by default this jbod's

222
00:15:37,360 --> 00:15:43,040
configured as a single zone now you can do kind of cool stuff with zoning in a

223
00:15:41,519 --> 00:15:46,720
jbot for instance you could split this up into two zones and have two

224
00:15:44,880 --> 00:15:50,880
controller servers being serviced by one jbot so you'd have 45 drives on either

225
00:15:49,279 --> 00:15:55,120
you can also do three zones so that'd be 30 30 30 do three controller servers but

226
00:15:53,440 --> 00:15:58,800
they can't access the other zone's data you'd only get the

227
00:15:57,360 --> 00:16:03,440
zone that you're actually physically attached to so you'd have

228
00:16:00,639 --> 00:16:07,519
a c or e on the back of this server now in our case we're going to be using

229
00:16:05,120 --> 00:16:13,360
three zones but to one controller server and the reason for this is it allows us

230
00:16:10,320 --> 00:16:16,079
instead of getting one cable which is 48

231
00:16:13,360 --> 00:16:21,199
gigabit bandwidth we now get two and then three for 144 gigabit which again

232
00:16:19,199 --> 00:16:25,199
is probably overkill but what do we do that's not overkill

233
00:16:23,199 --> 00:16:29,519
hi you might need these like like for realsies

234
00:16:27,680 --> 00:16:35,800
no don't just wait this is just the power stuff that's the power supplies

235
00:16:31,680 --> 00:16:35,800
that's like full tilt though

236
00:16:37,920 --> 00:16:44,720
whoa this is ludicrously loud so i put it in

237
00:16:42,399 --> 00:16:50,000
the server room with the door shut you can hear it in the bedroom

238
00:16:47,920 --> 00:16:54,320
but this is like they're actually at a hundred percent right now they should

239
00:16:51,680 --> 00:16:59,360
never run it never and with the 60 drives i already tried it like running a

240
00:16:56,320 --> 00:17:03,440
load and they barely even spin up okay

241
00:16:59,360 --> 00:17:05,679
that's good yeah i got a little worried

242
00:17:03,440 --> 00:17:09,199
is if the air conditioning ever fails in the server room

243
00:17:07,199 --> 00:17:12,400
we'll know because we'll hear this everywhere

244
00:17:11,280 --> 00:17:14,640
there we go oh

245
00:17:15,439 --> 00:17:21,839
give it a sec okay coming back to planet earth

246
00:17:19,600 --> 00:17:26,160
that's a lot more reasonable i mean i guess that's what fans that

247
00:17:24,240 --> 00:17:29,760
thick will do for you well i mean when you have 90 hard drives you're talking

248
00:17:27,919 --> 00:17:34,080
like at least a thousand watts yeah that's a lot of power i mean what are

249
00:17:31,600 --> 00:17:38,400
these power supplies uh they're 2 000 watt on 208 volt 2 000 watt power

250
00:17:37,280 --> 00:17:43,280
supplies um so to have like actual power supply

251
00:17:41,120 --> 00:17:47,039
redundancy you need to be running 208 volt wow otherwise you'll probably use

252
00:17:45,679 --> 00:17:52,720
enough power that it needs to split across the two so we might actually have to rewire our server room to be 208 volt

253
00:17:51,200 --> 00:17:57,679
because for some reason it wasn't wired that way in the first place i don't know i don't remember why brian there's

254
00:17:55,360 --> 00:18:02,400
probably a reason yeah there's a reason and i bet you there's a way to fix it oh

255
00:17:59,760 --> 00:18:05,760
my god yeah you can fix anything with money mm-hmm you have lots of that

256
00:18:04,559 --> 00:18:10,160
right now right no oh well that's fine we'll figure it out

257
00:18:08,559 --> 00:18:14,000
you know what no we should just put these in lab two because that one won't

258
00:18:12,000 --> 00:18:18,240
be too late right time to put more insulation on the walls

259
00:18:16,400 --> 00:18:21,520
jesus with nails yeah i'll show you that quick config here this is where you can

260
00:18:19,760 --> 00:18:25,039
change the zone so there's single zone two zone or three zone again like i

261
00:18:23,440 --> 00:18:28,480
explained before you could use that for multiple controllers or to increase

262
00:18:27,039 --> 00:18:32,000
throughput we're doing the throughput route of course that's pretty much all

263
00:18:30,320 --> 00:18:35,679
you can do in here there's no remote control like there's no there's no

264
00:18:33,840 --> 00:18:39,840
screen it doesn't have a display output right yeah power it's just a big

265
00:18:38,559 --> 00:18:44,080
dumb big discs

266
00:18:42,720 --> 00:18:48,799
system critical i don't know why probably because you pulled the power supply probably plug it back in i'm

267
00:18:46,559 --> 00:18:52,880
sorry about that buddy here you go yeah there's your medicine right in the butt

268
00:18:51,120 --> 00:18:57,440
suppository medicine power cable now tell me something if we

269
00:18:55,440 --> 00:19:01,200
decided to go high availability in the future nothing would actually prevent us

270
00:18:59,520 --> 00:19:06,559
from adding a second interface unit back there yes a

271
00:19:03,919 --> 00:19:10,480
second head server yes no problem right we need sas drives sorry not high

272
00:19:08,799 --> 00:19:14,960
availability at the oh cause dual path oh crap okay yeah the

273
00:19:13,600 --> 00:19:20,880
thing about this approach though is we've already proven we don't really depend on this yeah it's

274
00:19:19,360 --> 00:19:26,559
been like it's gotta be like eight months at this point that we haven't had it it's funny how many people were

275
00:19:24,000 --> 00:19:31,919
listening to us in the video saying yeah it's really optional retaining this data

276
00:19:29,280 --> 00:19:36,240
we lost it because we don't really care that much and we're like

277
00:19:33,440 --> 00:19:40,400
wise like no we haven't had it for like almost a year yeah at this point and

278
00:19:37,760 --> 00:19:44,720
it's been totally fine it is nice to have but here's nice to have if this

279
00:19:42,240 --> 00:19:48,480
controller server were to poop the bed or to have some problem well everything

280
00:19:46,320 --> 00:19:53,679
here should still be we can just unplug this we'll have some other server temporarily

281
00:19:51,600 --> 00:19:56,880
so like a cold spare put the hp yeah a basic spare server i mean you're talking

282
00:19:55,440 --> 00:20:01,440
maybe an hour of downtime like realistically i was looking at this and

283
00:19:59,039 --> 00:20:04,799
i thought maybe this was i saw four v devs i thought that was the four drives

284
00:20:03,280 --> 00:20:09,919
i i just saw it on the corner of my eye and i thought you had put the 30 terabyte nvmes in here

285
00:20:08,160 --> 00:20:13,039
we might though wendell's been talking about this special metadata device for a

286
00:20:12,080 --> 00:20:20,160
while and basically he was saying he has like half a petabyte with 24 terabytes of

287
00:20:17,760 --> 00:20:23,039
special metadata device in zfs and it fills

288
00:20:21,280 --> 00:20:27,120
so i'm like we might as well just put 430 terabytes in there and that'll give

289
00:20:25,039 --> 00:20:30,559
us 60 terabytes what does special metadata do so you know like file

290
00:20:29,039 --> 00:20:35,679
metadata and like the directory structure is that it's like searching really fast it makes searching way

291
00:20:33,760 --> 00:20:39,280
faster oh that's cool so stores metadata like where files are

292
00:20:38,000 --> 00:20:44,640
she's talking about it right here it's literally literally Windows this is the first thing that comes up hi wendell

293
00:20:42,480 --> 00:20:50,480
yeah yeah but here's the thing is we can have a level two arc we can have a log

294
00:20:47,120 --> 00:20:52,799
device and we can have special data yeah

295
00:20:50,480 --> 00:20:55,840
so he's saying 172 terabytes of space is five terabytes of metadata okay but

296
00:20:54,720 --> 00:20:59,280
here's the thing we don't have to have all of it metadata

297
00:20:57,840 --> 00:21:03,039
in there like it'll pull it when it needs it right okay yeah by default this

298
00:21:01,360 --> 00:21:08,400
includes all the metadata the indirect blocks of user data and any deduplication tables stored on that

299
00:21:06,240 --> 00:21:12,080
device so maybe deduplication if you are running deduplication this is probably

300
00:21:10,640 --> 00:21:16,000
really important to improving performance yeah

301
00:21:14,159 --> 00:21:21,520
but either way we'll try it as you can see it's about 70 full not ideal for

302
00:21:18,720 --> 00:21:25,120
performance and on top of that it's an existing pool

303
00:21:23,120 --> 00:21:29,679
um that it's set up in four v-depths right yeah so they're 15 wide yeah so

304
00:21:28,559 --> 00:21:34,480
really i'm expecting like maybe three to five

305
00:21:32,240 --> 00:21:38,159
gigabytes a second okay sequential like i i would not be surprised if that's

306
00:21:36,240 --> 00:21:40,720
where we end up because when we move to the final deployment we're gonna do 10

307
00:21:39,760 --> 00:21:45,679
wide which will be better for performance we'll do 10 wide raid z2s

308
00:21:43,600 --> 00:21:49,440
which is basically the same as 15 wide raid z3s yeah

309
00:21:47,679 --> 00:21:53,919
and then we'll obviously have a lot more drives if you've got 10 drives of which

310
00:21:51,840 --> 00:21:58,799
you could lose two of them before you any data loss that's kind of the same as

311
00:21:56,000 --> 00:22:03,679
having 15 drives of which you could lose three in terms of the ratio of the whole

312
00:22:01,200 --> 00:22:09,360
pool if you look at it from a wide perspective but it's probably slightly

313
00:22:06,720 --> 00:22:14,400
better to have more v devs if you run the probability analysis

314
00:22:12,000 --> 00:22:19,120
anyway so this no no we gotta explain the data set is set up with caching only

315
00:22:17,120 --> 00:22:22,880
metadata which is what we want the RAM cache is turned off uh let's

316
00:22:21,679 --> 00:22:27,679
give it a second there we go we're getting like let's just give it give it you know just warm up there bud come on

317
00:22:26,640 --> 00:22:31,840
okay that's about two two and two gigabytes

318
00:22:29,840 --> 00:22:35,039
john you can do better than that i didn't even look at it

319
00:22:33,120 --> 00:22:39,760
come on three three yeah it's not three sustained

320
00:22:36,720 --> 00:22:41,520
that's for sure 3.6 you can't just read

321
00:22:39,760 --> 00:22:45,360
the high numbers i'm reading just the high numbers that was 3.7 there for a

322
00:22:43,600 --> 00:22:50,159
second yeah it really doesn't work like that that's

323
00:22:47,200 --> 00:22:54,799
579. oh there's four thousand percent there's

324
00:22:52,159 --> 00:22:57,280
point three probably has something to do with the fact that the drives are pretty

325
00:22:56,080 --> 00:23:02,080
full already remember we have no NVMe caching we have

326
00:22:59,440 --> 00:23:08,159
no RAM caching like this is raw dog giant v dev just disc

327
00:23:06,000 --> 00:23:12,480
now tell me something hold on a second wait here's something i don't fully

328
00:23:10,080 --> 00:23:16,159
understand this is really cool for maintenance the fact that you can have

329
00:23:14,080 --> 00:23:20,320
the system running you don't have to pull all your cables

330
00:23:18,000 --> 00:23:24,720
out at the back in order to see yeah the cable's staying in the same spot super

331
00:23:22,080 --> 00:23:27,840
nice but how is all of this still connected these drives are blinking

332
00:23:26,240 --> 00:23:31,360
these are doing things oh it's just like a ribbon yeah giant

333
00:23:29,679 --> 00:23:38,080
ribbon cable in the back you see it back there oh that's super cool

334
00:23:34,880 --> 00:23:40,080
man that's amazing for maintenance

335
00:23:38,080 --> 00:23:44,320
so you can just go be like okay yeah i've got a bad drive and bay whatever

336
00:23:42,720 --> 00:23:47,760
pull this out you don't have to slide any servers out see you later because

337
00:23:46,000 --> 00:23:50,720
this is internal right you just pull it out and you grab that one we're gonna

338
00:23:49,120 --> 00:23:54,400
have to label them oh yeah or i might just do a spreadsheet honestly um yeah

339
00:23:52,559 --> 00:23:59,120
that would be fine if they were sas drives you have like uh sas

340
00:23:56,640 --> 00:24:04,240
enclosure service or something like that ses whatever anyways you can go and be

341
00:24:01,520 --> 00:24:08,080
like what's in that slot or by drive you can say what slot is that driving and

342
00:24:06,159 --> 00:24:11,440
it'll tell you but uh since they're saying it drives they'll just show us

343
00:24:09,440 --> 00:24:15,760
not connected let's put this back in this is making me

344
00:24:12,960 --> 00:24:21,600
anxious let's do that i'm like i just oh boy okay yeah so we're doing 16 jobs in

345
00:24:18,640 --> 00:24:25,360
the i o depth of 16 block size one mag it's gonna be probably basically the

346
00:24:23,360 --> 00:24:30,799
same oh two thousand whew

347
00:24:27,760 --> 00:24:32,400
blow my skirt up

348
00:24:30,799 --> 00:24:35,760
more consistent though yeah

349
00:24:34,000 --> 00:24:39,120
like nope there it goes see you later

350
00:24:37,760 --> 00:24:43,600
in the grand scheme of things we're going to be writing to this thing like

351
00:24:40,799 --> 00:24:47,520
basically peak one Gigabyte a second ever and it will be better when there is

352
00:24:46,000 --> 00:24:52,720
more v-divs okay then i think we're good

353
00:24:50,640 --> 00:24:56,400
good to tell you about our sponsor bessie bessie proves that waterproof

354
00:24:54,799 --> 00:25:00,320
shoes don't need to be ugly or uncomfortable thanks to their dymatex

355
00:24:58,240 --> 00:25:04,559
technology you can keep those toes dry while remaining sleek snug and stylish

356
00:25:02,640 --> 00:25:09,120
need even more comfort and breathability the everyday move line has added padding

357
00:25:06,640 --> 00:25:13,360
at the midsole and a looser knit to keep up with your busy active lifestyle do

358
00:25:11,120 --> 00:25:17,440
you hate tying laces tired of chasing the rabbit around the tree and down the

359
00:25:15,200 --> 00:25:22,400
hole well fear not because vessy's everyday move comes with handy pull tabs

360
00:25:19,679 --> 00:25:26,159
for easy slip-on action and are made 100 creature-free so you can be walking

361
00:25:24,240 --> 00:25:31,200
around on fluffy clouds with a clear conscience all day long and when vessy

362
00:25:28,799 --> 00:25:35,039
says everyday move they mean it hot cold wet dry stay comfortable in any weather

363
00:25:33,679 --> 00:25:38,799
just think of all the freedom you can get with your new shoes so treat those

364
00:25:36,960 --> 00:25:44,799
feet of yours to vessy everyday move shoes and save 25 with offer code Linus

365
00:25:41,440 --> 00:25:46,799
tech tips at vetsy.com linustechtips if

366
00:25:44,799 --> 00:25:51,039
you guys enjoyed this video you might also enjoy the one where we explain what

367
00:25:49,120 --> 00:25:56,559
happened to the old vault and how we lost nearly all of the data on it

368
00:25:54,080 --> 00:26:01,279
what we got it all back well yeah i know but first we lost it well like sort of