1
00:00:00,240 --> 00:00:06,879
as promised i have brought you guys the

2
00:00:03,760 --> 00:00:08,080
one and only petabyte project

3
00:00:06,879 --> 00:00:13,920
two and the one and only Anthony Anthony our

4
00:00:10,960 --> 00:00:18,800
task today is very simple we had two hours to have this petabyte project

5
00:00:16,720 --> 00:00:22,960
deployed so that the editors and the shooters can offload a bunch of footage

6
00:00:20,640 --> 00:00:29,320
off of the existing vault onto this one so we can load balance things a little

7
00:00:24,560 --> 00:00:29,320
bit easy right easy

8
00:00:37,280 --> 00:00:42,399
so uh go

9
00:00:40,879 --> 00:00:47,120
wave the wand what what do you want to do what you do well i don't know i can plug it i mean

10
00:00:45,200 --> 00:00:51,520
it's not it's not powered i can't like just cast a spell and we're going to do

11
00:00:49,440 --> 00:00:55,039
something really similar to what we've got for existing petabyte project where

12
00:00:53,600 --> 00:01:00,800
we're going to do four 15-drive v-devs in raid z2 so so that

13
00:00:58,320 --> 00:01:05,760
you guys know that allows us to lose up to eight drives total before we'd lose

14
00:01:03,760 --> 00:01:09,040
any data and that has come in handy yeah the re-silvering was a bit of a

15
00:01:07,040 --> 00:01:14,159
nail-biting process let me tell you that it took many weeks i was thinking of

16
00:01:11,520 --> 00:01:17,439
adding an optane pci express card as a slog

17
00:01:15,280 --> 00:01:21,040
so that would be a right cache effectively

18
00:01:18,720 --> 00:01:24,640
that doesn't depend on dram so that in the event of a sudden power loss we

19
00:01:22,799 --> 00:01:28,799
would still retain all the data i don't know how big we did that though 480 gigs

20
00:01:27,360 --> 00:01:34,000
that's the smallest obtained drive we have okay okay i'll go grab one

21
00:01:31,280 --> 00:01:38,640
don't forget to sign it out yay got em oh they break off yeah

22
00:01:36,479 --> 00:01:42,720
for a server chassis that's like hundreds of dollars this is not an

23
00:01:40,560 --> 00:01:45,990
amazing experience but so we ready to power it up

24
00:01:44,400 --> 00:01:47,840
uh sure

25
00:01:47,840 --> 00:01:54,320
it's actually not even crazy loud no i was expecting much louder did they send

26
00:01:52,240 --> 00:02:00,399
the credentials for login well let's try root and password

27
00:01:57,520 --> 00:02:04,880
okay we should probably change that i'm a hacker today boys first of all you

28
00:02:03,280 --> 00:02:11,440
shouldn't be logged in as root well i mean just for like initial

29
00:02:06,479 --> 00:02:12,800
configurations matter yeah do rm-rf

30
00:02:11,440 --> 00:02:17,280
don't do that oh wait you were gonna ruin my stuff

31
00:02:15,200 --> 00:02:22,400
now the original plan was for 45 drives to build me a custom 75 drive unit right

32
00:02:20,560 --> 00:02:26,080
but seagate did send 75 drives

33
00:02:24,319 --> 00:02:30,080
so i was kind of thinking you know what would be cool

34
00:02:27,760 --> 00:02:34,239
is if we took an old decommissioned storinator

35
00:02:31,440 --> 00:02:37,840
and actually created a new cluster with this one and that one i'd have to

36
00:02:36,000 --> 00:02:41,120
upgrade it obviously so we go epic with the other one as well and uh what do you

37
00:02:40,000 --> 00:02:44,879
think no yes good bad

38
00:02:43,040 --> 00:02:49,840
they said not to do it over a petabyte let's do it hold on a second we might

39
00:02:47,440 --> 00:02:51,920
need drivers for that melanox one what are those 10 gigabit cars that are in

40
00:02:51,200 --> 00:02:57,920
there they're just the onboard ones um uh um i

41
00:02:55,920 --> 00:03:01,280
could try another slot okay if you want to throw the power switch to make sure

42
00:02:58,959 --> 00:03:05,840
it doesn't wake back up yep and now

43
00:03:03,360 --> 00:03:08,640
ready that's not a power switch i ask you if

44
00:03:07,280 --> 00:03:12,400
there's a power switch well there is a power switch

45
00:03:10,959 --> 00:03:15,920
that's not hey how's it going i found another 60

46
00:03:14,319 --> 00:03:20,560
drive storinator so now we could have matching ones are they really matching well if i

47
00:03:19,120 --> 00:03:23,920
change out all the hardware in this one they would be somewhat matching yes

48
00:03:22,239 --> 00:03:27,440
somewhat yeah it's the older model though i was kind of thinking while

49
00:03:25,280 --> 00:03:31,200
you're working on this i refurbed this i don't think this should take very long

50
00:03:29,280 --> 00:03:35,360
all we have to do right now is last words

51
00:03:33,200 --> 00:03:39,840
let me see if i've got the melanox card now yeah there we go

52
00:03:38,080 --> 00:03:44,000
oh nice so just moving it to the other slot worked yeah i hope that just

53
00:03:41,920 --> 00:03:48,959
rebooted on its own probably fine

54
00:03:45,200 --> 00:03:50,799
famous last words yeah thanks for that

55
00:03:48,959 --> 00:03:56,159
oh man do you remember when 45 drives used to use non-ferrous screws

56
00:03:52,879 --> 00:03:57,840
me too this is super annoying

57
00:03:56,159 --> 00:04:01,680
i'm going to replace every screw in this case when i rebuild this okay when i

58
00:04:00,319 --> 00:04:06,239
said this probably wasn't going to take very long i didn't realize that we had

59
00:04:03,519 --> 00:04:11,439
to download so many updates so here we are oh failure wait god why

60
00:04:11,439 --> 00:04:18,720
meanwhile i'm extracting this motherboard oh shoot i didn't unplug the

61
00:04:15,519 --> 00:04:20,160
front panel connectors or the 24 pin

62
00:04:18,720 --> 00:04:23,840
we are totally not going to finish this today oh

63
00:04:22,240 --> 00:04:29,280
package g deploy because it's not available in the mirror anymore

64
00:04:26,479 --> 00:04:31,919
what's g deploy i have no idea what the crap

65
00:04:30,320 --> 00:04:36,880
oh boy that is a dusty server

66
00:04:35,199 --> 00:04:39,120
dusty boy ah

67
00:04:39,280 --> 00:04:44,639
go away they're saying to use gluster ansible so

68
00:04:42,960 --> 00:04:48,320
i'll just use that hey there we go

69
00:04:46,080 --> 00:04:51,360
i feel like i'm playing operation don't touch the sides

70
00:04:52,320 --> 00:04:57,360
okay hey is that all lit up yeah the drives are showed up wow yeah it was

71
00:04:55,759 --> 00:05:00,720
just a different command sweet thanks Anthony go nuts

72
00:05:01,759 --> 00:05:06,639
that was our motto at best buy was have fun while being the best how'd that work

73
00:05:05,040 --> 00:05:10,080
out for you we were the worst performing store in canada

74
00:05:08,400 --> 00:05:13,440
yeah like all the contacts are still there if you can get enough mounting

75
00:05:11,520 --> 00:05:16,320
pressure on the pins it should be able to work what if we jammed something in

76
00:05:15,520 --> 00:05:22,560
there that might do something non-conductive that just presses against there i found

77
00:05:19,600 --> 00:05:26,560
an old junk 3d print there's actually a bit on this side too whatever hit that

78
00:05:24,960 --> 00:05:31,039
thing hit it pretty hard somebody just ripped the memory out of it without pulling the tabs first no i think it was

79
00:05:29,360 --> 00:05:34,960
during transport at some point well they're physically in let's see what

80
00:05:32,720 --> 00:05:39,479
happens show me the 128 gigs

81
00:05:40,400 --> 00:05:45,680
deleting intensifies uh oh yeah

82
00:05:44,320 --> 00:05:50,479
we got it we repaired the memories a lot

83
00:05:47,919 --> 00:05:54,960
repaired it's such a dumb fix but it's this is totally gonna work thankfully

84
00:05:52,639 --> 00:05:59,280
hot glue is non-conductive and it melted the filament a little so

85
00:05:57,039 --> 00:06:04,000
didn't think of that hot glue it's it's hot filament it melts our hot glue fix

86
00:06:02,240 --> 00:06:08,000
is applied so this should boot up with 64 gigs of RAM right now

87
00:06:06,400 --> 00:06:11,759
good got it

88
00:06:10,160 --> 00:06:17,680
the scale of the project has changed a little bit so we're gonna cheat

89
00:06:14,319 --> 00:06:20,080
ever so slightly and pull three of our

90
00:06:17,680 --> 00:06:24,080
eight terabyte ssds and put them in this second server

91
00:06:21,759 --> 00:06:27,680
we actually pulled out all of the eight gig sticks of RAM that

92
00:06:25,840 --> 00:06:32,319
were in the servers and we are replacing them with 32 gig sticks so each of them

93
00:06:29,919 --> 00:06:36,960
is going to have 256 gigs of RAM now hey you only live once and uh i think that's

94
00:06:35,039 --> 00:06:40,720
pretty much all that i've changed other than that the last time you guys saw

95
00:06:38,560 --> 00:06:46,000
this it had old hardware in it and now it is running brand spanking new 16 core

96
00:06:43,680 --> 00:06:49,919
epic hardware and i've got uh another 256 gigs of RAM that i can throw in this

97
00:06:48,880 --> 00:06:54,479
puppy um oh

98
00:06:52,000 --> 00:06:58,479
and then oh okay but we got 256 gigs of RAM okay

99
00:06:55,919 --> 00:07:01,360
sweet this is working then theoretically there's nothing wrong with what i'm

100
00:07:00,000 --> 00:07:05,919
doing here but i am officially the jank king this

101
00:07:04,000 --> 00:07:09,680
power supply has only two SATA connectors and i need to plug in five

102
00:07:08,319 --> 00:07:14,960
drives so get this

103
00:07:11,360 --> 00:07:18,080
i'm going SATA male to say to female to

104
00:07:14,960 --> 00:07:21,120
molex male to molex female to quadruple

105
00:07:18,080 --> 00:07:23,440
seda over here that'll power the four uh

106
00:07:21,120 --> 00:07:27,759
regular SATA drives and then i'm using the other one

107
00:07:25,120 --> 00:07:31,599
for the octane drive just kind of jam that in there yeah i mean thankfully

108
00:07:29,680 --> 00:07:34,479
ssds don't take that much power exactly that's what i'm talking about who's

109
00:07:33,039 --> 00:07:39,120
going to say anything all of our viewers

110
00:07:36,560 --> 00:07:44,080
yeah i am just about done building this puppy i've got my SATA cables run just

111
00:07:41,840 --> 00:07:48,639
casually doing some cable management with my LTT Store.com cable test fans

112
00:07:46,800 --> 00:07:53,520
are clear fans are spinning everything's good to go all we got to do is load it up with some

113
00:07:51,280 --> 00:07:57,680
drives oh i guess i need to find all the drive mounting hardware i put away a

114
00:07:55,680 --> 00:08:01,840
panel that has some screws that go with it i put a little magnet on there with

115
00:07:59,919 --> 00:08:04,319
all the screws stuck to it see that free tech tip

116
00:08:05,199 --> 00:08:12,720
day three of this epic shoot what do you think is this just gonna boot up

117
00:08:09,680 --> 00:08:14,560
um i would say 95 yes i mean we haven't

118
00:08:12,720 --> 00:08:17,759
even checked the RAM compatibility yet okay well that's a bad sign go ahead and

119
00:08:16,080 --> 00:08:20,800
cut go ahead and cut go ahead and cut no dice on getting it to boot so i'm going

120
00:08:19,280 --> 00:08:24,800
to start by ripping some extraneous hardware out and just see if at least we

121
00:08:22,879 --> 00:08:29,199
know the CPU and RAM work nope i don't think that's doing

122
00:08:27,599 --> 00:08:32,479
any better go ahead and cut according to the manual i shouldn't need any of that

123
00:08:30,800 --> 00:08:35,519
stuff cut do you want to get a close-up of ah my foot

124
00:08:34,240 --> 00:08:41,479
cut all right we're back where we started and cut now to just do one hba

125
00:08:38,320 --> 00:08:41,479
at a time

126
00:08:42,320 --> 00:08:47,360
one and cut what does this g3 denote so

127
00:08:45,760 --> 00:08:52,160
i can cut go ahead and cut oh weird just cut this

128
00:08:49,839 --> 00:08:57,279
is very boring troubleshooting after a BIOS update and a cmos clear

129
00:08:55,920 --> 00:09:01,279
it's up everything is detected what do you think

130
00:08:59,600 --> 00:09:03,920
i'm just gonna friction mount it no no i'm gonna put some hot glue along the

131
00:09:02,640 --> 00:09:08,640
top of the card edges and i'm gonna stick it down on it don't worry about it it's all good i read about it in a

132
00:09:06,720 --> 00:09:13,440
textbook somewhere these babies ain't going to deliver themselves i i i

133
00:09:11,360 --> 00:09:17,600
literally cannot tell if you're joking people claim they don't love the gore

134
00:09:14,800 --> 00:09:22,000
but they love the gore it's in it's done we can put drives in it now right yeah i

135
00:09:20,000 --> 00:09:26,399
mean theoretically it's hot plug right heck yeah oh right okay so no we should

136
00:09:25,279 --> 00:09:31,440
just do this in the server and that one's a lot harder to remove the drives from we have to remove the drives in

137
00:09:28,959 --> 00:09:34,480
order to transport it so i think that's the extent of the

138
00:09:32,959 --> 00:09:38,480
testing we can do here what about this we're gonna have to remove all the

139
00:09:37,200 --> 00:09:41,519
drives and then put it back together in there

140
00:09:42,399 --> 00:09:49,440
okay do we have to put them back in the same order yeah i think so okay

141
00:09:46,640 --> 00:09:54,320
unfortunately we're out of power plugs the good news is i did buy a second one

142
00:09:51,680 --> 00:09:57,360
that isn't mounted so we can put that in but the reason i didn't bother doing it

143
00:09:56,000 --> 00:10:02,240
was that it was going to be a royal pain in the butt checklist

144
00:09:59,600 --> 00:10:08,160
put in new pdu for more power move albion server up

145
00:10:04,959 --> 00:10:10,800
figure out all the kvm cables and labels

146
00:10:08,160 --> 00:10:17,440
so that everything corresponds again and then add two more servers easy right

147
00:10:14,160 --> 00:10:19,839
easy when you put it like that yeah

148
00:10:17,440 --> 00:10:19,839
okay

149
00:10:20,480 --> 00:10:26,959
wait no missed up a bit good in

150
00:10:25,040 --> 00:10:31,120
i've had worse man am i gonna have to just kind of jam it in there i'm glad

151
00:10:28,880 --> 00:10:35,839
you guys can't see this right now okay we've got power we've got kvm

152
00:10:34,079 --> 00:10:39,839
so we can throw them in plug in network and then we're good to go

153
00:10:37,360 --> 00:10:43,279
all right this is awkward mind labeling that will i put the rails in pull out

154
00:10:41,200 --> 00:10:47,910
just a little there we go should be in easy peasy

155
00:10:46,000 --> 00:10:51,760
fill her up boys

156
00:10:51,760 --> 00:10:59,040
my ARM so strong they don't understand boom box so big carry it with two hands

157
00:10:56,800 --> 00:11:03,120
all i wanna do is sit back and kick it all the cops wanna do is write me a

158
00:11:01,040 --> 00:11:06,480
ticket wait this is four why'd i put it here we gotta move it up one go ahead

159
00:11:05,200 --> 00:11:10,160
and cut oh man now to make transporting this a

160
00:11:08,880 --> 00:11:14,560
little bit easier we're going to take out all the drives put them on this

161
00:11:12,560 --> 00:11:17,040
little cart here and transport this separately

162
00:11:17,839 --> 00:11:23,519
we're gonna make the biggest gluster cluster we can by filling it up with

163
00:11:22,160 --> 00:11:26,160
some of the drives that we're just gonna return later

164
00:11:27,040 --> 00:11:33,360
uh i guess yeah i'll just grab it sure yeah you know what i got it don't worry

165
00:11:30,640 --> 00:11:38,240
about it no problem okay okay yeah boom box so big carry it with two hands yeah

166
00:11:36,000 --> 00:11:42,720
it's fine i'll just do this that's fine i got it oh i thought you were being no

167
00:11:41,040 --> 00:11:46,480
now the main trick is to put these all in the right spots all the storage we'll

168
00:11:45,120 --> 00:11:51,200
ever need ever wow that's a total capacity of like

169
00:11:49,200 --> 00:11:54,160
300 drives i noticed that this one's flashing green

170
00:11:52,720 --> 00:11:59,200
though go ahead and cut let's see so you changed the melanox

171
00:11:55,760 --> 00:12:02,079
card out yep so we got the static ip 44

172
00:11:59,200 --> 00:12:06,160
and we are able to ping rock on now we just needed the same thing to delta

173
00:12:03,920 --> 00:12:11,200
three how long does it take though however

174
00:12:08,720 --> 00:12:14,800
long it takes for me to manually oh my god you have to key those in

175
00:12:13,440 --> 00:12:18,399
yeah so we're picking this up tomorrow

176
00:12:16,800 --> 00:12:22,839
yeah okay yeah

177
00:12:20,160 --> 00:12:28,000
okay we're so close the good news is that Anthony

178
00:12:25,440 --> 00:12:32,800
found a script that makes it easy to import all the drives and it doesn't say

179
00:12:30,720 --> 00:12:36,560
nothing here buds yeah no way

180
00:12:33,920 --> 00:12:42,000
they are the most eastern canadian tech company of all time wait so how do we

181
00:12:39,600 --> 00:12:45,760
have hold on a second um but that doesn't make any sense because

182
00:12:43,920 --> 00:12:51,360
that's whatever that's whatever that's doing i have no idea it's lifeline time

183
00:12:48,560 --> 00:12:54,639
yo brett long story short we tried to use the new script

184
00:12:55,200 --> 00:13:01,839
okay awesome so it's an older chassis and i repurposed it with new hardware so

185
00:12:59,920 --> 00:13:05,519
there's a drive they're just not showing up okay give me a second i'm gonna do

186
00:13:03,519 --> 00:13:09,519
some open heart surgery here moving all the cables is going to be a little dumb

187
00:13:08,240 --> 00:13:14,560
don't fix your hardware to fix your software no big deal okay all right

188
00:13:12,480 --> 00:13:19,040
you guys having fun yet oh yeah it's always fun

189
00:13:16,959 --> 00:13:23,120
so with you just i know it's kind of annoying but change the cables so that

190
00:13:21,200 --> 00:13:28,240
they're in this order and then we'll fix the order of the cards in the d-map

191
00:13:24,880 --> 00:13:30,560
thing and we're home 45 drives never fix

192
00:13:28,240 --> 00:13:34,800
the software problem with hardware also 45 drives can you unplug literally

193
00:13:33,279 --> 00:13:39,440
all the cables and put them in the other way in their defense this is not a

194
00:13:36,959 --> 00:13:42,800
supported configuration so are they just in reverse order on this motherboard

195
00:13:41,120 --> 00:13:47,600
because i think super micro numbers them from bottom to top i think so you're

196
00:13:45,279 --> 00:13:50,639
coming out for ltx right yeah hell yeah okay cool i'll see you then then okay

197
00:13:49,360 --> 00:13:53,200
sweet they're all showing up so it was just a matter of

198
00:13:54,000 --> 00:14:00,160
uh it was just a matter of uh reseeding all the cables in a different order

199
00:13:58,639 --> 00:14:04,639
sort of yeah sure yes so we're ready to pull

200
00:14:02,639 --> 00:14:08,959
them together and then start samba and then we can see over one petabyte

201
00:14:06,800 --> 00:14:12,000
accessible over the network we're ready to

202
00:14:10,240 --> 00:14:15,600
pull them together uh they recommend that a single brick shouldn't be more

203
00:14:13,440 --> 00:14:20,160
than 100 terabytes in size wait so this is good probes are good probes are very

204
00:14:17,839 --> 00:14:23,760
good okay we just probed our pure probe success probe success let me do the

205
00:14:21,760 --> 00:14:27,440
other way around pure probing do you want to see it i do want to see it now

206
00:14:28,000 --> 00:14:34,639
last time we had a petabyte of raw storage

207
00:14:32,480 --> 00:14:38,880
but because of overhead loss to redundancy and whatever

208
00:14:36,959 --> 00:14:43,120
the case may be we ended up with what like

209
00:14:40,079 --> 00:14:44,560
600 something yeah terabytes

210
00:14:43,120 --> 00:14:49,600
this time 1.2

211
00:14:46,800 --> 00:14:53,680
petabytes of storage it's ready would you say that's a new pb

212
00:14:52,160 --> 00:14:59,120
what's a pb personal best yeah definitely so i'm

213
00:14:56,320 --> 00:15:03,360
just gonna take a quick test file here how about this three gig file

214
00:15:02,000 --> 00:15:09,760
we going so yeah it's not quite 10 gig speeds around

215
00:15:06,639 --> 00:15:11,040
730 750 gigabytes a second but

216
00:15:09,760 --> 00:15:18,720
considering that we're writing to magnetic storage i am pretty pleased

217
00:15:16,240 --> 00:15:22,000
oh we never installed pulseway on the server andy just get Anthony installing

218
00:15:20,880 --> 00:15:25,680
pulseway it's up already just like that delta 3

219
00:15:24,000 --> 00:15:28,320
21 hours of uptime don't you wish the gloucester setup was that easy i know

220
00:15:27,440 --> 00:15:34,240
right so thanks for watching guys if you enjoyed this maybe check out our video

221
00:15:31,760 --> 00:15:37,120
rolling out our high speed NVMe server it was

222
00:15:35,199 --> 00:15:40,639
almost as headache inducing as this one actually more so this this was

223
00:15:38,720 --> 00:15:44,959
relatively painless most of the problems were uh caused by me deciding that we

224
00:15:43,360 --> 00:15:49,920
were gonna roll two servers at once instead of just one so yeah

225
00:15:48,320 --> 00:15:55,639
we're gonna have that video linked below thanks andy he's been a trooper he's

226
00:15:51,920 --> 00:15:55,639
been a trooper for this