1
00:00:00,240 --> 00:00:08,320
i'm not feeling great today but we have a dire situation here at Linus tech tips

2
00:00:04,960 --> 00:00:11,360
our wanik server our main production

3
00:00:08,320 --> 00:00:12,400
editing storage server is completely

4
00:00:11,360 --> 00:00:16,800
full fortunately i have been planning an

5
00:00:14,799 --> 00:00:20,080
upgrade to it for the last little while that is going to be freaking awesome

6
00:00:19,119 --> 00:00:24,640
this is new wanik it's gonna have more

7
00:00:22,720 --> 00:00:31,359
storage space whoa over 100 terabytes of NVMe storage it's

8
00:00:29,039 --> 00:00:35,440
gonna be faster it is going to have lower power consumption it is going to

9
00:00:33,280 --> 00:00:40,480
have more processing cores it is going to be epic and perhaps the most epic

10
00:00:38,239 --> 00:00:45,760
thing about it is going to be how many pci express lanes it has you see pci

11
00:00:43,920 --> 00:00:50,800
express gets used for all kinds of things graphics cards video capture

12
00:00:48,480 --> 00:00:56,719
devices peripheral expansion but thing is it's been fast enough for all those

13
00:00:53,520 --> 00:00:59,199
consumer uses for many years now so

14
00:00:56,719 --> 00:01:05,360
what's driving the creation of faster and faster pci express is actually the

15
00:01:02,559 --> 00:01:10,320
server world where it gets used for storage that's right that is a direct

16
00:01:07,680 --> 00:01:16,400
pci express by four connection at the back of all 24 of these drives and the

17
00:01:14,000 --> 00:01:20,960
truly unbelievable thing about this server is that unlike our existing wanik

18
00:01:18,720 --> 00:01:25,680
server which takes a handful of those pci express connections and splits it

19
00:01:23,600 --> 00:01:30,159
across 48 drives this one has a dedicated connection for

20
00:01:28,880 --> 00:01:33,920
every single one we are going to be putting up some crazy numbers today

21
00:01:32,320 --> 00:01:39,119
ladies and gentlemen and it's going to be brought to you by seasonic no

22
00:01:36,640 --> 00:01:42,880
marketing today just happy holidays from seasonic go check them out at the link

23
00:01:41,040 --> 00:01:49,479
in the video description really they they paid for a spot for that

24
00:01:46,159 --> 00:01:49,479
nice guys

25
00:01:54,640 --> 00:02:00,079
while i'm cracking these open let's talk about the drives that i'm using for the

26
00:01:58,399 --> 00:02:04,640
server and why reason number one is that wanik is full

27
00:02:02,960 --> 00:02:10,160
so i needed a capacity upgrade we're going from 1.2 terabyte drives to 4

28
00:02:08,239 --> 00:02:13,920
terabyte drives because that's what a couple of years of NVMe development does

29
00:02:12,080 --> 00:02:18,640
for you reason number two is we didn't want to go back to SATA because NVMe was

30
00:02:16,480 --> 00:02:23,520
such a great upgrade for us in terms of not just performance but stability last

31
00:02:21,280 --> 00:02:28,160
time thanks to the extremely low access latencies reason number three and really

32
00:02:26,400 --> 00:02:33,120
the main reason for these drives in particular is that we got a great tip

33
00:02:30,480 --> 00:02:37,599
from wendell over at level one techs that facebook was apparently flipping a

34
00:02:35,519 --> 00:02:44,480
bunch of these on ebay because they're upgrading to optane so these were a

35
00:02:40,879 --> 00:02:47,280
really great deal just a hair over 350

36
00:02:44,480 --> 00:02:52,560
dollars for Intel data center grade drives that apparently are still under

37
00:02:49,840 --> 00:02:56,879
warranty on top of everything else now they're not the fastest thing in the

38
00:02:53,920 --> 00:03:02,720
world by today's standards 3.2 gigabytes per second reads 1.8 gigabytes per

39
00:02:59,200 --> 00:03:06,000
second rights maximum read iops of 645

40
00:03:02,720 --> 00:03:08,400
000 and rights of 48 000 but remember

41
00:03:06,000 --> 00:03:15,760
guys that's almost not going to matter at all because i'm going to have 24

42
00:03:12,400 --> 00:03:17,280
of them in the same server and it's just

43
00:03:15,760 --> 00:03:21,879
a naz anyway now let's meet our server this

44
00:03:20,080 --> 00:03:27,040
is the Gigabyte r272 z32

45
00:03:24,800 --> 00:03:31,840
and we chose this particular model based on the glowing review that was given to

46
00:03:29,360 --> 00:03:35,760
it by our buddy patrick kennedy over at serve the home oh

47
00:03:34,480 --> 00:03:40,720
heavy boy this is a 2u server so that's to say it

48
00:03:38,239 --> 00:03:46,400
is two rack units in height and it was designed from the ground up for AMD's

49
00:03:43,360 --> 00:03:48,400
epic 7002 rome platform

50
00:03:46,400 --> 00:03:55,599
and the advantage we get from that is that it actually has compatibility with

51
00:03:50,720 --> 00:03:58,319
pci express gen 4 not all epic servers

52
00:03:55,599 --> 00:04:04,239
are PCIe gen 4 ready what gen 4 compatibility means for us in practical

53
00:04:00,319 --> 00:04:06,000
terms is not necessarily that much today

54
00:04:04,239 --> 00:04:09,920
but in the event that we wanted to upgrade it means that we could

55
00:04:07,280 --> 00:04:14,879
effectively double the bandwidth to almost the whole system so let's take a

56
00:04:12,080 --> 00:04:18,239
quick tour here we've got dual 1.2 000 watt power supplies here those are

57
00:04:16,320 --> 00:04:24,479
redundant in case one fails we've got our CPU socket here with support for up

58
00:04:20,160 --> 00:04:26,960
to 64 cores 128 threads we've got 16

59
00:04:24,479 --> 00:04:30,720
memory slots that run in eight channel mode freaking incredible

60
00:04:28,800 --> 00:04:36,639
bandwidth and on the subject of bandwidth most of the internal pci

61
00:04:33,520 --> 00:04:39,600
express slots are taken up of course by

62
00:04:36,639 --> 00:04:44,160
the 24 u.2 slots at the front so the way that they're fed is they're using their

63
00:04:41,759 --> 00:04:48,080
mezzanine card slot here to run four of them you can see that managed over here

64
00:04:46,400 --> 00:04:53,600
SATA actually goes to these two bays at the back so that's what you would typically boot off of now the CPU that

65
00:04:52,000 --> 00:05:00,000
we're using is pretty overkill but AMD sent over a

66
00:04:56,720 --> 00:05:01,919
bunch of cpus and unfortunately

67
00:05:00,000 --> 00:05:06,160
all the ones that kind of make more sense for this project like lower core

68
00:05:04,080 --> 00:05:11,919
count ones i sort of have earmarked for other projects so unfortunately i am

69
00:05:08,560 --> 00:05:14,080
locking away my 132 core

70
00:05:11,919 --> 00:05:17,360
basically forever that's okay though we really need the upgrade

71
00:05:16,160 --> 00:05:24,160
it really says a lot about the efficiency of these things that all we need is one of these basic

72
00:05:20,639 --> 00:05:26,240
passive heat pipe coolers thrown on here

73
00:05:24,160 --> 00:05:29,840
one of these shrouds and just the airflow from one of the chassis 80

74
00:05:27,840 --> 00:05:34,960
millimeter fans to keep it cool and remember guys it's validated for up to

75
00:05:31,759 --> 00:05:36,800
the 64 core one this is cool for my boot

76
00:05:34,960 --> 00:05:40,400
drive i had intended to use the SATA bays at the back of the chassis

77
00:05:38,479 --> 00:05:45,759
but then i realized i've actually got two m.2s that share

78
00:05:43,680 --> 00:05:50,479
their pci express lanes with this 8x slot over here so i figured if i have a

79
00:05:48,240 --> 00:05:53,360
couple of these old octane m.2s lying around

80
00:05:51,600 --> 00:05:58,240
what the hey might as well throw them in raid 1 and

81
00:05:54,880 --> 00:06:00,800
go full pci express on everything all

82
00:05:58,240 --> 00:06:04,800
right let's turn it around and oh oh that's funny Gigabyte sent over a uh a

83
00:06:03,520 --> 00:06:09,919
demo unit it seems to be a little bit broken

84
00:06:06,880 --> 00:06:11,759
that's okay oh they've got the stupid

85
00:06:09,919 --> 00:06:15,840
cable management things that make it really hard to plug in i mean it also

86
00:06:14,240 --> 00:06:21,609
makes it hard to come out accidentally which is nice

87
00:06:22,240 --> 00:06:28,080
now i've actually worked with this motherboard a little bit already for a

88
00:06:25,680 --> 00:06:31,680
whole home one CPU and it takes

89
00:06:30,400 --> 00:06:37,120
forever to post so i'm gonna fire it up while we check

90
00:06:35,280 --> 00:06:41,039
the compatibility of the sleds that our drives were pre-installed on

91
00:06:39,120 --> 00:06:44,960
and see if we're gonna have to swap all those out i really really hope i don't

92
00:06:42,800 --> 00:06:49,120
have to it's up

93
00:06:47,120 --> 00:06:53,360
that's good let's get in the BIOS and see uh if everything's detected

94
00:06:51,199 --> 00:06:56,560
correctly come on baby everything detector oh would that but

95
00:06:55,120 --> 00:07:03,840
memory training error what with the what now that is not all of our RAM that is

96
00:07:00,039 --> 00:07:06,479
458 gigs of RAM uh-oh okay brandon i've

97
00:07:03,840 --> 00:07:12,400
got good news and bad news um the good news is resetting those two memory

98
00:07:08,479 --> 00:07:14,400
modules boom 512 gigs of RAM so we're

99
00:07:12,400 --> 00:07:19,120
ready to rock as far as that goes the bad news is that due to AMD's

100
00:07:17,039 --> 00:07:24,240
architecture of their epic processors where there's no actual chipset with

101
00:07:21,680 --> 00:07:30,800
functionality like you know a raid controller or anything like that this

102
00:07:26,800 --> 00:07:34,080
has no support for hardware raid either

103
00:07:30,800 --> 00:07:35,599
on the m.2 drives or the SATA ones if

104
00:07:34,080 --> 00:07:40,000
you want to run raid on your state of drives you have to put an add-in raid

105
00:07:37,919 --> 00:07:43,680
card in and i ask Gigabyte well why didn't you guys just put

106
00:07:41,919 --> 00:07:48,639
a raid chip on the motherboard and they were like well it would have used up pci

107
00:07:45,759 --> 00:07:52,880
express lanes which i mean as part of the design of this board we were trying

108
00:07:50,319 --> 00:07:57,520
to reserve for all these NVMe drives so i kind of went ah okay fair enough now

109
00:07:54,720 --> 00:08:02,000
in Linux you can run your os on rated drives but Windows has no easy way to do

110
00:08:00,319 --> 00:08:06,370
that so that's just something we're going to have to consider as we

111
00:08:04,800 --> 00:08:11,120
build out this machine i'm going to shut it down for now though

112
00:08:09,199 --> 00:08:17,199
because the last thing that i really want to know is if i am remounting all

113
00:08:14,080 --> 00:08:18,879
of these drives to the included sleds

114
00:08:17,199 --> 00:08:22,800
let's see if they happen to be intercompatible that would be so cool

115
00:08:20,960 --> 00:08:25,599
that would be so cool oh

116
00:08:24,000 --> 00:08:31,199
yes yeah they are oh it is so tedious it's so

117
00:08:28,400 --> 00:08:36,240
tedious well that's not a good sign uh only my two m.2 drives are in here

118
00:08:34,240 --> 00:08:41,760
which means that maybe the sleds don't go all the way back

119
00:08:39,760 --> 00:08:44,880
that would blow all right

120
00:08:42,800 --> 00:08:48,480
i put one drive in each one and lined it up and

121
00:08:46,480 --> 00:08:52,080
you can see it doesn't go back far enough i gotta swap them all

122
00:08:52,240 --> 00:08:57,680
damage and i've had people ask me why so many

123
00:08:55,680 --> 00:09:01,200
people work here you thought i was gonna shock all these drives by myself and put

124
00:09:00,000 --> 00:09:07,279
them in mounts what one person can do slowly four people can do also slowly but less

125
00:09:04,880 --> 00:09:10,480
slowly pretty tedious now i just need to install a couple of roles and features

126
00:09:08,720 --> 00:09:13,279
including file server then

127
00:09:12,240 --> 00:09:17,680
yes we're going to shut it down and put all the drives in

128
00:09:16,000 --> 00:09:21,200
and we're going to hope they all work now really the most benefit for a

129
00:09:19,680 --> 00:09:26,560
solution like this is when you've got like a a storage

130
00:09:23,600 --> 00:09:32,320
attached network and you're using like banks and banks of these to act as

131
00:09:29,360 --> 00:09:36,080
the storage for like a whack of compute servers that are connected over fiber

132
00:09:34,160 --> 00:09:39,279
optics on you know the other end of the data center or whatever the case may be

133
00:09:38,240 --> 00:09:46,000
but come on guys this line is tech tips so we're using it

134
00:09:43,440 --> 00:09:49,839
as an 8k video editing nas i really hope these all pick up worst

135
00:09:48,240 --> 00:09:55,440
case scenario though we've got a couple of bad ones and the key night among you might have

136
00:09:52,640 --> 00:09:59,839
noticed there's two drives not installed these are either cold spares or to

137
00:09:57,920 --> 00:10:03,279
account for if a couple of our drives are just doa i mean that's the thing

138
00:10:01,120 --> 00:10:06,560
with used hardware you get a good deal you might get a couple bad ones all

139
00:10:04,959 --> 00:10:10,480
right that's a good sign every one of them has its light lit but

140
00:10:08,959 --> 00:10:14,640
we won't know if everything's picked up

141
00:10:12,240 --> 00:10:18,720
until we actually see them in here boom okay they're all there man storage

142
00:10:16,800 --> 00:10:23,839
spaces takes a lot of your capacity by default oh actually that's not that bad

143
00:10:20,640 --> 00:10:24,720
it's about 70 terabytes

144
00:10:23,839 --> 00:10:31,120
okay let's try that as long as it works oh you got to be

145
00:10:27,920 --> 00:10:32,880
kidding me this stupid error like

146
00:10:31,120 --> 00:10:37,360
five years or whatever it's had this problem and they don't just allow you to choose

147
00:10:35,760 --> 00:10:40,480
your columns in the gui you have to do it via

148
00:10:39,120 --> 00:10:44,399
powershell so dumb oh you can hear it ramping up

149
00:10:42,560 --> 00:10:48,399
now boys 5.5 gigs a second

150
00:10:46,480 --> 00:10:52,320
well that result is horrible worse across the board than old wanik

151
00:10:50,640 --> 00:10:56,320
clearly there's a configuration issue here all right so i did a simple

152
00:10:54,000 --> 00:11:01,200
mirrored space this time and we're gonna try running that we've

153
00:10:58,320 --> 00:11:04,160
got 44 terabytes here well that's better when it comes to

154
00:11:02,480 --> 00:11:09,120
sequential reads almost 11 gigs a second not even close

155
00:11:07,200 --> 00:11:15,360
to what we should be getting though okay that's it we're trying a simple virtual

156
00:11:11,680 --> 00:11:17,600
disk which is basically just striped

157
00:11:15,360 --> 00:11:22,959
if this doesn't perform well then i'm at a loss wow that is complete garbage town

158
00:11:20,800 --> 00:11:26,399
it's even worse how is that even possible

159
00:11:24,079 --> 00:11:30,560
and crystal disk mark is not the issue there's a microsoft tool called disk

160
00:11:28,079 --> 00:11:33,839
speed that you can use via command line to run sort of any mixture of loads that

161
00:11:32,880 --> 00:11:36,800
you want and it seems like there's just this hard

162
00:11:35,920 --> 00:11:43,279
cap at around 10 gigabytes a second which is like

163
00:11:41,360 --> 00:11:48,480
i mean obviously it's like good or whatever but for this hardware

164
00:11:46,000 --> 00:11:52,060
it's terrible so we're going to freenas

165
00:11:50,160 --> 00:11:54,560
that's it

166
00:11:54,560 --> 00:12:01,680
and it's been about four days so we did manage to push Windows

167
00:11:58,720 --> 00:12:05,600
performance a little bit further uh the drives needed firmware updates and they

168
00:12:03,839 --> 00:12:10,320
were doing some idle garbage collection stuff like that but we capped out at

169
00:12:07,680 --> 00:12:13,279
about 10 gigabytes a second which to be clear is plenty for anything you're

170
00:12:11,920 --> 00:12:17,440
going to access over the network i mean we would need a 100 gigabit per second

171
00:12:15,360 --> 00:12:20,000
network card in order to saturate that

172
00:12:18,880 --> 00:12:27,519
um but that's not enough for us we know this thing should be capable of so much

173
00:12:25,120 --> 00:12:33,680
more so we moved on to Linux Jake loaded it up with proxmox and built a pool with

174
00:12:30,880 --> 00:12:40,079
four v devs each with six drives in them running in raid z1 so raid z1 is kind of

175
00:12:37,519 --> 00:12:45,279
like raid 5 but without the right hole issue meaning that we should be giving

176
00:12:42,240 --> 00:12:48,240
up uh let's see four drives worth of

177
00:12:45,279 --> 00:12:53,040
capacity total but with the resiliency of being able to lose up to four drives

178
00:12:51,200 --> 00:12:57,040
as long as we don't lose more than one in a single v dev so you'd be well let's

179
00:12:55,600 --> 00:13:00,720
see six so four chunks like that of drives working

180
00:12:58,959 --> 00:13:05,279
together now that should have resulted in some rocking

181
00:13:02,720 --> 00:13:09,600
performance except it didn't it was actually complete

182
00:13:06,959 --> 00:13:15,760
dog crap like what did we max out at like a hundred giga megabytes a second

183
00:13:12,560 --> 00:13:20,720
maybe 200 200 megs a second

184
00:13:15,760 --> 00:13:23,200
terrible so turns out as soon as zfs was

185
00:13:20,720 --> 00:13:28,560
being loaded up our machine was actually spitting out kernel panics

186
00:13:26,240 --> 00:13:34,160
that's not a chicken emergency that's like really bad you liked it

187
00:13:31,040 --> 00:13:36,160
shut up you liked it that makes no sense

188
00:13:34,160 --> 00:13:41,279
a single one of these drives if i plug it into my computer should be doing 15

189
00:13:38,480 --> 00:13:45,600
to 20 times that performance and we verified this we checked in with our bud

190
00:13:43,360 --> 00:13:50,880
patrick over at serve the home he's got one of these boxes and he was getting

191
00:13:47,600 --> 00:13:53,120
great numbers with just 10 drives

192
00:13:50,880 --> 00:13:57,519
so what could we do well we called in the tech support for

193
00:13:55,200 --> 00:14:02,240
tech support wendell from level one tax got on the horn with Jake chatting it up

194
00:13:59,760 --> 00:14:06,800
on discord and between the two of them they managed to figure out that it looks

195
00:14:04,240 --> 00:14:11,120
like there is some kind of compatibility issue between the particular Linux

196
00:14:09,360 --> 00:14:16,880
kernel that the latest version of proxmox is running and zfs in our

197
00:14:14,320 --> 00:14:21,920
particular system that's causing these problems so we said you know what

198
00:14:19,519 --> 00:14:26,480
it we're just gonna run a benchmark where we load up each of the drives

199
00:14:23,839 --> 00:14:32,160
individually and slam them all at the same time Jake hit it

200
00:14:28,880 --> 00:14:32,160
he wandered away he'll be back

201
00:14:32,480 --> 00:14:37,839
okay you hear that thing ramping up

202
00:14:35,760 --> 00:14:42,399
oh my goodness oh my goodness what's going on here is

203
00:14:39,920 --> 00:14:48,000
that over a Gigabyte per second per drive 28 gigabytes a second

204
00:14:45,519 --> 00:14:53,680
at those kinds of speeds we are at the limits of AMD's infinity fabric and in

205
00:14:50,880 --> 00:14:58,079
fact to get to those speeds we actually had to overclock it a little bit thank

206
00:14:55,839 --> 00:15:00,560
you wendell for the tip there we pushed it too far it's actually spitting out

207
00:14:59,519 --> 00:15:05,519
some memory errors we're gonna have to dial it back a little bit but whatever you guys want to see impressive numbers

208
00:15:03,360 --> 00:15:09,199
right well here they are now i'm running a second script that he created that

209
00:15:07,199 --> 00:15:14,600
writes what was it 15 gigs of data so this is gonna write to the drives this

210
00:15:11,279 --> 00:15:18,240
time these numbers are nuts there you go

211
00:15:14,600 --> 00:15:21,600
23.7 gigabytes a second that are right

212
00:15:18,240 --> 00:15:25,199
up against the theoretical limits of

213
00:15:21,600 --> 00:15:26,720
what can be done so that's it guys that

214
00:15:25,199 --> 00:15:32,880
is basically the pinnacle of modern technology nearly

215
00:15:29,760 --> 00:15:36,160
30 gigabytes a second of reads and just

216
00:15:32,880 --> 00:15:38,240
over 20 gigabytes a second of rights

217
00:15:36,160 --> 00:15:42,199
guys to put that in perspective your home network is going to be probably

218
00:15:40,160 --> 00:15:48,240
gigabit this would be 300 times that to read it you could read

219
00:15:46,079 --> 00:15:51,440
one full blu-ray disc almost every second

220
00:15:50,240 --> 00:15:57,360
blu-ray blu-ray blu-ray it's crazy stuff now we just

221
00:15:55,440 --> 00:16:01,519
need to figure out some minor details like how we're actually gonna get an

222
00:15:59,680 --> 00:16:04,720
array loaded up on the thing because this is just

223
00:16:02,720 --> 00:16:07,920
i mean this is just theoretical stuff right now

224
00:16:05,839 --> 00:16:10,880
and theoretically i could have a good segue for a change but we all knew that

225
00:16:09,360 --> 00:16:15,120
wasn't going to happen thanks to our sponsor freshbooks for making this video

226
00:16:13,040 --> 00:16:18,720
possible today freshbooks is the cloud accounting solution that's built for how

227
00:16:16,880 --> 00:16:22,959
you want to work and you can work not just how you want but anywhere you want

228
00:16:20,959 --> 00:16:27,120
thanks to the freshbooks mobile app you can create professional looking invoices

229
00:16:24,959 --> 00:16:31,279
on the go snap pictures of your receipts so you don't lose them stay on top of

230
00:16:29,199 --> 00:16:35,199
important conversations and never miss an update for example being able to see

231
00:16:33,600 --> 00:16:39,040
when a client has viewed your invoice for the first time so you can get your

232
00:16:36,959 --> 00:16:44,320
money you know what i'm talking about yeah you do so visit freshbooks.com

233
00:16:42,079 --> 00:16:47,680
forward slash tech tips to get your free 30-day trial today we're going to have

234
00:16:45,920 --> 00:16:50,560
that linked in the video description so thanks again for watching guys if you're

235
00:16:49,040 --> 00:16:54,320
looking for something else to watch server related maybe check out the video

236
00:16:52,320 --> 00:16:59,519
where i lost all our data you know i was accused in the comments of that video of

237
00:16:55,920 --> 00:17:02,639
being a very bad actor uh brandon out of

238
00:16:59,519 --> 00:17:04,720
10 how much was i acting in that video

239
00:17:02,639 --> 00:17:07,280
negative 10. that happened that was a real thing