These Servers are TOO EXPENSIVE - Hybrid Storage Explored
Linus Tech Tips
·Linus Tech Tips
·2019-05-06
·
2,223 words · ~11 min read
0:00
so i've got a problem when it comes to building a storage server there are two
0:03
main options for your drives you can either go with ssds which are compact
0:08
and have amazing performance or you can go with traditional mechanical storage
0:14
which has the benefit of lower prices and outstanding capacity with the
0:19
drawback of being an order of magnitude
0:22
or more slower especially in highly
0:26
random workloads like if they were loaded up with a bunch of virtual
0:29
machines or in our case if they're pulling network storage duty for a whole
0:35
bunch of heavy users who are all trying to edit 8k videos at the same time if
0:41
only there were some way to
0:44
merge the benefits of both
0:47
with the drawbacks of neither
0:50
well there is it's called the apple fusion drive
0:54
and this episode is brought to you by ifixit the marlin screwdriver set from
0:58
ifixit features five specialty precision screwdrivers check it out at ifixit.com
1:03
forward slash Linus at the link below
1:13
the concept of tiered storage is nothing new it's been commonplace in the data
1:18
center for many years and has even made its way down to the consumer level in
1:22
the form of Intel's rapid storage technology or rst which uses a small
1:28
fast SSD to accelerate a larger slower
1:32
mechanical drive and more recently Intel's octane memory technology which
1:37
does pretty much the exact same thing but with even faster optane rather than
1:43
nand ssds the problem that i'm running
1:46
into here is that most of the tiering solutions out there have one of two
1:52
problems they are either inordinately complicated to set up requiring
1:57
extensive knowledge of not just Linux but even broader storage architecture or
2:03
they are expensive because you are paying the folks who have the
2:07
aforementioned knowledge to create a turnkey solution for you
2:12
there are some exceptions to this however and i would like to explore a
2:16
couple of them so the popular freebsd based freenas software from ix systems
2:22
uses your system's RAM and its ssds to cache frequently used
2:28
data that's stored on your mechanical hard drives but the issue is that zfs's
2:33
particular implementation is geared more towards accelerating performance of
2:39
large databases and doesn't look like it's going to be great for our workload
2:44
because the file sizes that we want cached that we're storing on our hard
2:48
drives are so large like these are 8k
2:51
red video files and there are practical
2:55
limits to how much level 2 arc that
2:59
would be the SSD tier i mean there's also practical limits to how much RAM
3:03
you can put in a given system
3:06
so then the one we're gonna try today is the
3:10
built-in tiering mode for storage spaces
3:13
in Windows server and what better way to test this out than to set up a tiered
3:19
storage space on the network so i'm going to use this test server right here
3:24
and then have our team of editors just switch over to it and see what happens
3:32
and it's at this point then that we wondered whether or not optane would
3:36
make a difference for some of the heavy tasks that we deal with around the
3:39
office no no it does not now our regular editing
3:44
server uses 31 1.2 terabyte NVMe drives
3:49
and it was really expensive for us to build and they've been complaining that
3:53
there's not enough space on it because we've only got about 24 terabytes of
3:58
usable space after all the lost space to
4:01
redundancy on it and they were like oh well you could just upgrade it i went yeah okay
4:06
technically i do have another 17 bays that i could
4:11
fill up but that's going to cost me like seven grand eight grand even if i buy
4:16
used drives on ebay so i was
4:20
looking for another way so our test server here the config is
4:25
pretty straightforward i've got four 10
4:28
terabyte drives then because i want to see exactly how much
4:34
cash you need this is what really got me thinking about it
4:37
Intel uses just 32 gigs of cash
4:42
for large multi-terabyte hard drives and we've
4:46
seen real world huge increases in performance so i went okay let's try
4:52
this so i took two of their 900 series
4:56
optane pci express drives and then i've got a total of 40 terabytes of
5:01
mechanical storage and i want to see if for a fraction of the price
5:06
we can get good enough performance that
5:09
the editors aren't gonna run into a bottleneck
5:12
so you can see i've got my four mechanical drives i've got my two ssds
5:18
uh allocation is all gonna be automatic so i go ahead and i try to create a pool
5:23
okay i've created a pool now i try to
5:27
create a virtual disk create storage tiers on this virtual disk it should
5:32
automatically detect which ones are your fast drives and which ones are your
5:38
capacity drives so we're going to use mirroring because
5:42
that way we're protected from a drive failure we're going to use fixed
5:46
provisioning and now we get to specify the size of our faster and our standard
5:51
tier we're going to go ahead and select the maximum sizes failed to create
5:55
virtual disks not supported so the issue is that the gui version of storage
5:59
spaces doesn't have some very important
6:02
options like selecting how many columns you want
6:06
so when you only have two drives and the
6:09
default is three columns it spits out this stupid error how did
6:14
nobody at microsoft notice this for
6:17
years anyway doesn't matter
6:20
i called in a lifeline led eater over on the forum as a guru when it comes to
6:25
this stuff so i'm just going to pull up his dm over there and follow his
6:29
instructions to hopefully do this in powershell so this will be fun you guys
6:33
are going to be learning right along with me i haven't done this before the
6:36
video so i can still use the gui to create my storage pool apparently
6:41
i'm going to go ahead and do that then i need to fire up powershell and
6:45
change the default number of columns
6:49
okay well i don't know what i just did
6:52
but i'm going to change it to mirror
6:56
number of columns default one i think i think we did that let's
7:02
find out so i think the command i just entered
7:08
changed the mirror layout default
7:12
to one column it did not
7:17
okay uh so i'm using powershell ise now
7:22
and this seems to be working better
7:26
mirror number of columns default one
7:30
come on so mad right now
7:35
i just i just want to change a stupid
7:39
setting it's definitely called
7:43
mirror it's definitely called storage pool one like i create
7:48
these commands using their own thing
7:52
we'll figure this out at some point the good news is there's a backup plan
7:56
i swear i had this working at one point
8:00
what is this oh okay oops actually yeah
8:05
oops all right
8:09
here we go so right now we've only got a 10 gig link i can put a faster card in
8:14
here if it ends up being a networking bottleneck but for now what we need to
8:18
do is grab a bunch of
8:22
just like video projects that are actually current and copy them to our
8:27
tiered storage drive here the idea is that i want to bring over more than
8:32
would comfortably fit in the cache and then i want the editors during our test
8:37
to open up first a whole set of completely different projects from each
8:40
other then close those open up a new set
8:44
go back open the other ones and evaluate the performance for themselves
8:49
so we're back and we settled into a steady transfer speed of anywhere from
8:54
200 to 300 plus megabytes a second
8:58
and all of our projects are over on here so
9:02
i guess all that's left for me to do then is go interrupt the editing team
9:08
no no no so i just need to know if you guys all
9:12
hit this and try to edit 8k footage at the same time
9:17
will you notice slow downs because i know some people are
9:21
sensitive to slow downs so here's the idea
9:26
i don't even know what a network share is this is the network share but i don't
9:30
know what that is uh so everyone kind of pick a project and
9:34
open it and start doing video editor things whatever those are work the way
9:37
you would normally work because full is going to be affected by like your system
9:41
as well so the idea
9:47
is that once it arrives it gets cached in the fast tier so i
9:53
just wanted to get a feel for what this is like ed are you opening a project oh
9:58
you're coding wait you're encoding it oh uh exporting probably isn't a very
10:03
stressful test okay what are you doing taren i uh
10:07
i don't know can you try doing some more editory things like pretend to edit this
10:11
i'm moving stuff around i you know keep pretending pretend to do your job
10:17
yes can you go back and play that sequence again
10:22
and see if it's any better how fast would you expect it to be
10:28
like is this normal or it's a little bit faster
10:33
it's not that way okay just hold on a second i haven't i
10:37
haven't given up on saving money yet so the thing i want to find out now is
10:42
if you guys keep doing stuff will it get any better
10:47
so i'm starting to come around to the realization that
10:51
maybe tiered storage solutions
10:54
aren't grab and go because in every environment you'd want
10:59
your tiers to behave in a completely different manner
11:04
like if there was some programmability in storage spaces where i could say okay
11:08
take all the newest files that come in and assume they are important
11:13
until such time as they do not get accessed for
11:17
a very long period or until such time as something new comes in and
11:22
bumps it out then it would be less likely for those
11:26
guys to open up a project and have it chug until
11:30
storage spaces goes oh this is important and can promote it from the slow
11:34
mechanical tier so it's possible that if i had a much much larger fast NVMe tier
11:40
we wouldn't run into this issue because this is really a very heavy load for
11:44
four hard drives and two ssds but the problem is that that would
11:48
require me to basically build that entire server without even knowing that
11:53
it would work so you are dropping i'm dropping 90 of
11:57
my frames can you rate your editing experience out of 10 for me here there's
12:01
like little moments of seven out of ten
12:04
but then overall it's more like a three out of ten sorry you're saying this is
12:09
better than waning a little bit maybe all right everyone just go back to wanik
12:13
unmap the drive throw it all in the garbage
12:22
so that's pretty much it i am
12:26
coming to the the the acceptance stage
12:30
in the in the stages of grief here um
12:35
where i'm realizing that if we want
12:38
more capacity of our high performance network storage
12:42
i'm just going to have to pull out the wallet and make it happen
12:48
thankfully to take some of the sting out of it is our sponsor for this video
12:53
squarespace with squarespace you can make your own beautiful website without
12:58
the hassle they're all in one platform makes it easy to get up and running
13:02
quickly and they've got all kinds of great features from their logo designer
13:06
to their built-in e-commerce to the ability to publish in apple news format
13:10
so that your blog will be available immediately to millions of potential
13:14
readers worldwide and they offer all kinds of support so you can build the
13:18
best website you can build whether it's their seminars or contacting their 24 7
13:24
tech support via live chat and email so don't take my word for it go try out
13:28
squarespace at the link in the video description then when you decide to sign
13:32
up for real go to squarespace.com LTT and you can get 10 off your first
13:37
purchase so thanks for watching guys if you
13:40
disliked this video you can hit that button but if you liked it hit like get
13:43
subscribed or maybe consider checking out where to buy the stuff we featured at the link in the video description
13:48
also down there is our merch store which has cool shirts
13:53
it's winter like this one and our community forum which you should totally
13:57
join