{"video_id":"riSffWEtrF0","title":"These Servers are TOO EXPENSIVE - Hybrid Storage Explored","channel":"Linus Tech Tips","show":"Linus Tech Tips","published_at":"2019-05-06T14:53:29Z","duration_s":840,"segments":[{"start_s":0.16,"end_s":6.08,"text":"so i've got a problem when it comes to building a storage server there are two","speaker":null,"is_sponsor":0},{"start_s":3.919,"end_s":11.92,"text":"main options for your drives you can either go with ssds which are compact","speaker":null,"is_sponsor":0},{"start_s":8.96,"end_s":16.96,"text":"and have amazing performance or you can go with traditional mechanical storage","speaker":null,"is_sponsor":0},{"start_s":14.48,"end_s":22.8,"text":"which has the benefit of lower prices and outstanding capacity with the","speaker":null,"is_sponsor":0},{"start_s":19.6,"end_s":26.24,"text":"drawback of being an order of magnitude","speaker":null,"is_sponsor":0},{"start_s":22.8,"end_s":28.4,"text":"or more slower especially in highly","speaker":null,"is_sponsor":0},{"start_s":26.24,"end_s":32.399,"text":"random workloads like if they were loaded up with a bunch of virtual","speaker":null,"is_sponsor":0},{"start_s":29.84,"end_s":37.84,"text":"machines or in our case if they're pulling network storage duty for a whole","speaker":null,"is_sponsor":0},{"start_s":35.2,"end_s":44.399,"text":"bunch of heavy users who are all trying to edit 8k videos at the same time if","speaker":null,"is_sponsor":0},{"start_s":41.28,"end_s":47.68,"text":"only there were some way to","speaker":null,"is_sponsor":0},{"start_s":44.399,"end_s":50.8,"text":"merge the benefits of both","speaker":null,"is_sponsor":0},{"start_s":47.68,"end_s":52.8,"text":"with the drawbacks of neither","speaker":null,"is_sponsor":0},{"start_s":50.8,"end_s":56.399,"text":"well there is it's called the apple fusion drive","speaker":null,"is_sponsor":0},{"start_s":54.48,"end_s":61.039,"text":"and this episode is brought to you by ifixit the marlin screwdriver set from","speaker":null,"is_sponsor":0},{"start_s":58.8,"end_s":67.45,"text":"ifixit features five specialty precision screwdrivers check it out at ifixit.com","speaker":null,"is_sponsor":0},{"start_s":63.44,"end_s":70.499,"text":"forward slash Linus at the link below","speaker":null,"is_sponsor":0},{"start_s":73.439,"end_s":80.64,"text":"the concept of tiered storage is nothing new it's been commonplace in the data","speaker":null,"is_sponsor":0},{"start_s":78.08,"end_s":84.72,"text":"center for many years and has even made its way down to the consumer level in","speaker":null,"is_sponsor":0},{"start_s":82.64,"end_s":92.0,"text":"the form of Intel's rapid storage technology or rst which uses a small","speaker":null,"is_sponsor":0},{"start_s":88.32,"end_s":94.96,"text":"fast SSD to accelerate a larger slower","speaker":null,"is_sponsor":0},{"start_s":92.0,"end_s":100.479,"text":"mechanical drive and more recently Intel's octane memory technology which","speaker":null,"is_sponsor":0},{"start_s":97.84,"end_s":106.64,"text":"does pretty much the exact same thing but with even faster optane rather than","speaker":null,"is_sponsor":0},{"start_s":103.439,"end_s":109.439,"text":"nand ssds the problem that i'm running","speaker":null,"is_sponsor":0},{"start_s":106.64,"end_s":115.119,"text":"into here is that most of the tiering solutions out there have one of two","speaker":null,"is_sponsor":0},{"start_s":112.24,"end_s":119.6,"text":"problems they are either inordinately complicated to set up requiring","speaker":null,"is_sponsor":0},{"start_s":117.2,"end_s":125.84,"text":"extensive knowledge of not just Linux but even broader storage architecture or","speaker":null,"is_sponsor":0},{"start_s":123.52,"end_s":129.84,"text":"they are expensive because you are paying the folks who have the","speaker":null,"is_sponsor":0},{"start_s":127.52,"end_s":134.0,"text":"aforementioned knowledge to create a turnkey solution for you","speaker":null,"is_sponsor":0},{"start_s":132.239,"end_s":139.599,"text":"there are some exceptions to this however and i would like to explore a","speaker":null,"is_sponsor":0},{"start_s":136.72,"end_s":144.879,"text":"couple of them so the popular freebsd based freenas software from ix systems","speaker":null,"is_sponsor":0},{"start_s":142.72,"end_s":150.48,"text":"uses your system's RAM and its ssds to cache frequently used","speaker":null,"is_sponsor":0},{"start_s":148.64,"end_s":156.8,"text":"data that's stored on your mechanical hard drives but the issue is that zfs's","speaker":null,"is_sponsor":0},{"start_s":153.84,"end_s":162.319,"text":"particular implementation is geared more towards accelerating performance of","speaker":null,"is_sponsor":0},{"start_s":159.44,"end_s":166.319,"text":"large databases and doesn't look like it's going to be great for our workload","speaker":null,"is_sponsor":0},{"start_s":164.08,"end_s":171.92,"text":"because the file sizes that we want cached that we're storing on our hard","speaker":null,"is_sponsor":0},{"start_s":168.4,"end_s":175.519,"text":"drives are so large like these are 8k","speaker":null,"is_sponsor":0},{"start_s":171.92,"end_s":179.28,"text":"red video files and there are practical","speaker":null,"is_sponsor":0},{"start_s":175.519,"end_s":181.28,"text":"limits to how much level 2 arc that","speaker":null,"is_sponsor":0},{"start_s":179.28,"end_s":186.959,"text":"would be the SSD tier i mean there's also practical limits to how much RAM","speaker":null,"is_sponsor":0},{"start_s":183.92,"end_s":188.0,"text":"you can put in a given system","speaker":null,"is_sponsor":0},{"start_s":186.959,"end_s":193.92,"text":"so then the one we're gonna try today is the","speaker":null,"is_sponsor":0},{"start_s":190.56,"end_s":196.879,"text":"built-in tiering mode for storage spaces","speaker":null,"is_sponsor":0},{"start_s":193.92,"end_s":201.92,"text":"in Windows server and what better way to test this out than to set up a tiered","speaker":null,"is_sponsor":0},{"start_s":199.84,"end_s":206.64,"text":"storage space on the network so i'm going to use this test server right here","speaker":null,"is_sponsor":0},{"start_s":204.319,"end_s":211.4,"text":"and then have our team of editors just switch over to it and see what happens","speaker":null,"is_sponsor":0},{"start_s":212.56,"end_s":218.239,"text":"and it's at this point then that we wondered whether or not optane would","speaker":null,"is_sponsor":0},{"start_s":216.319,"end_s":221.44,"text":"make a difference for some of the heavy tasks that we deal with around the","speaker":null,"is_sponsor":0},{"start_s":219.84,"end_s":229.68,"text":"office no no it does not now our regular editing","speaker":null,"is_sponsor":0},{"start_s":224.879,"end_s":231.84,"text":"server uses 31 1.2 terabyte NVMe drives","speaker":null,"is_sponsor":0},{"start_s":229.68,"end_s":235.599,"text":"and it was really expensive for us to build and they've been complaining that","speaker":null,"is_sponsor":0},{"start_s":233.84,"end_s":241.84,"text":"there's not enough space on it because we've only got about 24 terabytes of","speaker":null,"is_sponsor":0},{"start_s":238.72,"end_s":243.439,"text":"usable space after all the lost space to","speaker":null,"is_sponsor":0},{"start_s":241.84,"end_s":248.08,"text":"redundancy on it and they were like oh well you could just upgrade it i went yeah okay","speaker":null,"is_sponsor":0},{"start_s":246.799,"end_s":254.0,"text":"technically i do have another 17 bays that i could","speaker":null,"is_sponsor":0},{"start_s":251.36,"end_s":260.56,"text":"fill up but that's going to cost me like seven grand eight grand even if i buy","speaker":null,"is_sponsor":0},{"start_s":256.639,"end_s":262.8,"text":"used drives on ebay so i was","speaker":null,"is_sponsor":0},{"start_s":260.56,"end_s":268.8,"text":"looking for another way so our test server here the config is","speaker":null,"is_sponsor":0},{"start_s":265.12,"end_s":270.88,"text":"pretty straightforward i've got four 10","speaker":null,"is_sponsor":0},{"start_s":268.8,"end_s":276.72,"text":"terabyte drives then because i want to see exactly how much","speaker":null,"is_sponsor":0},{"start_s":274.56,"end_s":279.919,"text":"cash you need this is what really got me thinking about it","speaker":null,"is_sponsor":0},{"start_s":277.919,"end_s":284.36,"text":"Intel uses just 32 gigs of cash","speaker":null,"is_sponsor":0},{"start_s":282.479,"end_s":289.52,"text":"for large multi-terabyte hard drives and we've","speaker":null,"is_sponsor":0},{"start_s":286.8,"end_s":296.32,"text":"seen real world huge increases in performance so i went okay let's try","speaker":null,"is_sponsor":0},{"start_s":292.32,"end_s":298.96,"text":"this so i took two of their 900 series","speaker":null,"is_sponsor":0},{"start_s":296.32,"end_s":304.0,"text":"optane pci express drives and then i've got a total of 40 terabytes of","speaker":null,"is_sponsor":0},{"start_s":301.04,"end_s":309.44,"text":"mechanical storage and i want to see if for a fraction of the price","speaker":null,"is_sponsor":0},{"start_s":306.4,"end_s":311.52,"text":"we can get good enough performance that","speaker":null,"is_sponsor":0},{"start_s":309.44,"end_s":315.12,"text":"the editors aren't gonna run into a bottleneck","speaker":null,"is_sponsor":0},{"start_s":312.96,"end_s":321.039,"text":"so you can see i've got my four mechanical drives i've got my two ssds","speaker":null,"is_sponsor":0},{"start_s":318.56,"end_s":327.12,"text":"uh allocation is all gonna be automatic so i go ahead and i try to create a pool","speaker":null,"is_sponsor":0},{"start_s":323.919,"end_s":329.84,"text":"okay i've created a pool now i try to","speaker":null,"is_sponsor":0},{"start_s":327.12,"end_s":335.12,"text":"create a virtual disk create storage tiers on this virtual disk it should","speaker":null,"is_sponsor":0},{"start_s":332.56,"end_s":340.16,"text":"automatically detect which ones are your fast drives and which ones are your","speaker":null,"is_sponsor":0},{"start_s":338.24,"end_s":344.32,"text":"capacity drives so we're going to use mirroring because","speaker":null,"is_sponsor":0},{"start_s":342.16,"end_s":348.479,"text":"that way we're protected from a drive failure we're going to use fixed","speaker":null,"is_sponsor":0},{"start_s":346.32,"end_s":353.199,"text":"provisioning and now we get to specify the size of our faster and our standard","speaker":null,"is_sponsor":0},{"start_s":351.52,"end_s":357.84,"text":"tier we're going to go ahead and select the maximum sizes failed to create","speaker":null,"is_sponsor":0},{"start_s":355.12,"end_s":362.8,"text":"virtual disks not supported so the issue is that the gui version of storage","speaker":null,"is_sponsor":0},{"start_s":359.52,"end_s":365.6,"text":"spaces doesn't have some very important","speaker":null,"is_sponsor":0},{"start_s":362.8,"end_s":369.6,"text":"options like selecting how many columns you want","speaker":null,"is_sponsor":0},{"start_s":366.4,"end_s":371.759,"text":"so when you only have two drives and the","speaker":null,"is_sponsor":0},{"start_s":369.6,"end_s":377.36,"text":"default is three columns it spits out this stupid error how did","speaker":null,"is_sponsor":0},{"start_s":374.08,"end_s":378.96,"text":"nobody at microsoft notice this for","speaker":null,"is_sponsor":0},{"start_s":377.36,"end_s":383.44,"text":"years anyway doesn't matter","speaker":null,"is_sponsor":0},{"start_s":380.8,"end_s":386.72,"text":"i called in a lifeline led eater over on the forum as a guru when it comes to","speaker":null,"is_sponsor":0},{"start_s":385.36,"end_s":391.28,"text":"this stuff so i'm just going to pull up his dm over there and follow his","speaker":null,"is_sponsor":0},{"start_s":389.6,"end_s":394.88,"text":"instructions to hopefully do this in powershell so this will be fun you guys","speaker":null,"is_sponsor":0},{"start_s":393.52,"end_s":398.639,"text":"are going to be learning right along with me i haven't done this before the","speaker":null,"is_sponsor":0},{"start_s":396.639,"end_s":402.8,"text":"video so i can still use the gui to create my storage pool apparently","speaker":null,"is_sponsor":0},{"start_s":401.12,"end_s":409.36,"text":"i'm going to go ahead and do that then i need to fire up powershell and","speaker":null,"is_sponsor":0},{"start_s":405.68,"end_s":410.4,"text":"change the default number of columns","speaker":null,"is_sponsor":0},{"start_s":409.36,"end_s":415.6,"text":"okay well i don't know what i just did","speaker":null,"is_sponsor":0},{"start_s":412.72,"end_s":419.199,"text":"but i'm going to change it to mirror","speaker":null,"is_sponsor":0},{"start_s":416.72,"end_s":425.599,"text":"number of columns default one i think i think we did that let's","speaker":null,"is_sponsor":0},{"start_s":422.72,"end_s":429.68,"text":"find out so i think the command i just entered","speaker":null,"is_sponsor":0},{"start_s":428.4,"end_s":433.36,"text":"changed the mirror layout default","speaker":null,"is_sponsor":0},{"start_s":432.479,"end_s":438.96,"text":"to one column it did not","speaker":null,"is_sponsor":0},{"start_s":437.12,"end_s":446.24,"text":"okay uh so i'm using powershell ise now","speaker":null,"is_sponsor":0},{"start_s":442.319,"end_s":446.24,"text":"and this seems to be working better","speaker":null,"is_sponsor":0},{"start_s":446.96,"end_s":452.4,"text":"mirror number of columns default one","speaker":null,"is_sponsor":0},{"start_s":450.88,"end_s":455.68,"text":"come on so mad right now","speaker":null,"is_sponsor":0},{"start_s":455.759,"end_s":463.039,"text":"i just i just want to change a stupid","speaker":null,"is_sponsor":0},{"start_s":459.44,"end_s":466.0,"text":"setting it's definitely called","speaker":null,"is_sponsor":0},{"start_s":463.039,"end_s":470.88,"text":"mirror it's definitely called storage pool one like i create","speaker":null,"is_sponsor":0},{"start_s":468.56,"end_s":473.44,"text":"these commands using their own thing","speaker":null,"is_sponsor":0},{"start_s":472.0,"end_s":480.24,"text":"we'll figure this out at some point the good news is there's a backup plan","speaker":null,"is_sponsor":0},{"start_s":476.639,"end_s":484.03,"text":"i swear i had this working at one point","speaker":null,"is_sponsor":0},{"start_s":480.24,"end_s":485.599,"text":"what is this oh okay oops actually yeah","speaker":null,"is_sponsor":0},{"start_s":485.599,"end_s":491.44,"text":"oops all right","speaker":null,"is_sponsor":0},{"start_s":489.44,"end_s":496.08,"text":"here we go so right now we've only got a 10 gig link i can put a faster card in","speaker":null,"is_sponsor":0},{"start_s":494.0,"end_s":502.24,"text":"here if it ends up being a networking bottleneck but for now what we need to","speaker":null,"is_sponsor":0},{"start_s":498.72,"end_s":504.56,"text":"do is grab a bunch of","speaker":null,"is_sponsor":0},{"start_s":502.24,"end_s":510.4,"text":"just like video projects that are actually current and copy them to our","speaker":null,"is_sponsor":0},{"start_s":507.759,"end_s":514.88,"text":"tiered storage drive here the idea is that i want to bring over more than","speaker":null,"is_sponsor":0},{"start_s":512.719,"end_s":519.44,"text":"would comfortably fit in the cache and then i want the editors during our test","speaker":null,"is_sponsor":0},{"start_s":517.279,"end_s":524.08,"text":"to open up first a whole set of completely different projects from each","speaker":null,"is_sponsor":0},{"start_s":520.959,"end_s":526.959,"text":"other then close those open up a new set","speaker":null,"is_sponsor":0},{"start_s":524.08,"end_s":532.0,"text":"go back open the other ones and evaluate the performance for themselves","speaker":null,"is_sponsor":0},{"start_s":529.6,"end_s":538.08,"text":"so we're back and we settled into a steady transfer speed of anywhere from","speaker":null,"is_sponsor":0},{"start_s":534.56,"end_s":540.56,"text":"200 to 300 plus megabytes a second","speaker":null,"is_sponsor":0},{"start_s":538.08,"end_s":544.64,"text":"and all of our projects are over on here so","speaker":null,"is_sponsor":0},{"start_s":542.32,"end_s":550.72,"text":"i guess all that's left for me to do then is go interrupt the editing team","speaker":null,"is_sponsor":0},{"start_s":548.8,"end_s":555.12,"text":"no no no so i just need to know if you guys all","speaker":null,"is_sponsor":0},{"start_s":552.64,"end_s":559.04,"text":"hit this and try to edit 8k footage at the same time","speaker":null,"is_sponsor":0},{"start_s":557.04,"end_s":562.399,"text":"will you notice slow downs because i know some people are","speaker":null,"is_sponsor":0},{"start_s":561.279,"end_s":568.16,"text":"sensitive to slow downs so here's the idea","speaker":null,"is_sponsor":0},{"start_s":566.72,"end_s":572.08,"text":"i don't even know what a network share is this is the network share but i don't","speaker":null,"is_sponsor":0},{"start_s":570.48,"end_s":576.08,"text":"know what that is uh so everyone kind of pick a project and","speaker":null,"is_sponsor":0},{"start_s":574.16,"end_s":579.68,"text":"open it and start doing video editor things whatever those are work the way","speaker":null,"is_sponsor":0},{"start_s":577.92,"end_s":584.16,"text":"you would normally work because full is going to be affected by like your system","speaker":null,"is_sponsor":0},{"start_s":581.92,"end_s":590.56,"text":"as well so the idea","speaker":null,"is_sponsor":0},{"start_s":587.76,"end_s":596.16,"text":"is that once it arrives it gets cached in the fast tier so i","speaker":null,"is_sponsor":0},{"start_s":593.76,"end_s":601.12,"text":"just wanted to get a feel for what this is like ed are you opening a project oh","speaker":null,"is_sponsor":0},{"start_s":598.399,"end_s":605.68,"text":"you're coding wait you're encoding it oh uh exporting probably isn't a very","speaker":null,"is_sponsor":0},{"start_s":603.68,"end_s":609.68,"text":"stressful test okay what are you doing taren i uh","speaker":null,"is_sponsor":0},{"start_s":607.6,"end_s":613.839,"text":"i don't know can you try doing some more editory things like pretend to edit this","speaker":null,"is_sponsor":0},{"start_s":611.76,"end_s":619.279,"text":"i'm moving stuff around i you know keep pretending pretend to do your job","speaker":null,"is_sponsor":0},{"start_s":617.04,"end_s":624.56,"text":"yes can you go back and play that sequence again","speaker":null,"is_sponsor":0},{"start_s":622.079,"end_s":629.839,"text":"and see if it's any better how fast would you expect it to be","speaker":null,"is_sponsor":0},{"start_s":628.079,"end_s":635.839,"text":"like is this normal or it's a little bit faster","speaker":null,"is_sponsor":0},{"start_s":633.839,"end_s":640.48,"text":"it's not that way okay just hold on a second i haven't i","speaker":null,"is_sponsor":0},{"start_s":637.76,"end_s":645.2,"text":"haven't given up on saving money yet so the thing i want to find out now is","speaker":null,"is_sponsor":0},{"start_s":642.8,"end_s":649.2,"text":"if you guys keep doing stuff will it get any better","speaker":null,"is_sponsor":0},{"start_s":647.6,"end_s":654.72,"text":"so i'm starting to come around to the realization that","speaker":null,"is_sponsor":0},{"start_s":651.2,"end_s":656.88,"text":"maybe tiered storage solutions","speaker":null,"is_sponsor":0},{"start_s":654.72,"end_s":660.48,"text":"aren't grab and go because in every environment you'd want","speaker":null,"is_sponsor":0},{"start_s":659.6,"end_s":666.24,"text":"your tiers to behave in a completely different manner","speaker":null,"is_sponsor":0},{"start_s":664.399,"end_s":671.04,"text":"like if there was some programmability in storage spaces where i could say okay","speaker":null,"is_sponsor":0},{"start_s":668.72,"end_s":676.079,"text":"take all the newest files that come in and assume they are important","speaker":null,"is_sponsor":0},{"start_s":673.76,"end_s":680.56,"text":"until such time as they do not get accessed for","speaker":null,"is_sponsor":0},{"start_s":677.76,"end_s":684.0,"text":"a very long period or until such time as something new comes in and","speaker":null,"is_sponsor":0},{"start_s":682.24,"end_s":688.72,"text":"bumps it out then it would be less likely for those","speaker":null,"is_sponsor":0},{"start_s":686.16,"end_s":692.48,"text":"guys to open up a project and have it chug until","speaker":null,"is_sponsor":0},{"start_s":690.56,"end_s":696.48,"text":"storage spaces goes oh this is important and can promote it from the slow","speaker":null,"is_sponsor":0},{"start_s":694.32,"end_s":702.48,"text":"mechanical tier so it's possible that if i had a much much larger fast NVMe tier","speaker":null,"is_sponsor":0},{"start_s":700.88,"end_s":707.12,"text":"we wouldn't run into this issue because this is really a very heavy load for","speaker":null,"is_sponsor":0},{"start_s":704.24,"end_s":711.2,"text":"four hard drives and two ssds but the problem is that that would","speaker":null,"is_sponsor":0},{"start_s":708.8,"end_s":715.12,"text":"require me to basically build that entire server without even knowing that","speaker":null,"is_sponsor":0},{"start_s":713.68,"end_s":719.519,"text":"it would work so you are dropping i'm dropping 90 of","speaker":null,"is_sponsor":0},{"start_s":717.76,"end_s":724.959,"text":"my frames can you rate your editing experience out of 10 for me here there's","speaker":null,"is_sponsor":0},{"start_s":721.6,"end_s":726.959,"text":"like little moments of seven out of ten","speaker":null,"is_sponsor":0},{"start_s":724.959,"end_s":731.2,"text":"but then overall it's more like a three out of ten sorry you're saying this is","speaker":null,"is_sponsor":0},{"start_s":729.12,"end_s":734.72,"text":"better than waning a little bit maybe all right everyone just go back to wanik","speaker":null,"is_sponsor":0},{"start_s":733.279,"end_s":737.76,"text":"unmap the drive throw it all in the garbage","speaker":null,"is_sponsor":0},{"start_s":742.8,"end_s":750.88,"text":"so that's pretty much it i am","speaker":null,"is_sponsor":0},{"start_s":746.639,"end_s":753.76,"text":"coming to the the the acceptance stage","speaker":null,"is_sponsor":0},{"start_s":750.88,"end_s":758.56,"text":"in the in the stages of grief here um","speaker":null,"is_sponsor":0},{"start_s":755.279,"end_s":761.2,"text":"where i'm realizing that if we want","speaker":null,"is_sponsor":0},{"start_s":758.56,"end_s":765.839,"text":"more capacity of our high performance network storage","speaker":null,"is_sponsor":0},{"start_s":762.88,"end_s":770.72,"text":"i'm just going to have to pull out the wallet and make it happen","speaker":null,"is_sponsor":0},{"start_s":768.88,"end_s":775.76,"text":"thankfully to take some of the sting out of it is our sponsor for this video","speaker":null,"is_sponsor":0},{"start_s":773.36,"end_s":780.56,"text":"squarespace with squarespace you can make your own beautiful website without","speaker":null,"is_sponsor":0},{"start_s":778.8,"end_s":783.839,"text":"the hassle they're all in one platform makes it easy to get up and running","speaker":null,"is_sponsor":0},{"start_s":782.399,"end_s":788.16,"text":"quickly and they've got all kinds of great features from their logo designer","speaker":null,"is_sponsor":0},{"start_s":786.079,"end_s":792.0,"text":"to their built-in e-commerce to the ability to publish in apple news format","speaker":null,"is_sponsor":0},{"start_s":790.32,"end_s":796.8,"text":"so that your blog will be available immediately to millions of potential","speaker":null,"is_sponsor":0},{"start_s":794.32,"end_s":801.279,"text":"readers worldwide and they offer all kinds of support so you can build the","speaker":null,"is_sponsor":0},{"start_s":798.639,"end_s":806.399,"text":"best website you can build whether it's their seminars or contacting their 24 7","speaker":null,"is_sponsor":0},{"start_s":804.32,"end_s":810.079,"text":"tech support via live chat and email so don't take my word for it go try out","speaker":null,"is_sponsor":0},{"start_s":808.639,"end_s":815.04,"text":"squarespace at the link in the video description then when you decide to sign","speaker":null,"is_sponsor":0},{"start_s":812.079,"end_s":819.279,"text":"up for real go to squarespace.com LTT and you can get 10 off your first","speaker":null,"is_sponsor":0},{"start_s":817.44,"end_s":822.399,"text":"purchase so thanks for watching guys if you","speaker":null,"is_sponsor":0},{"start_s":820.639,"end_s":825.36,"text":"disliked this video you can hit that button but if you liked it hit like get","speaker":null,"is_sponsor":0},{"start_s":823.839,"end_s":830.16,"text":"subscribed or maybe consider checking out where to buy the stuff we featured at the link in the video description","speaker":null,"is_sponsor":0},{"start_s":828.56,"end_s":832.959,"text":"also down there is our merch store which has cool shirts","speaker":null,"is_sponsor":0},{"start_s":833.2,"end_s":840.36,"text":"it's winter like this one and our community forum which you should totally","speaker":null,"is_sponsor":0},{"start_s":837.36,"end_s":840.36,"text":"join","speaker":null,"is_sponsor":0}],"full_text":"so i've got a problem when it comes to building a storage server there are two main options for your drives you can either go with ssds which are compact and have amazing performance or you can go with traditional mechanical storage which has the benefit of lower prices and outstanding capacity with the drawback of being an order of magnitude or more slower especially in highly random workloads like if they were loaded up with a bunch of virtual machines or in our case if they're pulling network storage duty for a whole bunch of heavy users who are all trying to edit 8k videos at the same time if only there were some way to merge the benefits of both with the drawbacks of neither well there is it's called the apple fusion drive and this episode is brought to you by ifixit the marlin screwdriver set from ifixit features five specialty precision screwdrivers check it out at ifixit.com forward slash Linus at the link below the concept of tiered storage is nothing new it's been commonplace in the data center for many years and has even made its way down to the consumer level in the form of Intel's rapid storage technology or rst which uses a small fast SSD to accelerate a larger slower mechanical drive and more recently Intel's octane memory technology which does pretty much the exact same thing but with even faster optane rather than nand ssds the problem that i'm running into here is that most of the tiering solutions out there have one of two problems they are either inordinately complicated to set up requiring extensive knowledge of not just Linux but even broader storage architecture or they are expensive because you are paying the folks who have the aforementioned knowledge to create a turnkey solution for you there are some exceptions to this however and i would like to explore a couple of them so the popular freebsd based freenas software from ix systems uses your system's RAM and its ssds to cache frequently used data that's stored on your mechanical hard drives but the issue is that zfs's particular implementation is geared more towards accelerating performance of large databases and doesn't look like it's going to be great for our workload because the file sizes that we want cached that we're storing on our hard drives are so large like these are 8k red video files and there are practical limits to how much level 2 arc that would be the SSD tier i mean there's also practical limits to how much RAM you can put in a given system so then the one we're gonna try today is the built-in tiering mode for storage spaces in Windows server and what better way to test this out than to set up a tiered storage space on the network so i'm going to use this test server right here and then have our team of editors just switch over to it and see what happens and it's at this point then that we wondered whether or not optane would make a difference for some of the heavy tasks that we deal with around the office no no it does not now our regular editing server uses 31 1.2 terabyte NVMe drives and it was really expensive for us to build and they've been complaining that there's not enough space on it because we've only got about 24 terabytes of usable space after all the lost space to redundancy on it and they were like oh well you could just upgrade it i went yeah okay technically i do have another 17 bays that i could fill up but that's going to cost me like seven grand eight grand even if i buy used drives on ebay so i was looking for another way so our test server here the config is pretty straightforward i've got four 10 terabyte drives then because i want to see exactly how much cash you need this is what really got me thinking about it Intel uses just 32 gigs of cash for large multi-terabyte hard drives and we've seen real world huge increases in performance so i went okay let's try this so i took two of their 900 series optane pci express drives and then i've got a total of 40 terabytes of mechanical storage and i want to see if for a fraction of the price we can get good enough performance that the editors aren't gonna run into a bottleneck so you can see i've got my four mechanical drives i've got my two ssds uh allocation is all gonna be automatic so i go ahead and i try to create a pool okay i've created a pool now i try to create a virtual disk create storage tiers on this virtual disk it should automatically detect which ones are your fast drives and which ones are your capacity drives so we're going to use mirroring because that way we're protected from a drive failure we're going to use fixed provisioning and now we get to specify the size of our faster and our standard tier we're going to go ahead and select the maximum sizes failed to create virtual disks not supported so the issue is that the gui version of storage spaces doesn't have some very important options like selecting how many columns you want so when you only have two drives and the default is three columns it spits out this stupid error how did nobody at microsoft notice this for years anyway doesn't matter i called in a lifeline led eater over on the forum as a guru when it comes to this stuff so i'm just going to pull up his dm over there and follow his instructions to hopefully do this in powershell so this will be fun you guys are going to be learning right along with me i haven't done this before the video so i can still use the gui to create my storage pool apparently i'm going to go ahead and do that then i need to fire up powershell and change the default number of columns okay well i don't know what i just did but i'm going to change it to mirror number of columns default one i think i think we did that let's find out so i think the command i just entered changed the mirror layout default to one column it did not okay uh so i'm using powershell ise now and this seems to be working better mirror number of columns default one come on so mad right now i just i just want to change a stupid setting it's definitely called mirror it's definitely called storage pool one like i create these commands using their own thing we'll figure this out at some point the good news is there's a backup plan i swear i had this working at one point what is this oh okay oops actually yeah oops all right here we go so right now we've only got a 10 gig link i can put a faster card in here if it ends up being a networking bottleneck but for now what we need to do is grab a bunch of just like video projects that are actually current and copy them to our tiered storage drive here the idea is that i want to bring over more than would comfortably fit in the cache and then i want the editors during our test to open up first a whole set of completely different projects from each other then close those open up a new set go back open the other ones and evaluate the performance for themselves so we're back and we settled into a steady transfer speed of anywhere from 200 to 300 plus megabytes a second and all of our projects are over on here so i guess all that's left for me to do then is go interrupt the editing team no no no so i just need to know if you guys all hit this and try to edit 8k footage at the same time will you notice slow downs because i know some people are sensitive to slow downs so here's the idea i don't even know what a network share is this is the network share but i don't know what that is uh so everyone kind of pick a project and open it and start doing video editor things whatever those are work the way you would normally work because full is going to be affected by like your system as well so the idea is that once it arrives it gets cached in the fast tier so i just wanted to get a feel for what this is like ed are you opening a project oh you're coding wait you're encoding it oh uh exporting probably isn't a very stressful test okay what are you doing taren i uh i don't know can you try doing some more editory things like pretend to edit this i'm moving stuff around i you know keep pretending pretend to do your job yes can you go back and play that sequence again and see if it's any better how fast would you expect it to be like is this normal or it's a little bit faster it's not that way okay just hold on a second i haven't i haven't given up on saving money yet so the thing i want to find out now is if you guys keep doing stuff will it get any better so i'm starting to come around to the realization that maybe tiered storage solutions aren't grab and go because in every environment you'd want your tiers to behave in a completely different manner like if there was some programmability in storage spaces where i could say okay take all the newest files that come in and assume they are important until such time as they do not get accessed for a very long period or until such time as something new comes in and bumps it out then it would be less likely for those guys to open up a project and have it chug until storage spaces goes oh this is important and can promote it from the slow mechanical tier so it's possible that if i had a much much larger fast NVMe tier we wouldn't run into this issue because this is really a very heavy load for four hard drives and two ssds but the problem is that that would require me to basically build that entire server without even knowing that it would work so you are dropping i'm dropping 90 of my frames can you rate your editing experience out of 10 for me here there's like little moments of seven out of ten but then overall it's more like a three out of ten sorry you're saying this is better than waning a little bit maybe all right everyone just go back to wanik unmap the drive throw it all in the garbage so that's pretty much it i am coming to the the the acceptance stage in the in the stages of grief here um where i'm realizing that if we want more capacity of our high performance network storage i'm just going to have to pull out the wallet and make it happen thankfully to take some of the sting out of it is our sponsor for this video squarespace with squarespace you can make your own beautiful website without the hassle they're all in one platform makes it easy to get up and running quickly and they've got all kinds of great features from their logo designer to their built-in e-commerce to the ability to publish in apple news format so that your blog will be available immediately to millions of potential readers worldwide and they offer all kinds of support so you can build the best website you can build whether it's their seminars or contacting their 24 7 tech support via live chat and email so don't take my word for it go try out squarespace at the link in the video description then when you decide to sign up for real go to squarespace.com LTT and you can get 10 off your first purchase so thanks for watching guys if you disliked this video you can hit that button but if you liked it hit like get subscribed or maybe consider checking out where to buy the stuff we featured at the link in the video description also down there is our merch store which has cool shirts it's winter like this one and our community forum which you should totally join"}