{"video_id":"6oiIMUrbilw","title":"Fixing our server's biggest flaw","channel":"Linus Tech Tips","show":"Linus Tech Tips","published_at":"2024-05-04T14:53:29Z","duration_s":845,"segments":[{"start_s":0.0,"end_s":5.759,"text":"when we did our last update on the server room I said that our days of","speaker":null,"is_sponsor":0},{"start_s":3.84,"end_s":10.62,"text":"janky Solutions were over it was all proper from here on out but if you","speaker":null,"is_sponsor":0},{"start_s":7.44,"end_s":13.08,"text":"freeze frame you can see that I had my","speaker":null,"is_sponsor":0},{"start_s":10.62,"end_s":19.38,"text":"fingers crossed our current live video editing server new new wanak is filled","speaker":null,"is_sponsor":0},{"start_s":15.66,"end_s":21.9,"text":"up with 32 4 terabyte NVMe ssds split","speaker":null,"is_sponsor":0},{"start_s":19.38,"end_s":25.32,"text":"across four of these liquid honey badger carrier cards all we really cared about","speaker":null,"is_sponsor":0},{"start_s":23.699,"end_s":29.699,"text":"at the time was building something that was fast enough to support our 10 plus","speaker":null,"is_sponsor":0},{"start_s":27.359,"end_s":33.78,"text":"editors all hitting it at the same time and it did in spite of all the hate it's","speaker":null,"is_sponsor":0},{"start_s":32.04,"end_s":36.96,"text":"been running for nearly two years now without missing a beat and that's","speaker":null,"is_sponsor":0},{"start_s":35.52,"end_s":40.92,"text":"despite the fact that our team has nearly doubled in size since then did I","speaker":null,"is_sponsor":0},{"start_s":39.18,"end_s":45.12,"text":"say without missing a beat it missed one beat a couple of months ago conveniently","speaker":null,"is_sponsor":0},{"start_s":43.2,"end_s":51.0,"text":"on the same day the LTT Channel Got Hijacked one of the drives on one of the","speaker":null,"is_sponsor":0},{"start_s":48.96,"end_s":54.539,"text":"carrier cards died dropping it from the array like any other sane person we set","speaker":null,"is_sponsor":0},{"start_s":52.98,"end_s":59.28,"text":"up the server in a raid configuration that protects us from losing drives so","speaker":null,"is_sponsor":0},{"start_s":57.0,"end_s":63.18,"text":"no biggie all we should have had to do is swap out the dead drive with a new","speaker":null,"is_sponsor":0},{"start_s":61.199,"end_s":66.54,"text":"one rebuild the array with the parity data and we should be back to normal","speaker":null,"is_sponsor":0},{"start_s":64.68,"end_s":71.76,"text":"except there was a massive oversight in our setup each SSD is in a carrier card","speaker":null,"is_sponsor":0},{"start_s":69.6,"end_s":75.84,"text":"which is mounted into a PCIe tray which which is inside the server how do we","speaker":null,"is_sponsor":0},{"start_s":73.92,"end_s":80.4,"text":"replace a drive in this thing without taking the whole thing apart well we","speaker":null,"is_sponsor":0},{"start_s":78.299,"end_s":84.6,"text":"never documented which SSD is in which carrier card so even if I tore the","speaker":null,"is_sponsor":0},{"start_s":82.38,"end_s":88.92,"text":"entire server apart I'd have a hell of a time finding the Dead one so the answer","speaker":null,"is_sponsor":0},{"start_s":86.52,"end_s":93.659,"text":"is you don't you just leave it in there and replace it with a mismatch u.2 SSD","speaker":null,"is_sponsor":0},{"start_s":91.38,"end_s":98.04,"text":"in one of the empty front Bays that is an awful solution but at least it's a","speaker":null,"is_sponsor":0},{"start_s":96.36,"end_s":102.06,"text":"good transition into how we're going to fix this now if we weren't crazy we'd","speaker":null,"is_sponsor":0},{"start_s":100.38,"end_s":106.74,"text":"probably reconfigure our current Dell server to have more u.2 slots and then","speaker":null,"is_sponsor":0},{"start_s":104.46,"end_s":112.56,"text":"fill those with Enterprise drives but Enterprise drives are really expensive","speaker":null,"is_sponsor":0},{"start_s":108.96,"end_s":115.32,"text":"and we happen to have 22 of sabrin's","speaker":null,"is_sponsor":0},{"start_s":112.56,"end_s":119.04,"text":"absolutely baller eight terabyte Gen 4 plus drives left over from a previous","speaker":null,"is_sponsor":0},{"start_s":116.88,"end_s":123.42,"text":"video that were pretty much begging to be used for something like this but if","speaker":null,"is_sponsor":0},{"start_s":120.899,"end_s":127.32,"text":"we just swapped these ssds in sure we'd be faster and we'd have double the","speaker":null,"is_sponsor":0},{"start_s":125.1,"end_s":133.14,"text":"capacity but we would have the same maintenance problem enter these These","speaker":null,"is_sponsor":0},{"start_s":130.259,"end_s":137.52,"text":"are m.2 to u.2 adapters from Star Tech and icy Dock and what they allow us to","speaker":null,"is_sponsor":0},{"start_s":135.3,"end_s":143.04,"text":"do is take practically any consumer grade m.2 drive and install it into a","speaker":null,"is_sponsor":0},{"start_s":141.12,"end_s":146.94,"text":"hot swappable Bay in the front of a server just like this it's the best of","speaker":null,"is_sponsor":0},{"start_s":145.319,"end_s":151.2,"text":"every solution because we can use cheap consumer ssds they're easily serviceable","speaker":null,"is_sponsor":0},{"start_s":149.58,"end_s":156.239,"text":"without even powering down the machine and we get the full performance of each","speaker":null,"is_sponsor":0},{"start_s":154.08,"end_s":161.94,"text":"Drive instead of only half like we do with the carrier cards oh and you guys","speaker":null,"is_sponsor":0},{"start_s":158.76,"end_s":164.459,"text":"get this segue to our sponsor AMD get","speaker":null,"is_sponsor":1},{"start_s":161.94,"end_s":169.14,"text":"great deals during the game on AMD event while supplies last on AMD ryzen","speaker":null,"is_sponsor":1},{"start_s":166.739,"end_s":173.4,"text":"processors and AMD Radeon graphics cards including amazing game bundles for a","speaker":null,"is_sponsor":1},{"start_s":171.0,"end_s":177.66,"text":"limited time from now until July 1st Canada's birthday check them out at the","speaker":null,"is_sponsor":1},{"start_s":175.68,"end_s":182.16,"text":"link in the description even if we had all the parts on hand to turn our","speaker":null,"is_sponsor":1},{"start_s":179.22,"end_s":187.379,"text":"current production Dell server into new new wanek wanak 4. it wouldn't really be","speaker":null,"is_sponsor":0},{"start_s":185.459,"end_s":191.64,"text":"a good idea because we'd have to do all of this outside of ours and even fully","speaker":null,"is_sponsor":0},{"start_s":189.9,"end_s":196.98,"text":"populated we would still be eight days short of fitting all 32 of the m.2","speaker":null,"is_sponsor":0},{"start_s":194.459,"end_s":201.379,"text":"drives so instead we're going to use a new server specifically we're going to","speaker":null,"is_sponsor":0},{"start_s":198.9,"end_s":205.14,"text":"be using the Gigabyte r282-z9g and you might remember this","speaker":null,"is_sponsor":0},{"start_s":203.459,"end_s":210.12,"text":"from a previous video that we did on g-raid that weird","speaker":null,"is_sponsor":0},{"start_s":207.019,"end_s":212.879,"text":"GPU-based NVMe raid that was wildly fast","speaker":null,"is_sponsor":0},{"start_s":210.12,"end_s":216.48,"text":"but we ultimately didn't use because it doesn't verify the data at all so if you","speaker":null,"is_sponsor":0},{"start_s":214.8,"end_s":220.86,"text":"like went in and like hex edited a file and then turned it back on it wouldn't","speaker":null,"is_sponsor":0},{"start_s":218.879,"end_s":224.659,"text":"catch that that's okay this machine is still perfectly good for software raid","speaker":null,"is_sponsor":0},{"start_s":222.36,"end_s":230.04,"text":"with plenty of AMD epic PCIe Lanes so what we're going","speaker":null,"is_sponsor":0},{"start_s":227.94,"end_s":236.099,"text":"to do is build it up and then just do a quick IP swap from new new wannick to","speaker":null,"is_sponsor":0},{"start_s":232.799,"end_s":238.08,"text":"new new new wanak no new mnemonic to new","speaker":null,"is_sponsor":0},{"start_s":236.099,"end_s":241.379,"text":"new mnemonic this has been in mothballs for a while though so it's time to talk","speaker":null,"is_sponsor":0},{"start_s":239.4,"end_s":247.019,"text":"about the config for CPUs we've gone with AMD epic 75 f3s these are third gen","speaker":null,"is_sponsor":0},{"start_s":244.98,"end_s":252.18,"text":"epics so that's the Milan family of processors 32 cores each the f is for","speaker":null,"is_sponsor":0},{"start_s":250.08,"end_s":256.26,"text":"high frequency these will boost up to 4 gigahertz meaning that they are perfect","speaker":null,"is_sponsor":0},{"start_s":253.86,"end_s":261.239,"text":"for high-speed storage applications like this the bottleneck is going to be our","speaker":null,"is_sponsor":0},{"start_s":258.419,"end_s":265.199,"text":"network interface and our application is that a fast yeah long before it will be","speaker":null,"is_sponsor":0},{"start_s":263.34,"end_s":269.58,"text":"these CPUs and especially long before it'll be those drives we're also missing","speaker":null,"is_sponsor":0},{"start_s":267.66,"end_s":273.08,"text":"a bunch of other essential gear here we have no network cards ah I got network","speaker":null,"is_sponsor":0},{"start_s":272.1,"end_s":279.0,"text":"cards no RAM Jake catch there's no coolers oh","speaker":null,"is_sponsor":0},{"start_s":276.72,"end_s":283.52,"text":"my God what are you doing 100 these are fancy ah those are connect X4 those are","speaker":null,"is_sponsor":0},{"start_s":281.22,"end_s":283.52,"text":"old now","speaker":null,"is_sponsor":0},{"start_s":286.759,"end_s":293.82,"text":"the good news is that these are actually really really inexpensive eight gig","speaker":null,"is_sponsor":0},{"start_s":291.84,"end_s":297.96,"text":"dimms that we have too many of to know what to do with and the reason we're","speaker":null,"is_sponsor":0},{"start_s":295.32,"end_s":303.419,"text":"using this is that while normally in a ZFS array you want a ton of RAM to use","speaker":null,"is_sponsor":0},{"start_s":300.479,"end_s":308.94,"text":"as an arc cash in a high speed NVMe array your drives are so fast that","speaker":null,"is_sponsor":0},{"start_s":306.78,"end_s":312.54,"text":"you're actually adding more overhead by using that cash than you are benefiting","speaker":null,"is_sponsor":0},{"start_s":311.1,"end_s":316.979,"text":"from it that's not to say that there's no need for RAM you could also use it as","speaker":null,"is_sponsor":0},{"start_s":314.639,"end_s":321.84,"text":"a metadata cache which will help accelerate NVMe storage like this it","speaker":null,"is_sponsor":0},{"start_s":319.44,"end_s":326.4,"text":"just doesn't need a ton of space I suspect it'll probably use between 5 and","speaker":null,"is_sponsor":0},{"start_s":323.94,"end_s":329.759,"text":"10 gigs so we're just putting the absolute smallest dims we have to","speaker":null,"is_sponsor":0},{"start_s":328.259,"end_s":333.9,"text":"populate all the memory channels so we get all the speed just one dim per","speaker":null,"is_sponsor":0},{"start_s":331.38,"end_s":336.78,"text":"Channel there's 16 slots per CPU but that would be two dims per Channel we're","speaker":null,"is_sponsor":0},{"start_s":335.46,"end_s":339.56,"text":"just gonna do one well now they're sucking response","speaker":null,"is_sponsor":0},{"start_s":343.82,"end_s":350.1,"text":"I love how many adapters This Server","speaker":null,"is_sponsor":0},{"start_s":347.46,"end_s":355.86,"text":"comes with by default in order to get full PCIe by four operation to all of","speaker":null,"is_sponsor":0},{"start_s":353.759,"end_s":360.9,"text":"these front Bays here they basically take every slot in here every mezzanine","speaker":null,"is_sponsor":0},{"start_s":358.919,"end_s":364.56,"text":"card spot and then convert them to these Slimline ports and run them up to the","speaker":null,"is_sponsor":0},{"start_s":362.94,"end_s":369.539,"text":"front and that's in spite of the fact that with a two socket configuration you","speaker":null,"is_sponsor":0},{"start_s":366.36,"end_s":370.63,"text":"get a total of 160 usable PCIe Lanes","speaker":null,"is_sponsor":0},{"start_s":369.539,"end_s":373.73,"text":"okay","speaker":null,"is_sponsor":0},{"start_s":374.759,"end_s":380.94,"text":"please stop I wouldn't usually bother doing this on like a little consumer CPU","speaker":null,"is_sponsor":0},{"start_s":379.259,"end_s":384.9,"text":"but these things are so huge because there's multiple dies under that heat","speaker":null,"is_sponsor":0},{"start_s":382.8,"end_s":387.84,"text":"spreader it's not like traditional CPUs where if you put some thermal compound","speaker":null,"is_sponsor":0},{"start_s":386.16,"end_s":391.979,"text":"in the middle realistically you've got the hot spots covered there are lots of","speaker":null,"is_sponsor":0},{"start_s":390.06,"end_s":396.419,"text":"hot spots down there and they're very widely spread have you seen these yeah","speaker":null,"is_sponsor":0},{"start_s":394.44,"end_s":400.5,"text":"they're so yeah the Epic ones are green I know they never gave us one until we","speaker":null,"is_sponsor":0},{"start_s":398.039,"end_s":403.5,"text":"went to Texas for Genoa oh yeah lab use only that's awesome I know oh yeah those","speaker":null,"is_sponsor":0},{"start_s":402.3,"end_s":408.72,"text":"are special ones too those are like actually from the lab wait did you steal","speaker":null,"is_sponsor":0},{"start_s":406.139,"end_s":414.78,"text":"this no no I mean sort of oh I should probably start doing this hey oh","speaker":null,"is_sponsor":0},{"start_s":412.16,"end_s":418.62,"text":"oh why have I just been standing here this whole time oh right I was making","speaker":null,"is_sponsor":0},{"start_s":416.16,"end_s":423.72,"text":"fun of him there is one drawback to using these adapters every one of our","speaker":null,"is_sponsor":0},{"start_s":421.139,"end_s":427.8,"text":"m.2 drives needs to be installed on a sled now you probably noticed we've got","speaker":null,"is_sponsor":0},{"start_s":425.639,"end_s":431.52,"text":"two different options for our sleds this one is from Star Tech and honestly","speaker":null,"is_sponsor":0},{"start_s":429.539,"end_s":438.18,"text":"speaking is probably a little faster to use but doesn't have any additional","speaker":null,"is_sponsor":0},{"start_s":433.919,"end_s":441.18,"text":"cooling for the m.2 drive this one is","speaker":null,"is_sponsor":0},{"start_s":438.18,"end_s":443.58,"text":"from Icy Dock and is a bit more of a","speaker":null,"is_sponsor":0},{"start_s":441.18,"end_s":448.08,"text":"bear to deal with but comes with a large heatsink that will","speaker":null,"is_sponsor":0},{"start_s":445.8,"end_s":451.74,"text":"hopefully help with cooling I tested it it does make a difference if you were to","speaker":null,"is_sponsor":0},{"start_s":450.3,"end_s":454.68,"text":"do this in your personal computer for whatever reason or like a home lab where","speaker":null,"is_sponsor":0},{"start_s":453.3,"end_s":457.979,"text":"there's not a lot of airflow you should probably buy these ones for us this is","speaker":null,"is_sponsor":0},{"start_s":456.9,"end_s":462.72,"text":"going to be right in the front of a server with tens of thousands of RPM of","speaker":null,"is_sponsor":0},{"start_s":460.62,"end_s":467.58,"text":"fan I want to do this one this is way better oh we have to peel like 20","speaker":null,"is_sponsor":0},{"start_s":464.819,"end_s":472.8,"text":"thermal pads but it's toolless Nick he's trying to not promote the screwdriver","speaker":null,"is_sponsor":0},{"start_s":470.06,"end_s":476.52,"text":"I have secret screwdriver things to show you oh secret screwdriver things that","speaker":null,"is_sponsor":1},{"start_s":474.78,"end_s":479.819,"text":"can't go on camera yeah they're in my pockets take your hands out of my pocket","speaker":null,"is_sponsor":1},{"start_s":478.139,"end_s":485.22,"text":"sir I don't appreciate this here you can show that oh sick what the hell is that","speaker":null,"is_sponsor":1},{"start_s":482.52,"end_s":490.38,"text":"don't worry about it oh yeah hey make sure we don't open too many","speaker":null,"is_sponsor":1},{"start_s":488.34,"end_s":493.979,"text":"yeah we can open them it's fine well yeah but like our","speaker":null,"is_sponsor":0},{"start_s":492.419,"end_s":499.639,"text":"um we're gonna run out of drives here at some point oh they're actually we're gonna run out of slots let's make sure","speaker":null,"is_sponsor":0},{"start_s":496.5,"end_s":499.639,"text":"we don't open too many of them","speaker":null,"is_sponsor":0},{"start_s":500.039,"end_s":506.58,"text":"and yeah these are not great sleds nope","speaker":null,"is_sponsor":0},{"start_s":503.759,"end_s":510.18,"text":"I am so glad the industry has moved on from using screws for these sleds though","speaker":null,"is_sponsor":0},{"start_s":508.62,"end_s":513.659,"text":"let's put it in there and it goes a gloop and then if you want to take it","speaker":null,"is_sponsor":0},{"start_s":511.68,"end_s":517.44,"text":"out you just go you have to make the noise when you do","speaker":null,"is_sponsor":0},{"start_s":515.099,"end_s":523.62,"text":"it otherwise it doesn't work see look uh it's not working clue ah see Magic","speaker":null,"is_sponsor":0},{"start_s":521.399,"end_s":527.16,"text":"these are freaking awesome you know what's even better I was looking at this","speaker":null,"is_sponsor":0},{"start_s":525.36,"end_s":531.42,"text":"earlier today and I was like wow I wonder if they make ones that go into","speaker":null,"is_sponsor":0},{"start_s":528.42,"end_s":533.82,"text":"the ruler stick ones like the e1s drives","speaker":null,"is_sponsor":0},{"start_s":531.42,"end_s":538.14,"text":"oh shut up they do the other thing I realized on these Star Tech ones is you","speaker":null,"is_sponsor":0},{"start_s":536.04,"end_s":541.98,"text":"can put a thermal pad between the m.2 and this PCB which obviously has a bunch","speaker":null,"is_sponsor":0},{"start_s":540.18,"end_s":546.24,"text":"of copper and it lowers the temps like five degrees but then the icy dock ones","speaker":null,"is_sponsor":0},{"start_s":544.26,"end_s":551.1,"text":"come down another three to five degrees after that yeah so the heatsink is","speaker":null,"is_sponsor":0},{"start_s":548.76,"end_s":556.2,"text":"definitely more better to be fair that was with the fans not really spinning","speaker":null,"is_sponsor":0},{"start_s":553.92,"end_s":560.16,"text":"that much in the server I suspect if you were to put a really big load on the","speaker":null,"is_sponsor":0},{"start_s":558.839,"end_s":563.7,"text":"server it would make less of a difference that's fair yeah we can put","speaker":null,"is_sponsor":0},{"start_s":562.08,"end_s":568.68,"text":"the network card in now it's a connect x400 gig card which is a bit old now you","speaker":null,"is_sponsor":0},{"start_s":566.399,"end_s":572.58,"text":"can get connect X6 which has a little bit more acceleration you can get it in","speaker":null,"is_sponsor":0},{"start_s":570.72,"end_s":575.64,"text":"200 gig capacity I don't have any more connect X6 cards","speaker":null,"is_sponsor":0},{"start_s":574.5,"end_s":582.899,"text":"so that's what's going in there for now like to be clear the the most our Network throughput on wanna ever gets is","speaker":null,"is_sponsor":0},{"start_s":579.66,"end_s":584.519,"text":"like 30 gig maybe I mean that's kind of","speaker":null,"is_sponsor":0},{"start_s":582.899,"end_s":587.82,"text":"a lie it's a fair bit but three gigabytes a second it's not bad what is","speaker":null,"is_sponsor":0},{"start_s":586.019,"end_s":591.42,"text":"wrong with you what do you mean I don't think you dropped it so much and I just","speaker":null,"is_sponsor":0},{"start_s":589.62,"end_s":595.26,"text":"no did well don't touch it though I wanted to can I just show them what's","speaker":null,"is_sponsor":0},{"start_s":592.74,"end_s":599.959,"text":"going to happen no why it's a framework you can fix it I don't want no walks in","speaker":null,"is_sponsor":0},{"start_s":597.54,"end_s":603.779,"text":"oh that's perfectly fine","speaker":null,"is_sponsor":0},{"start_s":602.16,"end_s":611.04,"text":"we didn't even really like show off the server what kind of power supplies are in here 1600 Watts holy Christ here we","speaker":null,"is_sponsor":0},{"start_s":609.12,"end_s":615.779,"text":"go you're just taking in the noise","speaker":null,"is_sponsor":0},{"start_s":613.68,"end_s":618.54,"text":"this is double useful I don't have to hear the server and I don't have to","speaker":null,"is_sponsor":0},{"start_s":617.1,"end_s":624.86,"text":"listen to him look at all the lights I don't see any","speaker":null,"is_sponsor":0},{"start_s":621.0,"end_s":624.86,"text":"activity on them yet can you not","speaker":null,"is_sponsor":0},{"start_s":625.8,"end_s":631.56,"text":"what are these it's like a whole sled placeholders","speaker":null,"is_sponsor":0},{"start_s":630.48,"end_s":637.019,"text":"everyone this is manufactured garbage see okay so","speaker":null,"is_sponsor":0},{"start_s":634.08,"end_s":642.24,"text":"if we do four five wide raid Z ones we get 160. oh that's so much wasted","speaker":null,"is_sponsor":0},{"start_s":639.12,"end_s":644.519,"text":"capacity it's like 40 terabytes of Boise","speaker":null,"is_sponsor":0},{"start_s":642.24,"end_s":650.279,"text":"just do 10. 10 wide be a man I want the toxically","speaker":null,"is_sponsor":0},{"start_s":648.72,"end_s":654.66,"text":"masculine array I don't even have a Bugatti don't even talk","speaker":null,"is_sponsor":0},{"start_s":652.26,"end_s":658.44,"text":"to him packing the name frame DFS okay so we're going to turn the arc cache to","speaker":null,"is_sponsor":0},{"start_s":656.7,"end_s":662.88,"text":"be metadata only it's running kind of warm that's really hot though it's like","speaker":null,"is_sponsor":0},{"start_s":660.899,"end_s":668.399,"text":"wow oh this is gonna be a long command I'll provide moral support um yeah go","speaker":null,"is_sponsor":0},{"start_s":664.86,"end_s":669.899,"text":"Jake yeah Jake type it type it yeah come","speaker":null,"is_sponsor":0},{"start_s":668.399,"end_s":678.12,"text":"on Jake think about what oh this is awesome Jake you're doing so great that's still 15. 16 gig of 12 12 11.","speaker":null,"is_sponsor":0},{"start_s":675.72,"end_s":682.519,"text":"it's going down that's two that's nine that's eight is it possible we should","speaker":null,"is_sponsor":0},{"start_s":679.86,"end_s":688.38,"text":"have done with concealer ssds what is happening","speaker":null,"is_sponsor":0},{"start_s":685.279,"end_s":690.06,"text":"18. what is happening right now you know","speaker":null,"is_sponsor":0},{"start_s":688.38,"end_s":696.56,"text":"parody calculations are a hell of a drug especially when your thing is 10 wide","speaker":null,"is_sponsor":0},{"start_s":692.88,"end_s":700.04,"text":"wow that is sucking back some CPU there","speaker":null,"is_sponsor":0},{"start_s":696.56,"end_s":703.92,"text":"67 usage of","speaker":null,"is_sponsor":0},{"start_s":700.04,"end_s":706.38,"text":"64. high frequency epic course 18","speaker":null,"is_sponsor":0},{"start_s":703.92,"end_s":710.1,"text":"gigabytes a second is very fast this was 20 that's 20 per second man well this is","speaker":null,"is_sponsor":0},{"start_s":708.36,"end_s":714.6,"text":"awesome and this is only a q depth of four I mean I guess for a sequential","speaker":null,"is_sponsor":0},{"start_s":712.26,"end_s":718.68,"text":"load this is not the most accurate so here let's try two threads per drive it","speaker":null,"is_sponsor":0},{"start_s":716.94,"end_s":722.76,"text":"might go a little faster it's going like 24 and then 18 and then seems like it","speaker":null,"is_sponsor":0},{"start_s":720.839,"end_s":727.019,"text":"needs a second to settle into how the CPUs want to handle this it's really not","speaker":null,"is_sponsor":0},{"start_s":724.98,"end_s":732.18,"text":"that loud I know that sounds like a crazy thing to say while it's gone but","speaker":null,"is_sponsor":0},{"start_s":729.72,"end_s":736.26,"text":"he's not wrong we're probably kind of getting to the limit of ZFS here I'm","speaker":null,"is_sponsor":0},{"start_s":734.519,"end_s":740.82,"text":"impressed though 20 gigs a second is like very respectable for software raid","speaker":null,"is_sponsor":0},{"start_s":738.24,"end_s":744.18,"text":"like our CPU is not at 100 is that like 70. so there's still some Headroom for","speaker":null,"is_sponsor":0},{"start_s":742.5,"end_s":748.8,"text":"other things to run and these are not even Enterprise SSD these I know right","speaker":null,"is_sponsor":0},{"start_s":746.579,"end_s":752.339,"text":"I'm trying to think of editors yeah like what kind of workload could we possibly","speaker":null,"is_sponsor":0},{"start_s":750.779,"end_s":758.1,"text":"be hitting it with we would need to overwhelm that cash yeah you'd have to","speaker":null,"is_sponsor":0},{"start_s":754.92,"end_s":759.899,"text":"be copying from every station Red mags","speaker":null,"is_sponsor":0},{"start_s":758.1,"end_s":763.38,"text":"at their full transfer because remember they're not even that fast and that's","speaker":null,"is_sponsor":0},{"start_s":761.64,"end_s":767.22,"text":"assuming Samba could keep up which it can't although what we're going to use","speaker":null,"is_sponsor":0},{"start_s":764.94,"end_s":770.639,"text":"for high availability wantic weka FS they built their own Samba","speaker":null,"is_sponsor":0},{"start_s":768.6,"end_s":775.2,"text":"implementation that has SMB direct so the CPU usage from writing files to this","speaker":null,"is_sponsor":0},{"start_s":773.459,"end_s":780.0,"text":"file system would be like almost nothing when's that coming the server's here you","speaker":null,"is_sponsor":0},{"start_s":777.6,"end_s":785.339,"text":"know what else is here this message from our sponsor racing are you tired of slow","speaker":null,"is_sponsor":1},{"start_s":783.0,"end_s":789.899,"text":"and inefficient file transfers well have you heard about racing with exceptional","speaker":null,"is_sponsor":1},{"start_s":787.44,"end_s":794.16,"text":"speed and efficiency racing can transfer large files with ease plus with","speaker":null,"is_sponsor":1},{"start_s":792.3,"end_s":797.76,"text":"unlimited transfer volume and file size you'll never have to worry about","speaker":null,"is_sponsor":1},{"start_s":795.48,"end_s":802.74,"text":"limitations again but that's not all racing also offers Rich features like","speaker":null,"is_sponsor":1},{"start_s":800.279,"end_s":806.639,"text":"peer-to-peer transfer auto file sync and real-time data backup and with the","speaker":null,"is_sponsor":1},{"start_s":804.66,"end_s":810.779,"text":"ability to sync files of any size you can rest easy knowing your data is","speaker":null,"is_sponsor":1},{"start_s":808.86,"end_s":814.5,"text":"always safe and secure tucked into its bed like a cute little data that it is","speaker":null,"is_sponsor":1},{"start_s":812.639,"end_s":818.279,"text":"need to collaborate with your team racing has got you covered with group","speaker":null,"is_sponsor":1},{"start_s":816.42,"end_s":822.12,"text":"folders and real-time audit logs of users and transfers and with","speaker":null,"is_sponsor":1},{"start_s":820.019,"end_s":826.62,"text":"cross-platform compatibility you can seamlessly transfer files between","speaker":null,"is_sponsor":1},{"start_s":823.5,"end_s":828.72,"text":"Windows Mac Linux Android and iOS","speaker":null,"is_sponsor":1},{"start_s":826.62,"end_s":833.04,"text":"that almost feels wrong so what are you waiting for upgrade your file transfer","speaker":null,"is_sponsor":1},{"start_s":830.82,"end_s":837.24,"text":"game with racing today at the link below if you guys enjoyed this video you","speaker":null,"is_sponsor":1},{"start_s":835.5,"end_s":841.26,"text":"should probably get some more details on what's changed in our infrastructure in","speaker":null,"is_sponsor":0},{"start_s":838.92,"end_s":846.62,"text":"our last server room tour it's done it made it five minutes staying above 10","speaker":null,"is_sponsor":0},{"start_s":843.36,"end_s":846.62,"text":"gigs a second oh sick","speaker":null,"is_sponsor":0}],"full_text":"when we did our last update on the server room I said that our days of janky Solutions were over it was all proper from here on out but if you freeze frame you can see that I had my fingers crossed our current live video editing server new new wanak is filled up with 32 4 terabyte NVMe ssds split across four of these liquid honey badger carrier cards all we really cared about at the time was building something that was fast enough to support our 10 plus editors all hitting it at the same time and it did in spite of all the hate it's been running for nearly two years now without missing a beat and that's despite the fact that our team has nearly doubled in size since then did I say without missing a beat it missed one beat a couple of months ago conveniently on the same day the LTT Channel Got Hijacked one of the drives on one of the carrier cards died dropping it from the array like any other sane person we set up the server in a raid configuration that protects us from losing drives so no biggie all we should have had to do is swap out the dead drive with a new one rebuild the array with the parity data and we should be back to normal except there was a massive oversight in our setup each SSD is in a carrier card which is mounted into a PCIe tray which which is inside the server how do we replace a drive in this thing without taking the whole thing apart well we never documented which SSD is in which carrier card so even if I tore the entire server apart I'd have a hell of a time finding the Dead one so the answer is you don't you just leave it in there and replace it with a mismatch u.2 SSD in one of the empty front Bays that is an awful solution but at least it's a good transition into how we're going to fix this now if we weren't crazy we'd probably reconfigure our current Dell server to have more u.2 slots and then fill those with Enterprise drives but Enterprise drives are really expensive and we happen to have 22 of sabrin's absolutely baller eight terabyte Gen 4 plus drives left over from a previous video that were pretty much begging to be used for something like this but if we just swapped these ssds in sure we'd be faster and we'd have double the capacity but we would have the same maintenance problem enter these These are m.2 to u.2 adapters from Star Tech and icy Dock and what they allow us to do is take practically any consumer grade m.2 drive and install it into a hot swappable Bay in the front of a server just like this it's the best of every solution because we can use cheap consumer ssds they're easily serviceable without even powering down the machine and we get the full performance of each Drive instead of only half like we do with the carrier cards oh and you guys get this segue to our sponsor AMD get great deals during the game on AMD event while supplies last on AMD ryzen processors and AMD Radeon graphics cards including amazing game bundles for a limited time from now until July 1st Canada's birthday check them out at the link in the description even if we had all the parts on hand to turn our current production Dell server into new new wanek wanak 4. it wouldn't really be a good idea because we'd have to do all of this outside of ours and even fully populated we would still be eight days short of fitting all 32 of the m.2 drives so instead we're going to use a new server specifically we're going to be using the Gigabyte r282-z9g and you might remember this from a previous video that we did on g-raid that weird GPU-based NVMe raid that was wildly fast but we ultimately didn't use because it doesn't verify the data at all so if you like went in and like hex edited a file and then turned it back on it wouldn't catch that that's okay this machine is still perfectly good for software raid with plenty of AMD epic PCIe Lanes so what we're going to do is build it up and then just do a quick IP swap from new new wannick to new new new wanak no new mnemonic to new new mnemonic this has been in mothballs for a while though so it's time to talk about the config for CPUs we've gone with AMD epic 75 f3s these are third gen epics so that's the Milan family of processors 32 cores each the f is for high frequency these will boost up to 4 gigahertz meaning that they are perfect for high-speed storage applications like this the bottleneck is going to be our network interface and our application is that a fast yeah long before it will be these CPUs and especially long before it'll be those drives we're also missing a bunch of other essential gear here we have no network cards ah I got network cards no RAM Jake catch there's no coolers oh my God what are you doing 100 these are fancy ah those are connect X4 those are old now the good news is that these are actually really really inexpensive eight gig dimms that we have too many of to know what to do with and the reason we're using this is that while normally in a ZFS array you want a ton of RAM to use as an arc cash in a high speed NVMe array your drives are so fast that you're actually adding more overhead by using that cash than you are benefiting from it that's not to say that there's no need for RAM you could also use it as a metadata cache which will help accelerate NVMe storage like this it just doesn't need a ton of space I suspect it'll probably use between 5 and 10 gigs so we're just putting the absolute smallest dims we have to populate all the memory channels so we get all the speed just one dim per Channel there's 16 slots per CPU but that would be two dims per Channel we're just gonna do one well now they're sucking response I love how many adapters This Server comes with by default in order to get full PCIe by four operation to all of these front Bays here they basically take every slot in here every mezzanine card spot and then convert them to these Slimline ports and run them up to the front and that's in spite of the fact that with a two socket configuration you get a total of 160 usable PCIe Lanes okay please stop I wouldn't usually bother doing this on like a little consumer CPU but these things are so huge because there's multiple dies under that heat spreader it's not like traditional CPUs where if you put some thermal compound in the middle realistically you've got the hot spots covered there are lots of hot spots down there and they're very widely spread have you seen these yeah they're so yeah the Epic ones are green I know they never gave us one until we went to Texas for Genoa oh yeah lab use only that's awesome I know oh yeah those are special ones too those are like actually from the lab wait did you steal this no no I mean sort of oh I should probably start doing this hey oh oh why have I just been standing here this whole time oh right I was making fun of him there is one drawback to using these adapters every one of our m.2 drives needs to be installed on a sled now you probably noticed we've got two different options for our sleds this one is from Star Tech and honestly speaking is probably a little faster to use but doesn't have any additional cooling for the m.2 drive this one is from Icy Dock and is a bit more of a bear to deal with but comes with a large heatsink that will hopefully help with cooling I tested it it does make a difference if you were to do this in your personal computer for whatever reason or like a home lab where there's not a lot of airflow you should probably buy these ones for us this is going to be right in the front of a server with tens of thousands of RPM of fan I want to do this one this is way better oh we have to peel like 20 thermal pads but it's toolless Nick he's trying to not promote the screwdriver I have secret screwdriver things to show you oh secret screwdriver things that can't go on camera yeah they're in my pockets take your hands out of my pocket sir I don't appreciate this here you can show that oh sick what the hell is that don't worry about it oh yeah hey make sure we don't open too many yeah we can open them it's fine well yeah but like our um we're gonna run out of drives here at some point oh they're actually we're gonna run out of slots let's make sure we don't open too many of them and yeah these are not great sleds nope I am so glad the industry has moved on from using screws for these sleds though let's put it in there and it goes a gloop and then if you want to take it out you just go you have to make the noise when you do it otherwise it doesn't work see look uh it's not working clue ah see Magic these are freaking awesome you know what's even better I was looking at this earlier today and I was like wow I wonder if they make ones that go into the ruler stick ones like the e1s drives oh shut up they do the other thing I realized on these Star Tech ones is you can put a thermal pad between the m.2 and this PCB which obviously has a bunch of copper and it lowers the temps like five degrees but then the icy dock ones come down another three to five degrees after that yeah so the heatsink is definitely more better to be fair that was with the fans not really spinning that much in the server I suspect if you were to put a really big load on the server it would make less of a difference that's fair yeah we can put the network card in now it's a connect x400 gig card which is a bit old now you can get connect X6 which has a little bit more acceleration you can get it in 200 gig capacity I don't have any more connect X6 cards so that's what's going in there for now like to be clear the the most our Network throughput on wanna ever gets is like 30 gig maybe I mean that's kind of a lie it's a fair bit but three gigabytes a second it's not bad what is wrong with you what do you mean I don't think you dropped it so much and I just no did well don't touch it though I wanted to can I just show them what's going to happen no why it's a framework you can fix it I don't want no walks in oh that's perfectly fine we didn't even really like show off the server what kind of power supplies are in here 1600 Watts holy Christ here we go you're just taking in the noise this is double useful I don't have to hear the server and I don't have to listen to him look at all the lights I don't see any activity on them yet can you not what are these it's like a whole sled placeholders everyone this is manufactured garbage see okay so if we do four five wide raid Z ones we get 160. oh that's so much wasted capacity it's like 40 terabytes of Boise just do 10. 10 wide be a man I want the toxically masculine array I don't even have a Bugatti don't even talk to him packing the name frame DFS okay so we're going to turn the arc cache to be metadata only it's running kind of warm that's really hot though it's like wow oh this is gonna be a long command I'll provide moral support um yeah go Jake yeah Jake type it type it yeah come on Jake think about what oh this is awesome Jake you're doing so great that's still 15. 16 gig of 12 12 11. it's going down that's two that's nine that's eight is it possible we should have done with concealer ssds what is happening 18. what is happening right now you know parody calculations are a hell of a drug especially when your thing is 10 wide wow that is sucking back some CPU there 67 usage of 64. high frequency epic course 18 gigabytes a second is very fast this was 20 that's 20 per second man well this is awesome and this is only a q depth of four I mean I guess for a sequential load this is not the most accurate so here let's try two threads per drive it might go a little faster it's going like 24 and then 18 and then seems like it needs a second to settle into how the CPUs want to handle this it's really not that loud I know that sounds like a crazy thing to say while it's gone but he's not wrong we're probably kind of getting to the limit of ZFS here I'm impressed though 20 gigs a second is like very respectable for software raid like our CPU is not at 100 is that like 70. so there's still some Headroom for other things to run and these are not even Enterprise SSD these I know right I'm trying to think of editors yeah like what kind of workload could we possibly be hitting it with we would need to overwhelm that cash yeah you'd have to be copying from every station Red mags at their full transfer because remember they're not even that fast and that's assuming Samba could keep up which it can't although what we're going to use for high availability wantic weka FS they built their own Samba implementation that has SMB direct so the CPU usage from writing files to this file system would be like almost nothing when's that coming the server's here you know what else is here this message from our sponsor racing are you tired of slow and inefficient file transfers well have you heard about racing with exceptional speed and efficiency racing can transfer large files with ease plus with unlimited transfer volume and file size you'll never have to worry about limitations again but that's not all racing also offers Rich features like peer-to-peer transfer auto file sync and real-time data backup and with the ability to sync files of any size you can rest easy knowing your data is always safe and secure tucked into its bed like a cute little data that it is need to collaborate with your team racing has got you covered with group folders and real-time audit logs of users and transfers and with cross-platform compatibility you can seamlessly transfer files between Windows Mac Linux Android and iOS that almost feels wrong so what are you waiting for upgrade your file transfer game with racing today at the link below if you guys enjoyed this video you should probably get some more details on what's changed in our infrastructure in our last server room tour it's done it made it five minutes staying above 10 gigs a second oh sick"}