{"video_id":"uY_jeaxVgIE","title":"Where Intel is in REAL Trouble... - AMD EPYC Server Upgrade","channel":"Linus Tech Tips","show":"Linus Tech Tips","published_at":"2020-05-05T14:53:29Z","duration_s":1026,"segments":[{"start_s":0.24,"end_s":8.32,"text":"i'm not feeling great today but we have a dire situation here at Linus tech tips","speaker":null,"is_sponsor":0},{"start_s":4.96,"end_s":11.36,"text":"our wanik server our main production","speaker":null,"is_sponsor":0},{"start_s":8.32,"end_s":12.4,"text":"editing storage server is completely","speaker":null,"is_sponsor":0},{"start_s":11.36,"end_s":16.8,"text":"full fortunately i have been planning an","speaker":null,"is_sponsor":0},{"start_s":14.799,"end_s":20.08,"text":"upgrade to it for the last little while that is going to be freaking awesome","speaker":null,"is_sponsor":0},{"start_s":19.119,"end_s":24.64,"text":"this is new wanik it's gonna have more","speaker":null,"is_sponsor":0},{"start_s":22.72,"end_s":31.359,"text":"storage space whoa over 100 terabytes of NVMe storage it's","speaker":null,"is_sponsor":0},{"start_s":29.039,"end_s":35.44,"text":"gonna be faster it is going to have lower power consumption it is going to","speaker":null,"is_sponsor":0},{"start_s":33.28,"end_s":40.48,"text":"have more processing cores it is going to be epic and perhaps the most epic","speaker":null,"is_sponsor":0},{"start_s":38.239,"end_s":45.76,"text":"thing about it is going to be how many pci express lanes it has you see pci","speaker":null,"is_sponsor":0},{"start_s":43.92,"end_s":50.8,"text":"express gets used for all kinds of things graphics cards video capture","speaker":null,"is_sponsor":0},{"start_s":48.48,"end_s":56.719,"text":"devices peripheral expansion but thing is it's been fast enough for all those","speaker":null,"is_sponsor":0},{"start_s":53.52,"end_s":59.199,"text":"consumer uses for many years now so","speaker":null,"is_sponsor":0},{"start_s":56.719,"end_s":65.36,"text":"what's driving the creation of faster and faster pci express is actually the","speaker":null,"is_sponsor":0},{"start_s":62.559,"end_s":70.32,"text":"server world where it gets used for storage that's right that is a direct","speaker":null,"is_sponsor":0},{"start_s":67.68,"end_s":76.4,"text":"pci express by four connection at the back of all 24 of these drives and the","speaker":null,"is_sponsor":0},{"start_s":74.0,"end_s":80.96,"text":"truly unbelievable thing about this server is that unlike our existing wanik","speaker":null,"is_sponsor":0},{"start_s":78.72,"end_s":85.68,"text":"server which takes a handful of those pci express connections and splits it","speaker":null,"is_sponsor":0},{"start_s":83.6,"end_s":90.159,"text":"across 48 drives this one has a dedicated connection for","speaker":null,"is_sponsor":0},{"start_s":88.88,"end_s":93.92,"text":"every single one we are going to be putting up some crazy numbers today","speaker":null,"is_sponsor":0},{"start_s":92.32,"end_s":99.119,"text":"ladies and gentlemen and it's going to be brought to you by seasonic no","speaker":null,"is_sponsor":0},{"start_s":96.64,"end_s":102.88,"text":"marketing today just happy holidays from seasonic go check them out at the link","speaker":null,"is_sponsor":0},{"start_s":101.04,"end_s":109.479,"text":"in the video description really they they paid for a spot for that","speaker":null,"is_sponsor":0},{"start_s":106.159,"end_s":109.479,"text":"nice guys","speaker":null,"is_sponsor":0},{"start_s":114.64,"end_s":120.079,"text":"while i'm cracking these open let's talk about the drives that i'm using for the","speaker":null,"is_sponsor":0},{"start_s":118.399,"end_s":124.64,"text":"server and why reason number one is that wanik is full","speaker":null,"is_sponsor":0},{"start_s":122.96,"end_s":130.16,"text":"so i needed a capacity upgrade we're going from 1.2 terabyte drives to 4","speaker":null,"is_sponsor":0},{"start_s":128.239,"end_s":133.92,"text":"terabyte drives because that's what a couple of years of NVMe development does","speaker":null,"is_sponsor":0},{"start_s":132.08,"end_s":138.64,"text":"for you reason number two is we didn't want to go back to SATA because NVMe was","speaker":null,"is_sponsor":0},{"start_s":136.48,"end_s":143.52,"text":"such a great upgrade for us in terms of not just performance but stability last","speaker":null,"is_sponsor":0},{"start_s":141.28,"end_s":148.16,"text":"time thanks to the extremely low access latencies reason number three and really","speaker":null,"is_sponsor":0},{"start_s":146.4,"end_s":153.12,"text":"the main reason for these drives in particular is that we got a great tip","speaker":null,"is_sponsor":0},{"start_s":150.48,"end_s":157.599,"text":"from wendell over at level one techs that facebook was apparently flipping a","speaker":null,"is_sponsor":0},{"start_s":155.519,"end_s":164.48,"text":"bunch of these on ebay because they're upgrading to optane so these were a","speaker":null,"is_sponsor":0},{"start_s":160.879,"end_s":167.28,"text":"really great deal just a hair over 350","speaker":null,"is_sponsor":0},{"start_s":164.48,"end_s":172.56,"text":"dollars for Intel data center grade drives that apparently are still under","speaker":null,"is_sponsor":0},{"start_s":169.84,"end_s":176.879,"text":"warranty on top of everything else now they're not the fastest thing in the","speaker":null,"is_sponsor":0},{"start_s":173.92,"end_s":182.72,"text":"world by today's standards 3.2 gigabytes per second reads 1.8 gigabytes per","speaker":null,"is_sponsor":0},{"start_s":179.2,"end_s":186.0,"text":"second rights maximum read iops of 645","speaker":null,"is_sponsor":0},{"start_s":182.72,"end_s":188.4,"text":"000 and rights of 48 000 but remember","speaker":null,"is_sponsor":0},{"start_s":186.0,"end_s":195.76,"text":"guys that's almost not going to matter at all because i'm going to have 24","speaker":null,"is_sponsor":0},{"start_s":192.4,"end_s":197.28,"text":"of them in the same server and it's just","speaker":null,"is_sponsor":0},{"start_s":195.76,"end_s":201.879,"text":"a naz anyway now let's meet our server this","speaker":null,"is_sponsor":0},{"start_s":200.08,"end_s":207.04,"text":"is the Gigabyte r272 z32","speaker":null,"is_sponsor":0},{"start_s":204.8,"end_s":211.84,"text":"and we chose this particular model based on the glowing review that was given to","speaker":null,"is_sponsor":0},{"start_s":209.36,"end_s":215.76,"text":"it by our buddy patrick kennedy over at serve the home oh","speaker":null,"is_sponsor":0},{"start_s":214.48,"end_s":220.72,"text":"heavy boy this is a 2u server so that's to say it","speaker":null,"is_sponsor":0},{"start_s":218.239,"end_s":226.4,"text":"is two rack units in height and it was designed from the ground up for AMD's","speaker":null,"is_sponsor":0},{"start_s":223.36,"end_s":228.4,"text":"epic 7002 rome platform","speaker":null,"is_sponsor":0},{"start_s":226.4,"end_s":235.599,"text":"and the advantage we get from that is that it actually has compatibility with","speaker":null,"is_sponsor":0},{"start_s":230.72,"end_s":238.319,"text":"pci express gen 4 not all epic servers","speaker":null,"is_sponsor":0},{"start_s":235.599,"end_s":244.239,"text":"are PCIe gen 4 ready what gen 4 compatibility means for us in practical","speaker":null,"is_sponsor":0},{"start_s":240.319,"end_s":246.0,"text":"terms is not necessarily that much today","speaker":null,"is_sponsor":0},{"start_s":244.239,"end_s":249.92,"text":"but in the event that we wanted to upgrade it means that we could","speaker":null,"is_sponsor":0},{"start_s":247.28,"end_s":254.879,"text":"effectively double the bandwidth to almost the whole system so let's take a","speaker":null,"is_sponsor":0},{"start_s":252.08,"end_s":258.239,"text":"quick tour here we've got dual 1.2 000 watt power supplies here those are","speaker":null,"is_sponsor":0},{"start_s":256.32,"end_s":264.479,"text":"redundant in case one fails we've got our CPU socket here with support for up","speaker":null,"is_sponsor":0},{"start_s":260.16,"end_s":266.96,"text":"to 64 cores 128 threads we've got 16","speaker":null,"is_sponsor":0},{"start_s":264.479,"end_s":270.72,"text":"memory slots that run in eight channel mode freaking incredible","speaker":null,"is_sponsor":0},{"start_s":268.8,"end_s":276.639,"text":"bandwidth and on the subject of bandwidth most of the internal pci","speaker":null,"is_sponsor":0},{"start_s":273.52,"end_s":279.6,"text":"express slots are taken up of course by","speaker":null,"is_sponsor":0},{"start_s":276.639,"end_s":284.16,"text":"the 24 u.2 slots at the front so the way that they're fed is they're using their","speaker":null,"is_sponsor":0},{"start_s":281.759,"end_s":288.08,"text":"mezzanine card slot here to run four of them you can see that managed over here","speaker":null,"is_sponsor":0},{"start_s":286.4,"end_s":293.6,"text":"SATA actually goes to these two bays at the back so that's what you would typically boot off of now the CPU that","speaker":null,"is_sponsor":0},{"start_s":292.0,"end_s":300.0,"text":"we're using is pretty overkill but AMD sent over a","speaker":null,"is_sponsor":0},{"start_s":296.72,"end_s":301.919,"text":"bunch of cpus and unfortunately","speaker":null,"is_sponsor":0},{"start_s":300.0,"end_s":306.16,"text":"all the ones that kind of make more sense for this project like lower core","speaker":null,"is_sponsor":0},{"start_s":304.08,"end_s":311.919,"text":"count ones i sort of have earmarked for other projects so unfortunately i am","speaker":null,"is_sponsor":0},{"start_s":308.56,"end_s":314.08,"text":"locking away my 132 core","speaker":null,"is_sponsor":0},{"start_s":311.919,"end_s":317.36,"text":"basically forever that's okay though we really need the upgrade","speaker":null,"is_sponsor":0},{"start_s":316.16,"end_s":324.16,"text":"it really says a lot about the efficiency of these things that all we need is one of these basic","speaker":null,"is_sponsor":0},{"start_s":320.639,"end_s":326.24,"text":"passive heat pipe coolers thrown on here","speaker":null,"is_sponsor":0},{"start_s":324.16,"end_s":329.84,"text":"one of these shrouds and just the airflow from one of the chassis 80","speaker":null,"is_sponsor":0},{"start_s":327.84,"end_s":334.96,"text":"millimeter fans to keep it cool and remember guys it's validated for up to","speaker":null,"is_sponsor":0},{"start_s":331.759,"end_s":336.8,"text":"the 64 core one this is cool for my boot","speaker":null,"is_sponsor":0},{"start_s":334.96,"end_s":340.4,"text":"drive i had intended to use the SATA bays at the back of the chassis","speaker":null,"is_sponsor":0},{"start_s":338.479,"end_s":345.759,"text":"but then i realized i've actually got two m.2s that share","speaker":null,"is_sponsor":0},{"start_s":343.68,"end_s":350.479,"text":"their pci express lanes with this 8x slot over here so i figured if i have a","speaker":null,"is_sponsor":0},{"start_s":348.24,"end_s":353.36,"text":"couple of these old octane m.2s lying around","speaker":null,"is_sponsor":0},{"start_s":351.6,"end_s":358.24,"text":"what the hey might as well throw them in raid 1 and","speaker":null,"is_sponsor":0},{"start_s":354.88,"end_s":360.8,"text":"go full pci express on everything all","speaker":null,"is_sponsor":0},{"start_s":358.24,"end_s":364.8,"text":"right let's turn it around and oh oh that's funny Gigabyte sent over a uh a","speaker":null,"is_sponsor":0},{"start_s":363.52,"end_s":369.919,"text":"demo unit it seems to be a little bit broken","speaker":null,"is_sponsor":0},{"start_s":366.88,"end_s":371.759,"text":"that's okay oh they've got the stupid","speaker":null,"is_sponsor":0},{"start_s":369.919,"end_s":375.84,"text":"cable management things that make it really hard to plug in i mean it also","speaker":null,"is_sponsor":0},{"start_s":374.24,"end_s":381.609,"text":"makes it hard to come out accidentally which is nice","speaker":null,"is_sponsor":0},{"start_s":382.24,"end_s":388.08,"text":"now i've actually worked with this motherboard a little bit already for a","speaker":null,"is_sponsor":0},{"start_s":385.68,"end_s":391.68,"text":"whole home one CPU and it takes","speaker":null,"is_sponsor":0},{"start_s":390.4,"end_s":397.12,"text":"forever to post so i'm gonna fire it up while we check","speaker":null,"is_sponsor":0},{"start_s":395.28,"end_s":401.039,"text":"the compatibility of the sleds that our drives were pre-installed on","speaker":null,"is_sponsor":0},{"start_s":399.12,"end_s":404.96,"text":"and see if we're gonna have to swap all those out i really really hope i don't","speaker":null,"is_sponsor":0},{"start_s":402.8,"end_s":409.12,"text":"have to it's up","speaker":null,"is_sponsor":0},{"start_s":407.12,"end_s":413.36,"text":"that's good let's get in the BIOS and see uh if everything's detected","speaker":null,"is_sponsor":0},{"start_s":411.199,"end_s":416.56,"text":"correctly come on baby everything detector oh would that but","speaker":null,"is_sponsor":0},{"start_s":415.12,"end_s":423.84,"text":"memory training error what with the what now that is not all of our RAM that is","speaker":null,"is_sponsor":0},{"start_s":420.039,"end_s":426.479,"text":"458 gigs of RAM uh-oh okay brandon i've","speaker":null,"is_sponsor":0},{"start_s":423.84,"end_s":432.4,"text":"got good news and bad news um the good news is resetting those two memory","speaker":null,"is_sponsor":0},{"start_s":428.479,"end_s":434.4,"text":"modules boom 512 gigs of RAM so we're","speaker":null,"is_sponsor":0},{"start_s":432.4,"end_s":439.12,"text":"ready to rock as far as that goes the bad news is that due to AMD's","speaker":null,"is_sponsor":0},{"start_s":437.039,"end_s":444.24,"text":"architecture of their epic processors where there's no actual chipset with","speaker":null,"is_sponsor":0},{"start_s":441.68,"end_s":450.8,"text":"functionality like you know a raid controller or anything like that this","speaker":null,"is_sponsor":0},{"start_s":446.8,"end_s":454.08,"text":"has no support for hardware raid either","speaker":null,"is_sponsor":0},{"start_s":450.8,"end_s":455.599,"text":"on the m.2 drives or the SATA ones if","speaker":null,"is_sponsor":0},{"start_s":454.08,"end_s":460.0,"text":"you want to run raid on your state of drives you have to put an add-in raid","speaker":null,"is_sponsor":0},{"start_s":457.919,"end_s":463.68,"text":"card in and i ask Gigabyte well why didn't you guys just put","speaker":null,"is_sponsor":0},{"start_s":461.919,"end_s":468.639,"text":"a raid chip on the motherboard and they were like well it would have used up pci","speaker":null,"is_sponsor":0},{"start_s":465.759,"end_s":472.88,"text":"express lanes which i mean as part of the design of this board we were trying","speaker":null,"is_sponsor":0},{"start_s":470.319,"end_s":477.52,"text":"to reserve for all these NVMe drives so i kind of went ah okay fair enough now","speaker":null,"is_sponsor":0},{"start_s":474.72,"end_s":482.0,"text":"in Linux you can run your os on rated drives but Windows has no easy way to do","speaker":null,"is_sponsor":0},{"start_s":480.319,"end_s":486.37,"text":"that so that's just something we're going to have to consider as we","speaker":null,"is_sponsor":0},{"start_s":484.8,"end_s":491.12,"text":"build out this machine i'm going to shut it down for now though","speaker":null,"is_sponsor":0},{"start_s":489.199,"end_s":497.199,"text":"because the last thing that i really want to know is if i am remounting all","speaker":null,"is_sponsor":0},{"start_s":494.08,"end_s":498.879,"text":"of these drives to the included sleds","speaker":null,"is_sponsor":0},{"start_s":497.199,"end_s":502.8,"text":"let's see if they happen to be intercompatible that would be so cool","speaker":null,"is_sponsor":0},{"start_s":500.96,"end_s":505.599,"text":"that would be so cool oh","speaker":null,"is_sponsor":0},{"start_s":504.0,"end_s":511.199,"text":"yes yeah they are oh it is so tedious it's so","speaker":null,"is_sponsor":0},{"start_s":508.4,"end_s":516.24,"text":"tedious well that's not a good sign uh only my two m.2 drives are in here","speaker":null,"is_sponsor":0},{"start_s":514.24,"end_s":521.76,"text":"which means that maybe the sleds don't go all the way back","speaker":null,"is_sponsor":0},{"start_s":519.76,"end_s":524.88,"text":"that would blow all right","speaker":null,"is_sponsor":0},{"start_s":522.8,"end_s":528.48,"text":"i put one drive in each one and lined it up and","speaker":null,"is_sponsor":0},{"start_s":526.48,"end_s":532.08,"text":"you can see it doesn't go back far enough i gotta swap them all","speaker":null,"is_sponsor":0},{"start_s":532.24,"end_s":537.68,"text":"damage and i've had people ask me why so many","speaker":null,"is_sponsor":0},{"start_s":535.68,"end_s":541.2,"text":"people work here you thought i was gonna shock all these drives by myself and put","speaker":null,"is_sponsor":0},{"start_s":540.0,"end_s":547.279,"text":"them in mounts what one person can do slowly four people can do also slowly but less","speaker":null,"is_sponsor":0},{"start_s":544.88,"end_s":550.48,"text":"slowly pretty tedious now i just need to install a couple of roles and features","speaker":null,"is_sponsor":0},{"start_s":548.72,"end_s":553.279,"text":"including file server then","speaker":null,"is_sponsor":0},{"start_s":552.24,"end_s":557.68,"text":"yes we're going to shut it down and put all the drives in","speaker":null,"is_sponsor":0},{"start_s":556.0,"end_s":561.2,"text":"and we're going to hope they all work now really the most benefit for a","speaker":null,"is_sponsor":0},{"start_s":559.68,"end_s":566.56,"text":"solution like this is when you've got like a a storage","speaker":null,"is_sponsor":0},{"start_s":563.6,"end_s":572.32,"text":"attached network and you're using like banks and banks of these to act as","speaker":null,"is_sponsor":0},{"start_s":569.36,"end_s":576.08,"text":"the storage for like a whack of compute servers that are connected over fiber","speaker":null,"is_sponsor":0},{"start_s":574.16,"end_s":579.279,"text":"optics on you know the other end of the data center or whatever the case may be","speaker":null,"is_sponsor":0},{"start_s":578.24,"end_s":586.0,"text":"but come on guys this line is tech tips so we're using it","speaker":null,"is_sponsor":0},{"start_s":583.44,"end_s":589.839,"text":"as an 8k video editing nas i really hope these all pick up worst","speaker":null,"is_sponsor":0},{"start_s":588.24,"end_s":595.44,"text":"case scenario though we've got a couple of bad ones and the key night among you might have","speaker":null,"is_sponsor":0},{"start_s":592.64,"end_s":599.839,"text":"noticed there's two drives not installed these are either cold spares or to","speaker":null,"is_sponsor":0},{"start_s":597.92,"end_s":603.279,"text":"account for if a couple of our drives are just doa i mean that's the thing","speaker":null,"is_sponsor":0},{"start_s":601.12,"end_s":606.56,"text":"with used hardware you get a good deal you might get a couple bad ones all","speaker":null,"is_sponsor":0},{"start_s":604.959,"end_s":610.48,"text":"right that's a good sign every one of them has its light lit but","speaker":null,"is_sponsor":0},{"start_s":608.959,"end_s":614.64,"text":"we won't know if everything's picked up","speaker":null,"is_sponsor":0},{"start_s":612.24,"end_s":618.72,"text":"until we actually see them in here boom okay they're all there man storage","speaker":null,"is_sponsor":0},{"start_s":616.8,"end_s":623.839,"text":"spaces takes a lot of your capacity by default oh actually that's not that bad","speaker":null,"is_sponsor":0},{"start_s":620.64,"end_s":624.72,"text":"it's about 70 terabytes","speaker":null,"is_sponsor":0},{"start_s":623.839,"end_s":631.12,"text":"okay let's try that as long as it works oh you got to be","speaker":null,"is_sponsor":0},{"start_s":627.92,"end_s":632.88,"text":"kidding me this stupid error like","speaker":null,"is_sponsor":0},{"start_s":631.12,"end_s":637.36,"text":"five years or whatever it's had this problem and they don't just allow you to choose","speaker":null,"is_sponsor":0},{"start_s":635.76,"end_s":640.48,"text":"your columns in the gui you have to do it via","speaker":null,"is_sponsor":0},{"start_s":639.12,"end_s":644.399,"text":"powershell so dumb oh you can hear it ramping up","speaker":null,"is_sponsor":0},{"start_s":642.56,"end_s":648.399,"text":"now boys 5.5 gigs a second","speaker":null,"is_sponsor":0},{"start_s":646.48,"end_s":652.32,"text":"well that result is horrible worse across the board than old wanik","speaker":null,"is_sponsor":0},{"start_s":650.64,"end_s":656.32,"text":"clearly there's a configuration issue here all right so i did a simple","speaker":null,"is_sponsor":0},{"start_s":654.0,"end_s":661.2,"text":"mirrored space this time and we're gonna try running that we've","speaker":null,"is_sponsor":0},{"start_s":658.32,"end_s":664.16,"text":"got 44 terabytes here well that's better when it comes to","speaker":null,"is_sponsor":0},{"start_s":662.48,"end_s":669.12,"text":"sequential reads almost 11 gigs a second not even close","speaker":null,"is_sponsor":0},{"start_s":667.2,"end_s":675.36,"text":"to what we should be getting though okay that's it we're trying a simple virtual","speaker":null,"is_sponsor":0},{"start_s":671.68,"end_s":677.6,"text":"disk which is basically just striped","speaker":null,"is_sponsor":0},{"start_s":675.36,"end_s":682.959,"text":"if this doesn't perform well then i'm at a loss wow that is complete garbage town","speaker":null,"is_sponsor":0},{"start_s":680.8,"end_s":686.399,"text":"it's even worse how is that even possible","speaker":null,"is_sponsor":0},{"start_s":684.079,"end_s":690.56,"text":"and crystal disk mark is not the issue there's a microsoft tool called disk","speaker":null,"is_sponsor":0},{"start_s":688.079,"end_s":693.839,"text":"speed that you can use via command line to run sort of any mixture of loads that","speaker":null,"is_sponsor":0},{"start_s":692.88,"end_s":696.8,"text":"you want and it seems like there's just this hard","speaker":null,"is_sponsor":0},{"start_s":695.92,"end_s":703.279,"text":"cap at around 10 gigabytes a second which is like","speaker":null,"is_sponsor":0},{"start_s":701.36,"end_s":708.48,"text":"i mean obviously it's like good or whatever but for this hardware","speaker":null,"is_sponsor":0},{"start_s":706.0,"end_s":712.06,"text":"it's terrible so we're going to freenas","speaker":null,"is_sponsor":0},{"start_s":710.16,"end_s":714.56,"text":"that's it","speaker":null,"is_sponsor":0},{"start_s":714.56,"end_s":721.68,"text":"and it's been about four days so we did manage to push Windows","speaker":null,"is_sponsor":0},{"start_s":718.72,"end_s":725.6,"text":"performance a little bit further uh the drives needed firmware updates and they","speaker":null,"is_sponsor":0},{"start_s":723.839,"end_s":730.32,"text":"were doing some idle garbage collection stuff like that but we capped out at","speaker":null,"is_sponsor":0},{"start_s":727.68,"end_s":733.279,"text":"about 10 gigabytes a second which to be clear is plenty for anything you're","speaker":null,"is_sponsor":0},{"start_s":731.92,"end_s":737.44,"text":"going to access over the network i mean we would need a 100 gigabit per second","speaker":null,"is_sponsor":0},{"start_s":735.36,"end_s":740.0,"text":"network card in order to saturate that","speaker":null,"is_sponsor":0},{"start_s":738.88,"end_s":747.519,"text":"um but that's not enough for us we know this thing should be capable of so much","speaker":null,"is_sponsor":0},{"start_s":745.12,"end_s":753.68,"text":"more so we moved on to Linux Jake loaded it up with proxmox and built a pool with","speaker":null,"is_sponsor":0},{"start_s":750.88,"end_s":760.079,"text":"four v devs each with six drives in them running in raid z1 so raid z1 is kind of","speaker":null,"is_sponsor":0},{"start_s":757.519,"end_s":765.279,"text":"like raid 5 but without the right hole issue meaning that we should be giving","speaker":null,"is_sponsor":0},{"start_s":762.24,"end_s":768.24,"text":"up uh let's see four drives worth of","speaker":null,"is_sponsor":0},{"start_s":765.279,"end_s":773.04,"text":"capacity total but with the resiliency of being able to lose up to four drives","speaker":null,"is_sponsor":0},{"start_s":771.2,"end_s":777.04,"text":"as long as we don't lose more than one in a single v dev so you'd be well let's","speaker":null,"is_sponsor":0},{"start_s":775.6,"end_s":780.72,"text":"see six so four chunks like that of drives working","speaker":null,"is_sponsor":0},{"start_s":778.959,"end_s":785.279,"text":"together now that should have resulted in some rocking","speaker":null,"is_sponsor":0},{"start_s":782.72,"end_s":789.6,"text":"performance except it didn't it was actually complete","speaker":null,"is_sponsor":0},{"start_s":786.959,"end_s":795.76,"text":"dog crap like what did we max out at like a hundred giga megabytes a second","speaker":null,"is_sponsor":0},{"start_s":792.56,"end_s":800.72,"text":"maybe 200 200 megs a second","speaker":null,"is_sponsor":0},{"start_s":795.76,"end_s":803.2,"text":"terrible so turns out as soon as zfs was","speaker":null,"is_sponsor":0},{"start_s":800.72,"end_s":808.56,"text":"being loaded up our machine was actually spitting out kernel panics","speaker":null,"is_sponsor":0},{"start_s":806.24,"end_s":814.16,"text":"that's not a chicken emergency that's like really bad you liked it","speaker":null,"is_sponsor":0},{"start_s":811.04,"end_s":816.16,"text":"shut up you liked it that makes no sense","speaker":null,"is_sponsor":0},{"start_s":814.16,"end_s":821.279,"text":"a single one of these drives if i plug it into my computer should be doing 15","speaker":null,"is_sponsor":0},{"start_s":818.48,"end_s":825.6,"text":"to 20 times that performance and we verified this we checked in with our bud","speaker":null,"is_sponsor":0},{"start_s":823.36,"end_s":830.88,"text":"patrick over at serve the home he's got one of these boxes and he was getting","speaker":null,"is_sponsor":0},{"start_s":827.6,"end_s":833.12,"text":"great numbers with just 10 drives","speaker":null,"is_sponsor":0},{"start_s":830.88,"end_s":837.519,"text":"so what could we do well we called in the tech support for","speaker":null,"is_sponsor":0},{"start_s":835.2,"end_s":842.24,"text":"tech support wendell from level one tax got on the horn with Jake chatting it up","speaker":null,"is_sponsor":0},{"start_s":839.76,"end_s":846.8,"text":"on discord and between the two of them they managed to figure out that it looks","speaker":null,"is_sponsor":0},{"start_s":844.24,"end_s":851.12,"text":"like there is some kind of compatibility issue between the particular Linux","speaker":null,"is_sponsor":0},{"start_s":849.36,"end_s":856.88,"text":"kernel that the latest version of proxmox is running and zfs in our","speaker":null,"is_sponsor":0},{"start_s":854.32,"end_s":861.92,"text":"particular system that's causing these problems so we said you know what","speaker":null,"is_sponsor":0},{"start_s":859.519,"end_s":866.48,"text":"it we're just gonna run a benchmark where we load up each of the drives","speaker":null,"is_sponsor":0},{"start_s":863.839,"end_s":872.16,"text":"individually and slam them all at the same time Jake hit it","speaker":null,"is_sponsor":0},{"start_s":868.88,"end_s":872.16,"text":"he wandered away he'll be back","speaker":null,"is_sponsor":0},{"start_s":872.48,"end_s":877.839,"text":"okay you hear that thing ramping up","speaker":null,"is_sponsor":0},{"start_s":875.76,"end_s":882.399,"text":"oh my goodness oh my goodness what's going on here is","speaker":null,"is_sponsor":0},{"start_s":879.92,"end_s":888.0,"text":"that over a Gigabyte per second per drive 28 gigabytes a second","speaker":null,"is_sponsor":0},{"start_s":885.519,"end_s":893.68,"text":"at those kinds of speeds we are at the limits of AMD's infinity fabric and in","speaker":null,"is_sponsor":0},{"start_s":890.88,"end_s":898.079,"text":"fact to get to those speeds we actually had to overclock it a little bit thank","speaker":null,"is_sponsor":0},{"start_s":895.839,"end_s":900.56,"text":"you wendell for the tip there we pushed it too far it's actually spitting out","speaker":null,"is_sponsor":0},{"start_s":899.519,"end_s":905.519,"text":"some memory errors we're gonna have to dial it back a little bit but whatever you guys want to see impressive numbers","speaker":null,"is_sponsor":0},{"start_s":903.36,"end_s":909.199,"text":"right well here they are now i'm running a second script that he created that","speaker":null,"is_sponsor":0},{"start_s":907.199,"end_s":914.6,"text":"writes what was it 15 gigs of data so this is gonna write to the drives this","speaker":null,"is_sponsor":0},{"start_s":911.279,"end_s":918.24,"text":"time these numbers are nuts there you go","speaker":null,"is_sponsor":0},{"start_s":914.6,"end_s":921.6,"text":"23.7 gigabytes a second that are right","speaker":null,"is_sponsor":0},{"start_s":918.24,"end_s":925.199,"text":"up against the theoretical limits of","speaker":null,"is_sponsor":0},{"start_s":921.6,"end_s":926.72,"text":"what can be done so that's it guys that","speaker":null,"is_sponsor":0},{"start_s":925.199,"end_s":932.88,"text":"is basically the pinnacle of modern technology nearly","speaker":null,"is_sponsor":0},{"start_s":929.76,"end_s":936.16,"text":"30 gigabytes a second of reads and just","speaker":null,"is_sponsor":0},{"start_s":932.88,"end_s":938.24,"text":"over 20 gigabytes a second of rights","speaker":null,"is_sponsor":0},{"start_s":936.16,"end_s":942.199,"text":"guys to put that in perspective your home network is going to be probably","speaker":null,"is_sponsor":0},{"start_s":940.16,"end_s":948.24,"text":"gigabit this would be 300 times that to read it you could read","speaker":null,"is_sponsor":0},{"start_s":946.079,"end_s":951.44,"text":"one full blu-ray disc almost every second","speaker":null,"is_sponsor":0},{"start_s":950.24,"end_s":957.36,"text":"blu-ray blu-ray blu-ray it's crazy stuff now we just","speaker":null,"is_sponsor":0},{"start_s":955.44,"end_s":961.519,"text":"need to figure out some minor details like how we're actually gonna get an","speaker":null,"is_sponsor":0},{"start_s":959.68,"end_s":964.72,"text":"array loaded up on the thing because this is just","speaker":null,"is_sponsor":0},{"start_s":962.72,"end_s":967.92,"text":"i mean this is just theoretical stuff right now","speaker":null,"is_sponsor":0},{"start_s":965.839,"end_s":970.88,"text":"and theoretically i could have a good segue for a change but we all knew that","speaker":null,"is_sponsor":0},{"start_s":969.36,"end_s":975.12,"text":"wasn't going to happen thanks to our sponsor freshbooks for making this video","speaker":null,"is_sponsor":0},{"start_s":973.04,"end_s":978.72,"text":"possible today freshbooks is the cloud accounting solution that's built for how","speaker":null,"is_sponsor":0},{"start_s":976.88,"end_s":982.959,"text":"you want to work and you can work not just how you want but anywhere you want","speaker":null,"is_sponsor":0},{"start_s":980.959,"end_s":987.12,"text":"thanks to the freshbooks mobile app you can create professional looking invoices","speaker":null,"is_sponsor":0},{"start_s":984.959,"end_s":991.279,"text":"on the go snap pictures of your receipts so you don't lose them stay on top of","speaker":null,"is_sponsor":0},{"start_s":989.199,"end_s":995.199,"text":"important conversations and never miss an update for example being able to see","speaker":null,"is_sponsor":0},{"start_s":993.6,"end_s":999.04,"text":"when a client has viewed your invoice for the first time so you can get your","speaker":null,"is_sponsor":0},{"start_s":996.959,"end_s":1004.32,"text":"money you know what i'm talking about yeah you do so visit freshbooks.com","speaker":null,"is_sponsor":0},{"start_s":1002.079,"end_s":1007.68,"text":"forward slash tech tips to get your free 30-day trial today we're going to have","speaker":null,"is_sponsor":0},{"start_s":1005.92,"end_s":1010.56,"text":"that linked in the video description so thanks again for watching guys if you're","speaker":null,"is_sponsor":0},{"start_s":1009.04,"end_s":1014.32,"text":"looking for something else to watch server related maybe check out the video","speaker":null,"is_sponsor":0},{"start_s":1012.32,"end_s":1019.519,"text":"where i lost all our data you know i was accused in the comments of that video of","speaker":null,"is_sponsor":0},{"start_s":1015.92,"end_s":1022.639,"text":"being a very bad actor uh brandon out of","speaker":null,"is_sponsor":0},{"start_s":1019.519,"end_s":1024.72,"text":"10 how much was i acting in that video","speaker":null,"is_sponsor":0},{"start_s":1022.639,"end_s":1027.28,"text":"negative 10. that happened that was a real thing","speaker":null,"is_sponsor":0}],"full_text":"i'm not feeling great today but we have a dire situation here at Linus tech tips our wanik server our main production editing storage server is completely full fortunately i have been planning an upgrade to it for the last little while that is going to be freaking awesome this is new wanik it's gonna have more storage space whoa over 100 terabytes of NVMe storage it's gonna be faster it is going to have lower power consumption it is going to have more processing cores it is going to be epic and perhaps the most epic thing about it is going to be how many pci express lanes it has you see pci express gets used for all kinds of things graphics cards video capture devices peripheral expansion but thing is it's been fast enough for all those consumer uses for many years now so what's driving the creation of faster and faster pci express is actually the server world where it gets used for storage that's right that is a direct pci express by four connection at the back of all 24 of these drives and the truly unbelievable thing about this server is that unlike our existing wanik server which takes a handful of those pci express connections and splits it across 48 drives this one has a dedicated connection for every single one we are going to be putting up some crazy numbers today ladies and gentlemen and it's going to be brought to you by seasonic no marketing today just happy holidays from seasonic go check them out at the link in the video description really they they paid for a spot for that nice guys while i'm cracking these open let's talk about the drives that i'm using for the server and why reason number one is that wanik is full so i needed a capacity upgrade we're going from 1.2 terabyte drives to 4 terabyte drives because that's what a couple of years of NVMe development does for you reason number two is we didn't want to go back to SATA because NVMe was such a great upgrade for us in terms of not just performance but stability last time thanks to the extremely low access latencies reason number three and really the main reason for these drives in particular is that we got a great tip from wendell over at level one techs that facebook was apparently flipping a bunch of these on ebay because they're upgrading to optane so these were a really great deal just a hair over 350 dollars for Intel data center grade drives that apparently are still under warranty on top of everything else now they're not the fastest thing in the world by today's standards 3.2 gigabytes per second reads 1.8 gigabytes per second rights maximum read iops of 645 000 and rights of 48 000 but remember guys that's almost not going to matter at all because i'm going to have 24 of them in the same server and it's just a naz anyway now let's meet our server this is the Gigabyte r272 z32 and we chose this particular model based on the glowing review that was given to it by our buddy patrick kennedy over at serve the home oh heavy boy this is a 2u server so that's to say it is two rack units in height and it was designed from the ground up for AMD's epic 7002 rome platform and the advantage we get from that is that it actually has compatibility with pci express gen 4 not all epic servers are PCIe gen 4 ready what gen 4 compatibility means for us in practical terms is not necessarily that much today but in the event that we wanted to upgrade it means that we could effectively double the bandwidth to almost the whole system so let's take a quick tour here we've got dual 1.2 000 watt power supplies here those are redundant in case one fails we've got our CPU socket here with support for up to 64 cores 128 threads we've got 16 memory slots that run in eight channel mode freaking incredible bandwidth and on the subject of bandwidth most of the internal pci express slots are taken up of course by the 24 u.2 slots at the front so the way that they're fed is they're using their mezzanine card slot here to run four of them you can see that managed over here SATA actually goes to these two bays at the back so that's what you would typically boot off of now the CPU that we're using is pretty overkill but AMD sent over a bunch of cpus and unfortunately all the ones that kind of make more sense for this project like lower core count ones i sort of have earmarked for other projects so unfortunately i am locking away my 132 core basically forever that's okay though we really need the upgrade it really says a lot about the efficiency of these things that all we need is one of these basic passive heat pipe coolers thrown on here one of these shrouds and just the airflow from one of the chassis 80 millimeter fans to keep it cool and remember guys it's validated for up to the 64 core one this is cool for my boot drive i had intended to use the SATA bays at the back of the chassis but then i realized i've actually got two m.2s that share their pci express lanes with this 8x slot over here so i figured if i have a couple of these old octane m.2s lying around what the hey might as well throw them in raid 1 and go full pci express on everything all right let's turn it around and oh oh that's funny Gigabyte sent over a uh a demo unit it seems to be a little bit broken that's okay oh they've got the stupid cable management things that make it really hard to plug in i mean it also makes it hard to come out accidentally which is nice now i've actually worked with this motherboard a little bit already for a whole home one CPU and it takes forever to post so i'm gonna fire it up while we check the compatibility of the sleds that our drives were pre-installed on and see if we're gonna have to swap all those out i really really hope i don't have to it's up that's good let's get in the BIOS and see uh if everything's detected correctly come on baby everything detector oh would that but memory training error what with the what now that is not all of our RAM that is 458 gigs of RAM uh-oh okay brandon i've got good news and bad news um the good news is resetting those two memory modules boom 512 gigs of RAM so we're ready to rock as far as that goes the bad news is that due to AMD's architecture of their epic processors where there's no actual chipset with functionality like you know a raid controller or anything like that this has no support for hardware raid either on the m.2 drives or the SATA ones if you want to run raid on your state of drives you have to put an add-in raid card in and i ask Gigabyte well why didn't you guys just put a raid chip on the motherboard and they were like well it would have used up pci express lanes which i mean as part of the design of this board we were trying to reserve for all these NVMe drives so i kind of went ah okay fair enough now in Linux you can run your os on rated drives but Windows has no easy way to do that so that's just something we're going to have to consider as we build out this machine i'm going to shut it down for now though because the last thing that i really want to know is if i am remounting all of these drives to the included sleds let's see if they happen to be intercompatible that would be so cool that would be so cool oh yes yeah they are oh it is so tedious it's so tedious well that's not a good sign uh only my two m.2 drives are in here which means that maybe the sleds don't go all the way back that would blow all right i put one drive in each one and lined it up and you can see it doesn't go back far enough i gotta swap them all damage and i've had people ask me why so many people work here you thought i was gonna shock all these drives by myself and put them in mounts what one person can do slowly four people can do also slowly but less slowly pretty tedious now i just need to install a couple of roles and features including file server then yes we're going to shut it down and put all the drives in and we're going to hope they all work now really the most benefit for a solution like this is when you've got like a a storage attached network and you're using like banks and banks of these to act as the storage for like a whack of compute servers that are connected over fiber optics on you know the other end of the data center or whatever the case may be but come on guys this line is tech tips so we're using it as an 8k video editing nas i really hope these all pick up worst case scenario though we've got a couple of bad ones and the key night among you might have noticed there's two drives not installed these are either cold spares or to account for if a couple of our drives are just doa i mean that's the thing with used hardware you get a good deal you might get a couple bad ones all right that's a good sign every one of them has its light lit but we won't know if everything's picked up until we actually see them in here boom okay they're all there man storage spaces takes a lot of your capacity by default oh actually that's not that bad it's about 70 terabytes okay let's try that as long as it works oh you got to be kidding me this stupid error like five years or whatever it's had this problem and they don't just allow you to choose your columns in the gui you have to do it via powershell so dumb oh you can hear it ramping up now boys 5.5 gigs a second well that result is horrible worse across the board than old wanik clearly there's a configuration issue here all right so i did a simple mirrored space this time and we're gonna try running that we've got 44 terabytes here well that's better when it comes to sequential reads almost 11 gigs a second not even close to what we should be getting though okay that's it we're trying a simple virtual disk which is basically just striped if this doesn't perform well then i'm at a loss wow that is complete garbage town it's even worse how is that even possible and crystal disk mark is not the issue there's a microsoft tool called disk speed that you can use via command line to run sort of any mixture of loads that you want and it seems like there's just this hard cap at around 10 gigabytes a second which is like i mean obviously it's like good or whatever but for this hardware it's terrible so we're going to freenas that's it and it's been about four days so we did manage to push Windows performance a little bit further uh the drives needed firmware updates and they were doing some idle garbage collection stuff like that but we capped out at about 10 gigabytes a second which to be clear is plenty for anything you're going to access over the network i mean we would need a 100 gigabit per second network card in order to saturate that um but that's not enough for us we know this thing should be capable of so much more so we moved on to Linux Jake loaded it up with proxmox and built a pool with four v devs each with six drives in them running in raid z1 so raid z1 is kind of like raid 5 but without the right hole issue meaning that we should be giving up uh let's see four drives worth of capacity total but with the resiliency of being able to lose up to four drives as long as we don't lose more than one in a single v dev so you'd be well let's see six so four chunks like that of drives working together now that should have resulted in some rocking performance except it didn't it was actually complete dog crap like what did we max out at like a hundred giga megabytes a second maybe 200 200 megs a second terrible so turns out as soon as zfs was being loaded up our machine was actually spitting out kernel panics that's not a chicken emergency that's like really bad you liked it shut up you liked it that makes no sense a single one of these drives if i plug it into my computer should be doing 15 to 20 times that performance and we verified this we checked in with our bud patrick over at serve the home he's got one of these boxes and he was getting great numbers with just 10 drives so what could we do well we called in the tech support for tech support wendell from level one tax got on the horn with Jake chatting it up on discord and between the two of them they managed to figure out that it looks like there is some kind of compatibility issue between the particular Linux kernel that the latest version of proxmox is running and zfs in our particular system that's causing these problems so we said you know what it we're just gonna run a benchmark where we load up each of the drives individually and slam them all at the same time Jake hit it he wandered away he'll be back okay you hear that thing ramping up oh my goodness oh my goodness what's going on here is that over a Gigabyte per second per drive 28 gigabytes a second at those kinds of speeds we are at the limits of AMD's infinity fabric and in fact to get to those speeds we actually had to overclock it a little bit thank you wendell for the tip there we pushed it too far it's actually spitting out some memory errors we're gonna have to dial it back a little bit but whatever you guys want to see impressive numbers right well here they are now i'm running a second script that he created that writes what was it 15 gigs of data so this is gonna write to the drives this time these numbers are nuts there you go 23.7 gigabytes a second that are right up against the theoretical limits of what can be done so that's it guys that is basically the pinnacle of modern technology nearly 30 gigabytes a second of reads and just over 20 gigabytes a second of rights guys to put that in perspective your home network is going to be probably gigabit this would be 300 times that to read it you could read one full blu-ray disc almost every second blu-ray blu-ray blu-ray it's crazy stuff now we just need to figure out some minor details like how we're actually gonna get an array loaded up on the thing because this is just i mean this is just theoretical stuff right now and theoretically i could have a good segue for a change but we all knew that wasn't going to happen thanks to our sponsor freshbooks for making this video possible today freshbooks is the cloud accounting solution that's built for how you want to work and you can work not just how you want but anywhere you want thanks to the freshbooks mobile app you can create professional looking invoices on the go snap pictures of your receipts so you don't lose them stay on top of important conversations and never miss an update for example being able to see when a client has viewed your invoice for the first time so you can get your money you know what i'm talking about yeah you do so visit freshbooks.com forward slash tech tips to get your free 30-day trial today we're going to have that linked in the video description so thanks again for watching guys if you're looking for something else to watch server related maybe check out the video where i lost all our data you know i was accused in the comments of that video of being a very bad actor uh brandon out of 10 how much was i acting in that video negative 10. that happened that was a real thing"}