{"video_id":"CcHevgjAnV0","title":"this server WONT break....  - Clustered Storage Server","channel":"Linus Tech Tips","show":"Linus Tech Tips","published_at":"2024-05-04T14:53:29Z","duration_s":1677,"segments":[{"start_s":0.0,"end_s":7.24,"text":"when you make as many videos as we do you need a lot of fast reliable storage","speaker":null,"is_sponsor":0},{"start_s":4.04,"end_s":9.76,"text":"and our main editing server wanic has","speaker":null,"is_sponsor":0},{"start_s":7.24,"end_s":13.4,"text":"checked all of those boxes for years it's a great little server it's built","speaker":null,"is_sponsor":0},{"start_s":11.639,"end_s":18.439,"text":"out of high quality components and it even looks cool but as our team is grown","speaker":null,"is_sponsor":0},{"start_s":16.279,"end_s":25.16,"text":"we've reached the point where even a minute one single minute of downtime","speaker":null,"is_sponsor":0},{"start_s":21.64,"end_s":27.8,"text":"costs over $50 and that's just in","speaker":null,"is_sponsor":0},{"start_s":25.16,"end_s":32.88,"text":"payroll now practically speaking the way to mitigate that is by adding redundant","speaker":null,"is_sponsor":0},{"start_s":30.679,"end_s":37.559,"text":"now our drives are already redundant we've got 20 drives in there with data","speaker":null,"is_sponsor":0},{"start_s":34.68,"end_s":43.28,"text":"striping but the problem is they all sit in one single server I'm sure you can","speaker":null,"is_sponsor":0},{"start_s":40.92,"end_s":48.92,"text":"see where this is going it's been over a year in the making but it's finally here","speaker":null,"is_sponsor":0},{"start_s":45.96,"end_s":56.0,"text":"wanic final form and I'm calling it wanic 10 because it's the last W ever","speaker":null,"is_sponsor":0},{"start_s":52.399,"end_s":58.12,"text":"avability W you this like 10 times","speaker":null,"is_sponsor":0},{"start_s":56.0,"end_s":63.32,"text":"nobody even knows what high availability means it means it's lus just go ahead","speaker":null,"is_sponsor":0},{"start_s":60.48,"end_s":66.439,"text":"unplug one do it go for it well okay I should probably tell you the stakes","speaker":null,"is_sponsor":0},{"start_s":64.839,"end_s":70.4,"text":"before you do that each of these two grand twin boxes has four entire servers","speaker":null,"is_sponsor":0},{"start_s":69.119,"end_s":77.08,"text":"inside of them that were provided by super micro who sponsored this whole thing and they're set up with WCA a","speaker":null,"is_sponsor":0},{"start_s":74.08,"end_s":79.72,"text":"redundant NVMe first file system in this","speaker":null,"is_sponsor":1},{"start_s":77.08,"end_s":84.159,"text":"config it should sustain two entire servers dropping out without anyone even","speaker":null,"is_sponsor":1},{"start_s":81.96,"end_s":87.84,"text":"noticing except that we moved the entire team onto it last night without telling","speaker":null,"is_sponsor":0},{"start_s":85.84,"end_s":91.72,"text":"anyone and it's the middle of the work day with a ton of high priority videos","speaker":null,"is_sponsor":0},{"start_s":89.4,"end_s":97.52,"text":"in progress do you really want to test it right now I like I haven't tried that","speaker":null,"is_sponsor":0},{"start_s":93.88,"end_s":99.96,"text":"all right here we go okay what could go","speaker":null,"is_sponsor":0},{"start_s":97.52,"end_s":103.26,"text":"wrong I mean a","speaker":null,"is_sponsor":0},{"start_s":107.88,"end_s":114.68,"text":"lot naturally a huge part of a project like this is the software the stuff","speaker":null,"is_sponsor":0},{"start_s":112.84,"end_s":121.24,"text":"that's going to handle Distributing all of ourish terabytes of video projects","speaker":null,"is_sponsor":0},{"start_s":118.0,"end_s":122.799,"text":"Word documents and Linux isos to the","speaker":null,"is_sponsor":0},{"start_s":121.24,"end_s":128.239,"text":"multiple machines that we just showed you but we can't install any software","speaker":null,"is_sponsor":0},{"start_s":125.68,"end_s":135.48,"text":"until we have some Hardware so why don't we start there meet the super micro","speaker":null,"is_sponsor":0},{"start_s":130.679,"end_s":138.2,"text":"Grand twin A+ server as- 2115 gt-","speaker":null,"is_sponsor":0},{"start_s":135.48,"end_s":141.92,"text":"hntr despite its sort of ordinary looking appearance and unexciting","speaker":null,"is_sponsor":0},{"start_s":140.08,"end_s":148.8,"text":"sounding name it is anything but ordinary and it is very","speaker":null,"is_sponsor":0},{"start_s":145.12,"end_s":151.72,"text":"exciting because inside this 2u is four","speaker":null,"is_sponsor":0},{"start_s":148.8,"end_s":159.2,"text":"independent Compu computers but for what we're doing four nodes please we want","speaker":null,"is_sponsor":0},{"start_s":156.28,"end_s":165.4,"text":"eight inside each of these is a completely independent motherboard 384","speaker":null,"is_sponsor":0},{"start_s":163.04,"end_s":174.159,"text":"gigs of memory an AMD epic Genoa processor with 64 cores dual m.2 slots","speaker":null,"is_sponsor":0},{"start_s":169.519,"end_s":177.56,"text":"for redundant boot drives six PCIe Gen 5","speaker":null,"is_sponsor":0},{"start_s":174.159,"end_s":181.599,"text":"2 and 1/2 in NVMe slots up front and","speaker":null,"is_sponsor":0},{"start_s":177.56,"end_s":183.12,"text":"we've got IO in the rear now this bit","speaker":null,"is_sponsor":0},{"start_s":181.599,"end_s":190.2,"text":"here could be a little confusing at first glance but that is because not","speaker":null,"is_sponsor":0},{"start_s":186.44,"end_s":193.92,"text":"only do we have USB but we have two full","speaker":null,"is_sponsor":0},{"start_s":190.2,"end_s":196.48,"text":"gen 5x6 PCIe connections back here along","speaker":null,"is_sponsor":0},{"start_s":193.92,"end_s":202.319,"text":"with display output and power for the entire server this whole thing slides","speaker":null,"is_sponsor":0},{"start_s":200.04,"end_s":206.159,"text":"into the chassis which holds a really cool modular backplane assembly that","speaker":null,"is_sponsor":0},{"start_s":204.519,"end_s":211.28,"text":"we'll take a look at in a minute and then passes through thank you Jake ah to","speaker":null,"is_sponsor":0},{"start_s":209.36,"end_s":216.2,"text":"the back at the server where you've got a Management Port a single USB port for","speaker":null,"is_sponsor":0},{"start_s":214.159,"end_s":222.12,"text":"each server nope it's two and they're shared what the I was about to ask cuz","speaker":null,"is_sponsor":0},{"start_s":219.64,"end_s":227.439,"text":"we've also got a single VGA you see the button for two servers there no way this","speaker":null,"is_sponsor":0},{"start_s":225.0,"end_s":234.079,"text":"button toggles yeah and okay before we talk about that","speaker":null,"is_sponsor":0},{"start_s":230.64,"end_s":237.28,"text":"a little bit more look at these power","speaker":null,"is_sponsor":0},{"start_s":234.079,"end_s":240.959,"text":"supplies each of these is","speaker":null,"is_sponsor":0},{"start_s":237.28,"end_s":242.68,"text":"2200 Watts 80 plus typ tianium which","speaker":null,"is_sponsor":0},{"start_s":240.959,"end_s":247.959,"text":"sounds like a lot but when you're potentially handling four 400 wat epic","speaker":null,"is_sponsor":0},{"start_s":245.92,"end_s":252.72,"text":"Genoa CPUs along with a bunch of RAM up to 24 NVMe drives and eight network","speaker":null,"is_sponsor":0},{"start_s":250.159,"end_s":257.239,"text":"cards well it seems downright reasonable doesn't it is it 24 drives can't be 6","speaker":null,"is_sponsor":0},{"start_s":255.56,"end_s":263.8,"text":"yes 6 * 4 is 24 and of course that's just one of them","speaker":null,"is_sponsor":0},{"start_s":261.28,"end_s":268.0,"text":"we've got two of those and that means that in the event that one of these dies","speaker":null,"is_sponsor":0},{"start_s":265.919,"end_s":273.32,"text":"the system should be able to continue to operate uninterrupted which is a big","speaker":null,"is_sponsor":0},{"start_s":270.68,"end_s":278.36,"text":"part of the high availability goal that we have for this deployment speaking of","speaker":null,"is_sponsor":0},{"start_s":276.44,"end_s":284.8,"text":"high availability let's move on to our network cards each of those PCIe gen 5x6","speaker":null,"is_sponsor":0},{"start_s":283.0,"end_s":290.52,"text":"slots I showed you guys before terminates in one of these ocp 3.0 small","speaker":null,"is_sponsor":0},{"start_s":287.759,"end_s":296.08,"text":"form factor mezzanine slots and what we're putting in them is these connectx","speaker":null,"is_sponsor":0},{"start_s":293.24,"end_s":303.639,"text":"6 200 gbit cards from melanox excuse me from NVIDIA that okay","speaker":null,"is_sponsor":0},{"start_s":300.68,"end_s":308.16,"text":"these are the older Gen 4 ones so they're going to be limited by the slot","speaker":null,"is_sponsor":0},{"start_s":305.16,"end_s":310.639,"text":"speed of around 250 gabit per second but","speaker":null,"is_sponsor":0},{"start_s":308.16,"end_s":316.88,"text":"if we had newer cards that means that each of these nodes could do 200 plus","speaker":null,"is_sponsor":0},{"start_s":314.24,"end_s":321.68,"text":"another 200 400 up to 800 gigabit which would of course be a","speaker":null,"is_sponsor":0},{"start_s":319.6,"end_s":328.36,"text":"complete waste for us a because our workload can't take advantage of it and","speaker":null,"is_sponsor":0},{"start_s":323.4,"end_s":330.919,"text":"B because our switch is only 100 gbit","speaker":null,"is_sponsor":0},{"start_s":328.36,"end_s":335.6,"text":"sorry of course the two ports are still helpful we do have redundant","speaker":null,"is_sponsor":0},{"start_s":333.36,"end_s":339.6,"text":"switches except there's kind of a problem here that's still a single point","speaker":null,"is_sponsor":0},{"start_s":337.44,"end_s":345.12,"text":"of failure in a perfect world we would have two single port Nicks so if a Nick","speaker":null,"is_sponsor":0},{"start_s":342.16,"end_s":349.919,"text":"were to die it would still be okay but because we have so many nodes we're not","speaker":null,"is_sponsor":0},{"start_s":347.919,"end_s":354.039,"text":"really worried about an individual node you know they could have one boot drive","speaker":null,"is_sponsor":0},{"start_s":351.8,"end_s":359.16,"text":"and it die or one Nick and it die we still have an extra backup how many","speaker":null,"is_sponsor":0},{"start_s":356.8,"end_s":364.759,"text":"nines do you want I mean I don't know like one would would be good 9% which","speaker":null,"is_sponsor":0},{"start_s":362.479,"end_s":369.44,"text":"Jokes Aside is a really good point if we were architecting this properly there","speaker":null,"is_sponsor":0},{"start_s":366.919,"end_s":373.36,"text":"are so many more considerations that we would need to make like the power coming","speaker":null,"is_sponsor":0},{"start_s":371.68,"end_s":378.0,"text":"into the rack would have to come from two independent backed up sources the","speaker":null,"is_sponsor":0},{"start_s":376.0,"end_s":382.24,"text":"connectivity to our clients would have to be redundant as well the connectivity","speaker":null,"is_sponsor":0},{"start_s":380.639,"end_s":385.56,"text":"between all of the systems would have to be architected in such a way that no","speaker":null,"is_sponsor":0},{"start_s":383.639,"end_s":390.479,"text":"matter what fails everything will stay up and realistically for us we're not","speaker":null,"is_sponsor":0},{"start_s":388.8,"end_s":395.199,"text":"going to get that deep into it because our goal is better than we had before","speaker":null,"is_sponsor":0},{"start_s":393.28,"end_s":399.52,"text":"which was a single machine with its own built-in redundancies but other than","speaker":null,"is_sponsor":0},{"start_s":397.319,"end_s":403.319,"text":"that nothing now at least we should be able to lose a full machine out of these","speaker":null,"is_sponsor":0},{"start_s":401.68,"end_s":408.759,"text":"eight we can restart one of our core switches totally fine two machines out","speaker":null,"is_sponsor":0},{"start_s":405.479,"end_s":410.919,"text":"of these eight and we can still be","speaker":null,"is_sponsor":0},{"start_s":408.759,"end_s":414.639,"text":"limping along I mean limping is a bit of a stretch it's going to be very fast now","speaker":null,"is_sponsor":0},{"start_s":413.24,"end_s":419.72,"text":"normally if you buy a super micro machine they're going to pre-build it for you they're going to validate it for","speaker":null,"is_sponsor":0},{"start_s":417.039,"end_s":424.479,"text":"you you can even have them pre-build an entire Rack or racks of these things and","speaker":null,"is_sponsor":0},{"start_s":422.639,"end_s":428.52,"text":"then validate your application on it before it ships to you in fact we've got","speaker":null,"is_sponsor":0},{"start_s":427.039,"end_s":436.68,"text":"a whole video that we did about that that was sponsored by super micro a little while back of course this is LT","speaker":null,"is_sponsor":0},{"start_s":433.319,"end_s":438.879,"text":"my friends so we will be assembling this","speaker":null,"is_sponsor":0},{"start_s":436.68,"end_s":443.16,"text":"one ourselves do you like that spin of the screwdriver above the server don't","speaker":null,"is_sponsor":0},{"start_s":440.639,"end_s":446.919,"text":"worry I won't miss I'll never miss see I could do this a hundred times and I","speaker":null,"is_sponsor":1},{"start_s":444.36,"end_s":451.639,"text":"would never miss why no it's fine it's good it's okay we have seven more any","speaker":null,"is_sponsor":1},{"start_s":449.16,"end_s":456.12,"text":"who for our CPU we've gone with an epic Genova","speaker":null,"is_sponsor":0},{"start_s":452.639,"end_s":460.8,"text":"9534 this is a 64 core","speaker":null,"is_sponsor":0},{"start_s":456.12,"end_s":464.28,"text":"128 thread monster of a CPU it'll do 3.7","speaker":null,"is_sponsor":0},{"start_s":460.8,"end_s":467.159,"text":"GHz Max boost it has A4 Gigabyte of","speaker":null,"is_sponsor":0},{"start_s":464.28,"end_s":475.12,"text":"level three cache a 300 wat TDP it supports ddr5 memory up to 12 channels","speaker":null,"is_sponsor":0},{"start_s":471.24,"end_s":478.8,"text":"and it supports a whopping 128 Lanes of","speaker":null,"is_sponsor":0},{"start_s":475.12,"end_s":481.4,"text":"PCIe Gen 5 originally we were intending","speaker":null,"is_sponsor":0},{"start_s":478.8,"end_s":486.8,"text":"to go with 32 core chips but they were out of stock so free upgrade lucky us","speaker":null,"is_sponsor":0},{"start_s":484.879,"end_s":492.84,"text":"compared to previous generation AMD epic CPUs dooa is a big step up in terms of","speaker":null,"is_sponsor":0},{"start_s":489.919,"end_s":496.84,"text":"IO performance which makes it perfect for this application and in the long","speaker":null,"is_sponsor":0},{"start_s":495.12,"end_s":501.759,"text":"term I mean if we've got all the extra CPU cores and a whole bunch of RAM","speaker":null,"is_sponsor":0},{"start_s":499.039,"end_s":506.199,"text":"anyway why run WCA on the bare metal when we could install prox Mox and then","speaker":null,"is_sponsor":0},{"start_s":503.919,"end_s":511.36,"text":"use the other cores for I don't know High","speaker":null,"is_sponsor":0},{"start_s":507.319,"end_s":513.08,"text":"availability Plex server yeah Linux isos","speaker":null,"is_sponsor":0},{"start_s":511.36,"end_s":516.599,"text":"more realistically it would be something like active directory yeah which we","speaker":null,"is_sponsor":0},{"start_s":515.32,"end_s":520.479,"text":"don't really want to do right now because if you run active directory on","speaker":null,"is_sponsor":0},{"start_s":518.479,"end_s":525.399,"text":"one server and it goes down you're going to have a really really bad time but if","speaker":null,"is_sponsor":0},{"start_s":522.719,"end_s":530.04,"text":"you run it on a bunch of servers yeah it's good great so normally server CPU","speaker":null,"is_sponsor":0},{"start_s":528.399,"end_s":533.92,"text":"coolers would come with their own thermal paste pre-applied but since","speaker":null,"is_sponsor":0},{"start_s":531.88,"end_s":537.48,"text":"we're doing this ourselves and uh if you look carefully it's not the first time","speaker":null,"is_sponsor":0},{"start_s":535.64,"end_s":544.079,"text":"that it's been installed we are going to be using okay thank you for that a piece","speaker":null,"is_sponsor":0},{"start_s":540.2,"end_s":547.0,"text":"of Honeywell PTM 7950 this stuff is","speaker":null,"is_sponsor":0},{"start_s":544.079,"end_s":552.32,"text":"freaking awesome it has great thermal transfer properties and it can handle","speaker":null,"is_sponsor":0},{"start_s":549.48,"end_s":556.839,"text":"varying temperatures like seriously I don't remember many not even just","speaker":null,"is_sponsor":0},{"start_s":553.88,"end_s":561.6,"text":"varying but like a lot of huge cycles for a very very long time now available","speaker":null,"is_sponsor":0},{"start_s":559.6,"end_s":566.32,"text":"LTD store.com is that big enough does that cover all of the ccds and","speaker":null,"is_sponsor":1},{"start_s":563.88,"end_s":569.68,"text":"cxs oh there's a second piece of PL am I stupid is there a second piece of","speaker":null,"is_sponsor":0},{"start_s":568.04,"end_s":572.8,"text":"plastic no there isn't should I put one in the fridge no no no it's totally fine","speaker":null,"is_sponsor":0},{"start_s":571.279,"end_s":576.8,"text":"I've done this like a bunch of times yeah oh she's Min look at that see all","speaker":null,"is_sponsor":0},{"start_s":575.0,"end_s":580.48,"text":"right easy I would recommend putting it in the fridge before you use it all","speaker":null,"is_sponsor":0},{"start_s":578.88,"end_s":585.12,"text":"right to ensure we're making the absolute most of our CPU especially in","speaker":null,"is_sponsor":0},{"start_s":582.959,"end_s":590.399,"text":"this High throughput storage workload we're going to be populating all 12 of","speaker":null,"is_sponsor":0},{"start_s":587.48,"end_s":597.56,"text":"our memory Channels with 32 gig dims of ddr5 ECC running at 4800 megga","speaker":null,"is_sponsor":0},{"start_s":593.44,"end_s":602.6,"text":"transitors per second that's a total","speaker":null,"is_sponsor":0},{"start_s":597.56,"end_s":605.279,"text":"of 384 three terabytes of memory what","speaker":null,"is_sponsor":0},{"start_s":602.6,"end_s":611.04,"text":"across all eight oh each of the cables Jake removing","speaker":null,"is_sponsor":0},{"start_s":607.68,"end_s":613.16,"text":"right now is a PCIe by8 cable that feeds","speaker":null,"is_sponsor":0},{"start_s":611.04,"end_s":618.2,"text":"two of the drive bays in the front but the reason he's taking them out is that","speaker":null,"is_sponsor":0},{"start_s":615.079,"end_s":620.6,"text":"we can install our boot drives these are","speaker":null,"is_sponsor":0},{"start_s":618.2,"end_s":626.399,"text":"consumer grade each system is getting two Sab 512 gig gen 3 rocket drives and","speaker":null,"is_sponsor":0},{"start_s":624.76,"end_s":630.56,"text":"it's not because they're particularly special in any meaningful way they're","speaker":null,"is_sponsor":0},{"start_s":628.519,"end_s":635.8,"text":"not even that fast by modern standards but what they are is from our experience","speaker":null,"is_sponsor":0},{"start_s":632.839,"end_s":639.32,"text":"reliable enough and they are fast enough for what we're going to be doing which","speaker":null,"is_sponsor":0},{"start_s":636.88,"end_s":643.399,"text":"is just booting our operating system off of them movie Magic all of the other","speaker":null,"is_sponsor":0},{"start_s":641.24,"end_s":646.8,"text":"nodes are already built so what do you mean movie Magic super micro built them","speaker":null,"is_sponsor":0},{"start_s":645.12,"end_s":651.2,"text":"Oh I thought you buil them super micro builds them for you I took it apart okay","speaker":null,"is_sponsor":0},{"start_s":649.079,"end_s":655.279,"text":"fine I took that one apart no secrets left anymore yep no Intrigue no mystery","speaker":null,"is_sponsor":0},{"start_s":653.959,"end_s":661.44,"text":"you know what is still mysterious is inside of here I've actually never opened this before Oh okay let's have a","speaker":null,"is_sponsor":0},{"start_s":657.959,"end_s":662.92,"text":"look woo holy oh that's power supplies","speaker":null,"is_sponsor":0},{"start_s":661.44,"end_s":667.279,"text":"yeah this is so cool so the whole computer is cooled by four fans no way","speaker":null,"is_sponsor":0},{"start_s":665.839,"end_s":672.76,"text":"there's the two power supply fans and then these fans in their what do they call this like IO module I think is what","speaker":null,"is_sponsor":0},{"start_s":670.639,"end_s":676.6,"text":"they call it look at the blades on this thing counter rotating you're serious","speaker":null,"is_sponsor":0},{"start_s":674.72,"end_s":682.0,"text":"that's what you're looking at not this the most delicate of spaghet oh my God","speaker":null,"is_sponsor":0},{"start_s":679.56,"end_s":688.32,"text":"there's not even connectors every one of these wires is soldered directly to the","speaker":null,"is_sponsor":0},{"start_s":684.72,"end_s":689.959,"text":"back of the ocp 3.0 what yeah for","speaker":null,"is_sponsor":0},{"start_s":688.32,"end_s":697.839,"text":"storage we're installing ing two of kok's Speedy cd6 Gen 4 and vme drives in","speaker":null,"is_sponsor":0},{"start_s":694.04,"end_s":700.32,"text":"each node so we've got one that is 7","speaker":null,"is_sponsor":0},{"start_s":697.839,"end_s":704.24,"text":"tabt and another one that is 15 terabytes they're kind of placeholders","speaker":null,"is_sponsor":0},{"start_s":702.76,"end_s":710.079,"text":"for now and in the long term we're going to switch to Something in the neighborhood of about 4 15 tab drives","speaker":null,"is_sponsor":0},{"start_s":708.48,"end_s":715.92,"text":"per node but the drives we want to use are currently occupied by oh that","speaker":null,"is_sponsor":0},{"start_s":712.72,"end_s":717.88,"text":"project by a top secret pastry related","speaker":null,"is_sponsor":0},{"start_s":715.92,"end_s":722.24,"text":"project so that's going to have to wait the good news is that when those drives","speaker":null,"is_sponsor":0},{"start_s":719.8,"end_s":726.16,"text":"become available WCA supports live upgrading and downgrading so we can just","speaker":null,"is_sponsor":0},{"start_s":724.56,"end_s":730.8,"text":"pull these drives swap in the new ones pull swap pull swap pull swap as long as","speaker":null,"is_sponsor":0},{"start_s":728.079,"end_s":733.92,"text":"we uh don't do it all at once are we ready to fire these things up okay","speaker":null,"is_sponsor":0},{"start_s":732.32,"end_s":737.959,"text":"there's a lot going on here what is that is that a switch y hey look you can see","speaker":null,"is_sponsor":0},{"start_s":735.6,"end_s":745.56,"text":"the button now oh that's cool what you're hearing so far is just","speaker":null,"is_sponsor":0},{"start_s":741.68,"end_s":748.079,"text":"the NVIDIA SN 3700 32 Port 200 gig","speaker":null,"is_sponsor":0},{"start_s":745.56,"end_s":752.12,"text":"switch oh my God it even says melanox on the front I know maybe it's an old like","speaker":null,"is_sponsor":0},{"start_s":750.0,"end_s":755.8,"text":"review sample demo univ we got it with the $1 million PC and I'm pretty sure","speaker":null,"is_sponsor":0},{"start_s":754.24,"end_s":762.0,"text":"that that was already NVIDIA at that point can you hear that you hear it getting louder yeah","speaker":null,"is_sponsor":0},{"start_s":759.16,"end_s":766.88,"text":"who well that one's just excited to see this is the WKA dashboard maybe if I go","speaker":null,"is_sponsor":0},{"start_s":764.44,"end_s":772.36,"text":"over here cluster servers we can see all of our servers we have two drives per","speaker":null,"is_sponsor":0},{"start_s":770.88,"end_s":776.8,"text":"and then course this is a very interesting part of how wo works it's","speaker":null,"is_sponsor":0},{"start_s":774.6,"end_s":781.399,"text":"not like trass let's say where it just uses the whole CPU for whatever you're","speaker":null,"is_sponsor":0},{"start_s":778.56,"end_s":786.88,"text":"trying to do they dedicate and like fence off specific cores for specific","speaker":null,"is_sponsor":0},{"start_s":784.199,"end_s":793.32,"text":"tasks for instance each Drive gets a core so we've got two Drive containers","speaker":null,"is_sponsor":0},{"start_s":789.279,"end_s":796.6,"text":"that means two a full core per Drive","speaker":null,"is_sponsor":0},{"start_s":793.32,"end_s":799.32,"text":"yeah damn yeah you also have compute","speaker":null,"is_sponsor":0},{"start_s":796.6,"end_s":802.88,"text":"cores which do like the par calculation and intercluster communication and then","speaker":null,"is_sponsor":0},{"start_s":801.639,"end_s":807.68,"text":"there's front end which you don't necessarily always have frontend cores","speaker":null,"is_sponsor":0},{"start_s":805.0,"end_s":811.44,"text":"managed connecting to a file system so if you just had drives and Compu compute","speaker":null,"is_sponsor":0},{"start_s":809.839,"end_s":814.88,"text":"you wouldn't be able to access the files on this machine so you would have your","speaker":null,"is_sponsor":0},{"start_s":812.839,"end_s":819.839,"text":"backend servers right those would run drives and compute which is the cluster","speaker":null,"is_sponsor":0},{"start_s":817.68,"end_s":823.399,"text":"and then on your like GPU box you would run just the front end and that would","speaker":null,"is_sponsor":0},{"start_s":821.639,"end_s":828.079,"text":"allow the GPU box to connect to the backend cluster servers oh the back-end","speaker":null,"is_sponsor":0},{"start_s":826.399,"end_s":834.36,"text":"cluster servers don't need to run a front end unless you want to be able to","speaker":null,"is_sponsor":0},{"start_s":830.92,"end_s":836.839,"text":"access the files on that machine or from","speaker":null,"is_sponsor":0},{"start_s":834.36,"end_s":842.12,"text":"that machine which we want to cuz we're using SMB we're using it as a a file","speaker":null,"is_sponsor":0},{"start_s":839.56,"end_s":847.279,"text":"server stupid NZ for our stupid Windows machines yeah you can also have a","speaker":null,"is_sponsor":0},{"start_s":845.0,"end_s":850.36,"text":"dedicated front end machine yes so if you had like a 100 backend servers but","speaker":null,"is_sponsor":0},{"start_s":849.12,"end_s":855.399,"text":"then that's adding a single point of failure which is what we're trying to avoid you could have multiple of them","speaker":null,"is_sponsor":0},{"start_s":853.399,"end_s":860.68,"text":"okay you thought they thought of that yeah I set it up so every single machine","speaker":null,"is_sponsor":0},{"start_s":858.519,"end_s":866.48,"text":"in the cluster all eight of them are part of our SMB cluster which means it","speaker":null,"is_sponsor":0},{"start_s":863.6,"end_s":870.079,"text":"cannot go down realistically there are a ton of other file systems out there that","speaker":null,"is_sponsor":0},{"start_s":868.399,"end_s":875.279,"text":"you could use for something like this traz has their scale out setup for","speaker":null,"is_sponsor":0},{"start_s":872.279,"end_s":877.079,"text":"clustered ZFS which only requires three","speaker":null,"is_sponsor":0},{"start_s":875.279,"end_s":880.88,"text":"nodes and is something we'd be quite interested in trying out or if you're","speaker":null,"is_sponsor":0},{"start_s":879.12,"end_s":885.56,"text":"looking for object storage there's a million options but the main open-","speaker":null,"is_sponsor":0},{"start_s":882.92,"end_s":889.56,"text":"source one Min iio requires only four nodes though when we saw how nuts WCA","speaker":null,"is_sponsor":0},{"start_s":888.16,"end_s":897.24,"text":"was when we set up the million dooll server cluster I mean we had to try it","speaker":null,"is_sponsor":0},{"start_s":892.88,"end_s":901.6,"text":"out for ourselves and try it out we did","speaker":null,"is_sponsor":0},{"start_s":897.24,"end_s":904.079,"text":"so this is each not no holy","speaker":null,"is_sponsor":0},{"start_s":901.6,"end_s":909.48,"text":"sh look okay the crazy thing is look at the read latency now guys look look hold","speaker":null,"is_sponsor":0},{"start_s":906.079,"end_s":912.399,"text":"on hold on hold on at 70 gabt a second","speaker":null,"is_sponsor":0},{"start_s":909.48,"end_s":917.399,"text":"we've seen numbers like this before but we're talking with in some cases double","speaker":null,"is_sponsor":0},{"start_s":915.0,"end_s":922.56,"text":"the number of drives and no file system without a file system like raw to each","speaker":null,"is_sponsor":0},{"start_s":919.68,"end_s":929.04,"text":"drive this is with a file system with a file system over a network and we're","speaker":null,"is_sponsor":0},{"start_s":925.6,"end_s":930.36,"text":"only using 100 Gig ports like usually","speaker":null,"is_sponsor":0},{"start_s":929.04,"end_s":936.319,"text":"with a WCA setup like this you'd probably use 200 yeah cuz we oh my God","speaker":null,"is_sponsor":0},{"start_s":933.8,"end_s":941.0,"text":"we didn't know cuz we didn't even have networking as a factor last time all the","speaker":null,"is_sponsor":0},{"start_s":939.399,"end_s":945.759,"text":"drives were in one box I know this is networking too and the crazy part is","speaker":null,"is_sponsor":0},{"start_s":943.319,"end_s":952.319,"text":"we're not using RDMA this is like um some fancy uh what's it called dpdk I","speaker":null,"is_sponsor":0},{"start_s":948.759,"end_s":955.959,"text":"think is the library this is wild yeah","speaker":null,"is_sponsor":0},{"start_s":952.319,"end_s":959.399,"text":"look at that so read latency 131 microc","speaker":null,"is_sponsor":0},{"start_s":955.959,"end_s":962.04,"text":"seconds that's 4 million read iops with","speaker":null,"is_sponsor":0},{"start_s":959.399,"end_s":967.199,"text":"a latency of 1 millisecond average are are we able to keep using W FS like this","speaker":null,"is_sponsor":0},{"start_s":964.639,"end_s":972.04,"text":"is a trial okay this software is quite expensive this is unreal 4 million iops","speaker":null,"is_sponsor":0},{"start_s":969.92,"end_s":977.72,"text":"this is like it is unreal it's way more than we could possibly ever need but","speaker":null,"is_sponsor":0},{"start_s":975.24,"end_s":980.639,"text":"it's cool it's so cool don't they support tearing and everything oh yeah","speaker":null,"is_sponsor":0},{"start_s":979.6,"end_s":987.92,"text":"here I'll show you actually what that looks like this is on mother vault which I think right now has 400 Tippy bytes","speaker":null,"is_sponsor":0},{"start_s":985.279,"end_s":993.199,"text":"left so let's say Max Capacity is 400 terab now once we run out of the 100","speaker":null,"is_sponsor":0},{"start_s":991.279,"end_s":998.0,"text":"terab of SSD capacity which you can see here it'll just it'll tear I mean it","speaker":null,"is_sponsor":0},{"start_s":995.959,"end_s":1002.199,"text":"automatically tear anyways and you do need to make sure that your object store","speaker":null,"is_sponsor":0},{"start_s":999.88,"end_s":1006.16,"text":"is at least the same size as the flash or bigger because they're going to","speaker":null,"is_sponsor":0},{"start_s":1004.16,"end_s":1013.199,"text":"automatically tear everything to it that makes sense so in theory we","speaker":null,"is_sponsor":0},{"start_s":1008.6,"end_s":1015.319,"text":"move manually copy everything from Vault","speaker":null,"is_sponsor":0},{"start_s":1013.199,"end_s":1020.319,"text":"one time to wo one time because it stores in like 64 megabyte chunks and","speaker":null,"is_sponsor":0},{"start_s":1018.56,"end_s":1023.92,"text":"then it just stays there forever stays there forever and then we just have one","speaker":null,"is_sponsor":0},{"start_s":1021.72,"end_s":1028.12,"text":"network share and when something needs to get vaed you just you just move it","speaker":null,"is_sponsor":0},{"start_s":1026.28,"end_s":1031.48,"text":"from allow it to Decay yeah you would probably move it from pending projects","speaker":null,"is_sponsor":0},{"start_s":1029.72,"end_s":1035.439,"text":"to like done or something like that we make a folder for done yeah sure um and","speaker":null,"is_sponsor":0},{"start_s":1033.64,"end_s":1040.24,"text":"then it will just do it automatically wow or if it's a video that like","speaker":null,"is_sponsor":0},{"start_s":1038.16,"end_s":1043.88,"text":"somebody was working on and then you know it's been on hold for 3 months and","speaker":null,"is_sponsor":0},{"start_s":1041.839,"end_s":1047.6,"text":"we shot you know a ter of footage it will just and then when we're ready to","speaker":null,"is_sponsor":0},{"start_s":1045.199,"end_s":1051.84,"text":"work on it it'll promote it back up holy we K net boot off of this","speaker":null,"is_sponsor":0},{"start_s":1049.48,"end_s":1056.799,"text":"followup video yeah I mean why not it's so fast you literally could not we we","speaker":null,"is_sponsor":0},{"start_s":1055.08,"end_s":1061.64,"text":"couldn't saturate this now a lot of you at this point must be thinking gosh","speaker":null,"is_sponsor":0},{"start_s":1059.12,"end_s":1068.16,"text":"Mister that's an awful lot of computers for high availability couldn't you do","speaker":null,"is_sponsor":0},{"start_s":1064.16,"end_s":1071.4,"text":"this with two and you're not that far","speaker":null,"is_sponsor":0},{"start_s":1068.16,"end_s":1073.2,"text":"off the old school high availability net","speaker":null,"is_sponsor":0},{"start_s":1071.4,"end_s":1079.76,"text":"app storage appliances like that one we looked at recently did have just two","speaker":null,"is_sponsor":0},{"start_s":1076.48,"end_s":1083.4,"text":"machines but those were both connected","speaker":null,"is_sponsor":0},{"start_s":1079.76,"end_s":1085.96,"text":"to the same storage drives if each","speaker":null,"is_sponsor":0},{"start_s":1083.4,"end_s":1090.2,"text":"system has its own drives when things can get out of sync like let's say if","speaker":null,"is_sponsor":0},{"start_s":1088.0,"end_s":1095.12,"text":"one machine has downtime you can run into a situation where each system","speaker":null,"is_sponsor":0},{"start_s":1092.64,"end_s":1101.6,"text":"believes with all the conviction in its heart that it has the correct data and","speaker":null,"is_sponsor":0},{"start_s":1098.44,"end_s":1104.159,"text":"then if all you have is two how will","speaker":null,"is_sponsor":0},{"start_s":1101.6,"end_s":1109.24,"text":"they decide who's right this is typically referred to as split brain and","speaker":null,"is_sponsor":0},{"start_s":1107.2,"end_s":1115.12,"text":"that's why the majority of High availability systems have at bare","speaker":null,"is_sponsor":0},{"start_s":1111.72,"end_s":1117.2,"text":"minimum three servers this allows the","speaker":null,"is_sponsor":0},{"start_s":1115.12,"end_s":1124.159,"text":"third system to be a tie breaker of sorts in the case of a disagreement now","speaker":null,"is_sponsor":0},{"start_s":1120.36,"end_s":1126.24,"text":"in our case WCA that stupid Ultra fast","speaker":null,"is_sponsor":0},{"start_s":1124.159,"end_s":1130.679,"text":"file system that we're using which unlike anything that we've used before","speaker":null,"is_sponsor":0},{"start_s":1128.32,"end_s":1136.919,"text":"has been built specifically for NVMe drives not hard drives well it requires","speaker":null,"is_sponsor":0},{"start_s":1133.88,"end_s":1139.76,"text":"a minimum of six nodes with a","speaker":null,"is_sponsor":0},{"start_s":1136.919,"end_s":1144.919,"text":"recommendation of eight but running WKA can still be an advantage video editing","speaker":null,"is_sponsor":0},{"start_s":1142.159,"end_s":1149.52,"text":"with Adobe Premiere like we use is very latency sensitive and even a small delay","speaker":null,"is_sponsor":0},{"start_s":1147.88,"end_s":1155.24,"text":"when going to access a clip can be enough to make the software crash so any","speaker":null,"is_sponsor":0},{"start_s":1152.039,"end_s":1157.88,"text":"Improvement there is huge not to mention","speaker":null,"is_sponsor":0},{"start_s":1155.24,"end_s":1165.84,"text":"that a pair of these Grand twins speced out to the max with 128 car epic Berg","speaker":null,"is_sponsor":0},{"start_s":1161.159,"end_s":1169.08,"text":"CPUs would get you just four rack units","speaker":null,"is_sponsor":0},{"start_s":1165.84,"end_s":1173.2,"text":"with 1,000 CPU cores actually actually a","speaker":null,"is_sponsor":0},{"start_s":1169.08,"end_s":1176.559,"text":"little more 24 terab of ddr5 and up to 3","speaker":null,"is_sponsor":0},{"start_s":1173.2,"end_s":1178.799,"text":"pedabytes of ndme storage I mean h that","speaker":null,"is_sponsor":0},{"start_s":1176.559,"end_s":1183.6,"text":"makes our setup seem downright reasonable now the average W customers","speaker":null,"is_sponsor":0},{"start_s":1181.52,"end_s":1189.28,"text":"are going to be a little more demanding than us visual effect Studios AI","speaker":null,"is_sponsor":0},{"start_s":1186.24,"end_s":1191.679,"text":"developers genomics Labs all the folks","speaker":null,"is_sponsor":0},{"start_s":1189.28,"end_s":1194.96,"text":"out there that need Stupid Fast low latency storage and WCA showed us","speaker":null,"is_sponsor":0},{"start_s":1193.36,"end_s":1200.24,"text":"screenshots of clusters that were reading in excess of 1 terte per second","speaker":null,"is_sponsor":0},{"start_s":1198.0,"end_s":1204.76,"text":"consistently obviously that was a bigger cluster but it shows you what can be","speaker":null,"is_sponsor":0},{"start_s":1202.32,"end_s":1212.08,"text":"achieved with this kind of Hardware running on I mean what used to be the","speaker":null,"is_sponsor":0},{"start_s":1207.6,"end_s":1214.679,"text":"crappier option software raid man I feel","speaker":null,"is_sponsor":0},{"start_s":1212.08,"end_s":1220.2,"text":"bad even calling it that these days I had a interesting idea with the super","speaker":null,"is_sponsor":0},{"start_s":1217.919,"end_s":1225.6,"text":"micro folks so you know how we have like two pedabytes of 13 years worth of","speaker":null,"is_sponsor":0},{"start_s":1222.84,"end_s":1231.2,"text":"footage thousands and thousands of hours of footage thousands it's really cool","speaker":null,"is_sponsor":0},{"start_s":1228.28,"end_s":1236.76,"text":"that we have it but it's really hard to use unless you just happen to know what","speaker":null,"is_sponsor":0},{"start_s":1234.0,"end_s":1241.76,"text":"video the thing you were looking for is in well what if you could just like","speaker":null,"is_sponsor":0},{"start_s":1238.76,"end_s":1243.24,"text":"search for something lonus Sebastian I","speaker":null,"is_sponsor":0},{"start_s":1241.76,"end_s":1247.919,"text":"want every clip with lonus Sebastian in it wow bam look at that shot up and","speaker":null,"is_sponsor":0},{"start_s":1246.159,"end_s":1251.44,"text":"let's say you know there's this one that's uh detected that it's you","speaker":null,"is_sponsor":0},{"start_s":1249.559,"end_s":1258.72,"text":"throughout the entire clip yeah you're in a chair so you could search for clips","speaker":null,"is_sponsor":0},{"start_s":1254.559,"end_s":1260.0,"text":"of lonus sitting down with a keyboard","speaker":null,"is_sponsor":0},{"start_s":1258.72,"end_s":1265.36,"text":"yeah like we're going to be able to actually find stuff yeah right now there","speaker":null,"is_sponsor":0},{"start_s":1262.799,"end_s":1269.64,"text":"is a a finite amount of objects that are trained I mean chihuahua let me scroll","speaker":null,"is_sponsor":0},{"start_s":1267.799,"end_s":1273.12,"text":"through this it's a lot eventually you'll be able to train it and tell it","speaker":null,"is_sponsor":0},{"start_s":1271.559,"end_s":1277.679,"text":"hey this is what a computer fan looks like or this is what an SSD looks like","speaker":null,"is_sponsor":0},{"start_s":1275.08,"end_s":1283.52,"text":"oh my God that is so cool so wait is this running on these extra CPU cores or","speaker":null,"is_sponsor":0},{"start_s":1281.039,"end_s":1289.24,"text":"okay no not right now faces and logos are running on CPU yeah objects OCR and","speaker":null,"is_sponsor":0},{"start_s":1286.559,"end_s":1292.64,"text":"scenes run on GPU got it but they're not running on any of those machines they're","speaker":null,"is_sponsor":0},{"start_s":1290.72,"end_s":1296.679,"text":"running on a GPU workstation that super micro sent that's sitting at my desk um","speaker":null,"is_sponsor":0},{"start_s":1294.44,"end_s":1302.52,"text":"it was Heavy anyways what is happening on that new server is proxies because if","speaker":null,"is_sponsor":0},{"start_s":1299.6,"end_s":1307.2,"text":"we were to analyze the original Clips oh AAL formatting is a huge problem when","speaker":null,"is_sponsor":0},{"start_s":1305.279,"end_s":1311.2,"text":"you go into an AI model it might not necessarily support the Kodak that","speaker":null,"is_sponsor":0},{"start_s":1308.679,"end_s":1316.08,"text":"you're filming in sure but also Clips are like hundreds of megabytes a second","speaker":null,"is_sponsor":0},{"start_s":1313.84,"end_s":1319.52,"text":"potentially that would take forever so instead it generates proxies of","speaker":null,"is_sponsor":0},{"start_s":1318.0,"end_s":1324.36,"text":"everything first first which we're dumping to that new server and then we","speaker":null,"is_sponsor":0},{"start_s":1322.279,"end_s":1331.4,"text":"can take advantage of the Lightning Fast storage yeah you can we have 2.6 massive","speaker":null,"is_sponsor":0},{"start_s":1328.799,"end_s":1338.2,"text":"compute and we can basically create like a proxy map of what everything is in the","speaker":null,"is_sponsor":0},{"start_s":1335.36,"end_s":1343.12,"text":"main archive right that is so cool so far I've generated 2.6 terab of proxies","speaker":null,"is_sponsor":0},{"start_s":1340.88,"end_s":1347.76,"text":"which might not sound like a lot but they're only 5 megabit so it's actually","speaker":null,"is_sponsor":0},{"start_s":1345.96,"end_s":1354.88,"text":"like a lot this is going to be a flipping game Cher News","speaker":null,"is_sponsor":0},{"start_s":1350.96,"end_s":1357.679,"text":"sports can you imagine your CNN you want","speaker":null,"is_sponsor":0},{"start_s":1354.88,"end_s":1362.279,"text":"that person wearing a red tie yeah but right now we've done 25,000 so 2.6","speaker":null,"is_sponsor":0},{"start_s":1359.84,"end_s":1366.799,"text":"terabyt is 25,000 Pro okay well let's try and find something oh hold on once","speaker":null,"is_sponsor":0},{"start_s":1364.679,"end_s":1372.6,"text":"you've generated a proxy you have to then analyze it right ah so the analysis","speaker":null,"is_sponsor":0},{"start_s":1370.4,"end_s":1376.76,"text":"is not done no not even close I've analyzed 22 Clips okay everything with","speaker":null,"is_sponsor":0},{"start_s":1374.88,"end_s":1381.159,"text":"Elijah Elijah and this is the every clip that","speaker":null,"is_sponsor":0},{"start_s":1379.12,"end_s":1387.36,"text":"Elijah's in and you can even see this is so cool this is the actual ma'am as they","speaker":null,"is_sponsor":0},{"start_s":1384.039,"end_s":1389.48,"text":"call it media asset manager the axle AI","speaker":null,"is_sponsor":0},{"start_s":1387.36,"end_s":1392.96,"text":"guys built this before it was like AI as far as I'm aware back when you would","speaker":null,"is_sponsor":0},{"start_s":1391.24,"end_s":1398.32,"text":"have had to make comments like this manually now it's just AI so all of the","speaker":null,"is_sponsor":0},{"start_s":1396.2,"end_s":1403.76,"text":"data is in here now and we can see here's Adam and Elijah oh that's so cool","speaker":null,"is_sponsor":0},{"start_s":1401.799,"end_s":1408.559,"text":"here's all the different objects chair flower pot microphone oh let me show you","speaker":null,"is_sponsor":0},{"start_s":1407.12,"end_s":1412.84,"text":"the scene understanding thing cuz that is so cool this is like brand new thing","speaker":null,"is_sponsor":0},{"start_s":1410.559,"end_s":1418.72,"text":"they barely even worked it in but it basic it basically takes a snapshot","speaker":null,"is_sponsor":0},{"start_s":1415.76,"end_s":1423.159,"text":"every seconds two men are working on a project in a room there is a speaker","speaker":null,"is_sponsor":0},{"start_s":1421.24,"end_s":1427.279,"text":"stereo equipment there's a faucet there's a tripod there's the tripod some","speaker":null,"is_sponsor":0},{"start_s":1425.2,"end_s":1431.919,"text":"of these are a little less accurate two men are working on a robot in a room it","speaker":null,"is_sponsor":0},{"start_s":1429.72,"end_s":1435.6,"text":"kind of looks like a robot you I mean yeah sure two men are in a workshop","speaker":null,"is_sponsor":0},{"start_s":1433.64,"end_s":1439.84,"text":"looking at a laptop computer looking at a machine there is person Alex Clark so","speaker":null,"is_sponsor":0},{"start_s":1438.12,"end_s":1446.52,"text":"this is just running right now in real time like more stuff is getting processed as see here processing logos 9","speaker":null,"is_sponsor":0},{"start_s":1444.64,"end_s":1449.52,"text":"there it is processing logos and faces it's going to take a while yeah it's","speaker":null,"is_sponsor":0},{"start_s":1448.32,"end_s":1455.279,"text":"going to take forever they're still working on making it function on","speaker":null,"is_sponsor":0},{"start_s":1451.799,"end_s":1457.44,"text":"multiple gpus so once we can get it","speaker":null,"is_sponsor":0},{"start_s":1455.279,"end_s":1462.08,"text":"running on like four gpus say one GPU is doing face detection one's doing scene","speaker":null,"is_sponsor":0},{"start_s":1459.919,"end_s":1465.2,"text":"analysis one's doing object detection or something like that we'll be able to go","speaker":null,"is_sponsor":0},{"start_s":1463.52,"end_s":1470.36,"text":"a lot faster but right now it's just one GPU got it but this is so cool all","speaker":null,"is_sponsor":0},{"start_s":1467.919,"end_s":1473.679,"text":"that's left is to deploy it lonus had to run away to do some other stuff so I've","speaker":null,"is_sponsor":0},{"start_s":1471.72,"end_s":1478.2,"text":"hired some backup Cavalry Sean our infrastructure administrator except","speaker":null,"is_sponsor":0},{"start_s":1476.159,"end_s":1482.88,"text":"we've run into a bit of a problem lonus and me and our Infinite Wisdom while we","speaker":null,"is_sponsor":0},{"start_s":1480.08,"end_s":1486.72,"text":"were making this rack so much better ran a bunch of cables right where we need to","speaker":null,"is_sponsor":0},{"start_s":1485.039,"end_s":1491.399,"text":"put the server did we just start unplugging no yeah how are we even going","speaker":null,"is_sponsor":0},{"start_s":1489.159,"end_s":1494.52,"text":"to do this we have to like part the seas exactly I started to try to move some of","speaker":null,"is_sponsor":0},{"start_s":1493.08,"end_s":1499.159,"text":"the cables out of the way but they're all twisted together so hopefully the","speaker":null,"is_sponsor":0},{"start_s":1496.919,"end_s":1504.6,"text":"LTT Cable Management thing which you can finally get at ltp store.com will save","speaker":null,"is_sponsor":1},{"start_s":1501.72,"end_s":1511.6,"text":"us beautiful cable managed we can slide a server in there now I hope you're in","speaker":null,"is_sponsor":1},{"start_s":1508.52,"end_s":1514.88,"text":"yeah it's on ow ow ow ow ow ow ow okay","speaker":null,"is_sponsor":0},{"start_s":1511.6,"end_s":1518.12,"text":"you're good just go that wasn't so bad","speaker":null,"is_sponsor":0},{"start_s":1514.88,"end_s":1518.12,"text":"like made for it","speaker":null,"is_sponsor":0},{"start_s":1518.32,"end_s":1524.84,"text":"next hey we're in now we just have to","speaker":null,"is_sponsor":0},{"start_s":1522.039,"end_s":1528.0,"text":"run a million cables uhoh do you notice anything different well it's loud most","speaker":null,"is_sponsor":0},{"start_s":1526.48,"end_s":1531.279,"text":"of that's actually just the vent is on one of the air conditioners is broken","speaker":null,"is_sponsor":0},{"start_s":1529.799,"end_s":1534.84,"text":"again but do you notice anything different I mean the sticker's here that","speaker":null,"is_sponsor":0},{"start_s":1533.36,"end_s":1539.6,"text":"that sticker's been there for years seriously you haven't noticed anything else well you guys uh screwed something","speaker":null,"is_sponsor":0},{"start_s":1537.64,"end_s":1544.64,"text":"onto the oh did you put sauna Pan behind it yeah but I thought this is supposed","speaker":null,"is_sponsor":0},{"start_s":1541.559,"end_s":1546.799,"text":"to be a vented door my original plan was","speaker":null,"is_sponsor":0},{"start_s":1544.64,"end_s":1552.159,"text":"to get rid of the vent that you put in but that vent was there as a backup in","speaker":null,"is_sponsor":0},{"start_s":1549.0,"end_s":1553.679,"text":"case the HVAC ever failed so that fan is","speaker":null,"is_sponsor":0},{"start_s":1552.159,"end_s":1557.48,"text":"the exhaust and that's the intake you see all the gaps F God there gaps but do","speaker":null,"is_sponsor":0},{"start_s":1555.799,"end_s":1561.559,"text":"you notice the sound difference yeah it's a big difference it's huge but that","speaker":null,"is_sponsor":0},{"start_s":1559.36,"end_s":1565.039,"text":"server is so loud we basically ended up where we","speaker":null,"is_sponsor":0},{"start_s":1563.24,"end_s":1571.159,"text":"started yeah but that's okay I was just trying to normalize I just mean I didn't","speaker":null,"is_sponsor":0},{"start_s":1566.88,"end_s":1574.799,"text":"make it worse it's not that okay look at","speaker":null,"is_sponsor":0},{"start_s":1571.159,"end_s":1577.679,"text":"that woo cute right God that's a lot of","speaker":null,"is_sponsor":0},{"start_s":1574.799,"end_s":1582.64,"text":"metal if all goes to plan we could get rid of this and this and just have these","speaker":null,"is_sponsor":0},{"start_s":1580.2,"end_s":1587.0,"text":"so no more additional rack taken up which is nice","speaker":null,"is_sponsor":0},{"start_s":1587.0,"end_s":1592.44,"text":"wow it should sustain two entire servers","speaker":null,"is_sponsor":0},{"start_s":1590.679,"end_s":1596.76,"text":"dropping out without anyone even noticing do you really want to test it","speaker":null,"is_sponsor":0},{"start_s":1594.919,"end_s":1602.559,"text":"right now I like I haven't tried that all right here we go what could go","speaker":null,"is_sponsor":0},{"start_s":1599.039,"end_s":1604.72,"text":"wrong u i mean a lot the fact that all","speaker":null,"is_sponsor":0},{"start_s":1602.559,"end_s":1609.159,"text":"the fans just like turned down a bit is a little scary let's go see if anyone","speaker":null,"is_sponsor":0},{"start_s":1606.84,"end_s":1613.76,"text":"noticed oh hi Mark hi I'm holding your file server how's your edit going uh","speaker":null,"is_sponsor":0},{"start_s":1611.48,"end_s":1617.799,"text":"what huh is it working it's working is this on Wi-Fi hey Emily hey how's your","speaker":null,"is_sponsor":0},{"start_s":1616.159,"end_s":1623.12,"text":"edit going I'm holding your server that's cool is it working are you sure yeah Hoffman","speaker":null,"is_sponsor":0},{"start_s":1621.76,"end_s":1626.84,"text":"what's up how's your edit going this is your server right here it's amazing look","speaker":null,"is_sponsor":0},{"start_s":1624.88,"end_s":1629.32,"text":"feel it it's still warm wow yeah it's it's still warm how well how's it","speaker":null,"is_sponsor":0},{"start_s":1627.919,"end_s":1634.76,"text":"working it's great you know I'm editing the video that we're shooting you are yeah uh we're going to pull another one","speaker":null,"is_sponsor":0},{"start_s":1632.159,"end_s":1637.88,"text":"wait no l you forgot one yeah here here's another here's another one of","speaker":null,"is_sponsor":0},{"start_s":1636.08,"end_s":1643.36,"text":"your servers is it working it's great though huhuh for reference you're not","speaker":null,"is_sponsor":0},{"start_s":1640.799,"end_s":1648.52,"text":"supposed to do this you should power off the system first but we're just trying","speaker":null,"is_sponsor":0},{"start_s":1645.36,"end_s":1651.44,"text":"to simulate it failing yeah a terrible","speaker":null,"is_sponsor":0},{"start_s":1648.52,"end_s":1655.279,"text":"catastrophic failure I can't believe how smoothly it handled that see all the","speaker":null,"is_sponsor":0},{"start_s":1653.159,"end_s":1658.48,"text":"lights they never stopped blinking big thanks to Super Micro for these awesome","speaker":null,"is_sponsor":0},{"start_s":1656.76,"end_s":1662.6,"text":"servers thanks to WCA for making this crazy software thanks to axle for the","speaker":null,"is_sponsor":1},{"start_s":1660.64,"end_s":1665.96,"text":"awesome AI detection if you like this video maybe check out the video series","speaker":null,"is_sponsor":1},{"start_s":1664.48,"end_s":1669.84,"text":"of us building our nearly three pedabytes of archival storage which we","speaker":null,"is_sponsor":0},{"start_s":1667.96,"end_s":1674.279,"text":"call the mother Vault that thing is awesome and we showed it to you and it's","speaker":null,"is_sponsor":0},{"start_s":1671.919,"end_s":1678.08,"text":"faster now oh and thanks to you for being an awesome viewer","speaker":null,"is_sponsor":0}],"full_text":"when you make as many videos as we do you need a lot of fast reliable storage and our main editing server wanic has checked all of those boxes for years it's a great little server it's built out of high quality components and it even looks cool but as our team is grown we've reached the point where even a minute one single minute of downtime costs over $50 and that's just in payroll now practically speaking the way to mitigate that is by adding redundant now our drives are already redundant we've got 20 drives in there with data striping but the problem is they all sit in one single server I'm sure you can see where this is going it's been over a year in the making but it's finally here wanic final form and I'm calling it wanic 10 because it's the last W ever avability W you this like 10 times nobody even knows what high availability means it means it's lus just go ahead unplug one do it go for it well okay I should probably tell you the stakes before you do that each of these two grand twin boxes has four entire servers inside of them that were provided by super micro who sponsored this whole thing and they're set up with WCA a redundant NVMe first file system in this config it should sustain two entire servers dropping out without anyone even noticing except that we moved the entire team onto it last night without telling anyone and it's the middle of the work day with a ton of high priority videos in progress do you really want to test it right now I like I haven't tried that all right here we go okay what could go wrong I mean a lot naturally a huge part of a project like this is the software the stuff that's going to handle Distributing all of ourish terabytes of video projects Word documents and Linux isos to the multiple machines that we just showed you but we can't install any software until we have some Hardware so why don't we start there meet the super micro Grand twin A+ server as- 2115 gt- hntr despite its sort of ordinary looking appearance and unexciting sounding name it is anything but ordinary and it is very exciting because inside this 2u is four independent Compu computers but for what we're doing four nodes please we want eight inside each of these is a completely independent motherboard 384 gigs of memory an AMD epic Genoa processor with 64 cores dual m.2 slots for redundant boot drives six PCIe Gen 5 2 and 1/2 in NVMe slots up front and we've got IO in the rear now this bit here could be a little confusing at first glance but that is because not only do we have USB but we have two full gen 5x6 PCIe connections back here along with display output and power for the entire server this whole thing slides into the chassis which holds a really cool modular backplane assembly that we'll take a look at in a minute and then passes through thank you Jake ah to the back at the server where you've got a Management Port a single USB port for each server nope it's two and they're shared what the I was about to ask cuz we've also got a single VGA you see the button for two servers there no way this button toggles yeah and okay before we talk about that a little bit more look at these power supplies each of these is 2200 Watts 80 plus typ tianium which sounds like a lot but when you're potentially handling four 400 wat epic Genoa CPUs along with a bunch of RAM up to 24 NVMe drives and eight network cards well it seems downright reasonable doesn't it is it 24 drives can't be 6 yes 6 * 4 is 24 and of course that's just one of them we've got two of those and that means that in the event that one of these dies the system should be able to continue to operate uninterrupted which is a big part of the high availability goal that we have for this deployment speaking of high availability let's move on to our network cards each of those PCIe gen 5x6 slots I showed you guys before terminates in one of these ocp 3.0 small form factor mezzanine slots and what we're putting in them is these connectx 6 200 gbit cards from melanox excuse me from NVIDIA that okay these are the older Gen 4 ones so they're going to be limited by the slot speed of around 250 gabit per second but if we had newer cards that means that each of these nodes could do 200 plus another 200 400 up to 800 gigabit which would of course be a complete waste for us a because our workload can't take advantage of it and B because our switch is only 100 gbit sorry of course the two ports are still helpful we do have redundant switches except there's kind of a problem here that's still a single point of failure in a perfect world we would have two single port Nicks so if a Nick were to die it would still be okay but because we have so many nodes we're not really worried about an individual node you know they could have one boot drive and it die or one Nick and it die we still have an extra backup how many nines do you want I mean I don't know like one would would be good 9% which Jokes Aside is a really good point if we were architecting this properly there are so many more considerations that we would need to make like the power coming into the rack would have to come from two independent backed up sources the connectivity to our clients would have to be redundant as well the connectivity between all of the systems would have to be architected in such a way that no matter what fails everything will stay up and realistically for us we're not going to get that deep into it because our goal is better than we had before which was a single machine with its own built-in redundancies but other than that nothing now at least we should be able to lose a full machine out of these eight we can restart one of our core switches totally fine two machines out of these eight and we can still be limping along I mean limping is a bit of a stretch it's going to be very fast now normally if you buy a super micro machine they're going to pre-build it for you they're going to validate it for you you can even have them pre-build an entire Rack or racks of these things and then validate your application on it before it ships to you in fact we've got a whole video that we did about that that was sponsored by super micro a little while back of course this is LT my friends so we will be assembling this one ourselves do you like that spin of the screwdriver above the server don't worry I won't miss I'll never miss see I could do this a hundred times and I would never miss why no it's fine it's good it's okay we have seven more any who for our CPU we've gone with an epic Genova 9534 this is a 64 core 128 thread monster of a CPU it'll do 3.7 GHz Max boost it has A4 Gigabyte of level three cache a 300 wat TDP it supports ddr5 memory up to 12 channels and it supports a whopping 128 Lanes of PCIe Gen 5 originally we were intending to go with 32 core chips but they were out of stock so free upgrade lucky us compared to previous generation AMD epic CPUs dooa is a big step up in terms of IO performance which makes it perfect for this application and in the long term I mean if we've got all the extra CPU cores and a whole bunch of RAM anyway why run WCA on the bare metal when we could install prox Mox and then use the other cores for I don't know High availability Plex server yeah Linux isos more realistically it would be something like active directory yeah which we don't really want to do right now because if you run active directory on one server and it goes down you're going to have a really really bad time but if you run it on a bunch of servers yeah it's good great so normally server CPU coolers would come with their own thermal paste pre-applied but since we're doing this ourselves and uh if you look carefully it's not the first time that it's been installed we are going to be using okay thank you for that a piece of Honeywell PTM 7950 this stuff is freaking awesome it has great thermal transfer properties and it can handle varying temperatures like seriously I don't remember many not even just varying but like a lot of huge cycles for a very very long time now available LTD store.com is that big enough does that cover all of the ccds and cxs oh there's a second piece of PL am I stupid is there a second piece of plastic no there isn't should I put one in the fridge no no no it's totally fine I've done this like a bunch of times yeah oh she's Min look at that see all right easy I would recommend putting it in the fridge before you use it all right to ensure we're making the absolute most of our CPU especially in this High throughput storage workload we're going to be populating all 12 of our memory Channels with 32 gig dims of ddr5 ECC running at 4800 megga transitors per second that's a total of 384 three terabytes of memory what across all eight oh each of the cables Jake removing right now is a PCIe by8 cable that feeds two of the drive bays in the front but the reason he's taking them out is that we can install our boot drives these are consumer grade each system is getting two Sab 512 gig gen 3 rocket drives and it's not because they're particularly special in any meaningful way they're not even that fast by modern standards but what they are is from our experience reliable enough and they are fast enough for what we're going to be doing which is just booting our operating system off of them movie Magic all of the other nodes are already built so what do you mean movie Magic super micro built them Oh I thought you buil them super micro builds them for you I took it apart okay fine I took that one apart no secrets left anymore yep no Intrigue no mystery you know what is still mysterious is inside of here I've actually never opened this before Oh okay let's have a look woo holy oh that's power supplies yeah this is so cool so the whole computer is cooled by four fans no way there's the two power supply fans and then these fans in their what do they call this like IO module I think is what they call it look at the blades on this thing counter rotating you're serious that's what you're looking at not this the most delicate of spaghet oh my God there's not even connectors every one of these wires is soldered directly to the back of the ocp 3.0 what yeah for storage we're installing ing two of kok's Speedy cd6 Gen 4 and vme drives in each node so we've got one that is 7 tabt and another one that is 15 terabytes they're kind of placeholders for now and in the long term we're going to switch to Something in the neighborhood of about 4 15 tab drives per node but the drives we want to use are currently occupied by oh that project by a top secret pastry related project so that's going to have to wait the good news is that when those drives become available WCA supports live upgrading and downgrading so we can just pull these drives swap in the new ones pull swap pull swap pull swap as long as we uh don't do it all at once are we ready to fire these things up okay there's a lot going on here what is that is that a switch y hey look you can see the button now oh that's cool what you're hearing so far is just the NVIDIA SN 3700 32 Port 200 gig switch oh my God it even says melanox on the front I know maybe it's an old like review sample demo univ we got it with the $1 million PC and I'm pretty sure that that was already NVIDIA at that point can you hear that you hear it getting louder yeah who well that one's just excited to see this is the WKA dashboard maybe if I go over here cluster servers we can see all of our servers we have two drives per and then course this is a very interesting part of how wo works it's not like trass let's say where it just uses the whole CPU for whatever you're trying to do they dedicate and like fence off specific cores for specific tasks for instance each Drive gets a core so we've got two Drive containers that means two a full core per Drive yeah damn yeah you also have compute cores which do like the par calculation and intercluster communication and then there's front end which you don't necessarily always have frontend cores managed connecting to a file system so if you just had drives and Compu compute you wouldn't be able to access the files on this machine so you would have your backend servers right those would run drives and compute which is the cluster and then on your like GPU box you would run just the front end and that would allow the GPU box to connect to the backend cluster servers oh the back-end cluster servers don't need to run a front end unless you want to be able to access the files on that machine or from that machine which we want to cuz we're using SMB we're using it as a a file server stupid NZ for our stupid Windows machines yeah you can also have a dedicated front end machine yes so if you had like a 100 backend servers but then that's adding a single point of failure which is what we're trying to avoid you could have multiple of them okay you thought they thought of that yeah I set it up so every single machine in the cluster all eight of them are part of our SMB cluster which means it cannot go down realistically there are a ton of other file systems out there that you could use for something like this traz has their scale out setup for clustered ZFS which only requires three nodes and is something we'd be quite interested in trying out or if you're looking for object storage there's a million options but the main open- source one Min iio requires only four nodes though when we saw how nuts WCA was when we set up the million dooll server cluster I mean we had to try it out for ourselves and try it out we did so this is each not no holy sh look okay the crazy thing is look at the read latency now guys look look hold on hold on hold on at 70 gabt a second we've seen numbers like this before but we're talking with in some cases double the number of drives and no file system without a file system like raw to each drive this is with a file system with a file system over a network and we're only using 100 Gig ports like usually with a WCA setup like this you'd probably use 200 yeah cuz we oh my God we didn't know cuz we didn't even have networking as a factor last time all the drives were in one box I know this is networking too and the crazy part is we're not using RDMA this is like um some fancy uh what's it called dpdk I think is the library this is wild yeah look at that so read latency 131 microc seconds that's 4 million read iops with a latency of 1 millisecond average are are we able to keep using W FS like this is a trial okay this software is quite expensive this is unreal 4 million iops this is like it is unreal it's way more than we could possibly ever need but it's cool it's so cool don't they support tearing and everything oh yeah here I'll show you actually what that looks like this is on mother vault which I think right now has 400 Tippy bytes left so let's say Max Capacity is 400 terab now once we run out of the 100 terab of SSD capacity which you can see here it'll just it'll tear I mean it automatically tear anyways and you do need to make sure that your object store is at least the same size as the flash or bigger because they're going to automatically tear everything to it that makes sense so in theory we move manually copy everything from Vault one time to wo one time because it stores in like 64 megabyte chunks and then it just stays there forever stays there forever and then we just have one network share and when something needs to get vaed you just you just move it from allow it to Decay yeah you would probably move it from pending projects to like done or something like that we make a folder for done yeah sure um and then it will just do it automatically wow or if it's a video that like somebody was working on and then you know it's been on hold for 3 months and we shot you know a ter of footage it will just and then when we're ready to work on it it'll promote it back up holy we K net boot off of this followup video yeah I mean why not it's so fast you literally could not we we couldn't saturate this now a lot of you at this point must be thinking gosh Mister that's an awful lot of computers for high availability couldn't you do this with two and you're not that far off the old school high availability net app storage appliances like that one we looked at recently did have just two machines but those were both connected to the same storage drives if each system has its own drives when things can get out of sync like let's say if one machine has downtime you can run into a situation where each system believes with all the conviction in its heart that it has the correct data and then if all you have is two how will they decide who's right this is typically referred to as split brain and that's why the majority of High availability systems have at bare minimum three servers this allows the third system to be a tie breaker of sorts in the case of a disagreement now in our case WCA that stupid Ultra fast file system that we're using which unlike anything that we've used before has been built specifically for NVMe drives not hard drives well it requires a minimum of six nodes with a recommendation of eight but running WKA can still be an advantage video editing with Adobe Premiere like we use is very latency sensitive and even a small delay when going to access a clip can be enough to make the software crash so any Improvement there is huge not to mention that a pair of these Grand twins speced out to the max with 128 car epic Berg CPUs would get you just four rack units with 1,000 CPU cores actually actually a little more 24 terab of ddr5 and up to 3 pedabytes of ndme storage I mean h that makes our setup seem downright reasonable now the average W customers are going to be a little more demanding than us visual effect Studios AI developers genomics Labs all the folks out there that need Stupid Fast low latency storage and WCA showed us screenshots of clusters that were reading in excess of 1 terte per second consistently obviously that was a bigger cluster but it shows you what can be achieved with this kind of Hardware running on I mean what used to be the crappier option software raid man I feel bad even calling it that these days I had a interesting idea with the super micro folks so you know how we have like two pedabytes of 13 years worth of footage thousands and thousands of hours of footage thousands it's really cool that we have it but it's really hard to use unless you just happen to know what video the thing you were looking for is in well what if you could just like search for something lonus Sebastian I want every clip with lonus Sebastian in it wow bam look at that shot up and let's say you know there's this one that's uh detected that it's you throughout the entire clip yeah you're in a chair so you could search for clips of lonus sitting down with a keyboard yeah like we're going to be able to actually find stuff yeah right now there is a a finite amount of objects that are trained I mean chihuahua let me scroll through this it's a lot eventually you'll be able to train it and tell it hey this is what a computer fan looks like or this is what an SSD looks like oh my God that is so cool so wait is this running on these extra CPU cores or okay no not right now faces and logos are running on CPU yeah objects OCR and scenes run on GPU got it but they're not running on any of those machines they're running on a GPU workstation that super micro sent that's sitting at my desk um it was Heavy anyways what is happening on that new server is proxies because if we were to analyze the original Clips oh AAL formatting is a huge problem when you go into an AI model it might not necessarily support the Kodak that you're filming in sure but also Clips are like hundreds of megabytes a second potentially that would take forever so instead it generates proxies of everything first first which we're dumping to that new server and then we can take advantage of the Lightning Fast storage yeah you can we have 2.6 massive compute and we can basically create like a proxy map of what everything is in the main archive right that is so cool so far I've generated 2.6 terab of proxies which might not sound like a lot but they're only 5 megabit so it's actually like a lot this is going to be a flipping game Cher News sports can you imagine your CNN you want that person wearing a red tie yeah but right now we've done 25,000 so 2.6 terabyt is 25,000 Pro okay well let's try and find something oh hold on once you've generated a proxy you have to then analyze it right ah so the analysis is not done no not even close I've analyzed 22 Clips okay everything with Elijah Elijah and this is the every clip that Elijah's in and you can even see this is so cool this is the actual ma'am as they call it media asset manager the axle AI guys built this before it was like AI as far as I'm aware back when you would have had to make comments like this manually now it's just AI so all of the data is in here now and we can see here's Adam and Elijah oh that's so cool here's all the different objects chair flower pot microphone oh let me show you the scene understanding thing cuz that is so cool this is like brand new thing they barely even worked it in but it basic it basically takes a snapshot every seconds two men are working on a project in a room there is a speaker stereo equipment there's a faucet there's a tripod there's the tripod some of these are a little less accurate two men are working on a robot in a room it kind of looks like a robot you I mean yeah sure two men are in a workshop looking at a laptop computer looking at a machine there is person Alex Clark so this is just running right now in real time like more stuff is getting processed as see here processing logos 9 there it is processing logos and faces it's going to take a while yeah it's going to take forever they're still working on making it function on multiple gpus so once we can get it running on like four gpus say one GPU is doing face detection one's doing scene analysis one's doing object detection or something like that we'll be able to go a lot faster but right now it's just one GPU got it but this is so cool all that's left is to deploy it lonus had to run away to do some other stuff so I've hired some backup Cavalry Sean our infrastructure administrator except we've run into a bit of a problem lonus and me and our Infinite Wisdom while we were making this rack so much better ran a bunch of cables right where we need to put the server did we just start unplugging no yeah how are we even going to do this we have to like part the seas exactly I started to try to move some of the cables out of the way but they're all twisted together so hopefully the LTT Cable Management thing which you can finally get at ltp store.com will save us beautiful cable managed we can slide a server in there now I hope you're in yeah it's on ow ow ow ow ow ow ow okay you're good just go that wasn't so bad like made for it next hey we're in now we just have to run a million cables uhoh do you notice anything different well it's loud most of that's actually just the vent is on one of the air conditioners is broken again but do you notice anything different I mean the sticker's here that that sticker's been there for years seriously you haven't noticed anything else well you guys uh screwed something onto the oh did you put sauna Pan behind it yeah but I thought this is supposed to be a vented door my original plan was to get rid of the vent that you put in but that vent was there as a backup in case the HVAC ever failed so that fan is the exhaust and that's the intake you see all the gaps F God there gaps but do you notice the sound difference yeah it's a big difference it's huge but that server is so loud we basically ended up where we started yeah but that's okay I was just trying to normalize I just mean I didn't make it worse it's not that okay look at that woo cute right God that's a lot of metal if all goes to plan we could get rid of this and this and just have these so no more additional rack taken up which is nice wow it should sustain two entire servers dropping out without anyone even noticing do you really want to test it right now I like I haven't tried that all right here we go what could go wrong u i mean a lot the fact that all the fans just like turned down a bit is a little scary let's go see if anyone noticed oh hi Mark hi I'm holding your file server how's your edit going uh what huh is it working it's working is this on Wi-Fi hey Emily hey how's your edit going I'm holding your server that's cool is it working are you sure yeah Hoffman what's up how's your edit going this is your server right here it's amazing look feel it it's still warm wow yeah it's it's still warm how well how's it working it's great you know I'm editing the video that we're shooting you are yeah uh we're going to pull another one wait no l you forgot one yeah here here's another here's another one of your servers is it working it's great though huhuh for reference you're not supposed to do this you should power off the system first but we're just trying to simulate it failing yeah a terrible catastrophic failure I can't believe how smoothly it handled that see all the lights they never stopped blinking big thanks to Super Micro for these awesome servers thanks to WCA for making this crazy software thanks to axle for the awesome AI detection if you like this video maybe check out the video series of us building our nearly three pedabytes of archival storage which we call the mother Vault that thing is awesome and we showed it to you and it's faster now oh and thanks to you for being an awesome viewer"}