{"video_id":"pLC0FUnko-M","title":"This is my Endgame - Mother Vault Server Room Update","channel":"Linus Tech Tips","show":"Linus Tech Tips","published_at":"2023-05-05T14:53:29Z","duration_s":1543,"segments":[{"start_s":0.32,"end_s":6.08,"text":"it's all here look at this it's been","speaker":null,"is_sponsor":0},{"start_s":3.439,"end_s":12.24,"text":"almost a year since we discovered the major issue that effectively put our","speaker":null,"is_sponsor":0},{"start_s":8.639,"end_s":14.48,"text":"entire video archive server offline","speaker":null,"is_sponsor":0},{"start_s":12.24,"end_s":17.6,"text":"that's right all the full quality footage since i was still working at","speaker":null,"is_sponsor":0},{"start_s":16.0,"end_s":22.8,"text":"ncix gone temporarily after the painstaking","speaker":null,"is_sponsor":0},{"start_s":20.72,"end_s":28.64,"text":"process of recovering nearly all that data love you wendell from level1tx we","speaker":null,"is_sponsor":0},{"start_s":25.68,"end_s":33.6,"text":"finally have all the hardware in place so today's the day we get our vault back","speaker":null,"is_sponsor":0},{"start_s":31.519,"end_s":40.0,"text":"up and running we're talking sprinting we're talking better than before we're","speaker":null,"is_sponsor":0},{"start_s":35.6,"end_s":41.68,"text":"talking an entire 3.6 petabyte array","speaker":null,"is_sponsor":0},{"start_s":40.0,"end_s":47.52,"text":"managed by one single head server instead of the five","speaker":null,"is_sponsor":0},{"start_s":44.64,"end_s":52.48,"text":"servers that we have now and i promise it's gonna be awesome","speaker":null,"is_sponsor":0},{"start_s":49.52,"end_s":56.48,"text":"like the segway to our sponsor i fix it looking to expand your selection of","speaker":null,"is_sponsor":0},{"start_s":54.0,"end_s":63.039,"text":"tools ifixit has you covered learn more about the manta toolkit and its 12 what","speaker":null,"is_sponsor":0},{"start_s":59.76,"end_s":64.96,"text":"112 steel bits what with two premium","speaker":null,"is_sponsor":0},{"start_s":63.039,"end_s":68.92,"text":"drivers learn all about it at the end of this video manta","speaker":null,"is_sponsor":0},{"start_s":76.32,"end_s":83.759,"text":"this is the one computer that is going to run the mother vault and the astute","speaker":null,"is_sponsor":0},{"start_s":81.439,"end_s":90.56,"text":"among you have probably already realized that we are not going to be able to put","speaker":null,"is_sponsor":0},{"start_s":86.56,"end_s":91.759,"text":"3.6 petabytes worth of storage inside it","speaker":null,"is_sponsor":0},{"start_s":90.56,"end_s":97.92,"text":"unless it's from hogwarts it's magic","speaker":null,"is_sponsor":0},{"start_s":95.36,"end_s":102.24,"text":"wingardium leviosa in here no it's one guardian leviosa oh sorry excuse me no","speaker":null,"is_sponsor":0},{"start_s":101.2,"end_s":106.64,"text":"in fact there are no hard drives in here this is","speaker":null,"is_sponsor":0},{"start_s":104.72,"end_s":111.84,"text":"where the magic happens the machine you saw just now that's only the brain so","speaker":null,"is_sponsor":0},{"start_s":109.52,"end_s":116.799,"text":"instead of trying to put 3.6 petabytes of hard drives into there they're gonna","speaker":null,"is_sponsor":0},{"start_s":114.0,"end_s":122.96,"text":"go in separate enclosures like this one called j buds or just a bunch of disks","speaker":null,"is_sponsor":0},{"start_s":120.399,"end_s":128.239,"text":"and you can think of an external jbot enclosure kind of like an external hard","speaker":null,"is_sponsor":0},{"start_s":125.759,"end_s":134.319,"text":"drive dock except instead of one hard drive they hold up to a whopping 90 hard","speaker":null,"is_sponsor":0},{"start_s":132.16,"end_s":139.04,"text":"drives each we'll talk more about these a little bit later","speaker":null,"is_sponsor":0},{"start_s":135.76,"end_s":141.12,"text":"just because the 2024 us-trt from","speaker":null,"is_sponsor":0},{"start_s":139.04,"end_s":146.959,"text":"supermicro is shirking its storage responsibilities doesn't mean that it's","speaker":null,"is_sponsor":0},{"start_s":143.36,"end_s":149.92,"text":"not packing inside here we find a dual","speaker":null,"is_sponsor":0},{"start_s":146.959,"end_s":154.239,"text":"socket AMD epic milan system that honestly has a bit of an unusual","speaker":null,"is_sponsor":0},{"start_s":151.76,"end_s":157.68,"text":"configuration typically a system with this much CPU horsepower so those are","speaker":null,"is_sponsor":0},{"start_s":156.8,"end_s":164.72,"text":"two 24 core AMD epic 74 f3s and the ability","speaker":null,"is_sponsor":0},{"start_s":162.319,"end_s":171.599,"text":"to run up to dual 64 cores would be decked out with a ton of NVMe slots or","speaker":null,"is_sponsor":0},{"start_s":168.319,"end_s":174.48,"text":"some enterprise gpus or something like","speaker":null,"is_sponsor":0},{"start_s":171.599,"end_s":177.92,"text":"that but this guy only has four","speaker":null,"is_sponsor":0},{"start_s":175.28,"end_s":182.159,"text":"NVMe slots and these are an optional accessory the rest of the storage up","speaker":null,"is_sponsor":0},{"start_s":180.08,"end_s":186.239,"text":"here and in fact these four slots by default are i kid you not","speaker":null,"is_sponsor":0},{"start_s":184.959,"end_s":191.84,"text":"six gigabit SATA three and a half inch slots","speaker":null,"is_sponsor":0},{"start_s":189.36,"end_s":195.44,"text":"like that you would use for a hard drive that's because this system is actually","speaker":null,"is_sponsor":0},{"start_s":193.599,"end_s":199.519,"text":"designed with a different purpose in mind say for example uh as a","speaker":null,"is_sponsor":0},{"start_s":197.68,"end_s":203.599,"text":"virtualization server where you might need a ton of CPU that's going to be","speaker":null,"is_sponsor":0},{"start_s":201.36,"end_s":208.08,"text":"split up to different clients or if you're like us a ton of CPU that you're","speaker":null,"is_sponsor":0},{"start_s":206.4,"end_s":212.959,"text":"going to use to drive a boatload of connected storage that","speaker":null,"is_sponsor":0},{"start_s":210.0,"end_s":218.48,"text":"isn't inside the chassis so instead of using up all those express lanes from","speaker":null,"is_sponsor":0},{"start_s":215.36,"end_s":220.599,"text":"those epic cpus which is","speaker":null,"is_sponsor":0},{"start_s":218.48,"end_s":227.68,"text":"buddy 128 lanes each with like NVMe drives in","speaker":null,"is_sponsor":0},{"start_s":224.72,"end_s":233.36,"text":"the front we instead want them allocated to PCIe slots in the back of the server","speaker":null,"is_sponsor":0},{"start_s":230.48,"end_s":238.64,"text":"back there we could install high-end networking adapters what is this dual","speaker":null,"is_sponsor":0},{"start_s":235.2,"end_s":238.64,"text":"100 gigabit this guy right here","speaker":null,"is_sponsor":0},{"start_s":238.799,"end_s":246.879,"text":"or the other piece of our puzzle host bus adapters just like with an external","speaker":null,"is_sponsor":0},{"start_s":244.0,"end_s":251.76,"text":"hard drive our jbods back there which the keynight among you will have noticed","speaker":null,"is_sponsor":0},{"start_s":248.48,"end_s":254.799,"text":"that there are two require some kind of","speaker":null,"is_sponsor":0},{"start_s":251.76,"end_s":257.12,"text":"connection to the actual server with the","speaker":null,"is_sponsor":0},{"start_s":254.799,"end_s":260.959,"text":"actual CPU in it and the actual operating system running on it in a","speaker":null,"is_sponsor":0},{"start_s":258.72,"end_s":265.36,"text":"modern jbod this is usually handled by mini sas hd cables like these ones from","speaker":null,"is_sponsor":0},{"start_s":263.919,"end_s":269.68,"text":"infinite cables we're going to have these linked down below","speaker":null,"is_sponsor":0},{"start_s":267.12,"end_s":276.0,"text":"so each of these ports on the back of our three cards can handle four lanes of","speaker":null,"is_sponsor":0},{"start_s":272.96,"end_s":278.56,"text":"12 gigabit sas that's the enterprise big","speaker":null,"is_sponsor":0},{"start_s":276.0,"end_s":283.68,"text":"brother of SATA for a total of 48 gigabits per second per cable your","speaker":null,"is_sponsor":0},{"start_s":281.28,"end_s":288.88,"text":"average computer even server probably doesn't have any factory mini sas hd","speaker":null,"is_sponsor":0},{"start_s":286.88,"end_s":294.96,"text":"ports so we're going to be adding them to this machine with three of broadcom's","speaker":null,"is_sponsor":0},{"start_s":291.6,"end_s":297.36,"text":"9500 16e host bus adapters that means","speaker":null,"is_sponsor":0},{"start_s":294.96,"end_s":301.52,"text":"we'll have a full dedicated card for each of our j-bonds oh yeah there are","speaker":null,"is_sponsor":0},{"start_s":299.84,"end_s":304.8,"text":"three of these right there's one more j-bud that's actually already in the","speaker":null,"is_sponsor":0},{"start_s":303.12,"end_s":309.68,"text":"server room we will show you guys that one later now each of our jbods allows","speaker":null,"is_sponsor":0},{"start_s":307.28,"end_s":313.68,"text":"us to connect up to three cables for increased throughput so that's exactly","speaker":null,"is_sponsor":0},{"start_s":311.6,"end_s":318.32,"text":"what we're going to do on the subject of throughput to push this thing to the","speaker":null,"is_sponsor":0},{"start_s":315.68,"end_s":323.28,"text":"absolute limit we want to pack this machine with as much RAM as possible","speaker":null,"is_sponsor":0},{"start_s":321.6,"end_s":328.479,"text":"since the file system that we're using called zfs can use RAM as a read caching","speaker":null,"is_sponsor":0},{"start_s":326.88,"end_s":332.639,"text":"layer to significantly improve the responsiveness of the array because","speaker":null,"is_sponsor":0},{"start_s":330.56,"end_s":338.0,"text":"mechanical drives on their own yeah even with this many of them not","speaker":null,"is_sponsor":0},{"start_s":335.44,"end_s":342.56,"text":"that great in terms of access times luckily the folks over at micron just","speaker":null,"is_sponsor":0},{"start_s":340.96,"end_s":348.88,"text":"sent over one terabyte of 3200 megatransfer per second","speaker":null,"is_sponsor":0},{"start_s":345.52,"end_s":351.199,"text":"registered ecc memory which last time i","speaker":null,"is_sponsor":0},{"start_s":348.88,"end_s":355.6,"text":"checked smells like performance but the","speaker":null,"is_sponsor":0},{"start_s":353.199,"end_s":361.68,"text":"performance improvements don't stop there we had some","speaker":null,"is_sponsor":0},{"start_s":357.759,"end_s":364.16,"text":"leftover keoxia 30 terabyte NVMe drives","speaker":null,"is_sponsor":0},{"start_s":361.68,"end_s":368.479,"text":"from the dream server build sorry and we figured they would be the perfect","speaker":null,"is_sponsor":0},{"start_s":366.4,"end_s":374.319,"text":"choice to configure as a special metadata device in our zfs pool this is","speaker":null,"is_sponsor":0},{"start_s":372.08,"end_s":380.479,"text":"a game changer for performance because it allows metadata like where the files","speaker":null,"is_sponsor":0},{"start_s":377.68,"end_s":386.4,"text":"are physically stored on those hundreds of hard drives the allocation tables and","speaker":null,"is_sponsor":0},{"start_s":383.52,"end_s":390.4,"text":"optionally even small complete files to be stored on a discrete higher","speaker":null,"is_sponsor":0},{"start_s":388.72,"end_s":395.28,"text":"performance device like these NVMe drives this can make","speaker":null,"is_sponsor":0},{"start_s":393.12,"end_s":400.8,"text":"navigating through large data structures like the thousands of video projects","speaker":null,"is_sponsor":0},{"start_s":397.6,"end_s":403.36,"text":"that we're archiving much much faster","speaker":null,"is_sponsor":0},{"start_s":400.8,"end_s":407.199,"text":"and that directly results in workflow improvements for our staff i'd say it's","speaker":null,"is_sponsor":0},{"start_s":405.68,"end_s":413.28,"text":"about as much of a quality of life improvement as our flow plane subscribers get with no baked in ads","speaker":null,"is_sponsor":0},{"start_s":411.12,"end_s":417.6,"text":"higher quality streaming and exclusive behind the scenes content we're going to","speaker":null,"is_sponsor":0},{"start_s":414.72,"end_s":423.199,"text":"be setting these up in a mirrored stripe leaving us with a total of 60 terabytes","speaker":null,"is_sponsor":0},{"start_s":421.28,"end_s":427.919,"text":"of capacity which is hilarious given that it's a raid 10 of just four drives","speaker":null,"is_sponsor":0},{"start_s":425.12,"end_s":433.919,"text":"which we think should be an ample amount of fast metadata storage for our 3.6","speaker":null,"is_sponsor":0},{"start_s":431.199,"end_s":437.52,"text":"petabyte array unfortunately because we only have these four NVMe slots we're","speaker":null,"is_sponsor":0},{"start_s":436.16,"end_s":440.96,"text":"not going to be able to expand that storage unless higher capacity drives","speaker":null,"is_sponsor":0},{"start_s":439.28,"end_s":445.84,"text":"come along in the future and the performance compared to some of our","speaker":null,"is_sponsor":0},{"start_s":442.639,"end_s":448.72,"text":"other all NVMe arrays is not going to be","speaker":null,"is_sponsor":0},{"start_s":445.84,"end_s":453.28,"text":"that amazing another problem is that this doesn't leave us any additional","speaker":null,"is_sponsor":0},{"start_s":450.639,"end_s":458.4,"text":"space for NVMe storage drives for a level 2 arc reed cache or an intent log","speaker":null,"is_sponsor":0},{"start_s":456.639,"end_s":464.08,"text":"at least at the front but there's a solution to that huh yes","speaker":null,"is_sponsor":0},{"start_s":461.44,"end_s":470.479,"text":"this cute little half height PCIe card from liquid allows us to install four","speaker":null,"is_sponsor":0},{"start_s":467.28,"end_s":472.639,"text":"m.2 drives in a single slot","speaker":null,"is_sponsor":0},{"start_s":470.479,"end_s":477.599,"text":"inside the server and this will be absolutely perfect for that roll what's","speaker":null,"is_sponsor":0},{"start_s":475.599,"end_s":480.96,"text":"the total capacity of this bad boy i don't actually know what ssds are in","speaker":null,"is_sponsor":0},{"start_s":479.12,"end_s":484.96,"text":"there we don't know what's in it there is one we have that is four or four","speaker":null,"is_sponsor":0},{"start_s":482.639,"end_s":488.319,"text":"terabyte ssds okay so it might be this one it could be this one it could also","speaker":null,"is_sponsor":0},{"start_s":486.8,"end_s":491.759,"text":"not be this one great the problem for later Jake this is","speaker":null,"is_sponsor":0},{"start_s":490.319,"end_s":496.639,"text":"pretty cool Jake did you design this three and a half to two and a half inch adapter thing no we found it on","speaker":null,"is_sponsor":0},{"start_s":494.319,"end_s":500.56,"text":"thingiverse oh cool printed it shut out thingiverse whoa","speaker":null,"is_sponsor":0},{"start_s":498.56,"end_s":503.44,"text":"what well that came out which slot did you want the liquid card in uh you gotta","speaker":null,"is_sponsor":0},{"start_s":502.24,"end_s":507.599,"text":"change this guy and then put it right here oh man can we","speaker":null,"is_sponsor":0},{"start_s":505.44,"end_s":510.24,"text":"find out if it's the right one before we do this because we're going to have to","speaker":null,"is_sponsor":0},{"start_s":509.12,"end_s":516.479,"text":"pull the whole thing off what do you mean what's the right one the the high capacity one oh","speaker":null,"is_sponsor":0},{"start_s":513.919,"end_s":520.36,"text":"no oh come on there's got to be a test bench not far","speaker":null,"is_sponsor":0},{"start_s":521.68,"end_s":527.2,"text":"no there's none over there there's none over there Jake says there is none well","speaker":null,"is_sponsor":0},{"start_s":525.68,"end_s":531.839,"text":"let's just go to my bench upstairs logistics","speaker":null,"is_sponsor":0},{"start_s":530.08,"end_s":535.839,"text":"i think it's important for you guys to know that Jake thinks you guys are","speaker":null,"is_sponsor":0},{"start_s":533.68,"end_s":540.64,"text":"utterly incompetent that's what he said whoa those are his words not mine he","speaker":null,"is_sponsor":0},{"start_s":538.48,"end_s":545.76,"text":"said those guys don't even have a test bench over there at all not one","speaker":null,"is_sponsor":0},{"start_s":543.12,"end_s":551.12,"text":"yeah that's what i thought yeah yeah Jake mere minutes later all right hey we","speaker":null,"is_sponsor":0},{"start_s":548.72,"end_s":554.64,"text":"got a foreign dynamic disc here these are what three terabytes","speaker":null,"is_sponsor":0},{"start_s":555.12,"end_s":559.68,"text":"okay right click on that one let's see if it's","speaker":null,"is_sponsor":0},{"start_s":558.24,"end_s":563.12,"text":"toshiba three terabytes 84. yeah perfect these","speaker":null,"is_sponsor":0},{"start_s":561.68,"end_s":566.959,"text":"are the four terabyte ones thank you fast and competent logistics team i just","speaker":null,"is_sponsor":0},{"start_s":565.279,"end_s":571.36,"text":"want you guys to know that i defended you i was the first to say no i would","speaker":null,"is_sponsor":0},{"start_s":569.279,"end_s":575.6,"text":"never use those words about our logistics team","speaker":null,"is_sponsor":0},{"start_s":572.8,"end_s":575.6,"text":"not like Jake","speaker":null,"is_sponsor":0},{"start_s":577.279,"end_s":582.32,"text":"goodbye get out of here","speaker":null,"is_sponsor":0},{"start_s":580.399,"end_s":586.48,"text":"ah here's what we've got next now i mentioned that this was 100 gigabit but","speaker":null,"is_sponsor":0},{"start_s":584.32,"end_s":593.12,"text":"what i didn't say i don't think is that it has dual 100 gigabit ports it can do","speaker":null,"is_sponsor":0},{"start_s":589.2,"end_s":596.399,"text":"a total of 200 gigabit it's PCIe gen 4","speaker":null,"is_sponsor":0},{"start_s":593.12,"end_s":600.399,"text":"by 16 that is a flipping ton of","speaker":null,"is_sponsor":0},{"start_s":596.399,"end_s":602.48,"text":"bandwidth and it uses these sick qsfp28","speaker":null,"is_sponsor":0},{"start_s":600.399,"end_s":606.399,"text":"connectors while Jake orders his lunch i'm going to go ahead and get our other","speaker":null,"is_sponsor":0},{"start_s":604.0,"end_s":610.8,"text":"two hbas installed here i'm getting mucho burrito editor you better put this","speaker":null,"is_sponsor":0},{"start_s":608.48,"end_s":613.6,"text":"in the goddamn cut okay i'm just gonna put this in because","speaker":null,"is_sponsor":0},{"start_s":612.0,"end_s":617.12,"text":"you're taking too long what do you mean i'm taking too long i'm not the one","speaker":null,"is_sponsor":0},{"start_s":615.2,"end_s":624.079,"text":"sitting here ordering lunch touche this is amazing so we've got our","speaker":null,"is_sponsor":0},{"start_s":620.48,"end_s":626.8,"text":"three hbas we've got our 100 gig network","speaker":null,"is_sponsor":0},{"start_s":624.079,"end_s":631.279,"text":"card we've got our additional SSD which is somewhere in the neighborhood of","speaker":null,"is_sponsor":0},{"start_s":628.079,"end_s":634.24,"text":"around 15 terabytes of storage for our","speaker":null,"is_sponsor":0},{"start_s":631.279,"end_s":638.16,"text":"slog and level 2 arc well they're 3.84 each oh i see so like somewhere in that","speaker":null,"is_sponsor":0},{"start_s":636.16,"end_s":640.8,"text":"15 point something oh","speaker":null,"is_sponsor":0},{"start_s":639.2,"end_s":644.72,"text":"and how do you forget about that i forgot about these what i forget 120","speaker":null,"is_sponsor":0},{"start_s":642.72,"end_s":648.88,"text":"terabytes of NVMe in the front yeah yeah are we ready to turn it on i'm","speaker":null,"is_sponsor":0},{"start_s":646.959,"end_s":652.88,"text":"turned on okay i could be more turned on if i was","speaker":null,"is_sponsor":0},{"start_s":650.56,"end_s":657.519,"text":"chilling for lttstore.com hey check out the RGB diode shirt it's like based on","speaker":null,"is_sponsor":0},{"start_s":655.76,"end_s":661.76,"text":"what the individual sub pixels look like but then it's also just a cool design so","speaker":null,"is_sponsor":0},{"start_s":659.44,"end_s":666.24,"text":"do you want the top yeah do a peel very graceful Jake you're doing a fan oh","speaker":null,"is_sponsor":0},{"start_s":664.959,"end_s":672.399,"text":"oh that's right i'm a monster","speaker":null,"is_sponsor":0},{"start_s":669.76,"end_s":676.88,"text":"this thing is sick do we get to power it on now got a laptop a laptop","speaker":null,"is_sponsor":0},{"start_s":675.12,"end_s":680.88,"text":"uh high this one is easy just wait it's actually","speaker":null,"is_sponsor":0},{"start_s":678.88,"end_s":685.519,"text":"really quiet give it a little bit of time are you","speaker":null,"is_sponsor":0},{"start_s":682.0,"end_s":688.519,"text":"gaslighting me sir no i would never","speaker":null,"is_sponsor":0},{"start_s":685.519,"end_s":688.519,"text":"bmw","speaker":null,"is_sponsor":0},{"start_s":691.92,"end_s":697.68,"text":"oh that is so much better and can we turn off that alarm as well i have a","speaker":null,"is_sponsor":0},{"start_s":696.16,"end_s":702.0,"text":"solution i don't think that's going to work it does pulling the power supply","speaker":null,"is_sponsor":0},{"start_s":699.519,"end_s":706.959,"text":"out right makes the alarm for oh wow it didn't work it worked this is a 1600","speaker":null,"is_sponsor":0},{"start_s":704.88,"end_s":712.0,"text":"watt power supply what i mean to be clear that's only a 200 volt input it's","speaker":null,"is_sponsor":0},{"start_s":709.44,"end_s":717.32,"text":"a thousand watts if you're from 100 227 volts","speaker":null,"is_sponsor":0},{"start_s":717.68,"end_s":723.6,"text":"that's what you wanted to do i want to go fast","speaker":null,"is_sponsor":0},{"start_s":721.2,"end_s":729.839,"text":"you said it was quiet so i thought full fan speed must not be that bad right i","speaker":null,"is_sponsor":0},{"start_s":726.56,"end_s":732.48,"text":"had him log me separately into the ipmi","speaker":null,"is_sponsor":0},{"start_s":729.839,"end_s":737.2,"text":"just so i could do that in other news it's booted we got our terabyte of RAM","speaker":null,"is_sponsor":0},{"start_s":735.36,"end_s":740.56,"text":"we got our oh yeah we got our melanox card in there that's good did you put","speaker":null,"is_sponsor":0},{"start_s":738.639,"end_s":744.61,"text":"the nvmes in the right slot i mean one of them is wrong what do you","speaker":null,"is_sponsor":0},{"start_s":742.079,"end_s":747.839,"text":"mean wrong like oh it's wrong","speaker":null,"is_sponsor":0},{"start_s":747.839,"end_s":753.2,"text":"try the bodies one of those is a boot drive might it","speaker":null,"is_sponsor":0},{"start_s":752.079,"end_s":757.44,"text":"might be good there was an activity flash yeah right","speaker":null,"is_sponsor":0},{"start_s":755.279,"end_s":760.88,"text":"okay it's cool and look because the boot device","speaker":null,"is_sponsor":0},{"start_s":758.48,"end_s":764.0,"text":"is a pool just immediately fine i mean it's","speaker":null,"is_sponsor":0},{"start_s":762.56,"end_s":769.44,"text":"probably mad actually this is going to be kind of complicated because some of the drives we like need to put in the","speaker":null,"is_sponsor":0},{"start_s":767.36,"end_s":773.76,"text":"right spot and not mix up because some of the arrays we're keeping and some of","speaker":null,"is_sponsor":0},{"start_s":771.6,"end_s":777.839,"text":"the drives are in another jbod that wendell's working on right now so we","speaker":null,"is_sponsor":0},{"start_s":775.12,"end_s":781.04,"text":"can't really touch those ones uh-huh it's gonna be fun let's go start","speaker":null,"is_sponsor":0},{"start_s":779.279,"end_s":785.6,"text":"carrying how am i supposed to move this this is a four-man carry it literally","speaker":null,"is_sponsor":0},{"start_s":783.519,"end_s":789.279,"text":"has a label on it that says four people need to keep it actually yeah dude these","speaker":null,"is_sponsor":0},{"start_s":787.76,"end_s":793.68,"text":"are like the best earmuffs put them on oh wow now with 80 percent","speaker":null,"is_sponsor":0},{"start_s":791.839,"end_s":796.72,"text":"less Jake i can endorse these","speaker":null,"is_sponsor":0},{"start_s":795.12,"end_s":800.959,"text":"we don't actually have to take all of these out we could just take this one","speaker":null,"is_sponsor":0},{"start_s":799.2,"end_s":804.32,"text":"and that one and then just move this so the head server will just go here","speaker":null,"is_sponsor":0},{"start_s":803.12,"end_s":809.44,"text":"and then we'll go one so we don't even have to move this one so we just have to take the temp vault out should i let the","speaker":null,"is_sponsor":0},{"start_s":807.519,"end_s":812.639,"text":"editors know temp fault's going away uh i already told them it was going to","speaker":null,"is_sponsor":0},{"start_s":810.639,"end_s":815.68,"text":"happen today how do you tell them it's happening now i'll tell them what's","speaker":null,"is_sponsor":0},{"start_s":813.76,"end_s":820.0,"text":"happening now hello hardworking and truly valued","speaker":null,"is_sponsor":0},{"start_s":817.839,"end_s":827.639,"text":"editing team the vault that you have come to know and love temp vault","speaker":null,"is_sponsor":0},{"start_s":822.639,"end_s":827.639,"text":"is now deadvault press f to pay respects","speaker":null,"is_sponsor":0},{"start_s":828.56,"end_s":834.72,"text":"they don't care at all all right bye kind of raises the question do they","speaker":null,"is_sponsor":0},{"start_s":833.519,"end_s":837.92,"text":"need a vault at all then why don't we just have no vault then forget it","speaker":null,"is_sponsor":0},{"start_s":836.639,"end_s":840.079,"text":"if you guys aren't even going to miss it then","speaker":null,"is_sponsor":0},{"start_s":840.399,"end_s":847.92,"text":"what do you mean you copied the files you guys aren't supposed to keep things on your local machines we copied it to","speaker":null,"is_sponsor":0},{"start_s":844.72,"end_s":849.76,"text":"the active server new new new new monik","speaker":null,"is_sponsor":0},{"start_s":847.92,"end_s":853.68,"text":"hopefully this clears everything up and is not confusing at all i think they're","speaker":null,"is_sponsor":0},{"start_s":851.279,"end_s":858.639,"text":"more confused look somebody is reading files from it","speaker":null,"is_sponsor":0},{"start_s":856.16,"end_s":863.44,"text":"right now like ed was lying look oh somebody's is reading from it or","speaker":null,"is_sponsor":0},{"start_s":861.6,"end_s":868.32,"text":"they're writing to it what are they doing oh my god","speaker":null,"is_sponsor":0},{"start_s":865.36,"end_s":873.279,"text":"at 350 megabytes a second this is a not insignificant data transfer","speaker":null,"is_sponsor":0},{"start_s":871.279,"end_s":876.56,"text":"the chart's like nothing and then somebody's like","speaker":null,"is_sponsor":0},{"start_s":875.04,"end_s":881.44,"text":"you want to shut it off now okay who's messing with the server are you messing","speaker":null,"is_sponsor":0},{"start_s":878.959,"end_s":884.24,"text":"with us ed somebody is vaulting right now","speaker":null,"is_sponsor":0},{"start_s":882.32,"end_s":886.48,"text":"did you copy things to the vault right now","speaker":null,"is_sponsor":0},{"start_s":887.44,"end_s":895.12,"text":"probationary employee number 17. you cut and paste it","speaker":null,"is_sponsor":0},{"start_s":892.639,"end_s":898.16,"text":"i should not i use the tara terra copy","speaker":null,"is_sponsor":0},{"start_s":896.639,"end_s":903.6,"text":"but it's still going it's still kind of taste with territory","speaker":null,"is_sponsor":0},{"start_s":900.959,"end_s":908.16,"text":"dang it guys it's such a good you're done you're good","speaker":null,"is_sponsor":0},{"start_s":905.6,"end_s":914.079,"text":"god damn it ignore the the discs with errors uh what there's um we've lost a","speaker":null,"is_sponsor":0},{"start_s":911.68,"end_s":919.04,"text":"few the original vaults 10 terabyte drives are still like in production and","speaker":null,"is_sponsor":0},{"start_s":916.72,"end_s":923.36,"text":"they're like six years old now and we've had problems too our cooling in that","speaker":null,"is_sponsor":0},{"start_s":921.76,"end_s":927.04,"text":"room didn't used to be super stable there were times when you would walk in","speaker":null,"is_sponsor":0},{"start_s":924.959,"end_s":931.839,"text":"there and it was like a wall of heat yeah because we had this","speaker":null,"is_sponsor":0},{"start_s":929.76,"end_s":936.639,"text":"air conditioning unit that would just whatever when we didn't have an air conditioner","speaker":null,"is_sponsor":0},{"start_s":934.56,"end_s":939.44,"text":"yeah yeah at one point i remember when i started here atlantis was like no we","speaker":null,"is_sponsor":0},{"start_s":938.0,"end_s":944.16,"text":"don't need it we don't need it and then all the ups exploded because it was too","speaker":null,"is_sponsor":0},{"start_s":941.92,"end_s":946.88,"text":"hot and he was like okay fine how are we going to move these","speaker":null,"is_sponsor":0},{"start_s":945.279,"end_s":950.399,"text":"fortunately these aren't really rails they're actually like shelves so all we","speaker":null,"is_sponsor":0},{"start_s":948.88,"end_s":954.8,"text":"really need to do is kind of put it on the ground and then just","speaker":null,"is_sponsor":0},{"start_s":952.8,"end_s":958.399,"text":"tilt one end and then slide it in and tilt the other end that's how we put the","speaker":null,"is_sponsor":0},{"start_s":956.56,"end_s":964.8,"text":"other one in there installation by two persons is recommended if these need two","speaker":null,"is_sponsor":0},{"start_s":961.279,"end_s":966.8,"text":"people that server needs 25.","speaker":null,"is_sponsor":0},{"start_s":964.8,"end_s":970.0,"text":"i think i can handle it i wonder when my burrito's going to arrive all the drives","speaker":null,"is_sponsor":0},{"start_s":968.8,"end_s":974.399,"text":"are going to have to come out of this thing before we lift it out no no it's","speaker":null,"is_sponsor":0},{"start_s":972.0,"end_s":979.6,"text":"fine how are we supposed to lift this we can lift it we've done it okay well how","speaker":null,"is_sponsor":0},{"start_s":977.199,"end_s":982.72,"text":"it's drunk no really though it's not off the ground i just use my","speaker":null,"is_sponsor":0},{"start_s":980.959,"end_s":986.56,"text":"feet like this usually your feet yeah and then you just walk it up walk it out","speaker":null,"is_sponsor":0},{"start_s":984.8,"end_s":990.8,"text":"yeah three two i'm holding the heavy side","speaker":null,"is_sponsor":0},{"start_s":987.839,"end_s":994.399,"text":"don't drop it okay i didn't drop it wow you just put all the weight on my what","speaker":null,"is_sponsor":0},{"start_s":992.32,"end_s":998.32,"text":"was on your foot cool well that's all the hard work yeah now","speaker":null,"is_sponsor":0},{"start_s":996.079,"end_s":1002.959,"text":"that we've got lab two over there we might actually have a purpose for all","speaker":null,"is_sponsor":0},{"start_s":1000.0,"end_s":1008.079,"text":"these additional storinators oh man i haven't gone spelunking in the uh server","speaker":null,"is_sponsor":0},{"start_s":1006.0,"end_s":1011.759,"text":"rack in ages Jake there's so much treasure in here you're","speaker":null,"is_sponsor":0},{"start_s":1009.759,"end_s":1015.519,"text":"giving everyone quite the view right now whatever usually you got to pay extra","speaker":null,"is_sponsor":0},{"start_s":1013.519,"end_s":1019.68,"text":"for that onlyplanes.com how do you like this setup i'm taking a","speaker":null,"is_sponsor":0},{"start_s":1017.44,"end_s":1024.72,"text":"little server cabinet nap here got my little uh pad on the back of my head","speaker":null,"is_sponsor":0},{"start_s":1021.92,"end_s":1027.839,"text":"okay bottom of the fifth not bad","speaker":null,"is_sponsor":0},{"start_s":1025.839,"end_s":1032.88,"text":"this was definitely the best way to do this the front of this shelf is out","speaker":null,"is_sponsor":0},{"start_s":1030.64,"end_s":1037.28,"text":"yeah that's what it appears uh that's why we screw","speaker":null,"is_sponsor":0},{"start_s":1034.799,"end_s":1042.0,"text":"how full is this uh it's got 60 drives in it okay what seems to have happened","speaker":null,"is_sponsor":0},{"start_s":1040.0,"end_s":1046.16,"text":"with this server because we didn't screw the shelf in from the front is this","speaker":null,"is_sponsor":0},{"start_s":1044.4,"end_s":1051.36,"text":"little tooth here that holds it into place seems to have slipped so it's","speaker":null,"is_sponsor":0},{"start_s":1049.6,"end_s":1057.679,"text":"sagging down like that which means that a ton of this weight is","speaker":null,"is_sponsor":0},{"start_s":1054.72,"end_s":1061.76,"text":"being borne by a part that's really not designed for that complicates our","speaker":null,"is_sponsor":0},{"start_s":1060.08,"end_s":1065.6,"text":"operation here a little bit because we can't just pop it back into place from","speaker":null,"is_sponsor":0},{"start_s":1063.84,"end_s":1069.039,"text":"here we can't really get at it looks like we need to shut this down","speaker":null,"is_sponsor":0},{"start_s":1067.6,"end_s":1073.32,"text":"pull it off put it back into place screw it in and","speaker":null,"is_sponsor":0},{"start_s":1070.799,"end_s":1078.96,"text":"then put it back in which sucks if i take the weight and you slide","speaker":null,"is_sponsor":0},{"start_s":1077.2,"end_s":1082.88,"text":"it forward or the other way around here i'll use my leg up there we go it's up","speaker":null,"is_sponsor":0},{"start_s":1081.2,"end_s":1088.48,"text":"just put the put the rail in don't move wait put what rail in","speaker":null,"is_sponsor":0},{"start_s":1085.76,"end_s":1091.6,"text":"well i can't see it oh my god like this oh i think we got it i think we got it","speaker":null,"is_sponsor":0},{"start_s":1089.76,"end_s":1095.44,"text":"let go let go yeah this actually went pretty okay","speaker":null,"is_sponsor":0},{"start_s":1093.28,"end_s":1099.44,"text":"situations like this are when i'm so glad we developed this screwdriver i got","speaker":null,"is_sponsor":0},{"start_s":1097.76,"end_s":1105.52,"text":"the nice strong magnet holding it in place almost no","speaker":null,"is_sponsor":0},{"start_s":1102.08,"end_s":1106.559,"text":"back force required to spin the ratchet","speaker":null,"is_sponsor":0},{"start_s":1105.52,"end_s":1110.88,"text":"like is it in there easy yeah it's in oh my","speaker":null,"is_sponsor":0},{"start_s":1108.799,"end_s":1115.2,"text":"god wait what how are we supposed to get it from here down to there we got it up","speaker":null,"is_sponsor":0},{"start_s":1112.88,"end_s":1118.32,"text":"there somehow so it's just like a really heavy storenator why don't we just put","speaker":null,"is_sponsor":0},{"start_s":1117.039,"end_s":1123.679,"text":"the other one on top that says this cart can take 600 pounds no we're not doing that okay","speaker":null,"is_sponsor":0},{"start_s":1121.12,"end_s":1128.32,"text":"that's right what are the odds uh that it just goes right in","speaker":null,"is_sponsor":0},{"start_s":1126.16,"end_s":1132.16,"text":"no the angle is still too much okay i'm getting under it boys put the right side","speaker":null,"is_sponsor":0},{"start_s":1130.64,"end_s":1138.0,"text":"in a bit more okay good good here oh my god it's it yeah it's on it's on the top","speaker":null,"is_sponsor":0},{"start_s":1134.559,"end_s":1138.72,"text":"okay go ahead you can push it in","speaker":null,"is_sponsor":0},{"start_s":1138.0,"end_s":1143.6,"text":"hi all right now for the hard one yeah","speaker":null,"is_sponsor":0},{"start_s":1141.36,"end_s":1147.919,"text":"this is more of a Linus sized server yolo","speaker":null,"is_sponsor":0},{"start_s":1146.0,"end_s":1151.84,"text":"well it's easy to yolo when somebody else already did all the work yeah good","speaker":null,"is_sponsor":0},{"start_s":1150.48,"end_s":1157.52,"text":"work Jake there goes the brain and what's really","speaker":null,"is_sponsor":0},{"start_s":1154.32,"end_s":1162.08,"text":"wild is that if we want more capacity","speaker":null,"is_sponsor":0},{"start_s":1157.52,"end_s":1164.0,"text":"more than 90 180 270 drives with the","speaker":null,"is_sponsor":0},{"start_s":1162.08,"end_s":1166.799,"text":"cards we have in now we could do another one and then i think there's room for","speaker":null,"is_sponsor":0},{"start_s":1165.36,"end_s":1170.32,"text":"one more card right i would say right now with no hbas we","speaker":null,"is_sponsor":0},{"start_s":1168.799,"end_s":1174.559,"text":"could probably add two and it would still be pretty good performance","speaker":null,"is_sponsor":0},{"start_s":1172.08,"end_s":1182.24,"text":"um all right let's uh let's put in the hard one ready one two three no yes","speaker":null,"is_sponsor":0},{"start_s":1180.32,"end_s":1186.32,"text":"what weak all right still weak now that they're in","speaker":null,"is_sponsor":0},{"start_s":1184.48,"end_s":1192.64,"text":"there working on these is glorious you just press the buttons","speaker":null,"is_sponsor":0},{"start_s":1189.84,"end_s":1196.16,"text":"pull them out boom funny story when we first put this in i was like ah it's so","speaker":null,"is_sponsor":0},{"start_s":1194.88,"end_s":1200.799,"text":"heavy it doesn't really need to be bolted down does it i was told yes in","speaker":null,"is_sponsor":0},{"start_s":1198.559,"end_s":1204.72,"text":"fact it does need to be bolted down and this","speaker":null,"is_sponsor":0},{"start_s":1201.84,"end_s":1210.24,"text":"is why because if we had a handful of these hanging off the front of it and it","speaker":null,"is_sponsor":0},{"start_s":1207.36,"end_s":1211.919,"text":"wasn't bolted into the concrete","speaker":null,"is_sponsor":0},{"start_s":1211.919,"end_s":1216.96,"text":"we'd have a bit of a problem bit of an update here we shucked all those drives","speaker":null,"is_sponsor":0},{"start_s":1215.36,"end_s":1220.799,"text":"put them in the servers i plugged a bunch of stuff in some power and we have","speaker":null,"is_sponsor":0},{"start_s":1219.12,"end_s":1223.76,"text":"blinky lights now do i get to press the button yeah it's on the right side give","speaker":null,"is_sponsor":0},{"start_s":1222.16,"end_s":1228.0,"text":"them a sec everybody in the next square mile is going to hear this they're just","speaker":null,"is_sponsor":0},{"start_s":1225.6,"end_s":1231.12,"text":"they're they're spinning up boys wow it's actually not as bad in here","speaker":null,"is_sponsor":0},{"start_s":1229.84,"end_s":1235.039,"text":"come outside let's uh let's shut the door yeah sure","speaker":null,"is_sponsor":0},{"start_s":1233.36,"end_s":1237.6,"text":"you can hear how awful it is still through the door","speaker":null,"is_sponsor":0},{"start_s":1238.799,"end_s":1244.64,"text":"oh yeah we wouldn't be able to shoot with those things going","speaker":null,"is_sponsor":0},{"start_s":1242.24,"end_s":1247.44,"text":"but they don't go okay that's fine oh you just got to add another layer of","speaker":null,"is_sponsor":0},{"start_s":1246.24,"end_s":1251.76,"text":"insulation there's no space Jake well this is the","speaker":null,"is_sponsor":0},{"start_s":1250.08,"end_s":1257.12,"text":"backup plan right because there's a thermostat in here that will start up a","speaker":null,"is_sponsor":0},{"start_s":1254.799,"end_s":1260.96,"text":"fan that exhausts out the top and will pull air passively through this grill in","speaker":null,"is_sponsor":0},{"start_s":1259.2,"end_s":1265.039,"text":"the event that the air conditioning completely fails do you remember that","speaker":null,"is_sponsor":0},{"start_s":1262.96,"end_s":1268.799,"text":"the shroud for this thing i made that in metal shop when i was still in high","speaker":null,"is_sponsor":0},{"start_s":1266.4,"end_s":1272.08,"text":"school that's hilarious okay that was it it's up so far only 20 drives are","speaker":null,"is_sponsor":0},{"start_s":1270.64,"end_s":1276.88,"text":"showing up but Jake tried a thing which is a what a","speaker":null,"is_sponsor":0},{"start_s":1274.159,"end_s":1280.64,"text":"domain change we changed some cables and i changed the jbods from three zone to","speaker":null,"is_sponsor":0},{"start_s":1279.52,"end_s":1287.84,"text":"one zone are you going to click it yes","speaker":null,"is_sponsor":0},{"start_s":1283.679,"end_s":1287.84,"text":"okay thank god okay one of these","speaker":null,"is_sponsor":0},{"start_s":1288.28,"end_s":1294.48,"text":"145 drives god damn brother i mean 45 drives is","speaker":null,"is_sponsor":0},{"start_s":1293.52,"end_s":1299.52,"text":"nice but 145 drives","speaker":null,"is_sponsor":0},{"start_s":1296.72,"end_s":1303.52,"text":"now that's walking around hard dress i'm just going to try this it's probably","speaker":null,"is_sponsor":0},{"start_s":1301.039,"end_s":1306.48,"text":"a bad idea oh i'm just going gonna do it i'm gonna try to import the pools it","speaker":null,"is_sponsor":0},{"start_s":1304.96,"end_s":1309.76,"text":"should find both of them there's two discreet pools on those existing drives","speaker":null,"is_sponsor":0},{"start_s":1308.4,"end_s":1314.96,"text":"yeah and there's a bunch of extra drives so let's see what happens what are the odds this is just gonna be like yeah","speaker":null,"is_sponsor":0},{"start_s":1312.64,"end_s":1318.48,"text":"your ancient ass pool and just well it's not they're not that ancient i think","speaker":null,"is_sponsor":0},{"start_s":1316.24,"end_s":1324.159,"text":"they've all been recreated now okay but actually the crazy thing is i took like","speaker":null,"is_sponsor":0},{"start_s":1321.28,"end_s":1328.559,"text":"2015 pools from a version of zfs that was not recommended for production and","speaker":null,"is_sponsor":0},{"start_s":1326.4,"end_s":1333.919,"text":"had not been updated and i put it in like 2022 ass trunas","speaker":null,"is_sponsor":0},{"start_s":1331.76,"end_s":1338.32,"text":"it just imported it just upgraded for some reason it was super slow like","speaker":null,"is_sponsor":0},{"start_s":1335.36,"end_s":1341.28,"text":"scrubs would take like a week which is really weird yeah but there could be","speaker":null,"is_sponsor":0},{"start_s":1339.76,"end_s":1345.44,"text":"kind of flaky drives it wasn't the drives the drives were fine i moved all","speaker":null,"is_sponsor":0},{"start_s":1343.12,"end_s":1348.799,"text":"the data off deleted the pool made it and then brought it back and it was fine","speaker":null,"is_sponsor":0},{"start_s":1346.799,"end_s":1353.44,"text":"it might just be like a fragmentation thing or","speaker":null,"is_sponsor":0},{"start_s":1350.08,"end_s":1356.32,"text":"oh let's see did something and","speaker":null,"is_sponsor":0},{"start_s":1353.44,"end_s":1361.84,"text":"oh look at that go fast is the 12 drives i think got it i wish i saved these ids","speaker":null,"is_sponsor":0},{"start_s":1358.64,"end_s":1364.0,"text":"before i did this let's see what happens","speaker":null,"is_sponsor":0},{"start_s":1361.84,"end_s":1368.64,"text":"does this mean vault is just going to be like well i'm back baby temp vault will","speaker":null,"is_sponsor":0},{"start_s":1366.48,"end_s":1371.679,"text":"be back the read-only new vault because we're still recovering a few things","speaker":null,"is_sponsor":0},{"start_s":1369.84,"end_s":1375.44,"text":"that's why one of the jbods is still mia we'll be back just like that and i can","speaker":null,"is_sponsor":0},{"start_s":1373.679,"end_s":1381.52,"text":"just recreate new shares i mean the ip will be different but i can just recreate the shares","speaker":null,"is_sponsor":0},{"start_s":1378.88,"end_s":1385.12,"text":"zfs is cool man don't put that there this is a really nice shirt though oh is","speaker":null,"is_sponsor":0},{"start_s":1382.88,"end_s":1388.559,"text":"that from ltdstore.com it sure is it will be oh is it not out yet i don't","speaker":null,"is_sponsor":0},{"start_s":1387.039,"end_s":1394.0,"text":"know i don't think so have you announced it on rancho i don't think so 30 hours","speaker":null,"is_sponsor":0},{"start_s":1391.039,"end_s":1397.52,"text":"later wow the pool's finally imported holy crap","speaker":null,"is_sponsor":0},{"start_s":1396.0,"end_s":1402.08,"text":"super micro massive shout out thank you for sending","speaker":null,"is_sponsor":0},{"start_s":1399.36,"end_s":1406.96,"text":"over the gear it's amazing we did have a little bit of trouble with setting it up","speaker":null,"is_sponsor":0},{"start_s":1404.48,"end_s":1411.12,"text":"with three domains yeah it's for some reason they're only working in single","speaker":null,"is_sponsor":0},{"start_s":1409.12,"end_s":1414.159,"text":"zone mode that means all 90 drives are going over one cable i don't know why","speaker":null,"is_sponsor":0},{"start_s":1412.96,"end_s":1418.4,"text":"because we had it set up on the bench like a couple weeks ago with three zones","speaker":null,"is_sponsor":0},{"start_s":1416.24,"end_s":1421.36,"text":"anyway it's accessible we're ready to copy some files and hopefully that's","speaker":null,"is_sponsor":0},{"start_s":1420.0,"end_s":1426.08,"text":"something they can sort out either through helping us set up the configuration or through firmware yeah","speaker":null,"is_sponsor":0},{"start_s":1424.08,"end_s":1429.2,"text":"it's probably just we're missing something this is one of those solutions","speaker":null,"is_sponsor":0},{"start_s":1427.919,"end_s":1434.559,"text":"where it's kind of like you just know what you're doing if you're doing it you're an integrator who","speaker":null,"is_sponsor":0},{"start_s":1431.84,"end_s":1440.159,"text":"sets these things up every day week after week and that's not us but i","speaker":null,"is_sponsor":0},{"start_s":1437.52,"end_s":1444.32,"text":"copied here i copy the file let's do it look i'm copying a file from it to this","speaker":null,"is_sponsor":0},{"start_s":1442.4,"end_s":1448.159,"text":"like hundreds of megabytes a second it's not the most impressive scenes ever this","speaker":null,"is_sponsor":0},{"start_s":1446.4,"end_s":1452.159,"text":"is a SATA SSD and we're screen recording to it now so","speaker":null,"is_sponsor":0},{"start_s":1449.76,"end_s":1457.919,"text":"it's like ah but hey remember guys this is with","speaker":null,"is_sponsor":0},{"start_s":1455.2,"end_s":1464.559,"text":"ancient like degraded degraded-ass fragmented v-devs once we strip all the","speaker":null,"is_sponsor":0},{"start_s":1462.159,"end_s":1468.559,"text":"data off of them rearrange them reshuffle them set them up fresh and","speaker":null,"is_sponsor":0},{"start_s":1466.64,"end_s":1471.44,"text":"dump all the data back onto them we should be able to be in the neighborhood","speaker":null,"is_sponsor":0},{"start_s":1470.0,"end_s":1475.279,"text":"of a Gigabyte a second which is all we're going to have for for sure well yeah but that's all we're going to have","speaker":null,"is_sponsor":0},{"start_s":1474.0,"end_s":1479.679,"text":"yeah that's all we're going to do network interface it's 100 oh that's","speaker":null,"is_sponsor":0},{"start_s":1477.279,"end_s":1482.799,"text":"right it is but here look at that 147 drives never in actual production that's","speaker":null,"is_sponsor":0},{"start_s":1481.279,"end_s":1487.12,"text":"amazing and then we'll add 60 more and it'll be","speaker":null,"is_sponsor":0},{"start_s":1484.4,"end_s":1491.36,"text":"over 200. and then we'll add 50 more and it'll be 250. and then we'll add this","speaker":null,"is_sponsor":0},{"start_s":1489.039,"end_s":1495.12,"text":"segway to our sponsor i fixed its manta kit has a 112 steel bits and comes","speaker":null,"is_sponsor":0},{"start_s":1493.679,"end_s":1499.76,"text":"complete with standard bits like phillips and flathead as well as more","speaker":null,"is_sponsor":0},{"start_s":1497.12,"end_s":1504.559,"text":"niche options like pentalobe hex and tri-point bits it also includes two","speaker":null,"is_sponsor":0},{"start_s":1502.08,"end_s":1508.4,"text":"aluminum screwdriver handles a weighty quarter inch drive for full size","speaker":null,"is_sponsor":0},{"start_s":1506.4,"end_s":1512.08,"text":"applications and a smaller four millimeter for precision work both","speaker":null,"is_sponsor":0},{"start_s":1510.24,"end_s":1516.24,"text":"drivers feature magnetic bit sockets knurled grips and silky smooth spinning","speaker":null,"is_sponsor":0},{"start_s":1514.32,"end_s":1520.559,"text":"tops you can even use the toolkit's lid as a built-in storage tray best of all","speaker":null,"is_sponsor":0},{"start_s":1518.48,"end_s":1524.159,"text":"every ifixit tool is backed by a lifetime warranty so if something breaks","speaker":null,"is_sponsor":0},{"start_s":1522.08,"end_s":1530.24,"text":"they will replace it for free so visit ifixit.com forward slash LTT and pick up","speaker":null,"is_sponsor":0},{"start_s":1526.799,"end_s":1532.32,"text":"your manta kit today monta if you guys","speaker":null,"is_sponsor":0},{"start_s":1530.24,"end_s":1537.679,"text":"enjoyed this video maybe you'll enjoy the one where we tried to deploy wanik","speaker":null,"is_sponsor":0},{"start_s":1534.559,"end_s":1538.96,"text":"no no that was awful okay sure yeah sure","speaker":null,"is_sponsor":0},{"start_s":1537.679,"end_s":1543.84,"text":"to watch that or the the previous one where we just looked at jbots because they're cool","speaker":null,"is_sponsor":0}],"full_text":"it's all here look at this it's been almost a year since we discovered the major issue that effectively put our entire video archive server offline that's right all the full quality footage since i was still working at ncix gone temporarily after the painstaking process of recovering nearly all that data love you wendell from level1tx we finally have all the hardware in place so today's the day we get our vault back up and running we're talking sprinting we're talking better than before we're talking an entire 3.6 petabyte array managed by one single head server instead of the five servers that we have now and i promise it's gonna be awesome like the segway to our sponsor i fix it looking to expand your selection of tools ifixit has you covered learn more about the manta toolkit and its 12 what 112 steel bits what with two premium drivers learn all about it at the end of this video manta this is the one computer that is going to run the mother vault and the astute among you have probably already realized that we are not going to be able to put 3.6 petabytes worth of storage inside it unless it's from hogwarts it's magic wingardium leviosa in here no it's one guardian leviosa oh sorry excuse me no in fact there are no hard drives in here this is where the magic happens the machine you saw just now that's only the brain so instead of trying to put 3.6 petabytes of hard drives into there they're gonna go in separate enclosures like this one called j buds or just a bunch of disks and you can think of an external jbot enclosure kind of like an external hard drive dock except instead of one hard drive they hold up to a whopping 90 hard drives each we'll talk more about these a little bit later just because the 2024 us-trt from supermicro is shirking its storage responsibilities doesn't mean that it's not packing inside here we find a dual socket AMD epic milan system that honestly has a bit of an unusual configuration typically a system with this much CPU horsepower so those are two 24 core AMD epic 74 f3s and the ability to run up to dual 64 cores would be decked out with a ton of NVMe slots or some enterprise gpus or something like that but this guy only has four NVMe slots and these are an optional accessory the rest of the storage up here and in fact these four slots by default are i kid you not six gigabit SATA three and a half inch slots like that you would use for a hard drive that's because this system is actually designed with a different purpose in mind say for example uh as a virtualization server where you might need a ton of CPU that's going to be split up to different clients or if you're like us a ton of CPU that you're going to use to drive a boatload of connected storage that isn't inside the chassis so instead of using up all those express lanes from those epic cpus which is buddy 128 lanes each with like NVMe drives in the front we instead want them allocated to PCIe slots in the back of the server back there we could install high-end networking adapters what is this dual 100 gigabit this guy right here or the other piece of our puzzle host bus adapters just like with an external hard drive our jbods back there which the keynight among you will have noticed that there are two require some kind of connection to the actual server with the actual CPU in it and the actual operating system running on it in a modern jbod this is usually handled by mini sas hd cables like these ones from infinite cables we're going to have these linked down below so each of these ports on the back of our three cards can handle four lanes of 12 gigabit sas that's the enterprise big brother of SATA for a total of 48 gigabits per second per cable your average computer even server probably doesn't have any factory mini sas hd ports so we're going to be adding them to this machine with three of broadcom's 9500 16e host bus adapters that means we'll have a full dedicated card for each of our j-bonds oh yeah there are three of these right there's one more j-bud that's actually already in the server room we will show you guys that one later now each of our jbods allows us to connect up to three cables for increased throughput so that's exactly what we're going to do on the subject of throughput to push this thing to the absolute limit we want to pack this machine with as much RAM as possible since the file system that we're using called zfs can use RAM as a read caching layer to significantly improve the responsiveness of the array because mechanical drives on their own yeah even with this many of them not that great in terms of access times luckily the folks over at micron just sent over one terabyte of 3200 megatransfer per second registered ecc memory which last time i checked smells like performance but the performance improvements don't stop there we had some leftover keoxia 30 terabyte NVMe drives from the dream server build sorry and we figured they would be the perfect choice to configure as a special metadata device in our zfs pool this is a game changer for performance because it allows metadata like where the files are physically stored on those hundreds of hard drives the allocation tables and optionally even small complete files to be stored on a discrete higher performance device like these NVMe drives this can make navigating through large data structures like the thousands of video projects that we're archiving much much faster and that directly results in workflow improvements for our staff i'd say it's about as much of a quality of life improvement as our flow plane subscribers get with no baked in ads higher quality streaming and exclusive behind the scenes content we're going to be setting these up in a mirrored stripe leaving us with a total of 60 terabytes of capacity which is hilarious given that it's a raid 10 of just four drives which we think should be an ample amount of fast metadata storage for our 3.6 petabyte array unfortunately because we only have these four NVMe slots we're not going to be able to expand that storage unless higher capacity drives come along in the future and the performance compared to some of our other all NVMe arrays is not going to be that amazing another problem is that this doesn't leave us any additional space for NVMe storage drives for a level 2 arc reed cache or an intent log at least at the front but there's a solution to that huh yes this cute little half height PCIe card from liquid allows us to install four m.2 drives in a single slot inside the server and this will be absolutely perfect for that roll what's the total capacity of this bad boy i don't actually know what ssds are in there we don't know what's in it there is one we have that is four or four terabyte ssds okay so it might be this one it could be this one it could also not be this one great the problem for later Jake this is pretty cool Jake did you design this three and a half to two and a half inch adapter thing no we found it on thingiverse oh cool printed it shut out thingiverse whoa what well that came out which slot did you want the liquid card in uh you gotta change this guy and then put it right here oh man can we find out if it's the right one before we do this because we're going to have to pull the whole thing off what do you mean what's the right one the the high capacity one oh no oh come on there's got to be a test bench not far no there's none over there there's none over there Jake says there is none well let's just go to my bench upstairs logistics i think it's important for you guys to know that Jake thinks you guys are utterly incompetent that's what he said whoa those are his words not mine he said those guys don't even have a test bench over there at all not one yeah that's what i thought yeah yeah Jake mere minutes later all right hey we got a foreign dynamic disc here these are what three terabytes okay right click on that one let's see if it's toshiba three terabytes 84. yeah perfect these are the four terabyte ones thank you fast and competent logistics team i just want you guys to know that i defended you i was the first to say no i would never use those words about our logistics team not like Jake goodbye get out of here ah here's what we've got next now i mentioned that this was 100 gigabit but what i didn't say i don't think is that it has dual 100 gigabit ports it can do a total of 200 gigabit it's PCIe gen 4 by 16 that is a flipping ton of bandwidth and it uses these sick qsfp28 connectors while Jake orders his lunch i'm going to go ahead and get our other two hbas installed here i'm getting mucho burrito editor you better put this in the goddamn cut okay i'm just gonna put this in because you're taking too long what do you mean i'm taking too long i'm not the one sitting here ordering lunch touche this is amazing so we've got our three hbas we've got our 100 gig network card we've got our additional SSD which is somewhere in the neighborhood of around 15 terabytes of storage for our slog and level 2 arc well they're 3.84 each oh i see so like somewhere in that 15 point something oh and how do you forget about that i forgot about these what i forget 120 terabytes of NVMe in the front yeah yeah are we ready to turn it on i'm turned on okay i could be more turned on if i was chilling for lttstore.com hey check out the RGB diode shirt it's like based on what the individual sub pixels look like but then it's also just a cool design so do you want the top yeah do a peel very graceful Jake you're doing a fan oh oh that's right i'm a monster this thing is sick do we get to power it on now got a laptop a laptop uh high this one is easy just wait it's actually really quiet give it a little bit of time are you gaslighting me sir no i would never bmw oh that is so much better and can we turn off that alarm as well i have a solution i don't think that's going to work it does pulling the power supply out right makes the alarm for oh wow it didn't work it worked this is a 1600 watt power supply what i mean to be clear that's only a 200 volt input it's a thousand watts if you're from 100 227 volts that's what you wanted to do i want to go fast you said it was quiet so i thought full fan speed must not be that bad right i had him log me separately into the ipmi just so i could do that in other news it's booted we got our terabyte of RAM we got our oh yeah we got our melanox card in there that's good did you put the nvmes in the right slot i mean one of them is wrong what do you mean wrong like oh it's wrong try the bodies one of those is a boot drive might it might be good there was an activity flash yeah right okay it's cool and look because the boot device is a pool just immediately fine i mean it's probably mad actually this is going to be kind of complicated because some of the drives we like need to put in the right spot and not mix up because some of the arrays we're keeping and some of the drives are in another jbod that wendell's working on right now so we can't really touch those ones uh-huh it's gonna be fun let's go start carrying how am i supposed to move this this is a four-man carry it literally has a label on it that says four people need to keep it actually yeah dude these are like the best earmuffs put them on oh wow now with 80 percent less Jake i can endorse these we don't actually have to take all of these out we could just take this one and that one and then just move this so the head server will just go here and then we'll go one so we don't even have to move this one so we just have to take the temp vault out should i let the editors know temp fault's going away uh i already told them it was going to happen today how do you tell them it's happening now i'll tell them what's happening now hello hardworking and truly valued editing team the vault that you have come to know and love temp vault is now deadvault press f to pay respects they don't care at all all right bye kind of raises the question do they need a vault at all then why don't we just have no vault then forget it if you guys aren't even going to miss it then what do you mean you copied the files you guys aren't supposed to keep things on your local machines we copied it to the active server new new new new monik hopefully this clears everything up and is not confusing at all i think they're more confused look somebody is reading files from it right now like ed was lying look oh somebody's is reading from it or they're writing to it what are they doing oh my god at 350 megabytes a second this is a not insignificant data transfer the chart's like nothing and then somebody's like you want to shut it off now okay who's messing with the server are you messing with us ed somebody is vaulting right now did you copy things to the vault right now probationary employee number 17. you cut and paste it i should not i use the tara terra copy but it's still going it's still kind of taste with territory dang it guys it's such a good you're done you're good god damn it ignore the the discs with errors uh what there's um we've lost a few the original vaults 10 terabyte drives are still like in production and they're like six years old now and we've had problems too our cooling in that room didn't used to be super stable there were times when you would walk in there and it was like a wall of heat yeah because we had this air conditioning unit that would just whatever when we didn't have an air conditioner yeah yeah at one point i remember when i started here atlantis was like no we don't need it we don't need it and then all the ups exploded because it was too hot and he was like okay fine how are we going to move these fortunately these aren't really rails they're actually like shelves so all we really need to do is kind of put it on the ground and then just tilt one end and then slide it in and tilt the other end that's how we put the other one in there installation by two persons is recommended if these need two people that server needs 25. i think i can handle it i wonder when my burrito's going to arrive all the drives are going to have to come out of this thing before we lift it out no no it's fine how are we supposed to lift this we can lift it we've done it okay well how it's drunk no really though it's not off the ground i just use my feet like this usually your feet yeah and then you just walk it up walk it out yeah three two i'm holding the heavy side don't drop it okay i didn't drop it wow you just put all the weight on my what was on your foot cool well that's all the hard work yeah now that we've got lab two over there we might actually have a purpose for all these additional storinators oh man i haven't gone spelunking in the uh server rack in ages Jake there's so much treasure in here you're giving everyone quite the view right now whatever usually you got to pay extra for that onlyplanes.com how do you like this setup i'm taking a little server cabinet nap here got my little uh pad on the back of my head okay bottom of the fifth not bad this was definitely the best way to do this the front of this shelf is out yeah that's what it appears uh that's why we screw how full is this uh it's got 60 drives in it okay what seems to have happened with this server because we didn't screw the shelf in from the front is this little tooth here that holds it into place seems to have slipped so it's sagging down like that which means that a ton of this weight is being borne by a part that's really not designed for that complicates our operation here a little bit because we can't just pop it back into place from here we can't really get at it looks like we need to shut this down pull it off put it back into place screw it in and then put it back in which sucks if i take the weight and you slide it forward or the other way around here i'll use my leg up there we go it's up just put the put the rail in don't move wait put what rail in well i can't see it oh my god like this oh i think we got it i think we got it let go let go yeah this actually went pretty okay situations like this are when i'm so glad we developed this screwdriver i got the nice strong magnet holding it in place almost no back force required to spin the ratchet like is it in there easy yeah it's in oh my god wait what how are we supposed to get it from here down to there we got it up there somehow so it's just like a really heavy storenator why don't we just put the other one on top that says this cart can take 600 pounds no we're not doing that okay that's right what are the odds uh that it just goes right in no the angle is still too much okay i'm getting under it boys put the right side in a bit more okay good good here oh my god it's it yeah it's on it's on the top okay go ahead you can push it in hi all right now for the hard one yeah this is more of a Linus sized server yolo well it's easy to yolo when somebody else already did all the work yeah good work Jake there goes the brain and what's really wild is that if we want more capacity more than 90 180 270 drives with the cards we have in now we could do another one and then i think there's room for one more card right i would say right now with no hbas we could probably add two and it would still be pretty good performance um all right let's uh let's put in the hard one ready one two three no yes what weak all right still weak now that they're in there working on these is glorious you just press the buttons pull them out boom funny story when we first put this in i was like ah it's so heavy it doesn't really need to be bolted down does it i was told yes in fact it does need to be bolted down and this is why because if we had a handful of these hanging off the front of it and it wasn't bolted into the concrete we'd have a bit of a problem bit of an update here we shucked all those drives put them in the servers i plugged a bunch of stuff in some power and we have blinky lights now do i get to press the button yeah it's on the right side give them a sec everybody in the next square mile is going to hear this they're just they're they're spinning up boys wow it's actually not as bad in here come outside let's uh let's shut the door yeah sure you can hear how awful it is still through the door oh yeah we wouldn't be able to shoot with those things going but they don't go okay that's fine oh you just got to add another layer of insulation there's no space Jake well this is the backup plan right because there's a thermostat in here that will start up a fan that exhausts out the top and will pull air passively through this grill in the event that the air conditioning completely fails do you remember that the shroud for this thing i made that in metal shop when i was still in high school that's hilarious okay that was it it's up so far only 20 drives are showing up but Jake tried a thing which is a what a domain change we changed some cables and i changed the jbods from three zone to one zone are you going to click it yes okay thank god okay one of these 145 drives god damn brother i mean 45 drives is nice but 145 drives now that's walking around hard dress i'm just going to try this it's probably a bad idea oh i'm just going gonna do it i'm gonna try to import the pools it should find both of them there's two discreet pools on those existing drives yeah and there's a bunch of extra drives so let's see what happens what are the odds this is just gonna be like yeah your ancient ass pool and just well it's not they're not that ancient i think they've all been recreated now okay but actually the crazy thing is i took like 2015 pools from a version of zfs that was not recommended for production and had not been updated and i put it in like 2022 ass trunas it just imported it just upgraded for some reason it was super slow like scrubs would take like a week which is really weird yeah but there could be kind of flaky drives it wasn't the drives the drives were fine i moved all the data off deleted the pool made it and then brought it back and it was fine it might just be like a fragmentation thing or oh let's see did something and oh look at that go fast is the 12 drives i think got it i wish i saved these ids before i did this let's see what happens does this mean vault is just going to be like well i'm back baby temp vault will be back the read-only new vault because we're still recovering a few things that's why one of the jbods is still mia we'll be back just like that and i can just recreate new shares i mean the ip will be different but i can just recreate the shares zfs is cool man don't put that there this is a really nice shirt though oh is that from ltdstore.com it sure is it will be oh is it not out yet i don't know i don't think so have you announced it on rancho i don't think so 30 hours later wow the pool's finally imported holy crap super micro massive shout out thank you for sending over the gear it's amazing we did have a little bit of trouble with setting it up with three domains yeah it's for some reason they're only working in single zone mode that means all 90 drives are going over one cable i don't know why because we had it set up on the bench like a couple weeks ago with three zones anyway it's accessible we're ready to copy some files and hopefully that's something they can sort out either through helping us set up the configuration or through firmware yeah it's probably just we're missing something this is one of those solutions where it's kind of like you just know what you're doing if you're doing it you're an integrator who sets these things up every day week after week and that's not us but i copied here i copy the file let's do it look i'm copying a file from it to this like hundreds of megabytes a second it's not the most impressive scenes ever this is a SATA SSD and we're screen recording to it now so it's like ah but hey remember guys this is with ancient like degraded degraded-ass fragmented v-devs once we strip all the data off of them rearrange them reshuffle them set them up fresh and dump all the data back onto them we should be able to be in the neighborhood of a Gigabyte a second which is all we're going to have for for sure well yeah but that's all we're going to have yeah that's all we're going to do network interface it's 100 oh that's right it is but here look at that 147 drives never in actual production that's amazing and then we'll add 60 more and it'll be over 200. and then we'll add 50 more and it'll be 250. and then we'll add this segway to our sponsor i fixed its manta kit has a 112 steel bits and comes complete with standard bits like phillips and flathead as well as more niche options like pentalobe hex and tri-point bits it also includes two aluminum screwdriver handles a weighty quarter inch drive for full size applications and a smaller four millimeter for precision work both drivers feature magnetic bit sockets knurled grips and silky smooth spinning tops you can even use the toolkit's lid as a built-in storage tray best of all every ifixit tool is backed by a lifetime warranty so if something breaks they will replace it for free so visit ifixit.com forward slash LTT and pick up your manta kit today monta if you guys enjoyed this video maybe you'll enjoy the one where we tried to deploy wanik no no that was awful okay sure yeah sure to watch that or the the previous one where we just looked at jbots because they're cool"}