1 00:00:00,440 --> 00:00:06,200 behind me right now is the biggest 2 00:00:03,640 --> 00:00:11,160 supercomputer in the country it'll be serving researchers across Canada 3 00:00:09,240 --> 00:00:16,440 studying the human genome in bioinformatics particle physics 4 00:00:13,320 --> 00:00:19,039 materials research even humani research 5 00:00:16,440 --> 00:00:23,480 it's called Cedar it cost the federal government through the Canadian 6 00:00:20,439 --> 00:00:26,560 foundation for Innovation over $16 7 00:00:23,480 --> 00:00:28,960 million and we get to be the first to 8 00:00:26,560 --> 00:00:31,960 unbox this Beast 9 00:00:40,320 --> 00:00:49,399 Savage jerky is created without the use of nitrates or preservatives use offer 10 00:00:46,000 --> 00:00:50,600 code LTT to save 10% at the link in the 11 00:00:49,399 --> 00:00:59,199 video description so Cedar is a big data 12 00:00:54,760 --> 00:01:01,640 machine it takes up a quar of the 5,000 13 00:00:59,199 --> 00:01:07,360 ft Data Center it occupies meaning actually that there's room for it to 14 00:01:03,480 --> 00:01:11,080 grow but right now it has 15 00:01:07,360 --> 00:01:18,080 27,000 Intel Zeon processing course 16 00:01:11,080 --> 00:01:24,000 190 terab of RAM 64 petabytes of storage 17 00:01:18,080 --> 00:01:26,159 584 gpus and a total power draw of 18 00:01:24,000 --> 00:01:32,399 560,000 wats though with that said it's 19 00:01:29,360 --> 00:01:37,200 efficiency is a shocking 20 00:01:32,399 --> 00:01:39,280 1.07 on the Pue scale where one would be 21 00:01:37,200 --> 00:01:44,520 perfect and a typical data center would be 1 and 1/2 to two we'll get into how 22 00:01:42,280 --> 00:01:49,439 they did that a little bit later though so our tour starts right here behind me 23 00:01:47,079 --> 00:01:55,920 are what they call the high availability racks so everything back there has dual 24 00:01:53,520 --> 00:02:02,079 power supplies for redundancy with a battery backup for that and a diesel 25 00:01:58,600 --> 00:02:04,680 generator back up that everything back 26 00:02:02,079 --> 00:02:11,480 here is Mission critical things like networking login servers and management 27 00:02:07,880 --> 00:02:14,680 servers are all here and this is also 28 00:02:11,480 --> 00:02:17,160 where you'll find the bulk of Cedar's 29 00:02:14,680 --> 00:02:22,959 storage let's get in for a closer look at Cedar's connection to the outside 30 00:02:19,440 --> 00:02:27,680 world this networking Appliance from 31 00:02:22,959 --> 00:02:32,599 Huawei has a street price of around a 32 00:02:27,680 --> 00:02:35,480 million doll W and right here this is 33 00:02:32,599 --> 00:02:41,800 where it gets really bananas these guys are seedar 34 00:02:37,400 --> 00:02:44,599 dual 100 gabit connections through 35 00:02:41,800 --> 00:02:52,080 Vancouver and then as if that wasn't enough these orange ones here are dual 36 00:02:48,680 --> 00:02:54,680 40 gbit connections through nearby sui 37 00:02:52,080 --> 00:02:58,400 just in case somebody puts a back ho through one of these other fiber lines 38 00:02:56,920 --> 00:03:00,879 and they would have otherwise lost their internet connectivity I mean that's 39 00:03:00,080 --> 00:03:08,000 their Backup backup but Ethernet is not really the 40 00:03:05,920 --> 00:03:17,799 way you want to connect high performance Computing nodes this this right here is 41 00:03:12,080 --> 00:03:21,560 the true networking heart of Cedar these 42 00:03:17,799 --> 00:03:23,319 are 48 Port omn path switches and 43 00:03:21,560 --> 00:03:32,519 they're configured in what's called an island topology so the island is in 44 00:03:26,959 --> 00:03:34,760 almost all cases 32 compute nodes each 45 00:03:32,519 --> 00:03:42,640 of those compute nodes is connected to 32 ports on one of these switches in its 46 00:03:38,680 --> 00:03:46,720 rack then the remaining 16 ports come 47 00:03:42,640 --> 00:03:51,239 back to here that means that every 48 00:03:46,720 --> 00:03:54,519 island gets a dedicated line to each of 49 00:03:51,239 --> 00:03:57,159 the core switches giving you failover 50 00:03:54,519 --> 00:04:04,280 and massive bandwidth each one of these fiber links 51 00:04:00,640 --> 00:04:07,640 right here is capable of 100 gbit per 52 00:04:04,280 --> 00:04:11,920 second so even though between islands we 53 00:04:07,640 --> 00:04:13,560 are let's say bottlenecked by our 16 54 00:04:11,920 --> 00:04:19,720 connections so that's only half the total theoretical speed within an island 55 00:04:15,959 --> 00:04:22,600 we're still talking over 100 gabyt per 56 00:04:19,720 --> 00:04:29,280 second so it's not really an issue okay now let's move on to SFU and 57 00:04:26,360 --> 00:04:35,160 compute Canada's version of petabyte project spoiler alert theirs is better 58 00:04:32,240 --> 00:04:43,080 in every conceivable way so in the five cabinets behind me we've got Cedar 50 59 00:04:39,199 --> 00:04:46,240 petabyte IBM tape Library System they 60 00:04:43,080 --> 00:04:50,560 have a 40 gabit link to the rest of the 61 00:04:46,240 --> 00:04:53,320 supercomputer and each of the 5,000 10 62 00:04:50,560 --> 00:05:00,039 tbte magnetic tapes inside can be grabbed out of storage moved with like a 63 00:04:57,039 --> 00:05:01,960 robotic ARM into a reader and the data 64 00:05:00,039 --> 00:05:05,680 can then be accessed when needed and this is done 65 00:05:03,639 --> 00:05:12,000 automatically cool right okay yeah but due to the slowness 66 00:05:08,840 --> 00:05:15,199 of that swapping process this is still 67 00:05:12,000 --> 00:05:16,919 what we would consider to be cold or 68 00:05:15,199 --> 00:05:24,039 archival storage next up here is general purpose 69 00:05:20,440 --> 00:05:27,080 storage land where any data that's being 70 00:05:24,039 --> 00:05:30,720 used for any current research project 71 00:05:27,080 --> 00:05:35,360 would be housed so here here they're 72 00:05:30,720 --> 00:05:37,080 using offthe shelf five U racks Each of 73 00:05:35,360 --> 00:05:43,039 which contains let's see if we can crack One open here a total of two kind of 74 00:05:40,400 --> 00:05:53,120 trays here and 84 8 terab uh let's have a look here 75 00:05:48,680 --> 00:05:55,639 Enterprise capacity SAS drives from 76 00:05:53,120 --> 00:06:02,120 Seagate but there's actually more to this system than meets the eye every 77 00:05:58,720 --> 00:06:05,639 four of of these storage nodes requires 78 00:06:02,120 --> 00:06:08,039 two nodes of what they're calling object 79 00:06:05,639 --> 00:06:15,280 storage servers these act as a high-speed cache with their SAS 10,000 80 00:06:11,080 --> 00:06:18,400 RPM drives as well as as kind of like a 81 00:06:15,280 --> 00:06:21,919 a a traffic cough for everything behind 82 00:06:18,400 --> 00:06:24,840 it so every single read or write to 83 00:06:21,919 --> 00:06:30,759 these hard drives actually goes through these nodes so right now General Storage 84 00:06:28,360 --> 00:06:38,720 Land is 10 ped byes but in the near to Mid future it will be expanding to 20 85 00:06:35,080 --> 00:06:41,479 20 now that DIY approach to storage is 86 00:06:38,720 --> 00:06:45,680 great for scaling up at a low cost but when it comes to Performance they went 87 00:06:43,440 --> 00:06:54,400 for this data direct network storage Appliance because it has got the real 88 00:06:50,000 --> 00:06:58,759 Goods now in the rack next to this brain 89 00:06:54,400 --> 00:07:01,479 you'll find a mere 4 pedabytes of actual 90 00:06:58,759 --> 00:07:06,000 storage due to its higher cost but thanks to its proprietary Hardware 91 00:07:03,400 --> 00:07:13,919 custom software and solid state burst buffers this thing can handle up to 40 92 00:07:10,319 --> 00:07:16,960 gabt per second of sustained throughput 93 00:07:13,919 --> 00:07:19,759 making it perfect for data intensive 94 00:07:16,960 --> 00:07:26,960 applications that rely on humongous data sets now let's get into 95 00:07:23,879 --> 00:07:30,199 compute there are about half a dozen 96 00:07:26,960 --> 00:07:32,800 different types of compute nodes all 97 00:07:30,199 --> 00:07:38,080 connected to the same high-speed Omni paath Network backbone that are 98 00:07:34,879 --> 00:07:41,720 optimized for different types of 99 00:07:38,080 --> 00:07:45,240 research we'll begin with the base 100 00:07:41,720 --> 00:07:49,479 compute note there are a whopping 101 00:07:45,240 --> 00:07:52,159 576 of these each of these is a computer 102 00:07:49,479 --> 00:07:59,840 so there's actually four in a single toou shell Each of which contains two 103 00:07:55,039 --> 00:08:02,800 Zeon E5 2683 16 core processors 128 gigs 104 00:07:59,840 --> 00:08:10,800 of RAM and about a terabyte of raid zero SSD storage for Scratch so each rack 105 00:08:06,879 --> 00:08:14,599 here contains two islands so that's a 106 00:08:10,800 --> 00:08:15,840 total of 64 compute nodes giving us a 107 00:08:14,599 --> 00:08:23,840 whopping 2,48 compute units per rack so these 108 00:08:20,400 --> 00:08:26,520 nodes are the basic Workhorse of Cedar 109 00:08:23,840 --> 00:08:32,000 handling everything from Monte Carlo simulations for Material Science to sim 110 00:08:29,639 --> 00:08:38,719 ating Dynamic processes in nature with a high degree of Randomness like snowfall 111 00:08:34,880 --> 00:08:42,000 or rainfall they would also be used in 112 00:08:38,719 --> 00:08:44,480 any highly parallelized workload because 113 00:08:42,000 --> 00:08:49,480 if you need you know 10,000 CPU cores for one job there 114 00:08:47,399 --> 00:08:55,800 aren't enough cores in any other class of server to handle that kind of 115 00:08:52,839 --> 00:09:01,880 load moving right on up we've got the big memory nodes there are 48 of these 116 00:08:59,079 --> 00:09:09,600 and half of them are just like the basic nodes except with 512 gigs of RAM while 117 00:09:06,240 --> 00:09:15,200 the other half of them these puppies 118 00:09:09,600 --> 00:09:18,399 have 1 and half terabytes of system 119 00:09:15,200 --> 00:09:21,399 memory these ones take up twice as much 120 00:09:18,399 --> 00:09:24,680 rack space though each of these one use 121 00:09:21,399 --> 00:09:27,079 is a single dual socket system because 122 00:09:24,680 --> 00:09:35,320 you know what there just wasn't enough go darn room for old 24 64 gig memory 123 00:09:32,120 --> 00:09:37,560 modules that are required for that much 124 00:09:35,320 --> 00:09:43,959 RAM first world problem 125 00:09:39,760 --> 00:09:47,480 yes these guys are really special these 126 00:09:43,959 --> 00:09:50,320 are the aptly named 3 tbte nodes there 127 00:09:47,480 --> 00:09:58,959 are only a handful of them but these are quad socket machines with Zeon 128 00:09:53,600 --> 00:10:01,640 4809 v4s four of them but wait a tick 129 00:09:58,959 --> 00:10:06,920 those are only eight core processors these don't even have more processing 130 00:10:04,560 --> 00:10:13,640 cores than those little tiny ones that take up half of you what's the deal here 131 00:10:10,600 --> 00:10:16,160 well it turns out that some 132 00:10:13,640 --> 00:10:22,200 bioinformatics workloads like genome sequencing don't actually scale very 133 00:10:19,000 --> 00:10:24,720 well with more processors they just need 134 00:10:22,200 --> 00:10:30,519 massive amounts of memory to hold the data sets that they need to work on so 135 00:10:28,360 --> 00:10:37,000 while the team here probably isn't super stoked on using up for use just so they 136 00:10:34,399 --> 00:10:41,720 can stuff more memory into the system until until optane reaches a higher 137 00:10:39,639 --> 00:10:47,440 level of maturity this is the only choice they have now finally we're 138 00:10:44,720 --> 00:10:56,800 getting to my favorite nodes the most expensive nodes these are the GPU nodes 139 00:10:53,720 --> 00:10:59,320 and while they're actually quite similar 140 00:10:56,800 --> 00:11:04,560 to the base nodes with respect to their CPU and RAM configurations what's got 141 00:11:02,959 --> 00:11:09,560 the researchers in the fields of molecular Dynamics Ai and machine 142 00:11:07,360 --> 00:11:16,519 learning all amped up about these are the quad NVIDIA Tesla p100 graphics 143 00:11:13,399 --> 00:11:19,120 cards that they have cram into each one 144 00:11:16,519 --> 00:11:24,240 I mean seriously with 1,500 watts of power being consumed by 145 00:11:22,480 --> 00:11:29,920 each one of these is it it is an engineering Marvel that they've crammed 146 00:11:27,040 --> 00:11:34,680 enough power and cool to make this whole thing work so 147 00:11:32,680 --> 00:11:40,680 actually now that you think about it how exactly did they do that so the Keen ey 148 00:11:38,360 --> 00:11:46,440 among you might have already caught a couple of hints earlier in this video 149 00:11:42,680 --> 00:11:49,959 but the secret lies in the rear doors on 150 00:11:46,440 --> 00:11:54,680 the server racks look how thick this is 151 00:11:49,959 --> 00:11:58,200 yes my friends this entire door is a 152 00:11:54,680 --> 00:12:00,800 gigantic heat exchanger so their servers 153 00:11:58,200 --> 00:12:04,880 don't actually have have water blocks that would be more expensive what 154 00:12:03,000 --> 00:12:08,920 they're doing is they've just got the fronts of the racks all sealed up so 155 00:12:06,839 --> 00:12:14,519 there's no Backdraft pressure and they've got normal air cooled servers 156 00:12:12,200 --> 00:12:19,199 that pass the air from the front where they just draw in room temperature air 157 00:12:16,360 --> 00:12:24,560 in here and it comes out hot like 30 plus de and push it through this heat 158 00:12:21,800 --> 00:12:29,959 exchanger where it is actually cool to my skin that's how efficient these are 159 00:12:28,279 --> 00:12:35,440 and that cooling system system is massively expandable too you can 160 00:12:32,240 --> 00:12:37,920 actually see above me I am standing 161 00:12:35,440 --> 00:12:46,120 where we've got a blue and green cooling pipe connected to a whole bunch of quick 162 00:12:42,040 --> 00:12:49,079 release fittings ready to add more racks 163 00:12:46,120 --> 00:12:52,360 right here but to see what they actually do with the heat we're actually going to 164 00:12:50,920 --> 00:13:01,880 have to go upstairs where we'll find the final and 165 00:12:57,279 --> 00:13:05,639 perhaps the uh coolest stop in our tour 166 00:13:01,880 --> 00:13:10,160 here this is the mechanical room where 167 00:13:05,639 --> 00:13:13,480 the pumps and these freaking pipes take 168 00:13:10,160 --> 00:13:17,120 all the water from downstairs and dump 169 00:13:13,480 --> 00:13:20,120 it into three cooling towers outside the 170 00:13:17,120 --> 00:13:21,880 building now right now the weather is 171 00:13:20,120 --> 00:13:28,880 favorable to cooling the ambient temperature is quite low so it's just 172 00:13:25,040 --> 00:13:31,480 operating as gigantic radiators but get 173 00:13:28,880 --> 00:13:37,440 this when the conditions become less favorable in the summer they kick things 174 00:13:34,399 --> 00:13:41,399 into high gear with an automated system 175 00:13:37,440 --> 00:13:44,320 that sprays water onto the fins of the 176 00:13:41,399 --> 00:13:48,760 radiators in the cooling towers and if you watched our bong cooling video which 177 00:13:47,079 --> 00:13:53,920 you can check out right here you'll be familiar with this concept already but 178 00:13:51,079 --> 00:13:59,639 this is called evaporative cooling and by these means even in ambient 179 00:13:55,959 --> 00:14:04,279 temperatures up to 30° C 180 00:13:59,639 --> 00:14:07,560 they can achieve the 17° coolant levels 181 00:14:04,279 --> 00:14:10,880 that they need to without employing the 182 00:14:07,560 --> 00:14:15,759 massive Chiller unit that they have over 183 00:14:10,880 --> 00:14:15,759 on the other side of the 184 00:14:17,199 --> 00:14:23,839 room Squarespace is the way to build a 185 00:14:21,519 --> 00:14:29,399 website whether it's for your small business or for your you know uh local 186 00:14:27,759 --> 00:14:35,000 freaking book club it does doesn't matter if you want a web presence 187 00:14:31,800 --> 00:14:36,880 affordably and quickly Squarespace gets 188 00:14:35,000 --> 00:14:42,120 it done for you it starts at just 12 bucks a month and you get a free domain 189 00:14:39,440 --> 00:14:47,680 if you buy Squarespace for the year you just pick one of their templates and 190 00:14:44,800 --> 00:14:53,920 boom you upload some pictures you fill in some text it's all cloud-based and 191 00:14:50,680 --> 00:14:57,440 your website will simple as that look 192 00:14:53,920 --> 00:14:59,959 great on any device every website comes 193 00:14:57,440 --> 00:15:05,279 with a free online store and their cover pages feature allows you to set up a 194 00:15:02,320 --> 00:15:10,759 beautiful one-page online presence in just minutes so start a trial with no 195 00:15:08,720 --> 00:15:16,560 credit card required and start building your website today then when you decide 196 00:15:14,120 --> 00:15:23,000 to sign up for Squarespace don't forget head over to squarespace.com 197 00:15:19,160 --> 00:15:24,399 LTT and use offer code LTT to get 10% 198 00:15:23,000 --> 00:15:30,759 off your first purchase so a massive thank you to SFU 199 00:15:27,839 --> 00:15:35,079 and compute Canada for allowing us to uh run a muck in their data center thanks 200 00:15:33,120 --> 00:15:38,160 to you guys for watching if you dislike this video you know what to do but if 201 00:15:36,399 --> 00:15:41,839 you liked it hit that like button get subscribed maybe consider checking out 202 00:15:39,639 --> 00:15:45,399 where to buy the stuff we featured at the link in the video description also 203 00:15:44,120 --> 00:15:50,759 down there you'll find a link to our merch store which has cool shirts like this one and our community Forum which 204 00:15:48,720 --> 00:15:53,759 you should totally join