just yank it ah come on you bastard
oh wow now it's really mad whole thing's still running
meet z16 the latest member of ibm's
mainframe family that's right not only do mainframes still exist
powering high frequency transactional industries like global banking and
travel but as of today you can get a brand spanking new mainframe configured
with up to 256 cores built on the latest seven
nanometer node from samsung and with up to 40 terabytes of memory
very few outsiders have ever been invited to ibm's 100 000 square foot z
test lab so we're gonna pack as much as
we can into the very short time that we get to spend on site today just like i
pack our videos with sponsors ridge wallet time to ditch that bulky wallet
ridge wallet can hold up to 12 cards comes with a cash strap or money clip
and is available in tons of different colors use the code Linus to save 10 and
get free worldwide shipping at the link below
the core pillar z is right in the name it's short for zero like
zero downtime and their target is seven
ninths of reliability or about three
seconds of down time a year on average and to achieve that takes an
architecture that's built from the ground up for resiliency
now out of the hundreds of z systems that they're beating the heck out of
here in the lab there's just one next-gen z16 that they got all gussied
up for us with the fancy covers and everything only for us to immediately
ignore them and start digging around in its guts
starting at the bottom i initially assumed that these heavy black chunguses
here were cooling lines but as it turns out z16 does not support being plumbed
into a building-wide cooling system instead they opted for a self-contained
system where every rack that has a compute drawer gets this pump and
radiator unit down here at the bottom if you look closely you can actually see
the two redundant cooling pumps that's in case one fails there's actually also
an empty spot where it looks like one of the pumps is missing that's because in
the validation phase they weren't sure if these pumps were going to meet their
reliability standards so the plan was to just tuck three of them in there just in
case these are actually power lines then routed through the three foot crawl
space that's all underneath us covering the entire test facility and each of
these two is capable of carrying 60 amps
of 208 volt three-phase power on this particular config
power that needs to be distributed that's where these come in each of these
power distribution units or pdus down the side here are meant to be fed from a
separate breaker in the event of a power loss and basically you can think of
these as the enterprise grade equivalent of a power strip
so using the onboard Ethernet port a technician can monitor power usage turn
off particular plugs for maintenance or even update the firmware does your power
strip have upgradable firmware no
gross next up things get really spicy each z16
system can be configured with up to four of these compute drawers and each
compute drawer contains up to four of their new telum chips and these things
are really cool let's go take a closer look
what a monster telum is a chiplet design and you can
actually see the two separate dies here
it's got support for ddr4 and a total of 16 cores and 64 lanes of pci express per
socket which is neat but that tells only a fraction of the
story while a consumer x86 CPU might contain dedicated hardware for
let's say decoding popular video codecs
telum brings some very different specializations to the table
each one of these cores so there's eight per die has a coprocessor to accelerate
storage crypto and ibm's own compression
algorithm then each die gets its own
gzip compression accelerator and this is one of their biggest announcements an ai
accelerator the reason for bringing that on die was that customers like financial
institutions have these complex machine
learning models that they built for for example fraud detection
but data scientists care about accuracy
not necessarily the downstream performance impact so what they found
out was that they had these great models but they didn't have the performance to
actually apply them to every transaction or well rather they could apply them to
every transaction if they had all the time in the world but they don't
so if your bank has an sla or a performance guarantee of nine
milliseconds on a transaction and there's not enough time to apply their
fraud detection model they either have to just let it go fraudulent or not or
decline it risking the customer just pulling out the next card in their
wallet effectively giving that business to their competitor by putting the ai
processor on die they're giving it direct access to the transaction data
that is sitting in the CPU's cache rather than forcing it to go out to
memory for it and compared to last gen z ibm is figuring on a 20 to 30 x
improvement in performance with and this is critical better consistency and
accuracy the cash on this chip is really special too but to properly talk about
that we need to zoom out a little bit
now we're looking at a full compute drawer here each of which contains four
telum chips actually this one's only got three in it so far i'm assuming i have
this esd strap on so i can install the last one right is it this one wait so you guys handed
me a working chip before oh well that was your first mistake
okay uh where's the nifty install tool ah yes
okay cool so i've seen this demo once which should be more than enough for me
to perform it for you the telum chip goes in a little something like that
then this doohickey majig lines up with
the dots on the thing then you grab the chip
all right you make sure it's not going to come off over something soft ish
holds on to a little something like that we really hope the CPU doesn't come out
if it comes out in the socket this boy's done
won't right just have a bit of a reputation i don't
need it right now i don't need the sass from my camera operator here
then i poke this in here i think
i poke in there and then i squeeze to lock it
we'll unlock it and boom it's in
we give it a little wiggle heck yeah
now we're halfway the next thing we got to do is install
our cooling and this is super cool one
of the first questions i asked them when i walked up to this compute drawer was
what the heck kind of thermal compound is that my initial gut feeling was that it
looked like some kind of liquid metal
like i mean it's not like it's not being used in commercial products the playstation 5 is using liquid metal but
actually it's solid metal so rather than being like an indium gallium mix of some
sort this is just an indium thermal pad
and what's great is you can actually see how
there you go they've got the size just
exactly right to cover where the dyes are creating hot spots under the
integrated heat spreader so these are
going to be the interface
between the integrated liquid cooling system in the chassis
there we go and our processors
and i guess uh i can't even tell is the latch on it look kind of looks like it
where's my torque wrench or screwdriver oh is this it that's it look at that
it's all configured for me and everything yes this feels like a lot
i mean i guess that's a lot of pins but still
oh this really feels like a lot is that
okay putting that kind of pressure on that
kind of spicy expensive stuff is stressful
i'm sure you guys are used to it though this is adorable little 3d printed hose
holders do you know how many times i've been working on a system and i wished i
had one of these to hold a block out of the way while i'm doing stuff i love it
i don't like this screwdriver we're going to have to make a better torx
screwdriver for you guys we're working on our own screwdriver
lttstore.com yeah he knew i was going to say it
bloody hell that looks freaking awesome
like that is hardware porn right there if i've ever seen it
this four CPU config gives us a total of 64 cores in this drawer and a whopping
256 pci express gen 4 lanes but you
might have noticed the motherboard that they're installed on is really unusual
looking where are the memory slots
where are the vrms let's answer the second one first
this is called a poll card or a point of
load card they haven't showed me how to take it out but i'm sure i can figure it
out ah yes there we go these are super cool these take the bulk
12 volt power that comes from the power supplies over on the end here and step
it down to the approximately one volt that we need for the cpus and the dram
modules and the craziest thing about these is that they've got 14 power
phases each and they can be dynamically
assigned depending on where they are needed with two of them
actually being completely redundant so these three do all of the work and these
two including one of the ones that i took out
are doing absolutely nothing unless something fails i love seeing that old
school ibm ass logo on like cutting
edge total combined output not to exceed
660 amps okay i'll be sure not to
now they get some additional help from ah yes here they are come on out little
buddy there we go this is called a voltage
regulator stick or vrs it's smaller with
fewer phases but it contributes to step down for io cards and things like that
and then oh one other really cool thing i want to show you guys is the
oscillator card in the front of the oh i think this thing is locked there we go
i'm moving it in the front of the compute drawer here
we talked in more detail about what cards like this do in our time card
video but essentially they take an external signal and ensure that all the
machines within a data center are running at exactly the same time to
avoid wasting clock cycles and operate more efficiently now let's move on to
memory and this is where things get really funky because i ain't never seen
a memory module it looks anything like this thing there's at least three crazy
things going on here number one is that even though this is ddr4
some of the power delivery is actually on the module itself like we've seen
with ddr5 second up is the ic configuration we've
got 10 of these dram chips per side
per side for a total of 40 which doesn't
really correspond with any kind of ecc error correction that i've ever seen in
my life and it turns out that that's because it's not done on the module
each memory controller in the system actually addresses eight of these memory
modules and the ecc is handled more like
raid with the parity data striped across
the eight different modules it's all managed at the memory
controller level finally we've got this bad boy right
here what's this thing called again with the copper chipset explorer it's the
explorer this is a proprietary buffer chip that basically adds a latency hit
but allows the memory controller to address vast vast amounts of memory with each of
these systems able to handle 10 terabytes of ddr4 memory absolutely
flipping insane
now let's talk about why i needed all four of these chips to explain telum's
cache configuration where a normal CPU would have a tiny lightning fast level
one cache sitting right next to the processing course followed by a larger
slower level two and then a larger slower level three and so on and so
forth telum has an already aggressive by
consumer standards 256 kilobyte private
level one cash per core so they've got eight of those per die 16 per socket
that is then backed up by a whopping 32
megabytes of level 2 cash per core to
give you some idea of what this means in practical terms look at this die shot
this thing has more cash than compute by
area and it does away with level three cash
all together out the window now ibm's engineers probably could have
said well the level two cache is so big we probably won't need level three
anyway fair enough but they didn't
instead when a line needs to be evicted from one of the core's private level two
caches it will actually look for empty space in
another course level two and then mark that as virtual level three cache
now for that core to then fetch it from another course cache does add latency
compared to if it had been in its own level two right next to the core
but that would be true of a shared level three cache anyway and this approach
affords them an incredible degree of flexibility particularly for cloud
customers who might have very different hardware needs from one to the next
and it gets even crazier evicted lines can also be pushed off of
the chip to another CPU all together in
the z16 system allowing a single core to
conceivably have access to up to two
gigabytes of virtual level four cache
from the other cpus in the drawer and up to eight gigabytes if we go beyond
just the beyond i suppose i believe i died and went to
hardware heaven to my right is a fully
kitted out z16 and now we can both continue and actually expand on the tour
that we started before with that single rack unit
this one has all four possible compute drawers populated you can see there's
one here and there's actually three in the second rack here and in order to
connect them ibm uses these super cool
smp9 active cables these things are
super badass they've actually got heat sinks on them and inside the sheath
you'll find 10 plus one fiber for redundancy with each fiber carrying 25
gigabit per second for a total of 250 gigabit per second oh and of course
all of the links are redundant two of them between each one of the computers
now over these links the system allows the cpus and the separate drawers to
share that level two cache i talked about before as a virtual level four
which can apparently in some cases
actually be faster to draw off of this CPU's cache over smp9 versus pulling off
of your own dram in your own compute drawer
it's insane i'm just getting permission to pull a coupling link card out but i
realized we didn't even talk about the reservoir how sick is this
just want to fill up your mainframe no big deal got your like giant looks like
stainless steel res going on here
absolutely gorgeous i want 04
just data center things how's it going
look at these chunky quick connects man
that is sick dare me to pull the collar back it'll be it'll be fine it'll be fine you'd have
to actually link them together to let the liquid flow hey the lights on look
at this and a one
and uh i don't know sure like that hey
there we go suckas this is a coupling link card it's using multi-mode fiber 12
lanes per four redundant for short range
high bandwidth linking of multiple z systems within the data center and it
uses pci express the idea here is that
depending on the customer's resiliency strategy which is often government
mandated they could have a whole web of
z systems configured for high availability which could allow an entire
drawer to go down with immediate failover and no data loss that's why
these plug directly into the compute drawers for less bandwidth intensive i o
devices we rely on these you can fit so
much PCIe in this bad boy
codename hemlock makes them sound a little sexier than they are but they're
still pretty cool using these direct attached copper cables actually ah cool
i've got one of these right here they have a 16x pci link to the compute
drawer and then you can stack these puppies up
with a ton of i o cards nbne storage
crypto accelerators network cards i mean you can actually run Linux on these
things so pretty much whatever add-in board the customer desires can go in one of these
and the pci express link between the hemlock right here
and the compute drawer goes a little something like that
pretty cool now you guys might have noticed that only the center two racks
contain compute while the outer ones are
all i o in fact this one over here on the right is just an entire tower of io
that's normal and most of the cards that you're looking at in here are either
ficon or sysplex cards and these work
together to handle everything from connecting to
storage racks within the data center to off-site synchronization and backup
they can go unboosted up to 10 kilometers and hundred kilometers with
boosters but in order to test that we're gonna
have to take a little field trip
server hairstyle just gotta
mainframe up my view here
welcome to the patch room in here you
will find 50 000 Ethernet connections alongside
200 000 i o connections the blue ones are
om3 multi-mode fiber for carrying pci express links across the data center
those are the ones we saw before and often these would end up at an ibm ds8k
which is their enterprise class storage while these ones are almost all ficon
and this is super cool if normal fiber channel drops a packet the origin is
going to retransmit it to the destination no big deal says i no
says ibm that is a big deal so ficon actually contains enough
storage on each adapter in the link that
only the last hop of the journey needs to be
retransmitted this apparently has both performance and security implications
sounds expensive then over oh crap i lost it um there it
is check this out inside this box is 50
kilometers of real actual fiber that
they can combine with another 50 kilometer roll you can actually see it
right there to test out their off-site capabilities
under real-world conditions and while we're over here this is super cool
each of these racks is set up kind of like a a site a
site b and what these do is they can
take hundreds of channels of 1310 fiber
convert them to different wavelengths transmit them hundreds of kilometers
away or whatever then split them all
back out on the other side like with a prism into your separate channels
freaking bananas right then there's the brains of the operation
the support element or as i call it the my wife unit although because it's z
there's two redundant ones and i don't know how much you'd like that
um the support element acts as a management interface for the system
making sure that everything is running smoothly by monitoring this intra-rack
Ethernet network that is checking temperatures and functionality and
reporting on failure events bringing us to the big question what does it
actually do well the biggest draw for ibm's z customers is reliability and
lightning fast accurate processing of transactional data that's what it does
way better than x86 and z16 aims to
improve over last gen by introducing tools that will help it do all of that
and faster thanks to a machine learning boost as well as adding new tools that
will help customers identify and streamline security and compliance
obstacles in the organization sounds fancy
so i'm sure you're wondering how much one of these costs well the truth is
that the number in the title there assuming i haven't changed it by now is
more of a guesstimate a bear cage actually starts in the neighborhood of
250 000 but every single system is custom built
to the spec of the purchaser and the ibm
folks here laughed and but by the time you've got all the
hardware and software to actually run it i'd say we're pretty comfortable with
that million dollar number as an approximate so you won't be having any
of these in your basement lab anytime soon but uh what you might have is
freshbooks thanks freshbooks for sponsoring this video now i'm going to
go ahead and guess that you're not an accountant which is why you're going to love this software freshbooks is built
for freelancers and small business owners who don't have time to waste on
invoicing accounting and payment processing in fact freshbooks users can
save up to 11 hours a week by streamlining and automating pesky admin
tasks like time tracking following up on invoices and expense tracking with
features like their new digital bills and receipt scanner over 24 million people have used
freshbooks and love it for its intuitive dashboard and reports it's easy to see
at a glance exactly where your business stands and it's even easier to turn
everything over to your accountant come tax season 94 of freshbooks users say
it's super easy to get up and running and with award-winning support you're
never alone so try freshbooks for free for 30 days no credit card required by
going to freshbooks.com Linus
my voice is shot so if you enjoyed this video maybe go
check out our tour of sfu's super computer it was super cool i feel like
this is like a scene at a gremlins