if you already know who equinex is you know the treat you're in for today for
everyone else equinex who sponsored this facility tour was founded in 1998 and
currently operates over 260 data centers
globally through which flows over half
of the tier one internet backbone in a nutshell if you're using a computer
right now which you probably are like this guy you are more likely than not
touching equinex infrastructure right now and guys I get to take take you deep
behind the curtain I'm talking power data cooling security it's pretty much
everything that we do on our Channel yes even pool server cooling except
legit you see as pretty as it might look
the building behind me was purpose-built from the ground up to be a function
First Data Center and and that manifests in a lot of interesting ways for
starters it was not built from the ground up but rather the first floor was
lifted a few feet off of ground level
why security like any equinex facility
that building needs to be able to withstand a breach attempt from a
vehicle ramming into it but the city wouldn't let them surround it with ugly
barriers so instead they lifted up the
foundation itself so if you drove anything less than that city bus into 45
Parliament Street Toronto you would eventually regain Consciousness and find
the building to be quite unbreached honestly speaking you'd probably have
better luck getting inside if you just walk through the front
door JK once you get in there you'll
find yourself in a series of airlock style doors like this one with at least
three biometric security checkpoints before you so much as catch a glimpse of
a customer rack oo but I get to go but
what's the big deal you might think it's just some servers and network switches
right well yeah but really important
ones equinix houses critical infrastructure for everyone from
telecommunications providers to cloud service providers to Securities
exchanges and even governments to put it in perspective some poor schlub
accidentally overcooked a bag of microwave popcorn at this location and
the city sent seven fir trucks on the off chance that equinex was on fire and
hadn't already Managed IT with their own fire suppression systems for more
context their service level agreement with diesel fuel providers is on par
with the local hospital because well think about it realistically is that
hospital getting anything done without data services probably not let's head up
here's something cool everything in this building except the generators we're
about to see on the fifth floor was brought up in this freight elevator it
has a full ful loaded capacity that's equivalent to about four loaded Mini
Coopers and this is really important for cloud providers who pre-build their
racks offsite it allows them to wheel a rack into their cage hook up power hook
up Cooling and walk away we'll take a look at some of those cages later but
first we need to go all the way up to
the fifth floor where possible equinex
uses a top to bottom approach to building design so we're going to go in
that order why do they design top down you might ask that's because if you
spring a leak it's going to go down so this floor contains everything that's
needed to run the whole building in the event of a major outside Interruption
and nothing can leak on top of it which is especially important for this inside
the apply named battery room is enough capacity to run the entire building at
full load for up to 15 minutes and if
they need more time that's where these come in these diesel generators are
designed to run continuously at a capacity of 2.5 megaw each yes my
friends 2.5 million watts each getting
these up here by the way took some special planning which I'm told involves
cutting a hole in the side of the building renting a crane and then
sliding them in absolutely badass but
there's one small problem now they've got to fuel them I mean they've got this
little Reservoir here but this one run one of these bad boys for what an
hour well as it turns out the main tanks
which include an onsite polisher to remove water and other impurities that
can spoil the fuel those are underground and this was done because diesel fuel is
lighter than water and equinex wanted to design this building such that if the
entire first floor was flooded the tanks wouldn't float away Wicked smart the
last thing they had to manage was the noise oh the noise noise noise
noise now obviously equinex isn't going
to hit their 2030 renewable energy goals by running diesel generators 247 but
they do need to perform monthly checks on them and I'm told that when you fire
those things up you can't even hear yourself think in here now given there
used to be a public library on the other side of this wall that was a problem so
equinix designed These acoustic baffles that Bring the Noise down to the same as
a typical City Road and you can oh wow can you can probably
hear the difference right when I go in there oh yeah crazy kind of necessary if
you consider that this is about the equivalent of having five bloody trains
running in here fun fact before we leave
this room by the way diesel generation kicks off enough smoke that conventional
smoke detectors would be giving off constant false alarms so this room uses
flame detectors special cameras that can see through smoke to see if there's
actual fire moving into the chiller room we've got the exact opposite problem you
can fit so much water in this bad boy
that is why absolutely no customer infrastructure is allowed under this
room in here are three main cooling Loops the one behind me is filled with
antifreeze that goes between these liquid to liquid Heat exchangers and the
large evaporative cooling fin Stacks that are Up on the Roof we're going to
take a closer look at those in a minute but for now now I want to look at the
other Loops those two are connected to what is in effect the biggest freaking
air conditioner that I have ever seen except instead of cooling air like your
air conditioner does it cools water the
hot side here carries heat to those same heat exchangers that we just talked
about while the cold side is connected to the rest of the building through what
equinex calls their bus architecture which is kind of just a fancy way of
saying that it's all redundant it's all interconnected and it's all controlled
by a BMS or building management system and I'll show you why that's cool right
above me is our chilled water return which is bringing heat up from the data
center floors below okay so far so simple but where does it go
well as you can see it gets a little
complicated when it's below 4° C in
Canadian winter it essentially passes straight through with the compressors
barely running and goes to the heat exchangers on the roof sweet free
cooling thanks Mother Nature but then spring rolls around and the outside air
needs some help that's when our highly efficient magnetic bearing compressors
ramp up their cooling efforts bringing our thermals under control but wait now
it's Midsummer and there's a heat Dome we need more cooling Captain thankfully
we've got it now most of the time the misters are not needed but for a
building like this you don't design for most of the time and these rooftop fin
Stacks can either be run bare or they
can be lightly coated in water to dramatically increase their cooling
capacity it works in the same way that your arm gets much colder if you lick it
before you blow on it the best part is that everything we've seen is fully
redundant including cold spares of key components that gives equinix's team
lots of options for load balancing during maintenance and it means that in
the event of an equipment failure or another event equinex won't risk giving
up their nice 69 uptime Target that's an
average of about 32 seconds of downtime per year by the way wow hilariously
sometime in the near future all the cooling we just looked at will be
rendered redundant though when the forth pipe goes online in the dlwc or deep
lake water cooling system if you haven't heard of it it's basically my pool water
cool cooling setup except done properly with massive pipes that draw 4° water
from the bottom of Lake Ontario and then pump it into the city to cool office
towers and Arenas and well data centers
that sort of thing super cool project since we're up here by the way these
freaking blew my mind now as consumers when we want the lowest latency say for
gaming or video conferencing we think fiber right well what if I told you that
there's a tier above the speed of light
well you'd probably tell me there is not
and all right you got me but here's the thing light does not always go the same
speed and fiber optic cable slows it down to about 2/3 of its theoretical
maximum in a vacuum now that's not enough to matter for your game but for
Capital Market transactions where micros seconds can be the difference between
making your trade and losing a huge opportunity every Competitive Edge
matters and that is where these microwave arrays come in these provide
direct line of sight links North to the Toronto Stock Exchange and South to New
York now of course for anything where throughput matters nobody's using these
so let's go back inside and look at good oldfashioned fiber by the way you guys
might have noticed some extra pads and footings for additional equipment up
here that's another nice thing about a purpose-built building you can plan out
your future capacity and then you can build it as needed this for instance is
a spot for another fin stack and here in the electrical room you'll see much of
the same thing look at the Glorious extra conduit for future installs isn't
it beautiful this room takes in
13,800 volt power steps it down and
distributes it throughout the building but all right enough about
infrastructure how about some customer systems right the next two floors are
very similar so let's go with number four this is the oldest floor and
therefore uses some older design paradigms for starters cooling is
handled using cold Isle containment rather than hot aisle containment and
let me show you what that means the water from above us is getting pumped
into this air handler Corridor that chills the air chilled air then gets
carried through ducts to the center of the customer's cage where a thermal
probe aha will inform the system when
cooling is needed it opens up the motorized Vents and then blasts cold air
downward it then gets sucked up into the rack which is going to be on either side
and then exhausted out into the room where it gets sucked into that Corridor
rinse and repeat now the main drawback of this approach is that it allows some
mixing of the hot and cold air but equinex still managed a solid power
usage effectiveness of 1.42 for this site since you're
wondering by the way I should probably acknowledge that this is an Unleashed
space and there are normally strict rules about access to these cages for
example exchanges cannot have their cages entered at all during trading
hours and even for equinex staff unless
there's an emergency or contracted work that needs to be done they are not
supposed to be entering customer spaces so they built these cool access panels
where they can run any connections that the customer requests to the outside of
the cage and then the customer can come in and Route whatever they need for
whatever they're doing access to the cages is controlled by access cards and
Biometrics either a hand or a fingerprint that's stored as a hash for
customer privacy and any use of your
card is logged on top of the logs for the biometric sensor the contact sensor
on the door the motion sensor near the door ah there it is and of course the
camera pointed at the back of the rack space and if all this seems Overkill You
Ain't Seen Nothing Yet many of the cages not only have physical access barriers
but even visual blockers so no one can see any of the equipment in use and for
the extra paranoid a top cage can be installed to ensure that nobody can
climb over the top though of course anyone trying to do that would have been
spotted by one of the hundreds of cameras in this place but then as you
probably picked up by now in this business redundancy is the name of the
game some cages go even further
requiring what's called double access meaning that to get inside you need both
an equinex employee and a customer employee and we even saw one with a
touch sensitive cage so if you so much as leaned against it a ticket would get
automatically created and equinex security would get an Irate phone call
but wait there's more they told me that some customers will even post a guard
who stands outside their cage 24 hours a day 7 days a week financial institutions
and government agencies will often operate with a high degree of security
like that speaking of financial institutions certain customers have
noticed that if they're closer to each other they can have a physical advantage
in connection times well that's not
necessarily a good thing to solve that problem then equinex has a system called
equidistant trading that neutralizes any latency Advantage by running the exact
same length of fiber to every customer who requires that the spools of excess
fiber optic cable are one of the most protected things in the entire facility
now I was able to convince them to open the door earlier for a quick b-roll shot
but they wouldn't let us any closer there's a camera pointed at the cabinet
and if anyone so much as goes near it security shows up and it's a whole to do
with reports to the SEC and the whole kazoo moving on to the second floor this
place is sick now we weren't able to shoot as much here due to customer
privacy concerns but we did manage to get some footage of one of the liquid
cooled cages here building cold water goes into a heat exchanger that pumps
chilled water to their highdensity servers these kinds of deployments are
becoming much more common but they're kind of an ongoing challenge for
standards lovers equinex at the moment
these guys dial in flipping everything down to the colors of the metal posts in
their facilities by the way blue means you hang stuff off them and gray means
they hold stuff up the more you know but
the lack of standards in the liquid cooling space right now means that these
cages often need special provisions and that's something that they're working
with others in the industry to solve as rack power density continues to scale to
l ludicrous levels looking at you AI
while we're here there's a couple other cool things about this cage there's leak
containment troughs in the ceiling under any water pipes to ensure that even a
small leak will be caught and can't fall onto the equipment below it and if you
look carefully at the floor there's actually little yellow ropes running
around the outside of this cage and here on the first floor I see another one now
I wouldn't blame you guys for thinking gee it seems uncharacteristically sloppy
of these guys to leave this kind of thing lying around all over the place
but it's actually not a rope they are
everywhere in this facility where there's water and they're part of a leak
detection system that tells their team not only if there's been a leak but
exactly exactly where it was boy would
that ever have been handy boy would that ever have been handy boy would that ever
been handy since we're here by the way this happens to be the newest floor in
the building remember top to bottom buildout right this one was done about a
year ago and they refer to this data Hall as a as a mark1 design but now that
AI has exploded power densities to numbers that would have been Unthinkable
even 2 to 3 years ago the mark 2 designs that they're currently constructing are
aiming at around 10 times the power per
square foot that they targeted here so maybe that's something we could see in
another video equinex andho this floor
uses a very different system for handling Air instead of pushing cold air
to the cages now they pull hot air out
of the cages into these chimneys that go up into the plenum which runs into the
air handlers now it might seem like an
unimportant distinction but the higher the temperature difference the more
efficient the thermal transfer and when you're dealing with very high draw racks
this ensures that the hottest possible air goes into the handling it also all
but completely prevents hot and cold air from mixing which is a major winre
efficiency now I can practically hear you guys shouting if you've got all the
hot air in a chimney why don't you do
what a chimney does best and blast it out of the building well you can do that
but then you need to bring in air to replace it air that needs to be filtered
and that needs to be humidity controlled to handle that at the kind of volume
that you would need it's impractical so
even under very extreme conditions like we saw last winter in Chicago I'm sorry
but you can't just swing open the doors of a Data Center and hope that it'll run
cooler by the way we didn't show this on the other customer floors but there's
actually two equivalent air handling systems on opposite sides of each floor
each with the capacity to handle the entire floor in the event of a failure
by the way before we leave the customer systems area I was very surprised to
hear that equinex does not use exotic fire suppression methods op instead for
standard water sprinklers but then I found out they're not quite standard see
the pipes are not flooded with water all the time but rather they're full of
inert gas and water only flows into them
if two out of three systems go off and a
plastic head melt so they use heat detection smoke detection and the
world's most expensive PVC runs these
are part of a vesda or very early smoke detection alarm system system that can
not only detect smoke but also triangulate where it's coming from to
ensure that only the necessary sprinkler head goes off they made this choice
because oxygen depletion fire suppression can pose very real risks to
any Personnel who get accidentally trapped so good guy equinex let's go to
the basement above me is where utility power comes into the building or rather
this is one of the two rooms because redundancy right now interestingly the
power company can actually control this unit remotely allowing them to turn it
on and off and even scale how much power comes in that is not a normal thing but
because there's so much power going into
this building in an emergency they might need to call equinix and say hey um we
need more power could you guys just turn all of this off and Island yourselves
this kind of collaboration is really important because equinex estimates that
this this building draws somewhere in the neighborhood of 3% of the city's
entire power budget if a multi megawatt
load kicks in suddenly without warning or drops suddenly without warning it is
a big problem for grid stability with that said they do turn it off and on
regularly though every year they do a scheduled pull the plug test and
quarterly they do load tests where they will bring in load Banks to ensure that
the building can actually power everything that it's rated for these
things sound pretty cool basically if they don't have enough servers running
they bring in Big Blocks of metal and they fire up the generators and make
them glow red hot flipping wild finally
there's the fiber in room it's funny they thought that this was going to be
underwhelming and they were all like are you sure you even want to see it but it
really goes to show you how numb you can get to cool stuff if you spend your
whole day around it now in fairness to them they are kind of right I mean it
pretty much looks exactly like the fiber room our building back home line comes
in from outside line loops around as is
tradition and then gets spliced or patched to wherever the difference is
that I could probably fit my entire family hula hooping inside these
freaking loops and where they go to is even cooler they go outside to one of
the seven manholes that get shared by the approximately 60 carriers that route
into this building that is what the carrier's access ends equinix splices
runs from the manholes to this room and then they run it up to the floors above
the carriers only get access again once it reaches their cage this is how
equinex enforces the principle of equity of access that inspired their name oh
and of course there are two of these rooms with fiber coming in from two
different sides of the building because in the event of a construction worker
accidentally cutting one of these or something they got to have redundancy I
got to tell you guys there is so much we weren't able to fully cover if you
noticed QR codes on everything those are cool they use AR to direct technicians
for fault Discovery and to map out customer connection requests um the
18-wheeler dock can be completely closed off for privacy and security when you're
unloading racks uh they have employee facilities on site so when an emergency
you can shower shave sleep you know your 3 s's um they turn off the lights
whenever no humans are detected to save power they've got grounding strips
freaking everywhere they have direct links to their other buildings to turn
their entire flock of data centers into a giant redundant mesh guys there is so
much more but I think it might have to wait for next time for now massive
thanks to equinix for sponsoring this video and giving us such an in-depth
look at the kind of space that most people are never going to set foot in
let alone get a comprehensive tour of hopefully we'll get a chance to do it
again sometime if you guys enjoyed this video hey you might enjoy this kind of
high security nobody gets to see it kind of stuff so maybe check out the Intel
Fab tour that we did last year