you guys have been telling us for years that our infrastructure the servers and
the networking gear that our 20 video editors not to mention the other 90
employees rely on is a little Jake you
weren't wrong then and realistically you're probably not going to be wrong a
few years from now but we haven't been standing still and in the last year alone we've effectively replaced or
rebuilt every single server in this building making them faster better more
reliable cooler far less power hungry in
fact we've done so much that most of the servers you've ever seen on camera are
in this graveyard right in front of me which means it's high time to give you
guys a full update on all of it including some brand new gear that is
unlike anything we've ever shown you before
first up is the main server room at the OG office where you can see there's been
some serious server shrinkage there was one point during the Vault recovery
process where we literally only had three rack units left and now this is
only about half full but the important
stuff is actually at the back where our old PF sense router has been
replaced or rather it's pulling Duty as a steam cache now our steam cache used
to be in a VM but we were having issues with it it'd be like weird performance
things would be like 10 gig download and be like one gig I mean 10 gig again and
one gig so we're back on bare metal now which is fine because we upgraded this
thing anyway only to discover that PF sense didn't run on the new box this
super micro one you is super cool it's
got 25 gig networking on board except no
drivers and PF sense yeah it's running a really old FreeBSD version fortunately
opensense is a fork of PF sense that
gets updated way more frequently now that could be a double-edged sword your
organization might prefer the slow plotting approach to updates but we live
on the bleeding edge where we love that we've got support for our 25 gig
networking in here and you update it like a couple weeks after the news yeah
I usually wait like a month after a big release before updating and it's been
Rock Solid for us zero issues if we wanted to go back to PF sense though we
totally could is what I would say except they still don't support this thing no
they don't the other changes back here are not so much Hardware but rather
topology we've still got that awesome Dell 100 Gig switch but now we're wired
directly into all of our editors and ingest stations via fiber running 25 gig
and we're using the 100 Gig ports to
connect to each of our Downstream switches where we used to have it going
from one to the other to the other to the other so that if we had to update
one or something uh it would knock out everything Downstream of it which to
their credit these Dell 48 part 10 gig switches other than updates have they
gone down at all I've never had an issue with any of these three Dell switches
ever they would have 100 up time if we didn't update them shout out ubiquity as well
though we've been really happy with their PoE switches this one right here
runs all of our cameras all of our Wi-Fi access points and then this guy up here
was actually added when we acquired the unit on I guess this runs to both other
sides right 101 and 105. so that's the shop and like where the WAN Show set is
and this one's got a 25 gig link to our 100 Gig switch now oh there is one other
new piece of Hardware we have an NVR back here now yeah we have two actually
we previously only had one but we had so
many cameras that it was like starting to like cause weird performance issues
so we split them out into two ubiquity keeps telling me that they're gonna add
NVR stacking or you can combine them together but for now I have to like
click between the we're really happy overall though yeah yeah yeah big
brother's watching yeah yeah yeah yeah wait I missed something else
what's this Cloud key doing up here well it used to be on this side let me
remember don't we have a dream machine here I have one in my house no oh open
sense right I can't remember why
I thought you can run this in a VM now
okay all right it was and then the
database corrupted so we couldn't adopt anything properly so instead of just
re-adopting all of the like 40 switches
we had I was like what if we just transferred to a cloud key and it fixed
it well it's not a cloud key now again like it was before love doing work twice
the other thing that particularly the Europeans have tripped us about in here
is our use of 120 volt power in a server room and while the output on this does
stay only 110 and 120 volt is the options it's got two 120 volt phases so
if you pair it with the right pdus or power distribution units you can totally
do 208 volt which we're now doing thanks to these super cool pdus from triplite
they're networked so you can not only monitor the total power usage but you
can actually switch off individual Outlets say if a network device became
unresponsive you can even create automations for it the main benefit of
switching to 208 volt is efficiency you've probably seen in power supply
marketing materials that there will be two lines for the efficiency curve one
for 120 volt and one for 200 plus and
the reason for that is the higher voltage allows your power supply to
operate more efficiently it's exactly the same for servers so we dropped our
power consumption in here by about 10 percent just from that changeover that
means that combined with the server consolidation we've done and the
additional battery units that we've added to our 9pxm this Eaton UPS can run
for almost an hour with the power out completely which seems to come up around
here a lot now let's talk about the great consolidation
after we talk about our sponsor the ridge looking for a stylish and
practical wallet the ridge wallet has got you covered it's minimalist design
allows you to carry your essential cards and cash without the bulk check out the
link below and use code Linus to save 10 off your purchase and get free shipping
where we used to have five 45 Drive storinators taking up half our rack
holding about 60 drives each we now have
three of these super micro J bods and what we call the mother Vault each of
these holds up to 90 drives and then we've got this single AMD epic Milan
head unit that uses two epic Milan 32 core CPUs and then it uses external SAS
connectors to attach to the j-bods right now we've only got a single j-bot
attached to each of the SAS hpas in here but theoretically nothing would prevent
us from adding another I don't know like three of these once we clear out Clover
server this is running ancient Hardware it says Xeon
2699v4 I mean that's 28 cores but they
are flipping slow and we're using this for all of our virtual machines which do
we even have that many virtual machines what else we got going on in here not
really this one's being deprecated soon in place of Clayton's server up here
which is running proxmox instead that will be where all of our VMS move in the
long term does that mean I can take this CPU from my home server
sure our Clayton proxmox Server doesn't really run a lot of VMS to be honest we
have a parsec relay which makes Park SEC run a little bit faster when you have a
bunch of different users we have a couple VMS for the Labs team and then
the scene cache was on it but then like I said it kind of had issues so yeah 64
core epic 24 Bays for four drives in
like three VMS wait we didn't mention the mother vault
is running true Nas yeah I mean basically all of our servers run true
now shout out trunas shout out ZFS and
Shout Out Wendell from level one text hey it's not like the Vault data is
essential that's why it wasn't redundant then and it's still not redundant now
but it's a super nice to have and thanks
to Wendell we managed to get back what like 98 and a half percent of what we
lost most of it and to be fair I haven't run into a situation yet where someone's
been like hey I need this file and I can't find it but I think it's freaking
awesome and it's wild being in Windows
Explorer seeing this single folder this single machine that now has everything
2.4 petabytes or whatever unbelievable
speaking of unbelievable we got hated on
a fair bit for this thing and you were right
but not about the performance yeah this thing has been crazy reliable crazy fast
it's basically the bee's knees I've only really went into one issue and this was
recent the day of the heckoning
one of the drives decided to shut off and if you want to change out an m.2
drive that's loaded onto a carrier card that's loaded into a module thing that's
loaded into a server it's a bit of a pain in the butt especially because we
didn't have the foresight to write down the serials and which carrier card they
were installed in absolutely zero idea which drive it is oh my God that's what
you did so what we ended up doing to restore the array's Integrity is putting
a mismatched seven terabyte kyocsia
drive into one of the front NVMe Bays Jake yeah and the plan is apparently to
do something about the serviceability of it soon TM well it hasn't needed any
servicing right so to be fair when I restarted the server the drive came back
online it has been online since we do have some ideas for improving the
serviceability we found out that you can get u.2 to m.2 carrier cards so we could
actually just take all those internal ssds load them into the u.2 slots in the
front I guess we'll have to add more modules for that maybe use a different
server or maybe use a difference oh okay so new new new monik is coming soon then
wanted four and we could use those new sabrant drives they're sending over 22
of their wild eight terabyte rocket 4
plus drives so that we can try them out with that crazy 21 by m.2 carrier card
that you probably saw in the news so wait no I am not thinking we use those
carrier cards hey you were the one that brought that up no we're not we just put
one it's the one one card server it's
got 160 terabytes on one card yeah entire One X server and an Intel Nook
yeah no no I mean you made it yeah it's not that
way I'm the one that's supposed to say your ideas are stupid that's stupid it's
not that stupid you like it I don't not after that meanwhile in Unit 101 where
all the inventory and Logistics Department used to be but now you'll
find only our makeshift ShortCircuit set we've removed the server rack
outright because we can't have servers sitting here
next to where we're trying to film instead we've got this simple six rack
unit wall mount here that's going to hold our two Network switches that
handle all the network connectivity down here and upstairs we've also got a udm
pro down here that's the router for our secondary backup internet connection
it's not nearly as fast it's one gig instead of 10 gig but realistically most
of what we're doing we don't need most of the time we just use it for like testing a router on ShortCircuit so
yeah it doesn't really matter now you might have noticed that this
looks pretty okay this looks extremely not okay this is
very much a work in progress these patch panels are going to come down here and
then presumably at some point we're going to shorten these runs oh yeah oh
yeah yeah because they're long enough to go into a rack well these will get
replaced with little like those little ultra thin six inch cables from infinite
cable Shadow infinite cables thanks for the thanks for these things and uh
speaking of you know shout out shout out ubiquity for this access point yeah
definitely oh my God is this the actual access point no no it's just like the
backup connection one the only other noteworthy thing here is our ups this is
to ensure that not only is our main internet connection battery backed up
but also our secondary one that way in the event that we lose power to the
building outright the WAN Show can continue as you guys actually saw not
that long ago that time we had to run it on jackeries because we actually ran out
of backup battery so we bought a second battery pack that just needs to get put
on here it's sitting over there it'll it'll get on there we gotta be able to do WAN Show for like at least two hours
yeah yeah on battery five
moving over to the lab the biggest change is probably the complete removal
of Windows from our infrastructure get
it there used to be a window here yeah but because of all the sound bleed from
the servers next door the poor social media team who's moving in here was
going to develop tinnitus social media theme isn't that who's coming in here I
hope this is my server room no what it what the server room's over here and now
it is properly sealed so let's head in this room this is this no Honestly
though removing the window and finishing up the drywall
it's not really enough and it's especially not going to be enough with
some of the gear that we're putting in here so we're doing it again but this
time better way better hanging insulation on the wall of the server
room yeah this time we're going to use this cool product called Sona par which
like is a recycled wood panel they're like 30 bucks for a four by eight sheet
which is really cheap for sand isolation and then we're gonna put rock wall on
top of it with these cool adapters that Dan Made In fairness to me the original
solution was messy but darn it did it ever work yeah it does work really well
I took so much Flack for that I don't even know if sonopar existed then yeah
and to be fair Nails is way easier than 3D printing hundreds of little brackets
yeah yeah I did that in a weekend yeah we're gonna put the duvetine on with
magnets it probably took like three months off my life with the amount of fibers I inhaled that yeah that's okay
he got good life insurance man this is where you can see the difference between
us eight and a half years ago architecting this stuff and then kind of
upgrading piecemeal as we go and when we
build something from scratch with all the knowledge we've gained since then
here's where our internet connection comes in we did end up having to go for
the expensive link over to the other building but hey now that the whole
writing team's here and the lab is here we definitely do need it and then we've
got these 100 Gig links that break out into 425s that go to these 10 gig
switches one two three four five of them
we might need one more too what the intention is for every single Connection
in this building to be 10 gig obviously not cameras and like APS and stuff
everything else is running off of these Enterprise 48 switches from ubiquity
these are super cool they're 48 ports two and a half gig and they do PoE so
these will do all the cameras and access points here which access points in
particular now actually can use two and
a half gig uplinks yeah so freaking fast and we put those Wi-Fi 60 Enterprise APS
everywhere in this building at least with what's done so far we have
ripping fast Wi-Fi here there's more than just networking over here though
we've got the NVR Pro from ubiquity this is handling all of the cameras in this
building and you may recognize this as
the storeinator that 45 drives sent over for our vault recovery well now it's
pulling completely different Duty as our Annie Dale server so this is our footage
Dumping Ground and is one of the backup targets for our main video editing
server wanick it's over at the lab now for more physical separation so that way
if one of the buildings were to burn down in the past they were just two
separate units of the same building now they are physically separate buildings
probably the coolest thing over here though is this our first attempt at a
high availability server that if all goes well is going to be the new wanek
editing Nas you mean new new new high
availability wanted yes yes that's the name we're gonna go for J wanted four
um how do I get these out uh there's like a little
each of these four units in here has a
64 core epic melon season these are
Genoa yeah yeah so it's like 384 gigs of RAM 64 core
Genoa six UDOT three slots in the front for PCIe Gen 5 storage if you want and
then there's two little m.2s for boot and there's four of them in this to you
holy crap so there's 256 cores the idea
here is that in the same way that you would run multiple hard drives so that
if one fails the rest of them can just take over you run multiple servers so
that if one fails for whatever reason the other three will just keep on
trucking now there's a number of different ways that we can combine the
storage of these servers whether we use something like Gloucester FS on trunes
scale or Seth or even weka FS yeah we
were going to use that weren't we we're working on it okay maybe yeah maybe we might need a
second one of these because weka apparently needs eight nodes minimum now
but still then we could have like three of them fail I think and still have
storage which is crazy the other cool thing is
you can update it in the middle of the workday like it's nothing oh that's
super cool yeah BIOS update stay tuned for that yeah shout out Eaton by the way
for this 9pxm UPS I think
is the same one that's on the other side yeah did they just send this yeah they
also sent us another two pdus which is super sick dude the pdus those aren't
even the final ones they didn't have stock of the ones I wanted that have
individual per Port energy monitoring no
way so they were like oh yeah you can just have these ones for now and we'll
send those ones when you want it and they're also providing full cable tray
for the entire Warehouse so we can actually run all the Ethernet in like
those fancy trays so eaten our just complete Bros now yeah man their stuff
is like aced here that's the that's this is what
you want speaking of cable tray I guess that's the plan for this yeah the reason
that that hasn't been cleaned up is we have to run like another 30 or 40 wires
you might as well just do it all at once got it we didn't mention this on the
other side but you might have noticed we have these color-coded power cables
coming off of our pdus that's so that we can ensure that each of our servers is
powered by two separate pdus so that in the event of a pdu failure it'll still
stay running thanks to the Redundant power supply the last part of the
infrastructure changes are on the client side specifically for our writers
ubiquity hooked up each of them with one of these cool I think they're called the
switchflex XG they're a PoE powered 10
gig switch but in our case we're just using wall power it gives each of our
Riders four 10 gig ports and a gigabit Port obviously we have test benches and
other devices that we're working on so just having one 10 gig connection to
each workstation was a bit of a bottleneck this way they can plug in a bunch of different devices and still get
that 10 gig speed which is obviously very important if we need to review
footage off this server or what have you it's pretty sick and they're not that
expensive either I think it's 2 99 for one of those we almost forgot about the
network closet that was here when we moved into the lab the intention for
this is for all of these cables to get moved over to the new server room the
reason it's still here is these are all the old Connections in the warehouse that haven't been redone yet because of
that cable tray so once that's all done these will probably just get cut out of
the wall and recycled and yeah there won't really be anything in this closet
we could probably just use it for cleaning products or something like that grammarly is throwing their hat into the
AI ring with grammarly go which offers generative AI to accelerate your
productivity while staying true to your personal style what makes grammarly go
so unique is that you can customize your preferred communication Style by setting
your voice to personalize your tone the business team uses grammarly go to
rewrite emails and talking points to be more concise all while keeping it
authentic to our voice it's me need help
with a video title just input a prompt and grammarly go will provide innovative
ideas to unlock your creative potential our favorite part about grammarly go is
that it's there to help you write not replace you and there's no need to leave
the platform you're on just download sign up and start typing you'll be
amazed at what you can do with grammarly go so sign up at grammarly.com LTT and
get 20 off grammarly premium and that's pretty much it big shout out to you
ubiquity for all the networking gear micro tick as well super micro for all
the servers infinite cables for the patch panels the networking they sent us
like 20 boxes of a thousand feet of cat6a they got really sick patch cables
45 are you pimping infinite cases again
yes yes 45 drives also makes awesome servers
we're just not really using them aside from this one but definitely check them
out God what's up what else is there oh I guess a video to watch probably the
one where we put all this in this rack because that was just like pure
distilled chaos or you could go watch the one on the
mother Vault yeah that was a big change for us oh my gosh it's so nice though
bye