you ever spend a lot of money on something only to have it arrive broken
or otherwise defective it's a pit
even worse is opening up the box for your replacement only to find out that
they didn't actually fix the problem and i think that's what just happened to me
with over ten thousand dollars worth of Intel's four
terabyte p 4500 ssds you guys remember
these we tried to roll them in a new server deployment over a year ago only
to conclude that they were fundamentally flawed and could never work in it Intel
graciously agreed to swap them all out for us
but here's the thing Intel i thought it was understood that i
didn't want the exact same broken thing back and yet
here we are telling you about our sponsor smart
deploy smart deploy enables it admins to manage pcs from the cloud you can push
Windows apps and security patches to any device anywhere without leaving your
desk get your exclusive software for free worth over eight hundred dollars at
smartdeploy.com Linus
getting Intel to agree to replace these took a very long time and wasn't
straightforward as far as i can tell their initial plan was to do nothing
about my problem until i pulled strings with my media contacts
in fairness to Intel these are second hand drives i bought on ebay but also in
fairness to me they are still within their warranty period so my request
isn't totally unreasonable and besides while Intel has never
publicly acknowledged any design flaw with the p 4500 i've heard of other
customers even getting cash refunds as compensation for their trouble and if
you've ever dealt with a computer hardware returns department you'll know
that cash refunds ain't the kind of thing that happens when it's the
customer that's screwed it up so
let's take a closer look at what they sent i don't actually know what to say here
as far as i can tell everything about these is identical to the drives that i
sent back to Intel model number capacity
yeah it's they're the same thing look at this
these actually have marks on them that would seem to indicate that they've been
installed in drive sleds before i'm not actually mad about that though
it's pretty typical to get a refurbished product back when you send in an rma and
it's not like they still make this drive or anything just
not exactly confidence inspiring and come to think of it do i have any way of
knowing that they actually sent back different drives and didn't just
box back up the same drives and send them back to me you dealt with the rma for this right
did they ask any questions about the issues we were having they just processed the rma
and sent us back those do we even know these are new drives
i have no clue these came from latin america oh
right of course we sent them a whole list of serial numbers before we ship
the drive so oh yeah 720 versus 807.
these would have been manufactured way later
assuming that until serial numbers are sequential
okay so they are different drives at least but
after double checking with wendell from level one tax who helped us diagnose the
problem in the first place he seems to be under the impression that
nothing about these new units should change the problems that we had before
and they were big problems individually the drives behaved fine but
as soon as you built them into a larger array of some sort and then hit it with
any kind of sustained load even just copying a file would cause individual
drives to randomly fritz out or drop out for a few
seconds at a time and then reappear absolutely tanking the performance of
the whole array and we tried everything Windows Linux zfs dropping down to pci
express gen 2 speeds in the BIOS wendell eventually narrowed it down to
the way this drive issues interrupts to the CPU
if you don't know an interrupt is basically a way for a device to tell
your computer's processor hey i've got something for you come pay attention to
me now since this is a storage device chances are that the thing it's got is
some data but small problem if the CPU were to
respond to that interrupt too quickly which is what we saw in Linux there's a
chance that the data won't be in the drive's buffer yet
missing data oh so the Linux kernel would keep retrying
the drive to see if the data turns up and lo and behold it would totally
uncorrupted which is good but with the aforementioned performance
problems which are bad now wendell's hacky workaround was to
tell the operating system to pull the drives essentially having the CPU
checking constantly got anything for me got anything for me yet how about now do
you got anything for me he's some kind of genius and it did work
but that approach comes with a bit of a performance penalty as well think of it
kind of like the postal service it's way more efficient for you to schedule pick
up with fedex whenever you want to send a package instead of just
having a courier show up every hour of every day on the off chance that you
have something to ship so poor cpus
checking in with 24 of these drives constantly
not great and obviously most drives don't have
this problem it's just that these ones do and it wasn't just one of
them it was the entire badge of all 26
drives so why would they send us back the same
bloody thing that wasn't working and if
that's not what happened then what on earth did they change about
these the plot thickens
this AMD epic roam server has been sitting completely untouched since the
last time that we tried to roll it out and gave up ultimately replacing it with
a dell loaded up with liquid honey badgers but
just because we found another solution doesn't mean we couldn't use more
capacity and we wouldn't like to freaking use it it's 96
raw terabytes of NVMe storage each with
their own dedicated pci express links
and the epic is a beast i want to freaking use the thing
so let's fire it up shall we let's see if we can figure out what they
changed obviously something they could have changed would be the firmware on
the drives but conveniently because i haven't touched this thing i've got the
exact same version of solid state drive toolbox from Intel that i had before and
i can check if it allows any kind of firmware
update here we go nope
exactly the same firmware even though they're running the same
firmware as the ones i shipped out maybe there's been an update we can check that
um no
okay no no firmware change then
i have mostly given up that this is going to do anything but
let's at least give it a chance we can use this custom view and
administrative events to see the drive dropping out or not you'll get
a little uh yellow warning a fatal hardware error has occurred in memory
we'll have to deal with that later
performance is as expected for a gen 3 drive
that's somewhat hopeful that's promising nine runs
zero dropouts of course as i mentioned before
we saw the biggest problems with multiple drives running at the same time
i mean you could kind of imagine that's how something like this makes it past qc on
AMD Gigabyte other server manufacturers as
well and Intel's side everything works fine in isolation
but you open up all these load them up with
drives and slam the pci express controller with enough high speed
storage that you're getting dangerously close to RAM like speeds and
well the wings start to burn up a little don't they
let's try four drives next
okay uh i think i'm probably gonna have to
reboot theoretically they're hot swappable but like
no dropouts so far i'm feeling good
hey you guys want a free tech tip you notice that in a mirror your read speeds
scale individually with drives but you only get the write speed of half of your
number of drives that's because you have to write two mirrored copies across the
entire array so you're effectively splitting your writes in half but with
reads you can actually take advantage of all the drives in parallel neat huh
okay big test time that's promising we've got
eight drives running in parity mode in storage spaces which i prefer because it
gives way more capacity than mirror now you'll see a couple of these are not
blinking but that's more likely just due to the fact that in
parity mode it writes in a very different way compared to mirror these
write speeds suck for how many drives are in there but from our experience it
seems to be a weird interaction between crystal disk mark and storage spaces
because we've seen better real world performance than what it measures but
this defies all logic i don't understand
how there's how it's working i mean
to be clear i'm happy also
we haven't actually loaded this thing up yet should we try 16
oh yeah
no wait wait disc 10 has been surprise removed
wait what
hold on a second i got one of those before i haven't removed any
i just thought it was something to do with while i was putting drives in but i
didn't take any drives out did i there it is
store NVMe reset to device blah blah
raid port 15 was issued this is exactly the problem
we had before need a drink
lttstore.com
prove its water in here or prove it's not water uh
go buy a water bottle they're insulated they're really nice let's go back for a second to the
questions i set out to answer today number one
what did they change okay well we've got that one
nothing question number two
what was up with this replacement process when they've indicated at least
to other customers that they're aware of this problem
and they know evidently that they can't
they have no way of fixing it because here's the thing
any other Intel drive at least according to wendell would have worked fine the
slightly newer 4501 doesn't have this problem the 4511 which
is fundamentally the same drive but in their ruler form factor
same thing why send me
stacks of broken drives when you know how much like it's not free you know the
cost would be the same to send me something that works
unless they do work
one of the most basic troubleshooting steps when you suspect a piece of
hardware is defective is to take it and put it in another system to isolate your
variables unfortunately testing just one of them
at a time wouldn't have told me anything because they worked fine and
NVMe servers that can take 16 or 24
drives don't exactly grow on trees so it
wasn't an option for me fortunately now that old wanik has been replaced
i can actually take these new drives chuck them in here and see if we see the
same behavior
locked and loaded all 24 drives
let's see if it works
performance is not as good um
even at its best this server has a fraction of the pci express connectivity
and its gen 3 compared to AMD epic but
if the drives don't drop out and we get that consistency that's
way more important because we're only accessing this for video editing over a
network you don't actually need you know 20 or 100 gigabytes a second
that's just like genital measuring at that point
wait there's a disc oh disc 26 has been surprise removed
okay yep yep that's fine that actually did happen so that means the whole thing
ran no errors
worked perfectly slowly but
perfectly so then now Intel's behavior looks like
actually more generous than stupid so a
tech probably without understanding the whole backstory saw these perfectly functional
drives in the rma pool and went i don't know the customer's always right and
replace them not realizing that the problem was because of this edge case
error that only shows up on some platforms under
some workloads so
thanks Intel it's a specific incompatibility between
this drive and second gen AMD epic cpus
dell has already fixed the timing behavior through a firmware update on at
least some of their boards maybe one or multiples of their
customers put in a big order for new servers and then had a bunch of these
they didn't want to replace and curiously AMD's daytona test platform
from quanta doesn't have the problem either so
maybe just Gigabyte couldn't be arsed to fix this but then in fairness to them
it's just this one outdated drive from Intel and therefore it probably doesn't
affect any of their actual customers like remember guys
this right here is an engineering sample unit meaning i didn't pay for it and
it's still technically the property of Gigabyte
so it's not really anyone's fault
but that doesn't change that i spent over ten thousand dollars on drives and
my server doesn't work and i want it to work so what i've decided to do is just
eat the performance penalty and keep these new drives in my old super micro
box even though the limited number of pci express lanes means they are
bottlenecked to hell and back i mean while we waited on a solution to
this liquid stepped up and set us up with their honey badger den which is
faster anyway and has been rock solid so thanks liquid thanks
Intel i think and uh now this will be the like
capacity SSD server there you go
and this will be my segue to our sponsored drop check out their enter
keyboard it's made with enthusiasts in mind making it easy to swap out keycaps
and even key switches it's got an aluminum top plate and plastic bottom
plate that feels great and white leds for visibility in dark conditions the
pbd keycaps are double shot so they shine through and it weighs
964 grams so you could basically kill a man with it not that i would recommend
that it's available in three colors with your choice of mechanical switches and
you can buy yours today at the link in the video description if you guys are
looking for another server video to watch you can check out the original
deployment nightmare with sorry not this with this one or you can check out we
actually did a few videos on the honey badger den that thing is so freaking
fast like a hundred gigabytes a second raw it's dangerously close to memory speeds