if you went to the grocery store gave them your money and left with 10% less
vomite than you paid for you'd probably be pretty ticked off but what if I told
you that happens every day except with
$350 CPUs well it's true these 12 chips
were obtained over a span of a month from eight different sources spread
across three separate countries and the real world difference between the best
and the worst of them is 8% in factorio
and as much as is 12% in csgo this has
caused a lot of problems for us over the last 6 months you see in April I gave
the Labs team the goal of increasing our testing throughput so that we could give
you guys more juicy benchmarks in our reviews unfortunately the math was not
in our favor you see the typical turnaround to troubleshoot Benchmark
write film edit and QC our coverage of a new product is 5 to8 business days at
about 5 minutes per test times three runs for consistency checks times
however many products we want to compare times however many games or applications
you guys want to see it is easy to run out of time now we could shorten each
test but we' found that longer tests are more consistent from run to run and end
up being more reflective of the real world experience of using the product so
that's out and I guess we could work through the nights like a bunch of AD
oral out Master students but come on guys we're not that young anymore
automation with mark bench does help a lot but if our system hangs in the
middle of the night which it happens sometimes we're right back where we
started which leaves us with one real option building up more of our test
benches and running our tests in parallel except as I already told you 8%
difference in factorio 12% in
csgo now when I signed the procurement authorization for 11 CPUs I knew that we
were likely to find an outlier or two but then it turned out that this Rabbit
Hole went way deeper than I could possibly have known like this deep
Saveway to our sponsor nexigo whether
you're in need of webcams or VR accessories nexigo has products that'll
make you nexigo wow those are cool learn more
about them at the link below now before you start sharpening your pitchforks and
demanding that AMD's Executives be turned into Al paste or something it's
worth noting that most of our chips were within a few percentage points of each
other even in csgo at 1440p which was
the test that saw the greatest overall spread also when we expand our
comparison to include our full Suite of games that maximum difference Falls to
around 2 and 1 12% far less outrageous
so you don't really have to worry about Buddy in front of you in line getting a
way better gaming experience for the same price but still too big big for us
to buy any two random 7800x 3DS use them
to test two different gpus in parallel and then say well these results should
be comparable now the obvious conclusion
at this point then is guys there's something wrong with your csgo test I
think it's time to get good but the thing is there aren't a lot of variables
here and that's by Design we used the same motherboard same memory same
Windows drive and install we even tested
using phase change thermal pads to ensure that our paste application wasn't
an issue and we chucked our bench in the thermal chamber just for good measure we
are very confident that our numbers are valid and we're going to have the process doc Linked In the video
description if you want to have a look which is all finding good but doesn't
answer the much bigger question of why
do these CPUs vary so much in the first place AMD gave them the same model
number and specifications AMD charges the same price so they should have the
same performance right well back in the
day that was true CPUs would run at
their rated speed and if they didn't it meant they were either broken or they
were about to be but thanks to a relatively recent Innovation called
Dynamic frequency scaling that's no longer the case you see no two pieces of
silicon are the same and whether it's through rolling improvements to
manufacturing or just sheer blind luck you can end up with a processor that is
capable of better than the advertised clock speed now in the old days you
could unlock this extra performance manually through overclocking but
nowadays processors just adjust their own speeds and they do it on the Fly
based on a whole host of factors including user configurable power
profiles and thermal limits AMD's approach is to allow the CPU to clock as
high as it's able until the CPU die average reaches about 90° C at which
point the clock will be dialed back until it reaches equilibrium sounds good
right I mean why be bound by some artificial performance limiter when I
got a golden chip that can go higher and
I actually agree but for the folks who end up with a lesser chip can feel a
little bit like missing out even if AMD is careful to only guarantee clock
speeds that 100% of the chips can hit
and as I mentioned before it's also very inconvenient for our parallel testing
endeavors so what do we do testing lots of it
after throwing cinch at our very first CPU we ran into our very first roadblock
the numbers from run to run can be vastly different I am talking 300 point
spreads on a single CPU run back to back
what the heck right how on Earth are we supposed to narrow down which two CPUs
are within 1% of each other if one CPU
isn't within 1% of itself
as it turns out software was the culprit and software is notoriously hard to
account for have you ever opened up task manager right when you boot up your
system there's no programs running nothing should be happening except wrong
what was that here's the thing even when you're doing nothing your operating
system is busy managing all the behind the-scenes work that keeps your system
running like updating the weather widget synchronizing the clock with a trusted
time server prepping the next thing it thinks you might installing updates and
so much more and we don't really get to
decide when that stuff happens which means that no one result can ever be
taken as gospel truth we do have custom
Windows images that are intentionally debloated to remove some startup
processes to help with this but it only partially mitigates the issue and it
introduces new ones like making our results slightly less representative of
the typical user we feel this trade-off is worthwhile because it helps us to
better isolate our variables in testing but it's not even enough to further
mitigate the amount of work that Windows is doing in the background we can also
increase a process's priority in cinch
we went from seeing points varying in the hundreds down to the tens on the
same CPU that's a big Improvement and enough to use cinebench for our binning
process but we're not out of the woods yet you see with some tests it's not
enough to use the same Hardware at the same process priority because the
benchmarks themselves have built in
inconsistencies Red Dead Redemption 2 for example simulates physics and AI
behavior during a bench run that's a really good thing because if that stuff
was canned our results would not be comparable to actually playing the game
but the bad news is it means that sometimes Arthur loses his hat sometimes
he doesn't sometimes the horse gets shot sometimes it doesn't
which can impact our run-to-run consistency can we ever fully account
for this unfortunately not but by running each test multiple times and
then taking an average we can get a pretty good picture and we can bake that
expectation of noise into our data analysis which finally happens now sorry
for all the Preamble first up gaming for
the sake of legibility we named each of our samples after a Pokémon why Pokémon
I don't know cuz it seemed better than deadly diseases any who looking at the
geometric mean of our gaming results we found a
2.07% spread in average frames per second between the best performing and
the worst performing CPUs and a 2.4 6%
spread in our 1% lows this puts all but
one of our CPU samples within three
frames per second of each other which gives us confidence that we'll be able
to find some close enough CPUs but but
given that 2.46% isn't 1% it also tells us that we
can't just pull any three chips at random nor can we simply look at the
average returnal for instance is a benchmarker dream because a it's
actually a good game that people might want to play and B it is a stunningly
consistent Benchmark which is great for producing results that we can trust when
we're comparing gpus but the real world
is a lot messier than returnal and while
most of our other games both at 1440p and 1080p showed a similar small level
of variance in CPU performance in a couple of games notably Total War
Warhammer 3 and cyber Punk we found larger variants in the 1% lows this
indicates that as run these games are more CPU bottleneck which better reveals
the deficiencies of our worst chips but as you're about to see not all CPU bound
games are bound in the same ways we went
into this process thinking ah CSG go
what a classic CPU gaming Benchmark it's
a shame that it's been replaced by CS2 and we came out of it thinking ah CS
good ridds I mean on the one hand it
does certainly separate the CPUs from each other and our slowest chip Corola
was the slowest s in csgo but on the
other hand the overall variance is so
high and so different from the entire
rest of our test Suite that it becomes almost an outlier data point having an
outsized impact on our results and this
could be for a number of reasons first csgo uses a game engine that is older
than YouTube which has been useful over the years since it was originally built
just for single core CPUs and it can make use of just about all the single
threaded performance that you can give it but it also means that its
performance requirements just aren't very similar to more modern games that
are going to want to see a number of fast cores rather than just one or two
second csgo itself is also old old
enough that any modern gaming CPU can run it so fast that no professional
Esports gamer even could tell the difference anyway and so fast that
limitations in the software itself can start to rear their ugly heads which
adds potential variables basically csgo
is having its Quake three Arena moment after a long run it's time to drop it
and when we reviewed the overall variance numbers without csgo it shows
just how much of an outsized influence that it had on our results the new
results show far less variance closing the spread to just 46% and 1.43% for the
average and 1% lows respectively which is somewhat reassuring for you the
consumer but still doesn't change that our runaway loser corsola is still a dud
Corola consistently underperformed the rest of the chips by so much that when
we remove it from our results our overall spread and performance goes from
1.43% in our lows to
86% that is a massive decrease so what
the heck is wrong with this thing we don't know for sure but one guess is
that the 3D V cach on this chip could be struggling some way because it fumbles
pretty hard in our factorio test where most of the Benchmark can actually fit
on that 3D V cache another possibility
is that it could be the PCIe controller the part of the CPU that communicates
with our PCIe lanes and consequently our GPU this idea comes from the fact that
when it comes to productivity performance sure it's still ain't top of
the class but it isn't flunking like it used to speaking of we actually found
greater variance between our chips in our productivity tests which kind of
makes sense since we no longer have the GPU getting in the way of raw CPU
performance szip brought us a spread of around 3 to 4% for compression and
decompression and blender hovers in the same realm along with our video and
audio encoding Suites the biggest contributor to the size of the spread of
our sample though is Lugia who takes up the bottom spot in pretty much every
productivity benchmark since it wasn't so bad in gaming this
leads us to believe that perhaps there's a problem with the integrated heat
spreader but AMD has made that much more
difficult to evaluate now that all of their CPUs just kind of run at the same
temperature and then adjust their clock speeds to reach their thermal limit so
across our small sample variance in performance is present but not egregious
of course we aren't trying to quantify variants what we're trying to find is
equivalence so how do we do that it turned out to be a bit tricky we ended
up using ukian distance to determine which CPUs were the most similar
unconventional but also kind of cool here's how it works first we scaled our
data so that our five-digit cinebench scores don't overshadow those low
flacking code numbers then we took those scaled numbers and treated each as a
coordinate for a point in multi-dimensional space think about it
kind of like this if we took a plane and chose two points those would each have
an X and y-coordinate well the ukian
distance is the distance between those two points the closer together the
points are the more similar they are and this can be applied for points that
exist in any dimension in our case a 12-dimensional space for productivity
and the 19th dimension for gaming since we're weighing all of our tests as equal
we can then do a bunch of comparisons to determine which CPUs are the most
similar to one another from that four emerge as extremely comparable EV Mewtwo
Raichu and Zapdos with Zapdos being the least equivalent so sorry bro just the
other three they are outside of our tolerance in csgo but the issue is that
the CPUs that did perform identically in that one game were not the tightest
across the rest of the suite meaning that they can't really be trusted on
games that you know you might actually be able to play in conclusion
productivity saw these CPUs perform within roughly 24% of one another and in
gaming we see a spread of 86% in the 1%
lows and just. 1% in average frame rates
now that's tight tight enough we figure that it will allow us to directly
compare GPU results across our test benches wait Ben jez oh yeah you see
where I'm going with this we found near identical CPUs but what about the other
components do they vary time for another
round of testing the main secondary
performance contributors in your GPU test bench are going to be your
motherboard and your RAM but since those still run at fixed clock speeds we're
not expecting nearly as much variance with RAM for example you set the speeds
in your BIOS and then it's either capable or it's not and your system is
unstable and probably crashes all of our
testing is done at the recommended RAM speed from AMD 6,000 megat transfers per
second and if you want to learn even more about how we test our Hardware
we've got a recent exclusive over on Floatplane.com where we have a
featurelength deep dive looking at the improvements we've made to our testing
processes anyway to validate our hypothesis we took one of our future
test CPUs EV and threw it into both of
our new parallel benches in gaming we
landed on. 45% variance in the 1% lows
and less than a tenth of a percent in average FPS that is more than acceptable
and in productivity we ended up in the. 13% neighborhood that means in our
upcoming GPU reviews performed on these three parallel benches we're going to
consider our results to be accurate within plus or minus about.
25% of course that doesn't mean that our
results will be identical to your CPU or to other media and this is one of the
big reasons that we have always encouraged our viewers to look at
reviews from multiple Outlets whenever making a purchase decision oh before you
ask by the way there does not appear to be any Foul Play from AMD with respect
to review sample selection so you don't
have to pick a reviewer for example that buys their own CPUs versus one that gets
seated at least we don't think so we'd have to buy hundreds of CPUs to know for
sure but it appears that the unit that was sent to us for a review which is
Raichu Falls somewhere in the good but
not exceptional range so I think we can put put that conspiracy theory to rest
again another before you ask is yes
driver updates operating system updates and new software that we add to our test
Suite could change our CPU performance spread in the future and we're going to
do our best to maintain our data Integrity by performing periodic what
we're going to call equivalence checks because you guys have asked for Reliable
trustworthy information and you deserve it which brings us to a big issue why is
this task falling to random YouTubers I
mean the automotive industry for instance has government bodies that are
dedicated to verifying the performance of vehicles and ensuring that companies
aren't cheating on their testing then they Dole out big fines when they
inevitably do cheat on their testing with computer hardware there's no such
oversight we and our peers are this thin
open-mouth thumbnail line between you
and getting ripped off and that's a big problem I mean for one thing we don't
have access to the types of testing that large tech companies have and we don't
operate at the kind of scale where we can say for sure if an observation is a
fluke or if it's the result of conniving suits that are trying to save a quick
Buck even buying 11 chips for this investigation that was a huge investment
from our side and not the sort of thing that we can do with every single review
unfortunately all I'm doing is Rand in right now I don't have a solution to
this other than well we're going to keep trying gosh darn it but it just struck
us as we worked on this project that the fact that these companies don't have to
report things like estimated performance in a regulated and standardized fashion
is kind of crazy especially if you consider the kind of money that they're
asking for their most expensive CPUs so what's next well first is going
to be going through the exact same rig remoral with however many 490s it takes
to parallelize our CPU test platforms
and then slowly but surely we're going to be improving our automations and
increasing our test volume especially once we get the lab's website up and
running but good things take time and we
aren't going to rush a good thing especially I'm not going to rush this
segue to our sponsor delete me your
personal information sounds like it should stay personal right I mean it's
right there in the name well data Brokers and other sketchy companies
disagree so they're sharing your data online like it's a family style dinner
eat eat your skin and bones is what it's
what they're saying to each other thankfully delete me is here to crash
the party they'll find out who's spilling your info and get it removed so
that scammers can't use it to batter you with Robo calls and spam emails never
mind that it can also lead to fraud or identity theft because delete me can
mind that for you now what wiping out data held by hundreds of sites by
yourself sounds borderline impossible which is why this whole time I've been
trying to tell you that delete me can do it you don't have to their Nifty
software and expert Squad can sweep it away in minutes not hours delete me
averages over 2,000 pieces of personal data gone for a customer in their first
two years yeah go on delete me get them
and you should get on over to the link below and use code LTT for a sweet 20%
off if you guys enjoyed this video why not check out our motherboard turbo nerd
Edition video where we went into what exactly are all those little things that
look like cities and towns on the PCB