Welcome to the Labs Process Video. We're going to give you a step-by-step breakdown of everything
we do when we test a product for a video, all the way from ideation to the final results.
Every test starts with an idea. For example, it's been a long time since we've reviewed the 7900XTX
and the 4080, and we want to know how do these perform today? The first step is filling out our
test request form. As much as it would be nice to just go down and ask Labs to test these cards,
they're not going to do anything unless I provide them with a filled out test request form. There
is no testing without requesting. The reason we have to do a request form is that it allows Labs
to track the project. If we don't give them a request form, they won't do the testing because
that's how we waste people's time. In here we put like the name of the project, so we'll say like
4080 versus 7900XTX. Then we'll put a link to the Trello just again for project tracking. And then
here's the first kind of important answer. This is an important delineation. Is it exploratory
testing or project testing? So exploratory testing is going like, oh, we don't know if this is going
to be an interesting story. So we need to test to see if the effect that we are talking about
is significant. Like how much does a browser extension slow down your browser? Maybe it's
not enough to be an interesting video, but maybe there's something there versus a project test,
which is like, it's a new GPE review. We know that we're going to test it. We don't need to explore.
In this scenario, we're doing a project test. Then we give them a due date, so a project timeline.
We can either do a hard deadline or a soft deadline. A soft deadline is kind of like,
if something else of higher priority comes in, Labs can push it back a little bit because maybe
it's a video that doesn't really need to come out on a specific date. A hard deadline is really
important. If it's something that's very timely, like maybe it's a sponsored project or like an
embargo GPE review, then we give them a list of all of the components that we want tested in the
video. For this scenario, we're just going to do the two cards, a 4080 Accelerate and a 7900 XTX.
Then we just let them know if we have them here or if they're coming. Basically it allows them to
be like, where do we need to listen for the next step? Is it like the writer or do we need to like
keep an eye on procurement or something like that? Sometimes we also might want to like
talk to Labs about any sort of materials or components that we need to actually get.
And that'll usually happen during like a kickoff meeting. Then we have to clearly state what our
questions are. It's really important to do this and like the more nailed down our questions are,
the faster the project's going to go and the faster the kickoff meeting will go.
There have been times where I've stepped into meetings where I did not have a solid question
and then it takes a long time, waste everyone's time trying to figure out what the heck I'm doing.
In this scenario, we're just trying to see how well does AMD's 7900 XTX compared to the 4080
is the 4080 really we're going to be worth the extra money. And then we can also select specific
tests. We have a whole test portfolio that we can usually run like where we're just run through
a gauntlet of games and productivity benchmarks, but maybe I just need to select few or there's
some that I absolutely must have. In this case, we don't need to run these through like a million
gaming benchmarks, just a few. We're going to do Cyberpunk 2077, Dota 2, Red Dead Redemption 2,
and they're all going to be at 1080. The next part is like any sort of notable
test boundary conditions. So say we wanted to have something that's like out of the
ordinary, like, you know, maybe we need old drivers, like we want the drivers that were
used when it came out to compare to the drivers that are out today. That's something we can put
in here. And then for any super general notes that might not really have another place to go,
there's a final form just at the bottom where we can fill out and be like,
I don't know, maybe like, you guys want to hang out sometime too? I don't know,
you can ask them whatever they need to submit it and it goes off to them.
Hi, it's Gary. We just received a test request from Adam. Normally this goes to Sharon, but Sharon
is busy doing some other work today, so I'm handling this request. Once it comes into our mailbox,
we simply click on it, brings up the lab's data request. This one's obviously from Adam.
We're testing the 4080 against the 7900XTX.
And then he's actually filled out some specific tests that he wants.
All of these, Cyberpunk, Red Dead Redemption, Blender and Dota are part of our standard test
suite, so that makes it a little bit easier. He doesn't want anything special on it, just straight
up testing. And apparently he appreciates all of us, so thank you, Adam. With that, we'll go
over to the assignment form. This is where we keep track of who's assigned the project,
and also if it's just been assigned or if it's already in progress. For this one,
we'll go ahead and mark it in progress as John's going to start it momentarily.
And yeah, we're giving it to John. And with that, I'm going to hand it over to John. Thank you.
Now we have our kickoff meeting. Basically what this does is that it allows us to
talk with Labs, make sure that our test plan is all agreed upon, and that we are all on the same
page, so that we can make sure that we're doing everything right. Labs will often bring some
concerns and questions. Writers might be like, notice things in the middle of the meeting,
be like, oh crap, I forgot to add that to the sheet. So this is probably one of the most
important parts of the whole process, because it allows for all of us to make sure that we are
aligned on our goals. So we just had the kickoff meeting for this project. Now that we've done
that, we can actually go about collecting the requirements, the project requirements for the
system. So we're going to be using our AMD benches today. And that's something that we're going to
be using going forward for all of our GPU testing. So what we're going to do first is give you a
rundown of all the hardware we use on our AMD benches, and then get into the nitty gritty of
actually building it and setting it up correctly. So first thing we do is we got a
CSONIC Prime PX1600 power supply. We've been using these for a little bit now. They're ATX 3.0
capable. So the nice thing is we can swap quickly between using a 12-volt high power and typical
8-pin power connectors for any of the GPUs that are made by like Intel or AMD without having to
use any extra like pigtails and stuff like that. So that's pretty useful. And for motherboard, we
are using X670E AORUS extremes. We've been using these for a little while on our AMD benches,
not that we've used them in many of the videos, but going forward that is going to be our standard
for our GPU testing. The reason we went with the extremes, we were using the masters for a while,
and then we ended up going back to the extremes because we found that we liked the VRM a little
bit further. It was a little bit more robust, and the BIOS settings we found that it could tweak just
that little bit better than the master for doing some exploration testing of overclocking and stuff
like that with some of the CPUs we've gotten in. As far as other brands, we have looked at ASUS for
a little while. We looked at a couple of ASRock boards, but ultimately we kind of ended up on the
AORUS extreme. So for CPUs, we're going to be using Ryzen 7 7800X3Ds. We actually have a video
upcoming that is about AMD variance testing that we did to basically determine the three 7800X3Ds
that we're actually going to go with going forward. They are the fastest one or the slowest one,
the three tightest ones that we could find to effectively try to reduce any sort of variance
in our FPS basically between the three benches. So we can kind of more easily verify that if we
were to take this card and put them on any of the three benches, we should get the same number.
For RAM, we are using G-Skill Trident Z5 new RGB. The stuff we're going with is got
6,000 megatransfers per second CL30. Even though AMD has come out with new versions of
AGSA that have basically unlocked the ability to use anything higher than 6,000, we still found
that this is kind of the best sweet spot for AMD in terms of performance. You can get up there
where you start getting your timings a little bit tighter and stuff like that and perform
basically the same on like 6,800 megatransfers and stuff like that. But
this seems to be a good price to performance ratio still, so that's kind of what we're going
with going forward. For SSDs, we are using Samsung 980 Pro 2TB SSDs. We haven't explored
anything as far as using PCI Express Gen 5 SSDs yet, but this seems to be a really good
performance ratio on both the reads and writes of both not just the sequential but as well as
the random reads and writes. So it kind of allows us to get the load times a little bit faster and
kind of have more benchmarks running in sequential order without having to wait for all those
extra load times. Sure, most SSDs on the market nowadays are fairly quick as long as we have
something as DRAM in it, but that's the one we've kind of been going with for a while and
seems to be good enough for us going forward. And if it ain't broke, don't fix it, right?
So there's that. And then lastly, to cool the 7800X3Ds, we are using Noctua
NHD 15 coolers. It's kind of been the standard for a long time for a lot of people.
As soon as the new gen version comes out, I'm sure we're going to be exploring that as well
and seeing just how much more beneficial it could be for us. But it's been basically the
workhorse for us for a while and it probably won't change for a little while afterwards. So
that's kind of where we're at this point. So for the specific project, we obviously have to get
the specific cards that we need to test with. So we have a reference, AMD Radeon 7900 XTX.
It's been the one that we've used in all of our videos that we've ever covered on it.
It's got a pretty sick looking cooler. And then the video card we're going to be using today is a
PNY Accelerate RTX 4080. This seems to be currently the one that is the cheapest on the market,
but that doesn't necessarily mean it's the worst. And this is the one that the requirements
of the test plan specifically outline. So that's the one we're going to be testing today. And lastly,
we obviously need some sort of thermal paste to go between the D15 and the 7800 X3D. And we use
Noctua NTH2 thermal paste. We've been using it for a while. As far as application, more is always
better than less to some degree. And so when I usually put thermal paste on, it's in an equal
sign, which is something that you'll probably see in the build part of the video. So I'm sure
everybody's going to have tons of arguments of, oh, you've used too much or used too little.
There's been so many discussions about that. It doesn't seem to affect the thermal performance
in any sort of way. So as long as we have enough, we're good to go. All right, now we've gone over
the actual hardware, let's get to actually building. Now we've got the system all built together and
up and posted into the BIOS, we're going to be going through making sure we have the correct BIOS
version as well as configuring it to the standardized configuration that we do for
all of our benches. So when we look at our test conditions, we usually outline both drivers
as well as BIOS revision, because there might be an instance where like we're doing some sort of
exploratory testing and being like, hey, we actually don't want the latest version of the BIOS. We want
to go back four or five revisions. That's why we outline it in the actual test conditions. In this
case, it's saying F11D, which is the current version of the BIOS, but obviously we're not
running F11D, so we're going to have to update it. Update your BIOS kids. Yeah, there's, especially
with like new AMD CPUs, 7800X3D in particular, a number of motherboards don't actually support it
correctly out of the box. So if you don't have the option of actually booting into the BIOS,
luckily most manufacturers include some sort of BIOS flashback now. Just slot the USB into the
correct port on the motherboard, make sure obviously the file is named correctly before you do that.
It's incredibly important if you want a computer to work right. Then once we finally get the BIOS
updated, we're going to be going ahead and configuring it and then setting up Windows.
Okay, now we've finished the actual BIOS update. Let's go ahead and start configuring things. First
thing we're going to go to advanced options, so we can actually see everything. We always turn on
expo or if it's Intel, it's XMP. Verify that rebar is on. Most AMD boards usually have it on by default
now, but it's always good to go to. It's usually under IO ports or some sort of IO naming scheme,
whatever manufacturer makes it. Inside here, we also turn off the IGP or integrated graphics,
especially if we are testing any sort of dedicated GPU. All it does is sometimes confuse some games
or programs into wondering which actual graphics card to be using at the time.
And then from there, the only other thing that we do is go into the fan curves
and we manually set it so that it is a linear scale.
So it basically looks like that. Then at that point, we can actually start installing Windows.
For Windows, we use a optimized Windows image that our software developers have
created. Basically strip back a bunch of stuff that we don't need to be running in the background,
as well as adding in some programs that we actively use during our testing. We actually
reinstall them every three months. We'll usually update them basically and create a new Windows
image that has our latest updates so that we don't have to constantly do this rigmarole of
get a million updates, a million drivers. We can just basically install the latest drivers
and then we have basically all of our programs. So now we just got to let this thing run.
Very exciting stuff, like watching paint dry. In our test conditions, we'll usually outline the
exact version of Windows we want to run. Going back to that, we might need to recreate an old
set of parameters. So we might not be using Windows 11 22H2. We might use 22H1 or that's why
on that list, we actually have all the other versions of our images still on there so that we
can actually go back and be like, oh, actually we need to use the version from May. And then we
basically, if we need to recreate that set of test conditions, we have to go and also stop
Windows updates from updating. Now we're in Windows. We are not going to install the Gigabyte
Control Center. So basically at this point, we have to go through update all of our drivers,
as well as install all of our programs that we're actually using for the project.
Okay, so now we've installed the chipset drivers. We can make sure that we get the latest NVIDIA
driver as, or not necessarily the latest, because if the project does require an older version of
the driver, we got to get that version instead. But in this case, in this instance, we are using
the latest version. So we're using 537.42. Let's get that downloaded.
Then we're just going to make sure all the video settings are set correctly,
so we essentially turn off G-Sync, any sort of potential accelerators, V-Sync,
so we can get the, basically the raw performance of the card. Update your cards.
There's a lot of things that NVIDIA doesn't post or advertise, like a number of three series cards
don't have rebar enabled out of the box. So there's actually a bunch of articles that NVIDIA
posted that are really useful, such as UEFI fixes, DisplayPort fixes, and things like rebar
patches and stuff like that. So if you find your card isn't matching up for whatever reason,
or is doing something funky, good to check those out, because sometimes it might be as simple as,
oh, I need to do this little update. Boom, I've got more performance, or oh, my video doesn't
seem to work in the BIOS, but it works fine in Windows. Do a firmware update. Oh, sweet, now
it's fine. Alrighty, now that we've got that installed, let's just make sure we don't have
any other weird drivers. Looks good to me. Usually if there's no yellows, we're happy campers.
We usually do all the Windows updates according to what our project plan is. In most cases,
it's usually just as simple as getting every single update that's in there. A lot of times,
what will happen is, you know, there will be driver updates and stuff like that, but if you
already have a later driver than what Windows sees, often you'll get like a download error a lot of
the times, because Windows is flagged, oh, you actually have a newer version, we don't need to
get this. And what we do is we tend to update our Windows image every three months to help include
a lot of these Windows updates. We essentially have to download a lot less of them. It is time
consuming and the less time we have to spend on constantly downloading things and updating things,
the better, right? Then we can get right into the actual testing. It saves us a lot of overhead,
basically. So one last thing we're going to do before we get all of our games downloaded on the
system is we're going to go through and just do some last minute Windows optimizations.
So first thing we're going to do is we're going to do some trimming on the actual drive itself.
Just helps speed it up just that little bit better. Then we're going to go and go to percent
temp percent to basically get to our temporary folder. And we're going to basically delete
everything we can in here. So one last thing we're going to do is we're going to set the recycling
bin to 15 megs. The reason we do this is just that if we delete any files on the system, it'll
just delete them right away. It won't sit in the recycling bin taking up extra space we don't need.
Basically, it skips a step of us happening at empty out the recycling bin in most cases. So
kind of useful in that regard. So once we have all this set up, the only last thing we have to do
is just make sure whether it's NVIDIA, Intel, or AMD, we have any sort of variable refresh rate
stuff turned off, as well as B-sync. The reason behind that is we don't want to potentially give
any sort of advantage to one manufacturer over another with any sort of technologies they
might have running in the background causing any optimizations that are effectively outside of
Windows itself doing hardware optimizations. That way it kind of keeps the playing field a little
bit more level. So we're going to pop into here. I doubt we'll have G-sync on here. No, because we
are currently plugged into our NDI viewer to get all of our capture. So for this specific instance,
we won't have G-sync enabled. But typically, because these monitors are actually G-sync capable,
anytime we pull up an NVIDIA card inside there, it'll immediately enable G-sync. So we have to
make sure we go in and actually disable that. And then what we do also is in here, we've got to
manage three settings, scroll down to vertical sync, and we make sure we force it off. That way,
because there's some games that they don't either give you the option or they tell you it's off,
but it's not actually off. It's kind of just ensures that no matter the game, no matter the
settings, if somebody forgot to turn off V-sync for whatever reason, this basically forces it off.
So there's no opportunity for the game to even interact with, hey, by the way, I need V-sync.
So it kind of takes that guesswork out of it. And then we go into display settings.
We're going to be running at 1080p, so we just leave the resolution as it is, but we change our scale
to 100% rather than 150. Some of our markbench benchmarks do still use CV in order to do character
recognition. So if the scaling is at 150% and you're on the desktop, for example,
the new version of Hitman has a launcher that it shows up beforehand that has all the options
and stuff in. How markbench works in that one is it actually will look for the specific word.
If our scaling is different than what it's supposed to be, markbench will be like,
I can't find this because it's effectively looking for a picture at a different scale
than what it actually is. So if we set it to 100%, basically we're going to have no problems.
And then normally what we do is we check to make sure we're at the highest hertz we can be for the
monitor. Because we're going through the NDI, we actually have to run at 60 hertz because that's
the maximum the actual NDI itself supports. But typically it's 144 for the actual monitors themselves.
So lastly, we go into change default graphics settings. We make sure that hardware accelerated
GPU scheduling is on. This is at Windows level. This isn't hardware specific such as
NVIDIA Reflex or AMD's Antilag. That is something specific to the hardware,
not Windows itself. So this one, it's the same across all the platforms of hardware. It doesn't
give an advantage to NVIDIA over AMD because it's scheduling at the exact same. Anything else that's
in the menu, usually we have like optimization for Windows games. We turn that off, as well as in
some cases, there is a variable refresh rate option in here. It usually shows up for Intel
and AMD because theirs interact a little bit differently than G-Sync does. And so we make
sure that's off as well. Because again, we don't want to give advantage to one specific card. We
want to see raw performance of each of the cards stacked against each other so we can understand
what is actually the best option. Now that we've gotten all the Windows updates installed,
next thing we got to do is we have to install all the programs as well as any games that we're
going to be using to do this project. So in this case, we need to install Cyberpunk, Dota 2,
and Red Dead. So we already have Steam pre-installed on our Windows image so we don't have to worry
about actually going to the website and getting that part, but unfortunately it always needs to
update anyways. So it's kind of a moot point. While that's happening, what we're also going to do is get
markbench from our local drive, as well as some patches that I need to install as well.
So we are going to install, nope, Cyberpunk. We're going to install Red Dead.
And then the most annoying thing is that Dota 2 doesn't stay in your library half the time,
so we are going to go here and do play now.
Okay, so those three guys are downloading.
We've got markbench extracted here. We need to add another harness in for the Dota 1,
so let's get that real quick. And there it is, ready to go for when we are ready to use it.
And then the other thing we need is Blender Benchmark, so let's quickly grab that as well.
So although markbench does have a Blender Benchmark actually integrated into it,
we do need to still download the application itself to download the files required for it to run.
There's a little bit of bugginess with the CLI. When it tries to download the files,
it effectively says I can't do this and quits out. So if we download the actual program and have it
download the files, it just makes the automation a little bit simpler on our side because it has
the files already completely installed. So effectively we want version 3.6. We're going to
download the benchmarks. Again, let's just make sure that markbench is all prepared, just like when
we have to go through any of our game settings. We have to actually go download the programs,
download the games, actually set up the settings ourselves. That stuff is not currently automated.
Now we have all the games downloading and everything's kind of getting set up.
I'm going to throw it over to Nick and he can talk a little bit about markbench.
Fundamentally, markbench is our test automation and orchestration framework. It's not just simply
a simple desktop GUI anymore. It's a constellation of services and components that work together
to form the ecosystem that we conduct our testing in. But it all does start with the test
harnesses, which most of ours are in Python because it's most easily automatable for us.
However, it doesn't need to be Python. They can be PowerShell scripts, Ruby, GoLang, what have you.
The important part is that each test harness gives back a exit code of zero or one, zero to indicate
a successful test run, anything greater such as one to indicate that it failed. Because that exit
code is then read by the next layer of markbench, which is what we call auto bench. It just comes
before the GoLang GUI and it really contains all of the major features of our orchestration.
This layer is what separates what we're doing from just a collection of PowerShell scripts
to something that is a lot easier to manage long term because inside of auto bench, that's
where we get our orchestration. So with the test harness, I have the capability of running
individual tests, but I want to be able to group those into a bunch of test suites such as let's
say I want to run Dota 2 four times at one resolution and then Red Dead Redemption 2 at a
different resolution. So that's where the auto bench side comes in. Auto bench is where we run
presentmon in system logger in conjunction with the test harness so that we get a similar timestamp
between all three because this eliminates the human error of starting a test and then starting
hardware info or like logging on the side, which then would separate our timestamps by
sometimes multiple seconds, which makes it easier to trim the sessions later downstream,
which John will explain later. And then so the auto bench is all command line interface,
so to make it easier for us and John and writers or whoever wants to run Markbench,
we added a Golang GUI. So we went with the Golang GUI because the framework that we used
was very successful for us for other GUIs such as the keyboard tester, the robot, and the latency
tests. So anytime you see Markbench on screen, that's the Golang part, but underneath all the
work is really being done inside the test harness as well as the Python orchestration layer.
So once the test is done running, let's say we just ran an individual test and once that test is
done running, it sends its data up to a service called data ingest server, which is a Golang web
API, which takes it in that data in a protobuf format and then writes it to our Postgres database.
It's from that Postgres database, all of our data sits at rest and then is disseminated or sent
to all the various tools that John can use to evaluate and report on that data. And we use a
couple tools, notably Grafana and Redash for a lot of visualizations. And then we have internal
tooling that we just kind of collectively call the Harbor Tools portal. And that contains anything
that Grafana or Redash or another off the shelf tool doesn't provide us. Now if we go back up to
the test harnesses, on their own, they're fairly dumb in that they can send keyboard and mouse input,
but they don't always know what's going on the screen. So we used to use OpenCV locally to do
template matching, but it's more of a brute force approach. And it's very flaky as it depends on
certain resolutions. So if you're doing template matching with a resolution of 1920 by 1080,
you need to select screenshots of the path or the flow you want to go through the menu of the
particular game. But then if you want to do it at a lower resolution like 720p,
nine times at a 10, you have to take a smaller screenshot. So it doesn't scale very well. And
that's where we start getting into some of the services that we're adding. The first one being
Keras optical character recognition or OCR. This allows a test harness to send a screenshot of the
system it's running on with a target word and Keras will come back within a few milliseconds
is that word on the screen or not. This reduces flakiness in the test harness,
which makes navigating menus a hundred times easier. And in the future, we're going to be adding
services that do a bunch of computer vision things such as depth estimation, object detection and
whatnot, all in the effort of making the test harnesses more a lot smarter. But we'll talk
about levels of automation in a bit. So the benefit that Mark Bench and its ecosystem gives us
over just a collection of PowerShell scripts for one is maintainability at the scale that we're
trying to achieve. PowerShell scripts would or some auto hockey scripts would become messy very
quickly because eventually we'd have probably like 100 to 200. And as best as you could organize
those in some directories, it's just not a it's a maintainable nightmare. Investing in such an
ecosystem and the automation of Mark Bench, it allows us to build it to support multiple platforms.
So not just Windows on a desktop, but handhelds and macOS and Linux as well. And it normalizes
data for us so that when we capture metrics and we set it at rest, it's a lot easier to
query that data. Back to you, John. Thanks Nick for that explanation of Mark Bench.
Now what we're going to do since all these games are finished downloading is get them all set up
and then we'll be able to get our benchmarks started. So we're going to start off with cyber
bunk here. We have documentation that we are working on that we will be publishing in the near
future of all of our settings and configuration for the various benchmarks that we include in our
videos. I believe we've already released a couple of them on the forums. So I'm sure we'll continue
to update that as time goes on. Okie dokie. Yeah, the new update for cyberpunk came out. So that's
cool. So first thing we do, we go to settings. We're going to go to video mode and we're going to
make sure we're at 1080p because that's what the project requires. You're going to be turning off
any sort of things like low latency stuff like that. I think that it could benefit that specific
card because we want the raw performance of the card. We don't want any sort of additional things
like DLSS or low latency that can certainly help benefit the card and it might get better
performance for sure. But what we're trying to determine is apples to apples between that 7900
xdx and this 4080. So we're going to be turning off low latency. We're going to be making sure
that we're on full screen. We don't want borderless. Then we're going to go to the graphics tab and for
our specific test we aren't calling for ray tracing. So we're going to actually be turning off
ray tracing. We're going to be going just strictly ultra and we're going to go ahead and turn down
FSR to zero. Go back up a tiny bit. Go back down a tiny bit. Each individual time we change something
we're going to hit apply. The settings for cyberpunk are still kind of a little bit weird
where you might change some sliders and you might hit apply and sometimes it's forgetting
that you hit apply or you change the quick preset. It doesn't actually use the preset because you
haven't applied it. So when we go up and down on the FSR that kind of helps with that and then
we're going to go ahead and turn FSR off because we don't actually want it. And that's basically how
we set up our non ray tracing version of cyberpunk. So at that point we can basically exit out of that
and move on to the next one which is Dota 2. Dota 2 is one of our more recent benchmarks that we've
been working on. It uses a replay from I believe Dreamhack. A lot of visuals on the screen so that
we can basically tax the card as much as possible. So in Dota we're going to go up to the cog, go to
video and we're not going to use monitor's resolution. We're going to use advanced settings.
In this case we need 1080p. We're going to be checking on to exclusive full screen
rather than that. Turning off low latency yet again.
And then for rendering side we just pulled the slider all the way to best looking.
It's basically a preset that has all these specific settings. These things always off.
That sort of thing. That's all we got to do for this guy here. So we go ahead and close that.
Okay now we've done Dota 2. Lastly we just need to set up Red Dead Redemption. So we're going to
hit Z to go into settings. Go to graphics. Make sure the resolution and the screen type are correct
which they are. We're going to be turning off v-sync. Now pull the slider all the way to the right that
usually gets us all the way there but we still have to actually go through and manually check these
because like I was saying before Red Dead's a little bit different in that this slider
doesn't necessarily mean you're going to go everything ultra if you hit that.
If you have lower end hardware it actually kind of tries to adjust that slider based on
your hardware. So on like a low end laptop you go all the way to favorite quality and
everything might still read as high. So it's something that that's why we have actual screenshots
of our settings for it so that we can actually make sure that they're the same even after we
change that slider. So we should be sitting at ultra 16 ultra ultra ultra high ultra high ultra
medium high off high and this is always the one that kind of gets a little bit fussy.
That looks right. Okay and then we're going to hit enter to apply the changes. I'll often
recommend don't alt f4. There are some games out there that the settings don't save until you
actually properly quit the game. It's like a fail safe so if you just exit out the right way
shouldn't be any problem. So first thing we're going to do is set up mark bench here so we
actually start running the benchmarks. So in order to do that we have to input a few values
first. So we're going to go through all the different boxes and options that we have for
the tests we're going to be running today. The first box that we have that we can fill
information into is called the project slug. What that usually entails is every time we create a
test plan we usually assign a sort of either a character value or an actual code name for that
project so we can later look up the data if we have to or reference it in a future video or
something like that. It makes it a little bit easier on our side and then that way the data doesn't
get lost in translation of oh I'm looking for this project but there's eight folders that are
named similarly. It helps kind of basically organize that into a more clear overview of which
project is which. Next is the session tag so if for example we are altering the values from our
standard testing run so we're turning off anti aliasing or we're you know messing around with
which preset we actually use. This helps us assign an actual tag to those specific sessions
so that when we're looking at them later we can be like oh actually it's because the reason why the
values are completely different is because this is the set that we that we ran with experimenting
on anti aliasing or experimenting on you know running low versus high shadows or something like
that so we can basically put in like no dash a and later when we go to look it up it'll actually
have that listed on those specific sessions. Thirdly we have what's called the test parameter
there's a bunch of different options in here that we have currently it's something we can edit
whenever we need to but basically it assigns those sessions what the test parameters we
hope to use for that specific session so for example for this project we're only running at
1080p so we're going to be setting just on here games 1080. There are other options in here like
DX11, RT, stuff like that. This doesn't actually alter any game settings themselves this is just
for our internal use on all of our sessions so we know if we if that run was for 1440p if it was
for ray tracing if it was for XCSS you know that's all those different types of visual parameters
that are outside the norm basically. So for the project slug we are going to be putting in the
one that's on the test plan which is 4080-vs-7900-XTX-2023. So once we have written our project
slug in as well as our test parameters now we can get into actually editing which tests we're
going to be running as well as some of them have different configurations that we can alter
that are non-game tests or for assigning certain values of what the how the game is supposed to
interact that sort of thing. So first we're going to cover blender benchmark we click on it here
any of these ones have different test arguments in most cases so if we click on the blender
benchmark we have a bunch of new values in here we can adjust so things like scheduler delay,
recording delay and recording timeout. These are all in seconds we usually run a scheduler delay
of five seconds just to make sure that the program or the game can have time to close and be
satisfactory with Windows and all copacetic to be able to open again especially when it comes to
steam when you have like cloud game saving enabled and stuff like that it has to take those few
seconds to actually do the cloud save and sync it up what can happen in some instances if we
don't set the scheduler delay is it will run into that bottleneck of oh but the cloud saving
hasn't synced yet are you sure you want to load the game and it kind of gets everything all hung
up so when we set it to five seconds it usually helps give steam enough time to finish that
option and then actually continue on with the next run. So the fourth box here is the number of times
that the test is meant to repeat so it's for that specific test not for the all the test as a whole
so in this case we usually run three for all of our tests that doesn't mean that we only run three
ever and that's it if we find that a result is outside of norm not within our our typical
variance which usually about one to two percent we will do another run till we have three valid runs
that are actually within that variance so we can confidently say that these are the numbers.
So in blender benchmark we also have options for things like the scenes that we use because there's
three different scenes that it actually runs through we're just going to be doing all of the
scenes so which entails classroom junk shop as well as monster that's typically what it runs
in the gooey version with the version that's available through markbench we actually use
the command line interface so we're not going to see anything pop up on the screen for blender
specifically but it is running in the background and we'll show that when we pop over to the actual
command lines to see all the stuff that markbench is actually running then we have which versions
that we can run blender on the blender benchmark on we're going to be doing 3.6 because that's what
the test actually outlines the test plan outlines and then we're going to be testing the GPU not the
CPU because that's what we want to test today is the actual GPUs then we're gonna go ahead and check
it off and move on to the next one which is cyberpunk same thing five and three and then down here we
have for what our keras ip's should be so i can actually talk to them some of our harnesses don't
use keras still um it's something that we're kind of developing as we've been getting into these
future automations and stuff like that to be able to add these tools back into the toolbox of
basically streamlining how each individual harness is so when we get down to red dead
for example it's not going to have that listed right now but we're hoping to have that in the
future so we don't have to do anything else on this one just five and three go down to dota
same thing five and three and we go to red dead we're just going to be setting red dead to five
and three again but here all we have is preset which is current this can't be changed it's a
legacy harness that we've we're currently in progress of updating to use keras as well as
some of the other things that nick will be discussing as far as our levels of automation
goes now we've got everything set up for mark bench it's time to hit the go button and while
this is running the benchmarks i'm going to kick it back over to nick and he's going to talk about
the levels of automation so coming back to automation we're going to focus on the test
harnesses and talk about the degrees of automation because one of the most annoying questions to try
and answer is is it automated when someone asks you that it really depends the answer to that
depends on the scope is a bunch of power shell scripts that run your game and then output something
to a log file considered automated or is it only automated if you can press a button and walk away
and it handles every edge case that you could possibly think of to help talk about automation
internally we've come up with a milestone system and so we come up with the milestones of one two
three and four so when we speak of a test harness taking a game for example a game harness that has
a level one automation it simply just launches and runs the benchmark id like akin to that just a
power shell script doing something very simple and then when we reach a milestone of level two
it can do a lot more such as dealing with advertisements or pop-ups in one of their
launchers or it can report on game settings and then the next level to that is level three is
where it can actually configure the game for itself and also reach out to maybe some more
services like Keras or whatnot our definitions of one two and three are fairly loose because every
game is a little different some games are a lot easier to report on resolution because the it
might be reported in a .ini file that we can grab from app data where others might only store their
settings in some proprietary binary format that we can't even read without some finagling so this
is roughly how we categorize it just so that we can at least internally when someone asks you is it
automated we can say yes to a particular degree and then we get down to number four which is the
future this this category encompasses things that we can't do yet we have something we call our
automation toolbox where we include all the different services such as the keras optical
character recognition uh and uh pie direct input or pie auto gooey it's all the different things
that we can use in our test scripts to achieve the automation of a game so one to three incorporates
those ones i've listed where four is some of the tools that we haven't come to yet the future is
where we'll start to talk about ndi and how we use it for video recording as well as computer
when we started automating our game suite we relied on games that had a canned benchmark
due to the fact that it's a lot easier to automate as you just have to navigate the menus and then
start the benchmark however the problem with that is that not all built-in benchmarking
utilities are representative gameplay and not all games come with a canned benchmark which limits
our selection for our game suite to expand our game suite we started looking at games such as
dota 2 and rocket league that have a very robust replay system in the future we're looking to
incorporate computer vision tools such as depth estimation and object detection to expand our game
selection to any game before we can use depth estimation or object detection we need to get a
video stream off of the test bench to another computer to run the machine learning to do so we
use ndi or network device interface to get video stream from the test bench to the machine learning
computer ndi allows us to convert HDMI signal to video over ip which then can be picked up by any
other client that's on the subnet for example on our computer on the cart we have a client that's
picking up the ndi stream being sent from the test bench which then runs our experiment with depth
estimation and depth estimation is the task of measuring the distance of each pixel relative
to the camera depth estimation could be used to instruct a player to walk down a hallway by
chasing the darkest point on the screen we are also experimenting with other techniques such as
optical character recognition and object detection on a video stream the ndi encoder that we're using
is an n6 from kill of you which has an HDMI in HDMI out and then your Ethernet a particular nice to
have on this unit is the small display on the front which allows you to see the activity of the
unit as well as the ip and what video stream it's sending out okay so all of our benchmarks have
finished let's take a look at the results as you can see here it actually says we've ran
zero zero for dota 2 so we didn't actually get any valid runs so we're gonna actually have to do some
troubleshooting figure out why so that's the thing is that markbench allows us to identify some very
early on possible bugs that might occur with the run and it'll usually terminate so if we scroll up
here you can actually see we get to a point where it says test failed and it's for this run here so
it says game didn't start in time check settings and try again so this is something that we've
programmed into some of our logging so we're gonna have to actually go into our dota 2 settings and
it could be that maybe something got reset when we swapped cables or something like that so we're
gonna take a look at that and get it set back up to the correct settings and try running it again
so we're in dota 2 let's take a quick look at the setting and see what might have went wrong here
so we go to video ah there's the problem it changed itself back to 4k it's a good chance that
while we were swapping some cables around or something like that when we were doing the initial
setup I probably swapped itself back to native resolution and so it caused our markbench runs
to fail so let's change that back and then we'll rerun it here the upside of this is is we don't
have to run every test we just ran again we can actually unmark everything else and just specifically
target dota 2 now so let's go ahead and get that started and hopefully we'll come back with some
actual runs
so looks like we got three runs done for dota 2 now so we can actually upload all this data to our
data ingress server and start trimming so let's get that done now we've uploaded the data to our
server we can pull it up on grafana which is our data visualization dashboard from here we can pull
up all of projects that we've run any sessions that have occurred from basically any of the uploads
so on grafana itself we have the ability to check our projects as well as in this specific
dashboard we're looking at GPUs so for our current project it lists the 7900 XTX as well as the 4080
for tests it lists all the different tests that we've run for a given project and then there's
parameters so in this case we only specify games 1080 so that's the only one that shows up on the list
and then there is the option for the bench which essentially that takes the actual PC name when
when markbench logs it to help us identify which system we actually ran on and then lastly is the
tag which is blank because we didn't label any tags for anything that we needed so from here we
can pull up any of the sessions so let's take a look at dota 2 first here so as you can see here
it's kind of a little bit of a mess if you don't know what you're looking at here but effectively
this is the beginning load this is the main menu and then we have our little load screen
and then we have that initial bit before the actual benchmark occurs the actual bits that we need
are from this valley to this valley and then this last bit here is when it goes back to the main
menu so that's how we how we kind of know which game it's all specific you kind of have to know
what you're looking for at the beginning of like determining oh which part do I actually
need to use so what we all all we have to do at this point is just drag from there to there
apply the correct time range sometimes it skews it by a second if we do that that gives us our run
and then basically I just do that for all the sessions that we have and then once we finish
trimming them we take the data and we put it into our trim session tool on the harbour
LTT Labs page and that basically gives us a graph ideally we want all three sessions to be
basically almost the same however as I outlined previously red deads a little different there's
a bit of rng that occurs during each run so they don't look the exact same on here but the end
result of the averages in the first should be the same or within that one to two percent so at this
point all we gotta do now is trim all the data so we can send it over to sammy and she can start
working on getting those graphs created so let's get to it once the game has finished and the data
is uploaded to the server we need to mark an in and out on the benchmark data to eliminate the
menus and any loading screens from when we calculate the FPS currently john's doing this
manually in grafana by literally looking at the pattern of the frame times in the FPS we're trying
to help them out by having machine learning doing something similar by training a model based on
the behavior of the GPU its temperature and core clocks and memory usage as well as the pattern
and frame times to detect the in and out of a benchmark however that still can be faulty so
we're looking at eliminating it all together by using computer vision from the ndi streams as you
saw earlier to mark an in and out very precisely as it's close to the frame when the game first
loads as we can find we're also looking to apply machine learning and models trained on GPU behavior
to identify erroneous runs or bad runs based on historical data now that we've finished trimming
all the sessions grafana will actually spit out a list of all the different trim session ids
so we can use them to generate our game test report they might ask us how do you know these
are good trim sessions maybe your settings were bad or maybe you know it was just every run was
terrible that's where our other dashboard comes in redash this basically shows historical trim
data that we can use to help verify that these runs are in fact the runs that we need this is
something that gets generated over time if we have a game that we haven't done a whole lot of
trimming on it will be there will be less numbers to be able to verify against but ideally we want
the trend to be the same or similar so in this case if we're looking at cyberpunk on a 4080
on 1080p we're getting 190 average ish to 180 average ish and 154 to 153 for a 1%
so what we can do now is we can use our we can compare our trimmed sessions versus historical
data to verify that it's all correct so what we're going to do is we're going to go on to our
harbor Labs tools again and we're going to go through and we're going to start creating an
actual test report so what we do is we want GPU by component we're not going to include a line
chart because we were just looking at those so we hit create now what we do is we take all of these
session IDs and we hit add let me do it for the other card and so what that does now is it actually
generates a report for us to be able to see visually both with a bar graph as well as just the numbers
what our actual results are so if we're looking here at cyberpunk 2077 on 1080p
the 4080 for these three runs that we just did has an average of 193 and a 1% low of 154
so we're actually above our previous average now these numbers were with a reference card
obviously the accelerates a little bit over a clock so it stands for reason that 193 seems
accurate and so we can keep scrolling down so taking a look at dota 2 we can see that the
numbers appear to line up with what i have experienced in the past we don't have a whole
lot of historical data on it currently because it is one of our newer harnesses that we actually
have been developing and i've kind of just been beta testing but based on my personal experience
with it these numbers do appear to be accurate okay so lastly let's take a look at red dead 2 here
so now we've taken a look at some of the numbers
we can publish this report and get it sent off to sammy to start generating the graphs
so what we need to do first is put the report name which is usually the project slug
um so 40 versus 7900 x tx 2023 results in some cases if we decide to have separate reports based
on like ray tracing versus non ray tracing stuff like that even though those headers of like the
games 1080 um help us filter that if in some cases we just want to look at those bar graphs separately
and the data that they've generated sometimes we will append uh rt results or dlss results or
it's just a description of effectively what the reports based on so we're just going to do results
for 40 80 dash vs dash 7900 x tx dash 203 person then for a reference component
basically what this does is it generates a relative percentage um based on the data so
effectively like oh we can see that the 40 80 is negative 1.62 percent in performance versus
the 7900 x tx we can choose whatever component we want it's helpful for when we are looking at
our big GPU reviews because we can actually be like okay we want to focus on this component it's
not the top of the stack but like for example the rx 7600 we want to take a look at and see
how it compares to everything else and it'll allow us to to basically find these relatives a
lot easier than doing the math ourselves so it's very helpful in that regard um so let's go ahead
and we're going to click publish now this will create a report so if we actually go over here
we can see we actually have the 40 80 versus 7900 x tx results we click on that that basically gives
us the same report um that we were just looking at so i'm just going to copy the url here
into an excel sheet that i was working on go ahead and save this we'll get this sent off to her
and then she can start creating our graphs hi my name is sammy i'm the data visualization specialist
for the lab um i started a few months ago and ever since i started we've made some changes to
the graphs um previously they were done by the writing team um but since then i've taken over
that role um and yeah i'm here to walk you through uh some of the design choices that we made as well
as um the process by which we make these graphs for our videos previously the graphs that we had
had a lot of limitations in design choices um in terms of placement or colors or fonts that we used
mostly because we were making them in excel nowadays we're making them in a combination
of python and uh doby illustrator as well as adobe illustrator plugins mainly um one plugin that's
specialized for graphing uh called datlion datlion allows us to make customizations that we weren't
able to do previously um so one of the first things i did uh was use the style guide that was
developed by our creator warehouse graphic designers um they had kind of set out a palette as well as
some fonts that i could use and toy around with i tried to maintain a design design philosophy
that focused on eligibility um so as we've kind of gone through multiple iterations of the graphs
since i started i've made changes to emphasize on that based off a lot of feedback that we've
gotten from our viewers in terms of design aesthetic um one of the things that i really like to focus on
was an almost laboratorial or chalkboard kind of vibe um we wanted to evoke something that was very
similar to the nasa kind of retro space age kind of look we also incorporated a more modern twist
um inspired by the cyberpunk genre as well as more sci-fi films that have come out in recent years
we've focused on using as well a palette that is uh as accessible as possible for example the choices
um of which colors come first in our bar graphs we chose between colors that were very contrasting
another thing we did was uh we put the colors into um different accessibility tools to kind of
measure how visible our color palette would be for people with different types of color blindness
um and we've essentially developed a order and a palette that is as accessible as possible across
as many different types of um color blindness as possible in order to make our graphs legible
so this is an example of one of the game graph templates that i've developed um one of the first
things that i noticed when i came in was that the previous graphs um their title didn't really jump
out at you one of the things i really wanted to do is make use of like kind of like the very like
stampy Labs font that we have to kind of grab attention um for me one of the things that i want
to focus on is like what is what am i looking at and number two what is this me and so the first
part what am i looking at is um kind of answered immediately by this very bold and very loud um
title or header um and so we've added some design elements sometimes they'll be added in especially
if a graph is you know maybe maybe a little more bare if they're only like two or three
things that we're comparing across but you know if they're 12 13 CPUs GPUs whatever that we're
comparing across then obviously we won't have no space for that we then have the bars themselves
um the bars themselves are sorted now by uh best to worst um we'll still keep the higher
is better um in the subtitle of the graph however um we just want to make it like at a glance more
obvious um and another design element that the editors have helped me incorporate was um a marker
that kind of pulses right next to the GPU that we're talking about previously one of our concerns
and the reason why we weren't able to uh sort by best to worst was because we were worried that if
we swapped around the order of every single GPU it'd be hard to follow oh which one is where um and
so that kind of solved our problem and ever since then we've kind of adopted this philosophy of sorting
by best to worst as best as possible we've kept the grid lines kind of sparse because we want to
make sure that the graph isn't overcrowded with different design elements um and another design
choice we've made particularly with the game graphs and the FPS um graphs is that we've put
one percent low on top and we've made it a bolder color than the average so we've decided that one
percent lows are more indicative of overall gaming performance because average values can get pulled
up by um high FPS anomalies um that might happen over the course of a game when the one percent
lows are really low uh the gameplay ends up appearing as really choppy and so that's one of
the reasons why we think that actually the one percent lows the statistic or the data point
that we really want to highlight instead and so one thing that's been added since we've made a few
changes around the lab is that we've added this uh project slug that follows a project from the
beginning to the end and another thing that we've added is also this version number um so we will
change and update the version number every time of new set of graphs is cut um and that will also
ensure that we have the most updated version of graphs in our video and finally we have our
Labs logo in the bottom bottom right uh this Labs logo essentially tells viewers that this graph was
developed by Labs with Labs data so now we're gonna get started with our graphs um i've already gotten
a uh spreadsheet from adam with the list of graphs that he wants me to do uh three of them being
the game graphs so one for each game and then uh the blender benchmark graph and as you can see here
it says um cyberpunk 2020 2077 uh red dead redemption 2 dota 2 blender 3.6.0 and higher
is better and includes all of the values for the presets as well as the resolution x-axis label
project label version number and all of that so that allows me to know what exactly writers want
included in their video and what data they need as well and there's also room for writers to add
descriptions like if they want to include some kind of like asterisks at the bottom with notes or if
they want to cut out a certain GPU or CPU or if they want like another version of the exact same
graph with extra information or less information included we're just gonna open up a new file
and that will be one layer for each graph as well and i will copy and paste um the layers from our
template so for our first three graphs it's going to be a game graph and so for graph number one
just gonna paste it in and align it you can see that there are already numbers in the template
that's just to help um serve as a placeholder so that i know what graph is which uh that data
will be scrubbed later and then the final graph is our blend blender benchmark so i'm just going
to copy our three series benchmark graph now i'm going to import all of these project labels into
illustrator using visual studio code and a python script that i've written so here i'm just gonna
load in the packages that i need and here in chart names i'm just gonna pull up my file and file
explorer and copy the path to the file and paste it into here um and we'll see here one two three
four graphs um with all the graph titles subtitles text and uh project label version number and now
we know that there are four graphs so i'm just gonna make sure that this number here is four if there
were 50 i'd do 50 since there are four i'm just going to type in four um and there are no y-axis
labels so we're just going to skip over that and then press enter and as we press enter we can see
that it'll man automatically change everything so we went from you know all the placeholder text to
all of this text and let's just double check that everything is right
i'm going to open up harbour to look up the game test report and i'm going to download it as the csv
we'll see here that the labels for the gpus are a little long um we don't really need the NVIDIA
gforce part so i'm just going to cut that out it's going to be rtx 4080
i'm just going to select the graph that i'm working on at the time let's make this full
screen select the graph and i'm just going to delete everything that i have and make sure everything
clears yep it's saying that i have no items at all to graph and i'm just going to paste in what i have
and so here we see um because with our game graphs we like to sort uh highest to lowest because higher
is better um and it does that automatically um the legend will stay the same uh one percent
low on average result so this is a benchmark graph it has three series i had a um three series template
ready
okay i'm done now i've put in the data for all of the graphs and then we're just going to make
sure that all the data is correct so what i'm going to do is um look over um the data based off of
the spreadsheet that i pulled a cyberpunk for example for the 4080 is um 154 193
okay so now all the data is included um we're just going to change up the uh x axis a little bit
so you can see here that there are one two three four five six seven
different lines and that's a little too much for my taste i feel like that's a little crowded
and by default our graphs have an interval of 30 we prefer to have it at 60 but sometimes when the
graph results are really low like having 60 doesn't really make sense so i'm just going to change it
here to 60 so that it's a little less crowded and we can see 60 120 and 180 um with this one i'm going
to do about the same because it's kind of within the same range
i'm going to pull up go back to illustrator instead of exporting everything and having
you know people look over it and then give me feedback individually what i'm going to do is
i'm going to um share it for review it what it's going to do is create essentially like a google
doc where people can add comments so i'm going to um set it to anyone with the link can comment and
copy it and i'm going to uh send it off to adam and john so i'm just going to show everybody
what it looks like and this is essentially what it looks like for john and adam um they're just
going to look over it for multiple things they're going to make sure that all the labels are correct
um and they're going to make sure that the uh data is correct it matches up with the right
card and it matches up with the right um game or test uh here if they do you find something for
example um they can leave a comment so they can highlight this and uh they can just write a comment
and say uh for this one this data is actually for dota 2 not cyberpunk theoretically if i made
that mistake okay so now i've sent off my um graphs to adam and john and they've reviewed it and adam
doesn't have any comments and john has a question about um the specifics of like when we're going to
use commas for the graphing numbers and so um you know i can reply to him and i'll say we're adding
in commas whenever uh numbers are above a thousand if we had any changes to make
then i would go in i would make the changes and then i'd just update the content um since we
don't have any changes to make what i'm going to do is prepare for export
and adam will be the one to send it off to the editing team and we're going to put it in the video
yeah that's pretty much it on my end okay so at this point i received the graphs from sammy
so what i do now along with adam as we go through when we kind of take a look at the graphs he does
sort of a subjective look on the presentation of the graphs i do more of a data analysis make sure
everything was inputted correctly maybe something got copied into the wrong line or something like that
so basically i just have to pour over the data here make sure everything looks right
so we just quickly scan through blender here 50 48 20 10
this blender looks good and then if we hop back over to the the game test report
we can quickly look at all of them up here so if we go to dota 2
so this point now i've basically verified that all the data looks correct um assuming that adam
doesn't have any other issues then we go ahead and get these all set up sent off to the editors
and that's when they start cutting the video after all of that we then go into post production
where we provide the assets and the footage to the editor we have several more steps of qc that
occurred during this um first off you have the editor and writer review where the writer sits
down with the editor watches what the editor has created and um points out any sort of fixes
whether they're stylistic or content fixes that need to be done then once those fixes are implemented
they send a pass to Labs Labs looks over it Labs points out any fixes typically are their fixes
are going to be more focused on the correctness and the data side of things and also maybe kind of
small nitpicks less so about the stylistic side of things but we still make sure that we provide
Labs input into that because sometimes you can lose the exact meaning of stuff just through like a
couple of different wardings and we want to make sure that that's always communicated correctly
then once Labs reviews it the editor implements all those fixes there is another review with the
writer and Labs to make sure that all the fixes have been implemented correctly once that is done
we then upload it to Floatplane for our ecc squad to take a look at the whole point of ecc squad
is to get like a ton of knowledgeable people to have eyes and ears on exactly what we've made
because they'll notice the smallest kind of details when you have like a dozen sets of eyes you'll
just find so many small things and they're always great and once we've done that we then
review the suggestions that the ecc squad has made and if they need to be implemented we do that again
we also check with Labs to make sure that those corrections are in fact correct implement them
make sure they're implemented correctly and then finally after all of that a video gets released
update your BIOS kids