1
00:00:00,000 --> 00:00:04,560
Welcome to the Labs Process Video. We're going to give you a step-by-step breakdown of everything

2
00:00:04,560 --> 00:00:10,000
we do when we test a product for a video, all the way from ideation to the final results.

3
00:00:10,000 --> 00:00:15,680
Every test starts with an idea. For example, it's been a long time since we've reviewed the 7900XTX

4
00:00:15,680 --> 00:00:21,440
and the 4080, and we want to know how do these perform today? The first step is filling out our

5
00:00:21,440 --> 00:00:26,160
test request form. As much as it would be nice to just go down and ask Labs to test these cards,

6
00:00:26,160 --> 00:00:30,880
they're not going to do anything unless I provide them with a filled out test request form. There

7
00:00:30,880 --> 00:00:35,680
is no testing without requesting. The reason we have to do a request form is that it allows Labs

8
00:00:35,680 --> 00:00:40,960
to track the project. If we don't give them a request form, they won't do the testing because

9
00:00:40,960 --> 00:00:44,880
that's how we waste people's time. In here we put like the name of the project, so we'll say like

10
00:00:44,880 --> 00:00:51,120
4080 versus 7900XTX. Then we'll put a link to the Trello just again for project tracking. And then

11
00:00:51,120 --> 00:00:56,160
here's the first kind of important answer. This is an important delineation. Is it exploratory

12
00:00:56,160 --> 00:01:01,680
testing or project testing? So exploratory testing is going like, oh, we don't know if this is going

13
00:01:01,680 --> 00:01:05,840
to be an interesting story. So we need to test to see if the effect that we are talking about

14
00:01:05,840 --> 00:01:10,400
is significant. Like how much does a browser extension slow down your browser? Maybe it's

15
00:01:10,400 --> 00:01:15,360
not enough to be an interesting video, but maybe there's something there versus a project test,

16
00:01:15,360 --> 00:01:20,960
which is like, it's a new GPE review. We know that we're going to test it. We don't need to explore.

17
00:01:21,600 --> 00:01:26,560
In this scenario, we're doing a project test. Then we give them a due date, so a project timeline.

18
00:01:26,560 --> 00:01:30,400
We can either do a hard deadline or a soft deadline. A soft deadline is kind of like,

19
00:01:30,400 --> 00:01:34,720
if something else of higher priority comes in, Labs can push it back a little bit because maybe

20
00:01:34,720 --> 00:01:38,960
it's a video that doesn't really need to come out on a specific date. A hard deadline is really

21
00:01:38,960 --> 00:01:43,360
important. If it's something that's very timely, like maybe it's a sponsored project or like an

22
00:01:43,360 --> 00:01:48,240
embargo GPE review, then we give them a list of all of the components that we want tested in the

23
00:01:48,240 --> 00:01:53,200
video. For this scenario, we're just going to do the two cards, a 4080 Accelerate and a 7900 XTX.

24
00:01:55,840 --> 00:02:00,560
Then we just let them know if we have them here or if they're coming. Basically it allows them to

25
00:02:00,560 --> 00:02:04,960
be like, where do we need to listen for the next step? Is it like the writer or do we need to like

26
00:02:04,960 --> 00:02:09,440
keep an eye on procurement or something like that? Sometimes we also might want to like

27
00:02:09,440 --> 00:02:14,320
talk to Labs about any sort of materials or components that we need to actually get.

28
00:02:14,320 --> 00:02:18,880
And that'll usually happen during like a kickoff meeting. Then we have to clearly state what our

29
00:02:18,880 --> 00:02:23,120
questions are. It's really important to do this and like the more nailed down our questions are,

30
00:02:24,240 --> 00:02:28,000
the faster the project's going to go and the faster the kickoff meeting will go.

31
00:02:28,000 --> 00:02:31,120
There have been times where I've stepped into meetings where I did not have a solid question

32
00:02:31,120 --> 00:02:35,280
and then it takes a long time, waste everyone's time trying to figure out what the heck I'm doing.

33
00:02:35,280 --> 00:02:43,280
In this scenario, we're just trying to see how well does AMD's 7900 XTX compared to the 4080

34
00:02:45,280 --> 00:02:49,840
is the 4080 really we're going to be worth the extra money. And then we can also select specific

35
00:02:49,840 --> 00:02:54,960
tests. We have a whole test portfolio that we can usually run like where we're just run through

36
00:02:54,960 --> 00:02:59,040
a gauntlet of games and productivity benchmarks, but maybe I just need to select few or there's

37
00:02:59,040 --> 00:03:03,920
some that I absolutely must have. In this case, we don't need to run these through like a million

38
00:03:03,920 --> 00:03:11,680
gaming benchmarks, just a few. We're going to do Cyberpunk 2077, Dota 2, Red Dead Redemption 2,

39
00:03:11,680 --> 00:03:15,840
and they're all going to be at 1080. The next part is like any sort of notable

40
00:03:15,840 --> 00:03:19,840
test boundary conditions. So say we wanted to have something that's like out of the

41
00:03:19,840 --> 00:03:24,880
ordinary, like, you know, maybe we need old drivers, like we want the drivers that were

42
00:03:24,880 --> 00:03:29,200
used when it came out to compare to the drivers that are out today. That's something we can put

43
00:03:29,200 --> 00:03:34,080
in here. And then for any super general notes that might not really have another place to go,

44
00:03:34,080 --> 00:03:37,200
there's a final form just at the bottom where we can fill out and be like,

45
00:03:38,160 --> 00:03:42,480
I don't know, maybe like, you guys want to hang out sometime too? I don't know,

46
00:03:42,480 --> 00:03:46,640
you can ask them whatever they need to submit it and it goes off to them.

47
00:03:46,640 --> 00:03:54,640
Hi, it's Gary. We just received a test request from Adam. Normally this goes to Sharon, but Sharon

48
00:03:54,640 --> 00:04:02,560
is busy doing some other work today, so I'm handling this request. Once it comes into our mailbox,

49
00:04:03,360 --> 00:04:12,640
we simply click on it, brings up the lab's data request. This one's obviously from Adam.

50
00:04:13,760 --> 00:04:17,760
We're testing the 4080 against the 7900XTX.

51
00:04:24,880 --> 00:04:29,200
And then he's actually filled out some specific tests that he wants.

52
00:04:29,200 --> 00:04:36,480
All of these, Cyberpunk, Red Dead Redemption, Blender and Dota are part of our standard test

53
00:04:36,480 --> 00:04:42,720
suite, so that makes it a little bit easier. He doesn't want anything special on it, just straight

54
00:04:42,720 --> 00:04:51,040
up testing. And apparently he appreciates all of us, so thank you, Adam. With that, we'll go

55
00:04:51,040 --> 00:04:57,920
over to the assignment form. This is where we keep track of who's assigned the project,

56
00:04:59,040 --> 00:05:04,320
and also if it's just been assigned or if it's already in progress. For this one,

57
00:05:05,680 --> 00:05:12,000
we'll go ahead and mark it in progress as John's going to start it momentarily.

58
00:05:13,440 --> 00:05:20,640
And yeah, we're giving it to John. And with that, I'm going to hand it over to John. Thank you.

59
00:05:21,440 --> 00:05:25,520
Now we have our kickoff meeting. Basically what this does is that it allows us to

60
00:05:25,520 --> 00:05:30,160
talk with Labs, make sure that our test plan is all agreed upon, and that we are all on the same

61
00:05:30,160 --> 00:05:34,880
page, so that we can make sure that we're doing everything right. Labs will often bring some

62
00:05:34,880 --> 00:05:38,080
concerns and questions. Writers might be like, notice things in the middle of the meeting,

63
00:05:38,080 --> 00:05:43,280
be like, oh crap, I forgot to add that to the sheet. So this is probably one of the most

64
00:05:43,280 --> 00:05:46,560
important parts of the whole process, because it allows for all of us to make sure that we are

65
00:05:46,560 --> 00:05:52,000
aligned on our goals. So we just had the kickoff meeting for this project. Now that we've done

66
00:05:52,000 --> 00:05:59,360
that, we can actually go about collecting the requirements, the project requirements for the

67
00:05:59,360 --> 00:06:04,720
system. So we're going to be using our AMD benches today. And that's something that we're going to

68
00:06:04,720 --> 00:06:10,480
be using going forward for all of our GPU testing. So what we're going to do first is give you a

69
00:06:10,480 --> 00:06:16,480
rundown of all the hardware we use on our AMD benches, and then get into the nitty gritty of

70
00:06:17,040 --> 00:06:22,160
actually building it and setting it up correctly. So first thing we do is we got a

71
00:06:22,160 --> 00:06:28,560
CSONIC Prime PX1600 power supply. We've been using these for a little bit now. They're ATX 3.0

72
00:06:28,560 --> 00:06:36,480
capable. So the nice thing is we can swap quickly between using a 12-volt high power and typical

73
00:06:36,480 --> 00:06:41,600
8-pin power connectors for any of the GPUs that are made by like Intel or AMD without having to

74
00:06:41,600 --> 00:06:47,120
use any extra like pigtails and stuff like that. So that's pretty useful. And for motherboard, we

75
00:06:47,120 --> 00:06:55,040
are using X670E AORUS extremes. We've been using these for a little while on our AMD benches,

76
00:06:55,040 --> 00:07:00,720
not that we've used them in many of the videos, but going forward that is going to be our standard

77
00:07:01,280 --> 00:07:06,560
for our GPU testing. The reason we went with the extremes, we were using the masters for a while,

78
00:07:06,560 --> 00:07:11,200
and then we ended up going back to the extremes because we found that we liked the VRM a little

79
00:07:11,200 --> 00:07:17,440
bit further. It was a little bit more robust, and the BIOS settings we found that it could tweak just

80
00:07:17,440 --> 00:07:23,600
that little bit better than the master for doing some exploration testing of overclocking and stuff

81
00:07:23,600 --> 00:07:30,160
like that with some of the CPUs we've gotten in. As far as other brands, we have looked at ASUS for

82
00:07:30,160 --> 00:07:36,880
a little while. We looked at a couple of ASRock boards, but ultimately we kind of ended up on the

83
00:07:36,880 --> 00:07:44,480
AORUS extreme. So for CPUs, we're going to be using Ryzen 7 7800X3Ds. We actually have a video

84
00:07:44,480 --> 00:07:51,920
upcoming that is about AMD variance testing that we did to basically determine the three 7800X3Ds

85
00:07:51,920 --> 00:07:58,000
that we're actually going to go with going forward. They are the fastest one or the slowest one,

86
00:07:58,000 --> 00:08:04,640
the three tightest ones that we could find to effectively try to reduce any sort of variance

87
00:08:04,640 --> 00:08:11,360
in our FPS basically between the three benches. So we can kind of more easily verify that if we

88
00:08:11,360 --> 00:08:15,760
were to take this card and put them on any of the three benches, we should get the same number.

89
00:08:15,760 --> 00:08:23,200
For RAM, we are using G-Skill Trident Z5 new RGB. The stuff we're going with is got

90
00:08:23,440 --> 00:08:31,840
6,000 megatransfers per second CL30. Even though AMD has come out with new versions of

91
00:08:31,840 --> 00:08:38,640
AGSA that have basically unlocked the ability to use anything higher than 6,000, we still found

92
00:08:38,640 --> 00:08:44,080
that this is kind of the best sweet spot for AMD in terms of performance. You can get up there

93
00:08:44,080 --> 00:08:47,920
where you start getting your timings a little bit tighter and stuff like that and perform

94
00:08:47,920 --> 00:08:53,680
basically the same on like 6,800 megatransfers and stuff like that. But

95
00:08:53,680 --> 00:08:57,440
this seems to be a good price to performance ratio still, so that's kind of what we're going

96
00:08:57,440 --> 00:09:05,520
with going forward. For SSDs, we are using Samsung 980 Pro 2TB SSDs. We haven't explored

97
00:09:05,520 --> 00:09:12,480
anything as far as using PCI Express Gen 5 SSDs yet, but this seems to be a really good

98
00:09:12,560 --> 00:09:18,800
performance ratio on both the reads and writes of both not just the sequential but as well as

99
00:09:18,800 --> 00:09:24,320
the random reads and writes. So it kind of allows us to get the load times a little bit faster and

100
00:09:24,320 --> 00:09:29,280
kind of have more benchmarks running in sequential order without having to wait for all those

101
00:09:30,080 --> 00:09:36,560
extra load times. Sure, most SSDs on the market nowadays are fairly quick as long as we have

102
00:09:36,560 --> 00:09:42,320
something as DRAM in it, but that's the one we've kind of been going with for a while and

103
00:09:42,320 --> 00:09:46,800
seems to be good enough for us going forward. And if it ain't broke, don't fix it, right?

104
00:09:47,520 --> 00:09:53,280
So there's that. And then lastly, to cool the 7800X3Ds, we are using Noctua

105
00:09:53,280 --> 00:09:58,240
NHD 15 coolers. It's kind of been the standard for a long time for a lot of people.

106
00:09:59,600 --> 00:10:03,920
As soon as the new gen version comes out, I'm sure we're going to be exploring that as well

107
00:10:03,920 --> 00:10:10,400
and seeing just how much more beneficial it could be for us. But it's been basically the

108
00:10:10,400 --> 00:10:15,680
workhorse for us for a while and it probably won't change for a little while afterwards. So

109
00:10:15,680 --> 00:10:21,280
that's kind of where we're at this point. So for the specific project, we obviously have to get

110
00:10:21,280 --> 00:10:29,760
the specific cards that we need to test with. So we have a reference, AMD Radeon 7900 XTX.

111
00:10:29,760 --> 00:10:33,520
It's been the one that we've used in all of our videos that we've ever covered on it.

112
00:10:34,160 --> 00:10:40,320
It's got a pretty sick looking cooler. And then the video card we're going to be using today is a

113
00:10:40,320 --> 00:10:48,000
PNY Accelerate RTX 4080. This seems to be currently the one that is the cheapest on the market,

114
00:10:48,000 --> 00:10:52,560
but that doesn't necessarily mean it's the worst. And this is the one that the requirements

115
00:10:52,560 --> 00:10:58,480
of the test plan specifically outline. So that's the one we're going to be testing today. And lastly,

116
00:10:59,360 --> 00:11:05,760
we obviously need some sort of thermal paste to go between the D15 and the 7800 X3D. And we use

117
00:11:05,760 --> 00:11:13,600
Noctua NTH2 thermal paste. We've been using it for a while. As far as application, more is always

118
00:11:13,600 --> 00:11:20,640
better than less to some degree. And so when I usually put thermal paste on, it's in an equal

119
00:11:20,640 --> 00:11:27,920
sign, which is something that you'll probably see in the build part of the video. So I'm sure

120
00:11:27,920 --> 00:11:32,000
everybody's going to have tons of arguments of, oh, you've used too much or used too little.

121
00:11:32,000 --> 00:11:36,640
There's been so many discussions about that. It doesn't seem to affect the thermal performance

122
00:11:37,280 --> 00:11:42,560
in any sort of way. So as long as we have enough, we're good to go. All right, now we've gone over

123
00:11:42,560 --> 00:12:04,480
the actual hardware, let's get to actually building. Now we've got the system all built together and

124
00:12:04,480 --> 00:12:09,200
up and posted into the BIOS, we're going to be going through making sure we have the correct BIOS

125
00:12:09,280 --> 00:12:17,040
version as well as configuring it to the standardized configuration that we do for

126
00:12:17,040 --> 00:12:23,840
all of our benches. So when we look at our test conditions, we usually outline both drivers

127
00:12:23,840 --> 00:12:29,200
as well as BIOS revision, because there might be an instance where like we're doing some sort of

128
00:12:29,200 --> 00:12:34,960
exploratory testing and being like, hey, we actually don't want the latest version of the BIOS. We want

129
00:12:34,960 --> 00:12:40,000
to go back four or five revisions. That's why we outline it in the actual test conditions. In this

130
00:12:40,000 --> 00:12:46,880
case, it's saying F11D, which is the current version of the BIOS, but obviously we're not

131
00:12:46,880 --> 00:13:00,240
running F11D, so we're going to have to update it. Update your BIOS kids. Yeah, there's, especially

132
00:13:00,240 --> 00:13:12,320
with like new AMD CPUs, 7800X3D in particular, a number of motherboards don't actually support it

133
00:13:12,320 --> 00:13:17,360
correctly out of the box. So if you don't have the option of actually booting into the BIOS,

134
00:13:17,360 --> 00:13:23,440
luckily most manufacturers include some sort of BIOS flashback now. Just slot the USB into the

135
00:13:23,440 --> 00:13:29,280
correct port on the motherboard, make sure obviously the file is named correctly before you do that.

136
00:13:29,360 --> 00:13:34,000
It's incredibly important if you want a computer to work right. Then once we finally get the BIOS

137
00:13:34,000 --> 00:13:39,840
updated, we're going to be going ahead and configuring it and then setting up Windows.

138
00:13:43,920 --> 00:13:49,200
Okay, now we've finished the actual BIOS update. Let's go ahead and start configuring things. First

139
00:13:49,200 --> 00:13:54,960
thing we're going to go to advanced options, so we can actually see everything. We always turn on

140
00:13:54,960 --> 00:14:04,640
expo or if it's Intel, it's XMP. Verify that rebar is on. Most AMD boards usually have it on by default

141
00:14:04,640 --> 00:14:13,280
now, but it's always good to go to. It's usually under IO ports or some sort of IO naming scheme,

142
00:14:13,280 --> 00:14:20,400
whatever manufacturer makes it. Inside here, we also turn off the IGP or integrated graphics,

143
00:14:21,120 --> 00:14:30,960
especially if we are testing any sort of dedicated GPU. All it does is sometimes confuse some games

144
00:14:30,960 --> 00:14:36,800
or programs into wondering which actual graphics card to be using at the time.

145
00:14:39,120 --> 00:14:45,040
And then from there, the only other thing that we do is go into the fan curves

146
00:14:45,280 --> 00:14:51,280
and we manually set it so that it is a linear scale.

147
00:14:54,720 --> 00:15:00,560
So it basically looks like that. Then at that point, we can actually start installing Windows.

148
00:15:04,560 --> 00:15:12,800
For Windows, we use a optimized Windows image that our software developers have

149
00:15:13,520 --> 00:15:19,520
created. Basically strip back a bunch of stuff that we don't need to be running in the background,

150
00:15:19,520 --> 00:15:26,000
as well as adding in some programs that we actively use during our testing. We actually

151
00:15:26,560 --> 00:15:31,680
reinstall them every three months. We'll usually update them basically and create a new Windows

152
00:15:31,680 --> 00:15:37,920
image that has our latest updates so that we don't have to constantly do this rigmarole of

153
00:15:38,640 --> 00:15:45,760
get a million updates, a million drivers. We can just basically install the latest drivers

154
00:15:45,760 --> 00:15:50,000
and then we have basically all of our programs. So now we just got to let this thing run.

155
00:15:51,440 --> 00:15:56,960
Very exciting stuff, like watching paint dry. In our test conditions, we'll usually outline the

156
00:15:56,960 --> 00:16:05,040
exact version of Windows we want to run. Going back to that, we might need to recreate an old

157
00:16:05,040 --> 00:16:13,200
set of parameters. So we might not be using Windows 11 22H2. We might use 22H1 or that's why

158
00:16:13,200 --> 00:16:19,040
on that list, we actually have all the other versions of our images still on there so that we

159
00:16:19,040 --> 00:16:24,240
can actually go back and be like, oh, actually we need to use the version from May. And then we

160
00:16:24,240 --> 00:16:28,880
basically, if we need to recreate that set of test conditions, we have to go and also stop

161
00:16:28,880 --> 00:16:34,080
Windows updates from updating. Now we're in Windows. We are not going to install the Gigabyte

162
00:16:34,080 --> 00:16:41,200
Control Center. So basically at this point, we have to go through update all of our drivers,

163
00:16:41,200 --> 00:16:48,560
as well as install all of our programs that we're actually using for the project.

164
00:16:54,080 --> 00:17:01,760
Okay, so now we've installed the chipset drivers. We can make sure that we get the latest NVIDIA

165
00:17:01,760 --> 00:17:09,280
driver as, or not necessarily the latest, because if the project does require an older version of

166
00:17:09,280 --> 00:17:14,880
the driver, we got to get that version instead. But in this case, in this instance, we are using

167
00:17:14,880 --> 00:17:20,960
the latest version. So we're using 537.42. Let's get that downloaded.

168
00:17:27,520 --> 00:17:30,560
Then we're just going to make sure all the video settings are set correctly,

169
00:17:30,560 --> 00:17:38,880
so we essentially turn off G-Sync, any sort of potential accelerators, V-Sync,

170
00:17:40,160 --> 00:17:45,680
so we can get the, basically the raw performance of the card. Update your cards.

171
00:17:47,040 --> 00:17:54,560
There's a lot of things that NVIDIA doesn't post or advertise, like a number of three series cards

172
00:17:54,560 --> 00:17:59,920
don't have rebar enabled out of the box. So there's actually a bunch of articles that NVIDIA

173
00:18:00,080 --> 00:18:09,680
posted that are really useful, such as UEFI fixes, DisplayPort fixes, and things like rebar

174
00:18:09,680 --> 00:18:14,960
patches and stuff like that. So if you find your card isn't matching up for whatever reason,

175
00:18:14,960 --> 00:18:20,720
or is doing something funky, good to check those out, because sometimes it might be as simple as,

176
00:18:20,720 --> 00:18:26,560
oh, I need to do this little update. Boom, I've got more performance, or oh, my video doesn't

177
00:18:26,560 --> 00:18:30,880
seem to work in the BIOS, but it works fine in Windows. Do a firmware update. Oh, sweet, now

178
00:18:30,880 --> 00:18:34,880
it's fine. Alrighty, now that we've got that installed, let's just make sure we don't have

179
00:18:34,880 --> 00:18:43,360
any other weird drivers. Looks good to me. Usually if there's no yellows, we're happy campers.

180
00:18:44,320 --> 00:18:50,560
We usually do all the Windows updates according to what our project plan is. In most cases,

181
00:18:50,560 --> 00:18:54,960
it's usually just as simple as getting every single update that's in there. A lot of times,

182
00:18:55,040 --> 00:18:58,640
what will happen is, you know, there will be driver updates and stuff like that, but if you

183
00:18:58,640 --> 00:19:04,960
already have a later driver than what Windows sees, often you'll get like a download error a lot of

184
00:19:04,960 --> 00:19:09,200
the times, because Windows is flagged, oh, you actually have a newer version, we don't need to

185
00:19:09,200 --> 00:19:16,560
get this. And what we do is we tend to update our Windows image every three months to help include

186
00:19:16,560 --> 00:19:20,640
a lot of these Windows updates. We essentially have to download a lot less of them. It is time

187
00:19:20,640 --> 00:19:25,680
consuming and the less time we have to spend on constantly downloading things and updating things,

188
00:19:25,680 --> 00:19:31,200
the better, right? Then we can get right into the actual testing. It saves us a lot of overhead,

189
00:19:31,200 --> 00:19:38,640
basically. So one last thing we're going to do before we get all of our games downloaded on the

190
00:19:38,640 --> 00:19:43,520
system is we're going to go through and just do some last minute Windows optimizations.

191
00:19:44,240 --> 00:19:50,720
So first thing we're going to do is we're going to do some trimming on the actual drive itself.

192
00:19:51,920 --> 00:19:57,920
Just helps speed it up just that little bit better. Then we're going to go and go to percent

193
00:19:57,920 --> 00:20:03,920
temp percent to basically get to our temporary folder. And we're going to basically delete

194
00:20:03,920 --> 00:20:08,560
everything we can in here. So one last thing we're going to do is we're going to set the recycling

195
00:20:08,560 --> 00:20:15,600
bin to 15 megs. The reason we do this is just that if we delete any files on the system, it'll

196
00:20:15,600 --> 00:20:20,480
just delete them right away. It won't sit in the recycling bin taking up extra space we don't need.

197
00:20:21,360 --> 00:20:25,680
Basically, it skips a step of us happening at empty out the recycling bin in most cases. So

198
00:20:26,400 --> 00:20:32,160
kind of useful in that regard. So once we have all this set up, the only last thing we have to do

199
00:20:32,160 --> 00:20:38,480
is just make sure whether it's NVIDIA, Intel, or AMD, we have any sort of variable refresh rate

200
00:20:38,480 --> 00:20:45,760
stuff turned off, as well as B-sync. The reason behind that is we don't want to potentially give

201
00:20:45,760 --> 00:20:52,640
any sort of advantage to one manufacturer over another with any sort of technologies they

202
00:20:52,640 --> 00:20:57,520
might have running in the background causing any optimizations that are effectively outside of

203
00:20:57,520 --> 00:21:04,080
Windows itself doing hardware optimizations. That way it kind of keeps the playing field a little

204
00:21:04,080 --> 00:21:10,960
bit more level. So we're going to pop into here. I doubt we'll have G-sync on here. No, because we

205
00:21:10,960 --> 00:21:16,960
are currently plugged into our NDI viewer to get all of our capture. So for this specific instance,

206
00:21:16,960 --> 00:21:24,240
we won't have G-sync enabled. But typically, because these monitors are actually G-sync capable,

207
00:21:25,200 --> 00:21:30,880
anytime we pull up an NVIDIA card inside there, it'll immediately enable G-sync. So we have to

208
00:21:30,880 --> 00:21:38,080
make sure we go in and actually disable that. And then what we do also is in here, we've got to

209
00:21:38,080 --> 00:21:47,200
manage three settings, scroll down to vertical sync, and we make sure we force it off. That way,

210
00:21:47,760 --> 00:21:52,240
because there's some games that they don't either give you the option or they tell you it's off,

211
00:21:52,240 --> 00:21:57,520
but it's not actually off. It's kind of just ensures that no matter the game, no matter the

212
00:21:57,520 --> 00:22:04,160
settings, if somebody forgot to turn off V-sync for whatever reason, this basically forces it off.

213
00:22:04,160 --> 00:22:09,920
So there's no opportunity for the game to even interact with, hey, by the way, I need V-sync.

214
00:22:10,720 --> 00:22:17,680
So it kind of takes that guesswork out of it. And then we go into display settings.

215
00:22:18,080 --> 00:22:24,000
We're going to be running at 1080p, so we just leave the resolution as it is, but we change our scale

216
00:22:24,000 --> 00:22:35,680
to 100% rather than 150. Some of our markbench benchmarks do still use CV in order to do character

217
00:22:35,680 --> 00:22:41,360
recognition. So if the scaling is at 150% and you're on the desktop, for example,

218
00:22:42,080 --> 00:22:47,440
the new version of Hitman has a launcher that it shows up beforehand that has all the options

219
00:22:47,520 --> 00:22:53,280
and stuff in. How markbench works in that one is it actually will look for the specific word.

220
00:22:53,280 --> 00:22:58,400
If our scaling is different than what it's supposed to be, markbench will be like,

221
00:22:58,400 --> 00:23:03,200
I can't find this because it's effectively looking for a picture at a different scale

222
00:23:03,200 --> 00:23:09,360
than what it actually is. So if we set it to 100%, basically we're going to have no problems.

223
00:23:09,360 --> 00:23:13,920
And then normally what we do is we check to make sure we're at the highest hertz we can be for the

224
00:23:13,920 --> 00:23:20,080
monitor. Because we're going through the NDI, we actually have to run at 60 hertz because that's

225
00:23:20,080 --> 00:23:26,800
the maximum the actual NDI itself supports. But typically it's 144 for the actual monitors themselves.

226
00:23:26,800 --> 00:23:33,760
So lastly, we go into change default graphics settings. We make sure that hardware accelerated

227
00:23:33,760 --> 00:23:41,040
GPU scheduling is on. This is at Windows level. This isn't hardware specific such as

228
00:23:42,000 --> 00:23:48,320
NVIDIA Reflex or AMD's Antilag. That is something specific to the hardware,

229
00:23:49,280 --> 00:23:56,560
not Windows itself. So this one, it's the same across all the platforms of hardware. It doesn't

230
00:23:57,680 --> 00:24:04,160
give an advantage to NVIDIA over AMD because it's scheduling at the exact same. Anything else that's

231
00:24:04,160 --> 00:24:09,840
in the menu, usually we have like optimization for Windows games. We turn that off, as well as in

232
00:24:09,840 --> 00:24:17,360
some cases, there is a variable refresh rate option in here. It usually shows up for Intel

233
00:24:17,360 --> 00:24:24,960
and AMD because theirs interact a little bit differently than G-Sync does. And so we make

234
00:24:24,960 --> 00:24:29,760
sure that's off as well. Because again, we don't want to give advantage to one specific card. We

235
00:24:29,760 --> 00:24:35,680
want to see raw performance of each of the cards stacked against each other so we can understand

236
00:24:36,320 --> 00:24:41,840
what is actually the best option. Now that we've gotten all the Windows updates installed,

237
00:24:42,880 --> 00:24:48,880
next thing we got to do is we have to install all the programs as well as any games that we're

238
00:24:48,880 --> 00:24:54,560
going to be using to do this project. So in this case, we need to install Cyberpunk, Dota 2,

239
00:24:55,440 --> 00:25:02,960
and Red Dead. So we already have Steam pre-installed on our Windows image so we don't have to worry

240
00:25:02,960 --> 00:25:06,960
about actually going to the website and getting that part, but unfortunately it always needs to

241
00:25:06,960 --> 00:25:14,400
update anyways. So it's kind of a moot point. While that's happening, what we're also going to do is get

242
00:25:15,520 --> 00:25:22,240
markbench from our local drive, as well as some patches that I need to install as well.

243
00:25:22,320 --> 00:25:32,640
So we are going to install, nope, Cyberpunk. We're going to install Red Dead.

244
00:25:34,880 --> 00:25:39,360
And then the most annoying thing is that Dota 2 doesn't stay in your library half the time,

245
00:25:39,360 --> 00:25:45,040
so we are going to go here and do play now.

246
00:25:45,440 --> 00:25:50,800
Okay, so those three guys are downloading.

247
00:25:52,880 --> 00:25:58,560
We've got markbench extracted here. We need to add another harness in for the Dota 1,

248
00:25:58,560 --> 00:26:07,040
so let's get that real quick. And there it is, ready to go for when we are ready to use it.

249
00:26:08,800 --> 00:26:14,080
And then the other thing we need is Blender Benchmark, so let's quickly grab that as well.

250
00:26:15,440 --> 00:26:22,320
So although markbench does have a Blender Benchmark actually integrated into it,

251
00:26:23,120 --> 00:26:28,880
we do need to still download the application itself to download the files required for it to run.

252
00:26:30,000 --> 00:26:34,880
There's a little bit of bugginess with the CLI. When it tries to download the files,

253
00:26:34,880 --> 00:26:41,520
it effectively says I can't do this and quits out. So if we download the actual program and have it

254
00:26:41,520 --> 00:26:47,920
download the files, it just makes the automation a little bit simpler on our side because it has

255
00:26:47,920 --> 00:26:53,680
the files already completely installed. So effectively we want version 3.6. We're going to

256
00:26:53,680 --> 00:27:00,080
download the benchmarks. Again, let's just make sure that markbench is all prepared, just like when

257
00:27:00,080 --> 00:27:05,120
we have to go through any of our game settings. We have to actually go download the programs,

258
00:27:05,120 --> 00:27:12,400
download the games, actually set up the settings ourselves. That stuff is not currently automated.

259
00:27:13,120 --> 00:27:16,720
Now we have all the games downloading and everything's kind of getting set up.

260
00:27:16,720 --> 00:27:20,000
I'm going to throw it over to Nick and he can talk a little bit about markbench.

261
00:27:20,000 --> 00:27:26,400
Fundamentally, markbench is our test automation and orchestration framework. It's not just simply

262
00:27:26,400 --> 00:27:32,560
a simple desktop GUI anymore. It's a constellation of services and components that work together

263
00:27:32,560 --> 00:27:37,680
to form the ecosystem that we conduct our testing in. But it all does start with the test

264
00:27:37,680 --> 00:27:43,440
harnesses, which most of ours are in Python because it's most easily automatable for us.

265
00:27:44,080 --> 00:27:50,640
However, it doesn't need to be Python. They can be PowerShell scripts, Ruby, GoLang, what have you.

266
00:27:50,640 --> 00:27:57,840
The important part is that each test harness gives back a exit code of zero or one, zero to indicate

267
00:27:57,840 --> 00:28:03,520
a successful test run, anything greater such as one to indicate that it failed. Because that exit

268
00:28:03,520 --> 00:28:10,080
code is then read by the next layer of markbench, which is what we call auto bench. It just comes

269
00:28:10,080 --> 00:28:15,120
before the GoLang GUI and it really contains all of the major features of our orchestration.

270
00:28:16,160 --> 00:28:21,280
This layer is what separates what we're doing from just a collection of PowerShell scripts

271
00:28:21,840 --> 00:28:27,600
to something that is a lot easier to manage long term because inside of auto bench, that's

272
00:28:27,600 --> 00:28:32,000
where we get our orchestration. So with the test harness, I have the capability of running

273
00:28:32,000 --> 00:28:38,000
individual tests, but I want to be able to group those into a bunch of test suites such as let's

274
00:28:38,000 --> 00:28:45,440
say I want to run Dota 2 four times at one resolution and then Red Dead Redemption 2 at a

275
00:28:45,440 --> 00:28:50,080
different resolution. So that's where the auto bench side comes in. Auto bench is where we run

276
00:28:50,080 --> 00:28:55,520
presentmon in system logger in conjunction with the test harness so that we get a similar timestamp

277
00:28:55,520 --> 00:29:00,960
between all three because this eliminates the human error of starting a test and then starting

278
00:29:00,960 --> 00:29:06,960
hardware info or like logging on the side, which then would separate our timestamps by

279
00:29:07,520 --> 00:29:12,640
sometimes multiple seconds, which makes it easier to trim the sessions later downstream,

280
00:29:12,640 --> 00:29:18,480
which John will explain later. And then so the auto bench is all command line interface,

281
00:29:18,480 --> 00:29:23,840
so to make it easier for us and John and writers or whoever wants to run Markbench,

282
00:29:23,840 --> 00:29:29,440
we added a Golang GUI. So we went with the Golang GUI because the framework that we used

283
00:29:30,320 --> 00:29:36,800
was very successful for us for other GUIs such as the keyboard tester, the robot, and the latency

284
00:29:36,800 --> 00:29:42,640
tests. So anytime you see Markbench on screen, that's the Golang part, but underneath all the

285
00:29:42,640 --> 00:29:47,840
work is really being done inside the test harness as well as the Python orchestration layer.

286
00:29:48,640 --> 00:29:54,320
So once the test is done running, let's say we just ran an individual test and once that test is

287
00:29:54,320 --> 00:30:01,600
done running, it sends its data up to a service called data ingest server, which is a Golang web

288
00:30:01,600 --> 00:30:07,920
API, which takes it in that data in a protobuf format and then writes it to our Postgres database.

289
00:30:08,880 --> 00:30:13,760
It's from that Postgres database, all of our data sits at rest and then is disseminated or sent

290
00:30:14,400 --> 00:30:20,240
to all the various tools that John can use to evaluate and report on that data. And we use a

291
00:30:20,240 --> 00:30:27,040
couple tools, notably Grafana and Redash for a lot of visualizations. And then we have internal

292
00:30:27,040 --> 00:30:33,520
tooling that we just kind of collectively call the Harbor Tools portal. And that contains anything

293
00:30:33,520 --> 00:30:39,440
that Grafana or Redash or another off the shelf tool doesn't provide us. Now if we go back up to

294
00:30:39,440 --> 00:30:47,680
the test harnesses, on their own, they're fairly dumb in that they can send keyboard and mouse input,

295
00:30:48,240 --> 00:30:54,480
but they don't always know what's going on the screen. So we used to use OpenCV locally to do

296
00:30:54,480 --> 00:31:00,080
template matching, but it's more of a brute force approach. And it's very flaky as it depends on

297
00:31:00,080 --> 00:31:05,920
certain resolutions. So if you're doing template matching with a resolution of 1920 by 1080,

298
00:31:06,560 --> 00:31:11,920
you need to select screenshots of the path or the flow you want to go through the menu of the

299
00:31:11,920 --> 00:31:16,560
particular game. But then if you want to do it at a lower resolution like 720p,

300
00:31:17,680 --> 00:31:22,800
nine times at a 10, you have to take a smaller screenshot. So it doesn't scale very well. And

301
00:31:22,800 --> 00:31:27,200
that's where we start getting into some of the services that we're adding. The first one being

302
00:31:27,200 --> 00:31:34,160
Keras optical character recognition or OCR. This allows a test harness to send a screenshot of the

303
00:31:34,240 --> 00:31:39,600
system it's running on with a target word and Keras will come back within a few milliseconds

304
00:31:39,600 --> 00:31:43,440
is that word on the screen or not. This reduces flakiness in the test harness,

305
00:31:44,000 --> 00:31:49,600
which makes navigating menus a hundred times easier. And in the future, we're going to be adding

306
00:31:49,600 --> 00:31:56,560
services that do a bunch of computer vision things such as depth estimation, object detection and

307
00:31:56,560 --> 00:32:01,600
whatnot, all in the effort of making the test harnesses more a lot smarter. But we'll talk

308
00:32:01,600 --> 00:32:08,160
about levels of automation in a bit. So the benefit that Mark Bench and its ecosystem gives us

309
00:32:08,160 --> 00:32:15,280
over just a collection of PowerShell scripts for one is maintainability at the scale that we're

310
00:32:15,280 --> 00:32:22,240
trying to achieve. PowerShell scripts would or some auto hockey scripts would become messy very

311
00:32:22,240 --> 00:32:27,600
quickly because eventually we'd have probably like 100 to 200. And as best as you could organize

312
00:32:27,600 --> 00:32:32,960
those in some directories, it's just not a it's a maintainable nightmare. Investing in such an

313
00:32:32,960 --> 00:32:40,640
ecosystem and the automation of Mark Bench, it allows us to build it to support multiple platforms.

314
00:32:40,640 --> 00:32:47,920
So not just Windows on a desktop, but handhelds and macOS and Linux as well. And it normalizes

315
00:32:47,920 --> 00:32:53,520
data for us so that when we capture metrics and we set it at rest, it's a lot easier to

316
00:32:53,520 --> 00:32:58,400
query that data. Back to you, John. Thanks Nick for that explanation of Mark Bench.

317
00:32:58,400 --> 00:33:02,640
Now what we're going to do since all these games are finished downloading is get them all set up

318
00:33:02,640 --> 00:33:06,880
and then we'll be able to get our benchmarks started. So we're going to start off with cyber

319
00:33:06,880 --> 00:33:14,560
bunk here. We have documentation that we are working on that we will be publishing in the near

320
00:33:14,560 --> 00:33:20,960
future of all of our settings and configuration for the various benchmarks that we include in our

321
00:33:20,960 --> 00:33:29,600
videos. I believe we've already released a couple of them on the forums. So I'm sure we'll continue

322
00:33:29,600 --> 00:33:35,040
to update that as time goes on. Okie dokie. Yeah, the new update for cyberpunk came out. So that's

323
00:33:35,040 --> 00:33:42,320
cool. So first thing we do, we go to settings. We're going to go to video mode and we're going to

324
00:33:42,320 --> 00:33:47,280
make sure we're at 1080p because that's what the project requires. You're going to be turning off

325
00:33:47,280 --> 00:33:52,400
any sort of things like low latency stuff like that. I think that it could benefit that specific

326
00:33:52,400 --> 00:33:58,400
card because we want the raw performance of the card. We don't want any sort of additional things

327
00:33:58,400 --> 00:34:05,840
like DLSS or low latency that can certainly help benefit the card and it might get better

328
00:34:05,840 --> 00:34:12,320
performance for sure. But what we're trying to determine is apples to apples between that 7900

329
00:34:12,400 --> 00:34:18,320
xdx and this 4080. So we're going to be turning off low latency. We're going to be making sure

330
00:34:18,320 --> 00:34:25,200
that we're on full screen. We don't want borderless. Then we're going to go to the graphics tab and for

331
00:34:25,200 --> 00:34:30,000
our specific test we aren't calling for ray tracing. So we're going to actually be turning off

332
00:34:30,000 --> 00:34:37,040
ray tracing. We're going to be going just strictly ultra and we're going to go ahead and turn down

333
00:34:37,120 --> 00:34:44,560
FSR to zero. Go back up a tiny bit. Go back down a tiny bit. Each individual time we change something

334
00:34:44,560 --> 00:34:51,360
we're going to hit apply. The settings for cyberpunk are still kind of a little bit weird

335
00:34:51,360 --> 00:34:56,320
where you might change some sliders and you might hit apply and sometimes it's forgetting

336
00:34:56,320 --> 00:35:01,600
that you hit apply or you change the quick preset. It doesn't actually use the preset because you

337
00:35:01,600 --> 00:35:07,920
haven't applied it. So when we go up and down on the FSR that kind of helps with that and then

338
00:35:07,920 --> 00:35:13,520
we're going to go ahead and turn FSR off because we don't actually want it. And that's basically how

339
00:35:13,520 --> 00:35:21,920
we set up our non ray tracing version of cyberpunk. So at that point we can basically exit out of that

340
00:35:22,800 --> 00:35:30,000
and move on to the next one which is Dota 2. Dota 2 is one of our more recent benchmarks that we've

341
00:35:30,000 --> 00:35:37,840
been working on. It uses a replay from I believe Dreamhack. A lot of visuals on the screen so that

342
00:35:37,840 --> 00:35:45,680
we can basically tax the card as much as possible. So in Dota we're going to go up to the cog, go to

343
00:35:45,680 --> 00:35:50,720
video and we're not going to use monitor's resolution. We're going to use advanced settings.

344
00:35:52,000 --> 00:35:57,760
In this case we need 1080p. We're going to be checking on to exclusive full screen

345
00:35:58,560 --> 00:36:01,520
rather than that. Turning off low latency yet again.

346
00:36:03,520 --> 00:36:08,240
And then for rendering side we just pulled the slider all the way to best looking.

347
00:36:10,080 --> 00:36:16,160
It's basically a preset that has all these specific settings. These things always off.

348
00:36:16,720 --> 00:36:20,800
That sort of thing. That's all we got to do for this guy here. So we go ahead and close that.

349
00:36:21,760 --> 00:36:27,520
Okay now we've done Dota 2. Lastly we just need to set up Red Dead Redemption. So we're going to

350
00:36:27,600 --> 00:36:35,920
hit Z to go into settings. Go to graphics. Make sure the resolution and the screen type are correct

351
00:36:35,920 --> 00:36:43,200
which they are. We're going to be turning off v-sync. Now pull the slider all the way to the right that

352
00:36:43,200 --> 00:36:50,560
usually gets us all the way there but we still have to actually go through and manually check these

353
00:36:50,560 --> 00:36:56,640
because like I was saying before Red Dead's a little bit different in that this slider

354
00:36:56,640 --> 00:36:59,760
doesn't necessarily mean you're going to go everything ultra if you hit that.

355
00:37:00,560 --> 00:37:05,760
If you have lower end hardware it actually kind of tries to adjust that slider based on

356
00:37:05,760 --> 00:37:10,880
your hardware. So on like a low end laptop you go all the way to favorite quality and

357
00:37:10,880 --> 00:37:17,280
everything might still read as high. So it's something that that's why we have actual screenshots

358
00:37:17,280 --> 00:37:24,880
of our settings for it so that we can actually make sure that they're the same even after we

359
00:37:24,880 --> 00:37:33,360
change that slider. So we should be sitting at ultra 16 ultra ultra ultra high ultra high ultra

360
00:37:33,360 --> 00:37:43,680
medium high off high and this is always the one that kind of gets a little bit fussy.

361
00:37:44,240 --> 00:37:56,080
That looks right. Okay and then we're going to hit enter to apply the changes. I'll often

362
00:37:56,080 --> 00:38:01,680
recommend don't alt f4. There are some games out there that the settings don't save until you

363
00:38:01,680 --> 00:38:07,040
actually properly quit the game. It's like a fail safe so if you just exit out the right way

364
00:38:07,760 --> 00:38:11,840
shouldn't be any problem. So first thing we're going to do is set up mark bench here so we

365
00:38:11,840 --> 00:38:17,760
actually start running the benchmarks. So in order to do that we have to input a few values

366
00:38:17,760 --> 00:38:22,880
first. So we're going to go through all the different boxes and options that we have for

367
00:38:22,880 --> 00:38:29,120
the tests we're going to be running today. The first box that we have that we can fill

368
00:38:29,120 --> 00:38:35,200
information into is called the project slug. What that usually entails is every time we create a

369
00:38:35,200 --> 00:38:43,280
test plan we usually assign a sort of either a character value or an actual code name for that

370
00:38:43,280 --> 00:38:49,760
project so we can later look up the data if we have to or reference it in a future video or

371
00:38:49,760 --> 00:38:54,480
something like that. It makes it a little bit easier on our side and then that way the data doesn't

372
00:38:54,480 --> 00:39:00,400
get lost in translation of oh I'm looking for this project but there's eight folders that are

373
00:39:00,400 --> 00:39:08,560
named similarly. It helps kind of basically organize that into a more clear overview of which

374
00:39:08,560 --> 00:39:16,800
project is which. Next is the session tag so if for example we are altering the values from our

375
00:39:16,800 --> 00:39:24,640
standard testing run so we're turning off anti aliasing or we're you know messing around with

376
00:39:24,640 --> 00:39:30,960
which preset we actually use. This helps us assign an actual tag to those specific sessions

377
00:39:31,600 --> 00:39:37,600
so that when we're looking at them later we can be like oh actually it's because the reason why the

378
00:39:37,600 --> 00:39:44,000
values are completely different is because this is the set that we that we ran with experimenting

379
00:39:44,000 --> 00:39:52,160
on anti aliasing or experimenting on you know running low versus high shadows or something like

380
00:39:52,160 --> 00:39:58,560
that so we can basically put in like no dash a and later when we go to look it up it'll actually

381
00:39:58,560 --> 00:40:06,400
have that listed on those specific sessions. Thirdly we have what's called the test parameter

382
00:40:07,440 --> 00:40:11,120
there's a bunch of different options in here that we have currently it's something we can edit

383
00:40:11,120 --> 00:40:18,560
whenever we need to but basically it assigns those sessions what the test parameters we

384
00:40:19,440 --> 00:40:26,640
hope to use for that specific session so for example for this project we're only running at

385
00:40:26,640 --> 00:40:35,360
1080p so we're going to be setting just on here games 1080. There are other options in here like

386
00:40:35,360 --> 00:40:42,640
DX11, RT, stuff like that. This doesn't actually alter any game settings themselves this is just

387
00:40:42,640 --> 00:40:49,200
for our internal use on all of our sessions so we know if we if that run was for 1440p if it was

388
00:40:49,200 --> 00:40:56,480
for ray tracing if it was for XCSS you know that's all those different types of visual parameters

389
00:40:56,480 --> 00:41:03,120
that are outside the norm basically. So for the project slug we are going to be putting in the

390
00:41:03,120 --> 00:41:14,640
one that's on the test plan which is 4080-vs-7900-XTX-2023. So once we have written our project

391
00:41:14,640 --> 00:41:20,320
slug in as well as our test parameters now we can get into actually editing which tests we're

392
00:41:20,320 --> 00:41:25,840
going to be running as well as some of them have different configurations that we can alter

393
00:41:26,160 --> 00:41:35,440
that are non-game tests or for assigning certain values of what the how the game is supposed to

394
00:41:35,440 --> 00:41:41,440
interact that sort of thing. So first we're going to cover blender benchmark we click on it here

395
00:41:41,440 --> 00:41:47,280
any of these ones have different test arguments in most cases so if we click on the blender

396
00:41:47,280 --> 00:41:53,040
benchmark we have a bunch of new values in here we can adjust so things like scheduler delay,

397
00:41:53,040 --> 00:41:59,840
recording delay and recording timeout. These are all in seconds we usually run a scheduler delay

398
00:41:59,840 --> 00:42:06,320
of five seconds just to make sure that the program or the game can have time to close and be

399
00:42:06,320 --> 00:42:11,680
satisfactory with Windows and all copacetic to be able to open again especially when it comes to

400
00:42:11,680 --> 00:42:17,120
steam when you have like cloud game saving enabled and stuff like that it has to take those few

401
00:42:17,120 --> 00:42:22,960
seconds to actually do the cloud save and sync it up what can happen in some instances if we

402
00:42:23,040 --> 00:42:30,800
don't set the scheduler delay is it will run into that bottleneck of oh but the cloud saving

403
00:42:30,800 --> 00:42:35,360
hasn't synced yet are you sure you want to load the game and it kind of gets everything all hung

404
00:42:35,360 --> 00:42:41,200
up so when we set it to five seconds it usually helps give steam enough time to finish that

405
00:42:41,200 --> 00:42:46,640
option and then actually continue on with the next run. So the fourth box here is the number of times

406
00:42:46,720 --> 00:42:54,400
that the test is meant to repeat so it's for that specific test not for the all the test as a whole

407
00:42:54,400 --> 00:43:00,720
so in this case we usually run three for all of our tests that doesn't mean that we only run three

408
00:43:00,720 --> 00:43:06,960
ever and that's it if we find that a result is outside of norm not within our our typical

409
00:43:06,960 --> 00:43:12,880
variance which usually about one to two percent we will do another run till we have three valid runs

410
00:43:12,880 --> 00:43:20,240
that are actually within that variance so we can confidently say that these are the numbers.

411
00:43:20,240 --> 00:43:25,760
So in blender benchmark we also have options for things like the scenes that we use because there's

412
00:43:25,760 --> 00:43:30,400
three different scenes that it actually runs through we're just going to be doing all of the

413
00:43:30,400 --> 00:43:38,160
scenes so which entails classroom junk shop as well as monster that's typically what it runs

414
00:43:38,160 --> 00:43:43,920
in the gooey version with the version that's available through markbench we actually use

415
00:43:43,920 --> 00:43:48,400
the command line interface so we're not going to see anything pop up on the screen for blender

416
00:43:48,400 --> 00:43:53,760
specifically but it is running in the background and we'll show that when we pop over to the actual

417
00:43:53,760 --> 00:43:59,360
command lines to see all the stuff that markbench is actually running then we have which versions

418
00:43:59,360 --> 00:44:04,800
that we can run blender on the blender benchmark on we're going to be doing 3.6 because that's what

419
00:44:04,800 --> 00:44:11,040
the test actually outlines the test plan outlines and then we're going to be testing the GPU not the

420
00:44:11,040 --> 00:44:16,160
CPU because that's what we want to test today is the actual GPUs then we're gonna go ahead and check

421
00:44:16,160 --> 00:44:24,880
it off and move on to the next one which is cyberpunk same thing five and three and then down here we

422
00:44:24,880 --> 00:44:32,240
have for what our keras ip's should be so i can actually talk to them some of our harnesses don't

423
00:44:32,320 --> 00:44:38,640
use keras still um it's something that we're kind of developing as we've been getting into these

424
00:44:38,640 --> 00:44:44,640
future automations and stuff like that to be able to add these tools back into the toolbox of

425
00:44:45,440 --> 00:44:51,040
basically streamlining how each individual harness is so when we get down to red dead

426
00:44:51,040 --> 00:44:54,800
for example it's not going to have that listed right now but we're hoping to have that in the

427
00:44:54,800 --> 00:44:59,360
future so we don't have to do anything else on this one just five and three go down to dota

428
00:44:59,360 --> 00:45:06,240
same thing five and three and we go to red dead we're just going to be setting red dead to five

429
00:45:06,240 --> 00:45:12,560
and three again but here all we have is preset which is current this can't be changed it's a

430
00:45:12,560 --> 00:45:19,040
legacy harness that we've we're currently in progress of updating to use keras as well as

431
00:45:19,040 --> 00:45:24,400
some of the other things that nick will be discussing as far as our levels of automation

432
00:45:24,400 --> 00:45:28,960
goes now we've got everything set up for mark bench it's time to hit the go button and while

433
00:45:28,960 --> 00:45:32,880
this is running the benchmarks i'm going to kick it back over to nick and he's going to talk about

434
00:45:32,880 --> 00:45:45,120
the levels of automation so coming back to automation we're going to focus on the test

435
00:45:45,120 --> 00:45:50,880
harnesses and talk about the degrees of automation because one of the most annoying questions to try

436
00:45:50,880 --> 00:45:57,200
and answer is is it automated when someone asks you that it really depends the answer to that

437
00:45:57,200 --> 00:46:03,600
depends on the scope is a bunch of power shell scripts that run your game and then output something

438
00:46:03,600 --> 00:46:09,040
to a log file considered automated or is it only automated if you can press a button and walk away

439
00:46:09,040 --> 00:46:15,280
and it handles every edge case that you could possibly think of to help talk about automation

440
00:46:15,280 --> 00:46:21,920
internally we've come up with a milestone system and so we come up with the milestones of one two

441
00:46:21,920 --> 00:46:28,240
three and four so when we speak of a test harness taking a game for example a game harness that has

442
00:46:28,240 --> 00:46:33,920
a level one automation it simply just launches and runs the benchmark id like akin to that just a

443
00:46:33,920 --> 00:46:38,640
power shell script doing something very simple and then when we reach a milestone of level two

444
00:46:39,280 --> 00:46:44,800
it can do a lot more such as dealing with advertisements or pop-ups in one of their

445
00:46:44,800 --> 00:46:51,200
launchers or it can report on game settings and then the next level to that is level three is

446
00:46:51,200 --> 00:46:56,480
where it can actually configure the game for itself and also reach out to maybe some more

447
00:46:56,480 --> 00:47:02,960
services like Keras or whatnot our definitions of one two and three are fairly loose because every

448
00:47:02,960 --> 00:47:08,240
game is a little different some games are a lot easier to report on resolution because the it

449
00:47:08,240 --> 00:47:14,800
might be reported in a .ini file that we can grab from app data where others might only store their

450
00:47:14,800 --> 00:47:22,720
settings in some proprietary binary format that we can't even read without some finagling so this

451
00:47:22,720 --> 00:47:28,240
is roughly how we categorize it just so that we can at least internally when someone asks you is it

452
00:47:28,240 --> 00:47:35,040
automated we can say yes to a particular degree and then we get down to number four which is the

453
00:47:35,040 --> 00:47:40,880
future this this category encompasses things that we can't do yet we have something we call our

454
00:47:40,880 --> 00:47:45,440
automation toolbox where we include all the different services such as the keras optical

455
00:47:45,440 --> 00:47:51,920
character recognition uh and uh pie direct input or pie auto gooey it's all the different things

456
00:47:51,920 --> 00:47:59,040
that we can use in our test scripts to achieve the automation of a game so one to three incorporates

457
00:47:59,040 --> 00:48:03,440
those ones i've listed where four is some of the tools that we haven't come to yet the future is

458
00:48:03,440 --> 00:48:08,000
where we'll start to talk about ndi and how we use it for video recording as well as computer

459
00:48:08,480 --> 00:48:13,680
when we started automating our game suite we relied on games that had a canned benchmark

460
00:48:13,680 --> 00:48:17,440
due to the fact that it's a lot easier to automate as you just have to navigate the menus and then

461
00:48:17,440 --> 00:48:22,480
start the benchmark however the problem with that is that not all built-in benchmarking

462
00:48:22,480 --> 00:48:27,840
utilities are representative gameplay and not all games come with a canned benchmark which limits

463
00:48:27,840 --> 00:48:32,560
our selection for our game suite to expand our game suite we started looking at games such as

464
00:48:32,560 --> 00:48:37,200
dota 2 and rocket league that have a very robust replay system in the future we're looking to

465
00:48:37,200 --> 00:48:42,400
incorporate computer vision tools such as depth estimation and object detection to expand our game

466
00:48:42,400 --> 00:48:48,000
selection to any game before we can use depth estimation or object detection we need to get a

467
00:48:48,000 --> 00:48:54,560
video stream off of the test bench to another computer to run the machine learning to do so we

468
00:48:54,560 --> 00:49:01,840
use ndi or network device interface to get video stream from the test bench to the machine learning

469
00:49:01,840 --> 00:49:10,000
computer ndi allows us to convert HDMI signal to video over ip which then can be picked up by any

470
00:49:10,000 --> 00:49:16,080
other client that's on the subnet for example on our computer on the cart we have a client that's

471
00:49:16,080 --> 00:49:21,840
picking up the ndi stream being sent from the test bench which then runs our experiment with depth

472
00:49:21,840 --> 00:49:26,640
estimation and depth estimation is the task of measuring the distance of each pixel relative

473
00:49:26,640 --> 00:49:31,920
to the camera depth estimation could be used to instruct a player to walk down a hallway by

474
00:49:31,920 --> 00:49:36,240
chasing the darkest point on the screen we are also experimenting with other techniques such as

475
00:49:36,240 --> 00:49:42,720
optical character recognition and object detection on a video stream the ndi encoder that we're using

476
00:49:42,720 --> 00:49:50,320
is an n6 from kill of you which has an HDMI in HDMI out and then your Ethernet a particular nice to

477
00:49:50,320 --> 00:49:55,680
have on this unit is the small display on the front which allows you to see the activity of the

478
00:49:55,680 --> 00:50:02,400
unit as well as the ip and what video stream it's sending out okay so all of our benchmarks have

479
00:50:02,400 --> 00:50:10,080
finished let's take a look at the results as you can see here it actually says we've ran

480
00:50:10,080 --> 00:50:16,160
zero zero for dota 2 so we didn't actually get any valid runs so we're gonna actually have to do some

481
00:50:16,160 --> 00:50:24,240
troubleshooting figure out why so that's the thing is that markbench allows us to identify some very

482
00:50:24,240 --> 00:50:30,160
early on possible bugs that might occur with the run and it'll usually terminate so if we scroll up

483
00:50:30,160 --> 00:50:36,400
here you can actually see we get to a point where it says test failed and it's for this run here so

484
00:50:36,400 --> 00:50:41,520
it says game didn't start in time check settings and try again so this is something that we've

485
00:50:41,520 --> 00:50:46,640
programmed into some of our logging so we're gonna have to actually go into our dota 2 settings and

486
00:50:46,640 --> 00:50:52,080
it could be that maybe something got reset when we swapped cables or something like that so we're

487
00:50:52,080 --> 00:50:57,760
gonna take a look at that and get it set back up to the correct settings and try running it again

488
00:50:58,720 --> 00:51:03,760
so we're in dota 2 let's take a quick look at the setting and see what might have went wrong here

489
00:51:04,640 --> 00:51:13,600
so we go to video ah there's the problem it changed itself back to 4k it's a good chance that

490
00:51:13,600 --> 00:51:18,160
while we were swapping some cables around or something like that when we were doing the initial

491
00:51:18,160 --> 00:51:24,320
setup I probably swapped itself back to native resolution and so it caused our markbench runs

492
00:51:24,320 --> 00:51:32,880
to fail so let's change that back and then we'll rerun it here the upside of this is is we don't

493
00:51:32,880 --> 00:51:39,360
have to run every test we just ran again we can actually unmark everything else and just specifically

494
00:51:39,360 --> 00:51:44,640
target dota 2 now so let's go ahead and get that started and hopefully we'll come back with some

495
00:51:44,640 --> 00:51:56,080
actual runs

496
00:52:06,400 --> 00:52:12,160
so looks like we got three runs done for dota 2 now so we can actually upload all this data to our

497
00:52:12,160 --> 00:52:18,800
data ingress server and start trimming so let's get that done now we've uploaded the data to our

498
00:52:18,800 --> 00:52:25,360
server we can pull it up on grafana which is our data visualization dashboard from here we can pull

499
00:52:25,360 --> 00:52:32,240
up all of projects that we've run any sessions that have occurred from basically any of the uploads

500
00:52:32,960 --> 00:52:39,920
so on grafana itself we have the ability to check our projects as well as in this specific

501
00:52:39,920 --> 00:52:47,680
dashboard we're looking at GPUs so for our current project it lists the 7900 XTX as well as the 4080

502
00:52:48,640 --> 00:52:54,000
for tests it lists all the different tests that we've run for a given project and then there's

503
00:52:54,000 --> 00:52:59,120
parameters so in this case we only specify games 1080 so that's the only one that shows up on the list

504
00:53:00,400 --> 00:53:07,120
and then there is the option for the bench which essentially that takes the actual PC name when

505
00:53:08,080 --> 00:53:15,680
when markbench logs it to help us identify which system we actually ran on and then lastly is the

506
00:53:15,680 --> 00:53:22,640
tag which is blank because we didn't label any tags for anything that we needed so from here we

507
00:53:22,640 --> 00:53:28,720
can pull up any of the sessions so let's take a look at dota 2 first here so as you can see here

508
00:53:28,720 --> 00:53:32,960
it's kind of a little bit of a mess if you don't know what you're looking at here but effectively

509
00:53:32,960 --> 00:53:38,720
this is the beginning load this is the main menu and then we have our little load screen

510
00:53:39,280 --> 00:53:45,520
and then we have that initial bit before the actual benchmark occurs the actual bits that we need

511
00:53:45,520 --> 00:53:54,160
are from this valley to this valley and then this last bit here is when it goes back to the main

512
00:53:54,160 --> 00:54:02,480
menu so that's how we how we kind of know which game it's all specific you kind of have to know

513
00:54:02,480 --> 00:54:07,440
what you're looking for at the beginning of like determining oh which part do I actually

514
00:54:07,440 --> 00:54:14,480
need to use so what we all all we have to do at this point is just drag from there to there

515
00:54:15,600 --> 00:54:22,800
apply the correct time range sometimes it skews it by a second if we do that that gives us our run

516
00:54:23,760 --> 00:54:30,560
and then basically I just do that for all the sessions that we have and then once we finish

517
00:54:30,560 --> 00:54:37,760
trimming them we take the data and we put it into our trim session tool on the harbour

518
00:54:38,720 --> 00:54:46,320
LTT Labs page and that basically gives us a graph ideally we want all three sessions to be

519
00:54:47,360 --> 00:54:54,480
basically almost the same however as I outlined previously red deads a little different there's

520
00:54:54,480 --> 00:55:01,280
a bit of rng that occurs during each run so they don't look the exact same on here but the end

521
00:55:01,280 --> 00:55:07,600
result of the averages in the first should be the same or within that one to two percent so at this

522
00:55:07,600 --> 00:55:12,960
point all we gotta do now is trim all the data so we can send it over to sammy and she can start

523
00:55:12,960 --> 00:55:18,240
working on getting those graphs created so let's get to it once the game has finished and the data

524
00:55:18,240 --> 00:55:23,120
is uploaded to the server we need to mark an in and out on the benchmark data to eliminate the

525
00:55:23,200 --> 00:55:28,080
menus and any loading screens from when we calculate the FPS currently john's doing this

526
00:55:28,080 --> 00:55:33,040
manually in grafana by literally looking at the pattern of the frame times in the FPS we're trying

527
00:55:33,040 --> 00:55:37,760
to help them out by having machine learning doing something similar by training a model based on

528
00:55:37,760 --> 00:55:44,000
the behavior of the GPU its temperature and core clocks and memory usage as well as the pattern

529
00:55:44,000 --> 00:55:50,000
and frame times to detect the in and out of a benchmark however that still can be faulty so

530
00:55:50,000 --> 00:55:54,880
we're looking at eliminating it all together by using computer vision from the ndi streams as you

531
00:55:54,880 --> 00:56:00,560
saw earlier to mark an in and out very precisely as it's close to the frame when the game first

532
00:56:00,560 --> 00:56:06,320
loads as we can find we're also looking to apply machine learning and models trained on GPU behavior

533
00:56:06,320 --> 00:56:12,960
to identify erroneous runs or bad runs based on historical data now that we've finished trimming

534
00:56:12,960 --> 00:56:19,600
all the sessions grafana will actually spit out a list of all the different trim session ids

535
00:56:20,000 --> 00:56:25,360
so we can use them to generate our game test report they might ask us how do you know these

536
00:56:25,360 --> 00:56:31,520
are good trim sessions maybe your settings were bad or maybe you know it was just every run was

537
00:56:31,520 --> 00:56:38,240
terrible that's where our other dashboard comes in redash this basically shows historical trim

538
00:56:38,240 --> 00:56:43,920
data that we can use to help verify that these runs are in fact the runs that we need this is

539
00:56:43,920 --> 00:56:48,640
something that gets generated over time if we have a game that we haven't done a whole lot of

540
00:56:48,640 --> 00:56:55,520
trimming on it will be there will be less numbers to be able to verify against but ideally we want

541
00:56:55,520 --> 00:57:02,240
the trend to be the same or similar so in this case if we're looking at cyberpunk on a 4080

542
00:57:03,200 --> 00:57:14,320
on 1080p we're getting 190 average ish to 180 average ish and 154 to 153 for a 1%

543
00:57:15,120 --> 00:57:22,800
so what we can do now is we can use our we can compare our trimmed sessions versus historical

544
00:57:22,800 --> 00:57:27,600
data to verify that it's all correct so what we're going to do is we're going to go on to our

545
00:57:27,600 --> 00:57:31,760
harbor Labs tools again and we're going to go through and we're going to start creating an

546
00:57:31,760 --> 00:57:37,120
actual test report so what we do is we want GPU by component we're not going to include a line

547
00:57:37,120 --> 00:57:44,000
chart because we were just looking at those so we hit create now what we do is we take all of these

548
00:57:44,080 --> 00:57:53,840
session IDs and we hit add let me do it for the other card and so what that does now is it actually

549
00:57:53,840 --> 00:58:00,320
generates a report for us to be able to see visually both with a bar graph as well as just the numbers

550
00:58:01,600 --> 00:58:08,000
what our actual results are so if we're looking here at cyberpunk 2077 on 1080p

551
00:58:08,880 --> 00:58:16,480
the 4080 for these three runs that we just did has an average of 193 and a 1% low of 154

552
00:58:17,360 --> 00:58:23,200
so we're actually above our previous average now these numbers were with a reference card

553
00:58:23,200 --> 00:58:29,440
obviously the accelerates a little bit over a clock so it stands for reason that 193 seems

554
00:58:29,440 --> 00:58:37,200
accurate and so we can keep scrolling down so taking a look at dota 2 we can see that the

555
00:58:37,200 --> 00:58:42,880
numbers appear to line up with what i have experienced in the past we don't have a whole

556
00:58:42,880 --> 00:58:48,320
lot of historical data on it currently because it is one of our newer harnesses that we actually

557
00:58:48,320 --> 00:58:53,840
have been developing and i've kind of just been beta testing but based on my personal experience

558
00:58:53,840 --> 00:59:00,880
with it these numbers do appear to be accurate okay so lastly let's take a look at red dead 2 here

559
00:59:01,680 --> 00:59:07,680
so now we've taken a look at some of the numbers

560
00:59:08,720 --> 00:59:15,040
we can publish this report and get it sent off to sammy to start generating the graphs

561
00:59:15,760 --> 00:59:20,640
so what we need to do first is put the report name which is usually the project slug

562
00:59:21,200 --> 00:59:36,160
um so 40 versus 7900 x tx 2023 results in some cases if we decide to have separate reports based

563
00:59:36,160 --> 00:59:42,640
on like ray tracing versus non ray tracing stuff like that even though those headers of like the

564
00:59:42,640 --> 00:59:50,320
games 1080 um help us filter that if in some cases we just want to look at those bar graphs separately

565
00:59:50,320 --> 00:59:58,240
and the data that they've generated sometimes we will append uh rt results or dlss results or

566
01:00:04,960 --> 01:00:11,200
it's just a description of effectively what the reports based on so we're just going to do results

567
01:00:11,200 --> 01:00:23,760
for 40 80 dash vs dash 7900 x tx dash 203 person then for a reference component

568
01:00:24,320 --> 01:00:32,400
basically what this does is it generates a relative percentage um based on the data so

569
01:00:32,400 --> 01:00:42,560
effectively like oh we can see that the 40 80 is negative 1.62 percent in performance versus

570
01:00:42,560 --> 01:00:48,640
the 7900 x tx we can choose whatever component we want it's helpful for when we are looking at

571
01:00:48,640 --> 01:00:53,760
our big GPU reviews because we can actually be like okay we want to focus on this component it's

572
01:00:53,760 --> 01:01:01,120
not the top of the stack but like for example the rx 7600 we want to take a look at and see

573
01:01:01,120 --> 01:01:06,640
how it compares to everything else and it'll allow us to to basically find these relatives a

574
01:01:06,640 --> 01:01:13,840
lot easier than doing the math ourselves so it's very helpful in that regard um so let's go ahead

575
01:01:13,840 --> 01:01:20,800
and we're going to click publish now this will create a report so if we actually go over here

576
01:01:20,800 --> 01:01:27,600
we can see we actually have the 40 80 versus 7900 x tx results we click on that that basically gives

577
01:01:27,680 --> 01:01:33,120
us the same report um that we were just looking at so i'm just going to copy the url here

578
01:01:34,560 --> 01:01:41,360
into an excel sheet that i was working on go ahead and save this we'll get this sent off to her

579
01:01:41,360 --> 01:01:46,720
and then she can start creating our graphs hi my name is sammy i'm the data visualization specialist

580
01:01:47,680 --> 01:01:52,800
for the lab um i started a few months ago and ever since i started we've made some changes to

581
01:01:52,800 --> 01:01:58,720
the graphs um previously they were done by the writing team um but since then i've taken over

582
01:01:58,720 --> 01:02:04,800
that role um and yeah i'm here to walk you through uh some of the design choices that we made as well

583
01:02:04,800 --> 01:02:11,040
as um the process by which we make these graphs for our videos previously the graphs that we had

584
01:02:11,040 --> 01:02:18,240
had a lot of limitations in design choices um in terms of placement or colors or fonts that we used

585
01:02:18,240 --> 01:02:22,400
mostly because we were making them in excel nowadays we're making them in a combination

586
01:02:22,480 --> 01:02:28,960
of python and uh doby illustrator as well as adobe illustrator plugins mainly um one plugin that's

587
01:02:28,960 --> 01:02:34,240
specialized for graphing uh called datlion datlion allows us to make customizations that we weren't

588
01:02:34,240 --> 01:02:40,320
able to do previously um so one of the first things i did uh was use the style guide that was

589
01:02:40,320 --> 01:02:46,160
developed by our creator warehouse graphic designers um they had kind of set out a palette as well as

590
01:02:46,160 --> 01:02:51,040
some fonts that i could use and toy around with i tried to maintain a design design philosophy

591
01:02:51,040 --> 01:02:56,720
that focused on eligibility um so as we've kind of gone through multiple iterations of the graphs

592
01:02:56,720 --> 01:03:01,360
since i started i've made changes to emphasize on that based off a lot of feedback that we've

593
01:03:01,360 --> 01:03:07,120
gotten from our viewers in terms of design aesthetic um one of the things that i really like to focus on

594
01:03:07,120 --> 01:03:14,960
was an almost laboratorial or chalkboard kind of vibe um we wanted to evoke something that was very

595
01:03:14,960 --> 01:03:22,160
similar to the nasa kind of retro space age kind of look we also incorporated a more modern twist

596
01:03:22,160 --> 01:03:28,640
um inspired by the cyberpunk genre as well as more sci-fi films that have come out in recent years

597
01:03:28,640 --> 01:03:34,560
we've focused on using as well a palette that is uh as accessible as possible for example the choices

598
01:03:34,560 --> 01:03:40,160
um of which colors come first in our bar graphs we chose between colors that were very contrasting

599
01:03:40,160 --> 01:03:46,080
another thing we did was uh we put the colors into um different accessibility tools to kind of

600
01:03:46,080 --> 01:03:52,880
measure how visible our color palette would be for people with different types of color blindness

601
01:03:52,880 --> 01:03:58,400
um and we've essentially developed a order and a palette that is as accessible as possible across

602
01:03:58,400 --> 01:04:04,880
as many different types of um color blindness as possible in order to make our graphs legible

603
01:04:04,880 --> 01:04:08,960
so this is an example of one of the game graph templates that i've developed um one of the first

604
01:04:08,960 --> 01:04:13,680
things that i noticed when i came in was that the previous graphs um their title didn't really jump

605
01:04:13,680 --> 01:04:18,240
out at you one of the things i really wanted to do is make use of like kind of like the very like

606
01:04:18,240 --> 01:04:24,640
stampy Labs font that we have to kind of grab attention um for me one of the things that i want

607
01:04:24,640 --> 01:04:30,880
to focus on is like what is what am i looking at and number two what is this me and so the first

608
01:04:30,880 --> 01:04:37,520
part what am i looking at is um kind of answered immediately by this very bold and very loud um

609
01:04:38,480 --> 01:04:44,960
title or header um and so we've added some design elements sometimes they'll be added in especially

610
01:04:44,960 --> 01:04:49,440
if a graph is you know maybe maybe a little more bare if they're only like two or three

611
01:04:49,440 --> 01:04:55,920
things that we're comparing across but you know if they're 12 13 CPUs GPUs whatever that we're

612
01:04:55,920 --> 01:05:01,920
comparing across then obviously we won't have no space for that we then have the bars themselves

613
01:05:02,000 --> 01:05:09,680
um the bars themselves are sorted now by uh best to worst um we'll still keep the higher

614
01:05:09,680 --> 01:05:16,560
is better um in the subtitle of the graph however um we just want to make it like at a glance more

615
01:05:16,560 --> 01:05:23,680
obvious um and another design element that the editors have helped me incorporate was um a marker

616
01:05:23,680 --> 01:05:27,680
that kind of pulses right next to the GPU that we're talking about previously one of our concerns

617
01:05:27,680 --> 01:05:32,720
and the reason why we weren't able to uh sort by best to worst was because we were worried that if

618
01:05:32,720 --> 01:05:38,160
we swapped around the order of every single GPU it'd be hard to follow oh which one is where um and

619
01:05:38,160 --> 01:05:43,120
so that kind of solved our problem and ever since then we've kind of adopted this philosophy of sorting

620
01:05:43,120 --> 01:05:47,840
by best to worst as best as possible we've kept the grid lines kind of sparse because we want to

621
01:05:47,840 --> 01:05:53,760
make sure that the graph isn't overcrowded with different design elements um and another design

622
01:05:53,760 --> 01:06:00,240
choice we've made particularly with the game graphs and the FPS um graphs is that we've put

623
01:06:00,240 --> 01:06:05,200
one percent low on top and we've made it a bolder color than the average so we've decided that one

624
01:06:05,200 --> 01:06:09,680
percent lows are more indicative of overall gaming performance because average values can get pulled

625
01:06:09,680 --> 01:06:15,520
up by um high FPS anomalies um that might happen over the course of a game when the one percent

626
01:06:15,520 --> 01:06:19,280
lows are really low uh the gameplay ends up appearing as really choppy and so that's one of

627
01:06:19,280 --> 01:06:23,200
the reasons why we think that actually the one percent lows the statistic or the data point

628
01:06:23,200 --> 01:06:28,400
that we really want to highlight instead and so one thing that's been added since we've made a few

629
01:06:28,400 --> 01:06:34,400
changes around the lab is that we've added this uh project slug that follows a project from the

630
01:06:34,400 --> 01:06:39,280
beginning to the end and another thing that we've added is also this version number um so we will

631
01:06:39,280 --> 01:06:45,280
change and update the version number every time of new set of graphs is cut um and that will also

632
01:06:45,280 --> 01:06:52,080
ensure that we have the most updated version of graphs in our video and finally we have our

633
01:06:52,080 --> 01:06:58,320
Labs logo in the bottom bottom right uh this Labs logo essentially tells viewers that this graph was

634
01:06:58,320 --> 01:07:05,280
developed by Labs with Labs data so now we're gonna get started with our graphs um i've already gotten

635
01:07:05,280 --> 01:07:11,840
a uh spreadsheet from adam with the list of graphs that he wants me to do uh three of them being

636
01:07:12,720 --> 01:07:19,680
the game graphs so one for each game and then uh the blender benchmark graph and as you can see here

637
01:07:20,320 --> 01:07:29,840
it says um cyberpunk 2020 2077 uh red dead redemption 2 dota 2 blender 3.6.0 and higher

638
01:07:29,840 --> 01:07:37,600
is better and includes all of the values for the presets as well as the resolution x-axis label

639
01:07:37,600 --> 01:07:44,960
project label version number and all of that so that allows me to know what exactly writers want

640
01:07:45,040 --> 01:07:52,480
included in their video and what data they need as well and there's also room for writers to add

641
01:07:52,480 --> 01:07:58,640
descriptions like if they want to include some kind of like asterisks at the bottom with notes or if

642
01:07:58,640 --> 01:08:04,240
they want to cut out a certain GPU or CPU or if they want like another version of the exact same

643
01:08:04,240 --> 01:08:10,400
graph with extra information or less information included we're just gonna open up a new file

644
01:08:10,880 --> 01:08:20,240
and that will be one layer for each graph as well and i will copy and paste um the layers from our

645
01:08:20,240 --> 01:08:26,960
template so for our first three graphs it's going to be a game graph and so for graph number one

646
01:08:26,960 --> 01:08:37,920
just gonna paste it in and align it you can see that there are already numbers in the template

647
01:08:37,920 --> 01:08:44,400
that's just to help um serve as a placeholder so that i know what graph is which uh that data

648
01:08:44,400 --> 01:08:51,120
will be scrubbed later and then the final graph is our blend blender benchmark so i'm just going

649
01:08:51,120 --> 01:09:02,080
to copy our three series benchmark graph now i'm going to import all of these project labels into

650
01:09:02,080 --> 01:09:08,800
illustrator using visual studio code and a python script that i've written so here i'm just gonna

651
01:09:08,800 --> 01:09:14,560
load in the packages that i need and here in chart names i'm just gonna pull up my file and file

652
01:09:14,560 --> 01:09:23,520
explorer and copy the path to the file and paste it into here um and we'll see here one two three

653
01:09:23,520 --> 01:09:34,720
four graphs um with all the graph titles subtitles text and uh project label version number and now

654
01:09:34,720 --> 01:09:38,560
we know that there are four graphs so i'm just gonna make sure that this number here is four if there

655
01:09:38,560 --> 01:09:44,480
were 50 i'd do 50 since there are four i'm just going to type in four um and there are no y-axis

656
01:09:44,480 --> 01:09:50,080
labels so we're just going to skip over that and then press enter and as we press enter we can see

657
01:09:50,080 --> 01:09:55,680
that it'll man automatically change everything so we went from you know all the placeholder text to

658
01:09:56,400 --> 01:09:59,600
all of this text and let's just double check that everything is right

659
01:10:05,360 --> 01:10:11,360
i'm going to open up harbour to look up the game test report and i'm going to download it as the csv

660
01:10:12,000 --> 01:10:19,680
we'll see here that the labels for the gpus are a little long um we don't really need the NVIDIA

661
01:10:19,760 --> 01:10:23,440
gforce part so i'm just going to cut that out it's going to be rtx 4080

662
01:10:28,880 --> 01:10:34,320
i'm just going to select the graph that i'm working on at the time let's make this full

663
01:10:35,280 --> 01:10:40,720
screen select the graph and i'm just going to delete everything that i have and make sure everything

664
01:10:40,720 --> 01:10:47,360
clears yep it's saying that i have no items at all to graph and i'm just going to paste in what i have

665
01:10:47,360 --> 01:10:56,240
and so here we see um because with our game graphs we like to sort uh highest to lowest because higher

666
01:10:56,240 --> 01:11:01,680
is better um and it does that automatically um the legend will stay the same uh one percent

667
01:11:01,680 --> 01:11:09,040
low on average result so this is a benchmark graph it has three series i had a um three series template

668
01:11:09,120 --> 01:11:16,720
ready

669
01:11:26,000 --> 01:11:31,040
okay i'm done now i've put in the data for all of the graphs and then we're just going to make

670
01:11:31,040 --> 01:11:37,680
sure that all the data is correct so what i'm going to do is um look over um the data based off of

671
01:11:37,680 --> 01:11:45,360
the spreadsheet that i pulled a cyberpunk for example for the 4080 is um 154 193

672
01:11:54,000 --> 01:12:02,400
okay so now all the data is included um we're just going to change up the uh x axis a little bit

673
01:12:02,400 --> 01:12:06,640
so you can see here that there are one two three four five six seven

674
01:12:07,360 --> 01:12:11,920
different lines and that's a little too much for my taste i feel like that's a little crowded

675
01:12:12,720 --> 01:12:19,440
and by default our graphs have an interval of 30 we prefer to have it at 60 but sometimes when the

676
01:12:19,440 --> 01:12:24,320
graph results are really low like having 60 doesn't really make sense so i'm just going to change it

677
01:12:24,320 --> 01:12:31,360
here to 60 so that it's a little less crowded and we can see 60 120 and 180 um with this one i'm going

678
01:12:31,440 --> 01:12:34,080
to do about the same because it's kind of within the same range

679
01:12:39,520 --> 01:12:42,960
i'm going to pull up go back to illustrator instead of exporting everything and having

680
01:12:44,160 --> 01:12:47,840
you know people look over it and then give me feedback individually what i'm going to do is

681
01:12:47,840 --> 01:12:53,040
i'm going to um share it for review it what it's going to do is create essentially like a google

682
01:12:53,040 --> 01:12:58,480
doc where people can add comments so i'm going to um set it to anyone with the link can comment and

683
01:12:58,480 --> 01:13:07,200
copy it and i'm going to uh send it off to adam and john so i'm just going to show everybody

684
01:13:07,200 --> 01:13:12,560
what it looks like and this is essentially what it looks like for john and adam um they're just

685
01:13:12,560 --> 01:13:16,960
going to look over it for multiple things they're going to make sure that all the labels are correct

686
01:13:16,960 --> 01:13:21,760
um and they're going to make sure that the uh data is correct it matches up with the right

687
01:13:21,760 --> 01:13:28,720
card and it matches up with the right um game or test uh here if they do you find something for

688
01:13:28,720 --> 01:13:37,040
example um they can leave a comment so they can highlight this and uh they can just write a comment

689
01:13:37,040 --> 01:13:50,480
and say uh for this one this data is actually for dota 2 not cyberpunk theoretically if i made

690
01:13:50,560 --> 01:13:59,920
that mistake okay so now i've sent off my um graphs to adam and john and they've reviewed it and adam

691
01:13:59,920 --> 01:14:06,640
doesn't have any comments and john has a question about um the specifics of like when we're going to

692
01:14:06,640 --> 01:14:15,600
use commas for the graphing numbers and so um you know i can reply to him and i'll say we're adding

693
01:14:15,600 --> 01:14:25,280
in commas whenever uh numbers are above a thousand if we had any changes to make

694
01:14:25,280 --> 01:14:30,160
then i would go in i would make the changes and then i'd just update the content um since we

695
01:14:30,160 --> 01:14:33,520
don't have any changes to make what i'm going to do is prepare for export

696
01:14:36,880 --> 01:14:42,240
and adam will be the one to send it off to the editing team and we're going to put it in the video

697
01:14:42,240 --> 01:14:48,080
yeah that's pretty much it on my end okay so at this point i received the graphs from sammy

698
01:14:48,080 --> 01:14:53,760
so what i do now along with adam as we go through when we kind of take a look at the graphs he does

699
01:14:53,760 --> 01:15:00,960
sort of a subjective look on the presentation of the graphs i do more of a data analysis make sure

700
01:15:00,960 --> 01:15:05,680
everything was inputted correctly maybe something got copied into the wrong line or something like that

701
01:15:06,480 --> 01:15:11,760
so basically i just have to pour over the data here make sure everything looks right

702
01:15:11,760 --> 01:15:18,160
so we just quickly scan through blender here 50 48 20 10

703
01:15:24,320 --> 01:15:30,000
this blender looks good and then if we hop back over to the the game test report

704
01:15:30,720 --> 01:15:35,920
we can quickly look at all of them up here so if we go to dota 2

705
01:15:42,000 --> 01:15:48,800
so this point now i've basically verified that all the data looks correct um assuming that adam

706
01:15:48,800 --> 01:15:54,640
doesn't have any other issues then we go ahead and get these all set up sent off to the editors

707
01:15:54,640 --> 01:16:00,560
and that's when they start cutting the video after all of that we then go into post production

708
01:16:00,560 --> 01:16:06,240
where we provide the assets and the footage to the editor we have several more steps of qc that

709
01:16:06,240 --> 01:16:11,040
occurred during this um first off you have the editor and writer review where the writer sits

710
01:16:11,040 --> 01:16:15,920
down with the editor watches what the editor has created and um points out any sort of fixes

711
01:16:15,920 --> 01:16:21,280
whether they're stylistic or content fixes that need to be done then once those fixes are implemented

712
01:16:21,280 --> 01:16:27,360
they send a pass to Labs Labs looks over it Labs points out any fixes typically are their fixes

713
01:16:27,360 --> 01:16:31,200
are going to be more focused on the correctness and the data side of things and also maybe kind of

714
01:16:31,200 --> 01:16:36,080
small nitpicks less so about the stylistic side of things but we still make sure that we provide

715
01:16:36,080 --> 01:16:42,160
Labs input into that because sometimes you can lose the exact meaning of stuff just through like a

716
01:16:42,160 --> 01:16:46,160
couple of different wardings and we want to make sure that that's always communicated correctly

717
01:16:46,880 --> 01:16:52,160
then once Labs reviews it the editor implements all those fixes there is another review with the

718
01:16:52,160 --> 01:16:57,760
writer and Labs to make sure that all the fixes have been implemented correctly once that is done

719
01:16:57,760 --> 01:17:02,960
we then upload it to Floatplane for our ecc squad to take a look at the whole point of ecc squad

720
01:17:02,960 --> 01:17:08,080
is to get like a ton of knowledgeable people to have eyes and ears on exactly what we've made

721
01:17:08,080 --> 01:17:11,360
because they'll notice the smallest kind of details when you have like a dozen sets of eyes you'll

722
01:17:11,360 --> 01:17:17,440
just find so many small things and they're always great and once we've done that we then

723
01:17:17,440 --> 01:17:24,080
review the suggestions that the ecc squad has made and if they need to be implemented we do that again

724
01:17:24,720 --> 01:17:30,320
we also check with Labs to make sure that those corrections are in fact correct implement them

725
01:17:30,320 --> 01:17:35,200
make sure they're implemented correctly and then finally after all of that a video gets released

726
01:17:41,360 --> 01:17:44,720
update your BIOS kids