WEBVTT 00:00:00.000 --> 00:00:04.560 Welcome to the Labs Process Video. We're going to give you a step-by-step breakdown of everything 00:00:04.560 --> 00:00:10.000 we do when we test a product for a video, all the way from ideation to the final results. 00:00:10.000 --> 00:00:15.680 Every test starts with an idea. For example, it's been a long time since we've reviewed the 7900XTX 00:00:15.680 --> 00:00:21.440 and the 4080, and we want to know how do these perform today? The first step is filling out our 00:00:21.440 --> 00:00:26.160 test request form. As much as it would be nice to just go down and ask Labs to test these cards, 00:00:26.160 --> 00:00:30.880 they're not going to do anything unless I provide them with a filled out test request form. There 00:00:30.880 --> 00:00:35.680 is no testing without requesting. The reason we have to do a request form is that it allows Labs 00:00:35.680 --> 00:00:40.960 to track the project. If we don't give them a request form, they won't do the testing because 00:00:40.960 --> 00:00:44.880 that's how we waste people's time. In here we put like the name of the project, so we'll say like 00:00:44.880 --> 00:00:51.120 4080 versus 7900XTX. Then we'll put a link to the Trello just again for project tracking. And then 00:00:51.120 --> 00:00:56.160 here's the first kind of important answer. This is an important delineation. Is it exploratory 00:00:56.160 --> 00:01:01.680 testing or project testing? So exploratory testing is going like, oh, we don't know if this is going 00:01:01.680 --> 00:01:05.840 to be an interesting story. So we need to test to see if the effect that we are talking about 00:01:05.840 --> 00:01:10.400 is significant. Like how much does a browser extension slow down your browser? Maybe it's 00:01:10.400 --> 00:01:15.360 not enough to be an interesting video, but maybe there's something there versus a project test, 00:01:15.360 --> 00:01:20.960 which is like, it's a new GPE review. We know that we're going to test it. We don't need to explore. 00:01:21.600 --> 00:01:26.560 In this scenario, we're doing a project test. Then we give them a due date, so a project timeline. 00:01:26.560 --> 00:01:30.400 We can either do a hard deadline or a soft deadline. A soft deadline is kind of like, 00:01:30.400 --> 00:01:34.720 if something else of higher priority comes in, Labs can push it back a little bit because maybe 00:01:34.720 --> 00:01:38.960 it's a video that doesn't really need to come out on a specific date. A hard deadline is really 00:01:38.960 --> 00:01:43.360 important. If it's something that's very timely, like maybe it's a sponsored project or like an 00:01:43.360 --> 00:01:48.240 embargo GPE review, then we give them a list of all of the components that we want tested in the 00:01:48.240 --> 00:01:53.200 video. For this scenario, we're just going to do the two cards, a 4080 Accelerate and a 7900 XTX. 00:01:55.840 --> 00:02:00.560 Then we just let them know if we have them here or if they're coming. Basically it allows them to 00:02:00.560 --> 00:02:04.960 be like, where do we need to listen for the next step? Is it like the writer or do we need to like 00:02:04.960 --> 00:02:09.440 keep an eye on procurement or something like that? Sometimes we also might want to like 00:02:09.440 --> 00:02:14.320 talk to Labs about any sort of materials or components that we need to actually get. 00:02:14.320 --> 00:02:18.880 And that'll usually happen during like a kickoff meeting. Then we have to clearly state what our 00:02:18.880 --> 00:02:23.120 questions are. It's really important to do this and like the more nailed down our questions are, 00:02:24.240 --> 00:02:28.000 the faster the project's going to go and the faster the kickoff meeting will go. 00:02:28.000 --> 00:02:31.120 There have been times where I've stepped into meetings where I did not have a solid question 00:02:31.120 --> 00:02:35.280 and then it takes a long time, waste everyone's time trying to figure out what the heck I'm doing. 00:02:35.280 --> 00:02:43.280 In this scenario, we're just trying to see how well does AMD's 7900 XTX compared to the 4080 00:02:45.280 --> 00:02:49.840 is the 4080 really we're going to be worth the extra money. And then we can also select specific 00:02:49.840 --> 00:02:54.960 tests. We have a whole test portfolio that we can usually run like where we're just run through 00:02:54.960 --> 00:02:59.040 a gauntlet of games and productivity benchmarks, but maybe I just need to select few or there's 00:02:59.040 --> 00:03:03.920 some that I absolutely must have. In this case, we don't need to run these through like a million 00:03:03.920 --> 00:03:11.680 gaming benchmarks, just a few. We're going to do Cyberpunk 2077, Dota 2, Red Dead Redemption 2, 00:03:11.680 --> 00:03:15.840 and they're all going to be at 1080. The next part is like any sort of notable 00:03:15.840 --> 00:03:19.840 test boundary conditions. So say we wanted to have something that's like out of the 00:03:19.840 --> 00:03:24.880 ordinary, like, you know, maybe we need old drivers, like we want the drivers that were 00:03:24.880 --> 00:03:29.200 used when it came out to compare to the drivers that are out today. That's something we can put 00:03:29.200 --> 00:03:34.080 in here. And then for any super general notes that might not really have another place to go, 00:03:34.080 --> 00:03:37.200 there's a final form just at the bottom where we can fill out and be like, 00:03:38.160 --> 00:03:42.480 I don't know, maybe like, you guys want to hang out sometime too? I don't know, 00:03:42.480 --> 00:03:46.640 you can ask them whatever they need to submit it and it goes off to them. 00:03:46.640 --> 00:03:54.640 Hi, it's Gary. We just received a test request from Adam. Normally this goes to Sharon, but Sharon 00:03:54.640 --> 00:04:02.560 is busy doing some other work today, so I'm handling this request. Once it comes into our mailbox, 00:04:03.360 --> 00:04:12.640 we simply click on it, brings up the lab's data request. This one's obviously from Adam. 00:04:13.760 --> 00:04:17.760 We're testing the 4080 against the 7900XTX. 00:04:24.880 --> 00:04:29.200 And then he's actually filled out some specific tests that he wants. 00:04:29.200 --> 00:04:36.480 All of these, Cyberpunk, Red Dead Redemption, Blender and Dota are part of our standard test 00:04:36.480 --> 00:04:42.720 suite, so that makes it a little bit easier. He doesn't want anything special on it, just straight 00:04:42.720 --> 00:04:51.040 up testing. And apparently he appreciates all of us, so thank you, Adam. With that, we'll go 00:04:51.040 --> 00:04:57.920 over to the assignment form. This is where we keep track of who's assigned the project, 00:04:59.040 --> 00:05:04.320 and also if it's just been assigned or if it's already in progress. For this one, 00:05:05.680 --> 00:05:12.000 we'll go ahead and mark it in progress as John's going to start it momentarily. 00:05:13.440 --> 00:05:20.640 And yeah, we're giving it to John. And with that, I'm going to hand it over to John. Thank you. 00:05:21.440 --> 00:05:25.520 Now we have our kickoff meeting. Basically what this does is that it allows us to 00:05:25.520 --> 00:05:30.160 talk with Labs, make sure that our test plan is all agreed upon, and that we are all on the same 00:05:30.160 --> 00:05:34.880 page, so that we can make sure that we're doing everything right. Labs will often bring some 00:05:34.880 --> 00:05:38.080 concerns and questions. Writers might be like, notice things in the middle of the meeting, 00:05:38.080 --> 00:05:43.280 be like, oh crap, I forgot to add that to the sheet. So this is probably one of the most 00:05:43.280 --> 00:05:46.560 important parts of the whole process, because it allows for all of us to make sure that we are 00:05:46.560 --> 00:05:52.000 aligned on our goals. So we just had the kickoff meeting for this project. Now that we've done 00:05:52.000 --> 00:05:59.360 that, we can actually go about collecting the requirements, the project requirements for the 00:05:59.360 --> 00:06:04.720 system. So we're going to be using our AMD benches today. And that's something that we're going to 00:06:04.720 --> 00:06:10.480 be using going forward for all of our GPU testing. So what we're going to do first is give you a 00:06:10.480 --> 00:06:16.480 rundown of all the hardware we use on our AMD benches, and then get into the nitty gritty of 00:06:17.040 --> 00:06:22.160 actually building it and setting it up correctly. So first thing we do is we got a 00:06:22.160 --> 00:06:28.560 CSONIC Prime PX1600 power supply. We've been using these for a little bit now. They're ATX 3.0 00:06:28.560 --> 00:06:36.480 capable. So the nice thing is we can swap quickly between using a 12-volt high power and typical 00:06:36.480 --> 00:06:41.600 8-pin power connectors for any of the GPUs that are made by like Intel or AMD without having to 00:06:41.600 --> 00:06:47.120 use any extra like pigtails and stuff like that. So that's pretty useful. And for motherboard, we 00:06:47.120 --> 00:06:55.040 are using X670E AORUS extremes. We've been using these for a little while on our AMD benches, 00:06:55.040 --> 00:07:00.720 not that we've used them in many of the videos, but going forward that is going to be our standard 00:07:01.280 --> 00:07:06.560 for our GPU testing. The reason we went with the extremes, we were using the masters for a while, 00:07:06.560 --> 00:07:11.200 and then we ended up going back to the extremes because we found that we liked the VRM a little 00:07:11.200 --> 00:07:17.440 bit further. It was a little bit more robust, and the BIOS settings we found that it could tweak just 00:07:17.440 --> 00:07:23.600 that little bit better than the master for doing some exploration testing of overclocking and stuff 00:07:23.600 --> 00:07:30.160 like that with some of the CPUs we've gotten in. As far as other brands, we have looked at ASUS for 00:07:30.160 --> 00:07:36.880 a little while. We looked at a couple of ASRock boards, but ultimately we kind of ended up on the 00:07:36.880 --> 00:07:44.480 AORUS extreme. So for CPUs, we're going to be using Ryzen 7 7800X3Ds. We actually have a video 00:07:44.480 --> 00:07:51.920 upcoming that is about AMD variance testing that we did to basically determine the three 7800X3Ds 00:07:51.920 --> 00:07:58.000 that we're actually going to go with going forward. They are the fastest one or the slowest one, 00:07:58.000 --> 00:08:04.640 the three tightest ones that we could find to effectively try to reduce any sort of variance 00:08:04.640 --> 00:08:11.360 in our FPS basically between the three benches. So we can kind of more easily verify that if we 00:08:11.360 --> 00:08:15.760 were to take this card and put them on any of the three benches, we should get the same number. 00:08:15.760 --> 00:08:23.200 For RAM, we are using G-Skill Trident Z5 new RGB. The stuff we're going with is got 00:08:23.440 --> 00:08:31.840 6,000 megatransfers per second CL30. Even though AMD has come out with new versions of 00:08:31.840 --> 00:08:38.640 AGSA that have basically unlocked the ability to use anything higher than 6,000, we still found 00:08:38.640 --> 00:08:44.080 that this is kind of the best sweet spot for AMD in terms of performance. You can get up there 00:08:44.080 --> 00:08:47.920 where you start getting your timings a little bit tighter and stuff like that and perform 00:08:47.920 --> 00:08:53.680 basically the same on like 6,800 megatransfers and stuff like that. But 00:08:53.680 --> 00:08:57.440 this seems to be a good price to performance ratio still, so that's kind of what we're going 00:08:57.440 --> 00:09:05.520 with going forward. For SSDs, we are using Samsung 980 Pro 2TB SSDs. We haven't explored 00:09:05.520 --> 00:09:12.480 anything as far as using PCI Express Gen 5 SSDs yet, but this seems to be a really good 00:09:12.560 --> 00:09:18.800 performance ratio on both the reads and writes of both not just the sequential but as well as 00:09:18.800 --> 00:09:24.320 the random reads and writes. So it kind of allows us to get the load times a little bit faster and 00:09:24.320 --> 00:09:29.280 kind of have more benchmarks running in sequential order without having to wait for all those 00:09:30.080 --> 00:09:36.560 extra load times. Sure, most SSDs on the market nowadays are fairly quick as long as we have 00:09:36.560 --> 00:09:42.320 something as DRAM in it, but that's the one we've kind of been going with for a while and 00:09:42.320 --> 00:09:46.800 seems to be good enough for us going forward. And if it ain't broke, don't fix it, right? 00:09:47.520 --> 00:09:53.280 So there's that. And then lastly, to cool the 7800X3Ds, we are using Noctua 00:09:53.280 --> 00:09:58.240 NHD 15 coolers. It's kind of been the standard for a long time for a lot of people. 00:09:59.600 --> 00:10:03.920 As soon as the new gen version comes out, I'm sure we're going to be exploring that as well 00:10:03.920 --> 00:10:10.400 and seeing just how much more beneficial it could be for us. But it's been basically the 00:10:10.400 --> 00:10:15.680 workhorse for us for a while and it probably won't change for a little while afterwards. So 00:10:15.680 --> 00:10:21.280 that's kind of where we're at this point. So for the specific project, we obviously have to get 00:10:21.280 --> 00:10:29.760 the specific cards that we need to test with. So we have a reference, AMD Radeon 7900 XTX. 00:10:29.760 --> 00:10:33.520 It's been the one that we've used in all of our videos that we've ever covered on it. 00:10:34.160 --> 00:10:40.320 It's got a pretty sick looking cooler. And then the video card we're going to be using today is a 00:10:40.320 --> 00:10:48.000 PNY Accelerate RTX 4080. This seems to be currently the one that is the cheapest on the market, 00:10:48.000 --> 00:10:52.560 but that doesn't necessarily mean it's the worst. And this is the one that the requirements 00:10:52.560 --> 00:10:58.480 of the test plan specifically outline. So that's the one we're going to be testing today. And lastly, 00:10:59.360 --> 00:11:05.760 we obviously need some sort of thermal paste to go between the D15 and the 7800 X3D. And we use 00:11:05.760 --> 00:11:13.600 Noctua NTH2 thermal paste. We've been using it for a while. As far as application, more is always 00:11:13.600 --> 00:11:20.640 better than less to some degree. And so when I usually put thermal paste on, it's in an equal 00:11:20.640 --> 00:11:27.920 sign, which is something that you'll probably see in the build part of the video. So I'm sure 00:11:27.920 --> 00:11:32.000 everybody's going to have tons of arguments of, oh, you've used too much or used too little. 00:11:32.000 --> 00:11:36.640 There's been so many discussions about that. It doesn't seem to affect the thermal performance 00:11:37.280 --> 00:11:42.560 in any sort of way. So as long as we have enough, we're good to go. All right, now we've gone over 00:11:42.560 --> 00:12:04.480 the actual hardware, let's get to actually building. Now we've got the system all built together and 00:12:04.480 --> 00:12:09.200 up and posted into the BIOS, we're going to be going through making sure we have the correct BIOS 00:12:09.280 --> 00:12:17.040 version as well as configuring it to the standardized configuration that we do for 00:12:17.040 --> 00:12:23.840 all of our benches. So when we look at our test conditions, we usually outline both drivers 00:12:23.840 --> 00:12:29.200 as well as BIOS revision, because there might be an instance where like we're doing some sort of 00:12:29.200 --> 00:12:34.960 exploratory testing and being like, hey, we actually don't want the latest version of the BIOS. We want 00:12:34.960 --> 00:12:40.000 to go back four or five revisions. That's why we outline it in the actual test conditions. In this 00:12:40.000 --> 00:12:46.880 case, it's saying F11D, which is the current version of the BIOS, but obviously we're not 00:12:46.880 --> 00:13:00.240 running F11D, so we're going to have to update it. Update your BIOS kids. Yeah, there's, especially 00:13:00.240 --> 00:13:12.320 with like new AMD CPUs, 7800X3D in particular, a number of motherboards don't actually support it 00:13:12.320 --> 00:13:17.360 correctly out of the box. So if you don't have the option of actually booting into the BIOS, 00:13:17.360 --> 00:13:23.440 luckily most manufacturers include some sort of BIOS flashback now. Just slot the USB into the 00:13:23.440 --> 00:13:29.280 correct port on the motherboard, make sure obviously the file is named correctly before you do that. 00:13:29.360 --> 00:13:34.000 It's incredibly important if you want a computer to work right. Then once we finally get the BIOS 00:13:34.000 --> 00:13:39.840 updated, we're going to be going ahead and configuring it and then setting up Windows. 00:13:43.920 --> 00:13:49.200 Okay, now we've finished the actual BIOS update. Let's go ahead and start configuring things. First 00:13:49.200 --> 00:13:54.960 thing we're going to go to advanced options, so we can actually see everything. We always turn on 00:13:54.960 --> 00:14:04.640 expo or if it's Intel, it's XMP. Verify that rebar is on. Most AMD boards usually have it on by default 00:14:04.640 --> 00:14:13.280 now, but it's always good to go to. It's usually under IO ports or some sort of IO naming scheme, 00:14:13.280 --> 00:14:20.400 whatever manufacturer makes it. Inside here, we also turn off the IGP or integrated graphics, 00:14:21.120 --> 00:14:30.960 especially if we are testing any sort of dedicated GPU. All it does is sometimes confuse some games 00:14:30.960 --> 00:14:36.800 or programs into wondering which actual graphics card to be using at the time. 00:14:39.120 --> 00:14:45.040 And then from there, the only other thing that we do is go into the fan curves 00:14:45.280 --> 00:14:51.280 and we manually set it so that it is a linear scale. 00:14:54.720 --> 00:15:00.560 So it basically looks like that. Then at that point, we can actually start installing Windows. 00:15:04.560 --> 00:15:12.800 For Windows, we use a optimized Windows image that our software developers have 00:15:13.520 --> 00:15:19.520 created. Basically strip back a bunch of stuff that we don't need to be running in the background, 00:15:19.520 --> 00:15:26.000 as well as adding in some programs that we actively use during our testing. We actually 00:15:26.560 --> 00:15:31.680 reinstall them every three months. We'll usually update them basically and create a new Windows 00:15:31.680 --> 00:15:37.920 image that has our latest updates so that we don't have to constantly do this rigmarole of 00:15:38.640 --> 00:15:45.760 get a million updates, a million drivers. We can just basically install the latest drivers 00:15:45.760 --> 00:15:50.000 and then we have basically all of our programs. So now we just got to let this thing run. 00:15:51.440 --> 00:15:56.960 Very exciting stuff, like watching paint dry. In our test conditions, we'll usually outline the 00:15:56.960 --> 00:16:05.040 exact version of Windows we want to run. Going back to that, we might need to recreate an old 00:16:05.040 --> 00:16:13.200 set of parameters. So we might not be using Windows 11 22H2. We might use 22H1 or that's why 00:16:13.200 --> 00:16:19.040 on that list, we actually have all the other versions of our images still on there so that we 00:16:19.040 --> 00:16:24.240 can actually go back and be like, oh, actually we need to use the version from May. And then we 00:16:24.240 --> 00:16:28.880 basically, if we need to recreate that set of test conditions, we have to go and also stop 00:16:28.880 --> 00:16:34.080 Windows updates from updating. Now we're in Windows. We are not going to install the Gigabyte 00:16:34.080 --> 00:16:41.200 Control Center. So basically at this point, we have to go through update all of our drivers, 00:16:41.200 --> 00:16:48.560 as well as install all of our programs that we're actually using for the project. 00:16:54.080 --> 00:17:01.760 Okay, so now we've installed the chipset drivers. We can make sure that we get the latest NVIDIA 00:17:01.760 --> 00:17:09.280 driver as, or not necessarily the latest, because if the project does require an older version of 00:17:09.280 --> 00:17:14.880 the driver, we got to get that version instead. But in this case, in this instance, we are using 00:17:14.880 --> 00:17:20.960 the latest version. So we're using 537.42. Let's get that downloaded. 00:17:27.520 --> 00:17:30.560 Then we're just going to make sure all the video settings are set correctly, 00:17:30.560 --> 00:17:38.880 so we essentially turn off G-Sync, any sort of potential accelerators, V-Sync, 00:17:40.160 --> 00:17:45.680 so we can get the, basically the raw performance of the card. Update your cards. 00:17:47.040 --> 00:17:54.560 There's a lot of things that NVIDIA doesn't post or advertise, like a number of three series cards 00:17:54.560 --> 00:17:59.920 don't have rebar enabled out of the box. So there's actually a bunch of articles that NVIDIA 00:18:00.080 --> 00:18:09.680 posted that are really useful, such as UEFI fixes, DisplayPort fixes, and things like rebar 00:18:09.680 --> 00:18:14.960 patches and stuff like that. So if you find your card isn't matching up for whatever reason, 00:18:14.960 --> 00:18:20.720 or is doing something funky, good to check those out, because sometimes it might be as simple as, 00:18:20.720 --> 00:18:26.560 oh, I need to do this little update. Boom, I've got more performance, or oh, my video doesn't 00:18:26.560 --> 00:18:30.880 seem to work in the BIOS, but it works fine in Windows. Do a firmware update. Oh, sweet, now 00:18:30.880 --> 00:18:34.880 it's fine. Alrighty, now that we've got that installed, let's just make sure we don't have 00:18:34.880 --> 00:18:43.360 any other weird drivers. Looks good to me. Usually if there's no yellows, we're happy campers. 00:18:44.320 --> 00:18:50.560 We usually do all the Windows updates according to what our project plan is. In most cases, 00:18:50.560 --> 00:18:54.960 it's usually just as simple as getting every single update that's in there. A lot of times, 00:18:55.040 --> 00:18:58.640 what will happen is, you know, there will be driver updates and stuff like that, but if you 00:18:58.640 --> 00:19:04.960 already have a later driver than what Windows sees, often you'll get like a download error a lot of 00:19:04.960 --> 00:19:09.200 the times, because Windows is flagged, oh, you actually have a newer version, we don't need to 00:19:09.200 --> 00:19:16.560 get this. And what we do is we tend to update our Windows image every three months to help include 00:19:16.560 --> 00:19:20.640 a lot of these Windows updates. We essentially have to download a lot less of them. It is time 00:19:20.640 --> 00:19:25.680 consuming and the less time we have to spend on constantly downloading things and updating things, 00:19:25.680 --> 00:19:31.200 the better, right? Then we can get right into the actual testing. It saves us a lot of overhead, 00:19:31.200 --> 00:19:38.640 basically. So one last thing we're going to do before we get all of our games downloaded on the 00:19:38.640 --> 00:19:43.520 system is we're going to go through and just do some last minute Windows optimizations. 00:19:44.240 --> 00:19:50.720 So first thing we're going to do is we're going to do some trimming on the actual drive itself. 00:19:51.920 --> 00:19:57.920 Just helps speed it up just that little bit better. Then we're going to go and go to percent 00:19:57.920 --> 00:20:03.920 temp percent to basically get to our temporary folder. And we're going to basically delete 00:20:03.920 --> 00:20:08.560 everything we can in here. So one last thing we're going to do is we're going to set the recycling 00:20:08.560 --> 00:20:15.600 bin to 15 megs. The reason we do this is just that if we delete any files on the system, it'll 00:20:15.600 --> 00:20:20.480 just delete them right away. It won't sit in the recycling bin taking up extra space we don't need. 00:20:21.360 --> 00:20:25.680 Basically, it skips a step of us happening at empty out the recycling bin in most cases. So 00:20:26.400 --> 00:20:32.160 kind of useful in that regard. So once we have all this set up, the only last thing we have to do 00:20:32.160 --> 00:20:38.480 is just make sure whether it's NVIDIA, Intel, or AMD, we have any sort of variable refresh rate 00:20:38.480 --> 00:20:45.760 stuff turned off, as well as B-sync. The reason behind that is we don't want to potentially give 00:20:45.760 --> 00:20:52.640 any sort of advantage to one manufacturer over another with any sort of technologies they 00:20:52.640 --> 00:20:57.520 might have running in the background causing any optimizations that are effectively outside of 00:20:57.520 --> 00:21:04.080 Windows itself doing hardware optimizations. That way it kind of keeps the playing field a little 00:21:04.080 --> 00:21:10.960 bit more level. So we're going to pop into here. I doubt we'll have G-sync on here. No, because we 00:21:10.960 --> 00:21:16.960 are currently plugged into our NDI viewer to get all of our capture. So for this specific instance, 00:21:16.960 --> 00:21:24.240 we won't have G-sync enabled. But typically, because these monitors are actually G-sync capable, 00:21:25.200 --> 00:21:30.880 anytime we pull up an NVIDIA card inside there, it'll immediately enable G-sync. So we have to 00:21:30.880 --> 00:21:38.080 make sure we go in and actually disable that. And then what we do also is in here, we've got to 00:21:38.080 --> 00:21:47.200 manage three settings, scroll down to vertical sync, and we make sure we force it off. That way, 00:21:47.760 --> 00:21:52.240 because there's some games that they don't either give you the option or they tell you it's off, 00:21:52.240 --> 00:21:57.520 but it's not actually off. It's kind of just ensures that no matter the game, no matter the 00:21:57.520 --> 00:22:04.160 settings, if somebody forgot to turn off V-sync for whatever reason, this basically forces it off. 00:22:04.160 --> 00:22:09.920 So there's no opportunity for the game to even interact with, hey, by the way, I need V-sync. 00:22:10.720 --> 00:22:17.680 So it kind of takes that guesswork out of it. And then we go into display settings. 00:22:18.080 --> 00:22:24.000 We're going to be running at 1080p, so we just leave the resolution as it is, but we change our scale 00:22:24.000 --> 00:22:35.680 to 100% rather than 150. Some of our markbench benchmarks do still use CV in order to do character 00:22:35.680 --> 00:22:41.360 recognition. So if the scaling is at 150% and you're on the desktop, for example, 00:22:42.080 --> 00:22:47.440 the new version of Hitman has a launcher that it shows up beforehand that has all the options 00:22:47.520 --> 00:22:53.280 and stuff in. How markbench works in that one is it actually will look for the specific word. 00:22:53.280 --> 00:22:58.400 If our scaling is different than what it's supposed to be, markbench will be like, 00:22:58.400 --> 00:23:03.200 I can't find this because it's effectively looking for a picture at a different scale 00:23:03.200 --> 00:23:09.360 than what it actually is. So if we set it to 100%, basically we're going to have no problems. 00:23:09.360 --> 00:23:13.920 And then normally what we do is we check to make sure we're at the highest hertz we can be for the 00:23:13.920 --> 00:23:20.080 monitor. Because we're going through the NDI, we actually have to run at 60 hertz because that's 00:23:20.080 --> 00:23:26.800 the maximum the actual NDI itself supports. But typically it's 144 for the actual monitors themselves. 00:23:26.800 --> 00:23:33.760 So lastly, we go into change default graphics settings. We make sure that hardware accelerated 00:23:33.760 --> 00:23:41.040 GPU scheduling is on. This is at Windows level. This isn't hardware specific such as 00:23:42.000 --> 00:23:48.320 NVIDIA Reflex or AMD's Antilag. That is something specific to the hardware, 00:23:49.280 --> 00:23:56.560 not Windows itself. So this one, it's the same across all the platforms of hardware. It doesn't 00:23:57.680 --> 00:24:04.160 give an advantage to NVIDIA over AMD because it's scheduling at the exact same. Anything else that's 00:24:04.160 --> 00:24:09.840 in the menu, usually we have like optimization for Windows games. We turn that off, as well as in 00:24:09.840 --> 00:24:17.360 some cases, there is a variable refresh rate option in here. It usually shows up for Intel 00:24:17.360 --> 00:24:24.960 and AMD because theirs interact a little bit differently than G-Sync does. And so we make 00:24:24.960 --> 00:24:29.760 sure that's off as well. Because again, we don't want to give advantage to one specific card. We 00:24:29.760 --> 00:24:35.680 want to see raw performance of each of the cards stacked against each other so we can understand 00:24:36.320 --> 00:24:41.840 what is actually the best option. Now that we've gotten all the Windows updates installed, 00:24:42.880 --> 00:24:48.880 next thing we got to do is we have to install all the programs as well as any games that we're 00:24:48.880 --> 00:24:54.560 going to be using to do this project. So in this case, we need to install Cyberpunk, Dota 2, 00:24:55.440 --> 00:25:02.960 and Red Dead. So we already have Steam pre-installed on our Windows image so we don't have to worry 00:25:02.960 --> 00:25:06.960 about actually going to the website and getting that part, but unfortunately it always needs to 00:25:06.960 --> 00:25:14.400 update anyways. So it's kind of a moot point. While that's happening, what we're also going to do is get 00:25:15.520 --> 00:25:22.240 markbench from our local drive, as well as some patches that I need to install as well. 00:25:22.320 --> 00:25:32.640 So we are going to install, nope, Cyberpunk. We're going to install Red Dead. 00:25:34.880 --> 00:25:39.360 And then the most annoying thing is that Dota 2 doesn't stay in your library half the time, 00:25:39.360 --> 00:25:45.040 so we are going to go here and do play now. 00:25:45.440 --> 00:25:50.800 Okay, so those three guys are downloading. 00:25:52.880 --> 00:25:58.560 We've got markbench extracted here. We need to add another harness in for the Dota 1, 00:25:58.560 --> 00:26:07.040 so let's get that real quick. And there it is, ready to go for when we are ready to use it. 00:26:08.800 --> 00:26:14.080 And then the other thing we need is Blender Benchmark, so let's quickly grab that as well. 00:26:15.440 --> 00:26:22.320 So although markbench does have a Blender Benchmark actually integrated into it, 00:26:23.120 --> 00:26:28.880 we do need to still download the application itself to download the files required for it to run. 00:26:30.000 --> 00:26:34.880 There's a little bit of bugginess with the CLI. When it tries to download the files, 00:26:34.880 --> 00:26:41.520 it effectively says I can't do this and quits out. So if we download the actual program and have it 00:26:41.520 --> 00:26:47.920 download the files, it just makes the automation a little bit simpler on our side because it has 00:26:47.920 --> 00:26:53.680 the files already completely installed. So effectively we want version 3.6. We're going to 00:26:53.680 --> 00:27:00.080 download the benchmarks. Again, let's just make sure that markbench is all prepared, just like when 00:27:00.080 --> 00:27:05.120 we have to go through any of our game settings. We have to actually go download the programs, 00:27:05.120 --> 00:27:12.400 download the games, actually set up the settings ourselves. That stuff is not currently automated. 00:27:13.120 --> 00:27:16.720 Now we have all the games downloading and everything's kind of getting set up. 00:27:16.720 --> 00:27:20.000 I'm going to throw it over to Nick and he can talk a little bit about markbench. 00:27:20.000 --> 00:27:26.400 Fundamentally, markbench is our test automation and orchestration framework. It's not just simply 00:27:26.400 --> 00:27:32.560 a simple desktop GUI anymore. It's a constellation of services and components that work together 00:27:32.560 --> 00:27:37.680 to form the ecosystem that we conduct our testing in. But it all does start with the test 00:27:37.680 --> 00:27:43.440 harnesses, which most of ours are in Python because it's most easily automatable for us. 00:27:44.080 --> 00:27:50.640 However, it doesn't need to be Python. They can be PowerShell scripts, Ruby, GoLang, what have you. 00:27:50.640 --> 00:27:57.840 The important part is that each test harness gives back a exit code of zero or one, zero to indicate 00:27:57.840 --> 00:28:03.520 a successful test run, anything greater such as one to indicate that it failed. Because that exit 00:28:03.520 --> 00:28:10.080 code is then read by the next layer of markbench, which is what we call auto bench. It just comes 00:28:10.080 --> 00:28:15.120 before the GoLang GUI and it really contains all of the major features of our orchestration. 00:28:16.160 --> 00:28:21.280 This layer is what separates what we're doing from just a collection of PowerShell scripts 00:28:21.840 --> 00:28:27.600 to something that is a lot easier to manage long term because inside of auto bench, that's 00:28:27.600 --> 00:28:32.000 where we get our orchestration. So with the test harness, I have the capability of running 00:28:32.000 --> 00:28:38.000 individual tests, but I want to be able to group those into a bunch of test suites such as let's 00:28:38.000 --> 00:28:45.440 say I want to run Dota 2 four times at one resolution and then Red Dead Redemption 2 at a 00:28:45.440 --> 00:28:50.080 different resolution. So that's where the auto bench side comes in. Auto bench is where we run 00:28:50.080 --> 00:28:55.520 presentmon in system logger in conjunction with the test harness so that we get a similar timestamp 00:28:55.520 --> 00:29:00.960 between all three because this eliminates the human error of starting a test and then starting 00:29:00.960 --> 00:29:06.960 hardware info or like logging on the side, which then would separate our timestamps by 00:29:07.520 --> 00:29:12.640 sometimes multiple seconds, which makes it easier to trim the sessions later downstream, 00:29:12.640 --> 00:29:18.480 which John will explain later. And then so the auto bench is all command line interface, 00:29:18.480 --> 00:29:23.840 so to make it easier for us and John and writers or whoever wants to run Markbench, 00:29:23.840 --> 00:29:29.440 we added a Golang GUI. So we went with the Golang GUI because the framework that we used 00:29:30.320 --> 00:29:36.800 was very successful for us for other GUIs such as the keyboard tester, the robot, and the latency 00:29:36.800 --> 00:29:42.640 tests. So anytime you see Markbench on screen, that's the Golang part, but underneath all the 00:29:42.640 --> 00:29:47.840 work is really being done inside the test harness as well as the Python orchestration layer. 00:29:48.640 --> 00:29:54.320 So once the test is done running, let's say we just ran an individual test and once that test is 00:29:54.320 --> 00:30:01.600 done running, it sends its data up to a service called data ingest server, which is a Golang web 00:30:01.600 --> 00:30:07.920 API, which takes it in that data in a protobuf format and then writes it to our Postgres database. 00:30:08.880 --> 00:30:13.760 It's from that Postgres database, all of our data sits at rest and then is disseminated or sent 00:30:14.400 --> 00:30:20.240 to all the various tools that John can use to evaluate and report on that data. And we use a 00:30:20.240 --> 00:30:27.040 couple tools, notably Grafana and Redash for a lot of visualizations. And then we have internal 00:30:27.040 --> 00:30:33.520 tooling that we just kind of collectively call the Harbor Tools portal. And that contains anything 00:30:33.520 --> 00:30:39.440 that Grafana or Redash or another off the shelf tool doesn't provide us. Now if we go back up to 00:30:39.440 --> 00:30:47.680 the test harnesses, on their own, they're fairly dumb in that they can send keyboard and mouse input, 00:30:48.240 --> 00:30:54.480 but they don't always know what's going on the screen. So we used to use OpenCV locally to do 00:30:54.480 --> 00:31:00.080 template matching, but it's more of a brute force approach. And it's very flaky as it depends on 00:31:00.080 --> 00:31:05.920 certain resolutions. So if you're doing template matching with a resolution of 1920 by 1080, 00:31:06.560 --> 00:31:11.920 you need to select screenshots of the path or the flow you want to go through the menu of the 00:31:11.920 --> 00:31:16.560 particular game. But then if you want to do it at a lower resolution like 720p, 00:31:17.680 --> 00:31:22.800 nine times at a 10, you have to take a smaller screenshot. So it doesn't scale very well. And 00:31:22.800 --> 00:31:27.200 that's where we start getting into some of the services that we're adding. The first one being 00:31:27.200 --> 00:31:34.160 Keras optical character recognition or OCR. This allows a test harness to send a screenshot of the 00:31:34.240 --> 00:31:39.600 system it's running on with a target word and Keras will come back within a few milliseconds 00:31:39.600 --> 00:31:43.440 is that word on the screen or not. This reduces flakiness in the test harness, 00:31:44.000 --> 00:31:49.600 which makes navigating menus a hundred times easier. And in the future, we're going to be adding 00:31:49.600 --> 00:31:56.560 services that do a bunch of computer vision things such as depth estimation, object detection and 00:31:56.560 --> 00:32:01.600 whatnot, all in the effort of making the test harnesses more a lot smarter. But we'll talk 00:32:01.600 --> 00:32:08.160 about levels of automation in a bit. So the benefit that Mark Bench and its ecosystem gives us 00:32:08.160 --> 00:32:15.280 over just a collection of PowerShell scripts for one is maintainability at the scale that we're 00:32:15.280 --> 00:32:22.240 trying to achieve. PowerShell scripts would or some auto hockey scripts would become messy very 00:32:22.240 --> 00:32:27.600 quickly because eventually we'd have probably like 100 to 200. And as best as you could organize 00:32:27.600 --> 00:32:32.960 those in some directories, it's just not a it's a maintainable nightmare. Investing in such an 00:32:32.960 --> 00:32:40.640 ecosystem and the automation of Mark Bench, it allows us to build it to support multiple platforms. 00:32:40.640 --> 00:32:47.920 So not just Windows on a desktop, but handhelds and macOS and Linux as well. And it normalizes 00:32:47.920 --> 00:32:53.520 data for us so that when we capture metrics and we set it at rest, it's a lot easier to 00:32:53.520 --> 00:32:58.400 query that data. Back to you, John. Thanks Nick for that explanation of Mark Bench. 00:32:58.400 --> 00:33:02.640 Now what we're going to do since all these games are finished downloading is get them all set up 00:33:02.640 --> 00:33:06.880 and then we'll be able to get our benchmarks started. So we're going to start off with cyber 00:33:06.880 --> 00:33:14.560 bunk here. We have documentation that we are working on that we will be publishing in the near 00:33:14.560 --> 00:33:20.960 future of all of our settings and configuration for the various benchmarks that we include in our 00:33:20.960 --> 00:33:29.600 videos. I believe we've already released a couple of them on the forums. So I'm sure we'll continue 00:33:29.600 --> 00:33:35.040 to update that as time goes on. Okie dokie. Yeah, the new update for cyberpunk came out. So that's 00:33:35.040 --> 00:33:42.320 cool. So first thing we do, we go to settings. We're going to go to video mode and we're going to 00:33:42.320 --> 00:33:47.280 make sure we're at 1080p because that's what the project requires. You're going to be turning off 00:33:47.280 --> 00:33:52.400 any sort of things like low latency stuff like that. I think that it could benefit that specific 00:33:52.400 --> 00:33:58.400 card because we want the raw performance of the card. We don't want any sort of additional things 00:33:58.400 --> 00:34:05.840 like DLSS or low latency that can certainly help benefit the card and it might get better 00:34:05.840 --> 00:34:12.320 performance for sure. But what we're trying to determine is apples to apples between that 7900 00:34:12.400 --> 00:34:18.320 xdx and this 4080. So we're going to be turning off low latency. We're going to be making sure 00:34:18.320 --> 00:34:25.200 that we're on full screen. We don't want borderless. Then we're going to go to the graphics tab and for 00:34:25.200 --> 00:34:30.000 our specific test we aren't calling for ray tracing. So we're going to actually be turning off 00:34:30.000 --> 00:34:37.040 ray tracing. We're going to be going just strictly ultra and we're going to go ahead and turn down 00:34:37.120 --> 00:34:44.560 FSR to zero. Go back up a tiny bit. Go back down a tiny bit. Each individual time we change something 00:34:44.560 --> 00:34:51.360 we're going to hit apply. The settings for cyberpunk are still kind of a little bit weird 00:34:51.360 --> 00:34:56.320 where you might change some sliders and you might hit apply and sometimes it's forgetting 00:34:56.320 --> 00:35:01.600 that you hit apply or you change the quick preset. It doesn't actually use the preset because you 00:35:01.600 --> 00:35:07.920 haven't applied it. So when we go up and down on the FSR that kind of helps with that and then 00:35:07.920 --> 00:35:13.520 we're going to go ahead and turn FSR off because we don't actually want it. And that's basically how 00:35:13.520 --> 00:35:21.920 we set up our non ray tracing version of cyberpunk. So at that point we can basically exit out of that 00:35:22.800 --> 00:35:30.000 and move on to the next one which is Dota 2. Dota 2 is one of our more recent benchmarks that we've 00:35:30.000 --> 00:35:37.840 been working on. It uses a replay from I believe Dreamhack. A lot of visuals on the screen so that 00:35:37.840 --> 00:35:45.680 we can basically tax the card as much as possible. So in Dota we're going to go up to the cog, go to 00:35:45.680 --> 00:35:50.720 video and we're not going to use monitor's resolution. We're going to use advanced settings. 00:35:52.000 --> 00:35:57.760 In this case we need 1080p. We're going to be checking on to exclusive full screen 00:35:58.560 --> 00:36:01.520 rather than that. Turning off low latency yet again. 00:36:03.520 --> 00:36:08.240 And then for rendering side we just pulled the slider all the way to best looking. 00:36:10.080 --> 00:36:16.160 It's basically a preset that has all these specific settings. These things always off. 00:36:16.720 --> 00:36:20.800 That sort of thing. That's all we got to do for this guy here. So we go ahead and close that. 00:36:21.760 --> 00:36:27.520 Okay now we've done Dota 2. Lastly we just need to set up Red Dead Redemption. So we're going to 00:36:27.600 --> 00:36:35.920 hit Z to go into settings. Go to graphics. Make sure the resolution and the screen type are correct 00:36:35.920 --> 00:36:43.200 which they are. We're going to be turning off v-sync. Now pull the slider all the way to the right that 00:36:43.200 --> 00:36:50.560 usually gets us all the way there but we still have to actually go through and manually check these 00:36:50.560 --> 00:36:56.640 because like I was saying before Red Dead's a little bit different in that this slider 00:36:56.640 --> 00:36:59.760 doesn't necessarily mean you're going to go everything ultra if you hit that. 00:37:00.560 --> 00:37:05.760 If you have lower end hardware it actually kind of tries to adjust that slider based on 00:37:05.760 --> 00:37:10.880 your hardware. So on like a low end laptop you go all the way to favorite quality and 00:37:10.880 --> 00:37:17.280 everything might still read as high. So it's something that that's why we have actual screenshots 00:37:17.280 --> 00:37:24.880 of our settings for it so that we can actually make sure that they're the same even after we 00:37:24.880 --> 00:37:33.360 change that slider. So we should be sitting at ultra 16 ultra ultra ultra high ultra high ultra 00:37:33.360 --> 00:37:43.680 medium high off high and this is always the one that kind of gets a little bit fussy. 00:37:44.240 --> 00:37:56.080 That looks right. Okay and then we're going to hit enter to apply the changes. I'll often 00:37:56.080 --> 00:38:01.680 recommend don't alt f4. There are some games out there that the settings don't save until you 00:38:01.680 --> 00:38:07.040 actually properly quit the game. It's like a fail safe so if you just exit out the right way 00:38:07.760 --> 00:38:11.840 shouldn't be any problem. So first thing we're going to do is set up mark bench here so we 00:38:11.840 --> 00:38:17.760 actually start running the benchmarks. So in order to do that we have to input a few values 00:38:17.760 --> 00:38:22.880 first. So we're going to go through all the different boxes and options that we have for 00:38:22.880 --> 00:38:29.120 the tests we're going to be running today. The first box that we have that we can fill 00:38:29.120 --> 00:38:35.200 information into is called the project slug. What that usually entails is every time we create a 00:38:35.200 --> 00:38:43.280 test plan we usually assign a sort of either a character value or an actual code name for that 00:38:43.280 --> 00:38:49.760 project so we can later look up the data if we have to or reference it in a future video or 00:38:49.760 --> 00:38:54.480 something like that. It makes it a little bit easier on our side and then that way the data doesn't 00:38:54.480 --> 00:39:00.400 get lost in translation of oh I'm looking for this project but there's eight folders that are 00:39:00.400 --> 00:39:08.560 named similarly. It helps kind of basically organize that into a more clear overview of which 00:39:08.560 --> 00:39:16.800 project is which. Next is the session tag so if for example we are altering the values from our 00:39:16.800 --> 00:39:24.640 standard testing run so we're turning off anti aliasing or we're you know messing around with 00:39:24.640 --> 00:39:30.960 which preset we actually use. This helps us assign an actual tag to those specific sessions 00:39:31.600 --> 00:39:37.600 so that when we're looking at them later we can be like oh actually it's because the reason why the 00:39:37.600 --> 00:39:44.000 values are completely different is because this is the set that we that we ran with experimenting 00:39:44.000 --> 00:39:52.160 on anti aliasing or experimenting on you know running low versus high shadows or something like 00:39:52.160 --> 00:39:58.560 that so we can basically put in like no dash a and later when we go to look it up it'll actually 00:39:58.560 --> 00:40:06.400 have that listed on those specific sessions. Thirdly we have what's called the test parameter 00:40:07.440 --> 00:40:11.120 there's a bunch of different options in here that we have currently it's something we can edit 00:40:11.120 --> 00:40:18.560 whenever we need to but basically it assigns those sessions what the test parameters we 00:40:19.440 --> 00:40:26.640 hope to use for that specific session so for example for this project we're only running at 00:40:26.640 --> 00:40:35.360 1080p so we're going to be setting just on here games 1080. There are other options in here like 00:40:35.360 --> 00:40:42.640 DX11, RT, stuff like that. This doesn't actually alter any game settings themselves this is just 00:40:42.640 --> 00:40:49.200 for our internal use on all of our sessions so we know if we if that run was for 1440p if it was 00:40:49.200 --> 00:40:56.480 for ray tracing if it was for XCSS you know that's all those different types of visual parameters 00:40:56.480 --> 00:41:03.120 that are outside the norm basically. So for the project slug we are going to be putting in the 00:41:03.120 --> 00:41:14.640 one that's on the test plan which is 4080-vs-7900-XTX-2023. So once we have written our project 00:41:14.640 --> 00:41:20.320 slug in as well as our test parameters now we can get into actually editing which tests we're 00:41:20.320 --> 00:41:25.840 going to be running as well as some of them have different configurations that we can alter 00:41:26.160 --> 00:41:35.440 that are non-game tests or for assigning certain values of what the how the game is supposed to 00:41:35.440 --> 00:41:41.440 interact that sort of thing. So first we're going to cover blender benchmark we click on it here 00:41:41.440 --> 00:41:47.280 any of these ones have different test arguments in most cases so if we click on the blender 00:41:47.280 --> 00:41:53.040 benchmark we have a bunch of new values in here we can adjust so things like scheduler delay, 00:41:53.040 --> 00:41:59.840 recording delay and recording timeout. These are all in seconds we usually run a scheduler delay 00:41:59.840 --> 00:42:06.320 of five seconds just to make sure that the program or the game can have time to close and be 00:42:06.320 --> 00:42:11.680 satisfactory with Windows and all copacetic to be able to open again especially when it comes to 00:42:11.680 --> 00:42:17.120 steam when you have like cloud game saving enabled and stuff like that it has to take those few 00:42:17.120 --> 00:42:22.960 seconds to actually do the cloud save and sync it up what can happen in some instances if we 00:42:23.040 --> 00:42:30.800 don't set the scheduler delay is it will run into that bottleneck of oh but the cloud saving 00:42:30.800 --> 00:42:35.360 hasn't synced yet are you sure you want to load the game and it kind of gets everything all hung 00:42:35.360 --> 00:42:41.200 up so when we set it to five seconds it usually helps give steam enough time to finish that 00:42:41.200 --> 00:42:46.640 option and then actually continue on with the next run. So the fourth box here is the number of times 00:42:46.720 --> 00:42:54.400 that the test is meant to repeat so it's for that specific test not for the all the test as a whole 00:42:54.400 --> 00:43:00.720 so in this case we usually run three for all of our tests that doesn't mean that we only run three 00:43:00.720 --> 00:43:06.960 ever and that's it if we find that a result is outside of norm not within our our typical 00:43:06.960 --> 00:43:12.880 variance which usually about one to two percent we will do another run till we have three valid runs 00:43:12.880 --> 00:43:20.240 that are actually within that variance so we can confidently say that these are the numbers. 00:43:20.240 --> 00:43:25.760 So in blender benchmark we also have options for things like the scenes that we use because there's 00:43:25.760 --> 00:43:30.400 three different scenes that it actually runs through we're just going to be doing all of the 00:43:30.400 --> 00:43:38.160 scenes so which entails classroom junk shop as well as monster that's typically what it runs 00:43:38.160 --> 00:43:43.920 in the gooey version with the version that's available through markbench we actually use 00:43:43.920 --> 00:43:48.400 the command line interface so we're not going to see anything pop up on the screen for blender 00:43:48.400 --> 00:43:53.760 specifically but it is running in the background and we'll show that when we pop over to the actual 00:43:53.760 --> 00:43:59.360 command lines to see all the stuff that markbench is actually running then we have which versions 00:43:59.360 --> 00:44:04.800 that we can run blender on the blender benchmark on we're going to be doing 3.6 because that's what 00:44:04.800 --> 00:44:11.040 the test actually outlines the test plan outlines and then we're going to be testing the GPU not the 00:44:11.040 --> 00:44:16.160 CPU because that's what we want to test today is the actual GPUs then we're gonna go ahead and check 00:44:16.160 --> 00:44:24.880 it off and move on to the next one which is cyberpunk same thing five and three and then down here we 00:44:24.880 --> 00:44:32.240 have for what our keras ip's should be so i can actually talk to them some of our harnesses don't 00:44:32.320 --> 00:44:38.640 use keras still um it's something that we're kind of developing as we've been getting into these 00:44:38.640 --> 00:44:44.640 future automations and stuff like that to be able to add these tools back into the toolbox of 00:44:45.440 --> 00:44:51.040 basically streamlining how each individual harness is so when we get down to red dead 00:44:51.040 --> 00:44:54.800 for example it's not going to have that listed right now but we're hoping to have that in the 00:44:54.800 --> 00:44:59.360 future so we don't have to do anything else on this one just five and three go down to dota 00:44:59.360 --> 00:45:06.240 same thing five and three and we go to red dead we're just going to be setting red dead to five 00:45:06.240 --> 00:45:12.560 and three again but here all we have is preset which is current this can't be changed it's a 00:45:12.560 --> 00:45:19.040 legacy harness that we've we're currently in progress of updating to use keras as well as 00:45:19.040 --> 00:45:24.400 some of the other things that nick will be discussing as far as our levels of automation 00:45:24.400 --> 00:45:28.960 goes now we've got everything set up for mark bench it's time to hit the go button and while 00:45:28.960 --> 00:45:32.880 this is running the benchmarks i'm going to kick it back over to nick and he's going to talk about 00:45:32.880 --> 00:45:45.120 the levels of automation so coming back to automation we're going to focus on the test 00:45:45.120 --> 00:45:50.880 harnesses and talk about the degrees of automation because one of the most annoying questions to try 00:45:50.880 --> 00:45:57.200 and answer is is it automated when someone asks you that it really depends the answer to that 00:45:57.200 --> 00:46:03.600 depends on the scope is a bunch of power shell scripts that run your game and then output something 00:46:03.600 --> 00:46:09.040 to a log file considered automated or is it only automated if you can press a button and walk away 00:46:09.040 --> 00:46:15.280 and it handles every edge case that you could possibly think of to help talk about automation 00:46:15.280 --> 00:46:21.920 internally we've come up with a milestone system and so we come up with the milestones of one two 00:46:21.920 --> 00:46:28.240 three and four so when we speak of a test harness taking a game for example a game harness that has 00:46:28.240 --> 00:46:33.920 a level one automation it simply just launches and runs the benchmark id like akin to that just a 00:46:33.920 --> 00:46:38.640 power shell script doing something very simple and then when we reach a milestone of level two 00:46:39.280 --> 00:46:44.800 it can do a lot more such as dealing with advertisements or pop-ups in one of their 00:46:44.800 --> 00:46:51.200 launchers or it can report on game settings and then the next level to that is level three is 00:46:51.200 --> 00:46:56.480 where it can actually configure the game for itself and also reach out to maybe some more 00:46:56.480 --> 00:47:02.960 services like Keras or whatnot our definitions of one two and three are fairly loose because every 00:47:02.960 --> 00:47:08.240 game is a little different some games are a lot easier to report on resolution because the it 00:47:08.240 --> 00:47:14.800 might be reported in a .ini file that we can grab from app data where others might only store their 00:47:14.800 --> 00:47:22.720 settings in some proprietary binary format that we can't even read without some finagling so this 00:47:22.720 --> 00:47:28.240 is roughly how we categorize it just so that we can at least internally when someone asks you is it 00:47:28.240 --> 00:47:35.040 automated we can say yes to a particular degree and then we get down to number four which is the 00:47:35.040 --> 00:47:40.880 future this this category encompasses things that we can't do yet we have something we call our 00:47:40.880 --> 00:47:45.440 automation toolbox where we include all the different services such as the keras optical 00:47:45.440 --> 00:47:51.920 character recognition uh and uh pie direct input or pie auto gooey it's all the different things 00:47:51.920 --> 00:47:59.040 that we can use in our test scripts to achieve the automation of a game so one to three incorporates 00:47:59.040 --> 00:48:03.440 those ones i've listed where four is some of the tools that we haven't come to yet the future is 00:48:03.440 --> 00:48:08.000 where we'll start to talk about ndi and how we use it for video recording as well as computer 00:48:08.480 --> 00:48:13.680 when we started automating our game suite we relied on games that had a canned benchmark 00:48:13.680 --> 00:48:17.440 due to the fact that it's a lot easier to automate as you just have to navigate the menus and then 00:48:17.440 --> 00:48:22.480 start the benchmark however the problem with that is that not all built-in benchmarking 00:48:22.480 --> 00:48:27.840 utilities are representative gameplay and not all games come with a canned benchmark which limits 00:48:27.840 --> 00:48:32.560 our selection for our game suite to expand our game suite we started looking at games such as 00:48:32.560 --> 00:48:37.200 dota 2 and rocket league that have a very robust replay system in the future we're looking to 00:48:37.200 --> 00:48:42.400 incorporate computer vision tools such as depth estimation and object detection to expand our game 00:48:42.400 --> 00:48:48.000 selection to any game before we can use depth estimation or object detection we need to get a 00:48:48.000 --> 00:48:54.560 video stream off of the test bench to another computer to run the machine learning to do so we 00:48:54.560 --> 00:49:01.840 use ndi or network device interface to get video stream from the test bench to the machine learning 00:49:01.840 --> 00:49:10.000 computer ndi allows us to convert HDMI signal to video over ip which then can be picked up by any 00:49:10.000 --> 00:49:16.080 other client that's on the subnet for example on our computer on the cart we have a client that's 00:49:16.080 --> 00:49:21.840 picking up the ndi stream being sent from the test bench which then runs our experiment with depth 00:49:21.840 --> 00:49:26.640 estimation and depth estimation is the task of measuring the distance of each pixel relative 00:49:26.640 --> 00:49:31.920 to the camera depth estimation could be used to instruct a player to walk down a hallway by 00:49:31.920 --> 00:49:36.240 chasing the darkest point on the screen we are also experimenting with other techniques such as 00:49:36.240 --> 00:49:42.720 optical character recognition and object detection on a video stream the ndi encoder that we're using 00:49:42.720 --> 00:49:50.320 is an n6 from kill of you which has an HDMI in HDMI out and then your Ethernet a particular nice to 00:49:50.320 --> 00:49:55.680 have on this unit is the small display on the front which allows you to see the activity of the 00:49:55.680 --> 00:50:02.400 unit as well as the ip and what video stream it's sending out okay so all of our benchmarks have 00:50:02.400 --> 00:50:10.080 finished let's take a look at the results as you can see here it actually says we've ran 00:50:10.080 --> 00:50:16.160 zero zero for dota 2 so we didn't actually get any valid runs so we're gonna actually have to do some 00:50:16.160 --> 00:50:24.240 troubleshooting figure out why so that's the thing is that markbench allows us to identify some very 00:50:24.240 --> 00:50:30.160 early on possible bugs that might occur with the run and it'll usually terminate so if we scroll up 00:50:30.160 --> 00:50:36.400 here you can actually see we get to a point where it says test failed and it's for this run here so 00:50:36.400 --> 00:50:41.520 it says game didn't start in time check settings and try again so this is something that we've 00:50:41.520 --> 00:50:46.640 programmed into some of our logging so we're gonna have to actually go into our dota 2 settings and 00:50:46.640 --> 00:50:52.080 it could be that maybe something got reset when we swapped cables or something like that so we're 00:50:52.080 --> 00:50:57.760 gonna take a look at that and get it set back up to the correct settings and try running it again 00:50:58.720 --> 00:51:03.760 so we're in dota 2 let's take a quick look at the setting and see what might have went wrong here 00:51:04.640 --> 00:51:13.600 so we go to video ah there's the problem it changed itself back to 4k it's a good chance that 00:51:13.600 --> 00:51:18.160 while we were swapping some cables around or something like that when we were doing the initial 00:51:18.160 --> 00:51:24.320 setup I probably swapped itself back to native resolution and so it caused our markbench runs 00:51:24.320 --> 00:51:32.880 to fail so let's change that back and then we'll rerun it here the upside of this is is we don't 00:51:32.880 --> 00:51:39.360 have to run every test we just ran again we can actually unmark everything else and just specifically 00:51:39.360 --> 00:51:44.640 target dota 2 now so let's go ahead and get that started and hopefully we'll come back with some 00:51:44.640 --> 00:51:56.080 actual runs 00:52:06.400 --> 00:52:12.160 so looks like we got three runs done for dota 2 now so we can actually upload all this data to our 00:52:12.160 --> 00:52:18.800 data ingress server and start trimming so let's get that done now we've uploaded the data to our 00:52:18.800 --> 00:52:25.360 server we can pull it up on grafana which is our data visualization dashboard from here we can pull 00:52:25.360 --> 00:52:32.240 up all of projects that we've run any sessions that have occurred from basically any of the uploads 00:52:32.960 --> 00:52:39.920 so on grafana itself we have the ability to check our projects as well as in this specific 00:52:39.920 --> 00:52:47.680 dashboard we're looking at GPUs so for our current project it lists the 7900 XTX as well as the 4080 00:52:48.640 --> 00:52:54.000 for tests it lists all the different tests that we've run for a given project and then there's 00:52:54.000 --> 00:52:59.120 parameters so in this case we only specify games 1080 so that's the only one that shows up on the list 00:53:00.400 --> 00:53:07.120 and then there is the option for the bench which essentially that takes the actual PC name when 00:53:08.080 --> 00:53:15.680 when markbench logs it to help us identify which system we actually ran on and then lastly is the 00:53:15.680 --> 00:53:22.640 tag which is blank because we didn't label any tags for anything that we needed so from here we 00:53:22.640 --> 00:53:28.720 can pull up any of the sessions so let's take a look at dota 2 first here so as you can see here 00:53:28.720 --> 00:53:32.960 it's kind of a little bit of a mess if you don't know what you're looking at here but effectively 00:53:32.960 --> 00:53:38.720 this is the beginning load this is the main menu and then we have our little load screen 00:53:39.280 --> 00:53:45.520 and then we have that initial bit before the actual benchmark occurs the actual bits that we need 00:53:45.520 --> 00:53:54.160 are from this valley to this valley and then this last bit here is when it goes back to the main 00:53:54.160 --> 00:54:02.480 menu so that's how we how we kind of know which game it's all specific you kind of have to know 00:54:02.480 --> 00:54:07.440 what you're looking for at the beginning of like determining oh which part do I actually 00:54:07.440 --> 00:54:14.480 need to use so what we all all we have to do at this point is just drag from there to there 00:54:15.600 --> 00:54:22.800 apply the correct time range sometimes it skews it by a second if we do that that gives us our run 00:54:23.760 --> 00:54:30.560 and then basically I just do that for all the sessions that we have and then once we finish 00:54:30.560 --> 00:54:37.760 trimming them we take the data and we put it into our trim session tool on the harbour 00:54:38.720 --> 00:54:46.320 LTT Labs page and that basically gives us a graph ideally we want all three sessions to be 00:54:47.360 --> 00:54:54.480 basically almost the same however as I outlined previously red deads a little different there's 00:54:54.480 --> 00:55:01.280 a bit of rng that occurs during each run so they don't look the exact same on here but the end 00:55:01.280 --> 00:55:07.600 result of the averages in the first should be the same or within that one to two percent so at this 00:55:07.600 --> 00:55:12.960 point all we gotta do now is trim all the data so we can send it over to sammy and she can start 00:55:12.960 --> 00:55:18.240 working on getting those graphs created so let's get to it once the game has finished and the data 00:55:18.240 --> 00:55:23.120 is uploaded to the server we need to mark an in and out on the benchmark data to eliminate the 00:55:23.200 --> 00:55:28.080 menus and any loading screens from when we calculate the FPS currently john's doing this 00:55:28.080 --> 00:55:33.040 manually in grafana by literally looking at the pattern of the frame times in the FPS we're trying 00:55:33.040 --> 00:55:37.760 to help them out by having machine learning doing something similar by training a model based on 00:55:37.760 --> 00:55:44.000 the behavior of the GPU its temperature and core clocks and memory usage as well as the pattern 00:55:44.000 --> 00:55:50.000 and frame times to detect the in and out of a benchmark however that still can be faulty so 00:55:50.000 --> 00:55:54.880 we're looking at eliminating it all together by using computer vision from the ndi streams as you 00:55:54.880 --> 00:56:00.560 saw earlier to mark an in and out very precisely as it's close to the frame when the game first 00:56:00.560 --> 00:56:06.320 loads as we can find we're also looking to apply machine learning and models trained on GPU behavior 00:56:06.320 --> 00:56:12.960 to identify erroneous runs or bad runs based on historical data now that we've finished trimming 00:56:12.960 --> 00:56:19.600 all the sessions grafana will actually spit out a list of all the different trim session ids 00:56:20.000 --> 00:56:25.360 so we can use them to generate our game test report they might ask us how do you know these 00:56:25.360 --> 00:56:31.520 are good trim sessions maybe your settings were bad or maybe you know it was just every run was 00:56:31.520 --> 00:56:38.240 terrible that's where our other dashboard comes in redash this basically shows historical trim 00:56:38.240 --> 00:56:43.920 data that we can use to help verify that these runs are in fact the runs that we need this is 00:56:43.920 --> 00:56:48.640 something that gets generated over time if we have a game that we haven't done a whole lot of 00:56:48.640 --> 00:56:55.520 trimming on it will be there will be less numbers to be able to verify against but ideally we want 00:56:55.520 --> 00:57:02.240 the trend to be the same or similar so in this case if we're looking at cyberpunk on a 4080 00:57:03.200 --> 00:57:14.320 on 1080p we're getting 190 average ish to 180 average ish and 154 to 153 for a 1% 00:57:15.120 --> 00:57:22.800 so what we can do now is we can use our we can compare our trimmed sessions versus historical 00:57:22.800 --> 00:57:27.600 data to verify that it's all correct so what we're going to do is we're going to go on to our 00:57:27.600 --> 00:57:31.760 harbor Labs tools again and we're going to go through and we're going to start creating an 00:57:31.760 --> 00:57:37.120 actual test report so what we do is we want GPU by component we're not going to include a line 00:57:37.120 --> 00:57:44.000 chart because we were just looking at those so we hit create now what we do is we take all of these 00:57:44.080 --> 00:57:53.840 session IDs and we hit add let me do it for the other card and so what that does now is it actually 00:57:53.840 --> 00:58:00.320 generates a report for us to be able to see visually both with a bar graph as well as just the numbers 00:58:01.600 --> 00:58:08.000 what our actual results are so if we're looking here at cyberpunk 2077 on 1080p 00:58:08.880 --> 00:58:16.480 the 4080 for these three runs that we just did has an average of 193 and a 1% low of 154 00:58:17.360 --> 00:58:23.200 so we're actually above our previous average now these numbers were with a reference card 00:58:23.200 --> 00:58:29.440 obviously the accelerates a little bit over a clock so it stands for reason that 193 seems 00:58:29.440 --> 00:58:37.200 accurate and so we can keep scrolling down so taking a look at dota 2 we can see that the 00:58:37.200 --> 00:58:42.880 numbers appear to line up with what i have experienced in the past we don't have a whole 00:58:42.880 --> 00:58:48.320 lot of historical data on it currently because it is one of our newer harnesses that we actually 00:58:48.320 --> 00:58:53.840 have been developing and i've kind of just been beta testing but based on my personal experience 00:58:53.840 --> 00:59:00.880 with it these numbers do appear to be accurate okay so lastly let's take a look at red dead 2 here 00:59:01.680 --> 00:59:07.680 so now we've taken a look at some of the numbers 00:59:08.720 --> 00:59:15.040 we can publish this report and get it sent off to sammy to start generating the graphs 00:59:15.760 --> 00:59:20.640 so what we need to do first is put the report name which is usually the project slug 00:59:21.200 --> 00:59:36.160 um so 40 versus 7900 x tx 2023 results in some cases if we decide to have separate reports based 00:59:36.160 --> 00:59:42.640 on like ray tracing versus non ray tracing stuff like that even though those headers of like the 00:59:42.640 --> 00:59:50.320 games 1080 um help us filter that if in some cases we just want to look at those bar graphs separately 00:59:50.320 --> 00:59:58.240 and the data that they've generated sometimes we will append uh rt results or dlss results or 01:00:04.960 --> 01:00:11.200 it's just a description of effectively what the reports based on so we're just going to do results 01:00:11.200 --> 01:00:23.760 for 40 80 dash vs dash 7900 x tx dash 203 person then for a reference component 01:00:24.320 --> 01:00:32.400 basically what this does is it generates a relative percentage um based on the data so 01:00:32.400 --> 01:00:42.560 effectively like oh we can see that the 40 80 is negative 1.62 percent in performance versus 01:00:42.560 --> 01:00:48.640 the 7900 x tx we can choose whatever component we want it's helpful for when we are looking at 01:00:48.640 --> 01:00:53.760 our big GPU reviews because we can actually be like okay we want to focus on this component it's 01:00:53.760 --> 01:01:01.120 not the top of the stack but like for example the rx 7600 we want to take a look at and see 01:01:01.120 --> 01:01:06.640 how it compares to everything else and it'll allow us to to basically find these relatives a 01:01:06.640 --> 01:01:13.840 lot easier than doing the math ourselves so it's very helpful in that regard um so let's go ahead 01:01:13.840 --> 01:01:20.800 and we're going to click publish now this will create a report so if we actually go over here 01:01:20.800 --> 01:01:27.600 we can see we actually have the 40 80 versus 7900 x tx results we click on that that basically gives 01:01:27.680 --> 01:01:33.120 us the same report um that we were just looking at so i'm just going to copy the url here 01:01:34.560 --> 01:01:41.360 into an excel sheet that i was working on go ahead and save this we'll get this sent off to her 01:01:41.360 --> 01:01:46.720 and then she can start creating our graphs hi my name is sammy i'm the data visualization specialist 01:01:47.680 --> 01:01:52.800 for the lab um i started a few months ago and ever since i started we've made some changes to 01:01:52.800 --> 01:01:58.720 the graphs um previously they were done by the writing team um but since then i've taken over 01:01:58.720 --> 01:02:04.800 that role um and yeah i'm here to walk you through uh some of the design choices that we made as well 01:02:04.800 --> 01:02:11.040 as um the process by which we make these graphs for our videos previously the graphs that we had 01:02:11.040 --> 01:02:18.240 had a lot of limitations in design choices um in terms of placement or colors or fonts that we used 01:02:18.240 --> 01:02:22.400 mostly because we were making them in excel nowadays we're making them in a combination 01:02:22.480 --> 01:02:28.960 of python and uh doby illustrator as well as adobe illustrator plugins mainly um one plugin that's 01:02:28.960 --> 01:02:34.240 specialized for graphing uh called datlion datlion allows us to make customizations that we weren't 01:02:34.240 --> 01:02:40.320 able to do previously um so one of the first things i did uh was use the style guide that was 01:02:40.320 --> 01:02:46.160 developed by our creator warehouse graphic designers um they had kind of set out a palette as well as 01:02:46.160 --> 01:02:51.040 some fonts that i could use and toy around with i tried to maintain a design design philosophy 01:02:51.040 --> 01:02:56.720 that focused on eligibility um so as we've kind of gone through multiple iterations of the graphs 01:02:56.720 --> 01:03:01.360 since i started i've made changes to emphasize on that based off a lot of feedback that we've 01:03:01.360 --> 01:03:07.120 gotten from our viewers in terms of design aesthetic um one of the things that i really like to focus on 01:03:07.120 --> 01:03:14.960 was an almost laboratorial or chalkboard kind of vibe um we wanted to evoke something that was very 01:03:14.960 --> 01:03:22.160 similar to the nasa kind of retro space age kind of look we also incorporated a more modern twist 01:03:22.160 --> 01:03:28.640 um inspired by the cyberpunk genre as well as more sci-fi films that have come out in recent years 01:03:28.640 --> 01:03:34.560 we've focused on using as well a palette that is uh as accessible as possible for example the choices 01:03:34.560 --> 01:03:40.160 um of which colors come first in our bar graphs we chose between colors that were very contrasting 01:03:40.160 --> 01:03:46.080 another thing we did was uh we put the colors into um different accessibility tools to kind of 01:03:46.080 --> 01:03:52.880 measure how visible our color palette would be for people with different types of color blindness 01:03:52.880 --> 01:03:58.400 um and we've essentially developed a order and a palette that is as accessible as possible across 01:03:58.400 --> 01:04:04.880 as many different types of um color blindness as possible in order to make our graphs legible 01:04:04.880 --> 01:04:08.960 so this is an example of one of the game graph templates that i've developed um one of the first 01:04:08.960 --> 01:04:13.680 things that i noticed when i came in was that the previous graphs um their title didn't really jump 01:04:13.680 --> 01:04:18.240 out at you one of the things i really wanted to do is make use of like kind of like the very like 01:04:18.240 --> 01:04:24.640 stampy Labs font that we have to kind of grab attention um for me one of the things that i want 01:04:24.640 --> 01:04:30.880 to focus on is like what is what am i looking at and number two what is this me and so the first 01:04:30.880 --> 01:04:37.520 part what am i looking at is um kind of answered immediately by this very bold and very loud um 01:04:38.480 --> 01:04:44.960 title or header um and so we've added some design elements sometimes they'll be added in especially 01:04:44.960 --> 01:04:49.440 if a graph is you know maybe maybe a little more bare if they're only like two or three 01:04:49.440 --> 01:04:55.920 things that we're comparing across but you know if they're 12 13 CPUs GPUs whatever that we're 01:04:55.920 --> 01:05:01.920 comparing across then obviously we won't have no space for that we then have the bars themselves 01:05:02.000 --> 01:05:09.680 um the bars themselves are sorted now by uh best to worst um we'll still keep the higher 01:05:09.680 --> 01:05:16.560 is better um in the subtitle of the graph however um we just want to make it like at a glance more 01:05:16.560 --> 01:05:23.680 obvious um and another design element that the editors have helped me incorporate was um a marker 01:05:23.680 --> 01:05:27.680 that kind of pulses right next to the GPU that we're talking about previously one of our concerns 01:05:27.680 --> 01:05:32.720 and the reason why we weren't able to uh sort by best to worst was because we were worried that if 01:05:32.720 --> 01:05:38.160 we swapped around the order of every single GPU it'd be hard to follow oh which one is where um and 01:05:38.160 --> 01:05:43.120 so that kind of solved our problem and ever since then we've kind of adopted this philosophy of sorting 01:05:43.120 --> 01:05:47.840 by best to worst as best as possible we've kept the grid lines kind of sparse because we want to 01:05:47.840 --> 01:05:53.760 make sure that the graph isn't overcrowded with different design elements um and another design 01:05:53.760 --> 01:06:00.240 choice we've made particularly with the game graphs and the FPS um graphs is that we've put 01:06:00.240 --> 01:06:05.200 one percent low on top and we've made it a bolder color than the average so we've decided that one 01:06:05.200 --> 01:06:09.680 percent lows are more indicative of overall gaming performance because average values can get pulled 01:06:09.680 --> 01:06:15.520 up by um high FPS anomalies um that might happen over the course of a game when the one percent 01:06:15.520 --> 01:06:19.280 lows are really low uh the gameplay ends up appearing as really choppy and so that's one of 01:06:19.280 --> 01:06:23.200 the reasons why we think that actually the one percent lows the statistic or the data point 01:06:23.200 --> 01:06:28.400 that we really want to highlight instead and so one thing that's been added since we've made a few 01:06:28.400 --> 01:06:34.400 changes around the lab is that we've added this uh project slug that follows a project from the 01:06:34.400 --> 01:06:39.280 beginning to the end and another thing that we've added is also this version number um so we will 01:06:39.280 --> 01:06:45.280 change and update the version number every time of new set of graphs is cut um and that will also 01:06:45.280 --> 01:06:52.080 ensure that we have the most updated version of graphs in our video and finally we have our 01:06:52.080 --> 01:06:58.320 Labs logo in the bottom bottom right uh this Labs logo essentially tells viewers that this graph was 01:06:58.320 --> 01:07:05.280 developed by Labs with Labs data so now we're gonna get started with our graphs um i've already gotten 01:07:05.280 --> 01:07:11.840 a uh spreadsheet from adam with the list of graphs that he wants me to do uh three of them being 01:07:12.720 --> 01:07:19.680 the game graphs so one for each game and then uh the blender benchmark graph and as you can see here 01:07:20.320 --> 01:07:29.840 it says um cyberpunk 2020 2077 uh red dead redemption 2 dota 2 blender 3.6.0 and higher 01:07:29.840 --> 01:07:37.600 is better and includes all of the values for the presets as well as the resolution x-axis label 01:07:37.600 --> 01:07:44.960 project label version number and all of that so that allows me to know what exactly writers want 01:07:45.040 --> 01:07:52.480 included in their video and what data they need as well and there's also room for writers to add 01:07:52.480 --> 01:07:58.640 descriptions like if they want to include some kind of like asterisks at the bottom with notes or if 01:07:58.640 --> 01:08:04.240 they want to cut out a certain GPU or CPU or if they want like another version of the exact same 01:08:04.240 --> 01:08:10.400 graph with extra information or less information included we're just gonna open up a new file 01:08:10.880 --> 01:08:20.240 and that will be one layer for each graph as well and i will copy and paste um the layers from our 01:08:20.240 --> 01:08:26.960 template so for our first three graphs it's going to be a game graph and so for graph number one 01:08:26.960 --> 01:08:37.920 just gonna paste it in and align it you can see that there are already numbers in the template 01:08:37.920 --> 01:08:44.400 that's just to help um serve as a placeholder so that i know what graph is which uh that data 01:08:44.400 --> 01:08:51.120 will be scrubbed later and then the final graph is our blend blender benchmark so i'm just going 01:08:51.120 --> 01:09:02.080 to copy our three series benchmark graph now i'm going to import all of these project labels into 01:09:02.080 --> 01:09:08.800 illustrator using visual studio code and a python script that i've written so here i'm just gonna 01:09:08.800 --> 01:09:14.560 load in the packages that i need and here in chart names i'm just gonna pull up my file and file 01:09:14.560 --> 01:09:23.520 explorer and copy the path to the file and paste it into here um and we'll see here one two three 01:09:23.520 --> 01:09:34.720 four graphs um with all the graph titles subtitles text and uh project label version number and now 01:09:34.720 --> 01:09:38.560 we know that there are four graphs so i'm just gonna make sure that this number here is four if there 01:09:38.560 --> 01:09:44.480 were 50 i'd do 50 since there are four i'm just going to type in four um and there are no y-axis 01:09:44.480 --> 01:09:50.080 labels so we're just going to skip over that and then press enter and as we press enter we can see 01:09:50.080 --> 01:09:55.680 that it'll man automatically change everything so we went from you know all the placeholder text to 01:09:56.400 --> 01:09:59.600 all of this text and let's just double check that everything is right 01:10:05.360 --> 01:10:11.360 i'm going to open up harbour to look up the game test report and i'm going to download it as the csv 01:10:12.000 --> 01:10:19.680 we'll see here that the labels for the gpus are a little long um we don't really need the NVIDIA 01:10:19.760 --> 01:10:23.440 gforce part so i'm just going to cut that out it's going to be rtx 4080 01:10:28.880 --> 01:10:34.320 i'm just going to select the graph that i'm working on at the time let's make this full 01:10:35.280 --> 01:10:40.720 screen select the graph and i'm just going to delete everything that i have and make sure everything 01:10:40.720 --> 01:10:47.360 clears yep it's saying that i have no items at all to graph and i'm just going to paste in what i have 01:10:47.360 --> 01:10:56.240 and so here we see um because with our game graphs we like to sort uh highest to lowest because higher 01:10:56.240 --> 01:11:01.680 is better um and it does that automatically um the legend will stay the same uh one percent 01:11:01.680 --> 01:11:09.040 low on average result so this is a benchmark graph it has three series i had a um three series template 01:11:09.120 --> 01:11:16.720 ready 01:11:26.000 --> 01:11:31.040 okay i'm done now i've put in the data for all of the graphs and then we're just going to make 01:11:31.040 --> 01:11:37.680 sure that all the data is correct so what i'm going to do is um look over um the data based off of 01:11:37.680 --> 01:11:45.360 the spreadsheet that i pulled a cyberpunk for example for the 4080 is um 154 193 01:11:54.000 --> 01:12:02.400 okay so now all the data is included um we're just going to change up the uh x axis a little bit 01:12:02.400 --> 01:12:06.640 so you can see here that there are one two three four five six seven 01:12:07.360 --> 01:12:11.920 different lines and that's a little too much for my taste i feel like that's a little crowded 01:12:12.720 --> 01:12:19.440 and by default our graphs have an interval of 30 we prefer to have it at 60 but sometimes when the 01:12:19.440 --> 01:12:24.320 graph results are really low like having 60 doesn't really make sense so i'm just going to change it 01:12:24.320 --> 01:12:31.360 here to 60 so that it's a little less crowded and we can see 60 120 and 180 um with this one i'm going 01:12:31.440 --> 01:12:34.080 to do about the same because it's kind of within the same range 01:12:39.520 --> 01:12:42.960 i'm going to pull up go back to illustrator instead of exporting everything and having 01:12:44.160 --> 01:12:47.840 you know people look over it and then give me feedback individually what i'm going to do is 01:12:47.840 --> 01:12:53.040 i'm going to um share it for review it what it's going to do is create essentially like a google 01:12:53.040 --> 01:12:58.480 doc where people can add comments so i'm going to um set it to anyone with the link can comment and 01:12:58.480 --> 01:13:07.200 copy it and i'm going to uh send it off to adam and john so i'm just going to show everybody 01:13:07.200 --> 01:13:12.560 what it looks like and this is essentially what it looks like for john and adam um they're just 01:13:12.560 --> 01:13:16.960 going to look over it for multiple things they're going to make sure that all the labels are correct 01:13:16.960 --> 01:13:21.760 um and they're going to make sure that the uh data is correct it matches up with the right 01:13:21.760 --> 01:13:28.720 card and it matches up with the right um game or test uh here if they do you find something for 01:13:28.720 --> 01:13:37.040 example um they can leave a comment so they can highlight this and uh they can just write a comment 01:13:37.040 --> 01:13:50.480 and say uh for this one this data is actually for dota 2 not cyberpunk theoretically if i made 01:13:50.560 --> 01:13:59.920 that mistake okay so now i've sent off my um graphs to adam and john and they've reviewed it and adam 01:13:59.920 --> 01:14:06.640 doesn't have any comments and john has a question about um the specifics of like when we're going to 01:14:06.640 --> 01:14:15.600 use commas for the graphing numbers and so um you know i can reply to him and i'll say we're adding 01:14:15.600 --> 01:14:25.280 in commas whenever uh numbers are above a thousand if we had any changes to make 01:14:25.280 --> 01:14:30.160 then i would go in i would make the changes and then i'd just update the content um since we 01:14:30.160 --> 01:14:33.520 don't have any changes to make what i'm going to do is prepare for export 01:14:36.880 --> 01:14:42.240 and adam will be the one to send it off to the editing team and we're going to put it in the video 01:14:42.240 --> 01:14:48.080 yeah that's pretty much it on my end okay so at this point i received the graphs from sammy 01:14:48.080 --> 01:14:53.760 so what i do now along with adam as we go through when we kind of take a look at the graphs he does 01:14:53.760 --> 01:15:00.960 sort of a subjective look on the presentation of the graphs i do more of a data analysis make sure 01:15:00.960 --> 01:15:05.680 everything was inputted correctly maybe something got copied into the wrong line or something like that 01:15:06.480 --> 01:15:11.760 so basically i just have to pour over the data here make sure everything looks right 01:15:11.760 --> 01:15:18.160 so we just quickly scan through blender here 50 48 20 10 01:15:24.320 --> 01:15:30.000 this blender looks good and then if we hop back over to the the game test report 01:15:30.720 --> 01:15:35.920 we can quickly look at all of them up here so if we go to dota 2 01:15:42.000 --> 01:15:48.800 so this point now i've basically verified that all the data looks correct um assuming that adam 01:15:48.800 --> 01:15:54.640 doesn't have any other issues then we go ahead and get these all set up sent off to the editors 01:15:54.640 --> 01:16:00.560 and that's when they start cutting the video after all of that we then go into post production 01:16:00.560 --> 01:16:06.240 where we provide the assets and the footage to the editor we have several more steps of qc that 01:16:06.240 --> 01:16:11.040 occurred during this um first off you have the editor and writer review where the writer sits 01:16:11.040 --> 01:16:15.920 down with the editor watches what the editor has created and um points out any sort of fixes 01:16:15.920 --> 01:16:21.280 whether they're stylistic or content fixes that need to be done then once those fixes are implemented 01:16:21.280 --> 01:16:27.360 they send a pass to Labs Labs looks over it Labs points out any fixes typically are their fixes 01:16:27.360 --> 01:16:31.200 are going to be more focused on the correctness and the data side of things and also maybe kind of 01:16:31.200 --> 01:16:36.080 small nitpicks less so about the stylistic side of things but we still make sure that we provide 01:16:36.080 --> 01:16:42.160 Labs input into that because sometimes you can lose the exact meaning of stuff just through like a 01:16:42.160 --> 01:16:46.160 couple of different wardings and we want to make sure that that's always communicated correctly 01:16:46.880 --> 01:16:52.160 then once Labs reviews it the editor implements all those fixes there is another review with the 01:16:52.160 --> 01:16:57.760 writer and Labs to make sure that all the fixes have been implemented correctly once that is done 01:16:57.760 --> 01:17:02.960 we then upload it to Floatplane for our ecc squad to take a look at the whole point of ecc squad 01:17:02.960 --> 01:17:08.080 is to get like a ton of knowledgeable people to have eyes and ears on exactly what we've made 01:17:08.080 --> 01:17:11.360 because they'll notice the smallest kind of details when you have like a dozen sets of eyes you'll 01:17:11.360 --> 01:17:17.440 just find so many small things and they're always great and once we've done that we then 01:17:17.440 --> 01:17:24.080 review the suggestions that the ecc squad has made and if they need to be implemented we do that again 01:17:24.720 --> 01:17:30.320 we also check with Labs to make sure that those corrections are in fact correct implement them 01:17:30.320 --> 01:17:35.200 make sure they're implemented correctly and then finally after all of that a video gets released 01:17:41.360 --> 01:17:44.720 update your BIOS kids