{"video_id":"fp_fZgBrnc5qp","title":"How LTT Labs Tests Products","channel":"FP Exclusives","show":"FP Exclusives","published_at":"2023-10-12T21:38:00.027Z","duration_s":9328,"segments":[{"start_s":0.0,"end_s":4.5600000000000005,"text":"Welcome to the Labs Process Video. We're going to give you a step-by-step breakdown of everything","speaker":null,"is_sponsor":0},{"start_s":4.5600000000000005,"end_s":10.0,"text":"we do when we test a product for a video, all the way from ideation to the final results.","speaker":null,"is_sponsor":0},{"start_s":10.0,"end_s":15.68,"text":"Every test starts with an idea. For example, it's been a long time since we've reviewed the 7900XTX","speaker":null,"is_sponsor":0},{"start_s":15.68,"end_s":21.44,"text":"and the 4080, and we want to know how do these perform today? The first step is filling out our","speaker":null,"is_sponsor":0},{"start_s":21.44,"end_s":26.16,"text":"test request form. As much as it would be nice to just go down and ask Labs to test these cards,","speaker":null,"is_sponsor":0},{"start_s":26.16,"end_s":30.88,"text":"they're not going to do anything unless I provide them with a filled out test request form. There","speaker":null,"is_sponsor":0},{"start_s":30.88,"end_s":35.68,"text":"is no testing without requesting. The reason we have to do a request form is that it allows Labs","speaker":null,"is_sponsor":0},{"start_s":35.68,"end_s":40.96,"text":"to track the project. If we don't give them a request form, they won't do the testing because","speaker":null,"is_sponsor":0},{"start_s":40.96,"end_s":44.88,"text":"that's how we waste people's time. In here we put like the name of the project, so we'll say like","speaker":null,"is_sponsor":0},{"start_s":44.88,"end_s":51.12,"text":"4080 versus 7900XTX. Then we'll put a link to the Trello just again for project tracking. And then","speaker":null,"is_sponsor":0},{"start_s":51.12,"end_s":56.16,"text":"here's the first kind of important answer. This is an important delineation. Is it exploratory","speaker":null,"is_sponsor":0},{"start_s":56.16,"end_s":61.68,"text":"testing or project testing? So exploratory testing is going like, oh, we don't know if this is going","speaker":null,"is_sponsor":0},{"start_s":61.68,"end_s":65.84,"text":"to be an interesting story. So we need to test to see if the effect that we are talking about","speaker":null,"is_sponsor":0},{"start_s":65.84,"end_s":70.4,"text":"is significant. Like how much does a browser extension slow down your browser? Maybe it's","speaker":null,"is_sponsor":0},{"start_s":70.4,"end_s":75.36,"text":"not enough to be an interesting video, but maybe there's something there versus a project test,","speaker":null,"is_sponsor":0},{"start_s":75.36,"end_s":80.96,"text":"which is like, it's a new GPE review. We know that we're going to test it. We don't need to explore.","speaker":null,"is_sponsor":0},{"start_s":81.6,"end_s":86.56,"text":"In this scenario, we're doing a project test. Then we give them a due date, so a project timeline.","speaker":null,"is_sponsor":0},{"start_s":86.56,"end_s":90.4,"text":"We can either do a hard deadline or a soft deadline. A soft deadline is kind of like,","speaker":null,"is_sponsor":0},{"start_s":90.4,"end_s":94.72,"text":"if something else of higher priority comes in, Labs can push it back a little bit because maybe","speaker":null,"is_sponsor":0},{"start_s":94.72,"end_s":98.96,"text":"it's a video that doesn't really need to come out on a specific date. A hard deadline is really","speaker":null,"is_sponsor":0},{"start_s":98.96,"end_s":103.36,"text":"important. If it's something that's very timely, like maybe it's a sponsored project or like an","speaker":null,"is_sponsor":0},{"start_s":103.36,"end_s":108.24,"text":"embargo GPE review, then we give them a list of all of the components that we want tested in the","speaker":null,"is_sponsor":0},{"start_s":108.24,"end_s":113.2,"text":"video. For this scenario, we're just going to do the two cards, a 4080 Accelerate and a 7900 XTX.","speaker":null,"is_sponsor":0},{"start_s":115.84,"end_s":120.56,"text":"Then we just let them know if we have them here or if they're coming. Basically it allows them to","speaker":null,"is_sponsor":0},{"start_s":120.56,"end_s":124.96,"text":"be like, where do we need to listen for the next step? Is it like the writer or do we need to like","speaker":null,"is_sponsor":0},{"start_s":124.96,"end_s":129.44,"text":"keep an eye on procurement or something like that? Sometimes we also might want to like","speaker":null,"is_sponsor":0},{"start_s":129.44,"end_s":134.32,"text":"talk to Labs about any sort of materials or components that we need to actually get.","speaker":null,"is_sponsor":0},{"start_s":134.32,"end_s":138.88,"text":"And that'll usually happen during like a kickoff meeting. Then we have to clearly state what our","speaker":null,"is_sponsor":0},{"start_s":138.88,"end_s":143.12,"text":"questions are. It's really important to do this and like the more nailed down our questions are,","speaker":null,"is_sponsor":0},{"start_s":144.24,"end_s":148.0,"text":"the faster the project's going to go and the faster the kickoff meeting will go.","speaker":null,"is_sponsor":0},{"start_s":148.0,"end_s":151.12,"text":"There have been times where I've stepped into meetings where I did not have a solid question","speaker":null,"is_sponsor":0},{"start_s":151.12,"end_s":155.28,"text":"and then it takes a long time, waste everyone's time trying to figure out what the heck I'm doing.","speaker":null,"is_sponsor":0},{"start_s":155.28,"end_s":163.28,"text":"In this scenario, we're just trying to see how well does AMD's 7900 XTX compared to the 4080","speaker":null,"is_sponsor":0},{"start_s":165.28,"end_s":169.84,"text":"is the 4080 really we're going to be worth the extra money. And then we can also select specific","speaker":null,"is_sponsor":0},{"start_s":169.84,"end_s":174.96,"text":"tests. We have a whole test portfolio that we can usually run like where we're just run through","speaker":null,"is_sponsor":0},{"start_s":174.96,"end_s":179.04,"text":"a gauntlet of games and productivity benchmarks, but maybe I just need to select few or there's","speaker":null,"is_sponsor":0},{"start_s":179.04,"end_s":183.92,"text":"some that I absolutely must have. In this case, we don't need to run these through like a million","speaker":null,"is_sponsor":0},{"start_s":183.92,"end_s":191.68,"text":"gaming benchmarks, just a few. We're going to do Cyberpunk 2077, Dota 2, Red Dead Redemption 2,","speaker":null,"is_sponsor":0},{"start_s":191.68,"end_s":195.84,"text":"and they're all going to be at 1080. The next part is like any sort of notable","speaker":null,"is_sponsor":0},{"start_s":195.84,"end_s":199.84,"text":"test boundary conditions. So say we wanted to have something that's like out of the","speaker":null,"is_sponsor":0},{"start_s":199.84,"end_s":204.88,"text":"ordinary, like, you know, maybe we need old drivers, like we want the drivers that were","speaker":null,"is_sponsor":0},{"start_s":204.88,"end_s":209.2,"text":"used when it came out to compare to the drivers that are out today. That's something we can put","speaker":null,"is_sponsor":0},{"start_s":209.2,"end_s":214.08,"text":"in here. And then for any super general notes that might not really have another place to go,","speaker":null,"is_sponsor":0},{"start_s":214.08,"end_s":217.2,"text":"there's a final form just at the bottom where we can fill out and be like,","speaker":null,"is_sponsor":0},{"start_s":218.16,"end_s":222.48,"text":"I don't know, maybe like, you guys want to hang out sometime too? I don't know,","speaker":null,"is_sponsor":0},{"start_s":222.48,"end_s":226.64,"text":"you can ask them whatever they need to submit it and it goes off to them.","speaker":null,"is_sponsor":0},{"start_s":226.64,"end_s":234.64,"text":"Hi, it's Gary. We just received a test request from Adam. Normally this goes to Sharon, but Sharon","speaker":null,"is_sponsor":0},{"start_s":234.64,"end_s":242.56,"text":"is busy doing some other work today, so I'm handling this request. Once it comes into our mailbox,","speaker":null,"is_sponsor":0},{"start_s":243.36,"end_s":252.64,"text":"we simply click on it, brings up the lab's data request. This one's obviously from Adam.","speaker":null,"is_sponsor":0},{"start_s":253.76,"end_s":257.76,"text":"We're testing the 4080 against the 7900XTX.","speaker":null,"is_sponsor":0},{"start_s":264.88,"end_s":269.2,"text":"And then he's actually filled out some specific tests that he wants.","speaker":null,"is_sponsor":0},{"start_s":269.2,"end_s":276.48,"text":"All of these, Cyberpunk, Red Dead Redemption, Blender and Dota are part of our standard test","speaker":null,"is_sponsor":0},{"start_s":276.48,"end_s":282.72,"text":"suite, so that makes it a little bit easier. He doesn't want anything special on it, just straight","speaker":null,"is_sponsor":0},{"start_s":282.72,"end_s":291.04,"text":"up testing. And apparently he appreciates all of us, so thank you, Adam. With that, we'll go","speaker":null,"is_sponsor":0},{"start_s":291.04,"end_s":297.92,"text":"over to the assignment form. This is where we keep track of who's assigned the project,","speaker":null,"is_sponsor":0},{"start_s":299.04,"end_s":304.32,"text":"and also if it's just been assigned or if it's already in progress. For this one,","speaker":null,"is_sponsor":0},{"start_s":305.68,"end_s":312.0,"text":"we'll go ahead and mark it in progress as John's going to start it momentarily.","speaker":null,"is_sponsor":0},{"start_s":313.44,"end_s":320.64,"text":"And yeah, we're giving it to John. And with that, I'm going to hand it over to John. Thank you.","speaker":null,"is_sponsor":0},{"start_s":321.44,"end_s":325.52,"text":"Now we have our kickoff meeting. Basically what this does is that it allows us to","speaker":null,"is_sponsor":0},{"start_s":325.52,"end_s":330.16,"text":"talk with Labs, make sure that our test plan is all agreed upon, and that we are all on the same","speaker":null,"is_sponsor":0},{"start_s":330.16,"end_s":334.88,"text":"page, so that we can make sure that we're doing everything right. Labs will often bring some","speaker":null,"is_sponsor":0},{"start_s":334.88,"end_s":338.08,"text":"concerns and questions. Writers might be like, notice things in the middle of the meeting,","speaker":null,"is_sponsor":0},{"start_s":338.08,"end_s":343.28,"text":"be like, oh crap, I forgot to add that to the sheet. So this is probably one of the most","speaker":null,"is_sponsor":0},{"start_s":343.28,"end_s":346.56,"text":"important parts of the whole process, because it allows for all of us to make sure that we are","speaker":null,"is_sponsor":0},{"start_s":346.56,"end_s":352.0,"text":"aligned on our goals. So we just had the kickoff meeting for this project. Now that we've done","speaker":null,"is_sponsor":0},{"start_s":352.0,"end_s":359.36,"text":"that, we can actually go about collecting the requirements, the project requirements for the","speaker":null,"is_sponsor":0},{"start_s":359.36,"end_s":364.72,"text":"system. So we're going to be using our AMD benches today. And that's something that we're going to","speaker":null,"is_sponsor":0},{"start_s":364.72,"end_s":370.48,"text":"be using going forward for all of our GPU testing. So what we're going to do first is give you a","speaker":null,"is_sponsor":0},{"start_s":370.48,"end_s":376.48,"text":"rundown of all the hardware we use on our AMD benches, and then get into the nitty gritty of","speaker":null,"is_sponsor":0},{"start_s":377.04,"end_s":382.16,"text":"actually building it and setting it up correctly. So first thing we do is we got a","speaker":null,"is_sponsor":0},{"start_s":382.16,"end_s":388.56,"text":"CSONIC Prime PX1600 power supply. We've been using these for a little bit now. They're ATX 3.0","speaker":null,"is_sponsor":0},{"start_s":388.56,"end_s":396.48,"text":"capable. So the nice thing is we can swap quickly between using a 12-volt high power and typical","speaker":null,"is_sponsor":0},{"start_s":396.48,"end_s":401.6,"text":"8-pin power connectors for any of the GPUs that are made by like Intel or AMD without having to","speaker":null,"is_sponsor":0},{"start_s":401.6,"end_s":407.12,"text":"use any extra like pigtails and stuff like that. So that's pretty useful. And for motherboard, we","speaker":null,"is_sponsor":0},{"start_s":407.12,"end_s":415.04,"text":"are using X670E AORUS extremes. We've been using these for a little while on our AMD benches,","speaker":null,"is_sponsor":0},{"start_s":415.04,"end_s":420.72,"text":"not that we've used them in many of the videos, but going forward that is going to be our standard","speaker":null,"is_sponsor":0},{"start_s":421.28,"end_s":426.56,"text":"for our GPU testing. The reason we went with the extremes, we were using the masters for a while,","speaker":null,"is_sponsor":0},{"start_s":426.56,"end_s":431.2,"text":"and then we ended up going back to the extremes because we found that we liked the VRM a little","speaker":null,"is_sponsor":0},{"start_s":431.2,"end_s":437.44,"text":"bit further. It was a little bit more robust, and the BIOS settings we found that it could tweak just","speaker":null,"is_sponsor":0},{"start_s":437.44,"end_s":443.6,"text":"that little bit better than the master for doing some exploration testing of overclocking and stuff","speaker":null,"is_sponsor":0},{"start_s":443.6,"end_s":450.16,"text":"like that with some of the CPUs we've gotten in. As far as other brands, we have looked at ASUS for","speaker":null,"is_sponsor":0},{"start_s":450.16,"end_s":456.88,"text":"a little while. We looked at a couple of ASRock boards, but ultimately we kind of ended up on the","speaker":null,"is_sponsor":0},{"start_s":456.88,"end_s":464.48,"text":"AORUS extreme. So for CPUs, we're going to be using Ryzen 7 7800X3Ds. We actually have a video","speaker":null,"is_sponsor":0},{"start_s":464.48,"end_s":471.92,"text":"upcoming that is about AMD variance testing that we did to basically determine the three 7800X3Ds","speaker":null,"is_sponsor":0},{"start_s":471.92,"end_s":478.0,"text":"that we're actually going to go with going forward. They are the fastest one or the slowest one,","speaker":null,"is_sponsor":0},{"start_s":478.0,"end_s":484.64,"text":"the three tightest ones that we could find to effectively try to reduce any sort of variance","speaker":null,"is_sponsor":0},{"start_s":484.64,"end_s":491.36,"text":"in our FPS basically between the three benches. So we can kind of more easily verify that if we","speaker":null,"is_sponsor":0},{"start_s":491.36,"end_s":495.76,"text":"were to take this card and put them on any of the three benches, we should get the same number.","speaker":null,"is_sponsor":0},{"start_s":495.76,"end_s":503.2,"text":"For RAM, we are using G-Skill Trident Z5 new RGB. The stuff we're going with is got","speaker":null,"is_sponsor":0},{"start_s":503.44,"end_s":511.84,"text":"6,000 megatransfers per second CL30. Even though AMD has come out with new versions of","speaker":null,"is_sponsor":0},{"start_s":511.84,"end_s":518.64,"text":"AGSA that have basically unlocked the ability to use anything higher than 6,000, we still found","speaker":null,"is_sponsor":0},{"start_s":518.64,"end_s":524.08,"text":"that this is kind of the best sweet spot for AMD in terms of performance. You can get up there","speaker":null,"is_sponsor":0},{"start_s":524.08,"end_s":527.92,"text":"where you start getting your timings a little bit tighter and stuff like that and perform","speaker":null,"is_sponsor":0},{"start_s":527.92,"end_s":533.68,"text":"basically the same on like 6,800 megatransfers and stuff like that. But","speaker":null,"is_sponsor":0},{"start_s":533.68,"end_s":537.44,"text":"this seems to be a good price to performance ratio still, so that's kind of what we're going","speaker":null,"is_sponsor":0},{"start_s":537.44,"end_s":545.52,"text":"with going forward. For SSDs, we are using Samsung 980 Pro 2TB SSDs. We haven't explored","speaker":null,"is_sponsor":0},{"start_s":545.52,"end_s":552.48,"text":"anything as far as using PCI Express Gen 5 SSDs yet, but this seems to be a really good","speaker":null,"is_sponsor":0},{"start_s":552.56,"end_s":558.8,"text":"performance ratio on both the reads and writes of both not just the sequential but as well as","speaker":null,"is_sponsor":0},{"start_s":558.8,"end_s":564.32,"text":"the random reads and writes. So it kind of allows us to get the load times a little bit faster and","speaker":null,"is_sponsor":0},{"start_s":564.32,"end_s":569.28,"text":"kind of have more benchmarks running in sequential order without having to wait for all those","speaker":null,"is_sponsor":0},{"start_s":570.08,"end_s":576.56,"text":"extra load times. Sure, most SSDs on the market nowadays are fairly quick as long as we have","speaker":null,"is_sponsor":0},{"start_s":576.56,"end_s":582.32,"text":"something as DRAM in it, but that's the one we've kind of been going with for a while and","speaker":null,"is_sponsor":0},{"start_s":582.32,"end_s":586.8,"text":"seems to be good enough for us going forward. And if it ain't broke, don't fix it, right?","speaker":null,"is_sponsor":0},{"start_s":587.52,"end_s":593.28,"text":"So there's that. And then lastly, to cool the 7800X3Ds, we are using Noctua","speaker":null,"is_sponsor":0},{"start_s":593.28,"end_s":598.24,"text":"NHD 15 coolers. It's kind of been the standard for a long time for a lot of people.","speaker":null,"is_sponsor":0},{"start_s":599.6,"end_s":603.92,"text":"As soon as the new gen version comes out, I'm sure we're going to be exploring that as well","speaker":null,"is_sponsor":0},{"start_s":603.92,"end_s":610.4,"text":"and seeing just how much more beneficial it could be for us. But it's been basically the","speaker":null,"is_sponsor":0},{"start_s":610.4,"end_s":615.68,"text":"workhorse for us for a while and it probably won't change for a little while afterwards. So","speaker":null,"is_sponsor":0},{"start_s":615.68,"end_s":621.28,"text":"that's kind of where we're at this point. So for the specific project, we obviously have to get","speaker":null,"is_sponsor":0},{"start_s":621.28,"end_s":629.76,"text":"the specific cards that we need to test with. So we have a reference, AMD Radeon 7900 XTX.","speaker":null,"is_sponsor":0},{"start_s":629.76,"end_s":633.52,"text":"It's been the one that we've used in all of our videos that we've ever covered on it.","speaker":null,"is_sponsor":0},{"start_s":634.16,"end_s":640.32,"text":"It's got a pretty sick looking cooler. And then the video card we're going to be using today is a","speaker":null,"is_sponsor":0},{"start_s":640.32,"end_s":648.0,"text":"PNY Accelerate RTX 4080. This seems to be currently the one that is the cheapest on the market,","speaker":null,"is_sponsor":0},{"start_s":648.0,"end_s":652.56,"text":"but that doesn't necessarily mean it's the worst. And this is the one that the requirements","speaker":null,"is_sponsor":0},{"start_s":652.56,"end_s":658.48,"text":"of the test plan specifically outline. So that's the one we're going to be testing today. And lastly,","speaker":null,"is_sponsor":0},{"start_s":659.36,"end_s":665.76,"text":"we obviously need some sort of thermal paste to go between the D15 and the 7800 X3D. And we use","speaker":null,"is_sponsor":0},{"start_s":665.76,"end_s":673.6,"text":"Noctua NTH2 thermal paste. We've been using it for a while. As far as application, more is always","speaker":null,"is_sponsor":0},{"start_s":673.6,"end_s":680.64,"text":"better than less to some degree. And so when I usually put thermal paste on, it's in an equal","speaker":null,"is_sponsor":0},{"start_s":680.64,"end_s":687.92,"text":"sign, which is something that you'll probably see in the build part of the video. So I'm sure","speaker":null,"is_sponsor":0},{"start_s":687.92,"end_s":692.0,"text":"everybody's going to have tons of arguments of, oh, you've used too much or used too little.","speaker":null,"is_sponsor":0},{"start_s":692.0,"end_s":696.64,"text":"There's been so many discussions about that. It doesn't seem to affect the thermal performance","speaker":null,"is_sponsor":0},{"start_s":697.28,"end_s":702.56,"text":"in any sort of way. So as long as we have enough, we're good to go. All right, now we've gone over","speaker":null,"is_sponsor":0},{"start_s":702.56,"end_s":724.48,"text":"the actual hardware, let's get to actually building. Now we've got the system all built together and","speaker":null,"is_sponsor":0},{"start_s":724.48,"end_s":729.2,"text":"up and posted into the BIOS, we're going to be going through making sure we have the correct BIOS","speaker":null,"is_sponsor":0},{"start_s":729.28,"end_s":737.04,"text":"version as well as configuring it to the standardized configuration that we do for","speaker":null,"is_sponsor":0},{"start_s":737.04,"end_s":743.84,"text":"all of our benches. So when we look at our test conditions, we usually outline both drivers","speaker":null,"is_sponsor":0},{"start_s":743.84,"end_s":749.2,"text":"as well as BIOS revision, because there might be an instance where like we're doing some sort of","speaker":null,"is_sponsor":0},{"start_s":749.2,"end_s":754.96,"text":"exploratory testing and being like, hey, we actually don't want the latest version of the BIOS. We want","speaker":null,"is_sponsor":0},{"start_s":754.96,"end_s":760.0,"text":"to go back four or five revisions. That's why we outline it in the actual test conditions. In this","speaker":null,"is_sponsor":0},{"start_s":760.0,"end_s":766.88,"text":"case, it's saying F11D, which is the current version of the BIOS, but obviously we're not","speaker":null,"is_sponsor":0},{"start_s":766.88,"end_s":780.24,"text":"running F11D, so we're going to have to update it. Update your BIOS kids. Yeah, there's, especially","speaker":null,"is_sponsor":0},{"start_s":780.24,"end_s":792.32,"text":"with like new AMD CPUs, 7800X3D in particular, a number of motherboards don't actually support it","speaker":null,"is_sponsor":0},{"start_s":792.32,"end_s":797.36,"text":"correctly out of the box. So if you don't have the option of actually booting into the BIOS,","speaker":null,"is_sponsor":0},{"start_s":797.36,"end_s":803.44,"text":"luckily most manufacturers include some sort of BIOS flashback now. Just slot the USB into the","speaker":null,"is_sponsor":0},{"start_s":803.44,"end_s":809.28,"text":"correct port on the motherboard, make sure obviously the file is named correctly before you do that.","speaker":null,"is_sponsor":0},{"start_s":809.36,"end_s":814.0,"text":"It's incredibly important if you want a computer to work right. Then once we finally get the BIOS","speaker":null,"is_sponsor":0},{"start_s":814.0,"end_s":819.84,"text":"updated, we're going to be going ahead and configuring it and then setting up Windows.","speaker":null,"is_sponsor":0},{"start_s":823.92,"end_s":829.2,"text":"Okay, now we've finished the actual BIOS update. Let's go ahead and start configuring things. First","speaker":null,"is_sponsor":0},{"start_s":829.2,"end_s":834.96,"text":"thing we're going to go to advanced options, so we can actually see everything. We always turn on","speaker":null,"is_sponsor":0},{"start_s":834.96,"end_s":844.64,"text":"expo or if it's Intel, it's XMP. Verify that rebar is on. Most AMD boards usually have it on by default","speaker":null,"is_sponsor":0},{"start_s":844.64,"end_s":853.28,"text":"now, but it's always good to go to. It's usually under IO ports or some sort of IO naming scheme,","speaker":null,"is_sponsor":0},{"start_s":853.28,"end_s":860.4,"text":"whatever manufacturer makes it. Inside here, we also turn off the IGP or integrated graphics,","speaker":null,"is_sponsor":0},{"start_s":861.12,"end_s":870.96,"text":"especially if we are testing any sort of dedicated GPU. All it does is sometimes confuse some games","speaker":null,"is_sponsor":0},{"start_s":870.96,"end_s":876.8,"text":"or programs into wondering which actual graphics card to be using at the time.","speaker":null,"is_sponsor":0},{"start_s":879.12,"end_s":885.04,"text":"And then from there, the only other thing that we do is go into the fan curves","speaker":null,"is_sponsor":0},{"start_s":885.28,"end_s":891.28,"text":"and we manually set it so that it is a linear scale.","speaker":null,"is_sponsor":0},{"start_s":894.72,"end_s":900.56,"text":"So it basically looks like that. Then at that point, we can actually start installing Windows.","speaker":null,"is_sponsor":0},{"start_s":904.56,"end_s":912.8,"text":"For Windows, we use a optimized Windows image that our software developers have","speaker":null,"is_sponsor":0},{"start_s":913.52,"end_s":919.52,"text":"created. Basically strip back a bunch of stuff that we don't need to be running in the background,","speaker":null,"is_sponsor":0},{"start_s":919.52,"end_s":926.0,"text":"as well as adding in some programs that we actively use during our testing. We actually","speaker":null,"is_sponsor":0},{"start_s":926.56,"end_s":931.68,"text":"reinstall them every three months. We'll usually update them basically and create a new Windows","speaker":null,"is_sponsor":0},{"start_s":931.68,"end_s":937.92,"text":"image that has our latest updates so that we don't have to constantly do this rigmarole of","speaker":null,"is_sponsor":0},{"start_s":938.64,"end_s":945.76,"text":"get a million updates, a million drivers. We can just basically install the latest drivers","speaker":null,"is_sponsor":0},{"start_s":945.76,"end_s":950.0,"text":"and then we have basically all of our programs. So now we just got to let this thing run.","speaker":null,"is_sponsor":0},{"start_s":951.44,"end_s":956.96,"text":"Very exciting stuff, like watching paint dry. In our test conditions, we'll usually outline the","speaker":null,"is_sponsor":0},{"start_s":956.96,"end_s":965.04,"text":"exact version of Windows we want to run. Going back to that, we might need to recreate an old","speaker":null,"is_sponsor":0},{"start_s":965.04,"end_s":973.2,"text":"set of parameters. So we might not be using Windows 11 22H2. We might use 22H1 or that's why","speaker":null,"is_sponsor":0},{"start_s":973.2,"end_s":979.04,"text":"on that list, we actually have all the other versions of our images still on there so that we","speaker":null,"is_sponsor":0},{"start_s":979.04,"end_s":984.24,"text":"can actually go back and be like, oh, actually we need to use the version from May. And then we","speaker":null,"is_sponsor":0},{"start_s":984.24,"end_s":988.88,"text":"basically, if we need to recreate that set of test conditions, we have to go and also stop","speaker":null,"is_sponsor":0},{"start_s":988.88,"end_s":994.08,"text":"Windows updates from updating. Now we're in Windows. We are not going to install the Gigabyte","speaker":null,"is_sponsor":0},{"start_s":994.08,"end_s":1001.2,"text":"Control Center. So basically at this point, we have to go through update all of our drivers,","speaker":null,"is_sponsor":0},{"start_s":1001.2,"end_s":1008.56,"text":"as well as install all of our programs that we're actually using for the project.","speaker":null,"is_sponsor":0},{"start_s":1014.08,"end_s":1021.76,"text":"Okay, so now we've installed the chipset drivers. We can make sure that we get the latest NVIDIA","speaker":null,"is_sponsor":0},{"start_s":1021.76,"end_s":1029.28,"text":"driver as, or not necessarily the latest, because if the project does require an older version of","speaker":null,"is_sponsor":0},{"start_s":1029.28,"end_s":1034.88,"text":"the driver, we got to get that version instead. But in this case, in this instance, we are using","speaker":null,"is_sponsor":0},{"start_s":1034.88,"end_s":1040.96,"text":"the latest version. So we're using 537.42. Let's get that downloaded.","speaker":null,"is_sponsor":0},{"start_s":1047.52,"end_s":1050.56,"text":"Then we're just going to make sure all the video settings are set correctly,","speaker":null,"is_sponsor":0},{"start_s":1050.56,"end_s":1058.88,"text":"so we essentially turn off G-Sync, any sort of potential accelerators, V-Sync,","speaker":null,"is_sponsor":0},{"start_s":1060.16,"end_s":1065.68,"text":"so we can get the, basically the raw performance of the card. Update your cards.","speaker":null,"is_sponsor":0},{"start_s":1067.04,"end_s":1074.56,"text":"There's a lot of things that NVIDIA doesn't post or advertise, like a number of three series cards","speaker":null,"is_sponsor":0},{"start_s":1074.56,"end_s":1079.92,"text":"don't have rebar enabled out of the box. So there's actually a bunch of articles that NVIDIA","speaker":null,"is_sponsor":0},{"start_s":1080.08,"end_s":1089.68,"text":"posted that are really useful, such as UEFI fixes, DisplayPort fixes, and things like rebar","speaker":null,"is_sponsor":0},{"start_s":1089.68,"end_s":1094.96,"text":"patches and stuff like that. So if you find your card isn't matching up for whatever reason,","speaker":null,"is_sponsor":0},{"start_s":1094.96,"end_s":1100.72,"text":"or is doing something funky, good to check those out, because sometimes it might be as simple as,","speaker":null,"is_sponsor":0},{"start_s":1100.72,"end_s":1106.56,"text":"oh, I need to do this little update. Boom, I've got more performance, or oh, my video doesn't","speaker":null,"is_sponsor":0},{"start_s":1106.56,"end_s":1110.88,"text":"seem to work in the BIOS, but it works fine in Windows. Do a firmware update. Oh, sweet, now","speaker":null,"is_sponsor":0},{"start_s":1110.88,"end_s":1114.88,"text":"it's fine. Alrighty, now that we've got that installed, let's just make sure we don't have","speaker":null,"is_sponsor":0},{"start_s":1114.88,"end_s":1123.36,"text":"any other weird drivers. Looks good to me. Usually if there's no yellows, we're happy campers.","speaker":null,"is_sponsor":0},{"start_s":1124.32,"end_s":1130.56,"text":"We usually do all the Windows updates according to what our project plan is. In most cases,","speaker":null,"is_sponsor":0},{"start_s":1130.56,"end_s":1134.96,"text":"it's usually just as simple as getting every single update that's in there. A lot of times,","speaker":null,"is_sponsor":0},{"start_s":1135.04,"end_s":1138.64,"text":"what will happen is, you know, there will be driver updates and stuff like that, but if you","speaker":null,"is_sponsor":0},{"start_s":1138.64,"end_s":1144.96,"text":"already have a later driver than what Windows sees, often you'll get like a download error a lot of","speaker":null,"is_sponsor":0},{"start_s":1144.96,"end_s":1149.2,"text":"the times, because Windows is flagged, oh, you actually have a newer version, we don't need to","speaker":null,"is_sponsor":0},{"start_s":1149.2,"end_s":1156.56,"text":"get this. And what we do is we tend to update our Windows image every three months to help include","speaker":null,"is_sponsor":0},{"start_s":1156.56,"end_s":1160.64,"text":"a lot of these Windows updates. We essentially have to download a lot less of them. It is time","speaker":null,"is_sponsor":0},{"start_s":1160.64,"end_s":1165.68,"text":"consuming and the less time we have to spend on constantly downloading things and updating things,","speaker":null,"is_sponsor":0},{"start_s":1165.68,"end_s":1171.2,"text":"the better, right? Then we can get right into the actual testing. It saves us a lot of overhead,","speaker":null,"is_sponsor":0},{"start_s":1171.2,"end_s":1178.64,"text":"basically. So one last thing we're going to do before we get all of our games downloaded on the","speaker":null,"is_sponsor":0},{"start_s":1178.64,"end_s":1183.52,"text":"system is we're going to go through and just do some last minute Windows optimizations.","speaker":null,"is_sponsor":0},{"start_s":1184.24,"end_s":1190.72,"text":"So first thing we're going to do is we're going to do some trimming on the actual drive itself.","speaker":null,"is_sponsor":0},{"start_s":1191.92,"end_s":1197.92,"text":"Just helps speed it up just that little bit better. Then we're going to go and go to percent","speaker":null,"is_sponsor":0},{"start_s":1197.92,"end_s":1203.92,"text":"temp percent to basically get to our temporary folder. And we're going to basically delete","speaker":null,"is_sponsor":0},{"start_s":1203.92,"end_s":1208.56,"text":"everything we can in here. So one last thing we're going to do is we're going to set the recycling","speaker":null,"is_sponsor":0},{"start_s":1208.56,"end_s":1215.6,"text":"bin to 15 megs. The reason we do this is just that if we delete any files on the system, it'll","speaker":null,"is_sponsor":0},{"start_s":1215.6,"end_s":1220.48,"text":"just delete them right away. It won't sit in the recycling bin taking up extra space we don't need.","speaker":null,"is_sponsor":0},{"start_s":1221.36,"end_s":1225.68,"text":"Basically, it skips a step of us happening at empty out the recycling bin in most cases. So","speaker":null,"is_sponsor":0},{"start_s":1226.4,"end_s":1232.16,"text":"kind of useful in that regard. So once we have all this set up, the only last thing we have to do","speaker":null,"is_sponsor":0},{"start_s":1232.16,"end_s":1238.48,"text":"is just make sure whether it's NVIDIA, Intel, or AMD, we have any sort of variable refresh rate","speaker":null,"is_sponsor":0},{"start_s":1238.48,"end_s":1245.76,"text":"stuff turned off, as well as B-sync. The reason behind that is we don't want to potentially give","speaker":null,"is_sponsor":0},{"start_s":1245.76,"end_s":1252.64,"text":"any sort of advantage to one manufacturer over another with any sort of technologies they","speaker":null,"is_sponsor":0},{"start_s":1252.64,"end_s":1257.52,"text":"might have running in the background causing any optimizations that are effectively outside of","speaker":null,"is_sponsor":0},{"start_s":1257.52,"end_s":1264.08,"text":"Windows itself doing hardware optimizations. That way it kind of keeps the playing field a little","speaker":null,"is_sponsor":0},{"start_s":1264.08,"end_s":1270.96,"text":"bit more level. So we're going to pop into here. I doubt we'll have G-sync on here. No, because we","speaker":null,"is_sponsor":0},{"start_s":1270.96,"end_s":1276.96,"text":"are currently plugged into our NDI viewer to get all of our capture. So for this specific instance,","speaker":null,"is_sponsor":0},{"start_s":1276.96,"end_s":1284.24,"text":"we won't have G-sync enabled. But typically, because these monitors are actually G-sync capable,","speaker":null,"is_sponsor":0},{"start_s":1285.2,"end_s":1290.88,"text":"anytime we pull up an NVIDIA card inside there, it'll immediately enable G-sync. So we have to","speaker":null,"is_sponsor":0},{"start_s":1290.88,"end_s":1298.08,"text":"make sure we go in and actually disable that. And then what we do also is in here, we've got to","speaker":null,"is_sponsor":0},{"start_s":1298.08,"end_s":1307.2,"text":"manage three settings, scroll down to vertical sync, and we make sure we force it off. That way,","speaker":null,"is_sponsor":0},{"start_s":1307.76,"end_s":1312.24,"text":"because there's some games that they don't either give you the option or they tell you it's off,","speaker":null,"is_sponsor":0},{"start_s":1312.24,"end_s":1317.52,"text":"but it's not actually off. It's kind of just ensures that no matter the game, no matter the","speaker":null,"is_sponsor":0},{"start_s":1317.52,"end_s":1324.16,"text":"settings, if somebody forgot to turn off V-sync for whatever reason, this basically forces it off.","speaker":null,"is_sponsor":0},{"start_s":1324.16,"end_s":1329.92,"text":"So there's no opportunity for the game to even interact with, hey, by the way, I need V-sync.","speaker":null,"is_sponsor":0},{"start_s":1330.72,"end_s":1337.68,"text":"So it kind of takes that guesswork out of it. And then we go into display settings.","speaker":null,"is_sponsor":0},{"start_s":1338.08,"end_s":1344.0,"text":"We're going to be running at 1080p, so we just leave the resolution as it is, but we change our scale","speaker":null,"is_sponsor":0},{"start_s":1344.0,"end_s":1355.68,"text":"to 100% rather than 150. Some of our markbench benchmarks do still use CV in order to do character","speaker":null,"is_sponsor":0},{"start_s":1355.68,"end_s":1361.36,"text":"recognition. So if the scaling is at 150% and you're on the desktop, for example,","speaker":null,"is_sponsor":0},{"start_s":1362.08,"end_s":1367.44,"text":"the new version of Hitman has a launcher that it shows up beforehand that has all the options","speaker":null,"is_sponsor":0},{"start_s":1367.52,"end_s":1373.28,"text":"and stuff in. How markbench works in that one is it actually will look for the specific word.","speaker":null,"is_sponsor":0},{"start_s":1373.28,"end_s":1378.4,"text":"If our scaling is different than what it's supposed to be, markbench will be like,","speaker":null,"is_sponsor":0},{"start_s":1378.4,"end_s":1383.2,"text":"I can't find this because it's effectively looking for a picture at a different scale","speaker":null,"is_sponsor":0},{"start_s":1383.2,"end_s":1389.36,"text":"than what it actually is. So if we set it to 100%, basically we're going to have no problems.","speaker":null,"is_sponsor":0},{"start_s":1389.36,"end_s":1393.92,"text":"And then normally what we do is we check to make sure we're at the highest hertz we can be for the","speaker":null,"is_sponsor":0},{"start_s":1393.92,"end_s":1400.08,"text":"monitor. Because we're going through the NDI, we actually have to run at 60 hertz because that's","speaker":null,"is_sponsor":0},{"start_s":1400.08,"end_s":1406.8,"text":"the maximum the actual NDI itself supports. But typically it's 144 for the actual monitors themselves.","speaker":null,"is_sponsor":0},{"start_s":1406.8,"end_s":1413.76,"text":"So lastly, we go into change default graphics settings. We make sure that hardware accelerated","speaker":null,"is_sponsor":0},{"start_s":1413.76,"end_s":1421.04,"text":"GPU scheduling is on. This is at Windows level. This isn't hardware specific such as","speaker":null,"is_sponsor":0},{"start_s":1422.0,"end_s":1428.32,"text":"NVIDIA Reflex or AMD's Antilag. That is something specific to the hardware,","speaker":null,"is_sponsor":0},{"start_s":1429.28,"end_s":1436.56,"text":"not Windows itself. So this one, it's the same across all the platforms of hardware. It doesn't","speaker":null,"is_sponsor":0},{"start_s":1437.68,"end_s":1444.16,"text":"give an advantage to NVIDIA over AMD because it's scheduling at the exact same. Anything else that's","speaker":null,"is_sponsor":0},{"start_s":1444.16,"end_s":1449.84,"text":"in the menu, usually we have like optimization for Windows games. We turn that off, as well as in","speaker":null,"is_sponsor":0},{"start_s":1449.84,"end_s":1457.36,"text":"some cases, there is a variable refresh rate option in here. It usually shows up for Intel","speaker":null,"is_sponsor":0},{"start_s":1457.36,"end_s":1464.96,"text":"and AMD because theirs interact a little bit differently than G-Sync does. And so we make","speaker":null,"is_sponsor":0},{"start_s":1464.96,"end_s":1469.76,"text":"sure that's off as well. Because again, we don't want to give advantage to one specific card. We","speaker":null,"is_sponsor":0},{"start_s":1469.76,"end_s":1475.68,"text":"want to see raw performance of each of the cards stacked against each other so we can understand","speaker":null,"is_sponsor":0},{"start_s":1476.32,"end_s":1481.84,"text":"what is actually the best option. Now that we've gotten all the Windows updates installed,","speaker":null,"is_sponsor":0},{"start_s":1482.88,"end_s":1488.88,"text":"next thing we got to do is we have to install all the programs as well as any games that we're","speaker":null,"is_sponsor":0},{"start_s":1488.88,"end_s":1494.56,"text":"going to be using to do this project. So in this case, we need to install Cyberpunk, Dota 2,","speaker":null,"is_sponsor":0},{"start_s":1495.44,"end_s":1502.96,"text":"and Red Dead. So we already have Steam pre-installed on our Windows image so we don't have to worry","speaker":null,"is_sponsor":0},{"start_s":1502.96,"end_s":1506.96,"text":"about actually going to the website and getting that part, but unfortunately it always needs to","speaker":null,"is_sponsor":0},{"start_s":1506.96,"end_s":1514.4,"text":"update anyways. So it's kind of a moot point. While that's happening, what we're also going to do is get","speaker":null,"is_sponsor":0},{"start_s":1515.52,"end_s":1522.24,"text":"markbench from our local drive, as well as some patches that I need to install as well.","speaker":null,"is_sponsor":0},{"start_s":1522.32,"end_s":1532.64,"text":"So we are going to install, nope, Cyberpunk. We're going to install Red Dead.","speaker":null,"is_sponsor":0},{"start_s":1534.88,"end_s":1539.36,"text":"And then the most annoying thing is that Dota 2 doesn't stay in your library half the time,","speaker":null,"is_sponsor":0},{"start_s":1539.36,"end_s":1545.04,"text":"so we are going to go here and do play now.","speaker":null,"is_sponsor":0},{"start_s":1545.44,"end_s":1550.8,"text":"Okay, so those three guys are downloading.","speaker":null,"is_sponsor":0},{"start_s":1552.88,"end_s":1558.56,"text":"We've got markbench extracted here. We need to add another harness in for the Dota 1,","speaker":null,"is_sponsor":0},{"start_s":1558.56,"end_s":1567.04,"text":"so let's get that real quick. And there it is, ready to go for when we are ready to use it.","speaker":null,"is_sponsor":0},{"start_s":1568.8,"end_s":1574.08,"text":"And then the other thing we need is Blender Benchmark, so let's quickly grab that as well.","speaker":null,"is_sponsor":0},{"start_s":1575.44,"end_s":1582.32,"text":"So although markbench does have a Blender Benchmark actually integrated into it,","speaker":null,"is_sponsor":0},{"start_s":1583.12,"end_s":1588.88,"text":"we do need to still download the application itself to download the files required for it to run.","speaker":null,"is_sponsor":0},{"start_s":1590.0,"end_s":1594.88,"text":"There's a little bit of bugginess with the CLI. When it tries to download the files,","speaker":null,"is_sponsor":0},{"start_s":1594.88,"end_s":1601.52,"text":"it effectively says I can't do this and quits out. So if we download the actual program and have it","speaker":null,"is_sponsor":0},{"start_s":1601.52,"end_s":1607.92,"text":"download the files, it just makes the automation a little bit simpler on our side because it has","speaker":null,"is_sponsor":0},{"start_s":1607.92,"end_s":1613.68,"text":"the files already completely installed. So effectively we want version 3.6. We're going to","speaker":null,"is_sponsor":0},{"start_s":1613.68,"end_s":1620.08,"text":"download the benchmarks. Again, let's just make sure that markbench is all prepared, just like when","speaker":null,"is_sponsor":0},{"start_s":1620.08,"end_s":1625.12,"text":"we have to go through any of our game settings. We have to actually go download the programs,","speaker":null,"is_sponsor":0},{"start_s":1625.12,"end_s":1632.4,"text":"download the games, actually set up the settings ourselves. That stuff is not currently automated.","speaker":null,"is_sponsor":0},{"start_s":1633.12,"end_s":1636.72,"text":"Now we have all the games downloading and everything's kind of getting set up.","speaker":null,"is_sponsor":0},{"start_s":1636.72,"end_s":1640.0,"text":"I'm going to throw it over to Nick and he can talk a little bit about markbench.","speaker":null,"is_sponsor":0},{"start_s":1640.0,"end_s":1646.4,"text":"Fundamentally, markbench is our test automation and orchestration framework. It's not just simply","speaker":null,"is_sponsor":0},{"start_s":1646.4,"end_s":1652.56,"text":"a simple desktop GUI anymore. It's a constellation of services and components that work together","speaker":null,"is_sponsor":0},{"start_s":1652.56,"end_s":1657.68,"text":"to form the ecosystem that we conduct our testing in. But it all does start with the test","speaker":null,"is_sponsor":0},{"start_s":1657.68,"end_s":1663.44,"text":"harnesses, which most of ours are in Python because it's most easily automatable for us.","speaker":null,"is_sponsor":0},{"start_s":1664.08,"end_s":1670.64,"text":"However, it doesn't need to be Python. They can be PowerShell scripts, Ruby, GoLang, what have you.","speaker":null,"is_sponsor":0},{"start_s":1670.64,"end_s":1677.84,"text":"The important part is that each test harness gives back a exit code of zero or one, zero to indicate","speaker":null,"is_sponsor":0},{"start_s":1677.84,"end_s":1683.52,"text":"a successful test run, anything greater such as one to indicate that it failed. Because that exit","speaker":null,"is_sponsor":0},{"start_s":1683.52,"end_s":1690.08,"text":"code is then read by the next layer of markbench, which is what we call auto bench. It just comes","speaker":null,"is_sponsor":0},{"start_s":1690.08,"end_s":1695.12,"text":"before the GoLang GUI and it really contains all of the major features of our orchestration.","speaker":null,"is_sponsor":0},{"start_s":1696.16,"end_s":1701.28,"text":"This layer is what separates what we're doing from just a collection of PowerShell scripts","speaker":null,"is_sponsor":0},{"start_s":1701.84,"end_s":1707.6,"text":"to something that is a lot easier to manage long term because inside of auto bench, that's","speaker":null,"is_sponsor":0},{"start_s":1707.6,"end_s":1712.0,"text":"where we get our orchestration. So with the test harness, I have the capability of running","speaker":null,"is_sponsor":0},{"start_s":1712.0,"end_s":1718.0,"text":"individual tests, but I want to be able to group those into a bunch of test suites such as let's","speaker":null,"is_sponsor":0},{"start_s":1718.0,"end_s":1725.44,"text":"say I want to run Dota 2 four times at one resolution and then Red Dead Redemption 2 at a","speaker":null,"is_sponsor":0},{"start_s":1725.44,"end_s":1730.08,"text":"different resolution. So that's where the auto bench side comes in. Auto bench is where we run","speaker":null,"is_sponsor":0},{"start_s":1730.08,"end_s":1735.52,"text":"presentmon in system logger in conjunction with the test harness so that we get a similar timestamp","speaker":null,"is_sponsor":0},{"start_s":1735.52,"end_s":1740.96,"text":"between all three because this eliminates the human error of starting a test and then starting","speaker":null,"is_sponsor":0},{"start_s":1740.96,"end_s":1746.96,"text":"hardware info or like logging on the side, which then would separate our timestamps by","speaker":null,"is_sponsor":0},{"start_s":1747.52,"end_s":1752.64,"text":"sometimes multiple seconds, which makes it easier to trim the sessions later downstream,","speaker":null,"is_sponsor":0},{"start_s":1752.64,"end_s":1758.48,"text":"which John will explain later. And then so the auto bench is all command line interface,","speaker":null,"is_sponsor":0},{"start_s":1758.48,"end_s":1763.84,"text":"so to make it easier for us and John and writers or whoever wants to run Markbench,","speaker":null,"is_sponsor":0},{"start_s":1763.84,"end_s":1769.44,"text":"we added a Golang GUI. So we went with the Golang GUI because the framework that we used","speaker":null,"is_sponsor":0},{"start_s":1770.32,"end_s":1776.8,"text":"was very successful for us for other GUIs such as the keyboard tester, the robot, and the latency","speaker":null,"is_sponsor":0},{"start_s":1776.8,"end_s":1782.64,"text":"tests. So anytime you see Markbench on screen, that's the Golang part, but underneath all the","speaker":null,"is_sponsor":0},{"start_s":1782.64,"end_s":1787.84,"text":"work is really being done inside the test harness as well as the Python orchestration layer.","speaker":null,"is_sponsor":0},{"start_s":1788.64,"end_s":1794.32,"text":"So once the test is done running, let's say we just ran an individual test and once that test is","speaker":null,"is_sponsor":0},{"start_s":1794.32,"end_s":1801.6,"text":"done running, it sends its data up to a service called data ingest server, which is a Golang web","speaker":null,"is_sponsor":0},{"start_s":1801.6,"end_s":1807.92,"text":"API, which takes it in that data in a protobuf format and then writes it to our Postgres database.","speaker":null,"is_sponsor":0},{"start_s":1808.88,"end_s":1813.76,"text":"It's from that Postgres database, all of our data sits at rest and then is disseminated or sent","speaker":null,"is_sponsor":0},{"start_s":1814.4,"end_s":1820.24,"text":"to all the various tools that John can use to evaluate and report on that data. And we use a","speaker":null,"is_sponsor":0},{"start_s":1820.24,"end_s":1827.04,"text":"couple tools, notably Grafana and Redash for a lot of visualizations. And then we have internal","speaker":null,"is_sponsor":0},{"start_s":1827.04,"end_s":1833.52,"text":"tooling that we just kind of collectively call the Harbor Tools portal. And that contains anything","speaker":null,"is_sponsor":0},{"start_s":1833.52,"end_s":1839.44,"text":"that Grafana or Redash or another off the shelf tool doesn't provide us. Now if we go back up to","speaker":null,"is_sponsor":0},{"start_s":1839.44,"end_s":1847.68,"text":"the test harnesses, on their own, they're fairly dumb in that they can send keyboard and mouse input,","speaker":null,"is_sponsor":0},{"start_s":1848.24,"end_s":1854.48,"text":"but they don't always know what's going on the screen. So we used to use OpenCV locally to do","speaker":null,"is_sponsor":0},{"start_s":1854.48,"end_s":1860.08,"text":"template matching, but it's more of a brute force approach. And it's very flaky as it depends on","speaker":null,"is_sponsor":0},{"start_s":1860.08,"end_s":1865.92,"text":"certain resolutions. So if you're doing template matching with a resolution of 1920 by 1080,","speaker":null,"is_sponsor":0},{"start_s":1866.56,"end_s":1871.92,"text":"you need to select screenshots of the path or the flow you want to go through the menu of the","speaker":null,"is_sponsor":0},{"start_s":1871.92,"end_s":1876.56,"text":"particular game. But then if you want to do it at a lower resolution like 720p,","speaker":null,"is_sponsor":0},{"start_s":1877.68,"end_s":1882.8,"text":"nine times at a 10, you have to take a smaller screenshot. So it doesn't scale very well. And","speaker":null,"is_sponsor":0},{"start_s":1882.8,"end_s":1887.2,"text":"that's where we start getting into some of the services that we're adding. The first one being","speaker":null,"is_sponsor":0},{"start_s":1887.2,"end_s":1894.16,"text":"Keras optical character recognition or OCR. This allows a test harness to send a screenshot of the","speaker":null,"is_sponsor":0},{"start_s":1894.24,"end_s":1899.6,"text":"system it's running on with a target word and Keras will come back within a few milliseconds","speaker":null,"is_sponsor":0},{"start_s":1899.6,"end_s":1903.44,"text":"is that word on the screen or not. This reduces flakiness in the test harness,","speaker":null,"is_sponsor":0},{"start_s":1904.0,"end_s":1909.6,"text":"which makes navigating menus a hundred times easier. And in the future, we're going to be adding","speaker":null,"is_sponsor":0},{"start_s":1909.6,"end_s":1916.56,"text":"services that do a bunch of computer vision things such as depth estimation, object detection and","speaker":null,"is_sponsor":0},{"start_s":1916.56,"end_s":1921.6,"text":"whatnot, all in the effort of making the test harnesses more a lot smarter. But we'll talk","speaker":null,"is_sponsor":0},{"start_s":1921.6,"end_s":1928.16,"text":"about levels of automation in a bit. So the benefit that Mark Bench and its ecosystem gives us","speaker":null,"is_sponsor":0},{"start_s":1928.16,"end_s":1935.28,"text":"over just a collection of PowerShell scripts for one is maintainability at the scale that we're","speaker":null,"is_sponsor":0},{"start_s":1935.28,"end_s":1942.24,"text":"trying to achieve. PowerShell scripts would or some auto hockey scripts would become messy very","speaker":null,"is_sponsor":0},{"start_s":1942.24,"end_s":1947.6,"text":"quickly because eventually we'd have probably like 100 to 200. And as best as you could organize","speaker":null,"is_sponsor":0},{"start_s":1947.6,"end_s":1952.96,"text":"those in some directories, it's just not a it's a maintainable nightmare. Investing in such an","speaker":null,"is_sponsor":0},{"start_s":1952.96,"end_s":1960.64,"text":"ecosystem and the automation of Mark Bench, it allows us to build it to support multiple platforms.","speaker":null,"is_sponsor":0},{"start_s":1960.64,"end_s":1967.92,"text":"So not just Windows on a desktop, but handhelds and macOS and Linux as well. And it normalizes","speaker":null,"is_sponsor":0},{"start_s":1967.92,"end_s":1973.52,"text":"data for us so that when we capture metrics and we set it at rest, it's a lot easier to","speaker":null,"is_sponsor":0},{"start_s":1973.52,"end_s":1978.4,"text":"query that data. Back to you, John. Thanks Nick for that explanation of Mark Bench.","speaker":null,"is_sponsor":0},{"start_s":1978.4,"end_s":1982.64,"text":"Now what we're going to do since all these games are finished downloading is get them all set up","speaker":null,"is_sponsor":0},{"start_s":1982.64,"end_s":1986.88,"text":"and then we'll be able to get our benchmarks started. So we're going to start off with cyber","speaker":null,"is_sponsor":0},{"start_s":1986.88,"end_s":1994.56,"text":"bunk here. We have documentation that we are working on that we will be publishing in the near","speaker":null,"is_sponsor":0},{"start_s":1994.56,"end_s":2000.96,"text":"future of all of our settings and configuration for the various benchmarks that we include in our","speaker":null,"is_sponsor":0},{"start_s":2000.96,"end_s":2009.6,"text":"videos. I believe we've already released a couple of them on the forums. So I'm sure we'll continue","speaker":null,"is_sponsor":0},{"start_s":2009.6,"end_s":2015.04,"text":"to update that as time goes on. Okie dokie. Yeah, the new update for cyberpunk came out. So that's","speaker":null,"is_sponsor":0},{"start_s":2015.04,"end_s":2022.32,"text":"cool. So first thing we do, we go to settings. We're going to go to video mode and we're going to","speaker":null,"is_sponsor":0},{"start_s":2022.32,"end_s":2027.28,"text":"make sure we're at 1080p because that's what the project requires. You're going to be turning off","speaker":null,"is_sponsor":0},{"start_s":2027.28,"end_s":2032.4,"text":"any sort of things like low latency stuff like that. I think that it could benefit that specific","speaker":null,"is_sponsor":0},{"start_s":2032.4,"end_s":2038.4,"text":"card because we want the raw performance of the card. We don't want any sort of additional things","speaker":null,"is_sponsor":0},{"start_s":2038.4,"end_s":2045.84,"text":"like DLSS or low latency that can certainly help benefit the card and it might get better","speaker":null,"is_sponsor":0},{"start_s":2045.84,"end_s":2052.32,"text":"performance for sure. But what we're trying to determine is apples to apples between that 7900","speaker":null,"is_sponsor":0},{"start_s":2052.4,"end_s":2058.32,"text":"xdx and this 4080. So we're going to be turning off low latency. We're going to be making sure","speaker":null,"is_sponsor":0},{"start_s":2058.32,"end_s":2065.2,"text":"that we're on full screen. We don't want borderless. Then we're going to go to the graphics tab and for","speaker":null,"is_sponsor":0},{"start_s":2065.2,"end_s":2070.0,"text":"our specific test we aren't calling for ray tracing. So we're going to actually be turning off","speaker":null,"is_sponsor":0},{"start_s":2070.0,"end_s":2077.04,"text":"ray tracing. We're going to be going just strictly ultra and we're going to go ahead and turn down","speaker":null,"is_sponsor":0},{"start_s":2077.12,"end_s":2084.56,"text":"FSR to zero. Go back up a tiny bit. Go back down a tiny bit. Each individual time we change something","speaker":null,"is_sponsor":0},{"start_s":2084.56,"end_s":2091.36,"text":"we're going to hit apply. The settings for cyberpunk are still kind of a little bit weird","speaker":null,"is_sponsor":0},{"start_s":2091.36,"end_s":2096.32,"text":"where you might change some sliders and you might hit apply and sometimes it's forgetting","speaker":null,"is_sponsor":0},{"start_s":2096.32,"end_s":2101.6,"text":"that you hit apply or you change the quick preset. It doesn't actually use the preset because you","speaker":null,"is_sponsor":0},{"start_s":2101.6,"end_s":2107.92,"text":"haven't applied it. So when we go up and down on the FSR that kind of helps with that and then","speaker":null,"is_sponsor":0},{"start_s":2107.92,"end_s":2113.52,"text":"we're going to go ahead and turn FSR off because we don't actually want it. And that's basically how","speaker":null,"is_sponsor":0},{"start_s":2113.52,"end_s":2121.92,"text":"we set up our non ray tracing version of cyberpunk. So at that point we can basically exit out of that","speaker":null,"is_sponsor":0},{"start_s":2122.8,"end_s":2130.0,"text":"and move on to the next one which is Dota 2. Dota 2 is one of our more recent benchmarks that we've","speaker":null,"is_sponsor":0},{"start_s":2130.0,"end_s":2137.84,"text":"been working on. It uses a replay from I believe Dreamhack. A lot of visuals on the screen so that","speaker":null,"is_sponsor":0},{"start_s":2137.84,"end_s":2145.68,"text":"we can basically tax the card as much as possible. So in Dota we're going to go up to the cog, go to","speaker":null,"is_sponsor":0},{"start_s":2145.68,"end_s":2150.72,"text":"video and we're not going to use monitor's resolution. We're going to use advanced settings.","speaker":null,"is_sponsor":0},{"start_s":2152.0,"end_s":2157.76,"text":"In this case we need 1080p. We're going to be checking on to exclusive full screen","speaker":null,"is_sponsor":0},{"start_s":2158.56,"end_s":2161.52,"text":"rather than that. Turning off low latency yet again.","speaker":null,"is_sponsor":0},{"start_s":2163.52,"end_s":2168.24,"text":"And then for rendering side we just pulled the slider all the way to best looking.","speaker":null,"is_sponsor":0},{"start_s":2170.08,"end_s":2176.16,"text":"It's basically a preset that has all these specific settings. These things always off.","speaker":null,"is_sponsor":0},{"start_s":2176.72,"end_s":2180.8,"text":"That sort of thing. That's all we got to do for this guy here. So we go ahead and close that.","speaker":null,"is_sponsor":0},{"start_s":2181.76,"end_s":2187.52,"text":"Okay now we've done Dota 2. Lastly we just need to set up Red Dead Redemption. So we're going to","speaker":null,"is_sponsor":0},{"start_s":2187.6,"end_s":2195.92,"text":"hit Z to go into settings. Go to graphics. Make sure the resolution and the screen type are correct","speaker":null,"is_sponsor":0},{"start_s":2195.92,"end_s":2203.2,"text":"which they are. We're going to be turning off v-sync. Now pull the slider all the way to the right that","speaker":null,"is_sponsor":0},{"start_s":2203.2,"end_s":2210.56,"text":"usually gets us all the way there but we still have to actually go through and manually check these","speaker":null,"is_sponsor":0},{"start_s":2210.56,"end_s":2216.64,"text":"because like I was saying before Red Dead's a little bit different in that this slider","speaker":null,"is_sponsor":0},{"start_s":2216.64,"end_s":2219.76,"text":"doesn't necessarily mean you're going to go everything ultra if you hit that.","speaker":null,"is_sponsor":0},{"start_s":2220.56,"end_s":2225.76,"text":"If you have lower end hardware it actually kind of tries to adjust that slider based on","speaker":null,"is_sponsor":0},{"start_s":2225.76,"end_s":2230.88,"text":"your hardware. So on like a low end laptop you go all the way to favorite quality and","speaker":null,"is_sponsor":0},{"start_s":2230.88,"end_s":2237.28,"text":"everything might still read as high. So it's something that that's why we have actual screenshots","speaker":null,"is_sponsor":0},{"start_s":2237.28,"end_s":2244.88,"text":"of our settings for it so that we can actually make sure that they're the same even after we","speaker":null,"is_sponsor":0},{"start_s":2244.88,"end_s":2253.36,"text":"change that slider. So we should be sitting at ultra 16 ultra ultra ultra high ultra high ultra","speaker":null,"is_sponsor":0},{"start_s":2253.36,"end_s":2263.68,"text":"medium high off high and this is always the one that kind of gets a little bit fussy.","speaker":null,"is_sponsor":0},{"start_s":2264.24,"end_s":2276.08,"text":"That looks right. Okay and then we're going to hit enter to apply the changes. I'll often","speaker":null,"is_sponsor":0},{"start_s":2276.08,"end_s":2281.68,"text":"recommend don't alt f4. There are some games out there that the settings don't save until you","speaker":null,"is_sponsor":0},{"start_s":2281.68,"end_s":2287.04,"text":"actually properly quit the game. It's like a fail safe so if you just exit out the right way","speaker":null,"is_sponsor":0},{"start_s":2287.76,"end_s":2291.84,"text":"shouldn't be any problem. So first thing we're going to do is set up mark bench here so we","speaker":null,"is_sponsor":0},{"start_s":2291.84,"end_s":2297.76,"text":"actually start running the benchmarks. So in order to do that we have to input a few values","speaker":null,"is_sponsor":0},{"start_s":2297.76,"end_s":2302.88,"text":"first. So we're going to go through all the different boxes and options that we have for","speaker":null,"is_sponsor":0},{"start_s":2302.88,"end_s":2309.12,"text":"the tests we're going to be running today. The first box that we have that we can fill","speaker":null,"is_sponsor":0},{"start_s":2309.12,"end_s":2315.2,"text":"information into is called the project slug. What that usually entails is every time we create a","speaker":null,"is_sponsor":0},{"start_s":2315.2,"end_s":2323.28,"text":"test plan we usually assign a sort of either a character value or an actual code name for that","speaker":null,"is_sponsor":0},{"start_s":2323.28,"end_s":2329.76,"text":"project so we can later look up the data if we have to or reference it in a future video or","speaker":null,"is_sponsor":0},{"start_s":2329.76,"end_s":2334.48,"text":"something like that. It makes it a little bit easier on our side and then that way the data doesn't","speaker":null,"is_sponsor":0},{"start_s":2334.48,"end_s":2340.4,"text":"get lost in translation of oh I'm looking for this project but there's eight folders that are","speaker":null,"is_sponsor":0},{"start_s":2340.4,"end_s":2348.56,"text":"named similarly. It helps kind of basically organize that into a more clear overview of which","speaker":null,"is_sponsor":0},{"start_s":2348.56,"end_s":2356.8,"text":"project is which. Next is the session tag so if for example we are altering the values from our","speaker":null,"is_sponsor":0},{"start_s":2356.8,"end_s":2364.64,"text":"standard testing run so we're turning off anti aliasing or we're you know messing around with","speaker":null,"is_sponsor":0},{"start_s":2364.64,"end_s":2370.96,"text":"which preset we actually use. This helps us assign an actual tag to those specific sessions","speaker":null,"is_sponsor":0},{"start_s":2371.6,"end_s":2377.6,"text":"so that when we're looking at them later we can be like oh actually it's because the reason why the","speaker":null,"is_sponsor":0},{"start_s":2377.6,"end_s":2384.0,"text":"values are completely different is because this is the set that we that we ran with experimenting","speaker":null,"is_sponsor":0},{"start_s":2384.0,"end_s":2392.16,"text":"on anti aliasing or experimenting on you know running low versus high shadows or something like","speaker":null,"is_sponsor":0},{"start_s":2392.16,"end_s":2398.56,"text":"that so we can basically put in like no dash a and later when we go to look it up it'll actually","speaker":null,"is_sponsor":0},{"start_s":2398.56,"end_s":2406.4,"text":"have that listed on those specific sessions. Thirdly we have what's called the test parameter","speaker":null,"is_sponsor":0},{"start_s":2407.44,"end_s":2411.12,"text":"there's a bunch of different options in here that we have currently it's something we can edit","speaker":null,"is_sponsor":0},{"start_s":2411.12,"end_s":2418.56,"text":"whenever we need to but basically it assigns those sessions what the test parameters we","speaker":null,"is_sponsor":0},{"start_s":2419.44,"end_s":2426.64,"text":"hope to use for that specific session so for example for this project we're only running at","speaker":null,"is_sponsor":0},{"start_s":2426.64,"end_s":2435.36,"text":"1080p so we're going to be setting just on here games 1080. There are other options in here like","speaker":null,"is_sponsor":0},{"start_s":2435.36,"end_s":2442.64,"text":"DX11, RT, stuff like that. This doesn't actually alter any game settings themselves this is just","speaker":null,"is_sponsor":0},{"start_s":2442.64,"end_s":2449.2,"text":"for our internal use on all of our sessions so we know if we if that run was for 1440p if it was","speaker":null,"is_sponsor":0},{"start_s":2449.2,"end_s":2456.48,"text":"for ray tracing if it was for XCSS you know that's all those different types of visual parameters","speaker":null,"is_sponsor":0},{"start_s":2456.48,"end_s":2463.12,"text":"that are outside the norm basically. So for the project slug we are going to be putting in the","speaker":null,"is_sponsor":0},{"start_s":2463.12,"end_s":2474.64,"text":"one that's on the test plan which is 4080-vs-7900-XTX-2023. So once we have written our project","speaker":null,"is_sponsor":0},{"start_s":2474.64,"end_s":2480.32,"text":"slug in as well as our test parameters now we can get into actually editing which tests we're","speaker":null,"is_sponsor":0},{"start_s":2480.32,"end_s":2485.84,"text":"going to be running as well as some of them have different configurations that we can alter","speaker":null,"is_sponsor":0},{"start_s":2486.16,"end_s":2495.44,"text":"that are non-game tests or for assigning certain values of what the how the game is supposed to","speaker":null,"is_sponsor":0},{"start_s":2495.44,"end_s":2501.44,"text":"interact that sort of thing. So first we're going to cover blender benchmark we click on it here","speaker":null,"is_sponsor":0},{"start_s":2501.44,"end_s":2507.28,"text":"any of these ones have different test arguments in most cases so if we click on the blender","speaker":null,"is_sponsor":0},{"start_s":2507.28,"end_s":2513.04,"text":"benchmark we have a bunch of new values in here we can adjust so things like scheduler delay,","speaker":null,"is_sponsor":0},{"start_s":2513.04,"end_s":2519.84,"text":"recording delay and recording timeout. These are all in seconds we usually run a scheduler delay","speaker":null,"is_sponsor":0},{"start_s":2519.84,"end_s":2526.32,"text":"of five seconds just to make sure that the program or the game can have time to close and be","speaker":null,"is_sponsor":0},{"start_s":2526.32,"end_s":2531.68,"text":"satisfactory with Windows and all copacetic to be able to open again especially when it comes to","speaker":null,"is_sponsor":0},{"start_s":2531.68,"end_s":2537.12,"text":"steam when you have like cloud game saving enabled and stuff like that it has to take those few","speaker":null,"is_sponsor":0},{"start_s":2537.12,"end_s":2542.96,"text":"seconds to actually do the cloud save and sync it up what can happen in some instances if we","speaker":null,"is_sponsor":0},{"start_s":2543.04,"end_s":2550.8,"text":"don't set the scheduler delay is it will run into that bottleneck of oh but the cloud saving","speaker":null,"is_sponsor":0},{"start_s":2550.8,"end_s":2555.36,"text":"hasn't synced yet are you sure you want to load the game and it kind of gets everything all hung","speaker":null,"is_sponsor":0},{"start_s":2555.36,"end_s":2561.2,"text":"up so when we set it to five seconds it usually helps give steam enough time to finish that","speaker":null,"is_sponsor":0},{"start_s":2561.2,"end_s":2566.64,"text":"option and then actually continue on with the next run. So the fourth box here is the number of times","speaker":null,"is_sponsor":0},{"start_s":2566.72,"end_s":2574.4,"text":"that the test is meant to repeat so it's for that specific test not for the all the test as a whole","speaker":null,"is_sponsor":0},{"start_s":2574.4,"end_s":2580.72,"text":"so in this case we usually run three for all of our tests that doesn't mean that we only run three","speaker":null,"is_sponsor":0},{"start_s":2580.72,"end_s":2586.96,"text":"ever and that's it if we find that a result is outside of norm not within our our typical","speaker":null,"is_sponsor":0},{"start_s":2586.96,"end_s":2592.88,"text":"variance which usually about one to two percent we will do another run till we have three valid runs","speaker":null,"is_sponsor":0},{"start_s":2592.88,"end_s":2600.24,"text":"that are actually within that variance so we can confidently say that these are the numbers.","speaker":null,"is_sponsor":0},{"start_s":2600.24,"end_s":2605.76,"text":"So in blender benchmark we also have options for things like the scenes that we use because there's","speaker":null,"is_sponsor":0},{"start_s":2605.76,"end_s":2610.4,"text":"three different scenes that it actually runs through we're just going to be doing all of the","speaker":null,"is_sponsor":0},{"start_s":2610.4,"end_s":2618.16,"text":"scenes so which entails classroom junk shop as well as monster that's typically what it runs","speaker":null,"is_sponsor":0},{"start_s":2618.16,"end_s":2623.92,"text":"in the gooey version with the version that's available through markbench we actually use","speaker":null,"is_sponsor":0},{"start_s":2623.92,"end_s":2628.4,"text":"the command line interface so we're not going to see anything pop up on the screen for blender","speaker":null,"is_sponsor":0},{"start_s":2628.4,"end_s":2633.76,"text":"specifically but it is running in the background and we'll show that when we pop over to the actual","speaker":null,"is_sponsor":0},{"start_s":2633.76,"end_s":2639.36,"text":"command lines to see all the stuff that markbench is actually running then we have which versions","speaker":null,"is_sponsor":0},{"start_s":2639.36,"end_s":2644.8,"text":"that we can run blender on the blender benchmark on we're going to be doing 3.6 because that's what","speaker":null,"is_sponsor":0},{"start_s":2644.8,"end_s":2651.04,"text":"the test actually outlines the test plan outlines and then we're going to be testing the GPU not the","speaker":null,"is_sponsor":0},{"start_s":2651.04,"end_s":2656.16,"text":"CPU because that's what we want to test today is the actual GPUs then we're gonna go ahead and check","speaker":null,"is_sponsor":0},{"start_s":2656.16,"end_s":2664.88,"text":"it off and move on to the next one which is cyberpunk same thing five and three and then down here we","speaker":null,"is_sponsor":0},{"start_s":2664.88,"end_s":2672.24,"text":"have for what our keras ip's should be so i can actually talk to them some of our harnesses don't","speaker":null,"is_sponsor":0},{"start_s":2672.32,"end_s":2678.64,"text":"use keras still um it's something that we're kind of developing as we've been getting into these","speaker":null,"is_sponsor":0},{"start_s":2678.64,"end_s":2684.64,"text":"future automations and stuff like that to be able to add these tools back into the toolbox of","speaker":null,"is_sponsor":0},{"start_s":2685.44,"end_s":2691.04,"text":"basically streamlining how each individual harness is so when we get down to red dead","speaker":null,"is_sponsor":0},{"start_s":2691.04,"end_s":2694.8,"text":"for example it's not going to have that listed right now but we're hoping to have that in the","speaker":null,"is_sponsor":0},{"start_s":2694.8,"end_s":2699.36,"text":"future so we don't have to do anything else on this one just five and three go down to dota","speaker":null,"is_sponsor":0},{"start_s":2699.36,"end_s":2706.24,"text":"same thing five and three and we go to red dead we're just going to be setting red dead to five","speaker":null,"is_sponsor":0},{"start_s":2706.24,"end_s":2712.56,"text":"and three again but here all we have is preset which is current this can't be changed it's a","speaker":null,"is_sponsor":0},{"start_s":2712.56,"end_s":2719.04,"text":"legacy harness that we've we're currently in progress of updating to use keras as well as","speaker":null,"is_sponsor":0},{"start_s":2719.04,"end_s":2724.4,"text":"some of the other things that nick will be discussing as far as our levels of automation","speaker":null,"is_sponsor":0},{"start_s":2724.4,"end_s":2728.96,"text":"goes now we've got everything set up for mark bench it's time to hit the go button and while","speaker":null,"is_sponsor":0},{"start_s":2728.96,"end_s":2732.88,"text":"this is running the benchmarks i'm going to kick it back over to nick and he's going to talk about","speaker":null,"is_sponsor":0},{"start_s":2732.88,"end_s":2745.12,"text":"the levels of automation so coming back to automation we're going to focus on the test","speaker":null,"is_sponsor":0},{"start_s":2745.12,"end_s":2750.88,"text":"harnesses and talk about the degrees of automation because one of the most annoying questions to try","speaker":null,"is_sponsor":0},{"start_s":2750.88,"end_s":2757.2,"text":"and answer is is it automated when someone asks you that it really depends the answer to that","speaker":null,"is_sponsor":0},{"start_s":2757.2,"end_s":2763.6,"text":"depends on the scope is a bunch of power shell scripts that run your game and then output something","speaker":null,"is_sponsor":0},{"start_s":2763.6,"end_s":2769.04,"text":"to a log file considered automated or is it only automated if you can press a button and walk away","speaker":null,"is_sponsor":0},{"start_s":2769.04,"end_s":2775.28,"text":"and it handles every edge case that you could possibly think of to help talk about automation","speaker":null,"is_sponsor":0},{"start_s":2775.28,"end_s":2781.92,"text":"internally we've come up with a milestone system and so we come up with the milestones of one two","speaker":null,"is_sponsor":0},{"start_s":2781.92,"end_s":2788.24,"text":"three and four so when we speak of a test harness taking a game for example a game harness that has","speaker":null,"is_sponsor":0},{"start_s":2788.24,"end_s":2793.92,"text":"a level one automation it simply just launches and runs the benchmark id like akin to that just a","speaker":null,"is_sponsor":0},{"start_s":2793.92,"end_s":2798.64,"text":"power shell script doing something very simple and then when we reach a milestone of level two","speaker":null,"is_sponsor":0},{"start_s":2799.28,"end_s":2804.8,"text":"it can do a lot more such as dealing with advertisements or pop-ups in one of their","speaker":null,"is_sponsor":0},{"start_s":2804.8,"end_s":2811.2,"text":"launchers or it can report on game settings and then the next level to that is level three is","speaker":null,"is_sponsor":0},{"start_s":2811.2,"end_s":2816.48,"text":"where it can actually configure the game for itself and also reach out to maybe some more","speaker":null,"is_sponsor":0},{"start_s":2816.48,"end_s":2822.96,"text":"services like Keras or whatnot our definitions of one two and three are fairly loose because every","speaker":null,"is_sponsor":0},{"start_s":2822.96,"end_s":2828.24,"text":"game is a little different some games are a lot easier to report on resolution because the it","speaker":null,"is_sponsor":0},{"start_s":2828.24,"end_s":2834.8,"text":"might be reported in a .ini file that we can grab from app data where others might only store their","speaker":null,"is_sponsor":0},{"start_s":2834.8,"end_s":2842.72,"text":"settings in some proprietary binary format that we can't even read without some finagling so this","speaker":null,"is_sponsor":0},{"start_s":2842.72,"end_s":2848.24,"text":"is roughly how we categorize it just so that we can at least internally when someone asks you is it","speaker":null,"is_sponsor":0},{"start_s":2848.24,"end_s":2855.04,"text":"automated we can say yes to a particular degree and then we get down to number four which is the","speaker":null,"is_sponsor":0},{"start_s":2855.04,"end_s":2860.88,"text":"future this this category encompasses things that we can't do yet we have something we call our","speaker":null,"is_sponsor":0},{"start_s":2860.88,"end_s":2865.44,"text":"automation toolbox where we include all the different services such as the keras optical","speaker":null,"is_sponsor":0},{"start_s":2865.44,"end_s":2871.92,"text":"character recognition uh and uh pie direct input or pie auto gooey it's all the different things","speaker":null,"is_sponsor":0},{"start_s":2871.92,"end_s":2879.04,"text":"that we can use in our test scripts to achieve the automation of a game so one to three incorporates","speaker":null,"is_sponsor":0},{"start_s":2879.04,"end_s":2883.44,"text":"those ones i've listed where four is some of the tools that we haven't come to yet the future is","speaker":null,"is_sponsor":0},{"start_s":2883.44,"end_s":2888.0,"text":"where we'll start to talk about ndi and how we use it for video recording as well as computer","speaker":null,"is_sponsor":0},{"start_s":2888.48,"end_s":2893.68,"text":"when we started automating our game suite we relied on games that had a canned benchmark","speaker":null,"is_sponsor":0},{"start_s":2893.68,"end_s":2897.44,"text":"due to the fact that it's a lot easier to automate as you just have to navigate the menus and then","speaker":null,"is_sponsor":0},{"start_s":2897.44,"end_s":2902.48,"text":"start the benchmark however the problem with that is that not all built-in benchmarking","speaker":null,"is_sponsor":0},{"start_s":2902.48,"end_s":2907.84,"text":"utilities are representative gameplay and not all games come with a canned benchmark which limits","speaker":null,"is_sponsor":0},{"start_s":2907.84,"end_s":2912.56,"text":"our selection for our game suite to expand our game suite we started looking at games such as","speaker":null,"is_sponsor":0},{"start_s":2912.56,"end_s":2917.2,"text":"dota 2 and rocket league that have a very robust replay system in the future we're looking to","speaker":null,"is_sponsor":0},{"start_s":2917.2,"end_s":2922.4,"text":"incorporate computer vision tools such as depth estimation and object detection to expand our game","speaker":null,"is_sponsor":0},{"start_s":2922.4,"end_s":2928.0,"text":"selection to any game before we can use depth estimation or object detection we need to get a","speaker":null,"is_sponsor":0},{"start_s":2928.0,"end_s":2934.56,"text":"video stream off of the test bench to another computer to run the machine learning to do so we","speaker":null,"is_sponsor":0},{"start_s":2934.56,"end_s":2941.84,"text":"use ndi or network device interface to get video stream from the test bench to the machine learning","speaker":null,"is_sponsor":0},{"start_s":2941.84,"end_s":2950.0,"text":"computer ndi allows us to convert HDMI signal to video over ip which then can be picked up by any","speaker":null,"is_sponsor":0},{"start_s":2950.0,"end_s":2956.08,"text":"other client that's on the subnet for example on our computer on the cart we have a client that's","speaker":null,"is_sponsor":0},{"start_s":2956.08,"end_s":2961.84,"text":"picking up the ndi stream being sent from the test bench which then runs our experiment with depth","speaker":null,"is_sponsor":0},{"start_s":2961.84,"end_s":2966.64,"text":"estimation and depth estimation is the task of measuring the distance of each pixel relative","speaker":null,"is_sponsor":0},{"start_s":2966.64,"end_s":2971.92,"text":"to the camera depth estimation could be used to instruct a player to walk down a hallway by","speaker":null,"is_sponsor":0},{"start_s":2971.92,"end_s":2976.24,"text":"chasing the darkest point on the screen we are also experimenting with other techniques such as","speaker":null,"is_sponsor":0},{"start_s":2976.24,"end_s":2982.72,"text":"optical character recognition and object detection on a video stream the ndi encoder that we're using","speaker":null,"is_sponsor":0},{"start_s":2982.72,"end_s":2990.32,"text":"is an n6 from kill of you which has an HDMI in HDMI out and then your Ethernet a particular nice to","speaker":null,"is_sponsor":0},{"start_s":2990.32,"end_s":2995.68,"text":"have on this unit is the small display on the front which allows you to see the activity of the","speaker":null,"is_sponsor":0},{"start_s":2995.68,"end_s":3002.4,"text":"unit as well as the ip and what video stream it's sending out okay so all of our benchmarks have","speaker":null,"is_sponsor":0},{"start_s":3002.4,"end_s":3010.08,"text":"finished let's take a look at the results as you can see here it actually says we've ran","speaker":null,"is_sponsor":0},{"start_s":3010.08,"end_s":3016.16,"text":"zero zero for dota 2 so we didn't actually get any valid runs so we're gonna actually have to do some","speaker":null,"is_sponsor":0},{"start_s":3016.16,"end_s":3024.24,"text":"troubleshooting figure out why so that's the thing is that markbench allows us to identify some very","speaker":null,"is_sponsor":0},{"start_s":3024.24,"end_s":3030.16,"text":"early on possible bugs that might occur with the run and it'll usually terminate so if we scroll up","speaker":null,"is_sponsor":0},{"start_s":3030.16,"end_s":3036.4,"text":"here you can actually see we get to a point where it says test failed and it's for this run here so","speaker":null,"is_sponsor":0},{"start_s":3036.4,"end_s":3041.52,"text":"it says game didn't start in time check settings and try again so this is something that we've","speaker":null,"is_sponsor":0},{"start_s":3041.52,"end_s":3046.64,"text":"programmed into some of our logging so we're gonna have to actually go into our dota 2 settings and","speaker":null,"is_sponsor":0},{"start_s":3046.64,"end_s":3052.08,"text":"it could be that maybe something got reset when we swapped cables or something like that so we're","speaker":null,"is_sponsor":0},{"start_s":3052.08,"end_s":3057.76,"text":"gonna take a look at that and get it set back up to the correct settings and try running it again","speaker":null,"is_sponsor":0},{"start_s":3058.72,"end_s":3063.76,"text":"so we're in dota 2 let's take a quick look at the setting and see what might have went wrong here","speaker":null,"is_sponsor":0},{"start_s":3064.64,"end_s":3073.6,"text":"so we go to video ah there's the problem it changed itself back to 4k it's a good chance that","speaker":null,"is_sponsor":0},{"start_s":3073.6,"end_s":3078.16,"text":"while we were swapping some cables around or something like that when we were doing the initial","speaker":null,"is_sponsor":0},{"start_s":3078.16,"end_s":3084.32,"text":"setup I probably swapped itself back to native resolution and so it caused our markbench runs","speaker":null,"is_sponsor":0},{"start_s":3084.32,"end_s":3092.88,"text":"to fail so let's change that back and then we'll rerun it here the upside of this is is we don't","speaker":null,"is_sponsor":0},{"start_s":3092.88,"end_s":3099.36,"text":"have to run every test we just ran again we can actually unmark everything else and just specifically","speaker":null,"is_sponsor":0},{"start_s":3099.36,"end_s":3104.64,"text":"target dota 2 now so let's go ahead and get that started and hopefully we'll come back with some","speaker":null,"is_sponsor":0},{"start_s":3104.64,"end_s":3116.08,"text":"actual runs","speaker":null,"is_sponsor":0},{"start_s":3126.4,"end_s":3132.16,"text":"so looks like we got three runs done for dota 2 now so we can actually upload all this data to our","speaker":null,"is_sponsor":0},{"start_s":3132.16,"end_s":3138.8,"text":"data ingress server and start trimming so let's get that done now we've uploaded the data to our","speaker":null,"is_sponsor":0},{"start_s":3138.8,"end_s":3145.36,"text":"server we can pull it up on grafana which is our data visualization dashboard from here we can pull","speaker":null,"is_sponsor":0},{"start_s":3145.36,"end_s":3152.24,"text":"up all of projects that we've run any sessions that have occurred from basically any of the uploads","speaker":null,"is_sponsor":0},{"start_s":3152.96,"end_s":3159.92,"text":"so on grafana itself we have the ability to check our projects as well as in this specific","speaker":null,"is_sponsor":0},{"start_s":3159.92,"end_s":3167.68,"text":"dashboard we're looking at GPUs so for our current project it lists the 7900 XTX as well as the 4080","speaker":null,"is_sponsor":0},{"start_s":3168.64,"end_s":3174.0,"text":"for tests it lists all the different tests that we've run for a given project and then there's","speaker":null,"is_sponsor":0},{"start_s":3174.0,"end_s":3179.12,"text":"parameters so in this case we only specify games 1080 so that's the only one that shows up on the list","speaker":null,"is_sponsor":0},{"start_s":3180.4,"end_s":3187.12,"text":"and then there is the option for the bench which essentially that takes the actual PC name when","speaker":null,"is_sponsor":0},{"start_s":3188.08,"end_s":3195.68,"text":"when markbench logs it to help us identify which system we actually ran on and then lastly is the","speaker":null,"is_sponsor":0},{"start_s":3195.68,"end_s":3202.64,"text":"tag which is blank because we didn't label any tags for anything that we needed so from here we","speaker":null,"is_sponsor":0},{"start_s":3202.64,"end_s":3208.72,"text":"can pull up any of the sessions so let's take a look at dota 2 first here so as you can see here","speaker":null,"is_sponsor":0},{"start_s":3208.72,"end_s":3212.96,"text":"it's kind of a little bit of a mess if you don't know what you're looking at here but effectively","speaker":null,"is_sponsor":0},{"start_s":3212.96,"end_s":3218.72,"text":"this is the beginning load this is the main menu and then we have our little load screen","speaker":null,"is_sponsor":0},{"start_s":3219.28,"end_s":3225.52,"text":"and then we have that initial bit before the actual benchmark occurs the actual bits that we need","speaker":null,"is_sponsor":0},{"start_s":3225.52,"end_s":3234.16,"text":"are from this valley to this valley and then this last bit here is when it goes back to the main","speaker":null,"is_sponsor":0},{"start_s":3234.16,"end_s":3242.48,"text":"menu so that's how we how we kind of know which game it's all specific you kind of have to know","speaker":null,"is_sponsor":0},{"start_s":3242.48,"end_s":3247.44,"text":"what you're looking for at the beginning of like determining oh which part do I actually","speaker":null,"is_sponsor":0},{"start_s":3247.44,"end_s":3254.48,"text":"need to use so what we all all we have to do at this point is just drag from there to there","speaker":null,"is_sponsor":0},{"start_s":3255.6,"end_s":3262.8,"text":"apply the correct time range sometimes it skews it by a second if we do that that gives us our run","speaker":null,"is_sponsor":0},{"start_s":3263.76,"end_s":3270.56,"text":"and then basically I just do that for all the sessions that we have and then once we finish","speaker":null,"is_sponsor":0},{"start_s":3270.56,"end_s":3277.76,"text":"trimming them we take the data and we put it into our trim session tool on the harbour","speaker":null,"is_sponsor":0},{"start_s":3278.72,"end_s":3286.32,"text":"LTT Labs page and that basically gives us a graph ideally we want all three sessions to be","speaker":null,"is_sponsor":0},{"start_s":3287.36,"end_s":3294.48,"text":"basically almost the same however as I outlined previously red deads a little different there's","speaker":null,"is_sponsor":0},{"start_s":3294.48,"end_s":3301.28,"text":"a bit of rng that occurs during each run so they don't look the exact same on here but the end","speaker":null,"is_sponsor":0},{"start_s":3301.28,"end_s":3307.6,"text":"result of the averages in the first should be the same or within that one to two percent so at this","speaker":null,"is_sponsor":0},{"start_s":3307.6,"end_s":3312.96,"text":"point all we gotta do now is trim all the data so we can send it over to sammy and she can start","speaker":null,"is_sponsor":0},{"start_s":3312.96,"end_s":3318.24,"text":"working on getting those graphs created so let's get to it once the game has finished and the data","speaker":null,"is_sponsor":0},{"start_s":3318.24,"end_s":3323.12,"text":"is uploaded to the server we need to mark an in and out on the benchmark data to eliminate the","speaker":null,"is_sponsor":0},{"start_s":3323.2,"end_s":3328.08,"text":"menus and any loading screens from when we calculate the FPS currently john's doing this","speaker":null,"is_sponsor":0},{"start_s":3328.08,"end_s":3333.04,"text":"manually in grafana by literally looking at the pattern of the frame times in the FPS we're trying","speaker":null,"is_sponsor":0},{"start_s":3333.04,"end_s":3337.76,"text":"to help them out by having machine learning doing something similar by training a model based on","speaker":null,"is_sponsor":0},{"start_s":3337.76,"end_s":3344.0,"text":"the behavior of the GPU its temperature and core clocks and memory usage as well as the pattern","speaker":null,"is_sponsor":0},{"start_s":3344.0,"end_s":3350.0,"text":"and frame times to detect the in and out of a benchmark however that still can be faulty so","speaker":null,"is_sponsor":0},{"start_s":3350.0,"end_s":3354.88,"text":"we're looking at eliminating it all together by using computer vision from the ndi streams as you","speaker":null,"is_sponsor":0},{"start_s":3354.88,"end_s":3360.56,"text":"saw earlier to mark an in and out very precisely as it's close to the frame when the game first","speaker":null,"is_sponsor":0},{"start_s":3360.56,"end_s":3366.32,"text":"loads as we can find we're also looking to apply machine learning and models trained on GPU behavior","speaker":null,"is_sponsor":0},{"start_s":3366.32,"end_s":3372.96,"text":"to identify erroneous runs or bad runs based on historical data now that we've finished trimming","speaker":null,"is_sponsor":0},{"start_s":3372.96,"end_s":3379.6,"text":"all the sessions grafana will actually spit out a list of all the different trim session ids","speaker":null,"is_sponsor":0},{"start_s":3380.0,"end_s":3385.36,"text":"so we can use them to generate our game test report they might ask us how do you know these","speaker":null,"is_sponsor":0},{"start_s":3385.36,"end_s":3391.52,"text":"are good trim sessions maybe your settings were bad or maybe you know it was just every run was","speaker":null,"is_sponsor":0},{"start_s":3391.52,"end_s":3398.24,"text":"terrible that's where our other dashboard comes in redash this basically shows historical trim","speaker":null,"is_sponsor":0},{"start_s":3398.24,"end_s":3403.92,"text":"data that we can use to help verify that these runs are in fact the runs that we need this is","speaker":null,"is_sponsor":0},{"start_s":3403.92,"end_s":3408.64,"text":"something that gets generated over time if we have a game that we haven't done a whole lot of","speaker":null,"is_sponsor":0},{"start_s":3408.64,"end_s":3415.52,"text":"trimming on it will be there will be less numbers to be able to verify against but ideally we want","speaker":null,"is_sponsor":0},{"start_s":3415.52,"end_s":3422.24,"text":"the trend to be the same or similar so in this case if we're looking at cyberpunk on a 4080","speaker":null,"is_sponsor":0},{"start_s":3423.2,"end_s":3434.32,"text":"on 1080p we're getting 190 average ish to 180 average ish and 154 to 153 for a 1%","speaker":null,"is_sponsor":0},{"start_s":3435.12,"end_s":3442.8,"text":"so what we can do now is we can use our we can compare our trimmed sessions versus historical","speaker":null,"is_sponsor":0},{"start_s":3442.8,"end_s":3447.6,"text":"data to verify that it's all correct so what we're going to do is we're going to go on to our","speaker":null,"is_sponsor":0},{"start_s":3447.6,"end_s":3451.76,"text":"harbor Labs tools again and we're going to go through and we're going to start creating an","speaker":null,"is_sponsor":0},{"start_s":3451.76,"end_s":3457.12,"text":"actual test report so what we do is we want GPU by component we're not going to include a line","speaker":null,"is_sponsor":0},{"start_s":3457.12,"end_s":3464.0,"text":"chart because we were just looking at those so we hit create now what we do is we take all of these","speaker":null,"is_sponsor":0},{"start_s":3464.08,"end_s":3473.84,"text":"session IDs and we hit add let me do it for the other card and so what that does now is it actually","speaker":null,"is_sponsor":0},{"start_s":3473.84,"end_s":3480.32,"text":"generates a report for us to be able to see visually both with a bar graph as well as just the numbers","speaker":null,"is_sponsor":0},{"start_s":3481.6,"end_s":3488.0,"text":"what our actual results are so if we're looking here at cyberpunk 2077 on 1080p","speaker":null,"is_sponsor":0},{"start_s":3488.88,"end_s":3496.48,"text":"the 4080 for these three runs that we just did has an average of 193 and a 1% low of 154","speaker":null,"is_sponsor":0},{"start_s":3497.36,"end_s":3503.2,"text":"so we're actually above our previous average now these numbers were with a reference card","speaker":null,"is_sponsor":0},{"start_s":3503.2,"end_s":3509.44,"text":"obviously the accelerates a little bit over a clock so it stands for reason that 193 seems","speaker":null,"is_sponsor":0},{"start_s":3509.44,"end_s":3517.2,"text":"accurate and so we can keep scrolling down so taking a look at dota 2 we can see that the","speaker":null,"is_sponsor":0},{"start_s":3517.2,"end_s":3522.88,"text":"numbers appear to line up with what i have experienced in the past we don't have a whole","speaker":null,"is_sponsor":0},{"start_s":3522.88,"end_s":3528.32,"text":"lot of historical data on it currently because it is one of our newer harnesses that we actually","speaker":null,"is_sponsor":0},{"start_s":3528.32,"end_s":3533.84,"text":"have been developing and i've kind of just been beta testing but based on my personal experience","speaker":null,"is_sponsor":0},{"start_s":3533.84,"end_s":3540.88,"text":"with it these numbers do appear to be accurate okay so lastly let's take a look at red dead 2 here","speaker":null,"is_sponsor":0},{"start_s":3541.68,"end_s":3547.68,"text":"so now we've taken a look at some of the numbers","speaker":null,"is_sponsor":0},{"start_s":3548.72,"end_s":3555.04,"text":"we can publish this report and get it sent off to sammy to start generating the graphs","speaker":null,"is_sponsor":0},{"start_s":3555.76,"end_s":3560.64,"text":"so what we need to do first is put the report name which is usually the project slug","speaker":null,"is_sponsor":0},{"start_s":3561.2,"end_s":3576.16,"text":"um so 40 versus 7900 x tx 2023 results in some cases if we decide to have separate reports based","speaker":null,"is_sponsor":0},{"start_s":3576.16,"end_s":3582.64,"text":"on like ray tracing versus non ray tracing stuff like that even though those headers of like the","speaker":null,"is_sponsor":0},{"start_s":3582.64,"end_s":3590.32,"text":"games 1080 um help us filter that if in some cases we just want to look at those bar graphs separately","speaker":null,"is_sponsor":0},{"start_s":3590.32,"end_s":3598.24,"text":"and the data that they've generated sometimes we will append uh rt results or dlss results or","speaker":null,"is_sponsor":0},{"start_s":3604.96,"end_s":3611.2,"text":"it's just a description of effectively what the reports based on so we're just going to do results","speaker":null,"is_sponsor":0},{"start_s":3611.2,"end_s":3623.76,"text":"for 40 80 dash vs dash 7900 x tx dash 203 person then for a reference component","speaker":null,"is_sponsor":0},{"start_s":3624.32,"end_s":3632.4,"text":"basically what this does is it generates a relative percentage um based on the data so","speaker":null,"is_sponsor":0},{"start_s":3632.4,"end_s":3642.56,"text":"effectively like oh we can see that the 40 80 is negative 1.62 percent in performance versus","speaker":null,"is_sponsor":0},{"start_s":3642.56,"end_s":3648.64,"text":"the 7900 x tx we can choose whatever component we want it's helpful for when we are looking at","speaker":null,"is_sponsor":0},{"start_s":3648.64,"end_s":3653.76,"text":"our big GPU reviews because we can actually be like okay we want to focus on this component it's","speaker":null,"is_sponsor":0},{"start_s":3653.76,"end_s":3661.12,"text":"not the top of the stack but like for example the rx 7600 we want to take a look at and see","speaker":null,"is_sponsor":0},{"start_s":3661.12,"end_s":3666.64,"text":"how it compares to everything else and it'll allow us to to basically find these relatives a","speaker":null,"is_sponsor":0},{"start_s":3666.64,"end_s":3673.84,"text":"lot easier than doing the math ourselves so it's very helpful in that regard um so let's go ahead","speaker":null,"is_sponsor":0},{"start_s":3673.84,"end_s":3680.8,"text":"and we're going to click publish now this will create a report so if we actually go over here","speaker":null,"is_sponsor":0},{"start_s":3680.8,"end_s":3687.6,"text":"we can see we actually have the 40 80 versus 7900 x tx results we click on that that basically gives","speaker":null,"is_sponsor":0},{"start_s":3687.68,"end_s":3693.12,"text":"us the same report um that we were just looking at so i'm just going to copy the url here","speaker":null,"is_sponsor":0},{"start_s":3694.56,"end_s":3701.36,"text":"into an excel sheet that i was working on go ahead and save this we'll get this sent off to her","speaker":null,"is_sponsor":0},{"start_s":3701.36,"end_s":3706.72,"text":"and then she can start creating our graphs hi my name is sammy i'm the data visualization specialist","speaker":null,"is_sponsor":0},{"start_s":3707.68,"end_s":3712.8,"text":"for the lab um i started a few months ago and ever since i started we've made some changes to","speaker":null,"is_sponsor":0},{"start_s":3712.8,"end_s":3718.72,"text":"the graphs um previously they were done by the writing team um but since then i've taken over","speaker":null,"is_sponsor":0},{"start_s":3718.72,"end_s":3724.8,"text":"that role um and yeah i'm here to walk you through uh some of the design choices that we made as well","speaker":null,"is_sponsor":0},{"start_s":3724.8,"end_s":3731.04,"text":"as um the process by which we make these graphs for our videos previously the graphs that we had","speaker":null,"is_sponsor":0},{"start_s":3731.04,"end_s":3738.24,"text":"had a lot of limitations in design choices um in terms of placement or colors or fonts that we used","speaker":null,"is_sponsor":0},{"start_s":3738.24,"end_s":3742.4,"text":"mostly because we were making them in excel nowadays we're making them in a combination","speaker":null,"is_sponsor":0},{"start_s":3742.48,"end_s":3748.96,"text":"of python and uh doby illustrator as well as adobe illustrator plugins mainly um one plugin that's","speaker":null,"is_sponsor":0},{"start_s":3748.96,"end_s":3754.24,"text":"specialized for graphing uh called datlion datlion allows us to make customizations that we weren't","speaker":null,"is_sponsor":0},{"start_s":3754.24,"end_s":3760.32,"text":"able to do previously um so one of the first things i did uh was use the style guide that was","speaker":null,"is_sponsor":0},{"start_s":3760.32,"end_s":3766.16,"text":"developed by our creator warehouse graphic designers um they had kind of set out a palette as well as","speaker":null,"is_sponsor":0},{"start_s":3766.16,"end_s":3771.04,"text":"some fonts that i could use and toy around with i tried to maintain a design design philosophy","speaker":null,"is_sponsor":0},{"start_s":3771.04,"end_s":3776.72,"text":"that focused on eligibility um so as we've kind of gone through multiple iterations of the graphs","speaker":null,"is_sponsor":0},{"start_s":3776.72,"end_s":3781.36,"text":"since i started i've made changes to emphasize on that based off a lot of feedback that we've","speaker":null,"is_sponsor":0},{"start_s":3781.36,"end_s":3787.12,"text":"gotten from our viewers in terms of design aesthetic um one of the things that i really like to focus on","speaker":null,"is_sponsor":0},{"start_s":3787.12,"end_s":3794.96,"text":"was an almost laboratorial or chalkboard kind of vibe um we wanted to evoke something that was very","speaker":null,"is_sponsor":0},{"start_s":3794.96,"end_s":3802.16,"text":"similar to the nasa kind of retro space age kind of look we also incorporated a more modern twist","speaker":null,"is_sponsor":0},{"start_s":3802.16,"end_s":3808.64,"text":"um inspired by the cyberpunk genre as well as more sci-fi films that have come out in recent years","speaker":null,"is_sponsor":0},{"start_s":3808.64,"end_s":3814.56,"text":"we've focused on using as well a palette that is uh as accessible as possible for example the choices","speaker":null,"is_sponsor":0},{"start_s":3814.56,"end_s":3820.16,"text":"um of which colors come first in our bar graphs we chose between colors that were very contrasting","speaker":null,"is_sponsor":0},{"start_s":3820.16,"end_s":3826.08,"text":"another thing we did was uh we put the colors into um different accessibility tools to kind of","speaker":null,"is_sponsor":0},{"start_s":3826.08,"end_s":3832.88,"text":"measure how visible our color palette would be for people with different types of color blindness","speaker":null,"is_sponsor":0},{"start_s":3832.88,"end_s":3838.4,"text":"um and we've essentially developed a order and a palette that is as accessible as possible across","speaker":null,"is_sponsor":0},{"start_s":3838.4,"end_s":3844.88,"text":"as many different types of um color blindness as possible in order to make our graphs legible","speaker":null,"is_sponsor":0},{"start_s":3844.88,"end_s":3848.96,"text":"so this is an example of one of the game graph templates that i've developed um one of the first","speaker":null,"is_sponsor":0},{"start_s":3848.96,"end_s":3853.68,"text":"things that i noticed when i came in was that the previous graphs um their title didn't really jump","speaker":null,"is_sponsor":0},{"start_s":3853.68,"end_s":3858.24,"text":"out at you one of the things i really wanted to do is make use of like kind of like the very like","speaker":null,"is_sponsor":0},{"start_s":3858.24,"end_s":3864.64,"text":"stampy Labs font that we have to kind of grab attention um for me one of the things that i want","speaker":null,"is_sponsor":0},{"start_s":3864.64,"end_s":3870.88,"text":"to focus on is like what is what am i looking at and number two what is this me and so the first","speaker":null,"is_sponsor":0},{"start_s":3870.88,"end_s":3877.52,"text":"part what am i looking at is um kind of answered immediately by this very bold and very loud um","speaker":null,"is_sponsor":0},{"start_s":3878.48,"end_s":3884.96,"text":"title or header um and so we've added some design elements sometimes they'll be added in especially","speaker":null,"is_sponsor":0},{"start_s":3884.96,"end_s":3889.44,"text":"if a graph is you know maybe maybe a little more bare if they're only like two or three","speaker":null,"is_sponsor":0},{"start_s":3889.44,"end_s":3895.92,"text":"things that we're comparing across but you know if they're 12 13 CPUs GPUs whatever that we're","speaker":null,"is_sponsor":0},{"start_s":3895.92,"end_s":3901.92,"text":"comparing across then obviously we won't have no space for that we then have the bars themselves","speaker":null,"is_sponsor":0},{"start_s":3902.0,"end_s":3909.68,"text":"um the bars themselves are sorted now by uh best to worst um we'll still keep the higher","speaker":null,"is_sponsor":0},{"start_s":3909.68,"end_s":3916.56,"text":"is better um in the subtitle of the graph however um we just want to make it like at a glance more","speaker":null,"is_sponsor":0},{"start_s":3916.56,"end_s":3923.68,"text":"obvious um and another design element that the editors have helped me incorporate was um a marker","speaker":null,"is_sponsor":0},{"start_s":3923.68,"end_s":3927.68,"text":"that kind of pulses right next to the GPU that we're talking about previously one of our concerns","speaker":null,"is_sponsor":0},{"start_s":3927.68,"end_s":3932.72,"text":"and the reason why we weren't able to uh sort by best to worst was because we were worried that if","speaker":null,"is_sponsor":0},{"start_s":3932.72,"end_s":3938.16,"text":"we swapped around the order of every single GPU it'd be hard to follow oh which one is where um and","speaker":null,"is_sponsor":0},{"start_s":3938.16,"end_s":3943.12,"text":"so that kind of solved our problem and ever since then we've kind of adopted this philosophy of sorting","speaker":null,"is_sponsor":0},{"start_s":3943.12,"end_s":3947.84,"text":"by best to worst as best as possible we've kept the grid lines kind of sparse because we want to","speaker":null,"is_sponsor":0},{"start_s":3947.84,"end_s":3953.76,"text":"make sure that the graph isn't overcrowded with different design elements um and another design","speaker":null,"is_sponsor":0},{"start_s":3953.76,"end_s":3960.24,"text":"choice we've made particularly with the game graphs and the FPS um graphs is that we've put","speaker":null,"is_sponsor":0},{"start_s":3960.24,"end_s":3965.2,"text":"one percent low on top and we've made it a bolder color than the average so we've decided that one","speaker":null,"is_sponsor":0},{"start_s":3965.2,"end_s":3969.68,"text":"percent lows are more indicative of overall gaming performance because average values can get pulled","speaker":null,"is_sponsor":0},{"start_s":3969.68,"end_s":3975.52,"text":"up by um high FPS anomalies um that might happen over the course of a game when the one percent","speaker":null,"is_sponsor":0},{"start_s":3975.52,"end_s":3979.28,"text":"lows are really low uh the gameplay ends up appearing as really choppy and so that's one of","speaker":null,"is_sponsor":0},{"start_s":3979.28,"end_s":3983.2,"text":"the reasons why we think that actually the one percent lows the statistic or the data point","speaker":null,"is_sponsor":0},{"start_s":3983.2,"end_s":3988.4,"text":"that we really want to highlight instead and so one thing that's been added since we've made a few","speaker":null,"is_sponsor":0},{"start_s":3988.4,"end_s":3994.4,"text":"changes around the lab is that we've added this uh project slug that follows a project from the","speaker":null,"is_sponsor":0},{"start_s":3994.4,"end_s":3999.28,"text":"beginning to the end and another thing that we've added is also this version number um so we will","speaker":null,"is_sponsor":0},{"start_s":3999.28,"end_s":4005.28,"text":"change and update the version number every time of new set of graphs is cut um and that will also","speaker":null,"is_sponsor":0},{"start_s":4005.28,"end_s":4012.08,"text":"ensure that we have the most updated version of graphs in our video and finally we have our","speaker":null,"is_sponsor":0},{"start_s":4012.08,"end_s":4018.32,"text":"Labs logo in the bottom bottom right uh this Labs logo essentially tells viewers that this graph was","speaker":null,"is_sponsor":0},{"start_s":4018.32,"end_s":4025.28,"text":"developed by Labs with Labs data so now we're gonna get started with our graphs um i've already gotten","speaker":null,"is_sponsor":0},{"start_s":4025.28,"end_s":4031.84,"text":"a uh spreadsheet from adam with the list of graphs that he wants me to do uh three of them being","speaker":null,"is_sponsor":0},{"start_s":4032.72,"end_s":4039.68,"text":"the game graphs so one for each game and then uh the blender benchmark graph and as you can see here","speaker":null,"is_sponsor":0},{"start_s":4040.32,"end_s":4049.84,"text":"it says um cyberpunk 2020 2077 uh red dead redemption 2 dota 2 blender 3.6.0 and higher","speaker":null,"is_sponsor":0},{"start_s":4049.84,"end_s":4057.6,"text":"is better and includes all of the values for the presets as well as the resolution x-axis label","speaker":null,"is_sponsor":0},{"start_s":4057.6,"end_s":4064.96,"text":"project label version number and all of that so that allows me to know what exactly writers want","speaker":null,"is_sponsor":0},{"start_s":4065.04,"end_s":4072.48,"text":"included in their video and what data they need as well and there's also room for writers to add","speaker":null,"is_sponsor":0},{"start_s":4072.48,"end_s":4078.64,"text":"descriptions like if they want to include some kind of like asterisks at the bottom with notes or if","speaker":null,"is_sponsor":0},{"start_s":4078.64,"end_s":4084.24,"text":"they want to cut out a certain GPU or CPU or if they want like another version of the exact same","speaker":null,"is_sponsor":0},{"start_s":4084.24,"end_s":4090.4,"text":"graph with extra information or less information included we're just gonna open up a new file","speaker":null,"is_sponsor":0},{"start_s":4090.88,"end_s":4100.24,"text":"and that will be one layer for each graph as well and i will copy and paste um the layers from our","speaker":null,"is_sponsor":0},{"start_s":4100.24,"end_s":4106.96,"text":"template so for our first three graphs it's going to be a game graph and so for graph number one","speaker":null,"is_sponsor":0},{"start_s":4106.96,"end_s":4117.92,"text":"just gonna paste it in and align it you can see that there are already numbers in the template","speaker":null,"is_sponsor":0},{"start_s":4117.92,"end_s":4124.4,"text":"that's just to help um serve as a placeholder so that i know what graph is which uh that data","speaker":null,"is_sponsor":0},{"start_s":4124.4,"end_s":4131.12,"text":"will be scrubbed later and then the final graph is our blend blender benchmark so i'm just going","speaker":null,"is_sponsor":0},{"start_s":4131.12,"end_s":4142.08,"text":"to copy our three series benchmark graph now i'm going to import all of these project labels into","speaker":null,"is_sponsor":0},{"start_s":4142.08,"end_s":4148.8,"text":"illustrator using visual studio code and a python script that i've written so here i'm just gonna","speaker":null,"is_sponsor":0},{"start_s":4148.8,"end_s":4154.56,"text":"load in the packages that i need and here in chart names i'm just gonna pull up my file and file","speaker":null,"is_sponsor":0},{"start_s":4154.56,"end_s":4163.52,"text":"explorer and copy the path to the file and paste it into here um and we'll see here one two three","speaker":null,"is_sponsor":0},{"start_s":4163.52,"end_s":4174.72,"text":"four graphs um with all the graph titles subtitles text and uh project label version number and now","speaker":null,"is_sponsor":0},{"start_s":4174.72,"end_s":4178.56,"text":"we know that there are four graphs so i'm just gonna make sure that this number here is four if there","speaker":null,"is_sponsor":0},{"start_s":4178.56,"end_s":4184.48,"text":"were 50 i'd do 50 since there are four i'm just going to type in four um and there are no y-axis","speaker":null,"is_sponsor":0},{"start_s":4184.48,"end_s":4190.08,"text":"labels so we're just going to skip over that and then press enter and as we press enter we can see","speaker":null,"is_sponsor":0},{"start_s":4190.08,"end_s":4195.68,"text":"that it'll man automatically change everything so we went from you know all the placeholder text to","speaker":null,"is_sponsor":0},{"start_s":4196.4,"end_s":4199.6,"text":"all of this text and let's just double check that everything is right","speaker":null,"is_sponsor":0},{"start_s":4205.36,"end_s":4211.36,"text":"i'm going to open up harbour to look up the game test report and i'm going to download it as the csv","speaker":null,"is_sponsor":0},{"start_s":4212.0,"end_s":4219.68,"text":"we'll see here that the labels for the gpus are a little long um we don't really need the NVIDIA","speaker":null,"is_sponsor":0},{"start_s":4219.76,"end_s":4223.44,"text":"gforce part so i'm just going to cut that out it's going to be rtx 4080","speaker":null,"is_sponsor":0},{"start_s":4228.88,"end_s":4234.32,"text":"i'm just going to select the graph that i'm working on at the time let's make this full","speaker":null,"is_sponsor":0},{"start_s":4235.28,"end_s":4240.72,"text":"screen select the graph and i'm just going to delete everything that i have and make sure everything","speaker":null,"is_sponsor":0},{"start_s":4240.72,"end_s":4247.36,"text":"clears yep it's saying that i have no items at all to graph and i'm just going to paste in what i have","speaker":null,"is_sponsor":0},{"start_s":4247.36,"end_s":4256.24,"text":"and so here we see um because with our game graphs we like to sort uh highest to lowest because higher","speaker":null,"is_sponsor":0},{"start_s":4256.24,"end_s":4261.68,"text":"is better um and it does that automatically um the legend will stay the same uh one percent","speaker":null,"is_sponsor":0},{"start_s":4261.68,"end_s":4269.04,"text":"low on average result so this is a benchmark graph it has three series i had a um three series template","speaker":null,"is_sponsor":0},{"start_s":4269.12,"end_s":4276.72,"text":"ready","speaker":null,"is_sponsor":0},{"start_s":4286.0,"end_s":4291.04,"text":"okay i'm done now i've put in the data for all of the graphs and then we're just going to make","speaker":null,"is_sponsor":0},{"start_s":4291.04,"end_s":4297.68,"text":"sure that all the data is correct so what i'm going to do is um look over um the data based off of","speaker":null,"is_sponsor":0},{"start_s":4297.68,"end_s":4305.36,"text":"the spreadsheet that i pulled a cyberpunk for example for the 4080 is um 154 193","speaker":null,"is_sponsor":0},{"start_s":4314.0,"end_s":4322.4,"text":"okay so now all the data is included um we're just going to change up the uh x axis a little bit","speaker":null,"is_sponsor":0},{"start_s":4322.4,"end_s":4326.64,"text":"so you can see here that there are one two three four five six seven","speaker":null,"is_sponsor":0},{"start_s":4327.36,"end_s":4331.92,"text":"different lines and that's a little too much for my taste i feel like that's a little crowded","speaker":null,"is_sponsor":0},{"start_s":4332.72,"end_s":4339.44,"text":"and by default our graphs have an interval of 30 we prefer to have it at 60 but sometimes when the","speaker":null,"is_sponsor":0},{"start_s":4339.44,"end_s":4344.32,"text":"graph results are really low like having 60 doesn't really make sense so i'm just going to change it","speaker":null,"is_sponsor":0},{"start_s":4344.32,"end_s":4351.36,"text":"here to 60 so that it's a little less crowded and we can see 60 120 and 180 um with this one i'm going","speaker":null,"is_sponsor":0},{"start_s":4351.44,"end_s":4354.08,"text":"to do about the same because it's kind of within the same range","speaker":null,"is_sponsor":0},{"start_s":4359.52,"end_s":4362.96,"text":"i'm going to pull up go back to illustrator instead of exporting everything and having","speaker":null,"is_sponsor":0},{"start_s":4364.16,"end_s":4367.84,"text":"you know people look over it and then give me feedback individually what i'm going to do is","speaker":null,"is_sponsor":0},{"start_s":4367.84,"end_s":4373.04,"text":"i'm going to um share it for review it what it's going to do is create essentially like a google","speaker":null,"is_sponsor":0},{"start_s":4373.04,"end_s":4378.48,"text":"doc where people can add comments so i'm going to um set it to anyone with the link can comment and","speaker":null,"is_sponsor":0},{"start_s":4378.48,"end_s":4387.2,"text":"copy it and i'm going to uh send it off to adam and john so i'm just going to show everybody","speaker":null,"is_sponsor":0},{"start_s":4387.2,"end_s":4392.56,"text":"what it looks like and this is essentially what it looks like for john and adam um they're just","speaker":null,"is_sponsor":0},{"start_s":4392.56,"end_s":4396.96,"text":"going to look over it for multiple things they're going to make sure that all the labels are correct","speaker":null,"is_sponsor":0},{"start_s":4396.96,"end_s":4401.76,"text":"um and they're going to make sure that the uh data is correct it matches up with the right","speaker":null,"is_sponsor":0},{"start_s":4401.76,"end_s":4408.72,"text":"card and it matches up with the right um game or test uh here if they do you find something for","speaker":null,"is_sponsor":0},{"start_s":4408.72,"end_s":4417.04,"text":"example um they can leave a comment so they can highlight this and uh they can just write a comment","speaker":null,"is_sponsor":0},{"start_s":4417.04,"end_s":4430.48,"text":"and say uh for this one this data is actually for dota 2 not cyberpunk theoretically if i made","speaker":null,"is_sponsor":0},{"start_s":4430.56,"end_s":4439.92,"text":"that mistake okay so now i've sent off my um graphs to adam and john and they've reviewed it and adam","speaker":null,"is_sponsor":0},{"start_s":4439.92,"end_s":4446.64,"text":"doesn't have any comments and john has a question about um the specifics of like when we're going to","speaker":null,"is_sponsor":0},{"start_s":4446.64,"end_s":4455.6,"text":"use commas for the graphing numbers and so um you know i can reply to him and i'll say we're adding","speaker":null,"is_sponsor":0},{"start_s":4455.6,"end_s":4465.28,"text":"in commas whenever uh numbers are above a thousand if we had any changes to make","speaker":null,"is_sponsor":0},{"start_s":4465.28,"end_s":4470.16,"text":"then i would go in i would make the changes and then i'd just update the content um since we","speaker":null,"is_sponsor":0},{"start_s":4470.16,"end_s":4473.52,"text":"don't have any changes to make what i'm going to do is prepare for export","speaker":null,"is_sponsor":0},{"start_s":4476.88,"end_s":4482.24,"text":"and adam will be the one to send it off to the editing team and we're going to put it in the video","speaker":null,"is_sponsor":0},{"start_s":4482.24,"end_s":4488.08,"text":"yeah that's pretty much it on my end okay so at this point i received the graphs from sammy","speaker":null,"is_sponsor":0},{"start_s":4488.08,"end_s":4493.76,"text":"so what i do now along with adam as we go through when we kind of take a look at the graphs he does","speaker":null,"is_sponsor":0},{"start_s":4493.76,"end_s":4500.96,"text":"sort of a subjective look on the presentation of the graphs i do more of a data analysis make sure","speaker":null,"is_sponsor":0},{"start_s":4500.96,"end_s":4505.68,"text":"everything was inputted correctly maybe something got copied into the wrong line or something like that","speaker":null,"is_sponsor":0},{"start_s":4506.48,"end_s":4511.76,"text":"so basically i just have to pour over the data here make sure everything looks right","speaker":null,"is_sponsor":0},{"start_s":4511.76,"end_s":4518.16,"text":"so we just quickly scan through blender here 50 48 20 10","speaker":null,"is_sponsor":0},{"start_s":4524.32,"end_s":4530.0,"text":"this blender looks good and then if we hop back over to the the game test report","speaker":null,"is_sponsor":0},{"start_s":4530.72,"end_s":4535.92,"text":"we can quickly look at all of them up here so if we go to dota 2","speaker":null,"is_sponsor":0},{"start_s":4542.0,"end_s":4548.8,"text":"so this point now i've basically verified that all the data looks correct um assuming that adam","speaker":null,"is_sponsor":0},{"start_s":4548.8,"end_s":4554.64,"text":"doesn't have any other issues then we go ahead and get these all set up sent off to the editors","speaker":null,"is_sponsor":0},{"start_s":4554.64,"end_s":4560.56,"text":"and that's when they start cutting the video after all of that we then go into post production","speaker":null,"is_sponsor":0},{"start_s":4560.56,"end_s":4566.24,"text":"where we provide the assets and the footage to the editor we have several more steps of qc that","speaker":null,"is_sponsor":0},{"start_s":4566.24,"end_s":4571.04,"text":"occurred during this um first off you have the editor and writer review where the writer sits","speaker":null,"is_sponsor":0},{"start_s":4571.04,"end_s":4575.92,"text":"down with the editor watches what the editor has created and um points out any sort of fixes","speaker":null,"is_sponsor":0},{"start_s":4575.92,"end_s":4581.28,"text":"whether they're stylistic or content fixes that need to be done then once those fixes are implemented","speaker":null,"is_sponsor":0},{"start_s":4581.28,"end_s":4587.36,"text":"they send a pass to Labs Labs looks over it Labs points out any fixes typically are their fixes","speaker":null,"is_sponsor":0},{"start_s":4587.36,"end_s":4591.2,"text":"are going to be more focused on the correctness and the data side of things and also maybe kind of","speaker":null,"is_sponsor":0},{"start_s":4591.2,"end_s":4596.08,"text":"small nitpicks less so about the stylistic side of things but we still make sure that we provide","speaker":null,"is_sponsor":0},{"start_s":4596.08,"end_s":4602.16,"text":"Labs input into that because sometimes you can lose the exact meaning of stuff just through like a","speaker":null,"is_sponsor":0},{"start_s":4602.16,"end_s":4606.16,"text":"couple of different wardings and we want to make sure that that's always communicated correctly","speaker":null,"is_sponsor":0},{"start_s":4606.88,"end_s":4612.16,"text":"then once Labs reviews it the editor implements all those fixes there is another review with the","speaker":null,"is_sponsor":0},{"start_s":4612.16,"end_s":4617.76,"text":"writer and Labs to make sure that all the fixes have been implemented correctly once that is done","speaker":null,"is_sponsor":0},{"start_s":4617.76,"end_s":4622.96,"text":"we then upload it to Floatplane for our ecc squad to take a look at the whole point of ecc squad","speaker":null,"is_sponsor":0},{"start_s":4622.96,"end_s":4628.08,"text":"is to get like a ton of knowledgeable people to have eyes and ears on exactly what we've made","speaker":null,"is_sponsor":0},{"start_s":4628.08,"end_s":4631.36,"text":"because they'll notice the smallest kind of details when you have like a dozen sets of eyes you'll","speaker":null,"is_sponsor":0},{"start_s":4631.36,"end_s":4637.44,"text":"just find so many small things and they're always great and once we've done that we then","speaker":null,"is_sponsor":0},{"start_s":4637.44,"end_s":4644.08,"text":"review the suggestions that the ecc squad has made and if they need to be implemented we do that again","speaker":null,"is_sponsor":0},{"start_s":4644.72,"end_s":4650.32,"text":"we also check with Labs to make sure that those corrections are in fact correct implement them","speaker":null,"is_sponsor":0},{"start_s":4650.32,"end_s":4655.2,"text":"make sure they're implemented correctly and then finally after all of that a video gets released","speaker":null,"is_sponsor":0},{"start_s":4661.36,"end_s":4664.72,"text":"update your BIOS kids","speaker":null,"is_sponsor":0}],"full_text":"Welcome to the Labs Process Video. We're going to give you a step-by-step breakdown of everything we do when we test a product for a video, all the way from ideation to the final results. Every test starts with an idea. For example, it's been a long time since we've reviewed the 7900XTX and the 4080, and we want to know how do these perform today? The first step is filling out our test request form. As much as it would be nice to just go down and ask Labs to test these cards, they're not going to do anything unless I provide them with a filled out test request form. There is no testing without requesting. The reason we have to do a request form is that it allows Labs to track the project. If we don't give them a request form, they won't do the testing because that's how we waste people's time. In here we put like the name of the project, so we'll say like 4080 versus 7900XTX. Then we'll put a link to the Trello just again for project tracking. And then here's the first kind of important answer. This is an important delineation. Is it exploratory testing or project testing? So exploratory testing is going like, oh, we don't know if this is going to be an interesting story. So we need to test to see if the effect that we are talking about is significant. Like how much does a browser extension slow down your browser? Maybe it's not enough to be an interesting video, but maybe there's something there versus a project test, which is like, it's a new GPE review. We know that we're going to test it. We don't need to explore. In this scenario, we're doing a project test. Then we give them a due date, so a project timeline. We can either do a hard deadline or a soft deadline. A soft deadline is kind of like, if something else of higher priority comes in, Labs can push it back a little bit because maybe it's a video that doesn't really need to come out on a specific date. A hard deadline is really important. If it's something that's very timely, like maybe it's a sponsored project or like an embargo GPE review, then we give them a list of all of the components that we want tested in the video. For this scenario, we're just going to do the two cards, a 4080 Accelerate and a 7900 XTX. Then we just let them know if we have them here or if they're coming. Basically it allows them to be like, where do we need to listen for the next step? Is it like the writer or do we need to like keep an eye on procurement or something like that? Sometimes we also might want to like talk to Labs about any sort of materials or components that we need to actually get. And that'll usually happen during like a kickoff meeting. Then we have to clearly state what our questions are. It's really important to do this and like the more nailed down our questions are, the faster the project's going to go and the faster the kickoff meeting will go. There have been times where I've stepped into meetings where I did not have a solid question and then it takes a long time, waste everyone's time trying to figure out what the heck I'm doing. In this scenario, we're just trying to see how well does AMD's 7900 XTX compared to the 4080 is the 4080 really we're going to be worth the extra money. And then we can also select specific tests. We have a whole test portfolio that we can usually run like where we're just run through a gauntlet of games and productivity benchmarks, but maybe I just need to select few or there's some that I absolutely must have. In this case, we don't need to run these through like a million gaming benchmarks, just a few. We're going to do Cyberpunk 2077, Dota 2, Red Dead Redemption 2, and they're all going to be at 1080. The next part is like any sort of notable test boundary conditions. So say we wanted to have something that's like out of the ordinary, like, you know, maybe we need old drivers, like we want the drivers that were used when it came out to compare to the drivers that are out today. That's something we can put in here. And then for any super general notes that might not really have another place to go, there's a final form just at the bottom where we can fill out and be like, I don't know, maybe like, you guys want to hang out sometime too? I don't know, you can ask them whatever they need to submit it and it goes off to them. Hi, it's Gary. We just received a test request from Adam. Normally this goes to Sharon, but Sharon is busy doing some other work today, so I'm handling this request. Once it comes into our mailbox, we simply click on it, brings up the lab's data request. This one's obviously from Adam. We're testing the 4080 against the 7900XTX. And then he's actually filled out some specific tests that he wants. All of these, Cyberpunk, Red Dead Redemption, Blender and Dota are part of our standard test suite, so that makes it a little bit easier. He doesn't want anything special on it, just straight up testing. And apparently he appreciates all of us, so thank you, Adam. With that, we'll go over to the assignment form. This is where we keep track of who's assigned the project, and also if it's just been assigned or if it's already in progress. For this one, we'll go ahead and mark it in progress as John's going to start it momentarily. And yeah, we're giving it to John. And with that, I'm going to hand it over to John. Thank you. Now we have our kickoff meeting. Basically what this does is that it allows us to talk with Labs, make sure that our test plan is all agreed upon, and that we are all on the same page, so that we can make sure that we're doing everything right. Labs will often bring some concerns and questions. Writers might be like, notice things in the middle of the meeting, be like, oh crap, I forgot to add that to the sheet. So this is probably one of the most important parts of the whole process, because it allows for all of us to make sure that we are aligned on our goals. So we just had the kickoff meeting for this project. Now that we've done that, we can actually go about collecting the requirements, the project requirements for the system. So we're going to be using our AMD benches today. And that's something that we're going to be using going forward for all of our GPU testing. So what we're going to do first is give you a rundown of all the hardware we use on our AMD benches, and then get into the nitty gritty of actually building it and setting it up correctly. So first thing we do is we got a CSONIC Prime PX1600 power supply. We've been using these for a little bit now. They're ATX 3.0 capable. So the nice thing is we can swap quickly between using a 12-volt high power and typical 8-pin power connectors for any of the GPUs that are made by like Intel or AMD without having to use any extra like pigtails and stuff like that. So that's pretty useful. And for motherboard, we are using X670E AORUS extremes. We've been using these for a little while on our AMD benches, not that we've used them in many of the videos, but going forward that is going to be our standard for our GPU testing. The reason we went with the extremes, we were using the masters for a while, and then we ended up going back to the extremes because we found that we liked the VRM a little bit further. It was a little bit more robust, and the BIOS settings we found that it could tweak just that little bit better than the master for doing some exploration testing of overclocking and stuff like that with some of the CPUs we've gotten in. As far as other brands, we have looked at ASUS for a little while. We looked at a couple of ASRock boards, but ultimately we kind of ended up on the AORUS extreme. So for CPUs, we're going to be using Ryzen 7 7800X3Ds. We actually have a video upcoming that is about AMD variance testing that we did to basically determine the three 7800X3Ds that we're actually going to go with going forward. They are the fastest one or the slowest one, the three tightest ones that we could find to effectively try to reduce any sort of variance in our FPS basically between the three benches. So we can kind of more easily verify that if we were to take this card and put them on any of the three benches, we should get the same number. For RAM, we are using G-Skill Trident Z5 new RGB. The stuff we're going with is got 6,000 megatransfers per second CL30. Even though AMD has come out with new versions of AGSA that have basically unlocked the ability to use anything higher than 6,000, we still found that this is kind of the best sweet spot for AMD in terms of performance. You can get up there where you start getting your timings a little bit tighter and stuff like that and perform basically the same on like 6,800 megatransfers and stuff like that. But this seems to be a good price to performance ratio still, so that's kind of what we're going with going forward. For SSDs, we are using Samsung 980 Pro 2TB SSDs. We haven't explored anything as far as using PCI Express Gen 5 SSDs yet, but this seems to be a really good performance ratio on both the reads and writes of both not just the sequential but as well as the random reads and writes. So it kind of allows us to get the load times a little bit faster and kind of have more benchmarks running in sequential order without having to wait for all those extra load times. Sure, most SSDs on the market nowadays are fairly quick as long as we have something as DRAM in it, but that's the one we've kind of been going with for a while and seems to be good enough for us going forward. And if it ain't broke, don't fix it, right? So there's that. And then lastly, to cool the 7800X3Ds, we are using Noctua NHD 15 coolers. It's kind of been the standard for a long time for a lot of people. As soon as the new gen version comes out, I'm sure we're going to be exploring that as well and seeing just how much more beneficial it could be for us. But it's been basically the workhorse for us for a while and it probably won't change for a little while afterwards. So that's kind of where we're at this point. So for the specific project, we obviously have to get the specific cards that we need to test with. So we have a reference, AMD Radeon 7900 XTX. It's been the one that we've used in all of our videos that we've ever covered on it. It's got a pretty sick looking cooler. And then the video card we're going to be using today is a PNY Accelerate RTX 4080. This seems to be currently the one that is the cheapest on the market, but that doesn't necessarily mean it's the worst. And this is the one that the requirements of the test plan specifically outline. So that's the one we're going to be testing today. And lastly, we obviously need some sort of thermal paste to go between the D15 and the 7800 X3D. And we use Noctua NTH2 thermal paste. We've been using it for a while. As far as application, more is always better than less to some degree. And so when I usually put thermal paste on, it's in an equal sign, which is something that you'll probably see in the build part of the video. So I'm sure everybody's going to have tons of arguments of, oh, you've used too much or used too little. There's been so many discussions about that. It doesn't seem to affect the thermal performance in any sort of way. So as long as we have enough, we're good to go. All right, now we've gone over the actual hardware, let's get to actually building. Now we've got the system all built together and up and posted into the BIOS, we're going to be going through making sure we have the correct BIOS version as well as configuring it to the standardized configuration that we do for all of our benches. So when we look at our test conditions, we usually outline both drivers as well as BIOS revision, because there might be an instance where like we're doing some sort of exploratory testing and being like, hey, we actually don't want the latest version of the BIOS. We want to go back four or five revisions. That's why we outline it in the actual test conditions. In this case, it's saying F11D, which is the current version of the BIOS, but obviously we're not running F11D, so we're going to have to update it. Update your BIOS kids. Yeah, there's, especially with like new AMD CPUs, 7800X3D in particular, a number of motherboards don't actually support it correctly out of the box. So if you don't have the option of actually booting into the BIOS, luckily most manufacturers include some sort of BIOS flashback now. Just slot the USB into the correct port on the motherboard, make sure obviously the file is named correctly before you do that. It's incredibly important if you want a computer to work right. Then once we finally get the BIOS updated, we're going to be going ahead and configuring it and then setting up Windows. Okay, now we've finished the actual BIOS update. Let's go ahead and start configuring things. First thing we're going to go to advanced options, so we can actually see everything. We always turn on expo or if it's Intel, it's XMP. Verify that rebar is on. Most AMD boards usually have it on by default now, but it's always good to go to. It's usually under IO ports or some sort of IO naming scheme, whatever manufacturer makes it. Inside here, we also turn off the IGP or integrated graphics, especially if we are testing any sort of dedicated GPU. All it does is sometimes confuse some games or programs into wondering which actual graphics card to be using at the time. And then from there, the only other thing that we do is go into the fan curves and we manually set it so that it is a linear scale. So it basically looks like that. Then at that point, we can actually start installing Windows. For Windows, we use a optimized Windows image that our software developers have created. Basically strip back a bunch of stuff that we don't need to be running in the background, as well as adding in some programs that we actively use during our testing. We actually reinstall them every three months. We'll usually update them basically and create a new Windows image that has our latest updates so that we don't have to constantly do this rigmarole of get a million updates, a million drivers. We can just basically install the latest drivers and then we have basically all of our programs. So now we just got to let this thing run. Very exciting stuff, like watching paint dry. In our test conditions, we'll usually outline the exact version of Windows we want to run. Going back to that, we might need to recreate an old set of parameters. So we might not be using Windows 11 22H2. We might use 22H1 or that's why on that list, we actually have all the other versions of our images still on there so that we can actually go back and be like, oh, actually we need to use the version from May. And then we basically, if we need to recreate that set of test conditions, we have to go and also stop Windows updates from updating. Now we're in Windows. We are not going to install the Gigabyte Control Center. So basically at this point, we have to go through update all of our drivers, as well as install all of our programs that we're actually using for the project. Okay, so now we've installed the chipset drivers. We can make sure that we get the latest NVIDIA driver as, or not necessarily the latest, because if the project does require an older version of the driver, we got to get that version instead. But in this case, in this instance, we are using the latest version. So we're using 537.42. Let's get that downloaded. Then we're just going to make sure all the video settings are set correctly, so we essentially turn off G-Sync, any sort of potential accelerators, V-Sync, so we can get the, basically the raw performance of the card. Update your cards. There's a lot of things that NVIDIA doesn't post or advertise, like a number of three series cards don't have rebar enabled out of the box. So there's actually a bunch of articles that NVIDIA posted that are really useful, such as UEFI fixes, DisplayPort fixes, and things like rebar patches and stuff like that. So if you find your card isn't matching up for whatever reason, or is doing something funky, good to check those out, because sometimes it might be as simple as, oh, I need to do this little update. Boom, I've got more performance, or oh, my video doesn't seem to work in the BIOS, but it works fine in Windows. Do a firmware update. Oh, sweet, now it's fine. Alrighty, now that we've got that installed, let's just make sure we don't have any other weird drivers. Looks good to me. Usually if there's no yellows, we're happy campers. We usually do all the Windows updates according to what our project plan is. In most cases, it's usually just as simple as getting every single update that's in there. A lot of times, what will happen is, you know, there will be driver updates and stuff like that, but if you already have a later driver than what Windows sees, often you'll get like a download error a lot of the times, because Windows is flagged, oh, you actually have a newer version, we don't need to get this. And what we do is we tend to update our Windows image every three months to help include a lot of these Windows updates. We essentially have to download a lot less of them. It is time consuming and the less time we have to spend on constantly downloading things and updating things, the better, right? Then we can get right into the actual testing. It saves us a lot of overhead, basically. So one last thing we're going to do before we get all of our games downloaded on the system is we're going to go through and just do some last minute Windows optimizations. So first thing we're going to do is we're going to do some trimming on the actual drive itself. Just helps speed it up just that little bit better. Then we're going to go and go to percent temp percent to basically get to our temporary folder. And we're going to basically delete everything we can in here. So one last thing we're going to do is we're going to set the recycling bin to 15 megs. The reason we do this is just that if we delete any files on the system, it'll just delete them right away. It won't sit in the recycling bin taking up extra space we don't need. Basically, it skips a step of us happening at empty out the recycling bin in most cases. So kind of useful in that regard. So once we have all this set up, the only last thing we have to do is just make sure whether it's NVIDIA, Intel, or AMD, we have any sort of variable refresh rate stuff turned off, as well as B-sync. The reason behind that is we don't want to potentially give any sort of advantage to one manufacturer over another with any sort of technologies they might have running in the background causing any optimizations that are effectively outside of Windows itself doing hardware optimizations. That way it kind of keeps the playing field a little bit more level. So we're going to pop into here. I doubt we'll have G-sync on here. No, because we are currently plugged into our NDI viewer to get all of our capture. So for this specific instance, we won't have G-sync enabled. But typically, because these monitors are actually G-sync capable, anytime we pull up an NVIDIA card inside there, it'll immediately enable G-sync. So we have to make sure we go in and actually disable that. And then what we do also is in here, we've got to manage three settings, scroll down to vertical sync, and we make sure we force it off. That way, because there's some games that they don't either give you the option or they tell you it's off, but it's not actually off. It's kind of just ensures that no matter the game, no matter the settings, if somebody forgot to turn off V-sync for whatever reason, this basically forces it off. So there's no opportunity for the game to even interact with, hey, by the way, I need V-sync. So it kind of takes that guesswork out of it. And then we go into display settings. We're going to be running at 1080p, so we just leave the resolution as it is, but we change our scale to 100% rather than 150. Some of our markbench benchmarks do still use CV in order to do character recognition. So if the scaling is at 150% and you're on the desktop, for example, the new version of Hitman has a launcher that it shows up beforehand that has all the options and stuff in. How markbench works in that one is it actually will look for the specific word. If our scaling is different than what it's supposed to be, markbench will be like, I can't find this because it's effectively looking for a picture at a different scale than what it actually is. So if we set it to 100%, basically we're going to have no problems. And then normally what we do is we check to make sure we're at the highest hertz we can be for the monitor. Because we're going through the NDI, we actually have to run at 60 hertz because that's the maximum the actual NDI itself supports. But typically it's 144 for the actual monitors themselves. So lastly, we go into change default graphics settings. We make sure that hardware accelerated GPU scheduling is on. This is at Windows level. This isn't hardware specific such as NVIDIA Reflex or AMD's Antilag. That is something specific to the hardware, not Windows itself. So this one, it's the same across all the platforms of hardware. It doesn't give an advantage to NVIDIA over AMD because it's scheduling at the exact same. Anything else that's in the menu, usually we have like optimization for Windows games. We turn that off, as well as in some cases, there is a variable refresh rate option in here. It usually shows up for Intel and AMD because theirs interact a little bit differently than G-Sync does. And so we make sure that's off as well. Because again, we don't want to give advantage to one specific card. We want to see raw performance of each of the cards stacked against each other so we can understand what is actually the best option. Now that we've gotten all the Windows updates installed, next thing we got to do is we have to install all the programs as well as any games that we're going to be using to do this project. So in this case, we need to install Cyberpunk, Dota 2, and Red Dead. So we already have Steam pre-installed on our Windows image so we don't have to worry about actually going to the website and getting that part, but unfortunately it always needs to update anyways. So it's kind of a moot point. While that's happening, what we're also going to do is get markbench from our local drive, as well as some patches that I need to install as well. So we are going to install, nope, Cyberpunk. We're going to install Red Dead. And then the most annoying thing is that Dota 2 doesn't stay in your library half the time, so we are going to go here and do play now. Okay, so those three guys are downloading. We've got markbench extracted here. We need to add another harness in for the Dota 1, so let's get that real quick. And there it is, ready to go for when we are ready to use it. And then the other thing we need is Blender Benchmark, so let's quickly grab that as well. So although markbench does have a Blender Benchmark actually integrated into it, we do need to still download the application itself to download the files required for it to run. There's a little bit of bugginess with the CLI. When it tries to download the files, it effectively says I can't do this and quits out. So if we download the actual program and have it download the files, it just makes the automation a little bit simpler on our side because it has the files already completely installed. So effectively we want version 3.6. We're going to download the benchmarks. Again, let's just make sure that markbench is all prepared, just like when we have to go through any of our game settings. We have to actually go download the programs, download the games, actually set up the settings ourselves. That stuff is not currently automated. Now we have all the games downloading and everything's kind of getting set up. I'm going to throw it over to Nick and he can talk a little bit about markbench. Fundamentally, markbench is our test automation and orchestration framework. It's not just simply a simple desktop GUI anymore. It's a constellation of services and components that work together to form the ecosystem that we conduct our testing in. But it all does start with the test harnesses, which most of ours are in Python because it's most easily automatable for us. However, it doesn't need to be Python. They can be PowerShell scripts, Ruby, GoLang, what have you. The important part is that each test harness gives back a exit code of zero or one, zero to indicate a successful test run, anything greater such as one to indicate that it failed. Because that exit code is then read by the next layer of markbench, which is what we call auto bench. It just comes before the GoLang GUI and it really contains all of the major features of our orchestration. This layer is what separates what we're doing from just a collection of PowerShell scripts to something that is a lot easier to manage long term because inside of auto bench, that's where we get our orchestration. So with the test harness, I have the capability of running individual tests, but I want to be able to group those into a bunch of test suites such as let's say I want to run Dota 2 four times at one resolution and then Red Dead Redemption 2 at a different resolution. So that's where the auto bench side comes in. Auto bench is where we run presentmon in system logger in conjunction with the test harness so that we get a similar timestamp between all three because this eliminates the human error of starting a test and then starting hardware info or like logging on the side, which then would separate our timestamps by sometimes multiple seconds, which makes it easier to trim the sessions later downstream, which John will explain later. And then so the auto bench is all command line interface, so to make it easier for us and John and writers or whoever wants to run Markbench, we added a Golang GUI. So we went with the Golang GUI because the framework that we used was very successful for us for other GUIs such as the keyboard tester, the robot, and the latency tests. So anytime you see Markbench on screen, that's the Golang part, but underneath all the work is really being done inside the test harness as well as the Python orchestration layer. So once the test is done running, let's say we just ran an individual test and once that test is done running, it sends its data up to a service called data ingest server, which is a Golang web API, which takes it in that data in a protobuf format and then writes it to our Postgres database. It's from that Postgres database, all of our data sits at rest and then is disseminated or sent to all the various tools that John can use to evaluate and report on that data. And we use a couple tools, notably Grafana and Redash for a lot of visualizations. And then we have internal tooling that we just kind of collectively call the Harbor Tools portal. And that contains anything that Grafana or Redash or another off the shelf tool doesn't provide us. Now if we go back up to the test harnesses, on their own, they're fairly dumb in that they can send keyboard and mouse input, but they don't always know what's going on the screen. So we used to use OpenCV locally to do template matching, but it's more of a brute force approach. And it's very flaky as it depends on certain resolutions. So if you're doing template matching with a resolution of 1920 by 1080, you need to select screenshots of the path or the flow you want to go through the menu of the particular game. But then if you want to do it at a lower resolution like 720p, nine times at a 10, you have to take a smaller screenshot. So it doesn't scale very well. And that's where we start getting into some of the services that we're adding. The first one being Keras optical character recognition or OCR. This allows a test harness to send a screenshot of the system it's running on with a target word and Keras will come back within a few milliseconds is that word on the screen or not. This reduces flakiness in the test harness, which makes navigating menus a hundred times easier. And in the future, we're going to be adding services that do a bunch of computer vision things such as depth estimation, object detection and whatnot, all in the effort of making the test harnesses more a lot smarter. But we'll talk about levels of automation in a bit. So the benefit that Mark Bench and its ecosystem gives us over just a collection of PowerShell scripts for one is maintainability at the scale that we're trying to achieve. PowerShell scripts would or some auto hockey scripts would become messy very quickly because eventually we'd have probably like 100 to 200. And as best as you could organize those in some directories, it's just not a it's a maintainable nightmare. Investing in such an ecosystem and the automation of Mark Bench, it allows us to build it to support multiple platforms. So not just Windows on a desktop, but handhelds and macOS and Linux as well. And it normalizes data for us so that when we capture metrics and we set it at rest, it's a lot easier to query that data. Back to you, John. Thanks Nick for that explanation of Mark Bench. Now what we're going to do since all these games are finished downloading is get them all set up and then we'll be able to get our benchmarks started. So we're going to start off with cyber bunk here. We have documentation that we are working on that we will be publishing in the near future of all of our settings and configuration for the various benchmarks that we include in our videos. I believe we've already released a couple of them on the forums. So I'm sure we'll continue to update that as time goes on. Okie dokie. Yeah, the new update for cyberpunk came out. So that's cool. So first thing we do, we go to settings. We're going to go to video mode and we're going to make sure we're at 1080p because that's what the project requires. You're going to be turning off any sort of things like low latency stuff like that. I think that it could benefit that specific card because we want the raw performance of the card. We don't want any sort of additional things like DLSS or low latency that can certainly help benefit the card and it might get better performance for sure. But what we're trying to determine is apples to apples between that 7900 xdx and this 4080. So we're going to be turning off low latency. We're going to be making sure that we're on full screen. We don't want borderless. Then we're going to go to the graphics tab and for our specific test we aren't calling for ray tracing. So we're going to actually be turning off ray tracing. We're going to be going just strictly ultra and we're going to go ahead and turn down FSR to zero. Go back up a tiny bit. Go back down a tiny bit. Each individual time we change something we're going to hit apply. The settings for cyberpunk are still kind of a little bit weird where you might change some sliders and you might hit apply and sometimes it's forgetting that you hit apply or you change the quick preset. It doesn't actually use the preset because you haven't applied it. So when we go up and down on the FSR that kind of helps with that and then we're going to go ahead and turn FSR off because we don't actually want it. And that's basically how we set up our non ray tracing version of cyberpunk. So at that point we can basically exit out of that and move on to the next one which is Dota 2. Dota 2 is one of our more recent benchmarks that we've been working on. It uses a replay from I believe Dreamhack. A lot of visuals on the screen so that we can basically tax the card as much as possible. So in Dota we're going to go up to the cog, go to video and we're not going to use monitor's resolution. We're going to use advanced settings. In this case we need 1080p. We're going to be checking on to exclusive full screen rather than that. Turning off low latency yet again. And then for rendering side we just pulled the slider all the way to best looking. It's basically a preset that has all these specific settings. These things always off. That sort of thing. That's all we got to do for this guy here. So we go ahead and close that. Okay now we've done Dota 2. Lastly we just need to set up Red Dead Redemption. So we're going to hit Z to go into settings. Go to graphics. Make sure the resolution and the screen type are correct which they are. We're going to be turning off v-sync. Now pull the slider all the way to the right that usually gets us all the way there but we still have to actually go through and manually check these because like I was saying before Red Dead's a little bit different in that this slider doesn't necessarily mean you're going to go everything ultra if you hit that. If you have lower end hardware it actually kind of tries to adjust that slider based on your hardware. So on like a low end laptop you go all the way to favorite quality and everything might still read as high. So it's something that that's why we have actual screenshots of our settings for it so that we can actually make sure that they're the same even after we change that slider. So we should be sitting at ultra 16 ultra ultra ultra high ultra high ultra medium high off high and this is always the one that kind of gets a little bit fussy. That looks right. Okay and then we're going to hit enter to apply the changes. I'll often recommend don't alt f4. There are some games out there that the settings don't save until you actually properly quit the game. It's like a fail safe so if you just exit out the right way shouldn't be any problem. So first thing we're going to do is set up mark bench here so we actually start running the benchmarks. So in order to do that we have to input a few values first. So we're going to go through all the different boxes and options that we have for the tests we're going to be running today. The first box that we have that we can fill information into is called the project slug. What that usually entails is every time we create a test plan we usually assign a sort of either a character value or an actual code name for that project so we can later look up the data if we have to or reference it in a future video or something like that. It makes it a little bit easier on our side and then that way the data doesn't get lost in translation of oh I'm looking for this project but there's eight folders that are named similarly. It helps kind of basically organize that into a more clear overview of which project is which. Next is the session tag so if for example we are altering the values from our standard testing run so we're turning off anti aliasing or we're you know messing around with which preset we actually use. This helps us assign an actual tag to those specific sessions so that when we're looking at them later we can be like oh actually it's because the reason why the values are completely different is because this is the set that we that we ran with experimenting on anti aliasing or experimenting on you know running low versus high shadows or something like that so we can basically put in like no dash a and later when we go to look it up it'll actually have that listed on those specific sessions. Thirdly we have what's called the test parameter there's a bunch of different options in here that we have currently it's something we can edit whenever we need to but basically it assigns those sessions what the test parameters we hope to use for that specific session so for example for this project we're only running at 1080p so we're going to be setting just on here games 1080. There are other options in here like DX11, RT, stuff like that. This doesn't actually alter any game settings themselves this is just for our internal use on all of our sessions so we know if we if that run was for 1440p if it was for ray tracing if it was for XCSS you know that's all those different types of visual parameters that are outside the norm basically. So for the project slug we are going to be putting in the one that's on the test plan which is 4080-vs-7900-XTX-2023. So once we have written our project slug in as well as our test parameters now we can get into actually editing which tests we're going to be running as well as some of them have different configurations that we can alter that are non-game tests or for assigning certain values of what the how the game is supposed to interact that sort of thing. So first we're going to cover blender benchmark we click on it here any of these ones have different test arguments in most cases so if we click on the blender benchmark we have a bunch of new values in here we can adjust so things like scheduler delay, recording delay and recording timeout. These are all in seconds we usually run a scheduler delay of five seconds just to make sure that the program or the game can have time to close and be satisfactory with Windows and all copacetic to be able to open again especially when it comes to steam when you have like cloud game saving enabled and stuff like that it has to take those few seconds to actually do the cloud save and sync it up what can happen in some instances if we don't set the scheduler delay is it will run into that bottleneck of oh but the cloud saving hasn't synced yet are you sure you want to load the game and it kind of gets everything all hung up so when we set it to five seconds it usually helps give steam enough time to finish that option and then actually continue on with the next run. So the fourth box here is the number of times that the test is meant to repeat so it's for that specific test not for the all the test as a whole so in this case we usually run three for all of our tests that doesn't mean that we only run three ever and that's it if we find that a result is outside of norm not within our our typical variance which usually about one to two percent we will do another run till we have three valid runs that are actually within that variance so we can confidently say that these are the numbers. So in blender benchmark we also have options for things like the scenes that we use because there's three different scenes that it actually runs through we're just going to be doing all of the scenes so which entails classroom junk shop as well as monster that's typically what it runs in the gooey version with the version that's available through markbench we actually use the command line interface so we're not going to see anything pop up on the screen for blender specifically but it is running in the background and we'll show that when we pop over to the actual command lines to see all the stuff that markbench is actually running then we have which versions that we can run blender on the blender benchmark on we're going to be doing 3.6 because that's what the test actually outlines the test plan outlines and then we're going to be testing the GPU not the CPU because that's what we want to test today is the actual GPUs then we're gonna go ahead and check it off and move on to the next one which is cyberpunk same thing five and three and then down here we have for what our keras ip's should be so i can actually talk to them some of our harnesses don't use keras still um it's something that we're kind of developing as we've been getting into these future automations and stuff like that to be able to add these tools back into the toolbox of basically streamlining how each individual harness is so when we get down to red dead for example it's not going to have that listed right now but we're hoping to have that in the future so we don't have to do anything else on this one just five and three go down to dota same thing five and three and we go to red dead we're just going to be setting red dead to five and three again but here all we have is preset which is current this can't be changed it's a legacy harness that we've we're currently in progress of updating to use keras as well as some of the other things that nick will be discussing as far as our levels of automation goes now we've got everything set up for mark bench it's time to hit the go button and while this is running the benchmarks i'm going to kick it back over to nick and he's going to talk about the levels of automation so coming back to automation we're going to focus on the test harnesses and talk about the degrees of automation because one of the most annoying questions to try and answer is is it automated when someone asks you that it really depends the answer to that depends on the scope is a bunch of power shell scripts that run your game and then output something to a log file considered automated or is it only automated if you can press a button and walk away and it handles every edge case that you could possibly think of to help talk about automation internally we've come up with a milestone system and so we come up with the milestones of one two three and four so when we speak of a test harness taking a game for example a game harness that has a level one automation it simply just launches and runs the benchmark id like akin to that just a power shell script doing something very simple and then when we reach a milestone of level two it can do a lot more such as dealing with advertisements or pop-ups in one of their launchers or it can report on game settings and then the next level to that is level three is where it can actually configure the game for itself and also reach out to maybe some more services like Keras or whatnot our definitions of one two and three are fairly loose because every game is a little different some games are a lot easier to report on resolution because the it might be reported in a .ini file that we can grab from app data where others might only store their settings in some proprietary binary format that we can't even read without some finagling so this is roughly how we categorize it just so that we can at least internally when someone asks you is it automated we can say yes to a particular degree and then we get down to number four which is the future this this category encompasses things that we can't do yet we have something we call our automation toolbox where we include all the different services such as the keras optical character recognition uh and uh pie direct input or pie auto gooey it's all the different things that we can use in our test scripts to achieve the automation of a game so one to three incorporates those ones i've listed where four is some of the tools that we haven't come to yet the future is where we'll start to talk about ndi and how we use it for video recording as well as computer when we started automating our game suite we relied on games that had a canned benchmark due to the fact that it's a lot easier to automate as you just have to navigate the menus and then start the benchmark however the problem with that is that not all built-in benchmarking utilities are representative gameplay and not all games come with a canned benchmark which limits our selection for our game suite to expand our game suite we started looking at games such as dota 2 and rocket league that have a very robust replay system in the future we're looking to incorporate computer vision tools such as depth estimation and object detection to expand our game selection to any game before we can use depth estimation or object detection we need to get a video stream off of the test bench to another computer to run the machine learning to do so we use ndi or network device interface to get video stream from the test bench to the machine learning computer ndi allows us to convert HDMI signal to video over ip which then can be picked up by any other client that's on the subnet for example on our computer on the cart we have a client that's picking up the ndi stream being sent from the test bench which then runs our experiment with depth estimation and depth estimation is the task of measuring the distance of each pixel relative to the camera depth estimation could be used to instruct a player to walk down a hallway by chasing the darkest point on the screen we are also experimenting with other techniques such as optical character recognition and object detection on a video stream the ndi encoder that we're using is an n6 from kill of you which has an HDMI in HDMI out and then your Ethernet a particular nice to have on this unit is the small display on the front which allows you to see the activity of the unit as well as the ip and what video stream it's sending out okay so all of our benchmarks have finished let's take a look at the results as you can see here it actually says we've ran zero zero for dota 2 so we didn't actually get any valid runs so we're gonna actually have to do some troubleshooting figure out why so that's the thing is that markbench allows us to identify some very early on possible bugs that might occur with the run and it'll usually terminate so if we scroll up here you can actually see we get to a point where it says test failed and it's for this run here so it says game didn't start in time check settings and try again so this is something that we've programmed into some of our logging so we're gonna have to actually go into our dota 2 settings and it could be that maybe something got reset when we swapped cables or something like that so we're gonna take a look at that and get it set back up to the correct settings and try running it again so we're in dota 2 let's take a quick look at the setting and see what might have went wrong here so we go to video ah there's the problem it changed itself back to 4k it's a good chance that while we were swapping some cables around or something like that when we were doing the initial setup I probably swapped itself back to native resolution and so it caused our markbench runs to fail so let's change that back and then we'll rerun it here the upside of this is is we don't have to run every test we just ran again we can actually unmark everything else and just specifically target dota 2 now so let's go ahead and get that started and hopefully we'll come back with some actual runs so looks like we got three runs done for dota 2 now so we can actually upload all this data to our data ingress server and start trimming so let's get that done now we've uploaded the data to our server we can pull it up on grafana which is our data visualization dashboard from here we can pull up all of projects that we've run any sessions that have occurred from basically any of the uploads so on grafana itself we have the ability to check our projects as well as in this specific dashboard we're looking at GPUs so for our current project it lists the 7900 XTX as well as the 4080 for tests it lists all the different tests that we've run for a given project and then there's parameters so in this case we only specify games 1080 so that's the only one that shows up on the list and then there is the option for the bench which essentially that takes the actual PC name when when markbench logs it to help us identify which system we actually ran on and then lastly is the tag which is blank because we didn't label any tags for anything that we needed so from here we can pull up any of the sessions so let's take a look at dota 2 first here so as you can see here it's kind of a little bit of a mess if you don't know what you're looking at here but effectively this is the beginning load this is the main menu and then we have our little load screen and then we have that initial bit before the actual benchmark occurs the actual bits that we need are from this valley to this valley and then this last bit here is when it goes back to the main menu so that's how we how we kind of know which game it's all specific you kind of have to know what you're looking for at the beginning of like determining oh which part do I actually need to use so what we all all we have to do at this point is just drag from there to there apply the correct time range sometimes it skews it by a second if we do that that gives us our run and then basically I just do that for all the sessions that we have and then once we finish trimming them we take the data and we put it into our trim session tool on the harbour LTT Labs page and that basically gives us a graph ideally we want all three sessions to be basically almost the same however as I outlined previously red deads a little different there's a bit of rng that occurs during each run so they don't look the exact same on here but the end result of the averages in the first should be the same or within that one to two percent so at this point all we gotta do now is trim all the data so we can send it over to sammy and she can start working on getting those graphs created so let's get to it once the game has finished and the data is uploaded to the server we need to mark an in and out on the benchmark data to eliminate the menus and any loading screens from when we calculate the FPS currently john's doing this manually in grafana by literally looking at the pattern of the frame times in the FPS we're trying to help them out by having machine learning doing something similar by training a model based on the behavior of the GPU its temperature and core clocks and memory usage as well as the pattern and frame times to detect the in and out of a benchmark however that still can be faulty so we're looking at eliminating it all together by using computer vision from the ndi streams as you saw earlier to mark an in and out very precisely as it's close to the frame when the game first loads as we can find we're also looking to apply machine learning and models trained on GPU behavior to identify erroneous runs or bad runs based on historical data now that we've finished trimming all the sessions grafana will actually spit out a list of all the different trim session ids so we can use them to generate our game test report they might ask us how do you know these are good trim sessions maybe your settings were bad or maybe you know it was just every run was terrible that's where our other dashboard comes in redash this basically shows historical trim data that we can use to help verify that these runs are in fact the runs that we need this is something that gets generated over time if we have a game that we haven't done a whole lot of trimming on it will be there will be less numbers to be able to verify against but ideally we want the trend to be the same or similar so in this case if we're looking at cyberpunk on a 4080 on 1080p we're getting 190 average ish to 180 average ish and 154 to 153 for a 1% so what we can do now is we can use our we can compare our trimmed sessions versus historical data to verify that it's all correct so what we're going to do is we're going to go on to our harbor Labs tools again and we're going to go through and we're going to start creating an actual test report so what we do is we want GPU by component we're not going to include a line chart because we were just looking at those so we hit create now what we do is we take all of these session IDs and we hit add let me do it for the other card and so what that does now is it actually generates a report for us to be able to see visually both with a bar graph as well as just the numbers what our actual results are so if we're looking here at cyberpunk 2077 on 1080p the 4080 for these three runs that we just did has an average of 193 and a 1% low of 154 so we're actually above our previous average now these numbers were with a reference card obviously the accelerates a little bit over a clock so it stands for reason that 193 seems accurate and so we can keep scrolling down so taking a look at dota 2 we can see that the numbers appear to line up with what i have experienced in the past we don't have a whole lot of historical data on it currently because it is one of our newer harnesses that we actually have been developing and i've kind of just been beta testing but based on my personal experience with it these numbers do appear to be accurate okay so lastly let's take a look at red dead 2 here so now we've taken a look at some of the numbers we can publish this report and get it sent off to sammy to start generating the graphs so what we need to do first is put the report name which is usually the project slug um so 40 versus 7900 x tx 2023 results in some cases if we decide to have separate reports based on like ray tracing versus non ray tracing stuff like that even though those headers of like the games 1080 um help us filter that if in some cases we just want to look at those bar graphs separately and the data that they've generated sometimes we will append uh rt results or dlss results or it's just a description of effectively what the reports based on so we're just going to do results for 40 80 dash vs dash 7900 x tx dash 203 person then for a reference component basically what this does is it generates a relative percentage um based on the data so effectively like oh we can see that the 40 80 is negative 1.62 percent in performance versus the 7900 x tx we can choose whatever component we want it's helpful for when we are looking at our big GPU reviews because we can actually be like okay we want to focus on this component it's not the top of the stack but like for example the rx 7600 we want to take a look at and see how it compares to everything else and it'll allow us to to basically find these relatives a lot easier than doing the math ourselves so it's very helpful in that regard um so let's go ahead and we're going to click publish now this will create a report so if we actually go over here we can see we actually have the 40 80 versus 7900 x tx results we click on that that basically gives us the same report um that we were just looking at so i'm just going to copy the url here into an excel sheet that i was working on go ahead and save this we'll get this sent off to her and then she can start creating our graphs hi my name is sammy i'm the data visualization specialist for the lab um i started a few months ago and ever since i started we've made some changes to the graphs um previously they were done by the writing team um but since then i've taken over that role um and yeah i'm here to walk you through uh some of the design choices that we made as well as um the process by which we make these graphs for our videos previously the graphs that we had had a lot of limitations in design choices um in terms of placement or colors or fonts that we used mostly because we were making them in excel nowadays we're making them in a combination of python and uh doby illustrator as well as adobe illustrator plugins mainly um one plugin that's specialized for graphing uh called datlion datlion allows us to make customizations that we weren't able to do previously um so one of the first things i did uh was use the style guide that was developed by our creator warehouse graphic designers um they had kind of set out a palette as well as some fonts that i could use and toy around with i tried to maintain a design design philosophy that focused on eligibility um so as we've kind of gone through multiple iterations of the graphs since i started i've made changes to emphasize on that based off a lot of feedback that we've gotten from our viewers in terms of design aesthetic um one of the things that i really like to focus on was an almost laboratorial or chalkboard kind of vibe um we wanted to evoke something that was very similar to the nasa kind of retro space age kind of look we also incorporated a more modern twist um inspired by the cyberpunk genre as well as more sci-fi films that have come out in recent years we've focused on using as well a palette that is uh as accessible as possible for example the choices um of which colors come first in our bar graphs we chose between colors that were very contrasting another thing we did was uh we put the colors into um different accessibility tools to kind of measure how visible our color palette would be for people with different types of color blindness um and we've essentially developed a order and a palette that is as accessible as possible across as many different types of um color blindness as possible in order to make our graphs legible so this is an example of one of the game graph templates that i've developed um one of the first things that i noticed when i came in was that the previous graphs um their title didn't really jump out at you one of the things i really wanted to do is make use of like kind of like the very like stampy Labs font that we have to kind of grab attention um for me one of the things that i want to focus on is like what is what am i looking at and number two what is this me and so the first part what am i looking at is um kind of answered immediately by this very bold and very loud um title or header um and so we've added some design elements sometimes they'll be added in especially if a graph is you know maybe maybe a little more bare if they're only like two or three things that we're comparing across but you know if they're 12 13 CPUs GPUs whatever that we're comparing across then obviously we won't have no space for that we then have the bars themselves um the bars themselves are sorted now by uh best to worst um we'll still keep the higher is better um in the subtitle of the graph however um we just want to make it like at a glance more obvious um and another design element that the editors have helped me incorporate was um a marker that kind of pulses right next to the GPU that we're talking about previously one of our concerns and the reason why we weren't able to uh sort by best to worst was because we were worried that if we swapped around the order of every single GPU it'd be hard to follow oh which one is where um and so that kind of solved our problem and ever since then we've kind of adopted this philosophy of sorting by best to worst as best as possible we've kept the grid lines kind of sparse because we want to make sure that the graph isn't overcrowded with different design elements um and another design choice we've made particularly with the game graphs and the FPS um graphs is that we've put one percent low on top and we've made it a bolder color than the average so we've decided that one percent lows are more indicative of overall gaming performance because average values can get pulled up by um high FPS anomalies um that might happen over the course of a game when the one percent lows are really low uh the gameplay ends up appearing as really choppy and so that's one of the reasons why we think that actually the one percent lows the statistic or the data point that we really want to highlight instead and so one thing that's been added since we've made a few changes around the lab is that we've added this uh project slug that follows a project from the beginning to the end and another thing that we've added is also this version number um so we will change and update the version number every time of new set of graphs is cut um and that will also ensure that we have the most updated version of graphs in our video and finally we have our Labs logo in the bottom bottom right uh this Labs logo essentially tells viewers that this graph was developed by Labs with Labs data so now we're gonna get started with our graphs um i've already gotten a uh spreadsheet from adam with the list of graphs that he wants me to do uh three of them being the game graphs so one for each game and then uh the blender benchmark graph and as you can see here it says um cyberpunk 2020 2077 uh red dead redemption 2 dota 2 blender 3.6.0 and higher is better and includes all of the values for the presets as well as the resolution x-axis label project label version number and all of that so that allows me to know what exactly writers want included in their video and what data they need as well and there's also room for writers to add descriptions like if they want to include some kind of like asterisks at the bottom with notes or if they want to cut out a certain GPU or CPU or if they want like another version of the exact same graph with extra information or less information included we're just gonna open up a new file and that will be one layer for each graph as well and i will copy and paste um the layers from our template so for our first three graphs it's going to be a game graph and so for graph number one just gonna paste it in and align it you can see that there are already numbers in the template that's just to help um serve as a placeholder so that i know what graph is which uh that data will be scrubbed later and then the final graph is our blend blender benchmark so i'm just going to copy our three series benchmark graph now i'm going to import all of these project labels into illustrator using visual studio code and a python script that i've written so here i'm just gonna load in the packages that i need and here in chart names i'm just gonna pull up my file and file explorer and copy the path to the file and paste it into here um and we'll see here one two three four graphs um with all the graph titles subtitles text and uh project label version number and now we know that there are four graphs so i'm just gonna make sure that this number here is four if there were 50 i'd do 50 since there are four i'm just going to type in four um and there are no y-axis labels so we're just going to skip over that and then press enter and as we press enter we can see that it'll man automatically change everything so we went from you know all the placeholder text to all of this text and let's just double check that everything is right i'm going to open up harbour to look up the game test report and i'm going to download it as the csv we'll see here that the labels for the gpus are a little long um we don't really need the NVIDIA gforce part so i'm just going to cut that out it's going to be rtx 4080 i'm just going to select the graph that i'm working on at the time let's make this full screen select the graph and i'm just going to delete everything that i have and make sure everything clears yep it's saying that i have no items at all to graph and i'm just going to paste in what i have and so here we see um because with our game graphs we like to sort uh highest to lowest because higher is better um and it does that automatically um the legend will stay the same uh one percent low on average result so this is a benchmark graph it has three series i had a um three series template ready okay i'm done now i've put in the data for all of the graphs and then we're just going to make sure that all the data is correct so what i'm going to do is um look over um the data based off of the spreadsheet that i pulled a cyberpunk for example for the 4080 is um 154 193 okay so now all the data is included um we're just going to change up the uh x axis a little bit so you can see here that there are one two three four five six seven different lines and that's a little too much for my taste i feel like that's a little crowded and by default our graphs have an interval of 30 we prefer to have it at 60 but sometimes when the graph results are really low like having 60 doesn't really make sense so i'm just going to change it here to 60 so that it's a little less crowded and we can see 60 120 and 180 um with this one i'm going to do about the same because it's kind of within the same range i'm going to pull up go back to illustrator instead of exporting everything and having you know people look over it and then give me feedback individually what i'm going to do is i'm going to um share it for review it what it's going to do is create essentially like a google doc where people can add comments so i'm going to um set it to anyone with the link can comment and copy it and i'm going to uh send it off to adam and john so i'm just going to show everybody what it looks like and this is essentially what it looks like for john and adam um they're just going to look over it for multiple things they're going to make sure that all the labels are correct um and they're going to make sure that the uh data is correct it matches up with the right card and it matches up with the right um game or test uh here if they do you find something for example um they can leave a comment so they can highlight this and uh they can just write a comment and say uh for this one this data is actually for dota 2 not cyberpunk theoretically if i made that mistake okay so now i've sent off my um graphs to adam and john and they've reviewed it and adam doesn't have any comments and john has a question about um the specifics of like when we're going to use commas for the graphing numbers and so um you know i can reply to him and i'll say we're adding in commas whenever uh numbers are above a thousand if we had any changes to make then i would go in i would make the changes and then i'd just update the content um since we don't have any changes to make what i'm going to do is prepare for export and adam will be the one to send it off to the editing team and we're going to put it in the video yeah that's pretty much it on my end okay so at this point i received the graphs from sammy so what i do now along with adam as we go through when we kind of take a look at the graphs he does sort of a subjective look on the presentation of the graphs i do more of a data analysis make sure everything was inputted correctly maybe something got copied into the wrong line or something like that so basically i just have to pour over the data here make sure everything looks right so we just quickly scan through blender here 50 48 20 10 this blender looks good and then if we hop back over to the the game test report we can quickly look at all of them up here so if we go to dota 2 so this point now i've basically verified that all the data looks correct um assuming that adam doesn't have any other issues then we go ahead and get these all set up sent off to the editors and that's when they start cutting the video after all of that we then go into post production where we provide the assets and the footage to the editor we have several more steps of qc that occurred during this um first off you have the editor and writer review where the writer sits down with the editor watches what the editor has created and um points out any sort of fixes whether they're stylistic or content fixes that need to be done then once those fixes are implemented they send a pass to Labs Labs looks over it Labs points out any fixes typically are their fixes are going to be more focused on the correctness and the data side of things and also maybe kind of small nitpicks less so about the stylistic side of things but we still make sure that we provide Labs input into that because sometimes you can lose the exact meaning of stuff just through like a couple of different wardings and we want to make sure that that's always communicated correctly then once Labs reviews it the editor implements all those fixes there is another review with the writer and Labs to make sure that all the fixes have been implemented correctly once that is done we then upload it to Floatplane for our ecc squad to take a look at the whole point of ecc squad is to get like a ton of knowledgeable people to have eyes and ears on exactly what we've made because they'll notice the smallest kind of details when you have like a dozen sets of eyes you'll just find so many small things and they're always great and once we've done that we then review the suggestions that the ecc squad has made and if they need to be implemented we do that again we also check with Labs to make sure that those corrections are in fact correct implement them make sure they're implemented correctly and then finally after all of that a video gets released update your BIOS kids"}