{"video_id":"fp_NRIKgUM2pw","title":"TQ: What exactly is upscaling? ","channel":"Techquickie","show":"Techquickie","published_at":"2020-02-19T17:19:47.328Z","duration_s":272,"segments":[{"start_s":0.0,"end_s":4.88,"text":"There's one big compromise that gamers have had to make for a long time.","speaker":null,"is_sponsor":0},{"start_s":4.88,"end_s":8.68,"text":"You want your games to look better, or do you want them to run faster?","speaker":null,"is_sponsor":0},{"start_s":8.68,"end_s":13.16,"text":"Typically, this has meant turning down your graphics setting to get more frames per second,","speaker":null,"is_sponsor":0},{"start_s":13.16,"end_s":20.8,"text":"especially if you don't have a high-end graphics card. But today, we're instead going to talk about the amount of input lag that gets introduced","speaker":null,"is_sponsor":0},{"start_s":20.8,"end_s":26.48,"text":"when you're trying to upscale a game. We consulted with our friend Amin Shabane over at Merci to put this video together, so we","speaker":null,"is_sponsor":0},{"start_s":26.48,"end_s":33.88,"text":"like to thank him for his help. Now to be clear, I am not talking about your GPU rendering frames from scratch.","speaker":null,"is_sponsor":0},{"start_s":33.88,"end_s":38.48,"text":"What I'm referring to instead is what happens after your GPU finishes rendering a frame,","speaker":null,"is_sponsor":0},{"start_s":38.48,"end_s":45.76,"text":"and either the GPU or your display resizes the image to make it fit a certain resolution.","speaker":null,"is_sponsor":0},{"start_s":45.76,"end_s":49.6,"text":"You can see this if you're running a PC game at below your monitor's native resolution","speaker":null,"is_sponsor":0},{"start_s":49.6,"end_s":55.08,"text":"to improve performance, or if you hooked up an older console to modern flat-panel TV.","speaker":null,"is_sponsor":0},{"start_s":55.36,"end_s":59.6,"text":"There are different forms of upscaling, some of which look nicer than others, but they","speaker":null,"is_sponsor":0},{"start_s":59.6,"end_s":65.52,"text":"all require a certain amount of post-processing time which can introduce noticeable input lag,","speaker":null,"is_sponsor":0},{"start_s":65.52,"end_s":68.84,"text":"meaning that there's a delay between when you press a button or move a thumbstick or","speaker":null,"is_sponsor":0},{"start_s":68.84,"end_s":72.68,"text":"move the mouse, and the corresponding action appearing on the screen.","speaker":null,"is_sponsor":0},{"start_s":72.68,"end_s":78.4,"text":"This can seriously hinder gameplay for obvious reasons, especially in older titles like classic","speaker":null,"is_sponsor":0},{"start_s":78.4,"end_s":84.24,"text":"platforms where responsiveness is a huge part of making the game feel like you remember.","speaker":null,"is_sponsor":0},{"start_s":84.24,"end_s":91.36,"text":"But why does it introduce so much lag? Well to get the image looking as nice as possible, some algorithms look at the frames that are","speaker":null,"is_sponsor":0},{"start_s":91.36,"end_s":97.28,"text":"rendered before and after the frame to be upscaled to better understand what a higher res version","speaker":null,"is_sponsor":0},{"start_s":97.28,"end_s":100.92,"text":"of the same image is supposed to look like.","speaker":null,"is_sponsor":0},{"start_s":100.92,"end_s":105.76,"text":"Then they apply what they think are correct changes to the frame.","speaker":null,"is_sponsor":0},{"start_s":105.76,"end_s":109.76,"text":"This method of analyzing multiple frames that are held in what's called the frame buffer","speaker":null,"is_sponsor":0},{"start_s":110.2,"end_s":114.08,"text":"before they're shown to the user can definitely yield visual improvements.","speaker":null,"is_sponsor":0},{"start_s":114.08,"end_s":120.04,"text":"But not only is this a computationally time-consuming process that adds lag, it can also result","speaker":null,"is_sponsor":0},{"start_s":120.04,"end_s":124.24,"text":"in worse image quality if the frames it's examining were highly compressed.","speaker":null,"is_sponsor":0},{"start_s":124.24,"end_s":129.72,"text":"For example, if you're watching a movie. An alternative approach to reduce lag is to, instead of relying on multiple frames at","speaker":null,"is_sponsor":0},{"start_s":129.72,"end_s":135.44,"text":"one time, have the algorithm look at certain elements of a single frame that human brains","speaker":null,"is_sponsor":0},{"start_s":135.44,"end_s":144.04,"text":"are typically sensitive to. For example, Mercedes-Mersai, Mercedes-Mersay, Mercedes-M classic smart HDMI cable has a built-in","speaker":null,"is_sponsor":0},{"start_s":144.04,"end_s":148.8,"text":"library of objects like edges and textures that we naturally key in on.","speaker":null,"is_sponsor":0},{"start_s":148.8,"end_s":154.72,"text":"Think about how jaggies caused by bad anti-aliasing of edges are often really noticeable to us.","speaker":null,"is_sponsor":0},{"start_s":154.72,"end_s":160.08,"text":"Interestingly, characters' eyes are also a focus as humans are psychologically programmed","speaker":null,"is_sponsor":0},{"start_s":160.08,"end_s":163.72,"text":"to be very sensitive to what someone else's eyes are doing.","speaker":null,"is_sponsor":0},{"start_s":163.72,"end_s":167.52,"text":"This kind of strategy of focusing mostly on key visual elements can greatly reduce lag","speaker":null,"is_sponsor":0},{"start_s":167.52,"end_s":172.68,"text":"time while improving visual quality due to its reliance on predetermined visual cues","speaker":null,"is_sponsor":0},{"start_s":172.68,"end_s":177.72,"text":"for the algorithm to focus on, as well as the fact it only examines one frame.","speaker":null,"is_sponsor":0},{"start_s":177.72,"end_s":182.92,"text":"But like other upsampling methods, it's not perfect, so can we do better?","speaker":null,"is_sponsor":0},{"start_s":182.92,"end_s":186.88,"text":"It turns out the answer is yes, though we might still be some years away from seeing","speaker":null,"is_sponsor":0},{"start_s":186.88,"end_s":193.64,"text":"it becoming widely available. Rather than programming a scaler to spot a few specific elements, computer scientists","speaker":null,"is_sponsor":0},{"start_s":193.64,"end_s":198.36,"text":"have been training artificial intelligences to recognize what more complex objects are","speaker":null,"is_sponsor":0},{"start_s":198.36,"end_s":206.56,"text":"supposed to look like. Accurately scaling an HD image to 4K or even 8K is a very computationally intensive problem,","speaker":null,"is_sponsor":0},{"start_s":206.56,"end_s":211.56,"text":"so large amounts of AI training will reduce the reliance of predefined features and allow","speaker":null,"is_sponsor":0},{"start_s":211.56,"end_s":216.6,"text":"a scaler to recognize anything from whether or not an object is a dog to how it handles","speaker":null,"is_sponsor":0},{"start_s":216.6,"end_s":222.52,"text":"scenes with complicated lighting. We're already seeing this to some extent with NVIDIA's Deep Learning Super Sampling","speaker":null,"is_sponsor":0},{"start_s":222.52,"end_s":227.92,"text":"or DLSS, where a supercomputer is fed with lots of frames from different games and figures","speaker":null,"is_sponsor":0},{"start_s":227.92,"end_s":232.96,"text":"out an algorithm to produce something close to an ideally anti-aliased image.","speaker":null,"is_sponsor":0},{"start_s":232.96,"end_s":236.92,"text":"These algorithms are then pushed out to individual users through software updates.","speaker":null,"is_sponsor":0},{"start_s":236.92,"end_s":240.56,"text":"Not only does this allow gamers to improve how their games look without lowering frame","speaker":null,"is_sponsor":0},{"start_s":240.56,"end_s":245.2,"text":"rates, the more efficient post-processing algorithms optimized through AI should hopefully","speaker":null,"is_sponsor":0},{"start_s":245.2,"end_s":252.04,"text":"make games feel more responsive as well. Don't remember that if you just suck at games like CSGO because you have straight-up terrible","speaker":null,"is_sponsor":0},{"start_s":252.04,"end_s":257.32,"text":"reflexes, AI probably won't help you, so you might want to just give turn-based games","speaker":null,"is_sponsor":0},{"start_s":257.32,"end_s":262.56,"text":"a shot. So thanks for watching, guys. If you liked this video, give it a thumbs up, subscribe, and be sure to hit us up in","speaker":null,"is_sponsor":0},{"start_s":262.56,"end_s":268.76,"text":"the comment section for your ideas about future videos that we should make, about tech topics","speaker":null,"is_sponsor":0},{"start_s":268.76,"end_s":271.64,"text":"that you want explained. We'll do it.","speaker":null,"is_sponsor":0}],"full_text":"There's one big compromise that gamers have had to make for a long time. You want your games to look better, or do you want them to run faster? Typically, this has meant turning down your graphics setting to get more frames per second, especially if you don't have a high-end graphics card. But today, we're instead going to talk about the amount of input lag that gets introduced when you're trying to upscale a game. We consulted with our friend Amin Shabane over at Merci to put this video together, so we like to thank him for his help. Now to be clear, I am not talking about your GPU rendering frames from scratch. What I'm referring to instead is what happens after your GPU finishes rendering a frame, and either the GPU or your display resizes the image to make it fit a certain resolution. You can see this if you're running a PC game at below your monitor's native resolution to improve performance, or if you hooked up an older console to modern flat-panel TV. There are different forms of upscaling, some of which look nicer than others, but they all require a certain amount of post-processing time which can introduce noticeable input lag, meaning that there's a delay between when you press a button or move a thumbstick or move the mouse, and the corresponding action appearing on the screen. This can seriously hinder gameplay for obvious reasons, especially in older titles like classic platforms where responsiveness is a huge part of making the game feel like you remember. But why does it introduce so much lag? Well to get the image looking as nice as possible, some algorithms look at the frames that are rendered before and after the frame to be upscaled to better understand what a higher res version of the same image is supposed to look like. Then they apply what they think are correct changes to the frame. This method of analyzing multiple frames that are held in what's called the frame buffer before they're shown to the user can definitely yield visual improvements. But not only is this a computationally time-consuming process that adds lag, it can also result in worse image quality if the frames it's examining were highly compressed. For example, if you're watching a movie. An alternative approach to reduce lag is to, instead of relying on multiple frames at one time, have the algorithm look at certain elements of a single frame that human brains are typically sensitive to. For example, Mercedes-Mersai, Mercedes-Mersay, Mercedes-M classic smart HDMI cable has a built-in library of objects like edges and textures that we naturally key in on. Think about how jaggies caused by bad anti-aliasing of edges are often really noticeable to us. Interestingly, characters' eyes are also a focus as humans are psychologically programmed to be very sensitive to what someone else's eyes are doing. This kind of strategy of focusing mostly on key visual elements can greatly reduce lag time while improving visual quality due to its reliance on predetermined visual cues for the algorithm to focus on, as well as the fact it only examines one frame. But like other upsampling methods, it's not perfect, so can we do better? It turns out the answer is yes, though we might still be some years away from seeing it becoming widely available. Rather than programming a scaler to spot a few specific elements, computer scientists have been training artificial intelligences to recognize what more complex objects are supposed to look like. Accurately scaling an HD image to 4K or even 8K is a very computationally intensive problem, so large amounts of AI training will reduce the reliance of predefined features and allow a scaler to recognize anything from whether or not an object is a dog to how it handles scenes with complicated lighting. We're already seeing this to some extent with NVIDIA's Deep Learning Super Sampling or DLSS, where a supercomputer is fed with lots of frames from different games and figures out an algorithm to produce something close to an ideally anti-aliased image. These algorithms are then pushed out to individual users through software updates. Not only does this allow gamers to improve how their games look without lowering frame rates, the more efficient post-processing algorithms optimized through AI should hopefully make games feel more responsive as well. Don't remember that if you just suck at games like CSGO because you have straight-up terrible reflexes, AI probably won't help you, so you might want to just give turn-based games a shot. So thanks for watching, guys. If you liked this video, give it a thumbs up, subscribe, and be sure to hit us up in the comment section for your ideas about future videos that we should make, about tech topics that you want explained. We'll do it."}