{"video_id":"fp_GPmCZIbpsM","title":"Apple Vision Pro and Real-Time Operating Systems","channel":"Techquickie","show":"Techquickie","published_at":"2024-03-05T20:54:00.230Z","duration_s":242,"segments":[{"start_s":0.0,"end_s":5.18,"text":"The main thing that makes the Apple Vision Pro different from VR headsets that came before","speaker":null,"is_sponsor":0},{"start_s":5.18,"end_s":8.94,"text":"is that even though it has a fully opaque covering in front of your eyes,","speaker":null,"is_sponsor":0},{"start_s":8.94,"end_s":13.5,"text":"it feels like you're looking right through it. Obviously, part of this is Apple's choice","speaker":null,"is_sponsor":0},{"start_s":13.5,"end_s":19.02,"text":"of cameras and screens, but a major contributor is its absurdly low latency.","speaker":null,"is_sponsor":0},{"start_s":19.02,"end_s":22.82,"text":"That is the delay between light hitting the cameras on the outside of the headset","speaker":null,"is_sponsor":0},{"start_s":22.82,"end_s":25.82,"text":"and the image being displayed on the internal screen.","speaker":null,"is_sponsor":0},{"start_s":25.82,"end_s":29.58,"text":"Apple says that 12 milliseconds is about the highest amount of latency","speaker":null,"is_sponsor":0},{"start_s":29.58,"end_s":33.18,"text":"the human brain can tolerate before you start to notice a mismatch","speaker":null,"is_sponsor":0},{"start_s":33.18,"end_s":37.1,"text":"between a camera feed and what's actually going on in the physical world.","speaker":null,"is_sponsor":0},{"start_s":37.1,"end_s":41.42,"text":"And lo and behold, the listed latency of the Vision Pro is 12 milliseconds.","speaker":null,"is_sponsor":0},{"start_s":41.42,"end_s":46.22,"text":"Although this sounds a bit like marketing speak, early reviews of the headset have been positive","speaker":null,"is_sponsor":0},{"start_s":46.22,"end_s":50.18,"text":"with reviewers noting that there was little to no perceptible latency.","speaker":null,"is_sponsor":0},{"start_s":50.18,"end_s":53.66,"text":"So how did they do this? Apple has apparently pulled us off","speaker":null,"is_sponsor":0},{"start_s":53.66,"end_s":59.7,"text":"with a chip they're calling the R1, along with a special operating system, VisionOS.","speaker":null,"is_sponsor":0},{"start_s":59.7,"end_s":64.46,"text":"Essentially, VisionOS is partially a real-time operating system","speaker":null,"is_sponsor":0},{"start_s":64.46,"end_s":69.06,"text":"that's quite a bit different from more familiar platforms like Windows or iOS.","speaker":null,"is_sponsor":0},{"start_s":69.06,"end_s":74.06,"text":"A real-time operating system or RTOS is used in situations where low latency","speaker":null,"is_sponsor":0},{"start_s":74.06,"end_s":78.18,"text":"is absolutely paramount. Think for example, about avionics systems","speaker":null,"is_sponsor":0},{"start_s":78.18,"end_s":82.86,"text":"that use real-time data to help control airliners like the Boeing 787","speaker":null,"is_sponsor":0},{"start_s":82.86,"end_s":86.78,"text":"or the software inside self-driving cars so they can react to hazards","speaker":null,"is_sponsor":0},{"start_s":86.78,"end_s":91.82,"text":"in a similar amount of time or faster to human drivers if they ever become a real thing.","speaker":null,"is_sponsor":0},{"start_s":91.82,"end_s":96.78,"text":"But how does an RTOS accomplish this? Fundamentally, a real-time operating system","speaker":null,"is_sponsor":0},{"start_s":96.78,"end_s":102.1,"text":"enforces rules that certain tasks are guaranteed to be completed in a fixed amount of time.","speaker":null,"is_sponsor":0},{"start_s":102.1,"end_s":107.78,"text":"Instead of a situation where most of the running processes have to wait their turn to go through the processor,","speaker":null,"is_sponsor":0},{"start_s":107.78,"end_s":112.04,"text":"an RTOS makes sure important tasks are completed first.","speaker":null,"is_sponsor":0},{"start_s":112.04,"end_s":115.56,"text":"An RTOS is also often deterministic,","speaker":null,"is_sponsor":0},{"start_s":115.56,"end_s":120.9,"text":"a fancy way of saying that the same inputs will result in the same outputs every time.","speaker":null,"is_sponsor":0},{"start_s":120.9,"end_s":124.24,"text":"In more concrete terms, this means that the RTOS","speaker":null,"is_sponsor":0},{"start_s":124.24,"end_s":128.32,"text":"has to ensure that every task going through the processor will meet its time constraint","speaker":null,"is_sponsor":0},{"start_s":128.32,"end_s":133.36,"text":"no matter what other tasks try to interrupt. Looking a bit more specifically at the Vision Pro,","speaker":null,"is_sponsor":0},{"start_s":133.36,"end_s":138.32,"text":"the headset can actually keep the visual pass-through of your actual surroundings working","speaker":null,"is_sponsor":0},{"start_s":138.32,"end_s":144.0,"text":"even if the rest of VisionOS crashes. It appears that the R1 chip is running an RTOS","speaker":null,"is_sponsor":0},{"start_s":144.0,"end_s":149.52,"text":"that handles data related to depth, eye-tracking, and head-tracking while the rest of the Vision Pro's","speaker":null,"is_sponsor":0},{"start_s":149.52,"end_s":152.68,"text":"non-real-time software is handled elsewhere,","speaker":null,"is_sponsor":0},{"start_s":152.68,"end_s":158.16,"text":"namely by the more general-purpose M2 chip that you can find in Apple's current lineup MacBooks.","speaker":null,"is_sponsor":0},{"start_s":158.16,"end_s":162.6,"text":"This makes sense as real-time operating systems are usually quite small.","speaker":null,"is_sponsor":0},{"start_s":162.6,"end_s":168.4,"text":"Think of a few megabytes of code instead of the many Gigabyte behemoths that run our PCs.","speaker":null,"is_sponsor":0},{"start_s":168.44,"end_s":171.52,"text":"It's a safe bet that the code running on the R1","speaker":null,"is_sponsor":0},{"start_s":171.52,"end_s":176.4,"text":"is fairly lightweight and focused only on keeping the crucial low-latency aspects","speaker":null,"is_sponsor":0},{"start_s":176.4,"end_s":179.8,"text":"of the headset working properly, allowing it to be more reliable","speaker":null,"is_sponsor":0},{"start_s":179.8,"end_s":183.0,"text":"even if something goes wrong with the rest of the headset software.","speaker":null,"is_sponsor":0},{"start_s":183.0,"end_s":186.08,"text":"Apple hasn't given us tons of details on the R1,","speaker":null,"is_sponsor":0},{"start_s":186.08,"end_s":189.92,"text":"but we do know that it has 250 gigabytes per second","speaker":null,"is_sponsor":0},{"start_s":189.92,"end_s":195.28,"text":"of memory bandwidth, the same as AMD's RX 6600 XT GPU.","speaker":null,"is_sponsor":0},{"start_s":195.28,"end_s":201.76,"text":"Although this might seem like a huge amount for a hyper-specialized chip that's running a small RTOS,","speaker":null,"is_sponsor":0},{"start_s":201.76,"end_s":206.56,"text":"the chip does have to process data from 13 different cameras,","speaker":null,"is_sponsor":0},{"start_s":206.56,"end_s":211.04,"text":"seven different microphones, and six different sensors all at once,","speaker":null,"is_sponsor":0},{"start_s":211.04,"end_s":216.52,"text":"and all that data flowing to the chip adds up. So even though RTOS's tend to be smaller","speaker":null,"is_sponsor":0},{"start_s":216.52,"end_s":221.12,"text":"in terms of absolute size, they can still potentially handle huge amounts of data.","speaker":null,"is_sponsor":0},{"start_s":221.12,"end_s":226.56,"text":"And this is a godsend if you've just spent 3,500 bucks on a Vision Pro because you don't want","speaker":null,"is_sponsor":0},{"start_s":226.56,"end_s":229.84,"text":"to be losing your lunch just because of excessive latency.","speaker":null,"is_sponsor":0},{"start_s":229.84,"end_s":234.44,"text":"So thanks for watching guys. If you liked this video, hit like, hit subscribe, and hit up one of our previous videos.","speaker":null,"is_sponsor":0},{"start_s":234.44,"end_s":239.48,"text":"If you're into the nitty gritty like this, maybe you'd like our video explaining what a kernel is.","speaker":null,"is_sponsor":0},{"start_s":239.48,"end_s":242.24,"text":"Not talking popcorn, though that is delicious.","speaker":null,"is_sponsor":0}],"full_text":"The main thing that makes the Apple Vision Pro different from VR headsets that came before is that even though it has a fully opaque covering in front of your eyes, it feels like you're looking right through it. Obviously, part of this is Apple's choice of cameras and screens, but a major contributor is its absurdly low latency. That is the delay between light hitting the cameras on the outside of the headset and the image being displayed on the internal screen. Apple says that 12 milliseconds is about the highest amount of latency the human brain can tolerate before you start to notice a mismatch between a camera feed and what's actually going on in the physical world. And lo and behold, the listed latency of the Vision Pro is 12 milliseconds. Although this sounds a bit like marketing speak, early reviews of the headset have been positive with reviewers noting that there was little to no perceptible latency. So how did they do this? Apple has apparently pulled us off with a chip they're calling the R1, along with a special operating system, VisionOS. Essentially, VisionOS is partially a real-time operating system that's quite a bit different from more familiar platforms like Windows or iOS. A real-time operating system or RTOS is used in situations where low latency is absolutely paramount. Think for example, about avionics systems that use real-time data to help control airliners like the Boeing 787 or the software inside self-driving cars so they can react to hazards in a similar amount of time or faster to human drivers if they ever become a real thing. But how does an RTOS accomplish this? Fundamentally, a real-time operating system enforces rules that certain tasks are guaranteed to be completed in a fixed amount of time. Instead of a situation where most of the running processes have to wait their turn to go through the processor, an RTOS makes sure important tasks are completed first. An RTOS is also often deterministic, a fancy way of saying that the same inputs will result in the same outputs every time. In more concrete terms, this means that the RTOS has to ensure that every task going through the processor will meet its time constraint no matter what other tasks try to interrupt. Looking a bit more specifically at the Vision Pro, the headset can actually keep the visual pass-through of your actual surroundings working even if the rest of VisionOS crashes. It appears that the R1 chip is running an RTOS that handles data related to depth, eye-tracking, and head-tracking while the rest of the Vision Pro's non-real-time software is handled elsewhere, namely by the more general-purpose M2 chip that you can find in Apple's current lineup MacBooks. This makes sense as real-time operating systems are usually quite small. Think of a few megabytes of code instead of the many Gigabyte behemoths that run our PCs. It's a safe bet that the code running on the R1 is fairly lightweight and focused only on keeping the crucial low-latency aspects of the headset working properly, allowing it to be more reliable even if something goes wrong with the rest of the headset software. Apple hasn't given us tons of details on the R1, but we do know that it has 250 gigabytes per second of memory bandwidth, the same as AMD's RX 6600 XT GPU. Although this might seem like a huge amount for a hyper-specialized chip that's running a small RTOS, the chip does have to process data from 13 different cameras, seven different microphones, and six different sensors all at once, and all that data flowing to the chip adds up. So even though RTOS's tend to be smaller in terms of absolute size, they can still potentially handle huge amounts of data. And this is a godsend if you've just spent 3,500 bucks on a Vision Pro because you don't want to be losing your lunch just because of excessive latency. So thanks for watching guys. If you liked this video, hit like, hit subscribe, and hit up one of our previous videos. If you're into the nitty gritty like this, maybe you'd like our video explaining what a kernel is. Not talking popcorn, though that is delicious."}