The word latency is so geeky I may’ve lost half of you already.
Video buffering? Not so much. You know what it is, and the thought alone may’ve raised your blood pressure.
But between those two words is a pile of invisible technical debt that, unless we solve it as an industry over the next 12 to 24 months, will cause economic leakage on the biggest “win-win” for the business of TV since cable news:
We can make streaming TV far more delightful for viewers—and more profitable for our media ecosystem—by solving streaming TV’s relatively unknown Achille’s heel: Latency.
This post seeks to share brand new consumer data on this topic, from my #FutureOfTV 7 Survey Report, which is available for free by signing up here.
Also, I’ll share a view on the problem and solutions for C-level media execs, live TV producers and chief innovation and product officers that are beginning to understand why tackling the technical challenge of streaming latency is key to unlocking greater viewer engagement and stakeholder profit.
Step one? Problem description
Compared to cable, satellite and broadcast TV signals that reach most homes in about six or seven seconds, internet video streaming takes much longer. When you stream over-the-top, that signal may arrive thirty, forty-five or even up to sixty seconds behind real-time onto computers, phones and connected TVs.
Does it matter?
It depends on the nature of the viewing experience we want for our audience. The richer and more delightful the possibilities, the more we need to care about precision-controlled multi-platform video latency. On-demand viewing? Latency matters less. Live sports? Award shows? Latency matters more.
For example: Some avid sports app and social media users may well know about streaming latency because they’ve suffered “spoilers” from it.
This could’ve been in the form of a sports Tweet push notification that arrived on device while they were watching a football stream forty-five seconds behind real-time. (Now they know what “prevent spoiler mode” is.)
Others might have read an article warning them of this phenomenon, like this piece from CNET published in February of 2020. Still more have suffered from the scenario the article warns about: Imagine you’re watching a baseball, football, or soccer match. The game is close and nearing the end. You’re streaming the game. Your neighbor has cable TV. Suddenly, his apartment erupts in wild cheer, making it clear that the home team won—and spoiling your excitement since you’re still forty-five seconds behind.
Don’t want to ruin someone’s Super Bowl? Broadcast synchronization latency
These “spoiler” ideas alone have mobilized some in our industry (like me, not that long ago, here!) to call for broadcast-synchronized latency across all video players—from mobile phones to streaming players to connected TVs—but we’re still in an early phase of debate about exactly what constitutes “low latency."
Why the hold-up?
First, low latency isn’t easy and it seems to be getting harder amidst a widening fragmentation of streaming video players, encryption schemes and more. Certainly, the business model will continue to grow for CDNs who will have plenty of lifting to do, but it’s not at the tipping point.
And here, my latest #FutureOfTV consumer data backs up the point.
Only about 26.8% of our “live sports and news” panel were aware of streaming’s high latency and found it annoying. To broaden the audience to “all sports viewers,” we used other data to adjust and extrapolate that around 28.9% of all sports viewers are aware—and annoyed—by streaming’s high latency.
My takeaway? Even narrowing the sample to sports viewers, it’s not a terribly high number. What it illuminates for me, is that “Avoiding Spoilers” isn’t a sufficiently generalized or painful use case to catalyze an industry upgrade to low latency streaming. “Spoilers sounds bad, but the severity of the problem seems a thin in the real world, from this dataset. It’s not causing sufficient aggregated grief to accelerate the adoption of agreed-on standards.
But here’s the rub. Guess who over indexes on “Yes I know and it’s annoying.” Males, 35 – 44. Precisely the target for sports betting and gamification use cases that are being placed at the center of major sports media investments today.
Watch out! We know from human-computer interaction research: There’s no right answer
Instructive on this point is an article titled “40 Years of Searching for the Best Computer System Response Time” by the prestigious Interacting with Computer journal: There is no single definition or guideline for what constitutes fast-enough. It depends on the task.
Among other things, the paper highlights human-computer interaction researcher Dr. Steven Seow along with his Response-time categorization framework along four basic tiers. Dr. Seow incorporates an important new idea into the academic understanding: The expectation of the user matters.
Expectation Response time
Instantaneous: 100 - 200 milliseconds
Immediate: .5 - 1 second
Continuous: 2 - 5 seconds
Captive: 7 - 10 seconds
It’s worth noting that computer technology has steadily improved video viewing experiences, but primarily along three axes:
- Better picture quality
- More content choices
- Easier access
The fourth dimension: Dawn of interactivity, multitasking and mobile
But mobile apps, social media, 360-degree video, augmented reality and video gaming has added sharing, interactivity, conversation and chat into our media consumption behavior universe as a fourth dimension of quality—and it’s changing viewing expectations in major ways.
Does video interactivity need to be instantaneous? Or will immediate or continuous suffice?
Again, it depends on the experience being designed. Is it a video call? Then yes. And given the pace and importance of remote live production, real-time communication protocols will increasingly be a part of production’s digital arsenal.
Or is it a mobile app experience featuring instant replays using TrueView immersive volumetric video to a fan who’s on the top deck of a gigantic football venue?
In that moment, the experience offers to the user an ability to directly manipulate video camera angles on their device. Using our human-computer research framework above, along with a bit of sports fan sensibility, we might say this:
The user experience needs to be available within a second or two behind real-time. And once it begins, it then needs to feel nearly instantaneous, in alignment with the direct low-level control that the user might expect in that context.
Likewise, for a COVID-19 era version of this replay feature delivered at home.
What about Twitch video game livestreams and eSports?
Video games move exceptionally fast and chat usage in these environments adds furious and fun to the speed. This is defining a new genre of TV interactivity. In it, six to seven seconds qualifies as an eternity.
Latency for interactivity: TV’s future use cases
What happens when sports leagues like the NBA and the NFL work with networks and sports apps to bring innovative, Twitch-like chat experiences to a live sporting event?
It’s happening. During the 2020 NBA Playoffs, NBA’s Digital app offered users such an experience, in partnership with TNT Overtime.
For chat-based Watch Party experiences, it’s important for the participants to have a feed that is low latency and also synchronized. That synchronization could be with the existing broadcast feeds—or possibly with other users invited to the party. It depends entirely on the creative vision behind the viewing experience being designed and innovated.
Latency for gamification: Daily fantasy, sports betting and free-to-play
Also coming? The gamification of TV.
It’s been over two years since May of 2018 when the U.S. Supreme Court struck down the anti-betting law PASPA as unconstitutional. Since DraftKings went public in April of 2020 it has surged from $19 to over $51 a share on projections that the market for sports betting will ultimately be worth $18 billion.
But download the apps and you’ll find both DraftKings and its major competitor FanDuel, owned by Flutter Entertainment, are building a business not only on traditional sports bets but also on real-time contests during the game more broadly.
These “gamified” interactive mechanics, like “predict-the-next-play,” or “who will score next,” have promise to delight viewers, perhaps helping channels retain viewers through less-than-exciting games.
They also—it bears repeating the obvious here—require precisely-controlled latency in order to meet the needs of all stakeholders, to say nothing of requirements that regulators or integrity partners have to ensure fairness in these experiences.
From sports to live news, awards, streaming auctions and general interactivity
Finally, while sports provide fertile ground for use cases demanding low latency video, the vast amount of television viewing is still what’s known as live and linear TV. National and local news, game shows, reality contests, major series premieres and finales can all benefit from these real-time, conversational and interactive video formats. After all, TV advertisers appreciate this:
Having millions of viewers tuned-in at the exact same time is more valuable than delivering the identical number of person-hour-viewing that might take place, for example, over months or years.
And that kind of tent-pole live broadcast with trivia and polling on a second-screen is still very much in development.
What’s next? Can it be done?
Both Twitch and YouTube enable high-performance low latency live streaming to their desktop browser apps, at scale. They accomplish this – at least on their desktop apps -- by optimizing their encoding pipeline, content delivery network, content protection workflows and video players.
And even still, with the hyperscale budget of Google and Amazon behind these apps, most devices in homes today can’t easily deliver this same low latency experience that the browser can.
Two pieces of good news.
First, there are two widespread efforts to solve this challenge in the context of adaptive bit rate streaming.
There’s Apple’s LL-HLS spec, or Low Latency HTTP Live Streaming. This is an extension of the HLS work done by Apple’s technology team to popularize streaming and it calls for—among other things—additional “hints” provided to the video player about what’s coming next.
Then, there’s an emerging standard container format known as Common Media Application Format which supports different flavors of adaptive bit rate streaming (including HLS) with a single set of encoded video files – and a way to break such a file into ever-tinier chunks to be encoded and transferred.
But Covid-19 hasn’t just accelerated trends, it’s significantly changed the trajectory of certain behaviors and technologies. One? Video calls and WebRTC. I’ve been following WebRTC for years and the idea of it as a core technology in the premium media landscape was a longshot in February of 2020. Now, it’s quite possibly the next big thing in video tech.
WebRTC and real-time streaming
The global pandemic has accelerated the use of tools like Zoom video even in live production. Video calling has long used a very different technical protocol known as WebRTC, or web real-time communications. WebRTC can achieve latencies much closer to the instantaneous or immediate range, depending on the audience scale at which it’s delivered.
For a video call with under fifty participants seeking to achieve complete fluidity in the conversation, users will feel best with the instantaneous latency provided by WebRTC.
But what about the some of the newest and most innovative viewing experiences previously discussed above?
At this point, you probably know my next line:
It depends.
So, why not switch the tables here. You tell me. What is your dream viewing experience?
Brian Ring is Publisher of FutureOfTV.Live, a quarterly D2C survey, report and zoomcast designed to generate unique insights into viewer experience. Mr. Ring also provides strategic and product marketing services to an impressive and growing list of world-class clients.
Industry Voices are opinion columns written by outside contributors—often industry experts or analysts—who are invited to the conversation by FierceVideo staff. They do not represent the opinions of FierceVideo.