HBO NOW outage proves OTT industry needs to look QoE problems in the teeth

Samantha Bookman, FierceOnlineVideo

In trying to corral all the industry issues raised during last week's OTT Executive Summit, I was reminded of a classic Monty Python skit featuring the Spanish Inquisition and its ever-growing list of priorities, which forces a band of inquisitors to continually halt, mid-spiel, and tack on to their bold speech.

There are monetization issues to resolve. New variants on business models to test. Audience engagement and retention concerns. Content licensing costs. Encoding standards to decide on.

But one important issue quickly raised its head above the others this week: quality of experience, which emerged as a central topic of discussion during the one-day meet and greet among online video heavyweights and startups alike.

That QoE focus stayed front and center because, much like the Spanish Inquisition, the unexpected -- and yet somehow, totally expected -- managed to happen over the weekend: HBO Now's stream of Game of Thrones sputtered and died just as the most anticipated episode of the season spooled up. The viewer rage that boiled up during the outage, compounded by a perception that HBO was slow to address the issue (it didn't post about the stream being down until 43 minutes into the penultimate season 6 episode) pointed to an issue that the OTT industry is finally beginning to own up to: QoE is a problem. A big problem.

Game of Thrones battle scene

Game of Thrones illustrates the classic problem of traffic spikes and network congestion. (Photo courtesy of HBO Now)

Failure to get a handle on streaming quality could threaten many OTT providers' existence -- because it could torpedo the perceived value with consumers that the industry has worked so hard to achieve.

Game of Thrones wasn't the only big-ticket, live-streamed event happening on Sunday night. While two fictional characters prepared to square off against each other in the uber-popular show's penultimate episode, LeBron James was playing perhaps the greatest game of his career with Cleveland against Steph Curry and Golden State in the NBA Final, streaming on WatchESPN. And a new WWE heavyweight champion was crowned in a match live-streamed by fans on the WWE Network.

According to a Conviva blog post, the NBA Finals Game 7, Game of Thrones, and the WWE championship ranked first, second and third, respectively, on Sunday night. Despite HBO Now's service going down, the "Battle of the Bastards" took a 27 percent share of online viewers, and nearly all its audience engaged with the show for the entire hour. The Cavaliers-Warriors matchup had twice as many viewers, however.

Conviva recorded an all-time high of 25 million total plays between 5 p.m. and 9 p.m. PDT on Sunday.

"As you might imagine, this drove unprecedented traffic in the world of live OTT streaming video. We saw an all-time peak in concurrent viewers that was 7 percent higher than our last peak during the 2014 World Cup when Germany took on the U.S.," said Ed Haslam, chief marketing officer at Conviva, in the blog post. The analyst firm collected data from over 2 billion cross-screen video players throughout its publisher network.

While Conviva doesn't typically provide detailed breakdowns of its measurements, WatchESPN confirmed that the viewer traffic for Game 7 outstripped its World Cup streaming on the TV Everywhere app. Audience numbers were pretty huge on WatchESPN, with 1.76 million unique viewers during the final.

So, why do those fantastic numbers matter when it comes to quality of experience? I don't really have to answer that for you. It's clear that more viewers than ever are engaging with their favorite content -- sports, scripted series, you name it -- than ever before, and the concept of destination or event-oriented viewing has truly caught on. That sets the stage for a new round of video traffic spikes, which could become as troublesome as they were as little as four years ago. Remember the infamous Christmas Eve Netflix meltdown of 2012?

"HBO's recent Game of Thrones streaming complications are the latest example of the challenges of OTT video delivery," said Kurt Michel, senior director of marketing at IneoQuest, in a statement. "With the increase of high-demand streaming for popular television shows and sporting events like the NBA finals and upcoming Olympics, OTT video providers must consider every part of their infrastructure, anticipate failures, and have tools and plans in place to detect and address them."

But that's only one part of the solution for online video providers. Figuring out a streaming problem when it occurs – and fast – is really critical. While HBO said in a statement about its outage on Monday that "HBO NOW issues were quickly recognized last night by MLBAM," its streaming platform provider, many subscribers were upset because HBO didn't offer any announcement or information about the outage until 43 minutes after it occurred, finally posting on its Twitter account that it was working on the issue. That left some fans wondering how quickly the SVOD service actually noticed a problem.

"Because the multi-vendor OTT streaming pipeline is complex, the first step is often identifying that there is a problem, and separating the known-good elements from the suspected ones," Michel explained. While some smaller OTT providers rely on Twitter or Facebook for outage reports from viewers, "Social media monitoring and detection is not a solution. A proactive, end-to-end monitoring system is the best insurance against failures occurring, and enables the fastest response when they do occur."

Of course, it's no coincidence that IneoQuest offers just such a monitoring system. But it's not the only player in the OTT game developing a more effective QoE offering. Akamai recently launched a Broadcast Operations Control Center (BOCC) that monitors live streams end to end. Cedexis relies on last-mile, endpoint monitoring of devices to detect issues.

Last year, I pointed to monetization, and particularly solving some of the issues around advertising, as a critical problem for OTT providers to work out. That's still an outstanding problem, but it needs to take a back seat temporarily to QoE. Viewers don't care whether your stream is monetized or not – they want to see the content they expected to see – and in the case of HBO, content that they paid to see.

This is something SVOD providers in particular need to get a handle on. It's evident that the pay-TV industry, with lots more money and content licensing agreements in hand, is closing in on OTT providers. Many pay-TV providers also deliver the broadband that makes OTT streaming possible. If they're able to provide IP-based video at better than the best-effort level that currently rules online video delivery, and do so at a competitive price, consumers are likely to flock back to the cableco fold just so they can get their content delivered reliably. -- Sam