NEW YORK -- Standardization of online video quality -- particularly quality of experience (QoE) -- is more important than ever as more OTT services enter the market, adding to the complexities of video delivery, attendees and panelists at the fourth annual OTT Executive Summit here agreed. The problem: no one can agree yet on what those standards should be.
"We need consistent measurements for quality in network," said Kurt Michel, head of marketing at IneoQuest, during a panel session on OTT video QoE. Further, "there needs to be agreement across the industry on measurement."
Currently, what qualifies as successful QoE differs at every single point along the online video delivery ecosystem, Michel noted, with each provider -- from encoding service to transport provider to CDN and so on -- utilizing different measurement methods, different criteria, and different benchmarks for success.
The fragmentation in OTT delivery monitoring and measurement can have OTT customers -- such as content distributors or owners -- relying on just a few QoE factors rather than getting an accurate picture of video delivery quality.
For example, Ryan Korte, principal architect of content and media for Level 3 Communications, said that "there are five or six KPIs," or key performance indicators, that Level 3 uses to measure the success of a video stream -- such as video start time, buffer rates, and other factors. However, customers tend to not take a holistic view. "Every customer thinks one is more important than the other," he said.
The companies each panelist belonged to each has its own set of measurement and monitoring tools to maintain QoE for the end user and make sure that streaming video is being received, is viewable and is not buffering. For example, IneoQuest offers provider-agnostic monitoring along the entire path of a stream to pinpoint QoE problems. On the other hand, Cedexis, a data traffic management provider, monitors only in the last-mile network. "A ton of problems are related to the last mile," said Pete Mastin, market strategy and product evangelist at Cedexis. "We definitely recommend against synthetic monitoring [of the last mile] from a data server."
Some OTT distributors and providers feel they're not able to accurately measure QoE for their customers, either. Sung Ho Choi, co-founder and chief product officer of fuboTV, said the OTT service utilizes the company's customer support wing to identify streaming problems. "We take a lot of end customer tickets thru our support channels … so we can look back into an analytics tool to figure out what happened to the user," Choi said. "Other times we still do not have a good picture of what went wrong. We're still working on that."
fuboTV also uses other monitoring tools to look for problems with its video stream. "We look for regional outages on the CDNs, on the ISPs. There are tools that can measure these kind of outages," Choi said. "But we want to figure out why this went wrong and what to do for next time to prevent it from happening again."
For smaller providers, using monitoring tools that might roughly be called "off the shelf" could provide additional low-cost or free insight. Panelists talked about the effectiveness of monitoring social media, particularly Twitter, for user comments and complaints about a streaming outage. Level 3's Korte even suggested that DownDetector -- the same website that end users access to report service outages -- was "the kind of thing we need to bring into the heuristics data. … That's the kind of stuff that's outside the CDN space," but offers additional user-based data that could help providers solve a QoE problem.
Regardless of the solutions offered, the different methods used to ensure QoE among just the summit panelists pointed to an issue that may need significantly more attention and action. fuboTV's Choi summed it up for providers both large and small: "Finding out exactly what's wrong has always been a big challenge."
Special Report: The quality conundrum: Data drives OTT improvement, but lack of standards is a problem
Related articles:
Transforming OTT from gee-whiz tech to mainstream entertainment source requires continued quality focus
Improved OTT user experience, increased devices boost video worldwide, reports find
Streaming Video Alliance gets busy, publishing open caching principles
Akamai takes proactive approach with new OTT monitoring center headed by NBCU veteran Azzarto
IneoQuest looks to catch early NFV wave with new video QoS solutions