Workshop on Monitoring Quality of Service and Quality of Experience of Multimedia Services in Broadband/Internet Networks (Maputo, Mozambique, 14-16 April 2014) Perceptual Quality Evaluation Of OTT Streaming Video Services Joachim Pomy [email protected] OPTICOM, Germany Perceptual Quality Evaluation Of OTT Streaming Video Services Dipl.-Ing. Michael Keyhl, MBA Founder & CEO, OPTICOM, Germany Session > OTT Technologies and Services < Broadcast Engineering Conference | NAB Show 2014, Monday April 7 Who We Are
Founded 1995 in Erlangen, Germany Spin-Off from the Fraunhofer-Institute IIS (Home of MP3) Originators of Perceptual Audio Quality Measurement (1988) Co-Authors of Six Major International Standards Leading Source of Voice and Video Quality Test Products and OEM Technology for Mobile and IP-based Network Testing > 100 Licensed OEM Customers > 40.000 Licensed PESQ Products NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 3 How Do You Measure Video How Do You Measure Video Quality? Quality.. ...to benchmark Multi-Screen QoE of ABR VoD OTT
Services as perceived by subscribers? NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 4 How Do You Measure Video Quality.. ...to benchmark Multi-Screen QoE of ABR VoD OTT Services as perceived by subscribers? Conclusion 1: Clear Need for a New Streaming Video Quality Measurement Technique Conclusion 2: Newly Interpret Video Quality in the Context of Multi-Screen Use Scenarios NAB Show 2014, Broadcast Engineering Conference, 07 April 2014
2014 Michael Keyhl, www.opticom.de 5 Overview 1. 2. 3. 4. 5. 6. ABR and Existing Video Metrics and Standards Stakeholders in the OTT QoE Rivalry Defining an >OTT Quality Model< Designing a Video Quality Measure for OTT Introduction to PEVQ-S Conclusions NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de
6 Overview 1. 2. 3. 4. 5. 6. ABR and Existing Video Metrics and Standards Stakeholders in the OTT QoE Rivalry Defining an >OTT Quality Model< Designing a Video Quality Measure for OTT Introduction to PEVQ-S Conclusions NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 7
Testing Typologies Full Reference (FR) No-Reference (NR) Principle: Differential Analysis comparing Degraded with the Original Source Video Access to Reference/Studio Source Pixel-based, Frame-by-Frame Comparison e.g. OPTICOMs PEVQ/ITU-T J.247 + Highest Accuracy and Repeatability + Standardized Processing Intensive Principle: Single-ended Analysis, NO Information on the Original Source Video Pixel-based, or NR/IP Transport Stream Analysis, only,
no Payload decoding Various Proprietary Metrics Low Accuracy Estimates only Failed in Standardization + Lightweight, easy to apply NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 8 Video Quality Standard s develope d by VQEG and ITU based on
NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 9 ABR Video Streaming Techniques (based on Jan Ozer: Video Compression for Flash, Apple Devices and HTML5 [2011]) NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 10 Example: YouTube Upload Upload: V_Res 1920x1080, V_Fps 24, V_BitRate 8Mbit/s
Protocol PD PD PD PD PD PD PD ABR ABR ABR ABR ABR ABR ABR V_Codec MPEG4 MPEG4 FLV VP8 H.264
H.264 H.264 H.264 H.264 H.264 H.264 H.264 V_BitRate 60kbit/s 200kbit/s 300kbit/s 600kbit/s 500kbit/s 2Mbit/s 100kbit/s 250kbit/s 400kbit/s 800kbit/s 1.6Mbit/s 3Mbit/s
V_Res 176x144 320x180 426x240 640x360 640x360 1280x720 256x144 426x240 640x360 854x480 1280x720 1920x1080 V_Fps 12 24 24 24 24 24 15
24 24 24 24 24 A_Codec AAC AAC MP3 OGG OGG AAC AAC AAC - A_Bitrate 24kbit/s 32kbit/s 64kbit/s 96kbit/s
110kbit/s 96kbit/s 192kbit/s 128kbit/s - NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 Reference/Source PD or ABR Protocols Format 3GP 3GP FLV WEBM WEBM MP4 MP4 MP4 MP4 MP4
MP4 MP4 MP4 MP4 PEVQ-S Various Codecs at Various Bitrates Compare Various Frame Sizes Compare Different Fps
Handle Common Bitstream Formats 2014 Michael Keyhl, www.opticom.de 11 Only PEVQ was validated for QCIF, CIF, VGA, SD, and HD based on BT.500/P.910 NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 PEVQ PEVQ PEVQ
2014 Michael Keyhl, www.opticom.de 12 Validation by Subjective Testing ITU-R BT.500: Originally CRT TV-Quality Testing (SD) ITU-T P.910: Multimedia (QCIF, CIF, VGA) and IPTV (HD 720/1080) Mostly ACR Tests Test Sequences 8 15 sec 5-Point Opinion Scale Averaging over a Large Enough Test Population MOS Conclusion 4: Need for New Subjective Testing Snapshots of 10 s Video are Inadequate to Assess Re-Buffering and Long-term Behavior of Streaming Video
NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 13 Overview 1. 2. 3. 4. 5. 6. Existing Video Metrics and Standards Stakeholders in the OTT QoE Rivalry Defining an >OTT Quality Model< Designing a Video Quality Measure for OTT Introduction to PEVQ-S Conclusions NAB Show 2014, Broadcast Engineering Conference, 07 April 2014
2014 Michael Keyhl, www.opticom.de 14 Stakeholders involved in a QoE Rivalry NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 15 Overview 1. 2. 3. 4. 5. 6. Existing Video Metrics and Standards Stakeholders in the OTT QoE Rivalry
Defining an >OTT Quality Model< Designing a Video Quality Measure for OTT Introduction to PEVQ-S Conclusions NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 16 Defining a 4layered OTT Quality Model NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 17
Stakeholders Different Quality Views Stakeholders for OTT QoE and related Quality Layers NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 18 Overview 1. 2. 3. 4. 5. 6. Existing Video Metrics and Standards Stakeholders in the OTT QoE Rivalry
Defining an >OTT Quality Model< Designing a Video Quality Measure for OTT Introduction to PEVQ-S Conclusions NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 19 OTT VQM Requirements VQM Should be related to the Content Quality as a Reference Accurately score encoding and transcoding Artifacts = Media Stream Quality Measure and compare the Picture Quality for different Frame Sizes and Frame Rates = Media Stream/Transmission Quality Continuously track the different Bit Rates and evaluate how smooth the Player is able to interact with the Server in a Congested Network = Transmission Quality Take into account Player and Device Characteristics as well
as Viewing Environment = Presentation Quality 20 NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de Overview 1. 2. 3. 4. 5. 6. Existing Video Metrics and Standards Stakeholders in the OTT QoE Rivalry Defining an >OTT Quality Model< Designing a Video Quality Measure for OTT Introduction to PEVQ-S Conclusions NAB Show 2014, Broadcast Engineering Conference, 07 April 2014
2014 Michael Keyhl, www.opticom.de 21 NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 22 Overview 1. 2. 3. 4. 5. 6. Existing Video Metrics and Standards Stakeholders in the OTT QoE Rivalry Defining an >OTT Quality Model< Designing a Video Quality Measure for OTT
Introduction to PEVQ-S Conclusions NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 23 Conclusions (1/2) Clear Need for a Streaming Video Quality Measurement Technique which accurately evaluates Subscriber QoE We must reformulate QoE Concepts and newly interpret Video Quality in the Context of Multi-Screen Use Scenarios Currently, no Standardized Subjective and Objective VQM Standard exists for ABR Video Streaming PEVQ-S is proposed to leapfrog this Shortcoming, based on advancing existing Standards, while maintaining maximum Backward-Compatibility and Validated Accuracy PEVQ-S is suited to evaluate all 4 OTT Quality Layers:
Content, Media Stream, Transmission and Presentation Quality NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 24 Conclusions (2/2) PEVQ-S allows for Analysis of Common ABR Protocols and Formats, Various Codecs at Various Bitrates PEVQ-S can analyze Video at Different Frame Sizes and Fps An Advanced Hybrid Architecture: (a) the PEVQ-S Media Stream Quality Database and (b) the PEVQ-S Client Probe combines the high Accuracy of a FR Measure with a Light-weight Realtime Client Implementation, even on Android Smartphones Measures Current Video Quality (CVQ) MOS and Headroom for Optimization for the Best Video Quality Available (BVQA) MOS PEVQ-S is made available as an OEM-Technology to leading OTT, Middleware and T&M Vendors and soon to be found in Products
NAB Show 2014, Broadcast Engineering Conference, 07 April 2014 2014 Michael Keyhl, www.opticom.de 25 Any questions ? Joachim Pomy [email protected]
