Audio/Video Quality
Measuring Audio Quality as per Internationally Standardised Test Methods
Modern audio-coding techniques featuring data reduction typically function with variable bit rates. A reduced data rate is by definition associated with lower quality. Unfortunately, sufficiently reliable results cannot be provided by objective methods for determining the audio quality of special audio-coding techniques or cascade chains (repeated coding/decoding and use of different audio-coding techniques within a transmission chain). For this reason, experts must rely on psychoacoustic measurements, which are based on subjective quality evaluations generated by experimental means.
Two internationally standardised test methods for measuring the quality of audio-coding techniques and cascade chains have been used successfully for several years in the field of broadcasting:
ITU-R BS.1116 Test Method
In the context of the ITU-R BS.1116 test method, a listener hears three stimuli: A, B, and C. “A” is always the reference stimulus. The test signal which is to be evaluated and the hidden reference are presented in a random sequence as the stimuli B and C. The listener must first recognise the reference as well as the test signal which requires assessing, and then compare the test signal with A on a scale from 5 (imperceptible) to 1 (very annoying).
MUSHRA Method
The MUSHRA method (MUlti-Stimulus test with Hidden Reference and Anchor, ITU-R BS.1534) makes it possible to select any test signal requiring evaluation from as many as twelve synchronous test signals. The chosen test signal is then reproduced via loudspeakers or headphones for assessment.
In addition to the test sequences themselves, MUSHRA features three further stimuli. These include a labelled reference (unmodified original signal which does not require evaluation) as well as a hidden reference and a hidden anchor (7 kHz low-pass filtered original), which are to be assessed as the other test sequences. Scores are assigned on the basis of a continuous scale of quality from 0 to 100 %, or in the form of attributes from “bad” to “excellent”. The “overall quality of audio” is evaluated in consideration of all perceptible forms of interference.
Studies on the Picture Quality of Video-over-IP Solutions
Conventional techniques for studying picture quality no longer lend themselves to digital signals, which are sometimes plagued by entirely different kinds of errors. Despite efforts to develop objective measuring methods, the results repeatedly fail to reflect quality as perceived by the human eye, which is uninterested in purely logical approaches. Subjective tests are therefore recommended for checking the quality of digital videos.
IRT plays an instrumental role in developing these subjective test methods and conducts its own tests of current video codecs.
SAMVIQ
SAMVIQ (Subjective Assessment Methodology for VIdeo Quality) was developed primarily for evaluating the quality of online video formats, yet can also be used for conventional TV formats.
Existing subjective test methods have been standardised in Recommendation ITU-R BT.500. These methods, however, were developed first and foremost for the TV sector. While these conventional methods are good tools for evaluating errors such as blurring and blocking, continuous picture streams are of the utmost importance for multimedia applications. The picture size in this context usually corresponds to a fourth (SIF) or 1/16 (QCIF) of standard resolution.
SAMVIQ, on the other hand, was custom-designed for multimedia purposes and permits the use of manufacturer-specific decoders (e.g. MPEG-4 Player, RealVideo, QuickTime, Windows Media) as well as various resolutions, including SDTV. Moreover, the optimised test procedure allows for more detailed findings and absolute quality assessments.
MOVIS
A Performance Monitoring System for Video Streaming Networks
The MOVIS project implements a system to perform online measurements of end user service quality for streamed content. A prototype will be developed which reports the service quality, and quality estimates from the end user to the content providers for the purpose of providing the right QoS level to the end user. Since the distribution of content will be done through several Internet Service Providers (ISP) beyond the content provider's control, and with different transport technologies, the MOVIS prototype will be designed independent of implementation issues at the ISPs transport mechanisms. The ISPs are participating in the project for defining, implementation, and testing of technical and perceived class of service metrics.
Currently, the Windows Media format is one of the most popular systems for video streaming and thus a strong candidate for being used in the Norwegian MOVIS project. However, a clear picture about the quality behaviour in a virtual error free as well as in an erroneous streaming condition is required for the project.
Subjective assessment of video quality depending on encoding parameters
The video quality in an transmission error-free condition depends very much on bit rate, temporal and spatial resolution as well as on the video content itself. The IRT measures the influence of those parameters on the perceived quality and defines the appropriate settings of the encoder for a desired subjective quality, dependent on the technical parameters of the video signal. This avoids coding artefacts already from beginning and helps to minimize the impact of network errors on the final quality. Testing are done with a new and very efficient procedure, which has been developed by IRT together with France Telecom and RAI and has been standardized by EBU as “SAMVIQ” and presently is considered to be standardised by ITU-R.
Subjective assessment of video quality depending on network parameters
During video transmissions over unreliable IP networks, various network errors can occur. Packet loss, delay, jitter and residual wrong packet order can reduce the perceived quality dramatically. IRT is testing the impact of each specific error as well as the impact of a combination of different network errors regarding the perceived quality. This work is done using special network simulators and the highly reliable subjective testing methods above-mentioned.
