Infrastructure










Overall Infrastructure
The project partners started with an infrastructure which has been continuously improved and optimised in order to allow handling, joint analysis, exchange of debuggung information and managment of code for several thousands of conformance tests.

The number of existing an new conformance tests to be dealt with by MHP-CONFIDENCE is in the order of 6000 and more. Validation and debugging of the tests needs close communication of all the partners involved. As a consequence, a dedicated infrastructure is required

  • to distribute any updates or new tests amongst partners,
  • to facilitate and harmonise the running of the tests at various labs of the partners,
  • to monitor the status of the tests
  • to provide a means for communication amongst partners, but also with sub-contractors
  • to track issues and discussions concerning individual tests

No such infrastructure existed at the beginning of the project. However, from their prvious work on test suites and also from certifying their existing products with the MHP 1.0.2.b Test Suite, most partners had an Automated Test Environment system (ATE) which allows to automatically apply and run a series tests on their MHP implementations, if these tests are provided in the technical format as delivered from ETSI to test suite licensees. This technical format is adequate for distribution of complete and finalised test packages, but less adequate for stepwise improvement and debugging of tests in a joint environment.

An advanced infrastructure was gradually developed and optimised during the course of the project and is now available to partners for continuous and simplified test development. This infrastructure consists of

  • the Automated Test Environment (ATE)
  • a Code Versioning System (CVS) where the set of tests are stored and jointly developed
  • a Bug Tracking Database (Bugzilla) where any interoperability issues or implementation issues with tests are recorded, and
  • supporting tools which extract information in order to prioritise and monitor ongoing work. These tools had to consider external information as well, such as information from DVB on released tests in error (so-called appeals), from specification changes, etc. The supporting tools consisted of a series of scripts which e.g. created spreadsheets and other information. This was migrated to a more advanced database during the second and third project year.

The CVS, the bugzilla system and the advanced database is installed at IRT and is available to and managed by all partners via the internet through secure SSH channels. This activity was driven in WP 2 which was in charge of prioritising the work. The following paragraphs describe that infrastructure and its development.

The figure below gives an overview of these elements as used in the workflow process applied by MHP-CONFIDENCE: Starting from the bottom of the diagram, tests to be validated were reviewed by each partner, applied to their implementation, and debugged by a partner in case that they do no pass. Once initial problems had been solved, the results from running the tests at various partners locations (and implementations) were collected in a so-called “result collation”, and further debugging work for non-working tests was jointly assigned to one partner that did not pass the tests. This partner was then in charge of a deeper analysis of the problem and communicated technical discussions through the bugzilla system. After solving the issue (or also in between for further result collations), the modified test code was committed as a new delivery to the CVS repository and was available to all other partner for further verification. A similar approach with restricted access rights was applied to reviewing the work of sub-contractors. Initially and without the advanced database, relevant status information was derived from all other elements through scripts and mainly managed by spreadsheets. However, an advanced database (see below) was created that also provides features to simplify the setup and configuration of the ATE for assembling small and frequently changing of test packages.

                     

It turned out during the course of the project that a more advanced database would simplify the debugging work and therefore such a database for long-term maintenance of conformance tests was specified, designed and developed. In principle, this database replaces the co-ordination applied via spreadsheets by more automated processes for many more purposes that statistics. The database itself allows to create summaries. 

A main reason for implementing this database was the ATE system shows disadvantages when individual tests need modifications: Several details for executing single tests are maintained in larger XML files that cover complete test bundles and thus may need simultaneous editing from various partners. That procedure provoked mistakes that need to be found by later manual validation of files.

 Furthermore, browsing of assertions and the creation of statistics was not easy to handle if that data is only available via single xmls. For creation of statistics scripts were needed that search the xmls and such scripts were not commonly used by all partners at the start of the project.

Therefore, a common database was designed that is accessible to all partners simultaneously and that forms the information base for configuration files applicable to more than one test. It also simplifies browsing, packaging, committing, generating of tests and creation of status informatio in spreadsheets.

Automated Test Environment
Conformance testing of MHP receivers requires a controlled environment, which implements the MHP chain: streaming of A/V, program information, test applications and data, as well as the user inputs. MHP receivers are tested in final consumer configuration except that they are placed in specific test mode whereby a test results can be fetched from the receiver via a test interface (API). The test interface is part of the MHP specification.

       

For controlled and repeatable test execution an Automated Test Environment (ATE) has been defined. The ATE allows:

  • Fully automatic test execution
  • Selection and sequencing of tests
  • Test result logging and unambiguous determination of PASS/FAIL result
  • Capture of test/receiver output for later scrutiny.

For test coding, it offers features that can be used to ensure the correct execution path of test code leading to a PASS result (path trace) and to implement structured outputs that log the steps of test execution, and generally offers a framework for coding tests that are consistent and reliably set the tests result. For more complex tests, interaction facilities are provided for test implementations that are composed of parts that run on the test server and part that runs on the receiver under test.