- 1 : Introduction – The Erlang Ecosystem
- 2 : Relating Development Artifacts to PROWESS tools and libraries
- 3 : The leading example: VoDKATV
- 4 : Property-Based Testing for VoDKATV
- 5 : Supporting evolution in property-based testing
- 6 : Load Testing VoDKA
- 7 : Inference and property-based testing
- 8 : Understanding properties and models
- 9 : Building on web service interface descriptions
- 10 : Scaling property-based testing
- 11 : Validating quality of test suites
- 12 : Testing non-functional requirements
- 13 : Continuous integration
- 14 : Comparing different implementations
- 15 : Task – Tool relationships
To be of value to the European software industry, the work of the PROWESS project needs not only to be grounded in state of the art research, but also to be tensioned by industrial practice. The resulting work has produced not only new tools and libraries but also extensions to existing tools that are already part of the “Erlang ecosystem”, which provides a context for our work (Section 1). After doing this we describe how the different PROWESS tools complement each other in fitting different parts of the web services and systems workflow (Section 2).
To emphasise how the tools are used in practice we use a key pilot study, VoDKATV, (Section 3) that shows how the different aspects of that system are supported by different PROWESS tools. To frame the rest of the document, Section 4 gives a high-level overview of how property-based testing with the PROWESS tools can be used to test a part of the VoDKATV API. This is based on QuickCheck, of course, but also uses WStoolkit to automate generation of QuickCheck models and data generators from WSDL descriptions; the support for system evolution in WStoolkit is described in Section 5, and this is integrated with the Wrangler refactoring tool for Erlang.
Addressing other aspects of testing, Section 6 describes how Megaload is used to load test VoDKATV, and Section 7 presents two tools that use inference to aid property-based testing and testing in general. Working from JUnit tests of a web service, James will infer models and from them further unit tests – shown here in action on VoDKA – while Synapse interfaces the StateChum model-inference and differencing framework to Erlang. To scale property-based testing, it is necessary to structure models, and this is done using components; complementing this is a mocking framework, so that certain aspects of a system can be “mocked” in a PBT-appropriate way (Section 10).
Non-functional requirements of systems include fault tolerance, concurrency, complexity and user-interface testing, each of which is addressed in Section 12: FaultCheck, PULSE, complexity checking and an Erlang implementation of WebDriver.
Property-based testing can prove to be a challenge to some. Automation, that builds on web service interface descriptions, is implemented in JSONgen and wsdl_dsl to make test data generation more straightforward (Section 9), as well as WStoolkit (see above). To facilitate better understanding of PBT, PROWESS provides tools to render tests into semi-natural language (ReadSpec) and to select representative examples (GoodExamples) as described in Section 8. Finally, Section 10 introduces graphical editing support for QuickCheck models, allowing them to be modified directly without accessing the underlying Erlang code.
PBT can be assessed by the quality of tests to which it gives rise. We provide in Section 11 a tool to assess MC/DC coverage of tests in Erlang (Smother) as well as the Mu2 system supporting mutation testing for Erlang. Outside the VoDKATV pilot (there’s only one VoDKATV) we report on the Ranker tool (Section 14) which ranks competing solutions by means of property-based testing.
To conclude, we present QuickCheck CI a web-based continuous integration server that runs QuviQ QuickCheck on a project, and which is available free to Open Source Developers to get free access to QuviQ QuickCheck for their quality assurance.