PROWESS Project Workshop
10 June 2015 – Stockholm
The PROWESS end project workshop will be co-located with the Erlang User Conference 2015 in Stockholm, Sweden. The workshop will take place on 10th June and additional PROWESS talks will be presented at the conference on 11th and 12th June. Attendees can register to the workshop tutorials through the tutorials registration form
The tutorials will be held at Ericsson in Kista. The address to this location is:Ericsson Torshamnsgatan 21 Stockholm, Sweden
- 09:00-10:30 : Property-Based Testing for Web Services — Simon Thompson, Thomas Arts
This tutorial will provide an introduction to using Property-Based Testing to test web services, based on the results of the EU-funded PROWESS project.
PROWESS has developed a range of tools and techniques to support testing web services, centred on QuickCheck. Taking the example of the VoDKATV system from Interoud Innovation, the tutorial will give an overview of how each of the PROWESS tools fits into the overall process of testing web services systems. After introducing how QuickCheck is used in the example, we give more details about some of the tools: – JSONgen and WSDL_dsl are generator libraries for JSON and WSDL; – WStoolkit automatically generates QuickCheck boiler-plate code from WSDL system descriptions; – James, your butler for creating additional unit tests in your Java Unit test suite; – scaling QuickCheck by clustering component libraries and mocking in QuickCheck suitable way; – the quality of test code can be assessed using the Smother coverage analyser and the Mu2 mutation testing tool; – MegaLoad for load testing your applications in the cloud. Other PROWESS tools are described in more detail in the tutorial “Showing QuickCheck results to stake-holders which follows this one, and in presentations in the main conference itself.
- 10:45-12:15 : Showing QuickCheck results to stake-holders — Laura M. Castro, Thomas Arts and Nick Smallbone
For some people, QuickCheck properties are not as easy to read or understand as unit tests. Presenting statistics on test results is also different, since you do not run the same tests each time. We address this challenge by a number of techniques. We designed QuickCheck-CI to experiment with continuous integration in QuickCheck projects. In QuickCheck-CI we store failing test cases automatically and are able to run them for each consecutive build. We also put effort in presenting more than one failing test case for a property if we can found such, using MoreBugs. After all, one does not want to see small variations of the same bug, but rather completely different bugs. We developed a technique to present a set of unit tests generated by a QuickCheck property that capture the typical kind of things you have tested. You can show this set of unit tests to people that want a glimpse of what kind of things your QuickCheck property is testing. And, if you want to show people that have a hard time reading Erlang code what you have been tested, there is ReadSpec that translates your QuickCheck properties and generate tests in readable English.
- 13:15-13:45 : WSDL-DSL + WSToolKit: full automatization of Web Services testing — Laura M. Castro
Web services have proliferated enormously in the last decades; they are likely one of the most common integration mechanism with third-party systems. You receive a PDF description of such web service, and off you go to achieve a seamless integration… or to design, write, and run, the relevant test cases for such system. The lucky ones get a machine-readable description, such a WSDL. They’re lucky because, from a WSDL description of a web service, WSToolKit is your push-button testing technology of choice, as we will demonstrate in this tutorial: how to get a fully-working QuickCheck model for your web service instantly! Now the not so lucky will have a powerful reason to demand a WSDL description of third party web-based services!
- 13:45-15:15 : Scalable Load Testing Using Properties — Diana Corbacho
Software components need to fulfil a number of non-functional requirements on top of the natural functional requirements. In particular for scalable systems, it is important that performance requirements can be met when popularity of the service dramatically increases. Thus, the possibility to experiment at a very early stage to predict how a service behaves with respect to certain non-functional requirements is highly desirable.
In this tutorial we show how to apply property-based testing for validation of non-functional requirements in a web service. This tutorial uses QuickCheck properties to create many different load and user profiles that exercise a wide variety of aspects of the system. Megaload* performs and automates the execution allowing to test large deployed systems.
*Megaload is a scalable and customizable load testing tool written by Erlang Solutions
- 15:15-15:45 : Jsongen – Testing JSON web services using QuickCheck — Lars-Åke Fredlund
This tutorial introduces the jsongen library which, with the help of QuickCheck, can automate part of the testing of a web service which communicates using JSON. The approach uses a JSON schema description of the web service to drive testing.
The tutorial aims to introduce the:
(i) the jsongen library, and
(ii) provide a brief overview of the JSON schema language.
Web programmers, test personnel.
- 11th June 15:50-16:35 : PROWESS: Property-Based Testing for Web Services — John Derrick
This presentation will provide an overview of the PROWESS project, that has spent the last three years applying Property-Based Testing to web services, funded by the EU. Using a case study of the VoDKATV system from Interoud Innovation, wel will give an overview of how each of the PROWESS tools fits into the overall process of testing web services, as well as showing how QuickCheck is used in such a case.
To give a high-level overview of how to apply property-based testing in the field of web services, as well as to demonstrate how the PROWESS tools support this process.
Those interested in web services, testing, and property-based testing.
- 11th June 16:25-17:10 : Graphical Models For QuickCheck — Kirill Bogdanov
Quickcheck is a tool that generates test data to fit a model provided by a tester. Such models describe both the kind of data to generate and the constraints it has to satisfy. Moreover, one would usually take stateful nature of systems under test into account by using a test model, containing both abstract state and operations that determine which API calls can be made to a system under test. When building such test models, it is necessary to keep track of the variables describing state transitions of the test model and update them depending on the intended outcome of the API calls. The talk introduces a new tool that makes it possible to define a test model graphically. Using a simple web-based interface, one can construct and edit a state machine that shows which API calls on a system under test are possible and when. The tool automatically updates the corresponding Erlang code, permitting one to immediately run tests and observe outcomes, such as a failed test or a distribution of API calls obtained from a test suite.
Introduces the upcoming extension to Quickcheck tool for graphical editing of models.
Everyone using Quickcheck.
- 12th June 14:55-15:40 : Can I Stop Testing Now? – Test Adequacy Metrics Beyond ‘Cover’ — Ramsay Taylor
There are various frameworks for writing tests (eunit, ct, etc.) and even for automatically generating tests (e.g. QuickCheck) but how do you know when you have done “enough” testing? Test adequacy metrics provide some measure of how extensively your test set tests your system. The line coverage tool ‘cover’ that is included in OTP is a very simple metric – a test set that doesn’t even execute some parts of the code is definitely not testing the system thoroughly. However, simply executing every line still doesn’t ensure that the system is “well tested”. This talk will cover two more advanced metrics: MC/DC analysis, provided by the Smother tool, and Mutation Testing, supported by the mu2 tool.
Give an introduction to the intention and application of MC/DC analysis and mutation analysis of test sets.
Anyone developing tests or responsible for testing/acceptance of Erlang systems.