What is the benefit of automated testing?
New code will have a better quality
Original goal: CHF 5000
1 CHF ≈ 0.83€ ≈ 1.05 US$
November 15, 2014
There are two possibilities to fund this project.
- The easy way is to use paypal to raise the progress bar.
- If it is not possible for your organization to participate this way, I can also write an offer and an invoice. Please contact me directly by mail matthias [dot] kuhn [at] gmx [dot] ch.
All funds will be returned if the minimum required amount of CHF 5000 cannot be raised by November, 15.
Every piece of source-code - and therefore every change made to QGIS - will be tested automatically before it is being included in nightly builds. That means that some issues can be detected automatically. These issues therefore don't frustrate testers which in turn can then focus on more important things than "sigh, let's create another issue report for the same thing again"!
This has been done before, but it has been well-hidden and therefore nobody noticed when something failed. With this initiative test results will be shown on the main development platform (github) and sent to the responsible developer via e-mail.
Read on for details...
Unit testing is known in the software world as a measure to guarantee code quality. This is done by writing tests, that run pieces of the software and check that it behaves in an expected way.
QGIS and unit testing
QGIS currently has a set of over 100 tests that have been written when new features have been implemented or bugs have been fixed to assure that the same bug is not going to happen again. Every developer is able to run this test suite on his computer whenever he changes something and check if the tests run ok. There used to be regular test runs on a central server run by linfinity that produced test statics and there are still tests being run for the nightly builds.
What's the problem?
In reality only few developers runs the tests ever, and even the ones that do it do not run the tests not after every change they make. The central server that produced statistics did this quite reliably, however, hardly ever somebody went there to check. Currently about 30% (update, 11.10.2014) between 10% and 50% (depending on the platform) of the tests do not pass. And nobody seems to care much.
How do we fix it
Instead of asking developers to manually trigger test runs by themselves or go to a specific page where statistics could be found information needs to be put where developers look. Every QGIS developer works with github. Every new feature and every bug that is fixed enters QGIS via github. So test results will be put directly there.
In a way that does not leave much room for discussion:
- Green: good
- Red: bad.
Travis CI is a continuous integration platform. This platform is able to work in conjunction with github. After every change in the source-code Travis CI will run all tests and put a red or a green label next to the change. It's then pretty easy to tell which change by which developer is responsible. And an automated e-mail will have been sent to the responsible developer(s) anyway.
Even better, there is a test-bed for changes called pull requests in github. A new change can be put there and be tested before it is integrated into the main software. That in turn means, that it does not even have to enter the nightly builds before it can be flagged as (potentially) bad. And it is then also pretty easy to ask a developer to take care of this before his commits are being built into main QGIS.
What needs to be done?
And therefore, what are you funding...
Travis CI Integration
Instructions for Travis CI about how to handle QGIS source code need to be implemented. This is not very complicated and can be done in a couple of hours.
As currently a not-so-small portion of the tests is failing there is not much reason to enable this system right now. If QGIS master (the state of source code that goes into the nightly builds that become the next version 2.6) is starting with build status red, we cannot tell when somebody broke something. Therefore it is important to start this system with a green build status. That involves either fixing the source-code when it is broken, or the test, when the source-code is fine, but the test is broken.
With the funding goal being hit, in addition to the Travis CI integration I will invest at least 3 days into fixing tests. Whatever test has not been fixed after this period will be disabled until somebody fixes it (better not having all of the tests run, than not realizing when somebody breaks one of the working tests).
And if there are too much funds available?
If there are more funds available than the minimum requirement, I will happily continue to fix tests as long as there are tests that can be fixed. And as soon as I run out of tests I will start to write new tests that cover parts of QGIS that have not yet been covered.
Still hesitating? Just write me an email with any questions you have
matthias [dot] kuhn [at] gmx [dot] ch