There should be an effort to encourage people to adopt a single single
test output format.
|
There should be an effort to encourage people to adopt a single single
test output format.
|
Candidate formats are:
* json - using a schema similar to what kernelci use
* TAP - test anything protocol
* *unit - for the xunit, junit, pyunit family of test frameworks
* are these are consistent in their output?
|
Candidate formats are:
* json - using a schema similar to what kernelci use
* TAP - test anything protocol
* *unit - for the xunit, junit, pyunit family of test frameworks
* are these are consistent in their output?
|
Here are projects that promote or use a single test output format:
* https://packages.debian.org/sid/autopkgtest
* https://wiki.yoctoproject.org/wiki/Ptest
* https://testanything.org/ (Test Anything Protocol)
* http://pyunit.sourceforge.net/
|
Here are projects that promote or use a single test output format:
* https://packages.debian.org/sid/autopkgtest
* https://wiki.yoctoproject.org/wiki/Ptest
* https://testanything.org/ (Test Anything Protocol)
* http://pyunit.sourceforge.net/
|
Notes on these:
= ptest =
Yocto/OpenEmbedded's ptest framework decouples the compiling phase
from the testing phase and produces "a consistent output format".
|
Notes on these:
= ptest =
Yocto/OpenEmbedded's ptest framework decouples the compiling phase
from the testing phase and produces "a consistent output format".
|
Ptest normally uses the test suite that comes with the original source code.
For example, for the openssh recipe it uses some of the tests inside openssh's
"regress" folder.
|
Ptest normally uses the test suite that comes with the original source code.
For example, for the openssh recipe it uses some of the tests inside openssh's
"regress" folder.
|
However, there are many recipes without their corresponding ptest definitions. I'm not sure if that is just because nobody added them yet, or because
there was no test suite in the original source code.
|
However, there are many recipes without their corresponding ptest definitions. I'm not sure if that is just because nobody added them yet, or because
there was no test suite in the original source code.
|
|
= TAP =
Test Anything Protocol
|
https://testanything.org/tap-version-13-specification.html
|
https://testanything.org/tap-version-13-specification.html
|
Example:
{{{
TAP version 13
1..6
#
# Create a new Board and Tile, then place
# the Tile onto the board.
#
ok 1 - The object isa Board
ok 2 - Board size is zero
ok 3 - The object isa Tile
ok 4 - Get possible places to put the Tile
not ok 5 - Placing the tile produces no error
---
message: Could not place tile at row 1, column 1
severity: fail
...
ok 6 - Board size is 1
}}}
|
Example:
{{{
TAP version 13
1..6
#
# Create a new Board and Tile, then place
# the Tile onto the board.
#
ok 1 - The object isa Board
ok 2 - Board size is zero
ok 3 - The object isa Tile
ok 4 - Get possible places to put the Tile
not ok 5 - Placing the tile produces no error
---
message: Could not place tile at row 1, column 1
severity: fail
...
ok 6 - Board size is 1
}}}
|