criteria.json
Introduction [edit section]
The criteria.json file is used to specify the criteria used to determine whether a test has passed or failed.For the purpose of this explanation, I'll group tests into roughly 3 groups:
- simple Functional tests
- complex Functional tests
- Benchmarks
I'll come back to these definitions in a moment.
The criteria.json file contains data that allows Fuego to interpret the test results, and indicate overall PASS or FAIL status for a test.
For functional tests, this includes things like counting the number of test_cases in the test that had "PASS" or "FAIL" status, as well as ignoring some specific test results.
For benchmark tests, this includes specifying threshold values for measurements taken by the benchmark, as well as operations (e.g. less than greater than), to use to determine if a specific measure passed or failed.
Fuego uses the results of the test along with the criteria, to determine the final result of the test.
If no criteria.json file is provided, then a default is constructed based on the test results, consisting of the following:
{
'tguid': <test_set_name>
'max_fail': 0
}
types of tests and pass criteria [edit section]
A simple functional test runs a short sequence of tests, and if any one of them fails, then the test is reported as a failure. Since this corresponds to the default criteria.json, then most simple Functional tests do not need to provide a criteria.json file.A complex Functional test (such as LTP or glib) has hundreds or possibly thousands of individual test cases. Such tests often have some number of individual test cases that fail, but which may be safely ignored (either temporarily or permamently). For example, some test cases may fail sporadically due to problems with the test infrastructure or environment. Other tests may fail due to configuration choices for the software on the board. (For example, a choice of kernel config may cause some tests to fail - but this is expected and these fail results should be ignored).
Complex Functional tests requires a criteria.json file, to avoid failing the entire test because of individual test_cases that should be ignored.
Finally, a Benchmark test is one that produces one or more "measurements", which are test results with numeric values. In order to determine whether a result indicates a PASS or a FAIL result, Fuego needs to compare the numeric result with some threshold value. The criteria.json file holds the threshold value and operator used for making this comparison.
Different boards, or boards with different software installations or configurations, may require different pass criteria for the same tests. Therefore, the pass criteria are broken out into a separate file that can be adjusted at each test site, and for each board. Ultimately, we would like testers to be able to share their pass criteria, so that each Fuego user does not have to determine these on their own.
compatibility with previous Fuego versions [edit section]
Previously, Fuego (and it's predecessor JTA) supported pass criteria functionality in two different ways.
Functional test pass/fail counts [edit section]
For functional tests counts of positive and negative results were either hard-coded into the base scripts for the test, as arguments to the log_compare() in each test's test_processing() function, or they were specified as variables, read from the board file, and applied in the test_processing() function.For example, the Functional.OpenSSL test used values of 176 pass and 86 fails (see fuego-core/engine/tests/Functional.OpenSSL/OpenSSL.sh in fuego-1.1) to evaluate the result of this test.
log_compare "$TESTDIR" "176" "${P_CRIT}" "p"
log_compare "$TESTDIR" "86" "${N_CRIT}" "n"
But tests in JTA, such as Functional.LTP.Open_Posix expected the variables LTP_OPEN_POSIX_SUBTEST_COUNT_POS and LTP_OPEN_POSIX_SUBTEST_COUNT_NEG to be defined in a the board file for the device under test.
For example, the board file might have lines like the following:
LTP_OPEN_POSIX_SUBTEST_COUNT_POS="1232" LTP_OPEN_POSIX_SUBTEST_COUNT_NEG="158"
These were used in the log_compare function of the base script of the test like so:
log_compare "$TESTDIR" $LTP_OPEN_POSIX_SUBTEST_COUNT_POS "${P_CRIT}" "p"
log_compare "$TESTDIR" $LTP_OPEN_POSIX_SUBTEST_COUNT_NEG "${N_CRIT}" "n"
Starting with Fuego version 1.2, these would be replaced with criteria.json files like the following:
For Functional.OpenSSL:
{
"schema_version":"1.0",
"criteria":[
'tguid': 'OpenSSL',
'min_pass': 176,
'max_fail': 86
]
}
For Functional.LTP.Open_Posix:
{
"schema_version":"1.0",
"criteria":[
'tguid': 'LTP.Open_Posix',
'min_pass': 1232,
'max_fail': 158
]
}
FIXTHIS - should there be 'default' somewhere in the preceding tguids?
Benchmark measure evaluations [edit section]
For Benchmark programs, the pass criteria consists of one or more measurement thresholds, along with operators, that are compared with the results produced by the Benchmark.In JTA and Fuego 1.1 this data was contained in the reference.log file.
examples [edit section]
Here are some example criteria.json files:Benchmark.dbench [edit section]
{
"schema_version":"1.0",
"criteria":[
{
"tguid":"default.dbench.Throughput",
"reference":{
"value":0,
"operator":"gt"
}
},
{
"tguid":"default.dbench",
"min_pass":1
}
]
}
simple count [edit section]
schema [edit section]
The schema for the criteria.json file is contained in the fuego-core repository at: engine/scripts/parser/fuego-criteria-schema.json.Here it is (as of Fuego 1.2):
{
"$schema":"http://json-schema.org/schema#",
"id":"http://www.fuegotest.org/download/fuego_criteria_schema_v1.0.json",
"title":"criteria",
"description":"Pass criteria for a test suite",
"definitions":{
"criterion":{
"title":"criterion ",
"description":"Criterion for deciding if a test (test_set, test_case or measure) passes",
"type":"object",
"properties":{
"tguid":{
"type":"string",
"description":"unique identifier of a test (e.g.: Sequential_Output.CPU)"
},
"min_pass":{
"type":"number",
"description":"Minimum number of tests that must pass"
},
"max_fail":{
"type":"number",
"description":"Maximum number of tests that can fail"
},
"must_pass_list":{
"type":"array",
"description":"Detailed list of tests that must pass",
"items":{
"type":"string"
}
},
"fail_ok_list":{
"type":"array",
"description":"Detailed list of tests that can fail",
"items":{
"type":"string"
}
},
"reference":{
"type":"object",
"description":"Reference measure that is compared to a measure to decide the status",
"properties":{
"value":{
"type":[
"string",
"number",
"integer"
],
"description":"A value (often a threshold) to compare against"
},
"operator":{
"type":"string",
"description":"Type of operation to compare against",
"enum":[
"eq",
"ne",
"gt",
"ge",
"lt",
"le"
]
}
},
"required":[
"value",
"operator"
]
}
},
"required":[
"tguid"
]
}
},
"type":"object",
"properties":{
"schema_version":{
"type":"string",
"description":"The version number of this JSON schema",
"enum":[
"1.0"
]
},
"criteria":{
"type":"array",
"description":"A list of criterion items",
"items":{
"$ref":"#/definitions/criterion"
}
}
},
"required":[
"schema_version",
"criteria"
]
}