Introduction [edit section]

= Introduction =
The criteria.json file is used to specify the criteria used to determine
whether a test has passed or failed.

For the purpose of this explanation, I'll group tests into roughly 3 groups:
 * simple Functional tests
 * complex Functional tests
 * Benchmarks

I'll come back to these definitions in a moment.

The criteria.json file contains data that allows Fuego to interpret the
test results, and indicate overall PASS or FAIL status for a test.

For functional tests, this includes things like counting the number of
test_cases in the test that had "PASS" or "FAIL" status, as well
as ignoring some specific test results.

For benchmark tests, this includes specifying threshold values for
measurements taken by the benchmark, as well as operations (e.g. less than
greater than), to use to determine if a specific measure passed or failed.

Fuego uses the results of the test along with the criteria, to determine
the final result of the test.

If no criteria.json file is provided, then a default is constructed based
on the test results, consisting of the following:
{{{
{
 'tguid': <test_set_name>
 'max_fail': 0
}
}}}

types of tests and pass criteria [edit section]

== types of tests and pass criteria ==
A simple functional test runs a short sequence of tests, and if any one
of them fails, then the test is reported as a failure.  Since this corresponds
to the default criteria.json, then most simple Functional tests do not need
to provide a criteria.json file.

A complex Functional test (such as LTP or glib) has hundreds or possibly thousands of individual test cases.  Such tests often have some number of
individual test cases that fail, but which may be safely ignored (either temporarily or
permamently).  For example, some test cases may fail sporadically due to problems
with the test infrastructure or environment. Other tests may fail due to configuration choices for the software on the board. (For example, a choice of kernel config may cause some tests to fail - but this is
expected and these fail results should be ignored).

Complex Functional tests requires a criteria.json file, to avoid failing the entire test because of individual test_cases that should be ignored.

Finally, a Benchmark test is one that produces one or more "measurements",
which are test results with numeric values.  In order to determine whether
a result indicates a PASS or a FAIL result, Fuego needs to compare the
numeric result with some threshold value.  The criteria.json file holds
the threshold value and operator used for making this comparison.

Different boards, or boards with different software installations or configurations, may require
different pass criteria for the same tests.  Therefore, the pass criteria
are broken out into a separate file that can be adjusted at each test site,
and for each board.  Ultimately, we would like testers to be able to share
their pass criteria, so that each Fuego user does not have to determine
these on their own.

compatibility with previous Fuego versions [edit section]

= compatibility with previous Fuego versions =
Previously, Fuego (and it's predecessor JTA) supported pass criteria functionality in two
different ways.

Functional test pass/fail counts [edit section]

== Functional test pass/fail counts ==
For functional tests counts of positive and negative
results were either hard-coded into the base scripts for the test,
as arguments to the log_compare() in each test's
test_processing() function, or they were specified as variables, read
from the board file, and applied in the test_processing() function.

For example, the Functional.OpenSSL test used values of 176 pass
and 86 fails (see fuego-core/engine/tests/Functional.OpenSSL/OpenSSL.sh in
fuego-1.1) to evaluate the result of this test.
{{{
    log_compare "$TESTDIR" "176" "${P_CRIT}" "p"
    log_compare "$TESTDIR" "86" "${N_CRIT}" "n"
}}}

But tests in JTA, such as Functional.LTP.Open_Posix expected
the variables LTP_OPEN_POSIX_SUBTEST_COUNT_POS and
LTP_OPEN_POSIX_SUBTEST_COUNT_NEG to be defined in a the board
file for the device under test.

For example, the board file might have lines like the following:
{{{
LTP_OPEN_POSIX_SUBTEST_COUNT_POS="1232"
LTP_OPEN_POSIX_SUBTEST_COUNT_NEG="158"
}}}

These were used in the log_compare function of the base script of the test
like so:
{{{
    log_compare "$TESTDIR" $LTP_OPEN_POSIX_SUBTEST_COUNT_POS "${P_CRIT}" "p"
    log_compare "$TESTDIR" $LTP_OPEN_POSIX_SUBTEST_COUNT_NEG "${N_CRIT}" "n"
}}}

Starting with Fuego version 1.2, these would be replaced with criteria.json files like the following:

For Functional.OpenSSL:
{{{
{
    "schema_version":"1.0",
    "criteria":[
        'tguid': 'OpenSSL',
        'min_pass': 176,
        'max_fail': 86
    ]
}
}}}

For Functional.LTP.Open_Posix:
{{{
{
    "schema_version":"1.0",
    "criteria":[
        'tguid': 'LTP.Open_Posix',
        'min_pass': 1232,
        'max_fail': 158
    ]
}
}}}

FIXTHIS - should there be 'default' somewhere in the preceding tguids?

Benchmark measure evaluations [edit section]

== Benchmark measure evaluations ==
For Benchmark programs, the pass criteria consists of one or more
measurement thresholds, along with operators, that are compared
with the results produced by the Benchmark.

In JTA and Fuego 1.1 this data was contained in the reference.log file.

examples [edit section]

= examples =
Here are some example criteria.json files:
== Benchmark.dbench ==
{{{#!YellowBox
{
    "schema_version":"1.0",
    "criteria":[
        {
            "tguid":"default.dbench.Throughput",
            "reference":{
                "value":0,
                "operator":"gt"
            }
        },
        {
            "tguid":"default.dbench",
            "min_pass":1
        }
    ]
}
}}}

simple count [edit section]

== simple count ==

schema [edit section]

= schema =
The schema for the criteria.json file is contained in the fuego-core
repository at: engine/scripts/parser/fuego-criteria-schema.json.

Here it is (as of Fuego 1.2):
{{{#!YellowBox
{
    "$schema":"http://json-schema.org/schema#",
    "id":"http://www.fuegotest.org/download/fuego_criteria_schema_v1.0.json",
    "title":"criteria",
    "description":"Pass criteria for a test suite",
    "definitions":{
        "criterion":{
            "title":"criterion ",
            "description":"Criterion for deciding if a test (test_set, test_case or measure) passes",
            "type":"object",
            "properties":{
                "tguid":{
                    "type":"string",
                    "description":"unique identifier of a test (e.g.: Sequential_Output.CPU)"
                },
                "min_pass":{
                    "type":"number",
                    "description":"Minimum number of tests that must pass"
                },
                "max_fail":{
                    "type":"number",
                    "description":"Maximum number of tests that can fail"
                },
                "must_pass_list":{
                    "type":"array",
                    "description":"Detailed list of tests that must pass",
                    "items":{
                        "type":"string"
                    }
                },
                "fail_ok_list":{
                    "type":"array",
                    "description":"Detailed list of tests that can fail",
                    "items":{
                        "type":"string"
                    }
                },
                "reference":{
                    "type":"object",
                    "description":"Reference measure that is compared to a measure to decide the status",
                    "properties":{
                        "value":{
                            "type":[
                                "string",
                                "number",
                                "integer"
                            ],
                            "description":"A value (often a threshold) to compare against"
                        },
                        "operator":{
                            "type":"string",
                            "description":"Type of operation to compare against",
                            "enum":[
                                "eq",
                                "ne",
                                "gt",
                                "ge",
                                "lt",
                                "le"
                            ]
                        }
                    },
                    "required":[
                        "value",
                        "operator"
                    ]
                }
            },
            "required":[
                "tguid"
            ]
        }
    },
    "type":"object",
    "properties":{
        "schema_version":{
            "type":"string",
            "description":"The version number of this JSON schema",
            "enum":[
                "1.0"
            ]
        },
        "criteria":{
            "type":"array",
            "description":"A list of criterion items",
            "items":{
                "$ref":"#/definitions/criterion"
            }
        }
    },
    "required":[
        "schema_version",
        "criteria"
    ]
}
}}}