{{TableOfContents}}

Fuego test results are presented in the Jenkins interface via a numberof mechanisms.

built-in Jenkins status [edit section]

= built-in Jenkins status =Jenkins automatically presents the status of the last few tests (jobs)that have been executed, on the job page for those jobs.

A list of previous builds of the job are shown in the left-hand paneof the page for the job, showing colored balls indicating the test status.

FIXTHIS - add more details here

flot plugin [edit section]

= flot plugin =The flot plugin for Jenkins provides visualization of test results.

In Fuego 1.0 and previous (JTA), these provided plot data for Benchmarkresults.

See [[flot]] for more information.

Desired behavior [edit section]

= Desired behavior =Here is an outline of the desired behavior of the Jenkins visualization(Fuego charts) for the 1.2 release:

There are 4 different chart output options: * 1) a plot of benchmark measures (called "measure_plot") * 2) a table of testcase results (called "testcase_table") * 3) a table of testcase summary counts per testset (called "testset_summary_table") * 4) a plot of testcase summary counts per testset (called testset_summary_plot)

Output is controlled by a file called: chart_data.json

Inside that, there is a data structure indicating the configuration forone or more charts, called the chart_config.

This can be specified in the test directory, in the file chart_config.json

If chart_config.json is missing, then defaults are used: * for a Benchmark, Fuego produces a plot of all measures (in a single chart)   * that is, it scans the test for all measures, plots them using flot in a "measure_plot" * for a Functional test, Fuego produces a table of testcase results.   * that is, it scans the test for all testcases, and produces an HTML table (using flot) in a "testcase_table"

(not currently implemented)The chart_config can also specify a listof tguids to process (either measures, testcases or a combination of those.

A job page shows the data for all specs and boards related to thattest.  e.g. min1.default.Benchmark.bonnie shows results for: * boards: min1, bbb, ren1, docker * specs: default, noroot * measures: (the ones specified in chart_config.json)

The module fuego_parser_results.py is used to generate results.json

The module prepare_chart_data.py is used to generate flat_plot_data.txt,which is then used to create chart_data.son.  It uses chart_config.json to determine what results to include in chart_data.json.

The flot program mod.js is used to draw the actual plots and tables basedon chart_data.json.  mod.js is included in the web page for the job viewby Jenkins (along with the base flot libraries and jquery library, whichflot uses).

Detailed chart information [edit section]

= Detailed chart information === measure_plot ==A measure_plot is a graph of measures for a benchmark, with the following attributes:{{{  title=<test>-<testset>  X series=build number  Y1 series=result  Y1 label=<board>-<spec>-<test>-<kernel>-<tguid>  Y2 series=ref  Y2 label=<board>-<spec>-<test>-<kernel>-<tguid>-ref}}}

It plots measures (y) versus build_numbers.

Here's example data for this:{{{#!YellowBox "charts": [    {  # definition of chart 1      "title": "Benchmark.fuego_check_plots-main.shell_random"      "chart": {         "chart_type": "measure_plot",         "data": [           {              "label": "min1-default_spec-Benchmark.fuego_check_plots-v4.4-main.shell_random",              "data": [ ["1","1006"],["2","1116"] ],              "points": {"symbol": "circle"}           },           {              "label": "min1-default_spec-Benchmark.fuego_check_plots-v4.4-main.shell_random-ref",              "data": [ ["1","800"],["2","800"] ],              "points": ["symbol":"cross"}           }         ]         # note: could put flot config object here      }  } ]}}}

FIXTHIS - add testset, and draw one plot per testset.

testcase_table [edit section]

== testcase_table ==A testcase_table is a table of testcases (usually for a functional test), with the following attributes:{{{  title=<board>-<spec>-<test>-<kernel>-<tguid>  headers:     board:     test:     kernel:     tguid:  row=(one per line with matching tguid in flat_chart_data.txt)  columns=build_number, start_time/timestamp, duration, result}}}

It shows testcase results by build_id (runs).

Daniel's table has:{{{  overall title=<test>    chart title=<board>-<spec>-<testset>-<testcase>    headers:       board:       kernel_version:       test_spec:       test_case:       test_plan:       test_set:       toolchain:   build number | status | start_time | duration}}}

Cai's table has:{{{   overall title=<test>   summary:      latest total:      latest pass:      latest fail      latest untest:   table:   "no" | <test-name>  | test time |                               | start-time |                               | end-time |                                | board version |                               | test dir |                               | test device |                               | filesystem |                               | command line |   --------------------------------------------   testcase number | testcase     | result |}}}

This shows the result of only one run (the latest)

Tim's testcase table has:(one table per board-testname-testset){{{   overall title=<test>   header:      board     kernel version     spec?     filesystem     test directory?     command line?   --------------------------------------------   tguid | results         | build_number |         | b1 | b2 | bn |    <tguid1>|result1|result2|resultn|        totals   pass: |    |    |    |   fail: |    |    |    |   skip: |    |    |    |   error:|    |    |    |   --------------------------------------------}}}

testset_summary_table [edit section]

== testset_summary_table ==A testset_summary_table is a table of testsets (usually for a complexfunctional test) with the following attributes: * row=(one per line with matching testset/build-number in flat_chart_data.txt) * columns=test set, build_number, start_time/timestamp, testset pass count, testset fail count, duration * sort rows by testset, then by build_number

  title=<board>  headers:     board:     kernel(s):  -----------------------------------------------------                            |    counts                 build number   | test set | pass | fail| skip | err |  <bn>           |  <ts1>   |    (row-span    |  <ts2>   |  as appropriate)|  <ts3>   |  <b2n>          |  <ts1>   |                 |  <ts2>   |

{{{  title=<board>  headers:     board:     kernel(s):  -----------------------------------------------------                            |    counts                 build number   | test set | pass | fail| skip | err |  <bn>           |  <ts1>   |    (row-span    |  <ts2>   |  as appropriate)|  <ts3>   |  <b2n>          |  <ts1>   |                 |  <ts2>   |

}}}

Here was the format of the first attempt:{{{  title=<board>-<spec>-<test>-<kernel>  headers:     board:     test:     kernel:  -------------------------------------------------------------------------                                                |    counts                 testset | build_number | start time| duration | pass | fail| skip | err |  <ts>    | ...|

}}}

It shows testset summary results by runs

Here's an alternate testset summary arrangement, that I'm not using at the moment:{{{   --------------------------------------------   testset | results           | b1                      | b2    | bn    |            | pass | fail | skip | err |p|f|s|e|p|f|s|e|   <ts>    | <cnt>| <cnt>| <cnt>| <cnt>...            |        totals   --------------------------------------------}}}

testset_summary_plot [edit section]

== testset_summary_plot ==A testset_summary_plot is a graph of testsets (usually for a complexfunctional test) with the following attributes:{{{  title=<board>-<spec>-<test>-<kernel>  X series=build number  Y1 series=pass_count  Y1 label=<board>-<spec>-<test>-<kernel>-<testset>-pass  Y2 series=fail_count  Y2 label=<board>-<spec>-<test>-<kernel>-<testset>-fail}}}

It graphs testset summary results versus build_ids

structure of chart_data.json [edit section]

== structure of chart_data.json ==Here's an example:

{{{#!YellowBox{  "chart_config": {     "type": "measure_plot"     "title:": "min1-Benchmark.fuego_check_plots-default"     "chart_data": {        data}}}}

feature deferred to a future release [edit section]

== feature deferred to a future release == * ability to specify the axes for plots * ability to specify multiple charts in chart_config   * current Daniel code tries to automatically do this based on test_sets

Architecture for generic charting [edit section]

= Architecture for generic charting =Assuming you have a flat list of entries with attributes forboard, testname, spec, tguid, result, etc., then you can use treat this likea sql database, and do the following: * make a list of charts to build   * have a chart-loopover-key = type of data to use for loop over charts   * or, specify a list of charts * define a key to use to extract data for a chart (the chart-key) * for each chart:   * make a list of rows to build     * have a row-loopover-key = filter for rows to include     * or, specify a list of rows   * define a key to use to extract data for each row   * if sub-columns are defined:     * make a sub-column-key     * make a two-dimensional array to hold the sub-column data   * for each entry:     * add the entry to the correct row and sub-column   * sort by the desired column   * output the data in table format     * loop over rows in sort order     * generate the html for each row       * loop over sub-columns, if defined   * return html

There's a similar set of data (keys, looping) for defining plot data.With keys selecting the axes.

chart_config.json [edit section]

= chart_config.json === example ==

{{{#!YellowBox{   "chart_type": "measure_plot"|"testcase_table"|"testset_summary_table"}}}}

schema [edit section]

== schema ==Here is the schema for chart_config.json:

{{{#!YellowBox

Not Documented Yet!!

}}}FIXTHIS - document the chart_config.json schema