{{TableOfContents}}

Fuego test results are presented in the Jenkins interface via a number
of mechanisms.

built-in Jenkins status [edit section]

= built-in Jenkins status =
Jenkins automatically presents the status of the last few tests (jobs)
that have been executed, on the job page for those jobs.

A list of previous builds of the job are shown in the left-hand pane
of the page for the job, showing colored balls indicating the test status.

FIXTHIS - add more details here

flot plugin [edit section]

= flot plugin =
The flot plugin for Jenkins provides visualization of test results.

In Fuego 1.0 and previous (JTA), these provided plot data for Benchmark
results.

See [[flot]] for more information.

Desired behavior [edit section]

= Desired behavior =
Here is an outline of the desired behavior of the Jenkins visualization
(Fuego charts) for the 1.2 release:

There are 4 different chart output options:
 * 1) a plot of benchmark measures (called "measure_plot")
 * 2) a table of testcase results (called "testcase_table")
 * 3) a table of testcase summary counts per testset (called "testset_summary_table")
 * 4) a plot of testcase summary counts per testset (called testset_summary_plot)

Output is controlled by a file called: chart_data.json

Inside that, there is a data structure indicating the configuration for
one or more charts, called the chart_config.

This can be specified in the test directory, in the file chart_config.json

If chart_config.json is missing, then defaults are used:
 * for a Benchmark, Fuego produces a plot of all measures (in a single chart)
   * that is, it scans the test for all measures, plots them using flot in a "measure_plot"
 * for a Functional test, Fuego produces a table of testcase results.
   * that is, it scans the test for all testcases, and produces an HTML table (using flot) in a "testcase_table"

(not currently implemented)The chart_config can also specify a list
of tguids to process (either measures, testcases or a combination of those.

A job page shows the data for all specs and boards related to that
test.  e.g. min1.default.Benchmark.bonnie shows results for:
 * boards: min1, bbb, ren1, docker
 * specs: default, noroot
 * measures: (the ones specified in chart_config.json)

The module fuego_parser_results.py is used to generate results.json

The module prepare_chart_data.py is used to generate flat_plot_data.txt,
which is then used to create chart_data.son.  It uses chart_config.json to determine what results to include in chart_data.json.

The flot program mod.js is used to draw the actual plots and tables based
on chart_data.json.  mod.js is included in the web page for the job view
by Jenkins (along with the base flot libraries and jquery library, which
flot uses).

Detailed chart information [edit section]

= Detailed chart information =
== measure_plot ==
A measure_plot is a graph of measures for a benchmark, with the following attributes:
{{{
  title=<test>-<testset>
  X series=build number
  Y1 series=result
  Y1 label=<board>-<spec>-<test>-<kernel>-<tguid>
  Y2 series=ref
  Y2 label=<board>-<spec>-<test>-<kernel>-<tguid>-ref
}}}

It plots measures (y) versus build_numbers.

Here's example data for this:
{{{#!YellowBox
 "charts": [
    {  # definition of chart 1
      "title": "Benchmark.fuego_check_plots-main.shell_random"
      "chart": {
         "chart_type": "measure_plot",
         "data": [
           {
              "label": "min1-default_spec-Benchmark.fuego_check_plots-v4.4-main.shell_random",
              "data": [ ["1","1006"],["2","1116"] ],
              "points": {"symbol": "circle"}
           },
           {
              "label": "min1-default_spec-Benchmark.fuego_check_plots-v4.4-main.shell_random-ref",
              "data": [ ["1","800"],["2","800"] ],
              "points": ["symbol":"cross"}
           }
         ]
         # note: could put flot config object here
      }
  }
 ]
}}}

FIXTHIS - add testset, and draw one plot per testset.

testcase_table [edit section]

== testcase_table ==
A testcase_table is a table of testcases (usually for a functional test), with the following attributes:
{{{
  title=<board>-<spec>-<test>-<kernel>-<tguid>
  headers:
     board:
     test:
     kernel:
     tguid:
  row=(one per line with matching tguid in flat_chart_data.txt)
  columns=build_number, start_time/timestamp, duration, result
}}}

It shows testcase results by build_id (runs).

Daniel's table has:
{{{
  overall title=<test>
    chart title=<board>-<spec>-<testset>-<testcase>
    headers:
       board:
       kernel_version:
       test_spec:
       test_case:
       test_plan:
       test_set:
       toolchain:
   build number | status | start_time | duration
}}}

Cai's table has:
{{{
   overall title=<test>
   summary:
      latest total:
      latest pass:
      latest fail
      latest untest:
   table:
   "no" | <test-name>  | test time |
                               | start-time |
                               | end-time | 
                               | board version |
                               | test dir |
                               | test device |
                               | filesystem |
                               | command line |
   --------------------------------------------
   testcase number | testcase     | result |
}}}

This shows the result of only one run (the latest)

Tim's testcase table has:
(one table per board-testname-testset)
{{{
   overall title=<test>
   header: 
     board
     kernel version
     spec?
     filesystem
     test directory?
     command line?
   --------------------------------------------
   tguid | results
         | build_number |
         | b1 | b2 | bn | 
   <tguid1>|result1|result2|resultn|
        totals
   pass: |    |    |    |
   fail: |    |    |    |
   skip: |    |    |    |
   error:|    |    |    |
   --------------------------------------------
}}}

testset_summary_table [edit section]

== testset_summary_table ==
A testset_summary_table is a table of testsets (usually for a complex
functional test) with the following attributes:
 * row=(one per line with matching testset/build-number in flat_chart_data.txt)
 * columns=test set, build_number, start_time/timestamp, testset pass count, testset fail count, duration
 * sort rows by testset, then by build_number

  title=<board>
  headers:
     board:
     kernel(s):
  -----------------------------------------------------
                            |    counts               
  build number   | test set | pass | fail| skip | err |
  <bn>           |  <ts1>   |
    (row-span    |  <ts2>   |
  as appropriate)|  <ts3>   |
  <b2n>          |  <ts1>   |
                 |  <ts2>   |

{{{
  title=<board>
  headers:
     board:
     kernel(s):
  -----------------------------------------------------
                            |    counts               
  build number   | test set | pass | fail| skip | err |
  <bn>           |  <ts1>   |
    (row-span    |  <ts2>   |
  as appropriate)|  <ts3>   |
  <b2n>          |  <ts1>   |
                 |  <ts2>   |

}}}

Here was the format of the first attempt:
{{{
  title=<board>-<spec>-<test>-<kernel>
  headers:
     board:
     test:
     kernel:
  -------------------------------------------------------------------------
                                                |    counts               
  testset | build_number | start time| duration | pass | fail| skip | err |
  <ts>    | ...|

}}}

It shows testset summary results by runs

Here's an alternate testset summary arrangement, that I'm not using at the moment:
{{{
   --------------------------------------------
   testset | results
           | b1                      | b2    | bn    | 
           | pass | fail | skip | err |p|f|s|e|p|f|s|e|
   <ts>    | <cnt>| <cnt>| <cnt>| <cnt>...            |
        totals
   --------------------------------------------
}}}

testset_summary_plot [edit section]

== testset_summary_plot ==
A testset_summary_plot is a graph of testsets (usually for a complex
functional test) with the following attributes:
{{{  title=<board>-<spec>-<test>-<kernel>
  X series=build number
  Y1 series=pass_count
  Y1 label=<board>-<spec>-<test>-<kernel>-<testset>-pass
  Y2 series=fail_count
  Y2 label=<board>-<spec>-<test>-<kernel>-<testset>-fail
}}}

It graphs testset summary results versus build_ids

structure of chart_data.json [edit section]

== structure of chart_data.json ==
Here's an example:

{{{#!YellowBox
{
  "chart_config": {
     "type": "measure_plot"
     "title:": "min1-Benchmark.fuego_check_plots-default"
     "chart_data": {
        data
}
}}}

feature deferred to a future release [edit section]

== feature deferred to a future release ==
 * ability to specify the axes for plots
 * ability to specify multiple charts in chart_config
   * current Daniel code tries to automatically do this based on test_sets

Architecture for generic charting [edit section]

= Architecture for generic charting =
Assuming you have a flat list of entries with attributes for
board, testname, spec, tguid, result, etc., then you can use treat this like
a sql database, and do the following:
 * make a list of charts to build
   * have a chart-loopover-key = type of data to use for loop over charts
   * or, specify a list of charts
 * define a key to use to extract data for a chart (the chart-key)
 * for each chart:
   * make a list of rows to build
     * have a row-loopover-key = filter for rows to include
     * or, specify a list of rows
   * define a key to use to extract data for each row
   * if sub-columns are defined:
     * make a sub-column-key
     * make a two-dimensional array to hold the sub-column data
   * for each entry:
     * add the entry to the correct row and sub-column
   * sort by the desired column
   * output the data in table format
     * loop over rows in sort order
     * generate the html for each row
       * loop over sub-columns, if defined
   * return html

There's a similar set of data (keys, looping) for defining plot data.
With keys selecting the axes.

chart_config.json [edit section]

= chart_config.json =
== example ==

{{{#!YellowBox
{
   "chart_type": "measure_plot"|"testcase_table"|"testset_summary_table"
}
}}}

schema [edit section]

== schema ==
Here is the schema for chart_config.json:

{{{#!YellowBox

Not Documented Yet!!

}}}
FIXTHIS - document the chart_config.json schema