Jenkins Visualization
Fuego test results are presented in the Jenkins interface via a number of mechanisms.
built-in Jenkins status [edit section]
Jenkins automatically presents the status of the last few tests (jobs) that have been executed, on the job page for those jobs.A list of previous builds of the job are shown in the left-hand pane of the page for the job, showing colored balls indicating the test status.
FIXTHIS - add more details here
flot plugin [edit section]
The flot plugin for Jenkins provides visualization of test results.In Fuego 1.0 and previous (JTA), these provided plot data for Benchmark results.
See flot for more information.
Desired behavior [edit section]
Here is an outline of the desired behavior of the Jenkins visualization (Fuego charts) for the 1.2 release:There are 4 different chart output options:
- 1) a plot of benchmark measures (called "measure_plot")
- 2) a table of testcase results (called "testcase_table")
- 3) a table of testcase summary counts per testset (called "testset_summary_table")
- 4) a plot of testcase summary counts per testset (called testset_summary_plot)
Output is controlled by a file called: chart_data.json
Inside that, there is a data structure indicating the configuration for one or more charts, called the chart_config.
This can be specified in the test directory, in the file chart_config.json
If chart_config.json is missing, then defaults are used:
- for a Benchmark, Fuego produces a plot of all measures (in a single chart)
- that is, it scans the test for all measures, plots them using flot in a "measure_plot"
- for a Functional test, Fuego produces a table of testcase results.
- that is, it scans the test for all testcases, and produces an HTML table (using flot) in a "testcase_table"
(not currently implemented)The chart_config can also specify a list of tguids to process (either measures, testcases or a combination of those.
A job page shows the data for all specs and boards related to that test. e.g. min1.default.Benchmark.bonnie shows results for:
- boards: min1, bbb, ren1, docker
- specs: default, noroot
- measures: (the ones specified in chart_config.json)
The module fuego_parser_results.py is used to generate results.json
The module prepare_chart_data.py is used to generate flat_plot_data.txt, which is then used to create chart_data.son. It uses chart_config.json to determine what results to include in chart_data.json.
The flot program mod.js is used to draw the actual plots and tables based on chart_data.json. mod.js is included in the web page for the job view by Jenkins (along with the base flot libraries and jquery library, which flot uses).
Detailed chart information [edit section]
measure_plot [edit section]
A measure_plot is a graph of measures for a benchmark, with the following attributes:title=<test>-<testset> X series=build number Y1 series=result Y1 label=<board>-<spec>-<test>-<kernel>-<tguid> Y2 series=ref Y2 label=<board>-<spec>-<test>-<kernel>-<tguid>-ref
It plots measures (y) versus build_numbers.
Here's example data for this:
"charts": [
{ # definition of chart 1
"title": "Benchmark.fuego_check_plots-main.shell_random"
"chart": {
"chart_type": "measure_plot",
"data": [
{
"label": "min1-default_spec-Benchmark.fuego_check_plots-v4.4-main.shell_random",
"data": [ ["1","1006"],["2","1116"] ],
"points": {"symbol": "circle"}
},
{
"label": "min1-default_spec-Benchmark.fuego_check_plots-v4.4-main.shell_random-ref",
"data": [ ["1","800"],["2","800"] ],
"points": ["symbol":"cross"}
}
]
# note: could put flot config object here
}
}
]FIXTHIS - add testset, and draw one plot per testset.
testcase_table [edit section]
A testcase_table is a table of testcases (usually for a functional test), with the following attributes:
title=<board>-<spec>-<test>-<kernel>-<tguid>
headers:
board:
test:
kernel:
tguid:
row=(one per line with matching tguid in flat_chart_data.txt)
columns=build_number, start_time/timestamp, duration, result
It shows testcase results by build_id (runs).
Daniel's table has:
overall title=<test>
chart title=<board>-<spec>-<testset>-<testcase>
headers:
board:
kernel_version:
test_spec:
test_case:
test_plan:
test_set:
toolchain:
build number | status | start_time | duration
Cai's table has:
overall title=<test>
summary:
latest total:
latest pass:
latest fail
latest untest:
table:
"no" | <test-name> | test time |
| start-time |
| end-time |
| board version |
| test dir |
| test device |
| filesystem |
| command line |
--------------------------------------------
testcase number | testcase | result |
This shows the result of only one run (the latest)
Tim's testcase table has: (one table per board-testname-testset)
overall title=<test>
header:
board
kernel version
spec?
filesystem
test directory?
command line?
--------------------------------------------
tguid | results
| build_number |
| b1 | b2 | bn |
<tguid1>|result1|result2|resultn|
totals
pass: | | | |
fail: | | | |
skip: | | | |
error:| | | |
--------------------------------------------
testset_summary_table [edit section]
A testset_summary_table is a table of testsets (usually for a complex functional test) with the following attributes:- row=(one per line with matching testset/build-number in flat_chart_data.txt)
- columns=test set, build_number, start_time/timestamp, testset pass count, testset fail count, duration
- sort rows by testset, then by build_number
title=<board>
headers:
board:
kernel(s):
-----------------------------------------------------
| counts
build number | test set | pass | fail| skip | err |
<bn> | <ts1> |
(row-span | <ts2> |
as appropriate)| <ts3> |
<b2n> | <ts1> |
| <ts2> |
Here was the format of the first attempt:
title=<board>-<spec>-<test>-<kernel>
headers:
board:
test:
kernel:
-------------------------------------------------------------------------
| counts
testset | build_number | start time| duration | pass | fail| skip | err |
<ts> | ...|
It shows testset summary results by runs
Here's an alternate testset summary arrangement, that I'm not using at the moment:
--------------------------------------------
testset | results
| b1 | b2 | bn |
| pass | fail | skip | err |p|f|s|e|p|f|s|e|
<ts> | <cnt>| <cnt>| <cnt>| <cnt>... |
totals
--------------------------------------------
testset_summary_plot [edit section]
A testset_summary_plot is a graph of testsets (usually for a complex functional test) with the following attributes:title=<board>-<spec>-<test>-<kernel> X series=build number Y1 series=pass_count Y1 label=<board>-<spec>-<test>-<kernel>-<testset>-pass Y2 series=fail_count Y2 label=<board>-<spec>-<test>-<kernel>-<testset>-fail
It graphs testset summary results versus build_ids
structure of chart_data.json [edit section]
Here's an example:
{
"chart_config": {
"type": "measure_plot"
"title:": "min1-Benchmark.fuego_check_plots-default"
"chart_data": {
data
}
feature deferred to a future release [edit section]
- ability to specify the axes for plots
- ability to specify multiple charts in chart_config
- current Daniel code tries to automatically do this based on test_sets
Architecture for generic charting [edit section]
Assuming you have a flat list of entries with attributes for board, testname, spec, tguid, result, etc., then you can use treat this like a sql database, and do the following:- make a list of charts to build
- have a chart-loopover-key = type of data to use for loop over charts
- or, specify a list of charts
- define a key to use to extract data for a chart (the chart-key)
- for each chart:
- make a list of rows to build
- have a row-loopover-key = filter for rows to include
- or, specify a list of rows
- define a key to use to extract data for each row
- if sub-columns are defined:
- make a sub-column-key
- make a two-dimensional array to hold the sub-column data
- for each entry:
- add the entry to the correct row and sub-column
- sort by the desired column
- output the data in table format
- loop over rows in sort order
- generate the html for each row
- loop over sub-columns, if defined
- return html
- make a list of rows to build
There's a similar set of data (keys, looping) for defining plot data. With keys selecting the axes.
chart_config.json [edit section]
example [edit section]
{
"chart_type": "measure_plot"|"testcase_table"|"testset_summary_table"
}
schema [edit section]
Here is the schema for chart_config.json:
Not Documented Yet!!