Jenkins Visualization in 'raw' format
{{TableOfContents}} Fuego test results are presented in the Jenkins interface via a number of mechanisms. = built-in Jenkins status = Jenkins automatically presents the status of the last few tests (jobs) that have been executed, on the job page for those jobs. A list of previous builds of the job are shown in the left-hand pane of the page for the job, showing colored balls indicating the test status. FIXTHIS - add more details here = flot plugin = The flot plugin for Jenkins provides visualization of test results. In Fuego 1.0 and previous (JTA), these provided plot data for Benchmark results. See [[flot]] for more information. = Desired behavior = Here is an outline of the desired behavior of the Jenkins visualization (Fuego charts) for the 1.2 release: There are 4 different chart output options: * 1) a plot of benchmark measures (called "measure_plot") * 2) a table of testcase results (called "testcase_table") * 3) a table of testcase summary counts per testset (called "testset_summary_table") * 4) a plot of testcase summary counts per testset (called testset_summary_plot) Output is controlled by a file called: chart_data.json Inside that, there is a data structure indicating the configuration for one or more charts, called the chart_config. This can be specified in the test directory, in the file chart_config.json If chart_config.json is missing, then defaults are used: * for a Benchmark, Fuego produces a plot of all measures (in a single chart) * that is, it scans the test for all measures, plots them using flot in a "measure_plot" * for a Functional test, Fuego produces a table of testcase results. * that is, it scans the test for all testcases, and produces an HTML table (using flot) in a "testcase_table" (not currently implemented)The chart_config can also specify a list of tguids to process (either measures, testcases or a combination of those. A job page shows the data for all specs and boards related to that test. e.g. min1.default.Benchmark.bonnie shows results for: * boards: min1, bbb, ren1, docker * specs: default, noroot * measures: (the ones specified in chart_config.json) The module fuego_parser_results.py is used to generate results.json The module prepare_chart_data.py is used to generate flat_plot_data.txt, which is then used to create chart_data.son. It uses chart_config.json to determine what results to include in chart_data.json. The flot program mod.js is used to draw the actual plots and tables based on chart_data.json. mod.js is included in the web page for the job view by Jenkins (along with the base flot libraries and jquery library, which flot uses). = Detailed chart information = == measure_plot == A measure_plot is a graph of measures for a benchmark, with the following attributes: {{{ title=<test>-<testset> X series=build number Y1 series=result Y1 label=<board>-<spec>-<test>-<kernel>-<tguid> Y2 series=ref Y2 label=<board>-<spec>-<test>-<kernel>-<tguid>-ref }}} It plots measures (y) versus build_numbers. Here's example data for this: {{{#!YellowBox "charts": [ { # definition of chart 1 "title": "Benchmark.fuego_check_plots-main.shell_random" "chart": { "chart_type": "measure_plot", "data": [ { "label": "min1-default_spec-Benchmark.fuego_check_plots-v4.4-main.shell_random", "data": [ ["1","1006"],["2","1116"] ], "points": {"symbol": "circle"} }, { "label": "min1-default_spec-Benchmark.fuego_check_plots-v4.4-main.shell_random-ref", "data": [ ["1","800"],["2","800"] ], "points": ["symbol":"cross"} } ] # note: could put flot config object here } } ] }}} FIXTHIS - add testset, and draw one plot per testset. == testcase_table == A testcase_table is a table of testcases (usually for a functional test), with the following attributes: {{{ title=<board>-<spec>-<test>-<kernel>-<tguid> headers: board: test: kernel: tguid: row=(one per line with matching tguid in flat_chart_data.txt) columns=build_number, start_time/timestamp, duration, result }}} It shows testcase results by build_id (runs). Daniel's table has: {{{ overall title=<test> chart title=<board>-<spec>-<testset>-<testcase> headers: board: kernel_version: test_spec: test_case: test_plan: test_set: toolchain: build number | status | start_time | duration }}} Cai's table has: {{{ overall title=<test> summary: latest total: latest pass: latest fail latest untest: table: "no" | <test-name> | test time | | start-time | | end-time | | board version | | test dir | | test device | | filesystem | | command line | -------------------------------------------- testcase number | testcase | result | }}} This shows the result of only one run (the latest) Tim's testcase table has: (one table per board-testname-testset) {{{ overall title=<test> header: board kernel version spec? filesystem test directory? command line? -------------------------------------------- tguid | results | build_number | | b1 | b2 | bn | <tguid1>|result1|result2|resultn| totals pass: | | | | fail: | | | | skip: | | | | error:| | | | -------------------------------------------- }}} == testset_summary_table == A testset_summary_table is a table of testsets (usually for a complex functional test) with the following attributes: * row=(one per line with matching testset/build-number in flat_chart_data.txt) * columns=test set, build_number, start_time/timestamp, testset pass count, testset fail count, duration * sort rows by testset, then by build_number {{{ title=<board> headers: board: kernel(s): ----------------------------------------------------- | counts build number | test set | pass | fail| skip | err | <bn> | <ts1> | (row-span | <ts2> | as appropriate)| <ts3> | <b2n> | <ts1> | | <ts2> | }}} Here was the format of the first attempt: {{{ title=<board>-<spec>-<test>-<kernel> headers: board: test: kernel: ------------------------------------------------------------------------- | counts testset | build_number | start time| duration | pass | fail| skip | err | <ts> | ...| }}} It shows testset summary results by runs Here's an alternate testset summary arrangement, that I'm not using at the moment: {{{ -------------------------------------------- testset | results | b1 | b2 | bn | | pass | fail | skip | err |p|f|s|e|p|f|s|e| <ts> | <cnt>| <cnt>| <cnt>| <cnt>... | totals -------------------------------------------- }}} == testset_summary_plot == A testset_summary_plot is a graph of testsets (usually for a complex functional test) with the following attributes: {{{ title=<board>-<spec>-<test>-<kernel> X series=build number Y1 series=pass_count Y1 label=<board>-<spec>-<test>-<kernel>-<testset>-pass Y2 series=fail_count Y2 label=<board>-<spec>-<test>-<kernel>-<testset>-fail }}} It graphs testset summary results versus build_ids == structure of chart_data.json == Here's an example: {{{#!YellowBox { "chart_config": { "type": "measure_plot" "title:": "min1-Benchmark.fuego_check_plots-default" "chart_data": { data } }}} == feature deferred to a future release == * ability to specify the axes for plots * ability to specify multiple charts in chart_config * current Daniel code tries to automatically do this based on test_sets = Architecture for generic charting = Assuming you have a flat list of entries with attributes for board, testname, spec, tguid, result, etc., then you can use treat this like a sql database, and do the following: * make a list of charts to build * have a chart-loopover-key = type of data to use for loop over charts * or, specify a list of charts * define a key to use to extract data for a chart (the chart-key) * for each chart: * make a list of rows to build * have a row-loopover-key = filter for rows to include * or, specify a list of rows * define a key to use to extract data for each row * if sub-columns are defined: * make a sub-column-key * make a two-dimensional array to hold the sub-column data * for each entry: * add the entry to the correct row and sub-column * sort by the desired column * output the data in table format * loop over rows in sort order * generate the html for each row * loop over sub-columns, if defined * return html There's a similar set of data (keys, looping) for defining plot data. With keys selecting the axes. = chart_config.json = == example == {{{#!YellowBox { "chart_type": "measure_plot"|"testcase_table"|"testset_summary_table" } }}} == schema == Here is the schema for chart_config.json: {{{#!YellowBox Not Documented Yet!! }}} FIXTHIS - document the chart_config.json schema