{{TableOfContents}}

Here is an outline of the flow of control for test execution in fuego:

This example if for the fictional test Functional.foo:

• Jenkins - decides there is a test to perform (maybe based on user input) * Functional.foo/config.xml has a shell fragment that is executed by Jenkins, this may do other setup, but ultimately it sources the base test script for this test. * test script * defines the test base functions, and then sources: * functional.sh * source overlays.sh * set_overlay_vars * run_python $OF_OVGEN $OF_CLASSDIR_ARGS $OF_OVFILES_ARGS $OF_TESTPLAN_ARGS $OF_SPECDIR_ARGS $OF_OUTPUT_FILE_ARGS * ovgen.py * source $OF_OUTPUT_FILE * source reports.sh * source functions.sh * pre_test * build * pre_build * test_build - base script function to build the test program from source * post_build * deploy * pre_deploy * test_deploy - base script function to deploy the test to the target * post_deploy * test_run - base script function to run the test * get_testlog * test_processing - base script function to analyze the test results * log_compare * post_test * test_cleanup * Jenkins post_build actions

 * '''Jenkins''' - decides there is a test to perform (maybe based on user input)  * '''Functional.foo/config.xml'''  has a shell fragment that is executed by Jenkins, this may do other setup, but ultimately it sources the base test script for this test.    * ''test script''      * defines the test base functions, and then sources:      * '''functional.sh'''        * source '''overlays.sh'''        * '''set_overlay_vars'''          * '''run_python $OF_OVGEN $OF_CLASSDIR_ARGS $OF_OVFILES_ARGS $OF_TESTPLAN_ARGS $OF_SPECDIR_ARGS $OF_OUTPUT_FILE_ARGS'''            * ovgen.py          * source $OF_OUTPUT_FILE        * source '''reports.sh'''        * source '''functions.sh'''        * '''pre_test'''        * '''build'''          * '''pre_build'''          * '''test_build''' - base script function to build the test program from source          * '''post_build'''        * '''deploy'''          * '''pre_deploy'''          * '''test_deploy''' - base script function to deploy the test to the target          * '''post_deploy'''        * '''test_run''' - base script function to run the test        * '''get_testlog'''        * '''test_processing''' - base script function to analyze the test results          * '''log_compare'''        * '''post_test'''          * '''test_cleanup'''   * Jenkins post_build actions

FIXTHIS - finish test execution flow outline

sample paths [edit section]

= sample paths = * run_python /home/jenkins/scripts/ovgen/ovgen.py --classdir /home/jenkins/overlays//base --ovfiles /home/jenkins/overlays//distribs/nologger.dist /home/jenkins/overlays//boards/bbb-poky-sdk.board --testplan /home/jenkins/overlays//testplans/testplan_default.json --specdir /home/jenkins/overlays//test_specs/ --output /home/jenkins/work/bbb-poky-sdk_prolog.sh

comparison of top-level scripts [edit section]

= comparison of top-level scripts =Here is a comparison of the 3 top-level scripts used by Fuego base scripts: * functional.sh * benchmark.sh * stress.sh

functional.sh outline: * source overlays.sh, set_overlay_vars * source reports.sh * source functions.sh * pre_test * if $Rebuild, build * deploy * test_run * get_testlog * test_processing

benchmark.sh outline: * source overlays.sh, set_overlay_vars * source reports.sh * source functions.sh * pre_test * if $Rebuild, build * deploy * test_run * set_testres_file * bench_processing * check_create_logrun

stress.sh outline: * source overlays.sh, set_overlay_vars * source reports.sh * source functions.sh * pre_test * if $Rebuild, build * deploy * test_run * get_testlog

stress.sh is only used by 2 tests, and is a strict subset of the functional test.  The 2 tests are: Functional.scrashme and Functional.pi_tests.

Here is a table showing the differences:{{{#!Tableshow_edit_links=0show_sort_links=0||step||stress.sh||functional.sh||benchmark.sh||||source overlays.sh, set_overlay_vars||x||x||x||||source reports.sh||x||x||x||||source functions.sh||x||x||x||||[[function_pre_test|pre_test]]||x||x||x||||if $Rebuild, [[function_build|build]]||x||x||x||||[[function_deploy|deploy]]||x||x||x||||[[function_test_run|test_run]]||x||x||x||||[[function_get_testlog|get_testlog]]||x||x||(called inside bench_processing)||||[[function_test_processing|test_processing]]||.||x||.||||[[function_set_testres_file|set_testres_file]]||.||.||x||||[[function_bench_processing|bench_processing]]||.||.||x||||[[function_check_create_logrun|check_create_logrun]]||.||.||x||}}}

test information [edit section]

= test information === jobs definitions ==As of February 2017, there are 67 job config.xml files under fuego-core/jobs

63 jobs source something besides functions.sh (functions.sh is sourced for the call to post_test)

The job Functional.libpng/config.xml sources 2 items (besides functions.sh)This test is very weird and runs all the phases itself, instead ofrunning a single base script.

The following jobs don't source any shell scripts: * Matrix.Nightly * Matrix.Official * Run ALL test on ALL targets * Run ALL tests on SELECTED targets * Service.ReloadConfiguration

There are 30 Functional job directories, and 30 Benchmark job directories.

The following job directly sources reports.sh, then does gen_report: * Reports.make_pdf

The following tests don't start with Functional or Benchmark: * LTP * netperf * OpenSSL

test definitions [edit section]

== test definitions ==There are 31 Functional test directories, and 33 Benchmark test directories. * Functional.fsfuzz is a test with no job * Functional.mesa-demos is a test with no job * Functional.libpng is a job with no test   * apparently the test mechanisms used to be implemented as individual scripts under scripts/<phase>     * e.g. scripts/build/Functional.libpng.sh     * and scripts/deploy/Functional.libpng.sh * Benchmark.fs_mark is a test with no job * Benchmark.nbench_byte is a test with no job   * but there's a Benchmark.nbench-byte * Benchmark.sysbench is a test with no job

.sh scripts [edit section]

=== .sh scripts ===There are 29 Functional tests with .sh files: * Functional.fsfuzz has no .sh file * Functional.mesa_demos has not .sh file

There are 31 Benchmark tests with .sh files: * Benchmark.fs_mark has no .sh file * Benchmark.sysbench has no .sh file

Functional tests not sourcing functional.sh [edit section]

=== Functional tests not sourcing functional.sh ===There are 21 Functional .sh files that call functional.shHere are the ones that don't: * Functional.LTP.Devices   * ltp-devices.sh sources a common script ../LTP/ltp.sh   * it calls functional phases itself * Functional.LTP.Filesystem   * ltp-filesystems.sh sources a common script ../LTP/ltp.sh   * it calls functional phases itself * Functional.LTP.Open_Posix   * ltp-open_posix.sh sources a common script ../LTP/ltp.sh   * it calls functional phases itself * Functional.ft2demos   * ft2demos.sh calls functional phases itself * Functional.netperf   * netperf-func.sh sources netperf/netperf.sh   * it calls test_run and get_testlog itself * Functional.OpenSSL   * openssl-func.sh sources OpenSSL/openssl.sh   * it calls test_run, get_testlog and test_processing itself * Functional.pi_tests   * calls stress.sh

• calls stress.sh

   * calls stress.sh

There are 2 Functional tests that source stress.sh

Benchmark tests not sourcing benchmark.sh [edit section]

=== Benchmark tests not sourcing benchmark.sh ===There are 28 Benchmark tests that source benchmark.sh.

Here are the ones that don't: * Benchmark.netperf   * netperf-bench.sh sources netperf/netperf.sh   * it calls test_run and bench_processing directly * Benchmark.NetPipe   * NetPipe.sh sources functions.sh and overlays.sh   * it calls pre_test, build, deploy, and test_run directly     * it does not call set_testres_file, bench_processing, check_create_logrun