ovgen feature notes
ovgen is used to construct a 'prolog' file, which is a shell script which has variables and definitions from the distro and board files, to control the execution of a test. The prolog script is generated when the base script is executed, and is sourced at some point during test execution.
Note that test execution is done in two separate phases. That is, Jenkins executes the test script itself to do the first phases of a test (pre_test, build, deploy, run), then does a "source functions.sh ; post_test" to execute the 'post_test' phase of the test.
to do [edit section]
- + add --testdir <arg> to ovgen command line
- + if testdir is specified, then read specs and plan from there
- + read spec file from test directory
- + If not present, read from overlays/test_specs
- + read test plan from test directory
- + only if it's present
- + read spec file from test directory
- + add quotes to testspec variables
- + move ovgen.py into 'scripts' directory
- + support variable definitions without quotes around values
- + modify parser to read overlay variables without quotes
- +if testplan name is testplan_default, then just assume each
test spec has: { testName: <test_name> and spec: default }
- test this with moved specs and plans
- + move all specs to their respective test directories
- figure out what to do about missing specs:
*
missing spec in Benchmark.aim7 missing spec in Benchmark.bfoo missing spec in Benchmark.fs_mark missing spec in Benchmark.nbench-byte missing spec in Benchmark.sysbench missing spec in Functional.arch_timer missing spec in Functional.cmt missing spec in Functional.fsfuzz missing spec in Functional.mesa-demos missing spec in Functional.scifab missing spec in Functional.sdhi_0
- figure out what to do about missing specs:
*
- move all plans to their respective test directories
- bc
- hello
- sata, mmc, usb
- tests are: Benchmark.fio, Benchmark.Bonnie, Benchmark.IOzone, Benchmakr.ffsb, Benchmark.Tiobench, Benchmark.aiostress, Functional.synctest, Benchmakr.Interbench, Benchmark.dbench
- specify per-test testplans in the Jenkins interface
- have groovy script parse the testdir for testplan names
- + if spec is 'default', and no test.spec file is found, then just synthesize one with the following:
{
"testName": "Functional.zlib",
"specs":
[
{
"name":"default"
}
]
}
- read testplans from overlays/testplans and testdir
information [edit section]
- Functional.bzip2.spec has no variables (no specs besides default)
- Functional.zlib.spec has no variables (no specs besides default)
- Functional.hello_world.spec has different specs
testplan file format [edit section]
testplan<name>.json file has:- testPlanName
- tests:
- testName
- spec
spec file format [edit section]
Functional.<test_name>.json file has:- testName:
- specs:
- name
- var1
- var2 ...
Intended features [edit section]
- avoid definining test variables for tests unrelated to this one.
- see if default specs are needed by unrelated tests:
- I know that Functional.hello_world does not need BENCHMARK_DBENCH_NPROCS
- How can I tell if any test needs variables from another spec?
- it's referenced in their base script?
- any required spec should be listed in the testplan!
- an empty testplan only supports variables for that test?
- an empty testplan doesn't support any spec variables
- see if default specs are needed by unrelated tests:
- support empty testplan
- parse testplan from test directory
- support empty test spec
- parse test specs from test directory
- use yaml instead of json for plans, specs and overlays
notes [edit section]
the overlay format is really just a shell script, with extensions for- inherit
- include
Are these really needed for the type of object orientation that ovgen supports?
ovgen features [edit section]
- parsing of 'fuegoclass' files (base shell script class?)
- parsing of .dist files (overlay file = shell script class extension files)
- parsing of .board files (overlay file = shell script class extension files)
The model is that ovgen reads the dist, board, plan and spec files and produces the prolog file.
questions [edit section]
- is the prolog file intended to be persistent?
- is it present after a test run? - yes
- why? for reference or for reuse?
- fact: it is overwritten when a new testplan is used
- fact: it is written every time a test is run (see overlays.sh - rm $OF_OUTPUT_FILE)
- it's main purpose seems to be for use by the post_test routine, which may be run in a separate invocation from the rest of the base script
- is it ever re-used after the test completes?
- does ovgen or any other thing read it or examine it?
- is it present after a test run? - yes
- the default arguments for every test are included in the default testplan
- However, only the arguments for a single spec are included in a custom testplan
- see if default specs are needed by unrelated tests:
- I know that Functional.hello_world does not need BENCHMARK_DBENCH_NPROCS
- How can I tell if any test needs variables from another spec?
- it's referenced in their base script?
- any required spec should be listed in the testplan!
- an empty testplan only supports variables for that test?
- an empty testplan doesn't support any spec variables
- do the following in /home/jenkins/fuego/engine/tests:
- find . -type f -print0 | xargs -0 grep BENCHMARK
- find . -type f -print0 | xargs -o grep FUNCTIONAL
- no test vars are used outside their own tests
- Thus, having a global testplan_default is bogus!
- what happens if a test spec is missing?
- ovgen.py aborts - if a test spec is mentioned in the testplan but it's not in the list of specs available, then ovgen.py errors out.
- what happens if a test plan is missing?
- overlays.sh checks for presence of testplan file and aborts the job if not found
- does anyone call ovgen.py besides overlays.sh
- answer: NO
- is there a use case for having all the default specs in the default testplan?
- each test has to run it's own overlay generation, so I don't think so
prolog file structure [edit section]
The prolog filename is '<board_name>-prolog.sh"It has the following sections:
- #class: base-distrib - stuff from the distribution
- #class: base-board - stuff from the board file
- #class: base-params - global/system-wide variables
- #class: base-funcs - global/system-wide functions
- #testplan: default - test variables for the indicated testplan
the new model [edit section]
- individual test shows list of available specs
- modify testplan.groovy
- show 'default' and any parsed spec names from the test dir
- put specs in separate files
- I think this is already supported!
- testplan_mmc is no longer required for individual execution of e.g. Benchmark.bonnie
- testplan lists test.spec tuples
- it already has a list of tuples
- have ftc 'run' a testplan
- job specifies a list of tests and a testplan