Contents: to do information testplan file format spec file format Intended features notes ovgen features questions prolog file structure the new model

This page has some miscellaneous notes about ovgen, how it is used, and how it might be modified in the future:

ovgen is used to construct a 'prolog' file, which is a shell script which has variables and definitions from the distro and board files, to control the execution of a test. The prolog script is generated when the base script is executed, and is sourced at some point during test execution.

Note that test execution is done in two separate phases. That is, Jenkins executes the test script itself to do the first phases of a test (pre_test, build, deploy, run), then does a "source functions.sh ; post_test" to execute the 'post_test' phase of the test.

to do [edit section]

• + add --testdir <arg> to ovgen command line
• + if testdir is specified, then read specs and plan from there
  • + read spec file from test directory
    • + If not present, read from overlays/test_specs
  • + read test plan from test directory
    • + only if it's present
• + add quotes to testspec variables
• + move ovgen.py into 'scripts' directory
• + support variable definitions without quotes around values
  • + modify parser to read overlay variables without quotes
• +if testplan name is testplan_default, then just assume each test spec has: { testName: <test_name> and spec: default }
  • test this with moved specs and plans
• + move all specs to their respective test directories
  • figure out what to do about missing specs: *

    missing spec in Benchmark.aim7
    missing spec in Benchmark.bfoo
    missing spec in Benchmark.fs_mark
    missing spec in Benchmark.nbench-byte
    missing spec in Benchmark.sysbench
    missing spec in Functional.arch_timer
    missing spec in Functional.cmt
    missing spec in Functional.fsfuzz
    missing spec in Functional.mesa-demos
    missing spec in Functional.scifab
    missing spec in Functional.sdhi_0
    

• move all plans to their respective test directories
  • bc
  • hello
  • sata, mmc, usb
    • tests are: Benchmark.fio, Benchmark.Bonnie, Benchmark.IOzone, Benchmakr.ffsb, Benchmark.Tiobench, Benchmark.aiostress, Functional.synctest, Benchmakr.Interbench, Benchmark.dbench

• specify per-test testplans in the Jenkins interface
  • have groovy script parse the testdir for testplan names
• + if spec is 'default', and no test.spec file is found, then just synthesize one with the following:

{
    "testName": "Functional.zlib",
    "specs":
    [
        {
            "name":"default"
        }
    ]
}

• read testplans from overlays/testplans and testdir

information [edit section]

• Functional.bzip2.spec has no variables (no specs besides default)
• Functional.zlib.spec has no variables (no specs besides default)
• Functional.hello_world.spec has different specs

testplan file format [edit section]

testplan<name>.json file has:

• testPlanName
• tests:
  • testName
  • spec

spec file format [edit section]

Functional.<test_name>.json file has:

• testName:
• specs:
  • name
  • var1
  • var2 ...

Intended features [edit section]

• avoid definining test variables for tests unrelated to this one.
  • see if default specs are needed by unrelated tests:
    • I know that Functional.hello_world does not need BENCHMARK_DBENCH_NPROCS
    • How can I tell if any test needs variables from another spec?
      • it's referenced in their base script?
      • any required spec should be listed in the testplan!
    • an empty testplan only supports variables for that test?
      • an empty testplan doesn't support any spec variables

• support empty testplan
• parse testplan from test directory
• support empty test spec
• parse test specs from test directory

• use yaml instead of json for plans, specs and overlays

notes [edit section]

the overlay format is really just a shell script, with extensions for

• inherit
• include

Are these really needed for the type of object orientation that ovgen supports?

ovgen features [edit section]

• parsing of 'fuegoclass' files (base shell script class?)
• parsing of .dist files (overlay file = shell script class extension files)
• parsing of .board files (overlay file = shell script class extension files)

The model is that ovgen reads the dist, board, plan and spec files and produces the prolog file.

questions [edit section]

• is the prolog file intended to be persistent?
  • is it present after a test run? - yes
    • why? for reference or for reuse?
    • fact: it is overwritten when a new testplan is used
    • fact: it is written every time a test is run (see overlays.sh - rm $OF_OUTPUT_FILE)
  • it's main purpose seems to be for use by the post_test routine, which may be run in a separate invocation from the rest of the base script
  • is it ever re-used after the test completes?
    • does ovgen or any other thing read it or examine it?
• the default arguments for every test are included in the default testplan
  • However, only the arguments for a single spec are included in a custom testplan

• see if default specs are needed by unrelated tests:
  • I know that Functional.hello_world does not need BENCHMARK_DBENCH_NPROCS
  • How can I tell if any test needs variables from another spec?
    • it's referenced in their base script?
    • any required spec should be listed in the testplan!
  • an empty testplan only supports variables for that test?
    • an empty testplan doesn't support any spec variables
• do the following in /home/jenkins/fuego/engine/tests:
  • find . -type f -print0 | xargs -0 grep BENCHMARK
  • find . -type f -print0 | xargs -o grep FUNCTIONAL
• no test vars are used outside their own tests
  • Thus, having a global testplan_default is bogus!
• what happens if a test spec is missing?
  • ovgen.py aborts - if a test spec is mentioned in the testplan but it's not in the list of specs available, then ovgen.py errors out.
• what happens if a test plan is missing?
  • overlays.sh checks for presence of testplan file and aborts the job if not found
• does anyone call ovgen.py besides overlays.sh
  • answer: NO
• is there a use case for having all the default specs in the default testplan?
  • each test has to run it's own overlay generation, so I don't think so

prolog file structure [edit section]

The prolog filename is '<board_name>-prolog.sh"

It has the following sections:

• #class: base-distrib - stuff from the distribution
• #class: base-board - stuff from the board file
• #class: base-params - global/system-wide variables
• #class: base-funcs - global/system-wide functions
• #testplan: default - test variables for the indicated testplan

the new model [edit section]

• individual test shows list of available specs
  • modify testplan.groovy
  • show 'default' and any parsed spec names from the test dir
  • put specs in separate files
    • I think this is already supported!
  • testplan_mmc is no longer required for individual execution of e.g. Benchmark.bonnie

• testplan lists test.spec tuples
  • it already has a list of tuples
• have ftc 'run' a testplan
• job specifies a list of tests and a testplan