Report Generator

Fuego includes a report generator.

Here are some details about this system.

outline [edit section]

batch jobs can create 'logrun' files for use in generating reports
- logrun files are in json format and contain results from multiple tests
during execution, results are saved to the logrun file
a special Jenkins job can create reports (Report.make_pdf)
reports.sh has shell functions used to generate reports
reports are only created for batch jobs (it appears)
external tools are used to generate the log:
- loggen.py
  - creates logrun files, and appends data to them
  - also, outputs the login in report xml format
- gentexml.py
  - converts logrun file to report xml format
- texml
  - converts report xml format to report .tex format
- pdflatex
  - converts report .tex format to pdf

loggen.py and gentexml.py are Fuego programs. texml and pdflatex are generic programs for managing documents.

reports generation [edit section]

PDF reports can be generated from accumulated logrun results, by invoking a special Jenkins job: Reports.make_pdf
- this job has the command sequence:

See gen_report

file formats [edit section]

logrun file format [edit section]

There are two kinds of logrun entries: Functional and Benchmark.

A logrun file is in json format, with top level fields of:

device
testplan
runLogs - a list of entries of the form:
- testName, logFile
- testName, testResult
runNumber

logrun entries [edit section]

Each logrun entry is a dictionary containing two fields. For a Benchmark the second field is the logFile. For a Functional test, the second field is the testResult.

logFile is the filename containing benchmark results (xxx.res.json file)
testResult is a string with the overall result for a functional test
- the value comes from the log_compare function and is one of the following:
  - "test error"
  - "passed"
  - "failed"

res.json file format [edit section]

This file is produced for Benchmarks only. It is in json format, and contains a single object with a set of pairs

Here is an example res.json for Benchmark.Dhrystone:

{
    "Dhrystone": "12500000.0"
}

Here is the res.json for Benchmark.ffsb. (It's a bit more complicated than most others)

{
    "Main.append_fsync.TPercent": "1.393",
    "Main.create_fsync.TPercent": "22.452",
    "Main.delete.OpWeight": "9.302",
    "Main.create_fsync.Transactions": "15477",
    "Syscall_latency.write.Min": "0.000000",
    "Syscall_latency.open.Max": "55.584999",
    "Main.open_close.Transactions": "91",
    "Main.create_fsync.OpWeight": "9.725",
    "Main.stat.TPS": "8.11",
    "Main.delete.TPercent": "0.128",
    "Main.open_close.TPS": "8.03",
    "Syscall_latency.close.Avg": "0.002518",
    "Main.delete.TPS": "7.76",
    "Main.append.TPS": "89.08",
    "Main.append.Transactions": "1010",
    "Main.writeall.OpWeight": "10.888",
    "Main.writeall.Transactions": "13750",
    "Syscall_latency.read.Avg": "0.001175",
    "Main.create.OpWeight": "10.782",
    "Syscall_latency.open.Min": "0.002000",
    "Syscall_latency.unlink.Min": "0.016000",
    "Syscall_latency.write.Avg": "0.002091",
    "Main.writeall_fsync.Transactions": "13976",
    "Main.stat.TPercent": "0.133",
    "Main.stat.OpWeight": "9.725",
    "Main.stat.Transactions": "92",
    "Main.readall.OpWeight": "9.619",
    "Syscall_latency.unlink.Max": "3.539000",
    "Main.writeall.TPS": "1212.78",
    "Main.append.TPercent": "1.465",
    "Syscall_latency.read.Max": "0.440000",
    "Main.create.TPercent": "25.646",
    "Main.create.TPS": "1559.32",
    "Main.create.Transactions": "17679",
    "Syscall_latency.read.Min": "0.000000",
    "Main.readall.Transactions": "5812",
    "Syscall_latency.write.Max": "10.133000",
    "Throughput.Read": "2000.00",
    "Syscall_latency.unlink.Avg": "0.188614",
    "Main.writeall_fsync.OpWeight": "9.514",
    "Main.open_close.OpWeight": "9.619",
    "Syscall_latency.stat.Avg": "0.005761",
    "Throughput.Write": "21700.00",
    "Syscall_latency.open.Avg": "0.172316",
    "Main.writeall_fsync.TPS": "1232.71",
    "Syscall_latency.close.Max": "0.014000",
    "Main.append.OpWeight": "10.677",
    "Syscall_latency.stat.Max": "0.019000",
    "Syscall_latency.close.Min": "0.000000",
    "Main.readall.TPercent": "8.431",
    "Main.readall.TPS": "512.63",
    "Syscall_latency.stat.Min": "0.002000",
    "Main.writeall_fsync.TPercent": "20.274",
    "Main.delete.Transactions": "88",
    "Main.writeall.TPercent": "19.946",
    "Main.create_fsync.TPS": "1365.10",
    "Main.append_fsync.OpWeight": "10.148",
    "Main.open_close.TPercent": "0.132",
    "Main.append_fsync.Transactions": "960",
    "Main.append_fsync.TPS": "84.67"
}