|
Here is some information about testing started on August 24, 2017 for the 1.2release:
|
{{TableOfContents}}Here is some information about testing started on August 24, 2017 for the 1.2release:
|
|
.(make room for TOC on right)
|
.(make room for TOC on right)
|
|
.
|
.
|
|
.
|
.
|
|
.
|
.
|
|
.
|
.
|
|
.
|
.
|
|
.
|
.
|
|
|
= Tests conducted = - min1.testplan_ltsi{{{ Benchmark.Dhrystone 2017-08-24_18:09:14 timdesk:min1 PASS Benchmark.IOzone 2017-08-24_19:20:19 timdesk:min1 PASS Benchmark.Java 2017-08-24_18:10:37 timdesk:min1 UNFINISHED Benchmark.Whetstone 2017-08-24_18:07:07 timdesk:min1 PASS Benchmark.bonnie 2017-08-24_19:20:50 timdesk:min1 UNFINISHED Benchmark.dbench 2017-08-24_19:21:09 timdesk:min1 PASS Benchmark.ffsb 2017-08-24_18:07:59 timdesk:min1 PASS Benchmark.fio 2017-08-24_18:10:49 timdesk:min1 PASS Benchmark.hackbench 2017-08-24_18:07:39 timdesk:min1 PASS Benchmark.linpack 2017-08-24_18:16:56 timdesk:min1 PASS Benchmark.netperf 2017-08-24_18:31:37 timdesk:min1 PASS Benchmark.tiobench 2017-08-24_18:05:06 timdesk:min1 PASS Functional.LTP 2017-08-24_18:32:37 timdesk:min1 PASS Functional.aiostress 2017-08-24_18:17:35 timdesk:min1 UNFINISHED Functional.bc 2017-08-24_18:10:14 timdesk:min1 PASS Functional.bc 2017-08-24_18:17:54 timdesk:min1 PASS Functional.bzip2 2017-08-24_18:18:16 timdesk:min1 PASS Functional.crashme 2017-08-24_18:05:52 timdesk:min1 PASS Functional.expat 2017-08-24_18:18:35 timdesk:min1 UNFINISHED Functional.ft2demos 2017-08-24_18:19:05 timdesk:min1 PASS Functional.glib 2017-08-24_18:20:11 timdesk:min1 UNFINISHED Functional.hello_world 2017-08-24_18:14:26 timdesk:bbb PASS Functional.hello_world 2017-08-24_18:04:41 timdesk:min1 PASS Functional.hello_world 2017-08-24_18:06:49 timdesk:min1 PASS Functional.ipv6connect 2017-08-24_18:08:53 timdesk:min1 PASS Functional.jpeg 2017-08-24_18:06:22 timdesk:min1 PASS Functional.kernel_build 2017-08-24_18:22:57 timdesk:min1 PASS Functional.linus_stress 2017-08-24_18:27:33 timdesk:min1 PASS Functional.netperf 2017-08-24_19:18:06 timdesk:min1 PASS Functional.rmaptest 2017-08-24_18:28:33 timdesk:min1 PASS Functional.stress 2017-08-24_18:29:05 timdesk:min1 PASS Functional.synctest 2017-08-24_18:30:31 timdesk:min1 PASS Functional.zlib 2017-08-24_18:31:15 timdesk:min1 PASS }}}
|
|
|
= Issues found = - testplan_ltsi - uses noroot spec for bonnie - is probably wrong - tests - min1.default.Benchmark.Java - aborted - error message in console log is not human friendly - error is an abort from assert_define, from fuego_test.sh:test_pre_check() - this is not using need_check - error is not put into run.json file - min1.default.Functional.aiostress - failed - test execution error - error loading shared libraries: libaio.so.1 - min1.default.Functional.expat - failed - build failure - directory (empty) was left on target - Warning: deprecated conversion from string constant to 'char*' [-Wwrite-strings] - youtube says this can be fixed with a line like this: - #pragma GCC diagnostic ignored "-Wwrite-strings" - min1.default.Functional.glib - failed - runtime error with glib test - execution of bash script returns exit code 143 - FUEGO has a bug handling the return code: - TRB: RETURN_VALUE='' - generic_parser.py, line 19 invalid literal for int() with base 10: '' - value = "PASS" if int(RETURN_VAULE) == 0 else "FAIL" - result of "report" is not being handed to generic_parser.py - min1.default.Functional.kernel_build - failed - fuego_test.sh uses 'report "cat build.log"', but the build.log is generated on the host, not the target. You can't 'report' that. - need to add build.log to target-side test log, if that's what's wanted - it's not using the right kernel config - min1.docker.Functional.LTP - takes a very long time (45 minutes) - build duration was 584 seconds (~10 minutes) - subtest fork13 takes 337 seconds (~5 minutes) - subtest getrandom02 takes 296 seconds (~4 minutes) - min1.noroot.Benchmark.bonnie failed - Fuego uses root account on min1 - should make bonnie recognize situation and run correctly - min1.default.Functional.LTP - min1.selectionwithrt.Functional.LTP - run.json (4.4) does not have posix results - run.json (4.4) does not have realtime results - status of 'syscalls' testset is PASS, even though there are failures in individual testcases (e.g. sysctl01) - min1.default.Benchmark.IOZone - there's no run.json link in the Jenkins interface - there are lots of HTML tables with SKIP in the measure area (not just the status area) - The test doesn't actually perform anything but: - 2048_Kb_Read_Write.ReWrite - 2048_Kb_Read_Write.Write - that's odd, are the parameters to iozone correct for the default spec?
|
- Jenkins interface - results on Jenkins job page: - HTML table - test_plan is always "none" - start_time is in seconds since the epoch (not human friendly) - duration is in milliseconds (not human-friendly) - measures are reported as PASS instead of valid value - see Dhrystone Score, or linpack.linpack - ftc gen-report gets this right - Functional.LTP has too much data!! - plots for benchmarks - are completely missing - ftc - gen-report - columns are messy (columns don't auto-adjust to long field entries) - ftc gen-report --where test=.*LTP - tguid:status for syscalls are all None (grep fork13) - run.json has correct status (PASS) - general - failures are very difficult to diagnose - some tests (like LTP) hid the test output - interpreting the console log is very difficult
|
- Jenkins interface - results on Jenkins job page: - HTML table - test_plan is always "none" - start_time is in seconds since the epoch (not human friendly) - duration is in milliseconds (not human-friendly) - measures are reported as PASS instead of valid value - see Dhrystone Score, or linpack.linpack - ftc gen-report gets this right - Functional.LTP has too much data!! - plots for benchmarks - are completely missing - ftc - gen-report - columns are messy (columns don't auto-adjust to long field entries) - ftc gen-report --where test=.*LTP - tguid:status for syscalls are all None (grep fork13) - run.json has correct status (PASS) - general - failures are very difficult to diagnose - some tests (like LTP) hid the test output - interpreting the console log is very difficult
|
|
|
== issues table =={{{#!Table:test1show_edit_links=0show_sort_links=0||Test||problem||notes||||Benchmark.Java||abort not recorded||should put abort message in run.json file||||Functional.aiostress||missing libaio.so.1||is a toolchain/target mismatch||||Functional.expat||build error - warning about const char conversion||Looks like maybe a warning that turned into an error?||||Functional.glib||runtime error - gtester returns failure, and not the expected output||.||||Functional.kernel_build||error in fuego_test.sh - build.log is not on the target||Need to change build.log||||Functional.LTP||slow, result are too big for Jenkins page||.||||Functional.LTP||run.json test_set status is wrong||Should be fail, with missing criteria.json file||||Functional.LTP||run.json missing posix results||.||||Functional.LTP||run.json missing realtime results||.||||Functional.LTP||gen-report shows wrong values for tguids (None instead of PASS/FAIL)||.||||Benchmark.bonnie||noroot spec didn't work||switched to default spec, should handle root or not in test||||Benchmark.Dhrystone||Dhrystone.Score has PASS in Jenkins HTML table||.||||Benchmark.IOzone||only has values for 2 measures||.||}}}
|
|
|
== Functional.glib problem details == - on min1 and docker, gtester returns with exit code 142, which causes Fuego to freak out - fuego_test.sh test_run calls "report" - it returns 143, which is an error - the signal handler is called - fuego_test.sh test_processing is empty (normally it should have a log_compare) - the log_compares in this test don't match the test output - has this test every worked??? - post_test calls: - processing, which calls - test_processing(), which just returns 'true' - then it calls parser.py or generic_parser.py - if there were no errors, then RETURN_VALUE is undefined - on bbb, the build fails, trying to compile gatomic.c - The assembler says: - Error: selected processor does not support Thumb mode 'swp r3,r5,[r4]'
|
|
|
== Functional.LTP problem details ===== run.json: no posix results ===fuego-core/engine/tests/Functional.LTP/parser.py does not even parse posix test results.
|
|
Should look at JTA or Fuego 1.0 and see what Functional.LTP-posix did.Maybe just logcompare?
|
Should look at JTA or Fuego 1.0 and see what Functional.LTP-posix did.Maybe just logcompare?
|
|
|
=== run.json: no realtime results in file ====== run.json: incorrect status for test_set and test_suite ==run.json has overall status of PASS and test_set syscalls has status ofPASS, when some testcases under syscalls failed (and there's nocriteria.json file - the default should be max_fail=0)
|
|
Output from min1.selectionwithrt.Functional.LTP consolelog.txt:
|
Output from min1.selectionwithrt.Functional.LTP consolelog.txt:
|
|
Need to not show parsed_results - it's too big with LTP.
|
Need to not show parsed_results - it's too big with LTP.
|
Warning: missing or faulty criteria.json file, looking for reference.logApplying criterion {'max_fail': 0, 'tguid': 'syscalls'} ^^^^^ this should result in FAIL ^^^^^Applying criterion {'max_fail': 0, 'tguid': 'mm'}Applying criterion {'max_fail': 0, 'tguid': 'timers'}Applying criterion {'max_fail': 0, 'tguid': 'fs'}Applying criterion {'max_fail': 0, 'tguid': 'pty'}Applying criterion {'max_fail': 0, 'tguid': 'pipes'}Applying criterion {'max_fail': 0, 'tguid': 'sched'}Applying criterion {'max_fail': 0, 'tguid': 'dio'}Applying criterion {'max_fail': 0, 'tguid': 'ipc'}Applying criterion {'max_fail': 0, 'tguid': 'Functional.LTP'}Writing run data to /fuego-rw/logs/Functional.LTP/min1.selectionwithrt.4.4/run.jsonreference.json not available}}}
|
{{{Warning: missing or faulty criteria.json file, looking for reference.logApplying criterion {'max_fail': 0, 'tguid': 'syscalls'} ^^^^^ this should result in FAIL ^^^^^Applying criterion {'max_fail': 0, 'tguid': 'mm'}Applying criterion {'max_fail': 0, 'tguid': 'timers'}Applying criterion {'max_fail': 0, 'tguid': 'fs'}Applying criterion {'max_fail': 0, 'tguid': 'pty'}Applying criterion {'max_fail': 0, 'tguid': 'pipes'}Applying criterion {'max_fail': 0, 'tguid': 'sched'}Applying criterion {'max_fail': 0, 'tguid': 'dio'}Applying criterion {'max_fail': 0, 'tguid': 'ipc'}Applying criterion {'max_fail': 0, 'tguid': 'Functional.LTP'}Writing run data to /fuego-rw/logs/Functional.LTP/min1.selectionwithrt.4.4/run.jsonreference.json not available}}}
|
|
|
= Things to check for = - failures are easy to diagnose - no, it requires a ton of Fuego knowledge - I can skip LTP tests that cause problems - I can save results to establish new baseline for benchmark thresholds - save as criteria file?
|
|
|
= Ideas for fixes = - Benchmark.Java should put abort message into run.json file - for ftc gen-report column sloppiness: - do two passes over data to get field lengths for column widths - need to collect all data first, and run over it twice - Benchmark.bonnie should autodetect if running as root, and use correct arguments to bonnie++ program - should add "duration_hint" to spec.json file, so each spec can specify a hint - idea is that this number could be multiplied by a board variable (timeout_factor?) to yield an approximate timeout for the test with this spec. The number in the testplan would override this guess, but this would be better than nothing.
|
|
|
= issues fixed = * ftc list-runs does not process '!=' where operator * ftc list-runs help does not document '!=' where operator * ftc gen-report help does not document '!=' where operator * pass smaller number to fork13 command line for LTP time reduction
|