|
This page describes the log_compare function:= Description =log_compare is called from a test script's test_processing() function toscan the test's logfile and indicate whether the test passed or failed.
|
This page describes the log_compare function:= Description =log_compare is called from a test script's test_processing() function toscan the test's logfile and indicate whether the test passed or failed.
|
|
It can be used several times in a test_processing() function, to checkdifferent categories of test results. This includes checking for boththe number of positive results (sub-tests passed) or negative results(sub-tests failed).
|
It can be used several times in a test_processing() function, to checkdifferent categories of test results. This includes checking for boththe number of positive results (sub-tests passed) or negative results(sub-tests failed).
|
|
|
= Arguments = Position arguments: * $1 = reflog_prefix = prefix of reference parsed log filename * $2 = match_count = number of matching results * $3 = results_expression * $4 = results_category (either 'p', 'n', or something else)
|
|
The first argument is a string used in the filename of the reference parsed log. Internally this is used as part of a filename, but externally, all callers pass in $TESTDIR. In the case of Functional.bzip2, the stringpassed in is "Functional.bzip2", so it appears to match the test name.However, in Functional.posixtestsuite, the prefix passed in is:"posixtestsuite-1.5.2", which matches the first part of the name of thetarball containing the source for the test suite. It appears thiscan be used to distinguish between multiple versions of the sametest. The reference logs appear to be placed in the test area, alongsidethe test script.
|
The first argument is a string used in the filename of the reference parsed log. Internally this is used as part of a filename, but externally, all callers pass in $TESTDIR. In the case of Functional.bzip2, the stringpassed in is "Functional.bzip2", so it appears to match the test name.However, in Functional.posixtestsuite, the prefix passed in is:"posixtestsuite-1.5.2", which matches the first part of the name of thetarball containing the source for the test suite. It appears thiscan be used to distinguish between multiple versions of the sametest. The reference logs appear to be placed in the test area, alongsidethe test script.
|
|
Hence, the posixtestsuite reference parsed logs are at: * /home/jenkins/fuego/engine/tests/Functional.posixtestsuite/posixtestsuite-1.5.2_p.log * /home/jenkins/fuego/engine/tests/Functional.posixtestsuite/posixtestsuite-1.5.2_n.log
|
Hence, the posixtestsuite reference parsed logs are at: * /home/jenkins/fuego/engine/tests/Functional.posixtestsuite/posixtestsuite-1.5.2_p.log * /home/jenkins/fuego/engine/tests/Functional.posixtestsuite/posixtestsuite-1.5.2_n.log
|
|
The second argument is the match_count, specifying the number of linesin the log file that should match the results_expression (described next).In some tests, the match_count is specified via an environment variablesuch as: $LTP_OPEN_POSIX_SUBTEST_COUNT_POS. For extremely complex tests that have board-dependent or platform-dependent results it is usefulto allow the match_count to be specified in the board file.
|
The second argument is the match_count, specifying the number of linesin the log file that should match the results_expression (described next).In some tests, the match_count is specified via an environment variablesuch as: $LTP_OPEN_POSIX_SUBTEST_COUNT_POS. For extremely complex tests that have board-dependent or platform-dependent results it is usefulto allow the match_count to be specified in the board file.
|
|
The third argument is a regular expressionx tosearch for in the log file. Internally this is referred to as the "Criteria",but I prefer to call it the results_expression. This is passed as anexpression to 'grep -E', so the syntax used by that external command applieshere. See the grep man page for details.
|
The third argument is a regular expressionx tosearch for in the log file. Internally this is referred to as the "Criteria",but I prefer to call it the results_expression. This is passed as anexpression to 'grep -E', so the syntax used by that external command applieshere. See [[http://www.gnu.org/software/grep/manual/grep.html#Regular-Expressions|the grep man page]] for details.
|
|
The results_expression is searched for in the logfile for this run.The number of results is the number of lines in the logfile that matchthat regular expression.
|
The results_expression is searched for in the logfile for this run.The number of results is the number of lines in the logfile that matchthat regular expression.
|
|
A "parsed" logfile is created, wich consists of the original test logfilefiltered through 'grep -E ${results_expression}'.
|
A "parsed" logfile is created, wich consists of the original test logfilefiltered through 'grep -E ${results_expression}'.
|
|
This parsed logfile is tested against a reference log for this test,to see if it matches. If it does not match, then an error is logged.
|
This parsed logfile is tested against a reference log for this test,to see if it matches. If it does not match, then an error is logged.
|
|
Two checks are made: * the number of lines in the original logfile that match the results_expression * whether the lines in the parsed logfile file exactly match the lines in a reference parsed logfile, for a specific category
|
Two checks are made: * the number of lines in the original logfile that match the results_expression * whether the lines in the parsed logfile file exactly match the lines in a reference parsed logfile, for a specific category
|
|
The final parameter to the function is the "result category". This is usuallyeither 'p' or 'n', for positive results and negative results. But it can beany string. The posixtestsuite uses 'unr' for items reported as unresolvedthe test results log.
|
The final parameter to the function is the "result category". This is usuallyeither 'p' or 'n', for positive results and negative results. But it can beany string. The posixtestsuite uses 'unr' for items reported as unresolvedthe test results log.
|
|
The result_category is used as part of the name of the reference parsed filename. Indeed, the name used for comparisons is:${WORKSPACE}/../ref_logs/${JOB_NAME}/${reflog_prefix}_${results_category}.log
|
The result_category is used as part of the name of the reference parsed filename. Indeed, the name used for comparisons is:${WORKSPACE}/../ref_logs/${JOB_NAME}/${reflog_prefix}_${results_category}.log
|
|
After determining the status of the results, the function: * check_create_functional_logrunis called, to record the status of the test.
|
After determining the status of the results, the function: * check_create_functional_logrunis called, to record the status of the test.
|
|
|
= Examples =From the bzip2.sh test script: * log_compare "$TESTDIR" "11" "^TEST.*OK" "p" * log_compare "$TESTDIR" "0" "^TEST.*FAILED" "n"
|
|
This basically says that 11 lines in the log should start with TEST and haveOK in them, and 0 lines in the log should start with TEST and have FAILED in them.
|
This basically says that 11 lines in the log should start with TEST and haveOK in them, and 0 lines in the log should start with TEST and have FAILED in them.
|
|
From the posixtestsuite.sh{{{ PASSED="PASSED: 922" FAILED="FAILED: 69" UNRESOLVED="UNRESOLVED: 7" UNSUPPORTED="UNSUPPORTED: 93" UNTESTED="UNTESTED: 66"
|
From the posixtestsuite.sh{{{ PASSED="PASSED: 922" FAILED="FAILED: 69" UNRESOLVED="UNRESOLVED: 7" UNSUPPORTED="UNSUPPORTED: 93" UNTESTED="UNTESTED: 66"
|
|
log_compare "$TESTDIR" "1" "${PASSED}" "p" log_compare "$TESTDIR" "1" "${FAILED}" "f" log_compare "$TESTDIR" "1" "${UNRESOLVED}" "unr" log_compare "$TESTDIR" "1" "${UNSUPPORTED}" "uns" log_compare "$TESTDIR" "1" "${UNTESTED}" "unt"}}}
|
log_compare "$TESTDIR" "1" "${PASSED}" "p" log_compare "$TESTDIR" "1" "${FAILED}" "f" log_compare "$TESTDIR" "1" "${UNRESOLVED}" "unr" log_compare "$TESTDIR" "1" "${UNSUPPORTED}" "uns" log_compare "$TESTDIR" "1" "${UNTESTED}" "unt"}}}
|
|
Note that this test sums the number of results in different categories,so the match_counts are always 1, and the results_expression for eachcategory includes the number of results expected.
|
Note that this test sums the number of results in different categories,so the match_counts are always 1, and the results_expression for eachcategory includes the number of results expected.
|
|
|
= Return value =A return value of 'true' indicates that the test succeeded.
|
|
A return value of 'false' indicates that the test failed.
|
A return value of 'false' indicates that the test failed.
|
|
If multiple log_compares are called in sequence, only the return valueof the last one is used as the return value for the callingfunction (test_processing).
|
If multiple log_compares are called in sequence, only the return valueof the last one is used as the return value for the callingfunction (test_processing).
|
|
|
= Notes = * this call should be made inside the "test_processing" functions of Functional tests. * how is this function called? * see the examples * what does check_create_functional_logrun do? * called with one of: "test error", "passed", "failed" * check_create_functional_logrun only works if BATCH_TESTPLAN is set * how is success or failure communicated to Jenkins for their icon? * it appears to be entirely based on the return code from the last call to log_compare * which becomes the return code for test_processing(), and "source functional.sh" and the last value returned by the test script itself. * where is the final result stored in Jenkins? * I don't know
|
|
|
== Location ==This function is located in /home/jenkins/scripts/functions.sh.It is called by about 2/3rds of the functional tests, at the time of this writing.
|
|
|
== dead functionality ==There appears to be an intended feature that checks the parsed logfileagainst a reference parsed logfile (either positive or negative). If theresult of the test does not match exactly the reference parsed log, thenan error is emitted with the difference, with the string: * "Fuego error reason: Unexpected test log output:\n$TMP\n"
|
|
The path for the reference parsed logfile used in the current systemdoesn't match any files. Howeverreference_logs are defined in each test directory. There's currently abug in the shell fragment that does this comparison, so that it alwaysyields a result of 0 (or success), so the test is completely meaningless.
|
The path for the reference parsed logfile used in the current systemdoesn't match any files. Howeverreference_logs are defined in each test directory. There's currently abug in the shell fragment that does this comparison, so that it alwaysyields a result of 0 (or success), so the test is completely meaningless.
|
|
This is the only place in the routine that uses the first positional parameter($1), and it's part of the reference parsed log filename. So I highly doubtthat the meaning for parameter $1 of "tarball template" is correct.
|
This is the only place in the routine that uses the first positional parameter($1), and it's part of the reference parsed log filename. So I highly doubtthat the meaning for parameter $1 of "tarball template" is correct.
|
|
|
== creating reference parsed log files ==FIXTHIS - should document how to create parsed reference log files in the system, or make a tool to do so.
|
|
Maybe you just copy the parsed log file, creating thefilename with "${reflog_prefix}_p.log" or "{reflog_prefix}_n.log".That's easy to do manually.
|
Maybe you just copy the parsed log file, creating thefilename with "${reflog_prefix}_p.log" or "{reflog_prefix}_n.log".That's easy to do manually.
|
|
|
== fragility of the script ==The call to source functional.sh should be the last thing in atest script. This seems fragile.
|
|
|
== call tree =={{{ <test_name>/config.xml <test_name>/test_script.sh source functional.sh - call to test_processing <test_name>/<test_script>.sh:function test_processing scripts/functions.sh:log_compare scripts/reports.sh:check_create_functional_logrun}}}
|
|
FIXTHIS - the log_compare and function_log_compare pages should be merged
|
FIXTHIS - the log_compare and function_log_compare pages should be merged
|