Each assignment in CSSE2310 is tested via an automated test system. There are two sets of tests created - one that is publically available to students to test their code while developing their solutions, and another set that is used for the marking. There may be an overlap of tests between these two sets.
Test scripts
For each assignment there will usually be two scripts made available to you for use on moss. Full details of these scripts will be published to the newsgroup, but here are some basics (replace X with the assignment number - 1, 3, 4):
- testaX.sh
- Must be run from the directory that contains your compiled assignment binary
- By default it is using the 'test' option (see below), but can accept other instructions in place of this
- reptestaX.sh
- Makes a checkout of your repository in a new directory
- Compiles your code using the command specified in the spec
- Runs the tests against this compiled program
- Accepts all the same arguments as testaX.sh does
Test Modes
The test script offers two subcommands to make it easier for you to check that your assignment solution is doing what you want. As with most unix programs, you can get help on what the script does by using the -h option.
$ testaX.sh -h
usage: grum.py [-h] [--version] {test,explain,update,mark} ...
positional arguments:
{test,explain,update,mark}
test Run test(s)
explain Explain what tests run and check
update Update files associated with test(s)
mark Run tests and calculate marks
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
To get you started with the test script, here is an explanation of the modes available.
test Mode
This mode will run the tests against your code. You can use it to run all the tests (default) or a subset of the tests.
When a test is run, your program is started with some command line arguments and then sent pre-generated input. The output that your program generates is then compared with the expected output. If this matches and your program exits with the expected code, you will pass the test. If something goes wrong, that test will be recorded as having failed.
# Run all of the tests
$ testaX.sh test
# Run all tests within a class
$ testaX.sh test Assignment
# Run a single test
$ testaX.sh test Assignment.test1
# Run a specific set of tests
$ testaX.sh test Assignment.test2 Assignment.test5
When you run tests, you can also specify some options to give you more information.
Verbose output: -v
Often you will want to see why a test failed by checking how your output is different to the expected output. You can see this by using the -v option when running tests. If your test fails with "stdout mismatch" or "stderr mismatch", it will print out a diff (truncated at the point of failure) that you can then use to find the problem.
$ testaX.sh test -v
$ testaX.sh test -v Assignment
$ testaX.sh test -v Assignment.test1
Save output: -s
When you use this option, a new directory is created to store the stderr and stdout output from your program. Instead of reading the output, checking it and then discarding it (as normally happens), the test system will save the output of your program to files so you can look at it later. This is particularly useful when combined with the explain mode.
$ testaX.sh test -s
$ testaX.sh test -s Assignment
$ testaX.sh test -s Assignment.test1
You can also combine the save and verbose options to see a full diff from a test that fails due to output mismatch.
$ testaX.sh test -s -v
$ testaX.sh test -s -v Assignment
$ testaX.sh test -s -v Assignment.test1
explain Mode
The explain subcommand will NOT test your code. Instead, it will tell you what a particular test is doing when it runs, which will make it easier for you to replicate the test by yourself outside the system. It is used in much the same way as the test mode, allowing you to explain all tests or a subset of them.
$ testaX.sh explain
$ testaX.sh explain Assignment
$ testaX.sh explain Assignment.test1
$ testaX.sh explain Assignment.test14 Assignment.test2 Assignment.test99