Caca Labs test suite

All our major projects feature an extensive test suite that checks the software for consistency, source code quality, features and regressions.

The testsuite can be run by typing make check in the project’s top directory.

Source code tests

Most source code checks rely on the project’s ability to list all source files. This is done by putting the following Makefile.am snippet in all directories that contain source code:

echo-sources: ; echo $(SOURCES)

... and the following snippet in the top directory Makefile.am (example taken from libcaca):

echo-dirs: ; echo caca kernel src cxx examples ruby tools

check-source performs a source code quality check. It checks for the following coding style errors:

• no source code file should contains tabs
• no source code file should contain trailing spaces

check-copyright performs a copyright statement check. The following tests are performed:

• all source code files must contain a “Copyright (c)” statement
• all source code files must mention the year of the last modification (this check is only performed when in a git-svn repository)

check-win32 performs a Win32 port consistency check. The following tests are performed:

• all preprocessor tokens in config.h.in must appear in msvc/config.h

Documentation tests

check-doxygen checks for problems in the Doxygen-built documentation:

• no warnings should appear in the Doxygen build log

libcaca-specific test suite

libcaca uses CppUnit for its unit tests. Coverage is pretty low, but increases with time. The following features are already tested:

• canvas creation, resizing, and basic character blitting
• dirty rectangle features
• driver list
• “caca” format import/export

zzuf-specific test suite

zzuf uses several custom scripts to test the program’s features.

check-div0 runs a misbehaving executable doing a division by zero when its input is corrupted, and checks that zzuf properly catches the failure.

check-overflow runs a misbehaving executable accessing an unallocated memory area when its input is corrupted, and checks that zzuf properly catches the failure.

check-utils is the most extensive zzuf test. It runs various system utilities and zzat programs with known 64kB files and checks that the input of all programs is fuzzed in exactly the same way. System utilities include the following:

• dd with different block sizes
• grep -a ''
• sed -e n
• tail -n +1

zzat programs include the following:

• repeat(-1,fread(1,1000),feof(1))
• repeat(-1,getc(),feof(10))
• fread(1,33000) rewind() repeat(-1,fseek(1,SEEK_CUR),fread(1,1),feof(1))
• repeat(8000,getc_unlocked()) fread(1,33000)

check-zzuf-A-autoinc is a check for zzuf -A (--autoinc): it runs zzat with two files, and checks that the second file read is fuzzed differently with -A.

check-zzuf-f-fuzzing is a check for zzuf -f (--fuzzing): it checks that xor always changes the input file, that set changes all files except when the input already has all bits set, and so on.

check-zzuf-M-max-memory is a check for zzuf -M (--max-memory): it runs a misbehaving executable allocating enormous massive chunks of memory when its input is corrupted, and checks that zzuf properly catches the failure.

check-zzuf-m-md5 is a check for zzuf -m (--md5): it checks that zzuf properly computes known MD5 checksums such as d41d8cd98f00b204e9800998ecf8427e (the empty string) or 5eb63bbbe01eeed093cb22bb8f5acdc3 (hello world).

check-zzuf-r-ratio is a check for zzuf -r (--ratio): it checks for consistency with the expected average number of flipped bits with a given ratio, n/2*R^{(1 - 2/n).}

check-zzuf-s-seed is a check for zzuf -s (--seed): it runs zzuf up to 200000 times on the same file and checks that the seed distribution causes each bit to be flipped at least once.