14. Failgrind: a memory allocation and syscall failure testing tool

Probably the most useful control option is --alloc-toggle. This allows you to specify a function where the global allocation failure switch will be toggled on entry and exit from the function. This provides a straightforward way of isolating Failgrind to only testing the parts of your program that you are interested in. If you use a toggle function, then Failgrind will start with allocation failures disabled. This means if you specify --alloc-toggle=test_suite, then all code prior to test_suite() will run as normal, the the code inside test_suite() will run with allocation failures enabled, and finally all code after test_suite() will run as normal. Global allocation failures can be enabled/disabled at the start of Failgrind using --alloc-fail-atstart, which can be set to yes or no.

The same functionality can be obtained in a more programmatic manner through the use of Failgrind Monitor Commands or Client request reference).

Allocations failures can be controlled based on the size of allocation being made. This is controlled by the --alloc-threshold-high and --alloc-threshold-low options, which specify high and low thresholds in bytes respectively. For example:

All allocations larger than 100 bytes will always succeed:

valgrind --tool=exp-failgrind --alloc-threshold-high=100 ./my_prog

All allocations smaller than 10 bytes will always succeed:

valgrind --tool=exp-failgrind --alloc-threshold-low=10 ./my_prog

All allocations larger than 100 bytes and smaller than 10 bytes will always succeed:

valgrind --tool=exp-failgrind --alloc-threshold-high=100 --alloc-threshold-low=10 ./my_prog

The sense of the thresholds can be inverted with --alloc-threshold-invert=yes. Only really makes sense when specifying both a high and low threshold. For example:

All allocations smaller than 100 bytes will always succeed:

valgrind --tool=exp-failgrind --alloc-threshold-high=100 --alloc-threshold-invert ./my_prog

All allocations larger than 10 bytes will always succeed:

valgrind --tool=exp-failgrind --alloc-threshold-low=10 --alloc-threshold-invert ./my_prog

All allocations greater than 10 bytes and smaller than 100 bytes will always succeed:

valgrind --tool=exp-failgrind --alloc-threshold-high=100 --alloc-threshold-low=10 --alloc-threshold-invert ./my_prog

If you wish to have a memory allocation function always succeed regardless of other options, use the --alloc-allow option. This can be specified multiple times. For example if you are not interested in failing strdup():

valgrind --tool=exp-failgrind --alloc-allow=strdup ./my_prog

The number of allocation failures in a Failgrind run can be limited using the --alloc-max-fails option, which is an integer. It can sometimes be useful to limit to a single allocation per run, at the cost of increasing the number of runs that are needed:

valgrind --tool=exp-failgrind --alloc-max-fails=1 ./my_prog

The --alloc-fail-chance option allows allocation failures to happen randomly. Set to an integer value 1-100 that represents the percentage chance an allocation that is due to be rejected (that hasn't been seen before and meets all other criteria) will actually fail. Use with the --seed to get reproduceable but random tests. When you use --alloc-fail-chance, the seed being used is displayed by Failgrind on start. For example, to fail 50% of the time:

valgrind --tool=exp-failgrind --alloc-fail-chance=50 ./my_prog

==15526== Failgrind, a memory allocation and syscall failure testing tool
==15526== NOTE: This is an Experimental-Class Valgrind Tool
==15526== Copyright (C) 2018-2021, and GNU GPL'd, by Roger Light.
==15526== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==15526== Command: ./my_prog
==15526==
==15526== Using random seed: 7937841
...

A trivial program:

 1      #include <stdio.h>
 2      #include <stdlib.h>
 3
 4      void func_A(void)
 5      {
 6          int *ptr1;
 7
 8          ptr1 = malloc(10);
 9          if(!ptr1){
10              return;
11          }
12      }
13
14      int main(int argc, char *argv[])
15      {
16          func_A();
17      }

Running through Failgrind for the first time, and using the --show-failed=yes option produces the following output:

valgrind --tool=exp-failgrind --show-failed=yes ./test
==17460== Failgrind, a memory allocation failure testing tool
==17460== NOTE: This is an Experimental-Class Valgrind Tool
==17460== Copyright (C) 2018-2021, and GNU GPL'd, by Roger Light.
==17460== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==17460== Command: ./test
==17460==
==17460==    at 0x4C2E08B: malloc (vg_replace_malloc.c:299)
==17460==    by 0x10865B: func_A (test.c:8)
==17460==    by 0x10867A: main (test.c:16)
==17460==
==17460==
--17460--  Failgrind: 0 call stacks loaded from file
--17460--             0 allocations succeeded
--17460--             1 allocations failed
--17460--             1 new allocation callstacks found
--17460--             0 syscalls succeeded
--17460--             0 syscalls failed
--17460--             0 new syscall callstacks found
--17460--

This is as expected - the single call to malloc has failed, and the stack trace tells us where it was. We can see that the callstack file was empty because no call stacks were loaded.

Running a second time:

valgrind --tool=exp-failgrind --show-failed=yes ./test
==17461== Failgrind, a memory allocation failure testing tool
==17461== NOTE: This is an Experimental-Class Valgrind Tool
==17461== Copyright (C) 2018-2021, and GNU GPL'd, by Roger Light.
==17461== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==17461== Command: ./test
==17461==
==17461==
--17461--  Failgrind: 1 call stacks loaded from file
--17461--             1 allocations succeeded
--17461--             0 allocations failed
--17461--             0 new allocation callstacks found
--17461--             0 syscalls succeeded
--17461--             0 syscalls failed
--17461--             0 new syscall callstacks found
--17461--

This shows a single call stack was loaded from the callstack file, and that our allocation matches that call stack because the allocation succeeded.

A possible outcome of memory failures is that the program being tested segfaults. The listing below extends our program to add a second memory allocation that isn't checked for success, then attempts to write to that memory.

 1      #include <stdlib.h>
 2
 3      void func_A(void)
 4      {
 5          int *ptr1;
 6          int *ptr2;
 7
 8          ptr1 = malloc(10);
 9          if(!ptr1){
10              return;
11          }
12
13          ptr2 = malloc(10);
14          ptr2[5] = 1;
15      }
16
17      int main(int argc, char *argv[])
18      {
19          func_A();
20      }

Before the testing begins again, the failgrind.callstacks file is removed to give a fresh start with no old callstacks. The first run of the program is identical to the previous first run. Running Failgrind a second time, this time using the default options (i.e. the failed allocation call stacks are not being printed) produces:

==17619== Failgrind, a memory allocation failure testing tool
==17619== NOTE: This is an Experimental-Class Valgrind Tool
==17619== Copyright (C) 2018-2021, and GNU GPL'd, by Roger Light.
==17619== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==17619== Command: ./a.out
==17619==
==17619==
==17619== Process terminating with default action of signal 11 (SIGSEGV)
==17619==  Access not within mapped region at address 0x14
==17619==    at 0x10867D: func_A (test.c:14)
==17619==    by 0x10869B: main (test.c:19)
==17619==  If you believe this happened as a result of a stack
==17619==  overflow in your program's main thread (unlikely but
==17619==  possible), you can try to increase the size of the
==17619==  main thread stack using the --main-stacksize= flag.
==17619==  The main thread stack size used in this run was 8388608.
==17619==
--17619--  Failgrind: 1 call stacks loaded from file
--17619--             1 allocations succeeded
--17619--             1 allocations failed
--17619--             1 new allocation callstacks found
--17619--             0 syscalls succeeded
--17619--             0 syscalls failed
--17619--             0 new syscall callstacks found
--17619--
Segmentation fault (core dumped)

As expected, the write to the invalid pointer from the unchecked allocation causes a crash. We can see where the crash occurred, but that doesn't necessarily tell us where the allocation failure was. In this example, only one allocation failed, so we know this must be the allocation that caused the crash, and it will be the final entry in the callstack file:

# ./test
at 0x4C2E08B: malloc (vg_replace_malloc.c:299)
by 0x108670: func_A (test.c:13)
by 0x10869B: main (test.c:19)

It is possible to repeat a test by removing call stacks from the callstack file. In this case, removing the final call stack effectively restores our testing environment to as it was before the current test.

As well as examining the call stack file to see allocation failures, it is also possible to have callstacks printed to the screen each time a failure occurs, as was seen earlier. This is enabled using --show-failed=yes, and produces an output like below:

valgrind --tool=exp-failgrind --show-failed=yes ./a.out
==17674== Failgrind, a memory allocation failure testing tool
==17674== NOTE: This is an Experimental-Class Valgrind Tool
==17674== Copyright (C) 2018-2021, and GNU GPL'd, by Roger Light.
==17674== Using Valgrind-3.16.1 *and LibVEX; rerun with -h for copyright info
==17674== Command: ./a.out
==17674==
==17674==    at 0x4C2E08B: malloc (vg_replace_malloc.c:299)
==17674==    by 0x108670: func_A (test.c:13)
==17674==    by 0x10869B: main (test.c:19)
...

The most straightforward method of controlling which parts of your program are tested, is through the --alloc-fail-atstart and --alloc-toggle options. By default, as soon as Failgrind starts it will begin rejecting heap memory allocations. Using --alloc-fail-atstart=no, no memory allocations will be rejected until allocation failures are re-enabled. To enable allocation failures you can use a gdb monitor command, a client request, as described in Failgrind Monitor Commands and Client request reference,, or tell Failgrind to enable itself automatically using the --alloc-toggle option. This option, which can be repeated in order to specify multiple functions, causes Failgrind to toggle the enabled/disabled state when a named function is entered and left. For example, running a program prog as follows:

valgrind --tool=exp-failgrind --alloc-toggle=run_test1 --alloc-toggle=run_test2 prog

The program listing:

 1      #include <failgrind.h>
 2      #include <test_funcs.h>
 3
 4      int main(int argc, char* argv[])
 5      {
 6         setup_test_suite();
 7
 8         run_test1();
 9
10         run_test2();
11
12         cleanup_test_suite();
13         return 0;
14      }

When --alloc-toggle is in use, --alloc-fail-atstart defaults to no, so Failgrind starts off disabled. The setup of the test (the setup_test_suite() function) is carried out with no memory allocations being rejected. The --alloc-toggle=run_test1 option means that when run_test1() is entered, the enabled/disabled state will be toggled - so during the execution of this function memory allocation failures are enabled - then after it is left they will be disabled again. The same is true for run_test2().

If you need more control than is possible with the simple toggle option, then you should look into monitor commands or client requests (see Failgrind Monitor Commands or Client request reference) where it is possible to enable/disable Failgrind, reset its state and other capabilities.

Probably the most useful control option is --syscall-toggle. This allows you to specify a function where the global syscall failure switch will be toggled on entry and exit from the function. This provides a straightforward way of isolating Failgrind to only testing the parts of your program that you are interested in. If you specify --syscall-toggle=test_suite, then all code prior to test_suite() will run as normal, the the code inside test_suite() will run with syscall failures enabled, and finally all code after test_suite() will run as normal. Global syscall failures can be enabled/disabled at the start of Failgrind using --syscall-fail-atstart, which can be set to yes or no.

The same functionality can be obtained in a more programmatic manner through the use of Failgrind Monitor Commands or Client request reference).

By default, Failgrind will have all syscalls fail with errno EINVAL. This can be changed globally or per-syscall. Use --syscall-errno=<error> to set the error value to use for all syscalls. Use --syscall-errno=<function>,<error> to set an error to be used for a particular syscall. For example:

All syscalls will fail with EINTR:

valgrind --tool=exp-failgrind --syscall-fail-atstart=yes --syscall-errno=EINTR ./my_prog

All syscalls will fail with EINVAL, except for open(), which will fail with EACCES:

valgrind --tool=exp-failgrind --syscall-fail-atstart=yes --syscall-errno=open,EACCES ./my_prog

All syscalls will fail with EINTR, except for open(), which will fail with EACCES:

valgrind --tool=exp-failgrind --syscall-fail-atstart=yes --syscall-errno=EINTR --syscall-errno=open,EACCES ./my_prog

A further option --syscall-specified-only, which can be set to yes or no and defaults to no. If set to yes, then only functions that have been configured with --syscall-errno=<function>,<error> will fail. This allows targetted testing of particular syscalls. For example:

Calls to open() will fail with EACCES, all other syscalls will succeed:

valgrind --tool=exp-failgrind --syscall-fail-atstart=yes --syscall-errno=open,EACCES --specified-only=yes ./my_prog

If you wish to have a syscall always succeed regardless of other options, use the --syscall-allow option. This can be specified multiple times. This option is particularly useful if you are using printf() for debugging and always want it to succeed by allowing the write syscall. For example:

valgrind --tool=exp-failgrind --syscall-fail-atstart=yes --syscall-allow=write ./my_prog

The number of syscall failures in a Failgrind run can be limited using the --syscall-max-fails option, which is an integer. It can sometimes be useful to limit to a single syscall per run, at the cost of increasing the number of runs that are needed:

valgrind --tool=exp-failgrind --syscall-max-fails=1 ./my_prog

The --syscall-fail-chance option allows syscall failures to happen randomly. Set to an integer value 1-100 that represents the percentage chance an syscall that is due to be rejected (that hasn't been seen before and meets all other criteria) will actually fail. Use with the --seed to get reproduceable but random tests. When you use --syscall-fail-chance, the seed being used is displayed by Failgrind on start. For example, to fail 50% of the time:

valgrind --tool=exp-failgrind --syscall-fail-chance=50 ./my_prog

==15526== Failgrind, a memory allocation and syscall failure testing tool
==15526== NOTE: This is an Experimental-Class Valgrind Tool
==15526== Copyright (C) 2018-2021, and GNU GPL'd, by Roger Light.
==15526== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==15526== Command: ./my_prog
==15526==
==15526== Using random seed: 7937841
...