Using strace as a troubleshooting tool

Taken from Using strace as a troubleshooting tool (linuxcluster.wordpress.com) Strace, when runs in conjunction with a program do output all the calls made to the kernel by the program.

One of quick way to found out what is going on in your program is to do

$ strace -c ./my_hello_world_program

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
74.80    0.002998        1499         2           wait4
21.91    0.000878           4       221           read
0.95    0.000038           0       237         2 mmap
0.77    0.000031          10         3         1 mkdir
0.67    0.000027           0       566       361 open
0.35    0.000014           0        81           mprotect
0.30    0.000012           0        62        37 stat
0.25    0.000010           0       225           close
0.00    0.000000           0        37         1 write
0.00    0.000000           0       132           fstat
0.00    0.000000           0         8           poll
0.00    0.000000           0         2           lseek
0.00    0.000000           0       120           munmap
0.00    0.000000           0        15           brk
0.00    0.000000           0        16           rt_sigaction
................

................

------ ----------- ----------- --------- --------- ----------------
100.00    0.004008                  1990       411 total

If you wish to do a tracing, just do a, you can easily find out the error if there was....

$ strace ./my_hello_world_program

............

............

open("/tmp/openmpi-sessions-root@starfruit-h00.cluster.spms.ntu.edu.sg_0/25979/1/0",
O_RDONLY|O_NONBLOCK|O_DIRECTORY) = -1 ENOENT (No such file or directory)
munmap(0x2b46e05ef000, 2111200)         = 0
munmap(0x2b46dffe5000, 2102312)         = 0
munmap(0x2b46dfdde000, 2123264)         = 0
munmap(0x2b46e103f000, 2106960)         = 0
munmap(0x2b46e1242000, 2104560)         = 0
munmap(0x2b46e269d000, 2114912)         = 0
munmap(0x2b46e41c9000, 2145008)         = 0
munmap(0x2b46e43d5000, 2162608)         = 0

If you wish the output of strace to a file instead, do use the argument -o

$ strace -o strace_output_file ./my_hello_world_program

If you wish to trace system call, process,network, you can use the "-e trace=file", "-e trace=process", "-e trace=network",

$ strace -e trace=open,close,read,write ./my_hello_w0rld_program

$ strace -e trace=stat,chmod,unlink ./my_hello_world_program

Further Information:

1. Solutions for tracing UNIX applications (IBM DeveloperWorks)
2. strace - A very powerful troubleshooting tool for all Linux users (linuxhelp.blogspot.com)
3. Ten commands every linux developer should know (Linux Journal)