DTrace Scripts

There’s a really handy article over at MacTech that lets you drop into DTrace on Mac OS X rather easily. If you’re interested in the technology but haven’t gotten around to playing with it, the author starts a little slow and then ramps up to some really interesting things.

Of note in the article are two bits I hadn’t come across yet:

• Apple’s ported some of the DTrace Toolkit scripts over to Mac OS X. Check the manpage for some of these with man -k dtrace
• There are some more examples of DTrace in the standard install at /usr/share/examples/DTTk.

There’s a really handy toolkit over at OpenSolaris.org called the DTrace Toolkit. Inside you’ll find a ton of example scripts for all manner of troubleshooting and investigation.

At the very least, it’s a great introduction to DT scripting in general and worth a look for the take-apart value.

Some quick one-liners for sysadmins that work on Mac OS X and Solaris:

New processes with arguments

dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

Files opened by process

dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

Pages paged in by process

dtrace -n 'vminfo:::pgpgin { @pg[execname] = sum(arg0); }'

Pages paged out by process

dtrace -n 'vminfo:::pgpgout { @pg[execname] = sum(arg0); }'

Minor faults by process

dtrace -n 'vminfo:::as_fault { @mem[execname] = sum(arg0); }'

Some quick development-related one-liners that work on Solaris and Mac OS X.

Syscall count by program

dtrace -n 'syscall:::entry { @num[execname] = count(); }'

Syscall count by syscall

dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'

Syscall count by process

dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

Successful signal details

dtrace -n 'proc:::signal-send /pid/ { printf("%s -%d %d",execname,args[2],args[1]->pr_pid); }'

A program to print disk I/O events as they happen, with useful details such as UID, PID, filename, command, etc.

Displays the top disk I/O events by process. This tracks disk I/O by process, and prints a summary report that is refreshed every interval.

When system calls fail, an errno variable is set to convey a meaningful message to the end user - so long as the program does something with it (eg, "ls" printing "No such file or directory"). This program fetches and prints details for all syscall failures along with their message, whether the failing program is already printing this info or not.

            EXEC          SYSCALL  ERR  DESC
             mds __semwait_signal   60  Operation timed out 
 DirectoryServic           access    2  No such file or directory 
 DirectoryServic           access    2  No such file or directory 
 DirectoryServic    open_nocancel    2  No such file or directory 
 DirectoryServic           access    2  No such file or directory 
 DirectoryServic    open_nocancel    2  No such file or directory 
 DirectoryServic           access    2  No such file or directory 
 DirectoryServic    open_nocancel    2  No such file or directory 
 DirectoryServic    open_nocancel    2  No such file or directory 
        mdworker            lstat   13  Permission denied 
        mdworker      getattrlist   13  Permission denied 
        mdworker            lstat   13  Permission denied 
         launchd            mkdir   17  File exists 

This script measures the seek distance between consecutive reads and writes, and provides a histogram with the seek distances. For applications that are using sequential access patterns (e.g., dd), the distribution will be small. For applications accessing data in a random nature, you will see a wide distribution.

Brendan Gregg developed prustat to display the top processes sorted by CPU, Memory, Disk or Network utilization:

$ prustat -c -t 10 5

  PID   %CPU   %Mem  %Disk   %Net  COMM
 7176   0.88   0.70   0.00   0.00  dtrace
 7141   0.00   0.43   0.00   0.00  sshd
 7144   0.11   0.24   0.00   0.00  sshd
    3   0.34   0.00   0.00   0.00  fsflush
 7153   0.03   0.19   0.00   0.00  bash
   99   0.00   0.22   0.00   0.00  nscd
 7146   0.00   0.19   0.00   0.00  bash
   52   0.00   0.17   0.00   0.00  vxconfigd
 7175   0.07   0.09   0.00   0.00  sh
   98   0.00   0.16   0.00   0.00  kcfd

This script is useful for getting a high level understanding of what is happening on a server.