Useful DTrace recipes

2014 Mar 19

Intro

If there ever was one tool I wanted to see in Linux, it is DTrace. Ever since its release by Sun Microsystems, it has been an incredibly useful tool to trace, measure, and troubleshoot kernel and application issues. Since I’ve been mostly working on an OS X environment for the past ~9 months, I’ve had the pleasure of using it again for various (un-)common tasks.

I’ve decided to band them all together and keep them in one place, to serve as a reference for me, and hopefully be useful to others as well:

Recipes

# Files opened by process. Very useful when reversing:
dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'

# Syscall count by program:
dtrace -n 'syscall:::entry { @num[execname] = count(); }'

# Syscall count by syscall:
dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'

# Syscall count by PID:
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

# Read bytes by process:
dtrace -n 'syscall::read:entry { @bytes[execname] = sum(arg0); }'

# Periodic breakdown of written bytes by process:
dtrace -n 'syscall::write:* { @bytes[execname] = sum(arg0); } tick-5sec { printa(@bytes); trunc(@bytes); }'

# Read size distribution by process:
dtrace -n 'syscall::read:entry { @dist[execname] = quantize(arg0); }'

# Write size distribution by process:
dtrace -n 'syscall::write:entry { @dist[execname] = quantize(arg0); }'

# Disk size (in blocks) by process:
dtrace -n 'io:::start { printf("%d %s %d", pid, execname, args[0]->b_bcount); }'

# Pages paged in by process:
dtrace -n 'vminfo::vm_fault_enter:pgin { @pg[execname] = sum(arg0); }'

# Minor faults by process:
dtrace -n 'vminfo:::as_fault { @mem[execname] = sum(arg0); }'

# Interrupts by CPU:
dtrace -n 'sdt:::interrupt-start { @num[cpu] = count(); }'

# New processes with arguments and time:
dtrace -qn 'syscall::exec*:return { printf("%Y %s\n",walltimestamp,curpsinfo->pr_psargs); }'

# Successful signal details:
dtrace -n 'proc:::signal-send /pid/ { printf("%s -%d %d",execname,args[2],args[1]->pr_pid); }'

# System call breakdown for the process with PID 31337:
dtrace -n syscall:::entry'/pid == 31337/{ @syscalls[probefunc] = count(); }'

# Tracking memory page faults for process:
dtrace -n 'vminfo:::as_fault { @mem[execname] = sum(arg0); }'

# iOS Intruments-like malloc size distribution plot:
dtrace -n 'pid$target::malloc:entry { @ = quantize(arg0); }' -p PID

# Memory allocation via malloc by stack trace and requested size:
dtrace -n 'pid$target::malloc:entry { @[ustack()] = sum(arg0); }' -p PID

# Rate of disk I/O:
dtrace -n 'io:::start { @io = count(); } tick-1sec { printa("Disk I/Os per second: %@d\n", @io); trunc(@io); }'

Examples

Files opened by process:

$ sudo dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'
dyld: DYLD_ environment variables being ignored because main executable (/usr/bin/sudo) is setuid or setgid
dtrace: description 'syscall::open*:entry ' matched 4 probes

CPU     ID                    FUNCTION:NAME
  4    149                       open:entry iTerm /Users/mfukar/Library/Saved Application State/com.googlecode.iterm2.savedState/window_1.data
  4    149                       open:entry iTerm /Users/mfukar/Library/Saved Application State/com.googlecode.iterm2.savedState/window_2.data
  2    149                       open:entry mdworker /Users/mfukar/Library/Saved Application State/com.googlecode.iterm2.savedState/data.data
  2    149                       open:entry mds .
  3    149                       open:entry mds .
  3    149                       open:entry mdworker /Users/mfukar/Library/Saved Application State/com.googlecode.iterm2.savedState/windows.plist
  4    149                       open:entry iTerm /Users/mfukar/Library/Saved Application State/com.googlecode.iterm2.savedState/data.data

Read bytes by process:

# dtrace -n 'syscall::read:entry { @bytes[execname] = sum(arg0); }'
dtrace: description 'syscall::read:entry ' matched 1 probe
^C
  syslogd                                                           8
  vim                                                              14
  iTerm                                                            42
  launchd                                                          91
  NetworkBrowserA                                                 204
  SystemUIServer                                                  204
  mdworker                                                        294
  gpg-agent                                                       568
  fseventsd                                                      1328
  mds                                                            2317
  configd                                                        3350
  UserEventAgent                                                 5966
  plugin-containe                                               45379
  firefox                                                      371532

Write size distribution by process:

# dtrace -n 'syscall::write:entry { @dist[execname] = quantize(arg0); }'
<snip>
firefox
         value  ------------- Distribution ------------- count
             2 |                                         0
             4 |@@@@@@@@@@@@                             4582
             8 |@@@@@@                                   2266
            16 |@@@@                                     1450
            32 |@@                                       895
            64 |@@@@@@@@@@@@@@@                          5665
           128 |                                         0

Interrupts by CPU:

The below does not work on OS X, unfortunately:

# dtrace -n 'sdt:::interrupt-start { @num[cpu] = count(); }'
dtrace: description 'sdt:::interrupt-start ' matched 1 probe
^C

      513                2
      515                4
        3               39
        2               39

Rate of disk I/O:

# sudo dtrace -n 'io:::start { @io = count(); } tick-1sec { printa("Disk I/Os per second: %@d", @io); trunc(@io); }' -p 428
dtrace: description 'io:::start ' matched 2 probes
CPU     ID                    FUNCTION:NAME
  6 183582                       :tick-1sec
  6 183582                       :tick-1sec
  6 183582                       :tick-1sec Disk I/Os per second: 5
  6 183582                       :tick-1sec Disk I/Os per second: 2
  6 183582                       :tick-1sec

If you’re interested beyond the scope of this tiny blog post, check out Brendan Gregg’s DTrace page, or his brilliant book.

« Past Future »

Tagged :


Comments:

comments powered by Disqus