Intro
If there ever was one tool I wanted to see in Linux, it is DTrace. Ever since its release by Sun Microsystems, it has been an incredibly useful tool to trace, measure, and troubleshoot kernel and application issues. Since I’ve been mostly working on an OS X environment for the past ~9 months, I’ve had the pleasure of using it again for various (un-)common tasks.
I’ve decided to band them all together and keep them in one place, to serve as a reference for me, and hopefully be useful to others as well:
Recipes
# Files opened by process. Very useful when reversing:
dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'
# Syscall count by program:
dtrace -n 'syscall:::entry { @num[execname] = count(); }'
# Syscall count by syscall:
dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'
# Syscall count by PID:
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'
# Read bytes by process:
dtrace -n 'syscall::read:entry { @bytes[execname] = sum(arg0); }'
# Periodic breakdown of written bytes by process:
dtrace -n 'syscall::write:* { @bytes[execname] = sum(arg0); } tick-5sec { printa(@bytes); trunc(@bytes); }'
# Read size distribution by process:
dtrace -n 'syscall::read:entry { @dist[execname] = quantize(arg0); }'
# Write size distribution by process:
dtrace -n 'syscall::write:entry { @dist[execname] = quantize(arg0); }'
# Disk size (in blocks) by process:
dtrace -n 'io:::start { printf("%d %s %d", pid, execname, args[0]->b_bcount); }'
# Pages paged in by process:
dtrace -n 'vminfo::vm_fault_enter:pgin { @pg[execname] = sum(arg0); }'
# Minor faults by process:
dtrace -n 'vminfo:::as_fault { @mem[execname] = sum(arg0); }'
# Interrupts by CPU:
dtrace -n 'sdt:::interrupt-start { @num[cpu] = count(); }'
# New processes with arguments and time:
dtrace -qn 'syscall::exec*:return { printf("%Y %s\n",walltimestamp,curpsinfo->pr_psargs); }'
# Successful signal details:
dtrace -n 'proc:::signal-send /pid/ { printf("%s -%d %d",execname,args[2],args[1]->pr_pid); }'
# System call breakdown for the process with PID 31337:
dtrace -n syscall:::entry'/pid == 31337/{ @syscalls[probefunc] = count(); }'
# Tracking memory page faults for process:
dtrace -n 'vminfo:::as_fault { @mem[execname] = sum(arg0); }'
# iOS Intruments-like malloc size distribution plot:
dtrace -n 'pid$target::malloc:entry { @ = quantize(arg0); }' -p PID
# Memory allocation via malloc by stack trace and requested size:
dtrace -n 'pid$target::malloc:entry { @[ustack()] = sum(arg0); }' -p PID
# Rate of disk I/O:
dtrace -n 'io:::start { @io = count(); } tick-1sec { printa("Disk I/Os per second: %@d\n", @io); trunc(@io); }'
Examples
Files opened by process:
$ sudo dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'
dyld: DYLD_ environment variables being ignored because main executable (/usr/bin/sudo) is setuid or setgid
dtrace: description 'syscall::open*:entry ' matched 4 probes
CPU ID FUNCTION:NAME
4 149 open:entry iTerm /Users/mfukar/Library/Saved Application State/com.googlecode.iterm2.savedState/window_1.data
4 149 open:entry iTerm /Users/mfukar/Library/Saved Application State/com.googlecode.iterm2.savedState/window_2.data
2 149 open:entry mdworker /Users/mfukar/Library/Saved Application State/com.googlecode.iterm2.savedState/data.data
2 149 open:entry mds .
3 149 open:entry mds .
3 149 open:entry mdworker /Users/mfukar/Library/Saved Application State/com.googlecode.iterm2.savedState/windows.plist
4 149 open:entry iTerm /Users/mfukar/Library/Saved Application State/com.googlecode.iterm2.savedState/data.data
Read bytes by process:
# dtrace -n 'syscall::read:entry { @bytes[execname] = sum(arg0); }'
dtrace: description 'syscall::read:entry ' matched 1 probe
^C
syslogd 8
vim 14
iTerm 42
launchd 91
NetworkBrowserA 204
SystemUIServer 204
mdworker 294
gpg-agent 568
fseventsd 1328
mds 2317
configd 3350
UserEventAgent 5966
plugin-containe 45379
firefox 371532
Write size distribution by process:
# dtrace -n 'syscall::write:entry { @dist[execname] = quantize(arg0); }'
<snip>
firefox
value ------------- Distribution ------------- count
2 | 0
4 |@@@@@@@@@@@@ 4582
8 |@@@@@@ 2266
16 |@@@@ 1450
32 |@@ 895
64 |@@@@@@@@@@@@@@@ 5665
128 | 0
Interrupts by CPU:
The below does not work on OS X, unfortunately:
# dtrace -n 'sdt:::interrupt-start { @num[cpu] = count(); }'
dtrace: description 'sdt:::interrupt-start ' matched 1 probe
^C
513 2
515 4
3 39
2 39
Rate of disk I/O:
# sudo dtrace -n 'io:::start { @io = count(); } tick-1sec { printa("Disk I/Os per second: %@d", @io); trunc(@io); }' -p 428
dtrace: description 'io:::start ' matched 2 probes
CPU ID FUNCTION:NAME
6 183582 :tick-1sec
6 183582 :tick-1sec
6 183582 :tick-1sec Disk I/Os per second: 5
6 183582 :tick-1sec Disk I/Os per second: 2
6 183582 :tick-1sec
If you’re interested beyond the scope of this tiny blog post, check out Brendan Gregg’s DTrace page, or his brilliant book.
« Past Future »