# Chapter 27. DTrace

## 27.1. Synopsis

DTrace, also known as Dynamic Tracing, was developed by Sun™ as a tool for locating performance bottlenecks in production and pre-production systems. In addition to diagnosing performance problems, DTrace can be used to help investigate and debug unexpected behavior in both the FreeBSD kernel and in userland programs.

DTrace is a remarkable profiling tool, with an impressive array of features for diagnosing system issues. It may also be used to run pre-written scripts to take advantage of its capabilities. Users can author their own utilities using the DTrace D Language, allowing them to customize their profiling based on specific needs.

The FreeBSD implementation provides full support for kernel DTrace and experimental support for userland DTrace. Userland DTrace allows users to perform function boundary tracing for userland programs using the `pid` provider, and to insert static probes into userland programs for later tracing. Some ports, such as databases/postgresql12-server and lang/php74 have a DTrace option to enable static probes.

The official guide to DTrace is maintained by the illumos project at illumos Dynamic Tracing Guide.

After reading this chapter, you will know:

- What DTrace is and what features it provides.

- Differences between the Solaris™ DTrace implementation and the one provided by FreeBSD.

- How to enable and use DTrace on FreeBSD.

Before reading this chapter, you should:

- Understand UNIX® and FreeBSD basics (FreeBSD Basics).

- Have some familiarity with security and how it pertains to FreeBSD (Security).

## 27.2. Implementation Differences

While the DTrace in FreeBSD is similar to that found in Solaris™, differences do exist. The primary difference is that in FreeBSD, DTrace is implemented as a set of kernel modules and DTrace can not be used until the modules are loaded. To load all of the necessary modules:

    # kldload dtraceall

Beginning with FreeBSD 10.0-RELEASE, the modules are automatically loaded when dtrace(1) is run.

FreeBSD uses the `DDB_CTF` kernel option to enable support for loading ctf(5) data from kernel modules and the kernel itself. `CTF` is the Solaris™ Compact C Type Format which encapsulates a reduced form of debugging information similar to `DWARF` and the venerable stabs. `CTF` data is added to binaries by the ctfconvert(1) and ctfmerge(1) build tools. The `ctfconvert` utility parses `DWARF` `ELF` debug sections created by the compiler and `ctfmerge` merges `CTF` `ELF` sections from objects into either executables or shared libraries.

Some different providers exist for FreeBSD than for Solaris™. Most notable is the `dtmalloc` provider, which allows tracing malloc(9) by type in the FreeBSD kernel. Some of the providers found in Solaris™, such as `cpc`, are not present in FreeBSD. These may appear in future versions of FreeBSD. Moreover, some of the providers available in both operating systems are not compatible, in the sense that their probes have different argument types. Thus, `D` scripts written on Solaris™ may or may not work unmodified on FreeBSD, and vice versa.

Due to security differences, only `root` may use DTrace on FreeBSD. Solaris™ has a few low level security checks which do not yet exist in FreeBSD. As such, the /dev/dtrace/dtrace is strictly limited to `root`.

DTrace falls under the Common Development and Distribution License (`CDDL`) license. To view this license on FreeBSD, see /usr/src/cddl/contrib/opensolaris/OPENSOLARIS.LICENSE or view it online at http://opensource.org/licenses/CDDL-1.0. While a FreeBSD kernel with DTrace support is `BSD` licensed, the `CDDL` is used when the modules are distributed in binary form or the binaries are loaded.

## 27.3. Enabling DTrace Support

In FreeBSD 9.2 and 10.0, DTrace support is built into the GENERIC kernel. Users of earlier versions of FreeBSD or who prefer to statically compile in DTrace support should add the following lines to a custom kernel configuration file and recompile the kernel using the instructions in Configuring the FreeBSD Kernel:

    options         KDTRACE_HOOKS
    options         DDB_CTF
    makeoptions	DEBUG=-g
    makeoptions	WITH_CTF=1

Users of the AMD64 architecture should also add this line:

    options         KDTRACE_FRAME

This option provides support for dtrace_fbt(4). While DTrace will work without this option, there will be limited support for function boundary tracing.

Once the FreeBSD system has rebooted into the new kernel, or the DTrace kernel modules have been loaded using `kldload dtraceall`, install the current DTrace Toolkit (sysutils/dtrace-toolkit), a collection of ready-made scripts for collecting system information. There are scripts to check open files, memory, CPU usage, and a lot more. FreeBSD includes some scripts in the base system as well; see /usr/share/dtrace.

| The scripts found in /usr/share/dtrace have been specifically ported to FreeBSD. Not all of the scripts found in the DTrace Toolkit will work as-is on FreeBSD and some scripts may require some effort in order for them to work on FreeBSD.

The DTrace Toolkit includes many scripts in the special language of DTrace. This language is called the D language and it is very similar to C++. An in depth discussion of the language is beyond the scope of this document. Refer to the FreeBSD d(7) manual page for an overview of the D language on FreeBSD. The D language is also covered extensively in the illumos Dynamic Tracing Guide.

## 27.4. Enabling DTrace in Out-of-Kernel Modules

To add DTrace support to an out-of-kernel module, which is useful for development and debugging, include the following line in the module’s Makefile:

    CFLAGS+= -DKDTRACE_HOOKS

This flag enables DTrace hooks during compilation, allowing for advanced debugging and monitoring of the module. Ensure to recompile the module after this modification to activate DTrace functionality.

## 27.5. Using DTrace

DTrace scripts consist of a list of one or more probes, or instrumentation points, where each probe is associated with an action. Whenever the condition for a probe is met, the associated action is executed. For example, an action may occur when a file is opened, a process is started, or a line of code is executed. The action might be to log some information or to modify context variables. The reading and writing of context variables allows probes to share information and to cooperatively analyze the correlation of different events.

To view all probes, the administrator can execute the following command:

    # dtrace -l | more

Each probe has an `ID`, a `PROVIDER` (e.g., `dtrace` or `fbt`), a `MODULE`, and a `FUNCTION NAME`. Refer to dtrace(1) for more information about this command.

The examples in this section provide an overview of how to use two of the fully supported scripts from the DTrace Toolkit: the hotkernel and procsystime scripts.

The hotkernel script is designed to identify which function is using the most kernel time. It will produce output similar to the following:

    # cd /usr/local/share/dtrace-toolkit
    # ./hotkernel
    Sampling... Hit Ctrl-C to end.

As instructed, use the `Ctrl`+`C` key combination to stop the process. Upon termination, the script will display a list of kernel functions and timing information, sorting the output in increasing order of time:

    kernel`_thread_lock_flags                                   2   0.0%
    0xc1097063                                                  2   0.0%
    kernel`sched_userret                                        2   0.0%
    kernel`kern_select                                          2   0.0%
    kernel`generic_copyin                                       3   0.0%
    kernel`_mtx_assert                                          3   0.0%
    kernel`vm_fault                                             3   0.0%
    kernel`sopoll_generic                                       3   0.0%
    kernel`fixup_filename                                       4   0.0%
    kernel`_isitmyx                                             4   0.0%
    kernel`find_instance                                        4   0.0%
    kernel`_mtx_unlock_flags                                    5   0.0%
    kernel`syscall                                              5   0.0%
    kernel`DELAY                                                5   0.0%
    0xc108a253                                                  6   0.0%
    kernel`witness_lock                                         7   0.0%
    kernel`read_aux_data_no_wait                                7   0.0%
    kernel`Xint0x80_syscall                                     7   0.0%
    kernel`witness_checkorder                                   7   0.0%
    kernel`sse2_pagezero                                        8   0.0%
    kernel`strncmp                                              9   0.0%
    kernel`spinlock_exit                                       10   0.0%
    kernel`_mtx_lock_flags                                     11   0.0%
    kernel`witness_unlock                                      15   0.0%
    kernel`sched_idletd                                       137   0.3%
    0xc10981a5                                              42139  99.3%

This script will also work with kernel modules. To use this feature, run the script with `-m`:

    # ./hotkernel -m
    Sampling... Hit Ctrl-C to end.
    ^C
    MODULE                                                  COUNT   PCNT
    0xc107882e                                                  1   0.0%
    0xc10e6aa4                                                  1   0.0%
    0xc1076983                                                  1   0.0%
    0xc109708a                                                  1   0.0%
    0xc1075a5d                                                  1   0.0%
    0xc1077325                                                  1   0.0%
    0xc108a245                                                  1   0.0%
    0xc107730d                                                  1   0.0%
    0xc1097063                                                  2   0.0%
    0xc108a253                                                 73   0.0%
    kernel                                                    874   0.4%
    0xc10981a5                                             213781  99.6%

The procsystime script captures and prints the system call time usage for a given process `ID` (`PID`) or process name. In the following example, a new instance of /bin/csh was spawned. Then, procsystime was executed and remained waiting while a few commands were typed on the other incarnation of `csh`. These are the results of this test:

    # ./procsystime -n csh
    Tracing... Hit Ctrl-C to end...
    ^C

    Elapsed Times for processes csh,

             SYSCALL          TIME (ns)
              getpid               6131
           sigreturn               8121
               close              19127
               fcntl              19959
                 dup              26955
             setpgid              28070
                stat              31899
           setitimer              40938
               wait4              62717
           sigaction              67372
         sigprocmask             119091
        gettimeofday             183710
               write             263242
              execve             492547
               ioctl             770073
               vfork            3258923
          sigsuspend            6985124
                read         3988049784

As shown, the read(2) system call used the most time in nanoseconds while the getpid(2) system call used the least amount of time.

Last modified on: July 18, 2025 by Mateusz Piotrowski

Prev
Home
Next