funcslower(8) System Manager's Manual funcslower(8)

# [NAME](#NAME)

funcslower - Trace slow kernel or user function calls.

# [SYNOPSIS](#SYNOPSIS)

**funcslower \[-hf] \[-p PID] \[-U | -K] \[-m MIN\_MS] \[-u MIN\_US] \[-a ARGUMENTS] \[-T] \[-t] \[-v] function \[function ...]**

# [DESCRIPTION](#DESCRIPTION)

This script traces a kernel or user function's entry and return points, and prints a message when the function's latency exceeded the specified threshold. Multiple functions are supported, and you can mix kernel functions with user functions in different libraries.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 1 is used. Recursive functions are not supported: only the inner-most recursive invocation will be traced.

Since this uses BPF, only the root user can use this tool.

# [REQUIREMENTS](#REQUIREMENTS)

CONFIG\_BPF and bcc.

# [OPTIONS](#OPTIONS)

-p PID Trace this PID only.

[-m MIN\_NS](#m)

Minimum duration to trace, in milliseconds. Default is 1 ms.

[-u MIN\_US](#u)

Minimum duration to trace, in microseconds.

[-a ARGUMENTS](#a)

Print the function's arguments, up to 6.

[-T](#T)

Print a HH:MM:SS timestamp with each entry.

[-t](#t)

Print a seconds timestamp with each entry, at microsecond resolution.

[-f](#f)

Print output in folded stack format.

[-U](#U)

Show stacks from user space only (no kernel space stacks).

[-K](#K)

Show stacks from kernel space only (no user space stacks).

[-v](#v)

Print the resulting BPF program, for debugging purposes.

[function](#function)

The function to trace -- multiple functions are supported. If a plain function name is provided, the function is assumed to be a kernel function. For user functions, provide the library name and the function name, e.g. bash:readline or c:malloc.

# [EXAMPLES](#EXAMPLES)

[Trace vfs\_write calls slower than 1ms:](#Trace)

\# **funcslower vfs\_write**

[Trace open() calls in libc slower than 10us:](#Trace~2)

\# **funcslower -u 10 c:open**

[Trace both malloc() and free() slower than 10us, in pid 135 only:](#Trace~3)

\# **funcslower -p 135 -u 10 c:malloc c:free**

[Trace the write syscall and print its first 4 arguments:](#Trace~4)

\# **funcslower -a 4 SyS\_write**

[Trace opens from libc and print the user and kernel stack frames:](#Trace~5)

\# **funcslower -UK c:open**

# [FIELDS](#FIELDS)

[TIME](#TIME)

Time of the event as a human-readable HH:MM:SS format, or a timestamp in seconds at microsecond-accuracy from the first event seen.

[COMM](#COMM)

Process name.

[PID](#PID)

Process ID.

[LAT](#LAT)

Latency of the operation in either microseconds (us) or milliseconds (ms).

[RVAL](#RVAL)

The return value from the function. Often useful for diagnosing a relationship between slow and failed function calls.

[FUNC](#FUNC)

The function name, followed by its arguments if requested.

# [OVERHEAD](#OVERHEAD)

Depending on the function(s) being traced, overhead can become severe. For example, tracing a common function like malloc() can slow down a C/C++ program by a factor of 2 or more. On the other hand, tracing a low-frequency event like the SyS\_setreuid() function will probably not be as prohibitive, and in fact negligible for functions that are called up to 100-1000 times per second.

You should first use the funclatency and argdist tools for investigation, because they summarize data in-kernel and have a much lower overhead than this tool. To get a general idea of the number of times a particular function is called (and estimate the overhead), use the funccount tool, e.g.:

\# funccount c:open

# [SOURCE](#SOURCE)

This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion \_examples.txt file containing example usage, output, and commentary for this tool.

# [OS](#OS)

Linux

# [STABILITY](#STABILITY)

Unstable - in development.

# [AUTHOR](#AUTHOR)

Sasha Goldshtein

# [SEE ALSO](#SEE_ALSO)

[funccount(8)](/man/enterprise-packages/bcc/funccount.8), [funclatency(8)](/man/enterprise-packages/bcc/funclatency.8), [argdist(8)](/man/enterprise-packages/bcc/argdist.8), [trace(8)](/man/enterprise-packages/bcc/trace.8)

2017-03-30 USER COMMANDS

---

Document: funcslower(8) from bcc 0.37.0-r2 (Chainguard OS)
Source: https://github.com/iovisor/bcc