Strace is a Linux command-line tool that intercepts system calls and signals in real time using the ptrace() system call. It reveals exactly what a process is doing on a server. This makes strace essential for debugging and troubleshooting, performance tuning, and security analysis. You can use strace to trace system calls, from basic commands to advanced output management. When debugging an application on a Linux server, strace shows every file open, network read, and memory allocation. This visibility turns guesswork into precise diagnosis. You will learn to trace system calls on a running service without restarting it.

How to Use strace to Trace System Calls

Tracing System Calls Made by a Command

The simplest way to use strace to trace system calls involves running a command directly through it. You type strace followed by the command you want to observe. For example, strace ls -l executes the ls command while capturing every system call it makes. The tool records each interaction between the program and the Linux kernel. You see file opens, directory reads, memory mappings, and process exits. This approach works well for short-lived commands where you want immediate results.

The basic syntax follows this pattern: strace /path/to/program. You can trace any executable this way. When you run the command, strace launches the program as a child process. The tool intercepts every request the program sends to the kernel. Each line of output shows the system call name, its arguments, and the return value. This output gives you a complete picture of the program’s behavior. You can identify missing files, permission errors, or unexpected network connections. The command runs until it exits normally or crashes. Then strace prints a final summary to your terminal.

This method works best for debugging scripts, testing new software, or understanding how a utility operates. You can combine it with other options to filter results. For instance, you might trace only file-related operations or network activity. The key point is that you control the entire execution from start to finish. You see every step the program takes without guessing.

Attaching to a Running Process

Server environments often require you to diagnose a service that is already running. You cannot simply restart it to trace its behavior. In this situation, you attach to an already-running process using the -p option. The syntax is strace -p PID, where PID stands for the process id. You must know the pid number of the process you want to examine. You can find this information using commands like ps or pgrep. For example, pgrep nginx returns the pid number for each nginx worker process.

Once you attach, strace begins capturing all system calls from that moment forward. The process being traced continues running normally. You do not interrupt its work or cause downtime. This capability proves invaluable on production servers. You can observe a web server handling requests, a database processing queries, or an application writing logs. The tool relies on the ptrace() system call to attach to another process. Linux permits this operation when you have appropriate permissions, typically as root or the same user.

You can also trace specific processes by attaching to their pid individually. If a service spawns multiple workers, you might attach to each one separately. The -f option helps here by following child processes automatically. However, for a single process, strace -p PID gives you focused visibility. You watch the system calls in real time as they happen. When you finish observing, you press Ctrl+C to detach. The process continues running unaffected. This non-intrusive approach makes strace a powerful diagnostic tool for busy servers. You gain insight into live behavior without disrupting operations.

Essential strace Options on Linux

Following Child Processes

Many server applications do not run as a single process. Web servers like Apache or Nginx create multiple worker processes to handle concurrent requests. A database may fork helper processes for background tasks. When you trace only the main process, you miss all the activity happening in these workers. The -f option solves this problem. It tells strace to follow child processes created by fork, clone, or vfork. Each new process gets traced automatically as soon as it appears.

You use the option like this: strace -f -p PID. The tool attaches to the main process and then watches every child it creates. This capability proves critical for tracing servers that spawn workers. Without -f, you see only the parent’s system calls. With it, you see the complete picture of the entire process tree. The output shows each system call with the pid number of the process that made it. You can distinguish between the main process and its children easily.

Consider a PHP-FPM server handling web requests. The master process receives connections and assigns them to worker processes. Tracing without -f shows only the master’s activity. Adding -f reveals exactly what each worker does with each request. You see file reads, database queries, and network operations across all workers. This visibility helps you identify bottlenecks or errors in specific worker processes.

Filtering System Calls

A busy server generates thousands of system calls per second. Tracing everything produces overwhelming output that hides the information you need. The -e trace= option filters the trace to specific system calls. You specify which calls interest you, and strace ignores everything else. For example, strace -e trace=open,read,write captures only file operations and data transfers. This focused approach reduces noise and highlights relevant activity.

You can also filter by categories instead of individual calls. The %process category includes process management calls like fork, wait, and exec. Use -e trace=%process to see how a program creates and manages child processes. The %signal category covers signal-related calls such as kill and sigaction. This helps you debug signal handling issues. The %pure category includes system calls that always succeed and take no arguments, like getpid. These calls rarely cause problems, so filtering them out reduces clutter.

Combining filters with the -f option gives you precise control. You might trace only network-related calls across all child processes. The command strace -f -e trace=network -p PID captures socket operations from every worker. This targeted approach works well for diagnosing connection issues or performance problems.

When you need to save the results, use the -o option followed by a filename. This directs the output trace information to a file instead of your terminal. The command strace -o trace.log -p PID writes all captured data to trace.log. You can review the file later or share it with colleagues. Writing trace information to a file also prevents output from mixing with your terminal’s normal display. For long-running traces, you can redirect the trace to a text file and analyze it after detaching. This method keeps your terminal clean and preserves the data for detailed examination. The -o option works with all other filters, so you can save only the specific system calls you care about.

Analyzing Performance with strace

System Call Summary

You can turn strace into a performance profiler with the -c option. This flag generates a summary table instead of a line-by-line trace. The table shows each system call, its count, the number of errors, and the total time spent. You see at a glance which operations dominate your workload. This information helps you identify bottlenecks without reading thousands of individual lines.

Run strace -c -p PID to attach to a running service and collect data. Let the trace run for a few seconds or minutes. When you press Ctrl+C, strace prints the summary table to your terminal. The table sorts system calls by time consumed. You might discover that read operations consume 80 percent of the time. Or you might see that poll calls dominate, indicating idle waiting. This insight guides your optimization efforts toward the right area.

The summary also shows error counts for each syscall. A high error count on open or connect signals a configuration problem. You can investigate those specific failures after seeing the pattern. The -c option works with -f to include child processes in the summary. You can profile an entire process tree this way. This approach gives you a complete performance picture across all workers.

Timestamps and Latency

Sometimes you need to know when each system call happens. The -t option adds a timestamp to every line of output. Each timestamp shows the time of day with second precision. You can correlate system calls with external events. For example, you might match a slow database query to a specific read call. The -tt option adds microseconds to each timestamp. This precision reveals short delays between related system calls.

The -r option provides relative timestamps. Each line shows the time elapsed since the previous system call. This format highlights latency between operations. You can spot a long gap between a write call and its corresponding read. That gap might indicate network latency or disk I/O delays. Relative timestamps make these gaps obvious.

Combine timestamps with filters to focus on specific processes. You can trace only network-related calls with -e trace=network and add -tt for microsecond precision. This combination reveals exactly where connection delays occur. You can also save this output to a file for later analysis. The timestamps remain in the saved file, preserving the timing information. This practice helps you document performance issues for your team. You can compare traces from different times to track improvements after making changes.

Managing Large Output on a Busy Server

Compressing Trace Output

A high-traffic server generates enormous amounts of data during a trace session. Every file read, network packet, and memory operation produces a line of text. You can fill your disk quickly when tracing a busy web server or database. The -C option solves this problem by compressing the trace output on the fly. This flag works with the -o option to write compressed data directly to a file. You get a .gz file instead of a plain text file. This approach reduces storage requirements dramatically.

You might trace a process handling hundreds of requests per second. Without compression, you could generate gigabytes of data within minutes. With -C, you shrink that volume substantially. The compression happens in real time as strace captures each system call. You do not need to compress the file afterward. This feature proves essential when you have limited disk space on a production server. You can run longer trace sessions without worrying about filling your filesystem.

Rotating Long-Running Traces

Long-running traces present another challenge. You might need to monitor a service for hours to catch an intermittent issue. A single output file grows continuously and becomes difficult to manage. You can combine -o with external tools like logrotate or split to handle this growth. The logrotate utility rotates files based on size or time intervals. You configure it to rename the current trace file and start a new one automatically. This process keeps each file manageable and preserves historical data.

The -ff option adds another layer of control. This flag creates one file per child process when you combine it with -o. Each worker process writes to its own file. You can rotate these individual files independently. This approach makes it easier to isolate problems in specific workers. You might see one worker making repeated errors while others operate normally. The separate files reveal this pattern clearly.

You should remember that tracing adds overhead to your system. Every captured system call requires processing time. On a busy server, this overhead can slow down the traced process noticeably. Use -e trace= filters to reduce the volume of captured data. Focus only on the system calls you need. This practice minimizes both the performance impact and the output size. You keep your server responsive while gathering the diagnostic information you require.

You now know how to use strace to trace system calls on a Linux server, from a simple command to deep performance analysis. This Linux tool captures signals and every kernel request a process makes. Options like -f, -e trace=, and -c keep your trace output focused. They also stop raw output from burying the one detail you need.

Practice on a test machine first. On production, remember that strace slows the target down, so filter aggressively and detach when finished.

Attach to an already-running process only when you must. With care, this Linux utility becomes one of your most reliable diagnostic tools.

FAQ

What exactly does strace do on a Linux server?

Strace intercepts every system call a program makes to the Linux kernel. It shows you file opens, network reads, and memory operations in real time. This visibility helps you diagnose errors, find bottlenecks, and understand application behavior without guessing. You can use strace to trace system calls for debugging or performance analysis.

How do I attach strace to an already-running service?

You use the -p option with the process id. First, find the pid number using ps or pgrep. Then run strace -p PID. The tool attaches to that process and begins capturing its system calls immediately. Press Ctrl+C to detach. This method works on any Linux distribution.

Does strace affect server performance?

Yes, tracing adds overhead. Every captured system call requires extra processing time. On a busy server, this overhead can slow the traced process noticeably. Use filters like -e trace= to limit what you capture. This reduces both performance impact and output volume. Linux system administrators should always filter aggressively.

What permissions do I need to trace a process?

You need root privileges or the same user account as the target process. Linux restricts ptrace() access for security reasons. Without proper permissions, strace returns an error. Run strace as root or use sudo when tracing services owned by other users.

Can I trace multiple processes at once?

Yes, use the -f option to follow child processes. This captures system calls from the main process and all its workers. Each output line shows the pid of the process that made the call. This approach works well for servers that spawn multiple worker processes. Linux servers often run many workers simultaneously.