How to tune server network queue count to fix packet loss
Packet drops are a frustrating network problem. An often overlooked cause is a server network card queue count improperly configured causing packet loss. This guide shows a systematic process. You run baseline tests like ping to measure loss, and iperf to simulate traffic. This establishes a reference point. You inspect hardware drop counters and compare them to the queue count to isolate the problem. Then you determine the optimal queue count for network interface card tuning. Match it to CPU cores and workload. You adjust the count and verify the fix. If drops persist, you tune the OS network stack and NIC settings, such as increasing ring buffer sizes and configuring irq affinity. The primary tool for these adjustments is ethtool. Each step includes verification actions. You verify before moving to the next step. This approach directly targets the root cause and fixes the issue efficiently.
Diagnose Packet Loss from Improper Queue Count
Before you change any setting, you must prove the queue count is the real problem. A server network card queue count improperly configured causing packet loss will show clear symptoms. Your job is to find those symptoms and rule out other causes.
Run a Baseline Packet Loss Test
Start with a simple reachability test. Run ping from a client machine to your server. Send a large number of packets, at least several thousand, so the result is meaningful. Look at the loss percentage. A healthy link shows zero percent loss under light load. Any steady loss on a quiet network points to a configuration problem.
Next, generate real traffic with iperf3. Run the server side on the target machine and the client side on a peer. Push traffic for a sustained period. Watch the reported loss and retransmit counts. If loss appears only under load, the queue count is a strong suspect. Light traffic uses few queues. Heavy traffic spreads across all of them. A mismatch between queue count and CPU cores shows up exactly this way.
Repeat the test multiple times. Intermittent packet loss that appears in every run gives you a reliable pattern. A single bad run may just be noise.
Inspect Drop Counters with ethtool -S
Now look at the hardware itself. Run ethtool -S against your network interface. This command prints every driver-level counter. You care about two values: rx_dropped and tx_dropped. The rx side counts packets the NIC received but could not hand to the kernel. The tx side counts packets the NIC could not transmit.
Record these numbers. Wait a short interval. Run the command again. Compare the two readings. A counter that climbs during idle time signals a real problem. A counter that stays flat under load is fine.
Now compare the drops against your queue count. Check the queue count with ethtool -l. If you see many queues but few CPU cores, the kernel cannot service them all. Interrupts pile up. Packets wait too long and get discarded. The table below shows how to read the pattern.
| Observation | Likely Cause |
|---|---|
| rx_dropped rises, queue count exceeds CPU cores | Too many queues for available cores |
| tx_dropped rises under load | Queue too small or ring buffer exhausted |
| Both counters flat, loss still present | Look beyond the NIC, check cables and switch |
One more check helps. Run ethtool -S and grep for per-queue drop fields. Many drivers report drops per queue. If drops are concentrated in a few queues, your traffic is not balanced. That points to IRQ affinity problems, which you will fix later.
At this point you have evidence. Rising drop counters plus a queue count that does not match your CPU layout confirm the diagnosis. Move to the next step and calculate the right number.
Determine Optimal Queue Count for Network Interface Card Tuning
Match Queue Count to CPU Cores
The first rule of network interface card tuning is simple: match your queue count to the number of CPU cores that handle network interrupts. Each queue needs its own interrupt vector. If you create more queues than cores, the kernel cannot service them all in time. Packets wait, buffers fill, and drops occur. A good starting point is one queue per physical core. On a server with multiple cores, start with one queue per physical core. Check your current setting with ethtool -l. This command shows the pre-set maximums and the current active count.
You must also account for hyper-threading. Logical cores share physical execution units. Assigning one queue per logical core often creates contention. Start with one queue per physical core instead. Then test. If CPU usage stays low and drops disappear, you have found a good balance. If drops persist, you may need fewer queues, not more. Fewer queues reduce vector consumption and can actually alleviate interrupt drops on busy systems.
Consider Workload and Queue Distribution
Your workload type shapes the ideal queue distribution. High-throughput workloads, such as bulk transfer, AI training, or storage traffic, benefit from more queues and aggressive coalescing. Coalescing batches packets per interrupt, which raises throughput. Latency-sensitive workloads, such as trading or real-time interactive traffic, need the opposite approach. Disable adaptive coalescing and minimize batching to cut interrupt delay.
The table below shows how each workload type maps to coalescing strategy and example settings.
| Workload Type | Coalescing Strategy | Example ethtool Settings |
|---|---|---|
| High-Throughput | Enable adaptive coalescing or set high manual values | ethtool -C ens1f0 adaptive-rx on adaptive-tx on |
| Latency-Sensitive | Disable adaptive coalescing and minimize batching | ethtool -C ens1f0 adaptive-rx off adaptive-tx off |
A large queue helps a busy system maintain high throughput. It also introduces latency. An interactive packet can sit behind bulk flow packets when queue distribution favors throughput. Disable TSO, GSO, UFO, and GRO if you want to optimize for latency over throughput. You will likely see no CPU impact or throughput decrease unless the system handles very high data rates. To place a hard upper limit on queued bytes, write the new value to the limit_max file.
Adjust Queue Count and Verify the Fix
Using ethtool -L to Change Queue Count
You have your target number. Now apply it. The ethtool utility changes the active queue count without a reboot. Use the -L flag with the combined parameter. This sets transmit and receive queues together.
Set combined NIC transmit and receive queues to 8 with
ethtool -L$ sudo ethtool -L eth0 combined 8
For a 16-core server, you might run ethtool -L ens1f0 combined 16 to set the maximum. Always confirm the change with ethtool -l. This command shows the current active count. If the value matches your target, the change took effect. If the driver rejects the request, check the pre-set maximums. Some adapters limit the combined count to a fixed number.
Change only this one variable. Do not touch ring buffers or coalescing settings yet. A single change keeps your test results clean.
Monitor and Re-test for Packet Loss
Immediately after the change, re-check your drop counters. Run ethtool -S and look for rising rx_dropped or tx_dropped values. A flat counter means the fix worked. You can also inspect kernel-level drops through /proc/net/softnet_stat. Column 2 in that file shows dropped packets. Column 3 shows time_squeeze, which means the CPU was too busy to process traffic. Watch these counters in real time to catch new drops.
Next, run a throughput test with iperf3. Start the server, then run a single-stream client test. Follow with parallel streams to saturate the link. Compare the results to your baseline. If throughput holds steady and loss disappears, the adjustment succeeded. A server network card queue count improperly configured causing packet loss will show clear improvement here.
Latency matters too. Use ping for a basic check. For a detailed histogram, run sockperf ping-pong. This test reveals latency instability that ping might miss.
Finally, watch for side effects. Monitor /proc/interrupts to see how interrupt rates change. This file helps you verify that each receive queue maps to an appropriate CPU. However, this metric alone is not reliable for data volume, because many drivers disable NIC interrupts as part of their contract with the NAPI subsystem. Interrupt coalescing also affects these numbers. For a fuller picture, check /proc/softirqs as well. Rising CPU usage or new latency spikes mean you need to revisit your queue count. Fewer queues can reduce vector consumption and ease interrupt pressure. If drops persist after all these checks, move to the next section for deeper tuning.
Tune OS Network Stack and NIC Settings
If packet loss persists after queue count adjustment, deeper tuning helps. Two issues remain: undersized rings and unbalanced interrupt handling. Fixing both resolves the problem.
Increase Ring Buffer Sizes to Reduce Drops
Ring buffers act as temporary storage between the network interface and kernel. When traffic bursts exceed buffer capacity, the NIC drops packets. You prevent this by increasing buffer size.
Start by checking your current nic ring buffer settings. Use ethtool -g to view values. The command shows separate rx and tx ring buffers for each queue. Small values indicate room for growth.
Increase ring buffer sizes with the -G flag. Run sudo ethtool -G eth0 rx 4096 tx 4096. Set both buffers as large as possible, up to a maximum of 4096. Confirm with ethtool -g to see the new ring buffer sizes. Larger ring buffer sizes give the system extra time during bursts. This reduces drops without changing queue counts. However, larger buffers add latency. For latency-sensitive workloads, test moderate values.
Configure IRQ Affinity for Balanced Load
Even with correct queue counts and nic ring buffer size set correctly, unbalanced interrupts cause loss. Each receive queue generates an interrupt request. When multiple IRQs land on one core, it overloads while others sit idle. Configuring irq affinity spreads the load evenly.
Check interrupt distribution. Run cat /proc/interrupts | grep ethX to see which cores handle network interrupts. Use top to monitor CPU usage. An uneven distribution confirms the problem.
Debug IRQ distribution with irqbalance --debug. If balancing is off, manually set CPU core affinity for each queue. Write a core mask to /sys/class/net/ethX/queues/rx-0/rps_cpus. For example, assign queue 0 to CPU core 0 by echoing value 1 into that file. List available cores with lscpu | grep '^CPU(s):' for the correct mask.
Balanced irq affinity reduces per-core pressure and prevents drop accumulation. Combined with proper nic ring buffers, your nic settings handle traffic efficiently. If loss continues, reduce queue count further. Fewer queues cut vector consumption and ease IRQ drops. Configure QoS on the network interface to prioritize critical traffic.
Adjust linux kernel receive/send buffers at the OS level. Use sysctl to modify net.core.rmem_max and net.core.wmem_max. Set high values to accommodate bursts. This catches packets that pass the NIC but exceed kernel capacity.
You now have a four-step process. First, diagnose the issue with drop counters. Second, pick the best queue count for your hardware. Third, apply the change with ethtool. Fourth, tune the OS stack when needed. Change one variable at a time, verify each result, and document every setting. A server network card queue count improperly configured causing packet loss is a common problem, yet you can fix it. Take a few minutes to check your server’s queue count today—it could save you hours of network troubleshooting later.
FAQ
How do I confirm the queue count caused packet loss?
Run ethtool -S and watch rx_dropped and tx_dropped. If these counters rise and the queue count exceeds your CPU cores, the mismatch is likely the root cause.
What is the best queue count for my server?
Start with one queue per physical core. Check your count with ethtool -l. Adjust based on workload. More queues help throughput. Fewer queues reduce interrupt pressure.
Does changing queue count require a reboot?
No. You change the active count immediately with ethtool -L. The new value takes effect on the fly. Confirm the result with ethtool -l.
Will adjusting queues affect application performance?
Yes. More queues boost throughput for bulk transfers. Latency-sensitive apps may perform better with fewer queues and coalescing disabled. Test both settings for your workload.
What do I do if loss continues after tuning queues?
Increase ring buffer sizes with ethtool -G. Configure IRQ affinity to balance interrupts across cores. Reduce queue count further if vector consumption stays high.
