Urgent recovery when a message queue server’s disk is full

Run df -h and du -sh right now. Confirm the mount point. Identify the top consumer of space. A message queue server’s disk is full when storage hits zero. This condition halts ingestion, triggers broker shutdown, and risks message loss. Transient queues lose everything after an index rebuild on restart. You face insufficient disk space, and every minute counts.

This runbook follows a strict order. Diagnose the problem. Clear space by hand. Recover the broker. Prevent the next outage. Check your log files and queues first. Verify storage, not queue counts. Move fast, but delete nothing before you know what fills the disk.

Diagnose a message queue server’s disk is full

Confirm disk usage and mount point

Start with df -h. This command shows every mount point and its usage percentage. You need the exact mount point that hit 100%. A message queue server’s disk is full on one volume, not all of them. Next, run du -sh against /var/lib, /var/log, and your broker data directories. This ranks consumers by size and points you toward the underlying cause.

Watch for a full volume that holds only a small queue directory. The message queuing folder can sit on a separate mount from the broker’s storage directory. Check both. A full disk often stems from insufficient disk space, permission problems, or file system errors in a path you did not expect. Confirm the mount point before you delete anything.

Find top space consumers and queue depth

Queue depth alone misleads you. Use management tools to inspect depth and message contents. Prometheus exposes the rabbitmq_queue_messages metric for parking lot and wait queues. Dead-letter rate counters reveal changes sooner than depth. Read the x-death header to count retries and separate transient failures from permanent ones. Page when the dead-letter rate exceeds 1% of publish rate over 5 minutes.

Storage can be full while queues look empty. This is the MSMQ paradox. Verify storage, not queue counts. Index overhead is a major cause:

Check disk queue length as an I/O signal. A high value means the disk subsystem is overwhelmed. Old log files and temporary files also consume space. Review the log directory and the queue directory for stale data. Insufficient disk space or memory on the host worsens the problem. Track disk space and memory together to find the root cause of the issue.

Manually clearing the message queue safely

Delete old logs and temporary files

Start with the easiest targets. Once you confirm the message queue server’s disk is full, the /var/log directory holds rotated logs that you can safely delete. Run du -sh /var/log/ to find the largest log files. Remove archives older than a week. The broker’s own rotation path — /var/log/rabbitmq/ or /var/log/kafka/ — also contains old *.gz files. Delete them. Temporary items in /tmp or the broker’s temp directory consume space too. Clear them.

If the broker is unreachable due to a full disk, reboot into recovery mode. On Ubuntu, edit the GRUB command line to boot into single-user mode. This gives you a shell to delete log files from /var/log/ without interference.

When a separate filesystem has free space, move older send-summary files instead of deleting them. For GreenArrow, use hvmail_move_old_send_summary_files:

1. Confirm the script exists at /var/hvmail/bin/hvmail_move_old_send_summary_files.
2. Estimate required free space with du -hs on the send-summary directory.
3. Create a destination directory, for example mkdir /media/scratch/var-hvmail-log-send-summary.
4. Run the command with a minimum age of 7 days and the destination path.
5. Ensure backups cover the new location. Managed Backups handle this automatically.

Respect the storage quota for each directory. Do not delete the active broker log if you cannot rotate it first. Deleting old rotated data can free significant space before you touch any queue data. This gives you enough space to proceed safely. Insufficient disk space in the storage volume can prevent the broker from writing new entries. This single step can recover enough space for the broker to restart cleanly. Check for insufficient disk space or memory on the host machine before restarting the broker.

Truncate dead-letter queues and expired messages

After clearing logs, address the queue data. The message queuing folder — /var/lib/rabbitmq/mnesia or equivalent — holds the queue data. Insufficient disk space in this location can corrupt indexes. Use broker management tools to purge dead-letter queues. For RabbitMQ, rabbitmqctl purge_queue works. For Kafka, use kafka-delete-records with a target offset. Confirm with the business that these messages are not needed for replay. Only then clear the queue.

Never delete active message data files directly from the filesystem. This corrupts the index and causes data loss. Use broker commands instead. For MSMQ, the storage directory can fill while queues appear empty. MSMQ uses a storage system with .stg and .mq files that consume space even when the message count shows zero. Verify the actual disk usage by running dir on that folder.

For PostgreSQL-backed brokers, old WAL files consume significant storage. Identify the root cause of WAL growth first — inactive replication slots, archive failure, or bulk loads. If a replication slot is the cause and the consumer is gone, drop the slot with SELECT pg_drop_replication_slot('slot_name') after confirming it is inactive. Then run CHECKPOINT; to recycle unneeded WAL segments. If the database stops due to a full disk, expand the WAL volume, start PostgreSQL, and investigate. As a last resort, use pg_archivecleanup with a dry run to remove old segments. A common trap is insufficient disk space, permission problems, or file system errors that masquerade as queue issues. Avoid resetting the queue by deleting data files directly.

After purging dead-letter queues, run the broker’s maintenance commands. For RabbitMQ, rabbitmqctl force_gc triggers compaction. For Kafka, use the partition size tool. Watch the queue depth for a few minutes to ensure new messages start flowing. Do not restart the broker yet. Verify the queue depth. The goal of manually clearing the message queue is to create a stable foundation for broker recovery.

Recover the broker and verify integrity

Restart the broker and confirm message flow

Start the broker only after you free enough space. Run your service manager command and watch the startup log for errors. A disk-full condition can trigger linear logging behavior, which changes how the broker processes messages on disk. Read the log output line by line. Confirm that producers reconnect and consumers resume delivery.

If rsyslog disk queues activated during the incident, verify they drain correctly. Check the queue depth after restart. A queue that stays flat signals a blocked consumer or a permission problem in the storage directory. Watch the log for repeated retry messages. Those retries point to a poisoned payload that needs routing to a dead-letter queue.

Rebalance partitions or queues if needed

A restart can leave partitions unevenly distributed. Check broker metrics for hot spots. Move partitions or queues to underused nodes when one broker carries most of the load. This step matters after you clear a large backlog, because uneven distribution can refill one volume fast.

Verify message integrity before you declare recovery complete. Dead-letter queues handle poisoned messages, and they gain renewed importance in agentic architectures. When an agent cannot validate a message’s semantic integrity, it routes the payload to a dead-letter queue for inspection instead of dropping it or retrying forever. This surfaces integrity failures before recovery ends.

Message signing and verification ensures authenticity and prevents tampering via digital signatures. A message hash is signed with a private key using PSS padding, producing a signature that is base64-encoded. The corresponding verification function would validate this signature before the message is processed — confirming integrity prior to declaring recovery complete.

For batch verification, each receiving module checks every message batch with the matching algorithm, verification tag, and key. A pass confirms correct processing and clears intermediate data. A fail triggers an alarm, inspects the error record, resends the message, and reprocesses it. Tag-based schemes select message data by identifiers and verify integrity against tag values. Confirm all three checks pass before you close the incident.

Prevent the next disk-full incident

Expand LVM volumes and set quotas

You survived this incident. Now you must stop the next one. The fastest preventive move is to expand your LVM volume before the disk fills up. This prevents crashes from disk space exhaustion. Follow these steps:

1. Check the current LVM setup by running sudo lvs, sudo vgs, and sudo pvs to verify free space in the volume group.
2. Extend the logical volume using all available free space: sudo lvextend -l +100%FREE /dev/mapper/vg-lv_root.
3. Resize the filesystem to match the extended volume. For ext4, run sudo resize2fs /dev/mapper/vg-lv_root. For XFS, run sudo xfs_growfs /.

Set disk quotas and retention policies per queue and per log directory. A storage quota caps growth at the source. Apply a queue quota threshold so no single queue consumes the whole volume. For Kafka, size-based and time-based retention work together. Kafka applies whichever condition is met first, giving you dual protection against time and storage constraints. Set log.retention.bytes to cap total segment size per partition and log.segment.bytes to limit individual segments. Set log.roll.hours to create new segments on a schedule. Align retention with business needs: real-time data at 7 to 30 days, analytical data at 90 to 365 days, and compliance data at 3 to 7 years. Retaining data over 90 days increases disk usage linearly, so 30 to 90 days is often the sweet spot.

Configure retention policies and email alerts

Linux does not natively emit an event when disk space nears full. Any solution requires polling. Use cron with a notification tool such as ntfy to check disk usage and send alerts when thresholds are exceeded. Search for existing scripts that poll disk space and trigger email notifications. Consider full monitoring solutions such as Zabbix or Netdata, which send email alerts at multiple thresholds. Define tiered notification flows by severity: 75% Warning, 80% Average, 85% High, 90% Critical, 95% Disaster. Route messages to different channels depending on the level.

Set monitoring alert thresholds at 70% and 85%. A warning fires when store disk usage exceeds 70% for 10 minutes. A critical alert fires when usage exceeds 85% for 5 minutes. At 100%, all persistent message sends fail. Alert on fill trajectory, or days-to-full, rather than raw fullness. The exact numbers matter less than having both a warning and a critical tier.

Automate cleanup tasks and monitor their effects. Apply a message TTL policy so messages older than the threshold are removed automatically. Set a maximum queue length policy with overflow behavior such as drop-head or reject-publish. These mechanisms keep disk usage bounded. Scrape broker metrics with Prometheus to track available disk space. Define a warning alert when free disk space drops below 10 GB for 5 minutes. Define a critical alert when free disk space drops below 3 GB for 1 minute. Route alerts through Alert Manager to PagerDuty or Slack. Verify no alarms are active by checking rabbitmqctl status. Confirm publishers are no longer blocked. Periodically monitor disk space with df -h combined with rabbitmqctl status | grep disk_free. Watch for insufficient disk space or memory on the host. Check your log files and storage directory for stale data. Delete old log files on a schedule. This keeps your queues healthy and your storage predictable.

Recover in four steps: diagnose, clear space by hand, restart the broker, prevent recurrence. The fastest path forward starts with knowing what consumes disk space before you delete anything. Transient queues and MSMQ storage can fill a volume while queue counts read zero, so verify the storage directory itself. You now hold a repeatable runbook.

Look ahead. Expand LVM volumes before they fill. Automate retention so old log files and stale files leave on schedule. Rotate every log on a timer. Set email alerts at 70% usage, not 100%. A message queue server’s disk is full only once if you catch it early. Keep it close.

FAQ

How do I confirm the exact cause of the full disk?

Run df -h to identify the full mount point. Use du -sh on /var/lib, /var/log, and broker data directories. This reveals whether old log files, queue indexes, or stored messages consume your space.

Can I delete message data files directly from the filesystem?

Never delete active message data files directly. This action corrupts broker indexes and causes permanent data loss. Use broker management commands such as rabbitmqctl purge_queue or kafka-delete-records instead. Clear space through approved tools only.

Why does MSMQ show empty queues but a full disk?

MSMQ uses .stg and .mq files that persist even when message counts read zero. Run dir on the storage folder to verify real usage. Always check the storage directory itself, not queue counts.

What monitoring thresholds should I set?

Set warnings at 70% usage and critical alerts at 85%. Monitor fill trajectory, not raw percentages. Configure notifications through Prometheus and Alert Manager for automated response.