Your GPT-6 training run on the ai training cluster just crashed after 10 days. The screen is dead. Your heart races. Is all that progress gone? Resume progress needs planning. Fear of losing weeks of compute time and facing downtime is real.

This playbook provides practical steps. First, understand what state was lost. Then save and restore it to resume training. Adjust the learning rate. Finally, verify success.

Understanding the Interrupted Training Session

Common Causes of AI Training Cluster Failures

Your training run stops for many reasons. Hardware failures top the list. A GPU can overheat and shut down. Memory modules can develop errors. Power supplies can fail without warning. Software bugs also cause crashes. A subtle coding error might corrupt the training loop after days of operation. Network issues can disconnect nodes from the cluster. Each failure type produces the same result: your training job halts mid-process.

The impact extends beyond the immediate stop. Your cluster sits idle. Other jobs wait in queue. Your team loses productive hours. This downtime costs more than compute credits. It disrupts your research momentum and delays your project timeline. Understanding these causes helps you prepare for the inevitable interruption. You cannot prevent every failure, but you can control your response.

What State is Lost When Training Stops?

Many engineers assume they only need the model weights. This assumption causes failed recovery attempts. True resumption requires restoring the entire training state. Your model parameters represent one piece. The optimizer state holds critical data. For Adam optimization, this includes momentum and variance estimates for every parameter. Without these values, your optimizer behaves like a fresh start. The learning rate scheduler also holds position information. It knows which step of the decay schedule you reached.

The random number generator state matters equally. Your training process uses randomness for data shuffling and augmentation. Each epoch shuffles your dataset differently. Image transformations apply random crops and flips. The RNG state determines these operations. Without restoring it, your data pipeline produces different sequences. This inconsistency creates subtle training variations. Your model might converge differently than expected. The validation results may not match your pre-crash logs.

A complete checkpoint captures all these components. You save the model, optimizer, scheduler, and RNG states together. This comprehensive approach makes recovery straightforward. You restore everything to the exact pre-crash condition. Your training continues as if the interruption never happened. This preparation transforms a potential disaster into a minor inconvenience. The time you invest in proper checkpointing pays dividends during every future failure.

Checkpointing to Resume Progress

Saving the Full Model and Optimizer State

A checkpoint must capture every piece of your training state. You cannot save only the model weights. Your optimizer holds momentum values for each parameter. Your scheduler tracks its position in the decay curve. Your random number generator determines data shuffling patterns. Each component plays a role in seamless recovery.

You create a comprehensive checkpoint with a single command. The dictionary structure keeps everything organized. You save the current epoch number to know where to restart. You store the model state dictionary for your architecture. You include the optimizer state dictionary with its accumulated gradients. You add the scheduler state to preserve your learning rate plan. You record the RNG state for reproducible data pipelines. This complete snapshot enables true resume progress after any failure.

torch.save({
    'epoch': epoch,
    'model_state_dict': model.state_dict(),
    'optimizer_state_dict': optimizer.state_dict(),
    'scheduler_state_dict': scheduler.state_dict(),
    'rng_state': torch.get_rng_state()
}, 'checkpoint.pth')

The size of this file grows quickly with your model. The pattern follows a simple rule: 12 bytes per parameter when you include optimizer states. Weights consume 4 bytes per parameter. Optimizer states consume 8 bytes. This scaling rule helps you plan storage capacity before you start your training job.

Storing Checkpoints for Fast Recovery

Your storage choice determines your recovery time. A distributed file system like HDFS or S3 offers durability. A fast local NVMe drive offers speed. You need both for different purposes. Active training writes to parallel file systems. Long-term archival moves to object storage at one-tenth the cost per terabyte.

The checkpoint interval requires careful thought. Frequent saving reduces re-computation after a crash. Infrequent saving reduces GPU-blocking time during the write operation. You must balance these competing demands. Saving every 5 minutes often proves excessive. Saving every hour works well for multi-day training runs with rare restarts.

The checkpoint interval should be proportional to the job’s crash rate (mean time to interrupt), which scales with the number of GPUs in use.

Large-scale training runs follow this principle in practice. A 405B-parameter model with a 150-minute mean time to interrupt checkpoints every 15 minutes. An 800B model checkpoints every 40 minutes. The interval equals roughly one-tenth of the expected time between failures. This approach minimizes wasted compute without excessive overhead.

Modern LLMs write 350–500 GB of model state during each checkpoint. The write must complete in under 5 minutes to minimize training interruption. This requirement translates to a sustained burst write bandwidth of 1–2 GB/s. Storage architecture must prioritize burst performance over sustained throughput.

Systems like TrainMover push recovery even further. The replacement machine compiles CUDA kernels and establishes NCCL groups before any failure occurs. This pre-warming eliminates checkpoint loading from the critical path. The delta-based design updates only leaver–joiner connections during recovery. All other machines remain untouched. This architecture keeps downtime nearly constant at 20 seconds or less, even at 1,024-GPU scale. Zero GPU memory overhead results from keeping preparation state in CPU memory or NVMe. GPU memory stays untouched until the final switchover.

Asynchronous checkpointing offers another mitigation strategy. You offload data from VRAM to host RAM. GPUs resume training while CPUs handle background saving. Analysis of 85,000 checkpoints shows this keeps global bandwidth under 1 TB/s. Local offloads operate at 50–200 GB/s. This approach prevents GPU stalls during save operations.

The cost of idle GPUs during checkpointing can exceed $4,000 per day for large-scale training. Downtime optimization directly reduces this expense. Every second you shave from the recovery process saves money across your entire cluster. Your checkpointing strategy deserves the same attention as your model architecture. The time you invest in robust saving practices pays returns during every future interruption.

Restoring the AI Training Cluster State

Loading the Checkpoint and Resuming the Loop

Your recovery begins with a single loading operation. You read the saved checkpoint file back into memory. This action restores every component you saved before the crash. The model weights return to their pre-failure values. The optimizer regains its momentum estimates. The scheduler recovers its position in the decay curve. Your training environment returns to its exact pre-crash condition.

checkpoint = torch.load('checkpoint.pth')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
scheduler.load_state_dict(checkpoint['scheduler_state_dict'])
start_epoch = checkpoint['epoch'] + 1

The critical detail appears in that final line. You must start from the saved epoch plus one, not from zero. Your training loop needs adjustment to honor this starting point. You modify the loop range to begin at the restored epoch. This simple change prevents the wasted computation of repeating completed epochs. Your job resumes exactly where it stopped, preserving all prior progress.

for epoch in range(start_epoch, total_epochs):
    train_one_epoch(model, train_loader, optimizer, scheduler)
    validate(model, val_loader)
    save_checkpoint(model, optimizer, scheduler, epoch)

Frameworks like fastai simplify this process further. The fit_one_cycle function applies a cyclical learning rate and momentum policy. Without proper handling, an interruption restarts this cycle from the beginning. Your results would differ from uninterrupted training. Fastai solves this problem with the start_epoch parameter. You re-instantiate your learner and call fit_one_cycle with the next epoch number. The framework automatically loads the previously saved file for that epoch. Your learning rate and momentum cycle continue from that exact point. No manual weight reloading required. The training policy continues correctly without disruption.

Re-establishing Data Loaders and Random Seeds

Your model restoration means nothing without matching data pipelines. The random number generator state determines how your data loader shuffles samples. Each epoch produces a different order. Image augmentations apply random transformations. These operations depend entirely on the RNG state you saved earlier.

You restore this state with a simple call. Pass the saved RNG values back to PyTorch. Your data loader then produces the identical sequence it would have generated without the crash. This reproducibility matters for validation consistency. Your loss curves remain comparable across the interruption boundary. Your gradient patterns stay predictable.

torch.set_rng_state(checkpoint['rng_state'])

You must also restore any worker-specific seeds. Data loaders often use multiple worker processes. Each worker maintains its own random state. You need to set these seeds before creating your new data loader instances. This step ensures every worker reproduces its original shuffling pattern. Your complete training environment now matches the pre-crash configuration.

The verification step confirms your restoration worked correctly. Run a few batches and compare the loss values. They should match your pre-crash logs closely. Gradient norms should fall within expected ranges. Any significant deviation signals a restoration error. You catch these problems early before wasting hours on corrupted training.

Your checkpoint loading strategy directly impacts your recovery time. A well-organized checkpoint structure enables rapid restoration. You locate the correct file quickly. You load it without parsing complications. Your job returns to full operation within minutes. This efficiency transforms a potential disaster into a brief interruption. Your training continues with minimal downtime and maximum confidence.

Adjusting the Learning Rate to Resume Training

The learning rate scheduler holds critical information about your training progression. It tracks your position in the decay sequence. Restoring the model and optimizer without the scheduler state does not complete your recovery. The scheduler must return to its pre-crash position. A fresh scheduler disrupts your entire plan. Your model expects the lower rate from the later stage.

Calculating the Correct Step and Epoch

Your scheduler uses steps or epochs to determine the rate value during training. You need the exact position when the crash occurred. Use this formula: current_step = epoch * len(train_loader) + batch_index. This calculation gives you the precise step number in your schedule. Set the last_epoch parameter to your calculated step value. The scheduler then continues from that exact location. The rate curve stays smooth across the interruption boundary. This approach ensures your model receives the correct rate for every remaining step.

The epoch number matters for multi-cycle schedules. Cosine annealing and one-cycle policies depend on total cycle length. Your saved checkpoint contains the last completed epoch. Add one to this value. Pass it to your scheduler constructor. Your scheduler then calculates the remaining decay correctly. This step removes guesswork from your recovery process.

Re-initializing the Scheduler for Continuity

Do not create a new scheduler after a crash. A fresh scheduler resets your learning rate to the initial value. Your model now expects a lower rate from a later stage. A sudden high rate causes gradient instability. Your loss may spike. Your weights may diverge. This mistake wastes your entire recovery effort and forces another restart.

Creating a fresh scheduler after a crash resets your learning rate schedule. This spike can destabilize your model and undo hours of training progress. Always restore the saved scheduler state instead.

Create your scheduler and load its saved state. The scheduler.load_state_dict() method restores all internal parameters. The scheduler returns to its exact step count. Your rate continues its planned trajectory as if no failure occurred. Your job proceeds without missing a step.

A proper scheduler restoration eliminates this risk. Your training continues with the same dynamics. Your validation metrics remain comparable. Apply this practice to every checkpoint. The time to save the scheduler state is minimal. The cost of omitting it is substantial. This attention to detail turns a crash from a significant setback into a brief pause.

Verifying Successful Resumption

Running a Sanity Check on Loss and Gradients

After you restore your checkpoint, run a small number of batches. Compare the loss values to the numbers you logged just before the crash. The loss should remain close to the last recorded value from that epoch. A close match confirms your training state returned correctly. A significant jump indicates a problem with your checkpoint loading procedure.

You also need to examine the gradient norms. These values measure the size of your weight updates. Compare them to your pre-crash logs. Similar norms indicate a successful restoration. Different norms signal a problem. Your scheduler state might not match your pre-crash configuration. Your random number generator could differ from the original state. These subtle mismatches cause inconsistent results.

A large jump in loss or gradient values means something went wrong. Your checkpoint might contain corrupted data. This quick check catches issues early. You avoid wasting hours on incorrect training. You save valuable compute time.

Use a comparison approach for consistency. Log the loss and gradient norms from your pre-crash step. Run your verification on the same number of batches. Record the results. Compare them side by side. The values should show close agreement. This process takes only a few minutes but saves substantial time later.

Monitoring for Divergence in Early Steps

Watching the first few steps after restoration provides the strongest signal. Your training should proceed exactly as it did before the interruption. Your learning curve should continue its downward trend. Your gradient patterns should remain stable.

Watch for specific warning signs. Exploding gradients appear as sudden large jumps in parameter updates. NaN values indicate numerical instability. Both conditions result from a restoration error. Your training will not recover from these problems on its own. You must catch them early before wasting further resources.

If you observe divergence, stop your job immediately. Do not let it continue without investigation. Check your checkpoint file for completeness. Verify each component loaded correctly. Confirm your RNG state matches your original environment.

A clean restoration shows no divergence. Your loss curve stays smooth across each epoch. Your gradient norms remain predictable. Your learning process remains stable. You have successfully recovered from the failure. Your job proceeds as expected. Your downtime ends. Your progress continues.

This verification process completes your recovery. You can trust your model results moving forward.

You now understand the complete recovery process. Save the full state—model, optimizer, scheduler, and RNG. Implement robust checkpointing with appropriate intervals. Restore your environment precisely. Adjust the learning rate schedule correctly. These steps transform a server failure into a manageable interruption.

Your training job survives crashes when you prepare properly. The time invested in checkpointing saves countless hours later. Recovery time shrinks from days to minutes. Your session resumes exactly where it stopped. No wasted compute. No lost progress.

Every epoch you save represents insurance against downtime. Every learning rate adjustment maintains stability. Every checkpoint protects your model’s trajectory.

With this playbook, you can face the next server failure with confidence, knowing that your progress is safe and your path forward is clear. Implement these practices before the next crash, not after.

FAQ

How often should I save checkpoints during training?

You should save checkpoints at intervals proportional to your expected crash rate. For a job with a 150-minute mean time to interrupt, save every 15 minutes. This approach minimizes wasted compute without excessive overhead. Your storage speed determines what interval works best.

What happens if I only save the model weights?

You lose critical optimizer state, including momentum values for Adam. Your learning rate scheduler resets to the beginning. Your random number generator produces different data shuffling patterns. These missing pieces cause inconsistent training results. Always save the complete state dictionary for true recovery.

Can I resume training on a different number of GPUs?

Yes, you can resume with a different GPU count. Your checkpoint stores model parameters and optimizer states that transfer across configurations. You must adjust your batch size and learning rate accordingly. The data loader distribution changes, but your saved epoch and model state remain valid.

How do I know my restoration worked correctly?

Run a few batches and compare loss values to your pre-crash logs. Gradient norms should match closely. Watch for NaN values or exploding gradients in the first steps. Any significant deviation signals a restoration error. Stop immediately and investigate before wasting compute time.

Does checkpointing slow down my training?

Checkpointing adds overhead, but modern systems minimize this cost. Traditional storage requires 5-10% of training time for saving. Fast NVMe tiers reduce this to under 1%. Asynchronous checkpointing offloads data to host RAM, letting GPUs continue working during the save operation.