Automatic Stability and Recovery for Neural Network Training
An optimizer-agnostic runtime controller that detects destabilizing neural-network training updates and recovers by rollback.
Abstract
Training modern neural networks can be fragile: rare destabilizing updates may cause divergence or silent performance degradation after the optimizer has already acted. This paper introduces a supervisory runtime stability layer that treats optimization as a controlled stochastic process. Rather than changing the optimizer, the controller monitors proposed updates using secondary measurement signals such as validation probes, forms an innovation signal against recent stable behavior, and decides whether to accept the update or roll back to the last accepted snapshot. The framework provides runtime safety guarantees for bounded degradation and one-step recovery, and is evaluated on representative vision and sequence models including ResNet-18 on CIFAR-10 and a character-level Transformer.
Key Innovations
Optimizer-agnostic stability layer
Operates above standard optimizers and evaluates update outcomes through external measurement signals without modifying the optimizer itself.
Innovation-triggered rollback
Converts validation or probe deviations into accept-or-rollback decisions that restore both model parameters and optimizer state.
Runtime safety guarantees
Formalizes bounded degradation and one-step recovery, focusing on operational safety rather than nonconvex convergence guarantees.