Introduction: A New Era of Resilience
Over the past six months, Cloudflare has undertaken a major engineering initiative, internally dubbed "Code Orange: Fail Small", aimed at making its infrastructure more resilient, secure, and reliable for every customer. The project concluded earlier this month, marking a significant milestone in preventing outages like those experienced on November 18 and December 5, 2025. While network resilience is an ongoing journey, the steps taken now ensure that future failures are detected and contained before they escalate.
This article explores the key improvements implemented, including safer configuration changes, reduced failure impact, and enhanced communication during incidents. Learn how these changes directly benefit your traffic and operations.
Safer Configuration Changes
One of the most critical upgrades addresses how configuration changes are deployed across Cloudflare’s global network. Previously, internal configuration updates could propagate instantly—sometimes causing widespread impact before anyone noticed a problem. Now, most changes undergo a health-mediated deployment process that gradually rolls them out while monitoring real-time system health.
The Core Innovation: Snapstone
At the heart of this improvement is a new internal component called Snapstone. This system packages configuration changes into bundles and releases them incrementally, applying health mediation principles. Before Snapstone, implementing such granular control was possible but required significant per-team effort and was inconsistently applied. Snapstone provides a unified framework that includes:
- Progressive rollout – Changes are phased across the network rather than deployed all at once.
- Real-time health monitoring – Observability tools continuously check for anomalies.
- Automated rollback – If a problem is detected, the system reverts the change before it affects customer traffic.
The flexibility of Snapstone is key: it can manage any unit of configuration, whether a data file (like the one behind the November outage) or a control flag (as in the December incident). Teams define these units on demand, making the tool adaptable to future failure modes.
Reducing the Impact of Failures
Beyond safer deployments, Cloudflare has implemented strategies to limit the blast radius of any single failure. This involves redesigning critical services so that a problem in one region or feature does not cascade across the entire network. For example, by isolating control-plane functions from data-plane operations, a configuration mistake in one area no longer compromises traffic for all customers. These architectural changes complement the health-mediated deployments and ensure that even if a change goes awry, only a small subset of traffic is affected—a core tenet of the "fail small" philosophy.
Revamped Incident Management and Break-Glass Procedures
When emergencies happen, rapid response is vital. Cloudflare has revised its "break-glass" procedures—the emergency access protocols used during major incidents. These now include clearer escalation paths, pre-authorized actions for known failure scenarios, and faster communication channels between engineering teams. Additionally, incident management has been strengthened with post-mortem automation, ensuring that lessons learned from each event are systematically captured and turned into actionable improvements.
Preventing Drift and Regressions
Resilience is not a one-time fix; it requires continuous vigilance. To prevent drift—where systems slowly deviate from safe configurations—and regressions, Cloudflare has introduced automated compliance checks and periodic resilience audits. These measures verify that all configuration pipelines adhere to the new health-mediated deployment standards. If a team attempts to bypass Snapstone or revert to older, riskier practices, the system flags the deviation. This ensures that the improvements from Code Orange remain effective over time.
Transparent Communication During Incidents
Finally, Cloudflare has overhauled how it communicates with customers during outages. Recognizing that silence breeds uncertainty, the company now provides more frequent, structured updates via status pages, email, and social media. These updates include clear timelines, expected resolution windows, and explanations of impact. For premium customers, dedicated incident channels offer real-time access to engineering leads. The goal is to turn every outage into an opportunity to build trust through transparency.
What This Means for You
The practical benefit for Cloudflare customers is immense. With health-mediated deployments and Snapstone, configuration mistakes are caught early—often before they reach your traffic. The automated rollback means that even if a problem emerges, it is contained and reversed within minutes. As a result, you can expect fewer unplanned outages, faster recovery times, and greater predictability in network behavior.
These improvements also mean that future incidents will be smaller in scope and shorter in duration. By adopting Snapstone and other measures, Cloudflare has built a stronger, more resilient network—one that is prepared to fail small and recover fast.
Looking Ahead
While Code Orange: Fail Small is complete, resilience remains a never-ending journey. Cloudflare will continue to refine these tools, expand health-mediated deployments to all configuration types, and learn from every incident. For customers, this commitment translates into a more reliable platform that takes proactive steps to protect your traffic and data.
Stay tuned for further updates as we build on this foundation, ensuring that the network you rely on grows stronger every day.