How We Managed a DNSSEC Failure at the .de TLD: A Step-by-Step Response Guide

Introduction

When a top-level domain like .de—one of the largest country-code TLDs globally—suffers a DNSSEC misconfiguration, the impact can cascade across the internet. On May 5, 2026, at roughly 19:30 UTC, DENIC, the registry for .de, began publishing incorrect DNSSEC signatures. Any validating resolver, including Cloudflare's 1.1.1.1, rejected those signatures per the DNSSEC specification and returned SERVFAIL to clients, making millions of domains unreachable. This guide walks through how we responded, step by step, so you can handle a similar TLD-wide DNSSEC failure if it ever occurs.

How We Managed a DNSSEC Failure at the .de TLD: A Step-by-Step Response Guide — Source: blog.cloudflare.com

What You Need

DNSSEC-aware DNS resolver infrastructure (e.g., 1.1.1.1, BIND, Unbound)
Monitoring and alerting system for DNS query failures (SERVFAIL rates, validation errors)
Access to resolver configuration to adjust validation settings
Communication channels with upstream registry (e.g., DENIC)
Internal playbook for emergency DNSSEC bypass procedures
Logging infrastructure to capture validation failure details

Step-by-Step Response

Step 1: Detect the Anomaly

Monitor SERVFAIL spikes. In our case, alerts on Cloudflare Radar showed a sharp increase in SERVFAIL responses for .de domains. Validating resolvers like 1.1.1.1 returned failures because the signatures from DENIC couldn't be verified. Set up thresholds: e.g., >10% increase in SERVFAIL for a specific TLD compared to baseline. Use tools like Prometheus, Grafana, or built-in resolver logs to catch this early.

Step 2: Isolate the Validation Failure

Confirm the issue is DNSSEC-related. Check resolver logs for messages like “validation failure” or “no valid RRSIG”. Query the affected zone manually using dig +dnssec .de SOA. If the response fails validation but non-DNSSEC queries succeed, the problem is with signatures. In our incident, the .de zone published signatures that did not match the published DNSKEY records, breaking the chain of trust.

Step 3: Identify the Root Cause

Determine which key is misused. DNSSEC uses Zone Signing Keys (ZSK) and Key Signing Keys (KSK). A common cause is a failed key rotation: the old key is removed before new signatures propagate. In the .de case, DENIC had likely started a KSK rotation but the DS record in the root zone still pointed to the old key, or the new signatures were signed with a key not yet anchored. Check the DS records at the parent (root for TLDs) and compare with the DNSKEY in the child zone. Use delv or validns to trace the chain.

Step 4: Implement a Temporary Mitigation

Disable DNSSEC validation for the affected TLD. This is a last resort but necessary to restore service while the registry fixes the issue. On Cloudflare's 1.1.1.1, we added an exception to bypass validation for the .de zone. In BIND, you would add a validation exemption in named.conf using trust-anchors with an empty key or set dnssec-validation no; for that zone. In Unbound, use val-override-trusted-keys with a manual trust anchor or val-permissive-mode yes;. Document the change and set a timer to revert once fixed.

Important: This makes queries for .de domains insecure—they lose integrity verification. Communicate this to internal teams and potentially to users via status pages.

Step 5: Monitor and Communicate

Watch for resolution recovery. After applying the bypass, confirm that SERVFAIL rates drop for .de. Simultaneously, contact the registry (DENIC) to report the issue and get an ETA. Use established escalation paths: emails, phone calls, or registry forums. Provide them with specific error details: which signatures failed, timestamps, and affected DNSKEYS. Keep internal stakeholders updated via incident channels.

Step 6: Re-enable DNSSEC Once Fixed

Remove the bypass after registry confirms fix. DENIC corrected the signatures by republishing the .de zone with valid RRSIG records. Validate the fix with a few test domains (e.g., example.de) using a non-bypassed resolver. Verify that dig +dnssec returns AD flag and no validation errors. Once confident, revert the configuration change. For Cloudflare, we removed the .de exception and confirmed that 1.1.1.1 began validating again without SERVFAIL spikes.

Tips for Future Preparedness

Automate detection: Use anomaly detection on validation failure rates per TLD to catch issues within minutes.
Predefine bypass zones: Have a configuration stub ready for critical TLDs to quickly disable validation if needed.
Test key rotations: If you operate a TLD or signed zone, always test KSK rotations in a staging environment before production. The .de outage likely came from a botched rotation.
Maintain contacts: Keep up-to-date contact information for all major TLD registries (DENIC, Verisign, etc.) to expedite communication.
Document every action: During an incident, log all changes and timestamps. After resolution, conduct a post-mortem to refine your response playbook.
Balance security and availability: Bypassing DNSSEC is a trade-off. Use it only as a temporary measure and always inform affected parties.

By following these steps, you can minimize disruption during a TLD-level DNSSEC outage. The .de incident taught us that even well-maintained registries can fail, but a rapid, structured response keeps the internet running.

Tags: