Reminder DNSSEC Root KSK roll today

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Reminder DNSSEC Root KSK roll today

Viktor Dukhovni

In case you've not seen this many other places, just a friendly
reminder that ICANN is rolling the DNSSEC root KSK today.  Make
sure your resolver (if it is validating) is ready.  If you're
forwarding queries to an upstream resolver, you might also check
that the upstream is ready.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

Viktor Dukhovni
On Thu, Oct 11, 2018 at 11:24:13AM -0400, Viktor Dukhovni wrote:

> In case you've not seen this many other places, just a friendly
> reminder that ICANN is rolling the DNSSEC root KSK today.  Make
> sure your resolver (if it is validating) is ready.  If you're
> forwarding queries to an upstream resolver, you might also check
> that the upstream is ready.

The new root zone is now live, with the DNSKEY RRset signed with
KSK2017 (id 20326), rather than KSK2010 (id 19036):

    Before: http://dnsviz.net/d/root/W79zYQ/dnssec/
    During: http://dnsviz.net/d/root/W79zmg/dnssec/
    After:  http://dnsviz.net/d/root/W790GQ/dnssec/

As cached data expires, this should make its way into all working
caches over the next day or two.

--
  Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

pg151


On Thu, Oct 11, 2018, at 9:40 AM, Viktor Dukhovni wrote:
> On Thu, Oct 11, 2018 at 11:24:13AM -0400, Viktor Dukhovni wrote:
>
> > In case you've not seen this many other places, just a friendly
> > reminder that ICANN is rolling the DNSSEC root KSK today.  Make
> > sure your resolver (if it is validating) is ready.  If you're
> > forwarding queries to an upstream resolver, you might also check
> > that the upstream is ready.

Thx for the reminder ... seems quite timely!

Can you comment just a bit further on 'ready'?

Literally not long after I received your notice above bout the roll, here, all queries stopped working, and server can't be restarted.

Logs on, e.g.,

        dig A google.com

contain

        ...
        Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.435 resolver: debug 1: fetch: google.com/A
        Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 dnssec: info: view internal:   validating com/DS: bad cache hit (./DNSKEY)
        Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 lame-servers: info: broken trust chain resolving 'google.com/A/IN': 2001:4860:4802:36::a#53
        Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 query-errors: debug 1: client @0x7efc441cd640 ::1#63498 (google.com): view internal: query failed (SERVFAIL) for google.com/IN/A at query.c:10692
        ...

Which seems related to the key roll.

Changing my local dns (named) config to

        - dnssec-enable     yes;
        + dnssec-enable     no;
                dnssec-lookaside  no;
        - dnssec-validation yes;
        + dnssec-validation no;

gets me back up & running, without DNSSEC of course.

> As cached data expires, this should make its way into all working
caches over the next day or two

Is 'ready' simply .... 'wait awhile' ?

Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

Jim Reid
On 11 Oct 2018, at 18:27, [hidden email] wrote:

>
> Changing my local dns (named) config to
>
> - dnssec-enable     yes;
> + dnssec-enable     no;
> dnssec-lookaside  no;
> - dnssec-validation yes;
> + dnssec-validation no;
>
> gets me back up & running, without DNSSEC of course.

Although switching off DNSSEC validation will keep the mail flowing, it only kludges around the underlying problem. Which might or might not be related to the rollover of the root KSK a few hours ago. It’s hard to tell from the information you’ve provided. That said, you do appear to have a DNS server misconfiguration which is causing DNSSEC validation to fail. Clearly it would be wise to fix that before turning DNSSEC validation on again.

The switch to the new KSK seems the most likely cause, assuming DNSSEC validation always worked for you before then.

> Is 'ready' simply .... 'wait awhile’ ?

Maybe, maybe not. It depends on what is broken in your DNSSEC setup. If you’ve hard-wired the now dead root KSK, waiting a while won’t help. That key will still be dead when you re-enable DNSSEC validation. No matter how long or short you wait.

Consult ICANN’s web pages on the root KSK rollover. They have info on how to check that DNS configurations handle the KSK rollover properly and how to troubleshoot them when they don’t.
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

Viktor Dukhovni
In reply to this post by pg151
On Thu, Oct 11, 2018 at 10:27:57AM -0700, [hidden email] wrote:

> Can you comment just a bit further on 'ready'?

By "ready" I mean that it has a working "rfc5011" key rollover
implementation, and so has already long added KSK2017 to its list
of root trust anchors.  Or alternatively, that some attentive
administrator has manually updated the trust-anchor file.

> Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.435 resolver: debug 1: fetch: google.com/A
> Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 dnssec: info: view internal:   validating com/DS: bad cache hit (./DNSKEY)
> Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 lame-servers: info: broken trust chain resolving 'google.com/A/IN': 2001:4860:4802:36::a#53
> Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 query-errors: debug 1: client @0x7efc441cd640 ::1#63498 (google.com): view internal: query failed (SERVFAIL) for google.com/IN/A at query.c:10692
> ...

This does not look like a forwarder problem, your own trusted key
list is not up to date.  Either it is manually maintained, or
automated updates are failing (perhaps permission problems to update
the files, the keys need to be writable by the running nameserver).

> Which seems related to the key roll.

Definitely.

> Changing my local dns (named) config to
>
> - dnssec-enable     yes;
> + dnssec-enable     no;
> dnssec-lookaside  no;
> - dnssec-validation yes;
> + dnssec-validation no;
>
> gets me back up & running, without DNSSEC of course.
>
> > As cached data expires, this should make its way into all working
> caches over the next day or two
>
> Is 'ready' simply .... 'wait awhile' ?

Waiting won't fix anything, you need to fix the trust anchor
management on your system.

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

Jamie Nelson
This may help

https://www.icann.org/dns-resolvers-checking-current-trust-anchors

Jamie


October 11, 2018 11:59 AM, "Viktor Dukhovni" <[hidden email]> wrote:

> On Thu, Oct 11, 2018 at 10:27:57AM -0700, [hidden email] wrote:
>
>> Can you comment just a bit further on 'ready'?
>
> By "ready" I mean that it has a working "rfc5011" key rollover
> implementation, and so has already long added KSK2017 to its list
> of root trust anchors. Or alternatively, that some attentive
> administrator has manually updated the trust-anchor file.
>
>> Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.435 resolver: debug 1: fetch: google.com/A
>> Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 dnssec: info: view internal: validating
>> com/DS: bad cache hit (./DNSKEY)
>> Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 lame-servers: info: broken trust chain
>> resolving 'google.com/A/IN': 2001:4860:4802:36::a#53
>> Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 query-errors: debug 1: client
>> @0x7efc441cd640 ::1#63498 (google.com): view internal: query failed (SERVFAIL) for google.com/IN/A
>> at query.c:10692
>> ...
>
> This does not look like a forwarder problem, your own trusted key
> list is not up to date. Either it is manually maintained, or
> automated updates are failing (perhaps permission problems to update
> the files, the keys need to be writable by the running nameserver).
>
>> Which seems related to the key roll.
>
> Definitely.
>
>> Changing my local dns (named) config to
>>
>> - dnssec-enable yes;
>> + dnssec-enable no;
>> dnssec-lookaside no;
>> - dnssec-validation yes;
>> + dnssec-validation no;
>>
>> gets me back up & running, without DNSSEC of course.
>>
>> As cached data expires, this should make its way into all working
>> caches over the next day or two
>>
>> Is 'ready' simply .... 'wait awhile' ?
>
> Waiting won't fix anything, you need to fix the trust anchor
> management on your system.
>
> --
> Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

pg151
In reply to this post by Jim Reid


On Thu, Oct 11, 2018, at 10:53 AM, Jim Reid wrote:

> Although switching off DNSSEC validation will keep the mail flowing, it
> only kludges around the underlying problem. Which might or might not be
> related to the rollover of the root KSK a few hours ago. It’s hard to
> tell from the information you’ve provided. That said, you do appear to
> have a DNS server misconfiguration which is causing DNSSEC validation to
> fail. Clearly it would be wise to fix that before turning DNSSEC
> validation on again.
>
> The switch to the new KSK seems the most likely cause, assuming DNSSEC
> validation always worked for you before then.

It's been 'working' for ages.  Yes, I could have been 'just lucky for a long time'.  Bears looking at certainly.

> > Is 'ready' simply .... 'wait awhile’ ?
>
> Maybe, maybe not. It depends on what is broken in your DNSSEC setup. If
> you’ve hard-wired the now dead root KSK, waiting a while won’t help.
> That key will still be dead when you re-enable DNSSEC validation. No
> matter how long or short you wait.
>
> Consult ICANN’s web pages on the root KSK rollover. They have info on
> how to check that DNS configurations handle the KSK rollover properly
> and how to troubleshoot them when they don’t.

Isn't 'hardwired' here afaict.  Looking at the ICANN site -- again -- is probably best advice.

Thx!
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

pg151
In reply to this post by Viktor Dukhovni


On Thu, Oct 11, 2018, at 10:58 AM, Viktor Dukhovni wrote:
> This does not look like a forwarder problem, your own trusted key
> list is not up to date.  Either it is manually maintained, or
> automated updates are failing (perhaps permission problems to update
> the files, the keys need to be writable by the running nameserver).
...
> Waiting won't fix anything, you need to fix the trust anchor
> management on your system.

noted.

on it.

thx.
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

pg151
In reply to this post by Jamie Nelson


On Thu, Oct 11, 2018, at 11:03 AM, Jamie Nelson wrote:
> https://www.icann.org/dns-resolvers-checking-current-trust-anchors

was JUST looking for that! thx.
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

Jim Reid
In reply to this post by pg151


> On 11 Oct 2018, at 19:07, [hidden email] wrote:
>
>> The switch to the new KSK seems the most likely cause, assuming DNSSEC
>> validation always worked for you before then.
>
> It's been 'working' for ages.  Yes, I could have been 'just lucky for a long time'.

DNSSEC is very brittle. Either it works perfectly or not at all. Luck has nothing to do with it. Ending up with a working DNSSEC setup is something that rarely if ever happens by accident. If your validators don’t/can’t maintain up to date trust anchors they *will* fail at some point. Today might well have been that day for you.

Ensuring trust anchor(s) are current is critical to DNSSEC validation. It’s not a matter of luck if this doesn’t get configured correctly. And it’s not a matter of luck if someone’s failed to plan for today’s KSK rollover or been unaware of this high profile event. Sorry.

Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

Bill Cole-3
In reply to this post by pg151
On 11 Oct 2018, at 14:07, [hidden email] wrote:

> Isn't 'hardwired' here afaict.  Looking at the ICANN site -- again --
> is probably best advice.

Since you're running BIND, https://kb.isc.org/docs/aa-01182 might be
more specifically helpful, although I'm not sure that you can recover
from the key having actually rolled at this point by just setting
"dnssec-validation auto;"

--
Bill Cole
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

pg151
On Thu, Oct 11, 2018, at 2:33 PM, Bill Cole wrote:
> > Isn't 'hardwired' here afaict.  Looking at the ICANN site -- again --
> > is probably best advice.
>
> Since you're running BIND, https://kb.isc.org/docs/aa-01182 might be
> more specifically helpful, although I'm not sure that you can recover
> from the key having actually rolled at this point by just setting
> "dnssec-validation auto;"


Managed to get it all sorted.

manually deleted the existing bind.keys, DL'd and copied over a current/updated copy, made sure all jnl's were synced, switched validation -> auto, and restarted.

resolver's up, running & working now, as least as verified with the usual

  dig @127.0.0.1 dnssec-failed.org a +dnssec

not clear if all of that^ was needed, but it apparently did the trick.

thanks all.
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

Viktor Dukhovni
On Thu, Oct 11, 2018 at 03:44:56PM -0700, [hidden email] wrote:

> resolver's up, running & working now, as least as verified with the usual
>
>   dig @127.0.0.1 dnssec-failed.org a +dnssec
>
> not clear if all of that^ was needed, but it apparently did the trick.
>
> thanks all.

Check the user "named" runs as after chroot and dropping privs has
write permissions to update the root trust-anchor file (may need
write permissions to the containing directory to make the update
atomic).

--
        Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: Reminder DNSSEC Root KSK roll today

pg151
On Thu, Oct 11, 2018, at 3:51 PM, Viktor Dukhovni wrote:
> Check the user "named" runs as after chroot and dropping privs has
> write permissions to update the root trust-anchor file (may need
> write permissions to the containing directory to make the update
> atomic).

thanks! I _think_ I'm set

        ps aux | grep named
                named     2561  0.0  0.3 243468 48724 ?        Ssl  13:47   0:05 /usr/local/bind9/sbin/named -f -t /var/chroot/named -n 2 -S 1024 -u named -c /etc/named.conf

        ls -al \
         /var/chroot/named/keys/managed-keys/external.mkeys

                -rw-r--r-- 1 named named 1.4K Oct 11 13:47 /var/chroot/named/keys/managed-keys/external.mkeys

where, given the bind.keys' init,

        ls -al /usr/local/etc/named/bind.keys
                -rw-r--r-- 1 named named 3.9K Oct 11 12:28 /usr/local/etc/named/bind.keys

matches in chroot,

        cat /var/chroot/named/keys/managed-keys/external.mkeys
                $ORIGIN .
                $TTL 0  ; 0 seconds
                @ IN SOA  . . (
                        2          ; serial
                        0          ; refresh (0 seconds)
                        0          ; retry (0 seconds)
                        0          ; expire (0 seconds)
                        0          ; minimum (0 seconds)
                        )
            KEYDATA 20181012204732 20181011204732 19700101000000 257 3 8 (
                    AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQ
                    bSEW0O8gcCjFFVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh
                    /RStIoO8g0NfnfL2MTJRkxoXbfDaUeVPQuYEhg37NZWA
                    JQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaDX6RS6CXp
                    oY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3
                    LQpzW5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGO
                    Yl7OyQdXfZ57relSQageu+ipAdTTJ25AsRTAoub8ONGc
                    LmqrAmRLKBP1dfwhYB4N7knNnulqQxA+Uk1ihz0=
                    ) ; KSK; alg = RSASHA256; key id = 19036
                    ; next refresh: Fri, 12 Oct 2018 20:47:32 GMT
                    ; trusted since: Thu, 11 Oct 2018 20:47:32 GMT
            KEYDATA 20181012204732 20181011204732 19700101000000 257 3 8 (
                    AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTO
                    iW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN
                    7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5
                    LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8
                    efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7
                    pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLY
                    A4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws
                    9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU=
                    ) ; KSK; alg = RSASHA256; key id = 20326
                    ; next refresh: Fri, 12 Oct 2018 20:47:32 GMT
                    ; trusted since: Thu, 11 Oct 2018 20:47:32 GMT