coredns 1.10.1 erratic plugin does not work #6112

jonatiao · 2023-05-25T01:31:38Z

jonatiao
May 25, 2023

Hello,

I just installed coredns v1.10.1.
Here's the plugin list:

docker run --rm -it registry.k8s.io/coredns/coredns:v1.10.1 -plugins
Server types:
  dns

Caddyfile loaders:
  flag
  default

Other plugins:
  dns.acl
  dns.any
  dns.auto
  dns.autopath
  dns.azure
  dns.bind
  dns.bufsize
  dns.cache
  dns.cancel
  dns.chaos
  dns.clouddns
  dns.debug
  dns.dns64
  dns.dnssec
  dns.dnstap
  dns.erratic
  dns.errors
  dns.etcd
  dns.file
  dns.forward
  dns.geoip
  dns.grpc
  dns.header
  dns.health
  dns.hosts
  dns.k8s_external
  dns.kubernetes
  dns.loadbalance
  dns.local
  dns.log
  dns.loop
  dns.metadata
  dns.minimal
  dns.nsid
  dns.pprof
  dns.prometheus
  dns.ready
  dns.reload
  dns.rewrite
  dns.root
  dns.route53
  dns.secondary
  dns.sign
  dns.template
  dns.timeouts
  dns.tls
  dns.trace
  dns.transfer
  dns.tsig
  dns.view
  dns.whoami
  on

I can see that erratic is installed.
Then I configured the system to use erratic:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        hosts {
           192.168.65.254 host.minikube.internal
           fallthrough
        }
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        erratic
        loop
        reload
        loadbalance
    }

I keep getting this error and coredns does not come up:

$ kubectl logs deployment/coredns -f
.:53
[INFO] plugin/reload: Running configuration SHA512 = 39ff4f3d9d3339f5cb003a89af38387d47c9a4d4e3412a477b50574fbf7f7196219134bbaf6c40b13ba966946ca524d48f7c131f424b5ac7287745177bc285ad
CoreDNS-1.10.1
linux/amd64, go1.20, 055b2c3
[INFO] 127.0.0.1:47953 - 61851 "HINFO IN 7722420337874340296.7024234791753781667. udp 57 false 512" NXDOMAIN qr,rd,ra 57 0.039358s
[INFO] plugin/ready: Still waiting on: "erratic"
[INFO] plugin/ready: Still waiting on: "erratic"
[INFO] plugin/ready: Still waiting on: "erratic"
[INFO] plugin/ready: Still waiting on: "erratic"
[INFO] plugin/ready: Still waiting on: "erratic"
[INFO] plugin/ready: Still waiting on: "erratic"
[INFO] plugin/ready: Still waiting on: "erratic"

Am I doing something wrong?

chrisohaver · 2023-05-25T02:57:08Z

chrisohaver
May 25, 2023
Maintainer

You are doing nothing wrong. The erratic plugin is bonkers. The readme should really elaborate on how it "implements" ready…

// Ready returns true if the number of received queries is in the range [3, 5). All other values return false.
// To aid in testing we want to this flip between ready and not ready.

What is your use case for using erratic?

2 replies

jonatiao May 25, 2023
Author

I would like to simulate a delay on some queries to test the best approach to a resolv.conf configuration that I'm testing.

chrisohaver May 25, 2023
Maintainer

I would try temporarily disabling the k8s readiness check in the coredns deployment and removing the ready plugin. Then restore things when done testing and erratic is removed.

johnbelamaric · 2023-05-25T14:12:48Z

johnbelamaric
May 25, 2023
Maintainer

That's erratic behavior, even for the erratic plugin. If we want something to help test with readiness, we should make that an explicit feature of erratic, not some weird side effect. Since erratic is billed as being for testing client behavior, this weird readiness thing (which is about testing...what...server behavior? K8s behavior?), doesn't fit, especially as an undocumented side effect.

1 reply

chrisohaver May 25, 2023
Maintainer

I agree. Furthermore, IMO the behavior shouldn't be exposed at all to users, even with an option. AFAICT, it exists only for the purposes of one unit test, which would be been better handled with an ad-hoc test plugin that implements ready, not actually implementing it in a real plugin.

jonatiao · 2023-05-27T05:00:30Z

jonatiao
May 27, 2023
Author

Hello Everyone, sorry for the delay on the response.

If I disable the readiness probe and ready plugin, coredns starts fine.
Now, other findings.

If I have cache enabled for the Zone, like this, for instance:

    google.com {
        log . "{remote}:{port} TT {>id} {type} {class} {name} {proto} {size} {>do} {>bufsize} {rcode} {>rflags} {rsize} {duration}"
        whoami
        errors
        cache 30
        erratic {
           delay 2 5s
        }
    }

Erratic works the first time, then it will work only again after the cache expires:

[INFO] 10.244.0.94:52663 TT 7134 A IN www.google.com. udp 32 false 512 NOERROR qr,aa,rd 62 5.0014277s
[INFO] 10.244.0.94:35765 TT 30942 A IN www.google.com. udp 32 false 512 NOERROR qr,aa,rd 62 0.0001643s
[INFO] 10.244.0.94:33703 TT 58958 A IN www.google.com. udp 32 false 512 NOERROR qr,aa,rd 62 0.0001595s
[INFO] 10.244.0.94:36297 TT 47088 A IN www.google.com. udp 32 false 512 NOERROR qr,aa,rd 62 0.0001858s
[INFO] 10.244.0.94:40423 TT 47892 A IN www.google.com. udp 32 false 512 NOERROR qr,aa,rd 62 0.0001829s

Which is fine, but I believe, if I may, worth to mention in the documentation.

If I have this forward entry here:

    google.com {
        log . "{remote}:{port} TT {>id} {type} {class} {name} {proto} {size} {>do} {>bufsize} {rcode} {>rflags} {rsize} {duration}"
        whoami
        errors
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        erratic {
           delay 2 5s
        }
   }

It does not work, ever.

[INFO] 10.244.0.94:43564 TT 16582 A IN www.google.com. udp 32 false 512 NOERROR qr,rd,ra 62 0.0035209s
[INFO] 10.244.0.94:38500 TT 40074 A IN www.google.com. udp 32 false 512 NOERROR qr,rd,ra 62 0.0031738s
[INFO] 10.244.0.94:53653 TT 10022 A IN www.google.com. udp 32 false 512 NOERROR qr,rd,ra 62 0.0033513s
[INFO] 10.244.0.94:33136 TT 38837 A IN www.google.com. udp 32 false 512 NOERROR qr,rd,ra 62 0.0030964s
[INFO] 10.244.0.94:49769 TT 7315 A IN www.google.com. udp 32 false 512 NOERROR qr,rd,ra 62 0.0031092s

Which is not fine I believe, it's not clear in the documentation anywhere that forward breaks erratic. This forward config is standard in k8s.

Which made me wonder.. What if cache prefetch is also being affected by this. You know, the one we have been discussing here:
#6103

I changed the forwader config FROM:

        forward . /etc/resolv.conf {
           max_concurrent 1000
        }

TO:

        forward . 8.8.8.8 8.8.4.4 {
           max_concurrent 1000
        }

Failed again, but it takes more time that the 60sec..
Then I removed max_concurrent..

Prefetch is working, finally. (I know it's working because I had at the same time running the same test using the same test on another machine, and there, every 60s sucks)
ps. Tried to do the same with erratic, no luck. We cannot have forward together with erratic.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

coredns 1.10.1 erratic plugin does not work #6112

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

coredns 1.10.1 erratic plugin does not work #6112

Uh oh!

jonatiao May 25, 2023

Replies: 3 comments · 3 replies

Uh oh!

Uh oh!

chrisohaver May 25, 2023 Maintainer

Uh oh!

jonatiao May 25, 2023 Author

Uh oh!

chrisohaver May 25, 2023 Maintainer

Uh oh!

johnbelamaric May 25, 2023 Maintainer

Uh oh!

Uh oh!

chrisohaver May 25, 2023 Maintainer

Uh oh!

jonatiao May 27, 2023 Author

jonatiao
May 25, 2023

Replies: 3 comments 3 replies

chrisohaver
May 25, 2023
Maintainer

jonatiao May 25, 2023
Author

chrisohaver May 25, 2023
Maintainer

johnbelamaric
May 25, 2023
Maintainer

chrisohaver May 25, 2023
Maintainer

jonatiao
May 27, 2023
Author