resolver: make "dns" the default scheme

[Merovius]

(Note: This is de-facto re-opening #2168, which is locked. See below for justification)

What version of gRPC are you using?

HEAD, as of writing this (commit ID d15f1a4)

What version of Go are you using (go version)?

go version go1.14rc1 linux/amd64

What operating system (Linux, Windows, …) and version?

Linux 5.5.10-200.fc31.x86_64 x86_64 GNU/Linux

What did you do?

• Set up grpc-test.merovius.de with 3 A records pointing at 127.0.0.{1,2,3}.
• Run this program

What did you expect to see?

Two lines with the same output.

According to grpc/naming.md, dns should be the default scheme, so grpc-test.merovius.de:1234 and dns:///grpc-test.merovius.de:1234 should be the same target.

What did you see instead?

A line containing 9 instances of 127.0.0.1:1234 and a line cycling between 127.0.0.{1,2,3} three times each.

---

I've found #2168, which suggests making dns the default resolver. It is closed and locked, with the decision to not change behavior. I would like to ask to reconsider. The discussion there isn't very in-depth, so I'd like to add

• The grpc docs IMO should be considered authoritative for behavior. IMO, grpc-go disagreeing with them should be considered a bug.
• I find the current behavior at least slightly harmful. I think it's reasonable to expect that most people, when building grpc clients, will just use a hostname (because that makes intuitive sense). But that means, if I operate a public grpc server, I can't load-balance properly - I always have to expect that the first A-record is returned (and my DNS provider might not support randomizing them)
• We ran into this when evaluating options for load-balancing μ-services on Kubernetes. Kubernetes either a) resolves hostnames to a single address, pointing at a Layer 4 load-balancer colocated with the client, or b) resolves them to multiple addresses, each pointing at a backend replica, so clients can do their own LB ("headless mode"). As we need Layer 7 load-balancing, we expected it would be enough to just enable the round_robin policy in the client and switch the servers to headless mode. We tested it and were surprised to find this not working. Luckily one of us vaguely remembered this behavior-difference, so we tested the dns:/// prefix to make it work. Not everyone will notice and if they do, they might have trouble figuring out what's going on, if they work from the grpc documentation (as I was).

I acknowledge that there is cost associated with changing behavior. But IMO there are good reasons to do so anyway.

[easwars]

@Merovius
We agree that the behavior of gRPC-Go is not consistent with other languages in this aspect. But given that this has been a day-0 behavior of gRPC-Go, it would be a backwards incompatible change to do this and we are not interested in pursuing this at this point in time. The team is working on other higher priority items.
Thank you.

[dfawley]

I think the best path forward would be to accelerate #1786. We could possibly delay the "starts in IDLE" behavior to help it along. Let's carry on any further conversation there.