+1

Both on iOS & macOS this behaviour persists

Same:
MacOS: 10.15.3
Tailscale: 0.97.78

scutil output:

/etc ❯❯❯ scutil --dns
DNS configuration

resolver #1
  search domain[0] : attlocal.net
  nameserver[0] : 8.8.8.8
  flags    : Request A records, Request AAAA records
  reach    : 0x00000002 (Reachable)

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : attlocal.net
  nameserver[0] : 8.8.8.8
  if_index : 6 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00000002 (Reachable)

resolver #2
  search domain[0] : myinternal.domain
  nameserver[0] : 10.255.0.1
  nameserver[1] : 10.255.0.2
  if_index : 19 (utun4)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

Also on 10.15.4 and 0.97.78

It seems like this might be a general wireguard issue, as I've found a couple of reddit posts mentioning similar issue in the MacOS wireguard client (though I haven't tried it). I did come across this:

scutil example

that walks through creating a resolver for a specific domain. In short if I do this

sudo scutil
d.init
d.add ServerAddresses * 1.2.3.4
d.add SupplementalMatchDomains * mydomain.com
set State:/Network/Service/<my uuid>/DNS
d.show
quit

where 1.2.3.4 is part of the subnet my relay advertises and mydomain.com is the domain I want to use that DNS host for. I can then do DNS lookups over the tunnel.

I've only tested it for 5 minutes, but it also seems to not break things (in my case where mydomain.com is unavailable w/o the tunnel) when I shutdown Tailscale.

I'm sure there is a way to track it down, but might be worth remembering what you set <my uuid> to be because if you want to undo this:

sudo scutil
remove State:/Network/Service/<my uuid>/DNS

Looking at the scutil --dns output above (mine looks similar) and the documents for SupplementalMatchDomains in the OS X profile reference, maybe what is going wrong is that they are setting "search domain" rather than SupplementalMatchDomains? That is a pure guess though.

Confirmed on the latest macOS 10.5.4 and iOS 13.4.

I set up a dnsmasq server on one of my Linux nodes running Tailscale I have connectivity:dig <fqdn> @<Tailscale-dns-ip> returns the expected result. However, using the default dns query does not work.

If I put the Tailscale DNS IP at the top of the nameserver list in advanced settings on a macOS interface, name resolution works (apart from the search domain). So it seems there’s something not correct with the way Tailscale is handling DNS on macOS and iOS. I couldn’t see the DNS entry when running macOS command networksetup -getdnsservers Wi-Fi , so they don’t seem to be added, but I don’t know if that’s the expected behaviour? I logged out and back in on my macOS Tailscale client, but this made no difference

Thanks for the feedback here, everybody.

I left a comment in a now-closed duplicate bug #281 (comment) but we'll keep using this bug for tracking & debugging.

@crawshaw, looks like one solution if all else fails is implementing NEDNSProxyProvider
(https://developer.apple.com/documentation/networkextension/nednsproxyprovider) ... "A DNS proxy allows your app to intercept all DNS traffic generated on a device."

Hi @bradfitz happy to help test on iOS, macOS once a fix is ready

@crawshaw fixed this at HEAD in our iOS app and I've just confirmed it's fixed. The iOS app with a version greater than or equal to v0.98-12 will have the fix.

I should clarify that the fix should equally apply to macOS (they share the same code). I just only tested it on iOS.

Does it work with cellular networks too?

(If you're just setting system DNS, and not proxying DNS, I think that iOS doesn't allow DNS to be set for cellular networks.)

@sa1, yes. I tested it on iOS with wifi off.

0.98.12 for iOS and macOS have been released. May take a few hours to reach everyone's devices.

Users report that the DNS server IPs are set correctly now, but not the search path.

We are setting searchDomains (just confirmed with some logging). Preliminary testing with 8.8.8.8 and a search domain shows it isn't working though. I suspect we are running into some limitation of the NetworkExtension: https://forums.developer.apple.com/thread/35027

@bradfitz, next time you're connected to your home domain with your own DNS server, could you try setting a search domain (e.g. ts.tailscale.com) and see if ping derp1 successfully appends the search domain? I'm curious if the DNS server being an IP that is routed by the NetworkExtension changes macOS's behavior.

scutil --dns shows the Tailscale-set search domain in the DNS configuration (for scoped queries) section (fitzpat.com below), which seems like it means it's only used for lookups bound to that interface, which in practice is approximately never.

DNS configuration

resolver #1
  nameserver[0] : 10.0.0.1
  flags    : Supplemental, Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
  order    : 102000

resolver #2
  nameserver[0] : 8.8.8.8
  flags    : Request A records
  reach    : 0x00000002 (Reachable)
  order    : 200000

...

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : 8.8.8.8
  if_index : 4 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)

resolver #2
  search domain[0] : fitzpat.com
  nameserver[0] : 10.0.0.1
  if_index : 17 (utun2)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

Well, if the domain doesn't have a DNS search domain (the common case?), we can default to not being the default route.

If the Tailscale domain does have a search domain set, then we still could if we detect a simple network config (for some heuristic) on the mac. Otherwise worst case we could make it a client option, which I'm sure @apenwarr loves.

With version 0.98.12 on OS X we are seeing:

1. Tailscale is not the default route (traceroute to a non-advertised route does not go through the tunnel)
2. The default domain set in the Tailscale admin interface is not appended to hostname-only queries (ping www fails when the Tailscale provided DNS host has an entry for www.tailscaledomain.com)
3. The Tailscale provided DNS servers are used for all queries...when I am on a network with its own local DNS and connected to tailscale this works:

ping host.tailscaledomain.com

but this fails:

ping host.localdomain.com

when that localdomain.com is an internal domain that is hosted only on the local DNS. If I quit tailscale the ping to the localdomain.com host works.

From our perspective we would very much like at least the ability to maintain #1 (not be the default route), and gain the ability to disable #3 (use local DNS for domains other than those specified in Tailscale). We are going to run into users in locations that have local domains that will lose access to them with Tailscale running I think. Thinking about it, #3 is probably more important.

@apenwarr I think for #4 we'd want the DHCP-provided DNS to get the query.

For #3 the real issue is when one of our users visits e.g. the office of another company and needs to access services in a domain that is only on their local DNS. If all queries went to our DNS via Tailscale they would be unable to get there.

#4 would basically be the same issue in the case where the service had a domain only on the DHCP DNS but was not the search domain provided by the DNS.

Thanks.

Not quite, 3 and (presumably) 4 from above don't work.

I'm currently on a network like we are discussing here. I have a local DNS handed out by my DHCP server and it is definitive for an internal domain.

When I am connected to Tailscale:

ping host1.tailscaledomain.com
works

ping host2.dhcp-domain.com
ping: cannot resolve host2.dhcp-domain.com: Unknown host

Then I quite the Tailscale client:

ping host2.dhcp-domain.com
works

This is a non-issue right now when all of our users are home (and no one has this setup), so no one is really noticing it. It would be somewhat of an issue once they get out of the house.

Thanks.

There is an iOS/macOS feature that would let us implement that, matchDomains. But to achieve those semantics on other platforms we will have to implement a local resolver. I would like to do that, but we should probably have a separate issue for it.

+1 for having the capability for forcing all DNS queries through the VPN, which is our primary use case

My biggest concern right now: we can not abbrevate domain names inspite of
the search path we have declared.

We have links and code that use names like consul-1, mongo-3,
etc. etc. rather than consul-1.example.com. Fixing all those isn't
likely. So we're stuck.

Some of the discussions here has made me curious. Is there Apple
doc and RFCs that spell out what seems like a tedious set of
choices around what goes where, what search really implies, etc.
etc.? DNS config used to seem so simple, LOL.

Someone else just told me that some apps are working with the redirected DNS server, but specifically nslookup, dig, and kubectl do not. kubectl prints an especially interesting error:

Unable to connect to the server: dial tcp: lookup <apiserver-url> <...local DNS...>: server misbehaving

A quick googling suggests that Go's resolver is probably doing something suspicious to cause that last part: golang/go#12712

@apenwarr This is true. It comes down to whether the app uses the OS X system lookup tools or lower level settings. e.g.:

host host.tailscaledomain.com

ping host.tailscaledomain.com

when connected to Tailscale.

I read elsewhere that nslookup and dig have their own resolvers and just read /etc/resolv.conf (the synthetic file) on mac and use that, not using Apple's libc.

And kubectl if cross-compiled is likewise not using Apple's libc, falling back to /etc/resolv.conf without cgo.

And when we're a scoped resolver we don't get put in /etc/resolv.conf.

@bradfitz I believe all of that is correct. It makes troubleshooting less than obvious, particularly on the iPhone.

I "read elsewhere" that dns-sd is the perfered tool for CLI name lookups. SD? - service discovery. It's a very frustrating tool, as you'll learn. But it's manual advises " You can also query for a unicast name like www.apple.com and monitor its status. dns-sd -q www.apple.com"

Mostly i use ping -c 1 consul-1.example.com.

@apenwarr this seems right to me (even as someone who needs the scoped setting in some cases).

Frankly both DNS and default route seem like there isn't a single answer here. Sending everything over Tailscale is best for privacy, but it could cause issues in some cases for some users.

Okay, I have a fix that makes search domains work. For reference, the trick was:
dnsSettings.searchDomains = domains
dnsSettings.matchDomains = [""] + domains
One would think matching against "" would work, and it does work for activating our DNS server everywhere, but it doesn't work for activating our searchDomains, which apparently don't kick in unless they appear in both sections.

/etc/resolv.conf is indeed entirely bogus on macOS, and there isn't much we can do about it. Both go's internal resolver and commands like host are apparently befuddled by this. If they parsed the output of scutil --dns instead they would do better, but the rules used by the macOS resolver are a lot more complicated than /etc/resolv.conf.

The good news is that, (unlike older macOS releases I remember?), most command line tools do use the proper resolver. So ping and python and ssh for example, all properly use the configured DNS server and search path.

These new settings cause macOS/iOS to always use the configured DNS server, not just for corporate DNS purposes, because there is no way yet for us to make the split-DNS configuration work on our other platforms. However, we've also added an option "Use corporate DNS" to the menu on macOS, to match the one on Windows-staging, which lets you disable Tailscale's DNS configuration entirely so you can use a local DNS server when you prefer to do so.

When MagicDNS (#416) is further along, we can consider addressing it there in a cross-platform way.

This fix should show up in the next macOS and iOS releases >= v0.99.1-5.

Running app version on MacOS 0.100.94 but still things do not work properly.

Set up in the Admin Panel the DNS 172.20.0.2 corresponding to the AWS VPC default DNS.

➜ scutil --dns                                                                                                  13:12:19
DNS configuration

resolver #1
  search domain[0] : skynet.local
  nameserver[0] : 172.20.0.2
  if_index : 22 (utun4)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 102400

resolver #2
  nameserver[0] : 192.168.10.1
  if_index : 6 (en0)
  flags    : Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
  order    : 200000

resolver #3
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #4
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #5
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #6
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #7
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #8
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

resolver #9
  domain   : test
  nameserver[0] : 127.0.0.1
  flags    : Request A records, Request AAAA records
  reach    : 0x00030002 (Reachable,Local Address,Directly Reachable Address)

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : skynet.local
  nameserver[0] : 192.168.10.1
  if_index : 6 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  nameserver[0] : 172.20.0.2
  if_index : 22 (utun4)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

but only forcing for instance dig to use the DNS server things work

➜ dig +short ip-172-20-90-205.ec2.internal @172.20.0.2                                                          
172.20.90.205

otherwise

dig +short ip-172-20-90-205.ec2.internal

Of course I ticked the option Use Corporate DNS

This might be just a characteristic of macOS. We have found that tools like dig and host do not respect the scoped query configurations, but higher level tools like ping, ssh or Safari do.

yeah but in my case it does not work neither ssh.

Ok, must be something different then.

@thehilll I confirm that it works. as just a local misconfiguration

hey @apenwarr it was just a local misconfig of my ssh config file. nothing related to Tailscale.