Override domain lookups for split-dns with NetworkManager
How to override DNS search domains with Network Manager and maintain a Split-DNS setup.
I've found myself in the situation where my customer has merged with another giant in the industry. This usually involves a lot of consolidation work on many levels, especially the IT infrastructure.
In certain scenarios, companies might choose for a smooth transition, not wanting to disrupt too many things. For example, allow each company to be able to access each other's intranet through their own VPNs by forwarding DNS requests to the other companies nameservers, and making sure that the network allows for such traffic to pass.
To illustrate the situation, let's give fictitious names and domains to these companies:
- DenseCloud - densecloud.com
- VMwale - vmwale.org
Disclaimer: Names have been ficticiously generated through: https://namelix.com/app/?keywords=Cloud
I'm able to connect to DenseCloud through a VPN. However, I've opted in NetworkManager to only use the VPN connection for accessing the local resources:
The "use this connection only for resources on its network" option will tell NetworkManager to not configure the VPN as your default gateway. All your normal traffic will go through your router at home, or the router of whatever place you are connecting from.
Let's look at what NetworkManager's configuration looks like:
10:22:56 carroarmato0@neon-flower ~ → nmcli
DenseCloud VPN connection
master wlp4s0, VPN
inet4 192.168.14.176/24
route4 10.121.0.0/16
route4 10.119.168.0/21
route4 10.119.160.0/21
route4 10.225.0.0/16
route4 192.168.8.0/24
...
DNS configuration:
servers: 192.168.8.254 192.168.8.253
domains: densecloud.com
interface: vpn0
type: vpn
servers: 192.168.50.2
domains: lan
interface: wlp4s0
As we can see, the VPN pushes a number of routes and DNS information through DHCP. NetworkManager gladly applied the routes and the DNS nameservers. However, it will only attempt to resolve addresses through the VPN which use densecloud.com as the base domain.
How is this possible in the first place? Well, the answer lies within our /etc/resolv.conf file.
10:28:59 carroarmato0@neon-flower ~ → cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
search densecloud.com lan
As we can see. The resolver of the system is set up to use systemd-resolved in this case to facilitate the lookup process.
Notice how there is no mentioning of vmwale.org!
I can prove that only lookups for accessing corporate resources at DenseCloud work, and no DNS leakages are taking place by running tcpdump, looking at traffic going towards DenseCloud's nameservers over the VPN.
10:42:39 carroarmato0@neon-flower ~ → dig +short carroarmato0.be
116.203.113.242
10:54:16 carroarmato0@neon-flower ~ → dig +short artifactory.densecloud.com
192.168.1.114
----
10:53:42 carroarmato0@neon-flower ~ → sudo tcpdump -n host 192.168.8.254 or host 192.168.8.252 -i any
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
22:54:25.188920 IP 192.168.14.176.44374 > 192.168.8.254.53: 59768+ [1au] A? artifactory.densecloud.com. (50)
22:54:25.215596 IP 192.168.8.254.53 > 192.168.14.176.44374: 59768* 1/0/1 A 192.168.1.114 (78)
22:55:58.824462 IP 192.168.14.176.47348 > 192.168.8.254.53: 61736+ [1au] AAAA? connectivity-check.ubuntu.com. (58)
22:55:58.905537 IP 192.168.8.254.53 > 192.168.14.176.47348: 61736 0/1/1 (119)
As we can see from this output, there are no traces of a query sent to carroarmato0.be to the corporate nameservers. And as expected, a query for a corporate resource such as artifactory.densecloud.com did result in a lookup through the VPN.
Interestingly we DO notice a query to connectivity-check.ubuntu.com! I believe this is a special case where NetworkManager will try to reach a predetermined address to determine if it is capable or not to reach the public internet through various network interfaces such as this VPN.
But, if we try to access something from VMwale, this will not work, as the resolver has no specific local knowledge on how to resolve that, and thus, defaults to using a recursive lookup to the DNS nameserver present on the local network (192.168.50.2 in this case on the lan for the wireless wlp4s0).
While publically exposed DNS names will work, hidden DNS names only exposed internally on the intranet through Split-Horizon won't unless you are querying them through the VPN.
10:56:43 carroarmato0@neon-flower ~ → dig +short confluence.vmwale.org
11:22:29 carroarmato0@neon-flower ~ →
---
11:21:56 carroarmato0@neon-flower ~ → sudo tcpdump -n host 192.168.8.254 or host 192.168.8.252 -i any
[sudo] password for carroarmato0:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
So how do we solve this issue?
1) Manage /etc/hosts manually for all existing services and systems that you need to access for VMwale.org and become frustrated and insane.
2) Ask the IT department nicely to also include vmwale.org in the DHCP options (that's DHCP option 119, you are welcome) passed to clients connecting.
3) Not use Split-DNS and configure your VPN profile to use it for all traffic (but that would defeat the purpose of this blog post and potentially slow down your normal internet usage, so please continue reading ahead for the solution :) )
4) Change the configuration yourself to NetworkManager through the command line (for whatever reason this is not supported in the GUI connection editor nm-applet).
For option 2), the IT department might understand adding it for completeness, however, astute network engineers/system administrators might inquire as why you might need that, as they would assume you, just like anyone else in the company, would be using option 3).
There might be a company wide IT policy which requires all external devices connecting to the corporate network to use that connection for all traffic. This can be for various reasons: detect suspicious connections due to malware infested laptops, or keeping statistics of how often and how many employees access social media websites for cat pictures and not accessing work related websites or applications. Who knows?
In any case, if the above didn't scare you off, and you have legitimate, professional or ethical reasons to use Split-DNS (or you really don't care about policies, I'm not judging, why do you think I'm writing this? :) ), then option 4) is for you!
11:46:36 carroarmato0@neon-flower ~ → nmcli c modify DenseCloud ipv4.dns-search 'densecloud.com,vmwale.org'
If we now take a look at our various outputs and configurations:
11:50:07 carroarmato0@neon-flower ~ → nmcli
...
DNS configuration:
servers: 192.168.8.254 192.168.8.253
domains: densecloud.com vmwale.org
interface: vpn0
type: vpn
servers: 192.168.50.2
domains: lan
interface: wlp4s0
11:51:22 carroarmato0@neon-flower ~ → cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
search densecloud.com vmwale.org lan
11:51:27 carroarmato0@neon-flower ~ → dig +short carroarmato0.be
116.203.113.242
11:52:50 carroarmato0@neon-flower ~ → dig +short artifactory.densecloud.net
192.168.1.114
11:53:43 carroarmato0@neon-flower ~ → dig +short confluence.vmwale.org
10.250.251.26
---
11:52:29 carroarmato0@neon-flower ~ → sudo tcpdump -n host 192.168.8.254 or host 192.168.8.252 -i any
[sudo] password for carroarmato0:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
23:53:43.693659 IP 192.168.14.176.37612 > 192.168.8.254.53: 62746+ [1au] A? artifactory.densecloud.net. (42)
23:53:43.715007 IP 192.168.8.254.53 > 192.168.14.176.37612: 62746* 1/0/1 A 192.168.1.114 (58)
23:53:54.176720 IP 192.168.14.176.49476 > 192.168.8.254.53: 14110+ [1au] A? confluence.vmwale.org. (46)
23:53:54.294463 IP 192.168.8.254.53 > 192.168.14.176.49476: 14110 1/0/1 A 10.250.251.26 (62)
As we can see, we've successfully overridden our NetworkManager profile for the VPN to add the missing additional search domain and maintained our Split-DNS setup!