“To give you an idea of scale, the entire set of all machines on the internet, someone came up with an ASCII art file that you could print out on 11 sheets of standard paper. And it was actually not a list. It was an actual map that showed every single machine that was on the internet and what their hostname was and what their IP address was. Because that was how many machines there were...”- Paul Ebersman
In episode 5, John dials in on the world of DNS to understand the important role it plays in modern enterprises and how the unprepared can be vulnerable to malicious DNS attacks. Special guest Paul Ebersman (Principal Software Engineer at Neustar) joins John in educating enterprises about the mission critical role DNS plays.
Have feedback or a cybersecurity topic you would like us to dig into on this podcast? We would love to hear from you! Drop us a quick note at email@example.com.
If you have any questions for Brian, you can contact him at firstname.lastname@example.org.
Paul Ebersman: To give you an idea of scale, the entire set of all machines on the internet, someone came up with an ASCII art file that you could print out on 11 sheets of standard paper. And it was actually not a list. It was an actual map that shows every single machine that was on the internet and what their hostname was and what their IP address was. Because that was how many machines there were.
John McArthur: Welcome back to the Lock and Shield podcast presented by Neustar where we connect the dots for you exposing threats and discussing security issues, both small businesses and large enterprises need to navigate. I'm your host John McArthur, Director of Security Intelligence within Neustar Security Solutions Group. We thought we'd pay special attention to one of the most important enabling technologies in the digital space - DNS. That is the domain name system, affectionately referred to as the phonebook used to find the various websites, applications and services we all use on the internet, whether we know it or not. Now Neustar has been in the DNS business since 2006, when it acquired UltraDNS founded by Rodney Joffe. Rodney is still with Neustar as a technologist and Senior Fellow. And joining us today, also one of our own is Paul Ebersman. Paul works as Neustar's Principal Engineer and DNS Architect. A quick bio on Pau. Paul first worked on the internet for the Air Force in 1984, supporting TCP/IP and OSI networks. He was employee number 10 at UUNET and help build alternate and the modem network used by MSN, AOL and EarthLink. He has maintained his roots in the internet in the open source community working for various internet infrastructure companies, including ISC, Infoblox, and Nominum. He maintains his involvement with a protocol development and use both internally and in the internet community through organizations such as DNS-OAR, NANOG IETF, IETF-ISOC, and ICANN, Paul, thanks for being on the podcast. It's great to have you here.
PE: Sure, thank you.
JM: And I should say, I've learned a tremendous amount for Paul, and been fortunate to attend a few industry conferences with you where we were at MAAWG, I remember and breaking down some of the discussions around DNS SEC, or DNS over HTTPS. We'll talk about those later. And I should also admit, we've stolen Paul's time from time to time to work on our UltraGeoPoint product to improve the quality there. So, before we sort of dive deep into some of the technical issues and discussions around DNS, let me ask you, how did you get involved in working on DNS?
PE: It was a sort of the side effect of doing Unix and TCP back in the Air Force. And then at various other places. DNS was actually just getting started. When I was in the service in ‘84, the first RFCs, which are essentially the protocol Handbook, the instructions for what the machines speak, were just coming out, Paul Mockapetris had put them out. And at the time, we actually were still using a single text file as the entire listing of all of the machines on the internet. It was the host dot txt file. And there was a university and government contractor ISI out in California. And one of the things that they did was they kept track of, and owned the file, that was the list of all of the host names and all of the IP addresses of all of the machines on the internet. And so when you had a new machine, which in those days, it was usually a mini or mainframe. And so it didn't happen very often, you would actually have either email or more far more likely Fax to SRI, the new hostname and the new IP address of this machine that you would just put onto the internet. And somewhere over the next month or so various folks would download that file that now had that in there. And once they had that new file they would know about you and be able to reach your machine. So, at the time, we thought that was probably pretty much okay. To give you an idea of scale, the entire set of all machines on the internet, someone came up with an ASCII art file that you could print out on 11 sheets of standard paper. And it was actually not a list. It was an actual map that showed every single machine that was on the internet and what their hostname was and what their IP address was. Because that was how many machines there were. So, we sort of looked at DNS and this hierarchical database distributed with distributed authority that Paul Mockapetris had come up with. And we sort of went, well, that's kind of cool. But that's really kind of complicated. And we're not really sure that we need that. Of course, by 1990, things were just starting to take off. That was when they were allowing commercial connections to the internet. There were talks of people getting in, other than government agencies in universities. And so at that point, we started to get some clue that perhaps there might be more hosts, and DNS might actually be finally had some use. And we weren't quite sure how long would last work for that matter how long the internet or TCP/IP was going to be the prevalent technology. But as with many things, we weren't sure. And the reason that none of us are really retired is, is we aren't terribly good at predicting the future.
JM: And it seems to me, if you're talking about a list of host names and IP addresses, this is really just for sort of machine to machine communication, right? These aren't necessarily websites, these are, you know, these are effectively allowing sounds like government institutions, and university institutions to sort of exchange information.
PE: Yes, what little interactivity there was, you logged into the remote machine and had a command line session, a shell. And you might transfer files back and forth FTP was around, email, somewhat predates this. And that was it. Any of the things that we would consider to be anything like the world wide web, even some of the earlier protocols, like WASE, and Gopher and Archie and some of those, were really ‘92, ‘93, ‘94 timeframe. And Mosaic when it first came out. That's another one of those, why we are not making stock recommendations. I remember being at work and having this email flip across the staffs. This is one of the young kids was 19, all enthusiastic, there's this thing Mosaic. And you can actually have pictures with your text. And it's really cool. And we sort of looked at him and went yeah, yeah, WASE is faster, next. And that was it. We figured that that HTTP was just more bells and whistles than most of the world needed.
JM: Got it got it, and then Mosaic, eventually, I think that's when Mosaic became popular, right? That's when Microsoft sort of took notice and developed Internet Explorer.
PE: Within the next two years, yes, they sort of, well, they still hadn't quite conceded. Microsoft in 1996, when they launch or sort of ‘95, when they launched Windows 95. They wanted to own the internet. So HTTP was an open protocol that everybody could implement. So they had this skunkworks project called Blackbird, which was their own proprietary markup language. And fortunately, for all of us, it was a flaming disaster. And it didn't work. And so, when, in August of 95, when the operating system launched, and when Internet Explorer launched, it launched with HTTP support in it. Yeah, the rest is, as they say, history.
JM: Got it. Got it. And, and I was gonna say, so we've gone from this system, where effectively, there were, there was a list, if you will, of available computers of actually machines, you have the host names and the IP addresses. To the world we live in now, where there are millions of websites. Certainly, there's email transactions that go on and other services operate on the web. How does how does DNS work in that context? How has it evolved from the, from the eight or 10 sheets of paper to what it is now? How does it sort of what are the nuts and nuts and bolts of it now?
PE: So there are what are called resource record types RR types, which say, this is a label, you know, something.example.com. And then depending on what RR type it is, it might say, this is where you send email, or this is an IP address v4 or v6 that you can connect to, that might have a web server running on it, or this is a service record that tells you what type of services this machine offers on the internet. And so over time, DNS has continued to evolve. And we have added more and more RR types that are very specific to be able to say, not only is this label this humanly, easily rememberable and logical label, what you're going to be putting into your browser or what puts into code are what your app connects to when your app does something for you when you click on a button. It also says what are the services that are offered. Because fundamentally back in the day, where were, you know, literally dozens of machines on the entire internet. I right remember that, you know, 137 dot 39 dot 1.3 happens to be my file server or my mail server, whatever. But that certainly didn't scale when there are literally billions of people using, you know, millions and millions of service points all over the internet. And then when you get into IPv6, where you have 2001, colon, db eight, colon, blah, blah, blah, blah, blah, blah, even I can't memorize my server's IP address. I remember what it's IPv4 addresses, but certainly not it's v6. And so the need for human parsable and logical names was one piece. And there was just the hierarchy in control. If I go to Neustar dot com or Neustar dot biz? I know that it relates to the company Neustar and some service they offer. It's not going to be Sony's movies. It's not going to be HBO streaming. It's not going to be Amazon's website. And so we were able to sort of carve up the namespace in a way that human beings think about things, who owns it, who controls it? And then once I've decided who is the entity I want to talk to, then we get into the what are what would I like to do with that entity? You know what I like to do online banking, what I like to shop what I'd like to send mail.
JM: Hey there, just a quick break from the podcast to talk to you about how securing DNS is vital to all organizations, including the United States government. The Cybersecurity and Infrastructure Security Agency, or CISA caution that late 2018 cybersecurity organizations across the globe started to detect an increase in malicious activity, targeting the domain name system, DNS infrastructure in which we all rely. The attackers were able to redirect and intercept web and email traffic and could have achieved the same for other network services. The threat was further realized when the Department of Homeland Security issued Emergency Directive 1901 in January 2019, on mitigating DNS infrastructure tampering, at which point CISA was made aware of multiple United States executive branch domains impacted by a tampering campaign. In order to prevent future attacks, the Department of Homeland Security made several recommendations including auditing DNS records to ensure they resolved to their intended location, changing DNS account passwords, and adding multi factor authentication to DNS accounts, all best practices to secure your own organization. To learn more about securing your DNS infrastructure, please visit home dot Neustar and navigate to the Security Solutions section.
We quickly saw that those names became part of a lot of these companies brands. And, and I would say we're talking about the hostname is being listed on a sheet of paper. And really, the availability of a hostname, the machine being available was predicated on being on this list and getting distributed to all the other folks operating hosts. Now we're talking about a matter of minutes.
PE: In some cases, seconds. If I told somebody right now, you are registering this domain name and sometime over the next month, it might be useful, they would be incensed. And look for somebody else. There’s no way that on today's world any would accept that in any way as a viable way of doing business. But at the time, it was you know, when, when it required a forklift and field engineers, and special power and everything else and took you three months to get it installed. You could do that all ahead of when it actually lit and was usable on the internet. And that was fine. But yeah, technology has rolled on.
JM: Got it. And from there. So you know, again, I use you know, when I talk about technology with friends and family, you know, I use the phone, the phone book analogy, right? Effectively you have this system of records, resource records in zone files, which identify where websites are so that, again, the phone book, if you will, the Internet, and everybody's sort of websites, their services applications being managed there. But that's all obviously that will make sense and is good. But DNS can be used maliciously as well. Right? And I know we've seen even, I guess for starters, how can it be misused? Right? I think we've heard about DDOS on DNS. So most of us would think about DDOS or thinking about attacking specific websites or applications shutting down a certain entity online with a flood of traffic. But how does that work with DNS?
PE: Sure. So DDOS Distributed Denial of Service what it means is that over a wide swath of the internet, you have manage to deny the intended users of some service, the use of that service. And one of the easiest ways to do that is just create noise. You know, if you want to talk to the person next to you, and everyone in the room is screaming at the top of their lungs and they're 500 people in that room, your conversation is going to be impeded, if possible at all. And so that's essentially what most DDOS does. And DNS is designed to answer a question for you. Some piece of code that started by some user taking some action, asks a question of whoever's authoritative for the area of the DNS that you've asked about. And there is some server that answers the question, you know, who, who's accepts mail for microsoft.com? And if when you answer, ask a question, that doesn't get to the server at all, or the server is unable to get an answer back to you, I've now broken that connection. And so I don't even have to get to your web server, I can simply break answering that question. So that's certainly one way of doing it this easy. DNS was also designed in the days where this also gets back to the dark old days. We literally knew each other, if you had a problem with a machine doing something, or not doing something somewhere else in the internet, we could actually make a query that said, who's logged into that machine right now? And we all had that open, so we could tell who was logged in at the time on some other machine. And we could actually chat or pick up the phone and call them and say, Hey, Bob, you know, you're hosing my mail server with the same request over and over again, that isn't working, can you please stop? Well, that doesn't really scale to where the internet is now. But that was the security model we had. We knew everybody. And if somebody was doing something bad, it was probably a mistake. And you called them up and they went, “oops”, and they apologized, and they stopped. And so DNS was designed with that level of security and trust built in - i.e. none. So, I can essentially forge the packet. So, I can claim that the person who asked for this DNS answer that has a fairly large response size, and take some overhead to receive and look through, came from you. And so if I decided that I want to make your life miserable, I can forge a whole series of DNS queries with a false “who asked this”, and the DNS servers that answer those questions will all send back the answer to who they think asked, which will be you. So I can, with a very small packet, create a situation where you are receiving a very, very large number of very large packets, that basically just clogs your link. And going back to the you know, I can create the 500 screaming people in your room. And so I don't have to just do it to disrupt DNS, I can also overwhelm your website with just lots of garbage traffic that will simply clog the link that gets the traffic to you. So legitimate web queries and web responses are going to get lost in the noise.
JM: Got it. Got it. So effectively, there's an ample amplification ability to take a small request and basically return something large and sort of clog the pipes, if you will. Now, one of the other sort of attack vectors we hear about too, with DNS is around EXFIL. And this concept exfiltration of data I should say, yeah. And we I guess termed DNS tunneling. Correct. And how did how does that work? You know, I'm assuming that deals with malicious actors in malware, etc.
PE: Sure. So if I am a bad person, I'm trying to exfiltrate data out. Most companies that have proprietary information, software, contracts, whatever, have firewalls and various other things that try and clamp down how easy it is to get information out. But they only do it based on what they know people are using. And so the problem is that with DNS, anything that is internet connected is going to wind up making DNS queries in order to be able to find service points, connect to things or whatever. So much like you can't just turn off some HTTP and HTTPS queries from the device because that breaks most apps and all web browsing. You can't turn off DNS. So what they wind up doing is they have code and a malicious piece of software that's within that enterprise, or victim, that makes special DNS queries to a special server that takes the DNS query and actually take stuff out of it and says, “Oh, that's a data packet, and I'm going to treat that as if they'd actually just sent me that data”. So suddenly, you now have a send and receive capability over a different channel than you normally expect. And I can put actual data in, I can put in files, or I can break up files into lots of packets and send it out. So essentially, I have the tunnel, it's much like using a virtual private network piece of software, VPN, is just another way of doing it over a different port. And while it is possible to detect that it is much more difficult, and many companies don't monitor DNS queries. And so they might not notice this. And the scary thing also is that it's a fairly low overhead. So, there was actually a demo where someone ran a Skype session over a DNS tunnel, and it was relatively usable.
JM: Oh, my God. So I was going to say its rather robust and flexible, seems like the right word, but the DNS protocol in terms of what it can do what it can answer?
PE: Well, it's, it's much like HTTP. HTTP fundamentally is a “get”, I’m asking for this. And a “put”, here you go, and what you put in the saddlebags. It was never designed for anything other than text and very simple graphics way back when it was originally designed. But as you can see, from the World Wide Web, and everything on every website that you see, you know, you can do video conferencing, you can do all sorts of stuff, you can stream movies. So, what you asked for, and what you get back, can be potentially quite flexible. And DNS is another it's designed for asking a question and getting an answer back. And so, if you control both ends, the querier or and the server, and are willing to do some creative things with what kinds of things you put in there, yeah, you can pretty much do what you want. And that is exactly what DNS tunneling is.
JM: Got it. And I was gonna say that's scary enough. But I know also, there's this concept and frankly, a malicious attack vector of a DNS hijack, where it's, it's not enough that you could be exfiltrating data via DNS, but there's also with an incidence of perpetrators literally taking over resource records or zone files for enterprises, or governments.
PE: And there are two ways of doing it. One of them is if you can get the credentials, or permissions to either change things with your DNS provider, or change zone data on the authoritative servers, you can change who are the mail hosts, you can change who are the web servers. And you can even do things like set up a fake web server that lets you log in and then sends all the data to the real web server. So you may not even know that there's someone in the middle who has just stolen your login and password. You can also, if you get the credentials that allow you to say, who are the who are the name servers for that zone, you can change or add additional name servers, which can give you the same false records. So you can intercept mail and read it potentially on the fly. You can intercept logins and passwords. And once of course, you do those, now you're that much closer and being able to all sorts of other evil things. Since an awful lot of enterprises don't use multi factor authentication, so once you've got someone's, you know, Active Directory or Office 365 credentials, you've also got their network credentials, you've got their email credentials, you've got their file access and file share credentials, all in one swoop.
JM: Got it. Got it. That sort of leads then to there’s several attack vectors when we're talking about DNS. I know that I've heard, discussions more and more conversation, and I think it's become an industry standard, but I’ll allow you elaborate on that. Please do. Is DNS SEC, the security around DNS. And can you can you describe for us what that is? What are the measures that are taken there?
PE: Sure. So, um, to try and not dive too deeply into the crypto, there are things called digital signatures where basically, I can sign something with what is called a private key. And I can publish a public key. And anybody who gets that public key and looks at that signature can validate that the only one who could have possibly made that signature for that object is the owner of the private key. And what it allows me to do is know both that I know who made that signature and what is being signed whatever the DNS answer the piece of email, a file, a checksum, whatever, that it has not been altered in transit. So DNSSEC basically uses digital signatures so that I know that whatever the owner of the actual zone file that has the actual zone data for that particular example.com. If I put in that dub dub dub.example.com is dot 1.2, dot 3.4. I know all the way to my machine, if I'm validating with DNSSEC all the way to my machine, that nobody has changed that answer and that nobody in the middle is changed to 5678 or said that there's no such record or whatever. So you can still see everything that goes by on the wire. Anybody who has network access anywhere along the way can see that particular client asked for this question and this DNS server gave that answer, but at least we have data integrity. And so we know that no one has changed or altered it. And because of the fact that it is end to end from the actual zone owner all the way to the original asker of the question that it hasn't been altered many of the attack surfaces, the best you can do is break it. Because what happens is you can make it if you change an answer, it no longer validates so that the original asker can go “that wasn't what the zone owner gave me”, therefore, that's bad, therefore I won't use it. So it does make it much harder to do hijacking where you were sending someone else to a false name server. It's much harder to set up a false name server that has bogus answers if the tree above them is DNSSEC signed. So it still doesn't cover every possible scenario and attack surface. And particularly denial of service. DNSSEC fails what's called “closed”. If you're thinking in terms of physical doors, if I have an electronic door with an electronic lock, and I lose power, there are two ways of doing it. I can fail “open”, which means as soon as the power goes away, the door is unlocked, and anybody can get in or out. Or I can fail “closed”, such that no one can open that door until the power comes back on. Okay, DNSSEC fails closed, the user gets no usable answer, therefore, they can't get to whatever that resource was. But at least they can't get to something malicious that will steal their login credentials.
JM: Got it. Got it. And let me ask what has been, is DNSSEC is that effectively? How is that been effectively adopted? By the DNS operators that are rather I don't know where the onus lies? Is it all the DNS operators or the enterprises and brands themselves?
PE: There are a bunch of places. So a very quick DNS lesson. So there are different types of DNS servers, there's what's called a stub resolver. That's code that's running on the device, you use your PC, your phone, and all it's capable of doing is taking what the application said, I want to know who is the web server for amazon.com. What is their IP address? And it sends it up to somebody smarter than it that's able to walk along the whole DNS tree and get an answer. There are what's called Recursive servers. And that's what they do they recurse they walk along the entire tree and answer any question and then they send that back to whoever asked them usually basically stub resolvers. There are Authoritative servers, and their only job is to answer questions about specific zones. So amazon.com will have specific servers that are the only servers in the world that know answers about who is the web server for amazon.com. So with DNSSEC, the zone owner has to have whoever's running the Authoritative servers for them, and they might run them themselves or they might contract to someone else, a DNS operator is what its usually called, to do DNSSEC signing of their zones for them. And every time they make a change, and signatures need to be redone, the DNS operator or zone owner will do the signature such that's valid. Stub resolvers tend to do nothing but talk to their Recursive and nothing else. Recursive resolvers are the ones that will do DNSSEC validation usually. So they will not only chase all along the DNS tree to find out who is the Authoritative server for Amazon.com and then go to Amazon.com and ask who is the web server. They will also check to see if there are DNSSEC signatures available for all of the links in the chain, and they will validate. And so, what will happen is, you need zone owners to have themselves or their operators sign zones. But also, more critically, people with devices need to use Recursive servers that do DNSSEC validation, and those resolvers need to do DNSSEC validation. Now in today's market Google and Cloudflare and Quad9 and a lot of the public free Recursive servers, and most of the very large ISPs, the Comcasts of the world, their Recursive servers that they offer to their customers, do DNSSEC validation. So right now the trailing piece of it is, more people need to sign their zones. Zone owners need to be aware of what security issues that protects them in regard to, and make sure that they are using operators that do support DNSSEC, that they're actually signing their zones.
JM: Yeah, as we think through security, we also know with additional security, and maybe investigation measures, there's also this idea of there's privacy, privacy is a big concern right now, everything we do on the internet. And I know one of my favorite terms in use right now with DNS is DOH, a shout out to Bart and to Homer Simpson. But DNS over HTTP or HTTPS. What is that? How does it relate to privacy? I know, it's sort of a hotly debated topic within the DNS community, what are the pros and cons of using something like DOH?
PE: Right, so privacy and information disclosure has become a very hot topic. And people are becoming more aware of things like even metadata. If I know what phone number you call, even if I don't know what you talked about on your phone. But I know from where you made a phone call, and I know when you made the phone call, and I know what phone number you made that call to, I can cull an awful lot of information. And DNS queries, this goes back again to the in the dark ages, where we had zero security, DNSSEC says that I have end to end data integrity. What the zone owner put in the zone that I asked for, I know I got the answer they wanted me to have. But as I said, every single hop along the way that can listen to packets on that network, know who asked, what server they asked, what question they asked, and what answer they got back. That's a lot of metadata. And so, as governments are starting to watch this stuff, and some governments are more oppressive than others. And depending on your outlook, as more and more people are looking at what queries you make and wanting to sell your personal information and give you personalized ads, and all the rest of that, what we do and who we want to know that information. Having that floating around in the clear for everyone to look at is not a good thing. So HTTPS is HTTP that has been encrypted with a secure certificate, just like you can do digital signatures with public private key encryption, you can also encrypt it so that only one half of the pair can decrypt with the other sent. So I can take a public certificate, and I can encrypt my HTTPS query, that I send up to the web server, and only that web server with the private half of that certificate will be able to unpack it and see what query I made. And so that's a fairly secure thing. It also by inference says, I have a pretty good idea of who is the other person I'm talking to you because they have the private key and they were able to send me a response back that matches the public key that I really believe is that website’s. So, when we were looking at ways to secure DNS, we said, okay, in the same way that you can tunnel DNS to get through firewalls, everything in network is layers. And so, by default, we send DNS over what's called UDP. UDP and TCP are the two things that will normally sit on top of IP that you will use in interactive sessions. And UDP is, I send out a question I might get an answer back. There's no reliability to it. And I don't know who answers and when we talked about DNS amplification, where I can forge who asked this question, so I can guarantee that my victim gets a very large response. The reason is because it's UDP. So instead, what DOH is, is simply nothing but asking the question over an HTTPS connection instead of over UDP, or simply use a different transport and DOT which is the other one you'll hear about, which is DNS over TLS. TLS is the underlying encryption that HTTPS uses. And so it's the encryption part but it doesn't have all of the session smarts that an HTTPS, or a web session would have, who you are and what browser you're using and some other things that you have. And so currently DOH because it can be implemented in browsers, particularly Mozilla for Firefox, and Google for Chrome, have implemented it very quickly. Microsoft has been doing it as part of their operating system and part of their stub resolver instead. So that not only the web browser, but every application on your Windows device will get the benefit of this more private version of DNS.
JM: Hi, it's john. As we're digging deeper into the history of DNS and its importance to modern enterprises, I wanted to talk about how your DNS is mission critical and keeping your business accessible and available, but likely your DNS setup has been around as long as your business has. Over that time, your business may have tried several strategies to manage your DNS infrastructure, which may have negatively impacted your DNS configuration. Your DNS may be working, but it may not be working well. Leaving your business open to vulnerabilities. Neustar's UltraDNS Health Check makes it easy to see critical issues at the click of a button running over 50 DNS configuration checks to provide detailed results and recommendations to keep your DNS running smoothly. To learn more about Neustar's UltraDNS, please visit home dot Neustar and navigate to the Security Solutions section. Now back to the podcast.
Understood and I think it was I think about myself in my use my family's use of the internet, I know, you know, we may not use our ISP’s Recursive resolver. Right, we might choose a different one for performance reasons, and we do. But I would say that the difference being right now even though I'm you know, if I'm if I'm not using DOH, despite even though I'm not using, say my ISP, Recursive resolver, they can still see the requests I'm making for websites for instance. But if I click that checkbox, for instance, in Firefox, in the settings area, to enable to enable DOH, at that point, the websites I'm requesting, those become encrypted and are no longer visible in this case to my ISP is I'm using a third party Recursive resolver.
PE: Right, they are still, by definition visible to your Recursive, because your Recursive has to be able to unpack them to know where to recurse across. And it does argue for why you should give some thought to who you're giving that data to. Some ISPs have much better privacy policies than others, or privacy policies that are more in line with your own personal biases. And that's really what it boils down to. But if I am at work, and I'm on my company's network, just like looking for DNS tunneling, there's a lot of useful security information and potential early warnings in the DNS queries, companies may lose visibility if they don't set up their own DOH servers, to be able to unpack the queries that their users are using in the enterprise. And certainly the trend I'm seeing this is DOT is on a particular port. And so that's easy to block. blocking all HTTPS is much more difficult, so DOH, is much more difficult. But so far, all of the implementations and all the browsers have a way of politely saying either please don't use DOH at all when you're on the enterprise, or please use this particular server. And so enterprise and security teams are probably going to want to look at that balance of sometimes poking and prying is a security feature and not a loss of privacy.
JM: All right, well, we've gone through sort of both the security and privacy aspects and bit of history around DNS. Let's talk about when it comes time to make a decision around choosing a DNS operator. What should customers be thinking about for their own brands and domains when they're being managed and services? What are the most important features of a DNS operator?
PE: Well, they're certainly, there's some basic things that any online service you should look for. You should look for levels of access and control individual login IDs, you should look for two factor authentication, you should look for what form of support for alerts and emails and other things, they have, to make sure that you can control it. Because if I can steal your operator account credentials, I can change your DNS zone data. And then I can point you at fake web servers, fake mail servers, etc. So, there's the generic security that I would look at for anything, I would look for that with your cloud provider or with your software as a service provider. And DNS operators are no different. The things that I would do make sure that you very specifically look at for DNS operations, how widely distributed are their DNS servers? How many do they have? Where are they? How broad is their attack service? You know, how hard is it to essentially DDoS them? What kind of DDoS mitigation do they have to be able to soak up abuse? If they're, there's the ITF, does what are called RFCs, Request For Comment. Those are basically the protocols, the language rules that people talk. And DNS has, at this point, a large number of them. Everybody has to support those or DNS doesn't work. There are vendor specific tricks beyond those, in terms of things like load balancing, being able to enforce geographic boundaries and various other contract things that you may want, in advance features that you should probably look at and ask. And you should look at what kinds of reporting what kind of data. And when you're looking, one of the things you need to do is, is do the risk benefit analysis. You know, if I'm Joe's fish shop and auto body parts, and my website goes down for six hours, once in a year, that's probably okay. If I am Ameritrade and I go down for 20 seconds during the trading day. And I have racked up $15 or $20 million in penalties to my customers in refunds or whatever, I have a very different risk of what does a DNS outage cost. And before you can go shopping, because there are different levels of quality of service at different price points. You know, like everything else, there are Ugo’s to Bentley's and space shuttles. And you may or may not really need that. You need to be able to go through. But you should. What does a DNS outage cost and how likely is that to happen, should help you inform how much are you willing to spend to avoid that. And then eventually, you get into the more complex things, like at a certain dollar value amount, it's worth having multiple providers, potentially, you know, the majority of the world can probably get by, just like with email, you know, you can go to Google or office 365. And most of the world's has very few companies can justify running their own mail server in today's market, because it requires a lot of very specialized and expensive expertise to do that well. And there aren't a lot of people that can do that. And so there's a limited supply. And you know, those engineers charge accordingly. DNS is in a similar boat at a certain level of complexity. So for much of the world, making a really good choice and making an informed choice, and then using all of the security things. It is far more common that someone stole login credentials, because they forgot about an email address that someone had and they left the company, and they use that email to reset a password, or they didn't use two factor authentication. Those exploits are far more common and far more devastating than the odds of most of the big providers having any significant outage.
JM: Understood, then I would guess, finally, we were talking about DNS zone files earlier. And I guess as I manage those, which, for my particular brand, or for my enterprise or small business, is it a certain sort of a set and forget sort of way of managing or is there something I should be checking on regularly around those?
PE: So, there's an analogy that someone I've been working with used recently, we all have a kitchen junk drawer. Yes. And originally, it starts out with a couple of knives and scissors and some scotch tape. And we know well intentioned junk drawer, right, exactly. And four or five years later as you're moving, and you're opening up this junk drawer, and you discover this dinosaur and a cat toy. And this tube of electrolyte gel that has sort of oozed all over everything, and there's no longer any tape. DNS zone files, the trend for most folks who don't spend a lot of time with it is, when you originally get the domain name and you originally and every time you make some major change to your infrastructure, your visible external infrastructure, you probably go through and make changes to your DNS zone file. And you do things with good hygiene and all the rest of it. But the reality is that you start out with the scissors and the knives and scotch tape, and then everyone in the world, every developer you have, every script you have, every marketing promotion you ever did, get stuffed into that junk drawer. And if you're not going through on a very regular basis and looking at that, at the point at which you desperately need it, you're going to discover that the one thing in that drawer that you desperately needed and took four hours to find, is covered in the electrolyte gel. And so not only for security reasons, because there are holes where you may not notice, but bad folks will get in. There's also just the, if things do break, you're guaranteeing that it's much harder to fix, and that it may be take much longer to notice. And when you make some major changes, you may discover there's so much trailing crap from some guy who left three years ago and did everything wrong. And that was why you fired him. But now you have to undo all of that stuff before you go through. So yes, it is not a come in and have a professional services and engagement. Have the consultants set it up all nice and pretty and walk away. You know, you own the building, you own the junk drawer, and renters come and go all the time. And so yes, having particularly a nice, automated set of consistent tests for well-known issues is just another belt and suspenders, you know. It will mean that you get to catch it during the weekdays before an emergency or an outage and plan your cleanup.
JM: Very good at being able to be a bit more proactive, as opposed to reactive when bad things can and do happen to be honest. So that makes sense. Yes. Well, with that, Paul, many thanks for joining us today and sharing your vast knowledge around DNS, DNS security, that privacy as well as some of the best practices around choosing a DNS operator. Really appreciate your time.
JM: It's been a pleasure. Great, and to our listeners. Thank you for checking in. We'll be talking again soon. Thanks.