OARC 42 (Charlotte, NC, USA)Workshop

8 Feb 2024, 09:00 → 9 Feb 2024, 18:40 US/Eastern

Salon A/B (Embassy Suites Charlotte Uptown)

Petr Špaček (Internet Systems Consortium (ISC)), Phil Regnauld (DNS-OARC)

OARC 42 is planned to be a hybrid in-person and online workshop.

The workshop will be held in Charlotte (North Carolina), USA on Thursday 8th & Friday 9th February, 2024

---

DNS-OARC is a non-profit, membership organization that seeks to improve the security, stability, and understanding of the Internet's DNS infrastructure. Part of these aims are achieved through workshops.

DNS-OARC Workshops are open to OARC members and to all other parties interested in DNS operations and research.

Social Media hashtag: #OARC42 #LoveDNS #LifeUniverseAndDNS

Mattermost Chatroom: Workshops on chat.dns-oarc.net (sign-up here)

---

OARC 42 SPONSORS

DELUXE+

DELUXE

ASSOCIATE - Social Event

---

Sponsorship opportunities for OARC 42 are available. Details at:

https://www.dns-oarc.net/workshop/sponsorship-opportunities

---

OARC PATRONS 2024

---

Annual Workshop Patrons for 2024 are available. Details at:

https://www.dns-oarc.net/workshop/patronage-opportunities

---

 

Thursday, 8 February

09:00 In-person attendees registration

10:00 Session 1 start

OARC 42 Day 1: Session 1

Conveners: Petr Špaček (Internet Systems Consortium (ISC)), Suzanne Woolf (Public Interest Registry (.ORG))

1 Welcome to OARC 42

OARC-42-Welcome.pdf

OARC-42-Welcome.pptx

2 Using Multiple Authoritative Vendors Does Not Work Like You Thought

Since the 2016 DDoS against Dyn, common wisdom has been to use multiple authoritative vendors for your edge.

We wanted to quantify the behavior of resolvers when one authoritative edge failed completely. To our surprise, some resolvers never shifted away from the failing authoritative servers.

This presentation covers this experiment and gives a summary of the results.

Speaker: Shane Kerr (NS1, an IBM Company)

Using Multiple Authoritative Vendors Does Not Work Like You Thought

3 DNS fingerprinting

The fpdns tool was first introduced more than 20 years ago. It gained popularity in both research and operational communities, and was actively used to fingerprint DNS software at scale. Unfortunately, it has not been updated for a while. We are currently working on its successor - fpdns2. We set up more than 670 instances of recursive resolver software from 7 vendors and issue a series of DNS queries that effectively distinguish vendors and versions. With this presentation, we wish to present the first prototype of our tool and hope to obtain some valuable feedback from the community.

Speaker: Ms Yevheniya Nosyk (Université Grenoble Alpes)

presentation.pdf

11:05 Mid-morning break

11:35 Session 2 start

OARC 42 Day 1: Session 2

Conveners: Petr Špaček (Internet Systems Consortium (ISC)), Suzanne Woolf (Public Interest Registry (.ORG))

4 Real world challenges with large responses, truncation, and TCP

Recent experience changing delegations for an akamai domain resulted in some unexpected outcomes and useful observations. To improve IPv6 reachability we recently updated delegations for a widely used domain, adding glue records.

Although it resulted in a DNS record size beyond 1223 bytes experienced team members conferred and felt the presence of numerous RFCs specifying the use of TCP for oversize answers would mitigate any potential issues.

What we discovered is while most resolvers switch to TCP because of the popularity of our domains there was measurable stress to certain nameservers.

In this talk we’ll explore what was changed, what should have happened, what actually happened, and how resolver setups affected these failures. And hopefully we can start a discussion about new best practices to prevent a recurrence!

Speakers: Duane Wessels (Verisign), Ralf Weber (Akamai Technologies)

challenges-truncation-tcp-combined v2.pdf

5 The Impact of Negative Caching and DNS Resolution Failures

Recursive resolvers can aggressively requery Root and TLD authoritative name servers when all authoritative name servers for a zone return REFUSED or SERVAIL. These resolution failures consist of the same query tuple <QNAME, QTYPE, QCLASS> being asked repeatedly at an unexpectedly high rate. The cause of the excessive traffic is almost always related to aggressive resolver retry logic and negative caching behavior. There are numerous conditions that can lead to resolution failures; regardless, this resolver behavior can significantly increase query load to the authoritative name servers.

This presentation will report on a longitudinal analysis of SERVFAIL-related requery traffic to two root name servers and the thirteen com/net authoritative name servers. The analysis will demonstrate that such traffic is consistent, persistent, and sizeable.

Speaker: Yannis Labrou (Verisign)

DNS-OARC42_Labrou_Thomas_v1_OARC42.pdf

6 DNSSEC signer upgrade at PCH.net

In my talk I plan to summarize the properties of
PCH's DNSSEC bump in the wire signer from 2010.
(which was based on bind9 and custom code in c/bash/perl)

What were our goals and motivations for the upgrade.

Why we ended up choosing knot as a replacement signer.

Our process include a keysigning ceremony where ZSKs are generated and RRSIGs for those keys years in advance.

I will talk about how we use the keysigning output with knot's offline-ksk functionality and how we utilize HSMs.

Another part to be presented is nsd and how we use its relatively recent functionality the verifier hook to check the correctness of the DNSSEC signed zones.

Speaker: Tamas Csillag (PCH.net)

DNS-OARC42-Tamás-2024-02-07-v4.pdf

DNS-OARC42-Tamás-2024-02-07-v6.pdf

12:45 Lunch

OARC 42 Day 1 Sponsor Session: Sponsor Session

Special Sponsor Talk Block

Convener: Denesh Bhabuta (DNS-OARC)

7 SPONSOR PRESENTATION: Using DNS to solve the latency challenge

Latency is a costly challenge that more enterprises are asking DNS to solve. In this session, we'll use data to make the case for expanding the traditional role of DNS to include traffic steering and Global Server Load Balancing. Using a reference architecture, we'll investigate the different options for selecting DNS pathways, and discuss weighting of various factors to improve DNS connection performance.

Speaker: Ekim Maurer (IBM)

20240202 -DNS OARC GSLB Presentation.pptx

14:15 Session 3 start

OARC 42 Day 1: Session 3

Conveners: Benjamin Schwartz (Meta), Puneet Sood (Google)

8 GOV multi-signer transition with NSEC/NSEC3

In 2023 operations for the .GOV TLD transitioned from Verisign to Cloudflare. One interesting aspect of this transition was the different approaches to DNSSEC signing by Verisign and Cloudflare. Whereas Verisign uses offline signing with RSA (algorithm 8) and NSEC3, Cloudflare generally uses online signing with ECDSA (algorithm 13) and NSEC.

Although the parties agreed to transition using only RSA, we wanted to test the statement in RFC 8901 ("Multi-Signer DNSSEC Models") that says "NSEC and NSEC3 can be used by different providers to serve the same zone." After extensive testing by both parties, we found no reasons why it shouldn't work, and this approach was used for the transition. To the best of our knowledge, this is likely to be the first time that a signed zone of such significance was operated using NSEC and NSEC3 at the same time.

Speaker: Christian Elmerot (Cloudflare)

gov-transition-nsec-nsec3.pdf

speaker ElmerotQuarter.png

9 Verisign's Transition to ECDSA

In 2023, Verisign changed the DNSSEC signing algorithm for the .EDU, .NET., and .COM TLDs from RSA (algorithm 8) to ECDSA (algorithm 13). In this presentation we describe our conservative, double-signing approach to the algorithm rollovers, and our observations on how DNS query traffic before, during, and after each rollover.

In particular, we make observations on how DNS glue truncation policies impact response sizes, and on the population of recursive resolvers that are unable to fall back to TCP for large, truncated UDP responses. We'll show metrics that we developed for our real-time dashboards to remain informed of potential problems and discuss options for mitigating any significant impacts.

Speaker: Duane Wessels (Verisign)

wessels-verisign-algrolls.pdf

10 KSK algorithm rollover for .nl

A presentation about the KSK algorithm rollover that was done for .nl.
This will be based on the information we already published a blog posts on our website.

https://www.sidn.nl/en/news-and-blogs/new-dnssec-algorithm-for-nl
https://www.sidnlabs.nl/en/news-and-blogs/algorithm-rollover-the-effects-on-our-network-traffic-and-resolvers
https://www.sidn.nl/en/news-and-blogs/looking-back-at-nls-algorithm-rollover

Speaker: Mr Stefan Ubbink (SIDN)

dns-oarc42_SIDN_KSK_algorithm_rollover_for_.nl.pdf

15:15 Mid-afternoon break

15:45 Session 4 start

OARC 42 Day 1: Session 4

Conveners: Benjamin Schwartz (Meta), Puneet Sood (Google)

11 DELEGate the Modern Way

A new delegation record is now being discussed for standardization at the IETF, the DELEG record. It is an extensible signal at the parent side of a delegation that modern resolvers can select authoritative nameservers based on features described in the delegation, not just their name. For example, support for existing standards like DNS over TLS or DNS over QUIC could be signaled efficiently in the normal resolution flow. Some day a new DNS message format could even be evolved.

Speaker: David Lawrence (Salesforce)

DNS-OARC 42 DELEGations++.pdf

12 Resolver Capability Testing Open Source Framework (DNS Research Federation)

Currently, the DNS community has limited visibility of the capabilities and deployed features of the millions of recursive resolvers in use across the internet. A helpful source of data has been provided by APNIC over the last decade or so by the use of Google Ads but it has been felt that having alternative more accessible methods of collecting this data would be advantageous and provide more flexibility going forward.

The DNS Research Federation have been commissioned by ICANN OCTO to develop an open source testbed suitable for use by organisations with high query traffic, utilising popular open source DNS and Web servers to facilitate the testing of Resolver Capabilities and rational collection of data for analysis. The intention is to provide a good range of DNS feature tests out of the box to allow organisations to perform specific feature testing for themselves as well as to encourage contribution of general data trends to the wider community.

As development of this tool is scheduled for completion in advance of DNS-OARC 42 we would like to take this opportunity to present both the architecture and design of the testing tool as well as to potentially share some initial findings arising from use of the tools in initial testing.

Speakers: Mr Mark Robertshaw (DNS Research Federation), Mr Peter Spain (DNS Research Federation)

DNS-OARC 42 - Resolver Capability Testing.pptx

18:00 Social Event - walking distance from venue

Details provided to in-person delegates

Friday, 9 February

09:30 Registration

10:00 Session 1 start

OARC 42 Day 2: Session 1

Conveners: Suzanne Woolf (Public Interest Registry (.ORG)), Puneet Sood (Google)

13 TuDoor Attack: Systematically Exploring and Exploiting Logic Vulnerabilities in DNS Response Pre-processing with Malformed Packets

DNS can be compared to a game of chess in that its rules are simple, yet the possibilities it presents are endless. While the fundamental rules of DNS are straightforward, DNS implementations can be extremely complex. In this study, we intend to explore the complexities and vulnerabilities in DNS response pre-processing by systematically analyzing DNS RFCs and DNS software implementations. We present the discovery of three new types of logic vulnerabilities, leading to the proposal of three novel attacks, namely the TuDoor attack. These attacks involve the use of malformed DNS response packets to carry out DNS cache poisoning, denial-of-service, and resource consuming attacks. By performing comprehensive experiments, we demonstrate the attack’s feasibility and significant real-world impacts of TuDoor. In total, 24 mainstream DNS software, including BIND, PowerDNS, and Microsoft DNS, are affected by TuDoor. Attackers can instigate cache poisoning and denial-of-service attacks against vulnerable resolvers using a handful of crafted packets within 1 second or circumvent the query limit to deplete resolution resources (e.g., CPU). Besides, to determine the vulnerable resolver population in the wild, we collect and evaluate 16 popular Wi-Fi routers, 6 prevalent router OSes, 42 public DNS services, and around 1.8M open DNS resolvers. Our measurement results indicate that TuDoor could exploit 7 routers (OSes), 18 public DNS services, and 424,652 (23.1%) open DNS resolvers. Following the best practice of responsible disclosure, we have reported these vulnerabilities to all affected vendors, and 18 of them, including BIND, Chrome, Cloudflare, and Microsoft, have acknowledged our findings and discussed mitigation solutions with us. Furthermore, 33 CVE IDs are assigned to our discovered vulnerabilities, and we provide an online detection tool as one of the mitigation measures. Our research highlights the urgent need for standardization of DNS response pre-processing logic to enhance the security of DNS.

Speaker: Qifan Zhang (University of California, Irvine)

oarc42-tudoor-li.pdf

14 Analysis of NXDOMAIN data from an openresolver perspective in China

The Domain Name System (DNS) is a fundamental protocol of the Internet. Enhancing its efficiency requires in-depth analysis of DNS data. The analysis of negative responses (with a focus on the NXDOMAIN response code of 3 in this paper) is a critical area of research, as it directly affects the security and performance of servers from the root to the recursive level. In September 2022, a bug in a Chinese top-level application caused users' MAC addresses being led to the DNS query system as domain name requests. This issue affected several DNS servers, including root servers.
As the operator of an open resolver that carries approximately 5% of DNS traffic in China, we have developed a monitoring system that tracks negative responses in the DNS system and identifies anomalies in a timely manner. This system also enables regional analysis of DNS efficiency, providing insight into performance in different areas.
To uncover anomalies within this data, we clustered NXDOMAIN names into distinct patterns based on their string characteristics and analysed the causes of these patterns. We found that the causes ranged from server misconfiguration such as DNS suffixes, to application-related causes, such as DNS blacklisting, reverse lookups, or chromids. By monitoring the long-tail, non-patterned NXDOMAIN query names, the proposed system can identify and actively monitor the domain name patterns of recent large-scale NXDOMAIN events. In addition, from a regional perspective, there are significant differences in the rate of NXDOMAIN responses between different provinces in China, likely due to the specific equipment used by local operators.
We hope to collaborate with the community to enhance the efficiency and security of the DNS system.

Speaker: Jinghua Bai (Qi An Xin Technology Group Inc.)

Deep Dive into NXDOMAIN Data in China

dns-oarc42_Deep Dive into NXDOMAIN Data in China.pdf

15 ResolverFuzz: Automated Discovery of DNS Resolver Vulnerabilities with Query-Response Fuzzing

Domain Name System (DNS) is a critical component of the Internet. DNS resolvers, which act as the cache between DNS clients and DNS nameservers, are the central piece of the DNS infrastructure, essential to the scalability of DNS. However, finding the resolver vulnerabilities is non-trivial, and this problem is not well addressed by the existing tools. To list a few reasons, first, most of the known resolver vulnerabilities are non-crash bugs that cannot be directly detected by the existing oracles (or sanitizers). Second, there lacks rigorous specifications to be used as references to classify a test case as a resolver bug. Third, DNS resolvers are stateful, and stateful fuzzing is still challenging due to the large input space.

In this paper, we present a new fuzzing system termed ResolverFuzz to address the aforementioned challenges related to DNS resolvers, with a suite of new techniques being developed. First, ResolverFuzz performs constrained stateful fuzzing by focusing on the short query-response sequence, which has been demonstrated as the most effective way to find resolver bugs, based on our study of the published DNS CVEs. Second, to generate test cases that are more likely to trigger resolver bugs, we combine probabilistic context-free grammar (PCFG) based input generation with byte-level mutation for both queries and responses. Third, we leverage differential testing and clustering to identify non-crash bugs like cache poisoning bugs. We evaluated ResolverFuzz against 6 mainstream DNS software under 4 resolver modes. Overall, we identify 23 vulnerabilities that can result in cache poisoning, resource consumption, and crash attacks. After responsible disclosure, 19 of them have been confirmed or fixed, and 15 CVE numbers have been assigned.

Speaker: Mr Qifan Zhang (University of California, Irvine)

ResolverFuzz-OARC42-zhang.pdf

11:10 Mid-morning break

11:40 Session 2 start

OARC 42 Day 2: Session 2

Conveners: Suzanne Woolf (Public Interest Registry (.ORG)), Puneet Sood (Google)

16 DNS Benchmarking 101: Essentials and Common Pitfalls

How to meaningfully benchmark DNS systems?

We will cover the main methodological differences between:

• resolvers
• authoritative servers
• normal traffic
• DoS traffic

By end of the talk the audience will learn which tools are suitable for what scenarios and how to avoid the most common pitfalls.

If you prefer lengthy wording, here it is!

Discover the art of conducting meaningful DNS system benchmarks as we delve into the intricacies of methodology. This talk will explore the key methodological distinctions among resolvers, authoritative servers, normal traffic, and DoS traffic.

Join us in unraveling the mysteries of suitable tools for various scenarios. Learn how to navigate the landscape of DNS benchmarking with insights into the most common pitfalls and valuable strategies to avoid them. Whether you're a beginner or an experienced professional, this session will equip you with the knowledge to benchmark DNS systems effectively.

Speaker: Petr Špaček (Internet Systems Consortium (ISC))

pspacek.pdf

video recording

17 DNS Server Performance Tuning in Linux

NS1 Managed DNS runs on a large anycast deployment with approximately 25 POPs. We frequently need to assess new hardware for upgrades and expansion, and to do so, we must understand how our software performs. This presentation will discuss how DNS packets are processed on Linux, how the software interacts with hardware, and how to configure the software for optimal performance. We will also describe how we validate the configuration with benchmarks and share interesting findings from our latest testing.

Speaker: Jan Včelák (NS1, an IBM Company)

dns_server_performance_tuning_on_linux_draft.pdf

18 Cloud DNS Monitoring in a Large Scale

Our presentation at DNS OARC 42 focuses on developing and operating a robust DNS monitoring system, across various environments, including traditional and cloud infrastructures. We will discuss our journey in managing a large-scale DNS monitoring setup, consisting of more than 2500 zones distributed across 500 service instances and 15 regions all over the world, each zone containing 2000 - 8000 records.
The session will present the operational strategies and challenges involved in monitoring such a complex and large-scale DNS infrastructure, with a focus on key areas like zone delegation, resolution, and query latency. Additionally, our talk will examine both public and private DNS query logging, emphasizing their role in enhancing resolution monitoring and improving overall system efficiency.
Lastly, we will explore strategies for building a robust and comprehensive DNS monitoring system, leveraging the strategic implementation of cloud platforms to create solutions that are operationally effective, scalable, and adaptable to various environments.

Speaker: Sidan Qi (Salesforce)

Cloud DNS Monitoring.pdf

DNS-OARC42-demo-Sile's part.mov

12:50 Lunch

14:20 Session 3 start

OARC 42 Day 2: Session 3

Conveners: Petr Špaček (Internet Systems Consortium (ISC)), Benjamin Schwartz (Meta)

19 Is the DNS ready for IPv6?

There is a new draft in the IETF that proposes that all recursive resolvers and authoritative servers SHOULD include IPv6 service. But is the DNS ready for IPv6? This presentations looks at the problems that the DNS has with IPv6, arond the issues of IP fragmentation using large UDP payloads and the consequences of this in terms of delayed resolution and increased query loads.

The intended method of mitigation for these issues is encapsulated in the DNS Flag Day 2020 recommendations, dropping the ENDS(0) Buffer size value to 1232. The question is how well is all this working in the DNS today. This presentation will look at the measurements of query-weighted distribution of recursive resolver EDNS Bufffer Sizes, the resolver query patterns as ween at an authoritative server and address the overall efficiency of IPv6 in this context.

Speaker: Geoff Huston (APNIC)

2024-02-01-dns-oarc.pdf

20 Traffic Analysis of Fluctuating Flows (TAFFy)

Network and security operators are continually bombarded by strange deviations in network traffic that are sometimes operationally problematic, sometimes a threat to security, and other times just plain odd. These show up as large traffic spikes sometimes, and other times are just low-level plateaus. It's often very hard to quickly figure out exactly what these spikes come from. Wouldn't it be wonderful to have a tool that accurately tells you exactly what has changed in these traffic profiles down to individual protocol fields? This new project is designed to do just that. Though generic by design, it is being most heavily tested by the author to look for anomalies received at DNS authoritative servers, which will be the focus of the presentation for DNS-OARC.

Speaker: Wes Hardaker (USC/ISI)

oarc42-hardaker-taffy.pdf

oarc42-hardaker-taffy.pptx

15:15 Mid-afternoon break

15:45 Session 4 start

OARC 42 Day 2: Session 4

Conveners: Petr Špaček (Internet Systems Consortium (ISC)), Benjamin Schwartz (Meta)

21 RIPE DNS Resolver Recommendations

The RIPE community created a task force in response to the DNS4EU effort, with the goal of producing a document which provides recommendations for operators interested in running DNS resolvers, especially public resolvers.

This lightning talk introduces this work, and gives a quick review of the document, and the status.

Speaker: Shane Kerr (NS1)

RIPE DNS Resolver Recommendations TF: OARC 42 Lightning Talk Edition

22 Estimate the size of the DNS

Help crowd source a high level estimate of the "size" of the world-wide DNS system, in order to help compare the growth and size of the DNS root, to the growth and size of the DNS overall.

If you were to draw four clouds, representing the traffic to All Authoritative Servers, All Resolvers, All TLDs, and the DNS Root, what would be the relative size of each cloud? It would be helpful to be able to display this in order to convey to less-technical audiences the relative size of each of these systems.

Speaker: Victoria Risk (Internet Systems Consortium)

ISClightningTalk42.pptx

23 Expired RRSIG - answer or not?

This brief talk is about the problem of expired RRSIGs. I will talk about a couple of scenarios which result in this situation, and how we might want to react.

Speaker: Mr Anand Buddhdev (RIPE NCC)

OARC42_Lightning.pdf

24 DNS features matrix, implementation quirks

Different DNS vendors implement different features (IXFR support, EDNS expiry support, ZONEMD verification).
These are not 100% the same. Some of these are documented, some are not.
For zonemd support one can go to a DNS oarc talk, catalog zones has a website.
dnstap has a website.

Maybe there should be a directory or wiki page for these?
Would it be a good idea to have such at DNS OARC?

Speaker: Tamas Csillag (PCH.net)

Implementation-differences-Tamas-OARC42_2024-02-09-v1.pdf

25 OARC 42 - Closing remarks

Speaker: Phil Regnauld (DNS-OARC)

OARC-42-Wrapup.pdf

OARC-42-Wrapup.pptx