OARC 36 (Online)

Jan Včelák (NS1), Keith Mitchell (DNS-OARC), Manu Bretelle (Facebook), Pallavi Aras (Salesforce), Willem Toorop (NLnet Labs)

OARC 36 will be an online Workshop starting at 14:00 UTC on Monday and 10:00 UTC on Tuesday.

DNS-OARC is a non-profit, membership organization that seeks to improve the security, stability, and understanding of the Internet's DNS infrastructure. Part of these aims are achieved through workshops.

DNS-OARC Workshops are open to OARC members and to all other parties interested in DNS operations and research.

Social Media hashtag: #OARC36

Mattermost Chatroom: Workshops on chat.dns-oarc.net (sign-up here)




Sponsorship opportunities for OARC 36 are available. Details at:



Your company name here?

Annual Workshop Patrons for 2021 are available. Details at:




  • Abel Martin
  • Abhijeet Gaiha
  • Adis Pinjic
  • Ali Saleh
  • Allison Mankin
  • Amanda Swain
  • Anand Buddhdev
  • Andreas Taudte
  • Andrew Kaizer
  • Ann Paul
  • Arnaud Jolivet
  • Arsen Stasic
  • Arun Mudiraj
  • Baula Xu
  • Benno Overeinder
  • Brad Ellis
  • Brett Carr
  • Brian Dickson
  • Brian King
  • Brian Somers
  • Bryan Hughes
  • Casey Deccio
  • Cathy Almond
  • Christian Petrasch
  • Claudia Erices
  • Daniel Boughton
  • Daniel Stirnimann
  • David Blacka
  • David Couture
  • David Feldman
  • David Lawrence
  • Denesh Bhabuta
  • Douglas Dickinson
  • Duane Wessels
  • Eduardo Mercader
  • Erik Bishop
  • Felipe Barbosa
  • Florian Obser
  • Florian Rosenegger
  • Francisco Arias
  • Gaetan Gautier
  • Gary Lu
  • Gavin Mccullagh
  • Geoffrey Huston
  • Gonzalo Romero
  • Greg Choules
  • Han Zhang
  • Hanieh Bagheri
  • Hazel Smith
  • Hirofumi Hotta
  • Hugo Salgado
  • Jacob Zack
  • Jakob Dhondt
  • James Richards
  • James Stadtmiller
  • Jan Horak
  • Jan Včelák
  • Jarle Fredrik Greipsland
  • Jaromir Talir
  • Jason Lavigne
  • Jerry Lundström
  • Joacim Sørheim
  • João Luis Silva Damas
  • Joeri De Ruiter
  • John Todd
  • Jonas Andersson
  • Karl Reuss
  • Kazunori Fujiwara
  • Keith Mitchell
  • Klaus Darilion
  • Kundana Pillari
  • Latour Latour
  • Linjian Song
  • Magnus /Mem/ Sandberg
  • Manu Bretelle
  • Marc Dacier
  • Marco Diaz
  • Mark Teodoro
  • Martin Holub
  • Mat Ford
  • Matthew Pounsett
  • Mauricio Vergara Ereche
  • Michael Braunöder
  • Michael Daly
  • Miles Mccredie
  • Moritz Müller
  • Nicolai Leymann
  • Otto Moerbeek
  • Pallavi Aras-Mathai
  • Patrick Mevzek
  • Paul Ebersman
  • Paul Hoffman
  • Peter Davies
  • Peter Hessler
  • Peter Janssen
  • Peter Koch
  • Peter Thomassen
  • Peter Van Dijk
  • Petr Špaček
  • Pieter Lexis
  • Prashanth Suvarna
  • Priya Mohan
  • Puneet Sood
  • Rafael De Elvira
  • Raffaele Sommese
  • Ralf Weber
  • Ray Bellis
  • Richard Wilhelm
  • Robert Story
  • Roger Murray
  • Roy Arends
  • Sam Weiler
  • Sara Dickinson
  • Shane Kerr
  • Shinta Sato
  • Shreyan Sarkar
  • Shuai Hao
  • Shumon Huque
  • Sidan Qi
  • Siôn Lloyd
  • Stefan Ubbink
  • Steve Sullivan
  • Suzanne Woolf
  • Thibaud Duble
  • Tijay Chung
  • Tim Wicinski
  • Ulrich Wisser
  • Vincent Levigneron
  • Vinothkumar Prabhakaran
  • Willem Toorop
  • William Fleetwood
  • Wouter De Vries
  • Yoshitaka Aharen
  • Monday, November 29
    • 1:00 PM 2:00 PM
      OARC Software AMA

      Welcome to another OARC Software Ask Me Anything (AMA) which will be hosted by our Software Engineer Jerry Lundström.

      In this session you can ask us anything about our open source software or services, and there is possibility to demonstrate features if so requested.

      Details on how to join this meeting will be sent to all delegates before the conference starts.

      Please see https://www.dns-oarc.net/oarc/software for more information about our Software Development & Funding opportunities, and feel free to join us on Mattermost.

    • 2:00 PM 2:45 PM
      OARC 36 Day 1: Session 1
      • 2:00 PM
        Welcome 15m
        Speaker: Mr Keith Mitchell (DNS-OARC)
      • 2:15 PM
        DNS-OARC Privacy Committee Report 15m
        Speakers: Benno Overeinder (NLnet Labs), Sara Dickinson (Sinodun IT)
      • 2:30 PM
        Characterization of Anycast Adoption in the DNS Authoritative Infrastructure 15m

        Anycast has proven to be an effective mechanism to
        enhance resilience in the DNS ecosystem and for scaling DNS
        nameserver capacity, both in authoritative and the recursive
        resolver infrastructure. Since its adoption for root servers,
        anycast has mitigated the impact of failures and DDoS attacks
        on the DNS ecosystem. In this work, we quantify the adoption
        of anycast to support authoritative domain name service for toplevel and second-level domains (TLDs and SLDs). Comparing two
        comprehensive anycast census datasets in 2017 and 2021, with
        DNS measurements captured over the same period, reveals that
        anycast adoption is increasing, driven by a few large operators.
        While anycast offers compelling resilience advantage, it also shifts
        some resilience risk to other aspects of the infrastructure. We
        discuss these aspects, and how the pervasive use of anycast merits
        a re-evaluation of how to measure DNS resilience.

        Speaker: Raffaele Sommese
    • 2:45 PM 3:00 PM
      15 Minutes Break 15m
    • 3:00 PM 4:10 PM
      OARC 36 Day 1: Session 2
      • 3:00 PM
        Advertising DNS Protocol Use to Mitigate DDoS Attacks 25m

        The Domain Name System (DNS) has been frequently abused for distributed denial-of-service (DDoS) attacks and cache poisoning because it relies on the User Datagram Protocol (UDP). Since UDP is connection-less, it is trivial for an attacker to spoof the source of a DNS query or response. While other secure transport mechanisms provide identity management, such as the Transmission Control Protocol (TCP) and DNS Cookies, there is currently no method for a client to state that they only use a given protocol. This paper presents a new method to allow protocol enforcement: DNS Protocol Advertisement Records (DPAR). Advertisement records allow Internet Protocol (IP) address subnets to post a public record in the reverse DNS zone stating which DNS mechanisms are used by their clients. DNS servers may then look up this record and require a client to use the stated mechanism, in turn preventing an attacker from sending spoofed messages over UDP. In this paper, we define the specification for DNS Protocol Advertisement Records, considerations that were made, and comparisons to alternative approaches. We additionally estimate the effectiveness of advertisements in preventing DDoS attacks and the expected
        burden to DNS servers.

        Speaker: Casey Deccio (Brigham Young University)
      • 3:25 PM
        A Measurement-based Investigation of DNS Hijacking 15m

        Attacks against DNS have long plagued the Internet, requiring continual investigation and vigilance to prevent the abuse of this critical infrastructure. In recent years, the severity of DNS hijacking has motivated renewed interest in developing more robust defenses. The size, dynamism, and diversity of the DNS ecosystem present nontrivial challenges to crafting an effective and scalable defense. Further, the relative rarity of documented DNS hijacking attacks makes them difficult to study in-depth. In this study, we attempt to address the challenges in two thrusts. We first conduct an analysis based on the reports of confirmed DNS hijacking attacks and passive DNS records to characterize known DNS hijacking attacks and identify features for building defense mechanisms. Then we explore the extent to which the characteristic features can be used to build a DNS hijacking detection mechanism and evaluate its effectiveness from the perspective of a network gateway.

        Speaker: Shuai Hao (Old Dominion University)
      • 3:40 PM
        Slack’s DNSSEC Rollout: Third Time’s the Outage 25m

        On September 30th 2021, Slack had an outage that impacted less than 1% of our online user base, and lasted for 24 hours. This outage was the result of our attempt to enable DNSSEC, but which ultimately led to a series of unfortunate events.

        On this talk we'll cover our DNSSEC rollout to all Slack critical domains and the three failed attempts to enable DNSSEC on slack.com ­– doing a deep dive into our third attempt (the Sept 30th outage) – where we'll cover what was done during the outage, why we did it and ultimately the root cause of the outage, which was a bug in the DNSSEC implementation on our cloud provider authoritative DNS server.

        Speaker: Rafael de Elvira (Senior Software Engineer @ Slack)
      • 4:05 PM
        Day 1 Wrap Up 5m
        Speaker: Mr Keith Mitchell (DNS-OARC)
    • 4:10 PM 5:10 PM
      Let's Chat! Social Event 1h
  • Tuesday, November 30
    • 10:00 AM 10:45 AM
      OARC 36 Day 2: Session 1
      • 10:00 AM
        Measurement of DNSSEC Validation with RSA-4096 20m

        IN this presentation we motivate why there is a need to consider using larger keys for RWSA in the context of DNSSEC. We then describe a measurement experiment that looks at the success rate of using 4,096 bit signing keys in DNSSEC. We conclude with some thoughts as to the option between using ECDSA P-256 and RSA-4096 to counter the potential threat of quantum computing to this form of application of cryptography.

        Speaker: Geoff Huston (APNIC)
      • 10:20 AM
        An analysis of DNSSEC signed domains appearing within a large web crawl dataset 25m

        This presentation will discuss the analysis of domain names found within the Common Crawl dataset. It will cover the method of analysis and will focus on the characteristics of DNSSEC with regard to different types of web page content. It will include discussion on the proportion of websites that use content from only signed domains, the kinds of content in websites that are more likely to be delivered via a signed domain, and the variance in different DNSSEC algorithms seen across the dataset. It is hoped this analysis will offer an interesting perspective on the rate of DNSSEC deployment within different website components.

        Speaker: James Richards (Nominet)
    • 10:45 AM 11:00 AM
      15 Minute Break 15m
    • 11:00 AM 11:55 AM
      OARC 36 Day 2: Session 2
      • 11:00 AM
        Status Update on Authenticated Bootstrapping of DNSSEC Delegations 20m

        We present an update to the development and implementation of the DNSSEC Bootstrapping protocol since OARConline 35a. Protocol developments include, in particular, some structural changes to enable straightforward discovery of delegations which are ready for bootstrapping, on a per-parent basis. On the implementation side, we show prototype implementations for both the child (DNS operator) as well as the parental (registry/registrar) sphere, and give an overview of the implementation plans and status at various DNS operators and registries/registrars.

        Speaker: Peter Thomassen (deSEC)
      • 11:20 AM
        The PROXY protocol 20m

        Over the years, DNS proxies (such as dnsdist) have cooperated with
        their backends to pass the real IP of the client to those backends.
        edns-client-subnet has been abused for this, there was an attempt in
        the IETF to standardise a new EDNS option (XPF), but DNSOP did not like

        Based on that, and other operational insights, PowerDNS (in dnsdist,
        the Authoritative, and the Recursor) has decided to adopt the PROXYv2
        protocol from the HAproxy developers (
        https://www.haproxy.org/download/2.4/doc/proxy-protocol.txt), and we
        know ISC has support for it on the roadmap as well.

        As it would be good to have interoperability between all vendors in
        this area, this talk will give a quick overview of the existing solutions, their drawbacks, and an introduction to the PROXY protocol.

        Speakers: Pieter Lexis (PowerDNS.COM / Open-Xchange), Peter van Dijk (PowerDNS)
      • 11:40 AM
        Querying the Public Suffix List via the DNS 15m

        Information from the Public Suffix List (PSL) is required in various contexts, for example for cookie scoping in browsers, for certificate issuance, and for the secure operation of authoritative multi-tenant nameservers. Applications depending on the PSL customarily bring their own copy of the list, and thus require mechanisms to parse and interpret the list, and to keep it up to date.

        The PSL Query Service removes the need for applications to parse or refresh the PSL altogether. Based on a mapping of the PSL onto the DNS, it instead facilitates single-query lookups to immediately retrieve the public suffix that is associated with a given name.

        In this short contribution, I discuss the motivation, design, and applications of the PSL Query Service and explain its internal structure, before reflecting on privacy concerns that may arise. The talk ends with considerations about whether and how to better integrate the service into the Internet community.

        Speaker: Peter Thomassen (deSEC)
    • 11:55 AM 1:30 PM
      95 Minute Break 1h 35m
    • 1:30 PM 2:15 PM
      OARC 36 Day 2: Session 3
      • 1:30 PM
        Understanding DNSSEC debugging patterns using DNSVIZ. 15m

        DNS Security Extensions (DNSSEC) were introduced nearly two decades ago to provide integrity and authenticity of DNS messages. There have been some studies focusing on how DNSSEC has been deployed over years using active scans, which commonly reported pervasive mismanagement such as missing DS records.

        From the domain administrator perspective, however, it is hard to understand what makes it really challenging to deploy and "manage" DNSSEC, or to fix errors; for example, answering the question of "how long do usually take for DNS administrators to resolve a specific DNSSEC error?" is nontrivial.

        To shed a light on these questions, we leverage DNSVIZ (dnsviz.net), which is one of the most extensive and popular tools for debugging DNSSEC errors. It helps domain name owners and others help understand the current DNSSEC status of a domain name and diagnoses the problem if exists.

        With 7 years of DNSViz dataset that contain DNS debugging histories of domains, we would like to share our preliminary findings.

        Speaker: Dr Tijay Chung (Virginia Tech)
      • 1:45 PM
        A DNS Anomaly Detection Model to Identify Unusual Lagging in Zone Updates 15m

        In large-scale DNS deployments, zone updates are made by DNS hosting services on short timescales to large numbers of servers. It’s normal for the updates to be somewhat asynchronous from each other because DNS is “eventually consistent” by design. The question for DNS operations is how much lag is OK. For our organization, customer sensitivity to stale information is high and we want to take action by asking the hosting services to check for server problems when there are anomalous lags, but which lags are anomalous? What threshold of difference in the SOA serial numbers indicates a problem?

        This talk presents a project to use an unsupervised machine learning algorithm to identify the anomalous points in lag data to help with this question of what is actionable. The approach was to use the PyCaret open-source Python machine learning library to train and build a model. We chose PyCaret to start with because it includes many unsupervised ML algorithms and then we selected the Isolation Forest (iforest) clustering ML algorithm, an algorithm commonly used for anomaly detection because it requires limited memory and is relatively fast in performance.
        Additionally, after the model was trained using unsupervised learning, it was fine-tuned with the probe_time of the data points given as its supervised_target. Fine-tuning performs additional training by focusing on update times to improve the model’s performance. 

        The model was trained on DNS monitor logs that include information about zone updates made by DNS hosting services. This model was trained on SOA serial number changes in the multiple anycast locations for three production zones, my.salesforce.com, salesforce.com, and force.com. The output from the model marks each data point as an anomaly or normal and also provides a specific anomaly score where the higher the score, the more anomalous the datapoint. Analyzing the model output gives information regarding zone updates. This information includes learning about trends and patterns of zones, ranges of lags, lag dependence on zone sizes, etc. 

        One of the results of the project was to provide good numeric thresholds for how much lag was normal, now much anomalous. The thresholds were extremely different for the three zones (see slide #7). The thresholds indicated that lagging depended on zone sizes and how frequently the zones were updated, which is not surprising, but the actual numbers were surprising. These thresholds are accurate because they mark where truly anomalous behaviors begin. The original default thresholds that were in use prior to the model were guesstimates by the team and were the same for all three zones and had not proven predictive. 

        The next step of the project is to automate the model prediction by periodically running the model and integrating the new thresholds into the monitoring system (as well as preserving the history). 

        ML has been used on DNS data frequently, especially focusing on DNS attacks and security. DNS anomaly detection plays a big role operationally in DDOS mitigation for example. But this work covers an area in DNS availability and it seems likely that ML models, both unsupervised and supervised, can be valuable to many other areas where DNS operators have data in real-time in the future.

        Speaker: Kundana Pillari
      • 2:00 PM
        Disaster Recovery with DNSSEC 15m

        What if you loose all your DNSSEC keys for your zone, what is the impact and how can you recover from this?

        Speaker: Mr Stefan Ubbink (SIDN)
    • 2:15 PM 2:30 PM
      15 Minute Break 15m
    • 2:30 PM 3:30 PM
      OARC 36 Day 2: Session 4
      • 2:30 PM
        Independent Performance Verification of XDP-accelerated Knot DNS 25m

        Over the last year, Knot DNS has added support for Linux’s eXpress Data Path. CIRA and CENGN, with the help of CZ.NIC, took a deep dive into characterizing Knot DNS’s performance with XDP on both UDP and TCP-based queries. On a 36-core server, 16M QPS was achieved on UDP-based queries, and 1.3M QPS on TCP-based queries.

        This presentation will detail the hardware, software, configurations and testing methodologies used to achieve these results. We will share the successes and failures we experienced along the way. We also propose a unit of measurement to allow rough performance comparisons between different processor types: QPS/core-GHz.

        Speaker: Mr Brad Ellis (CENGN (Centre of Excellence in Next Generation Networks))
        Slide decks
      • 2:55 PM
        Lightning Talk: NSEC breakage survey 15m

        Some DNS authoritative servers provide incorrect proofs of non-existence which correctly DNSSEC-validate but deny existence of data which actually do exist in the zone. Consequently, this causes silent resolution failures on resolvers which implement aggressive use of DNSSEC-validated cache (RFC 8198).

        This talk aims to provide very short glimpse to where the breakage can be found in wild on the Internet.

        Speaker: Petr Špaček (Internet Systems Consortium (ISC))
      • 3:10 PM
        Lightning Talk: Bringing up DoH 15m

        Bringing up DoH services at OpenDNS

        • Phase 1 - DoH as a proxy
        • Phase 2 - Native support
        • Phase 3 - Native deployment
        Speaker: Brian Somers (OpenDNS/Cisco)
      • 3:25 PM
        Wrap-Up 5m
        Speaker: Mr Keith Mitchell (DNS-OARC)