OARC 26 (Madrid)

14 May 2017, 08:30 → 15 May 2017, 20:00 Europe/Madrid

Keith Mitchell (DNS-OARC), Ray Bellis (Internet Systems Consortium, Inc.)

 

DNS-OARC is traveling to Madrid, Spain for its 26th Workshop!

DNS-OARC is a non-profit, membership organization that seeks to improve the security, stability, and understanding of the Internet's DNS infrastructure. Part of these aims are achieved through workshops.

The workshops are open to OARC members and to all other parties interested in DNS operations and research, with attendees from ICANN GDD Industry Summit 2017, the 6th Registration Operations Workshop, and the ICANN DNS Symposium particularly welcome this time around - as OARC 26 takes place just after these in the same venue in Madrid. Attendance is free for OARC Members, Speakers and Sponsors. There are charges for other attendees and late registrations.

We are thankful to our Host and sponsors for supporting OARC 26.

---

HOST

 

---

PREMIUM SPONSOR

 

---

PATRON SPONSOR

---

Webcast Video

• Sunday morning
• Sunday afternoon
• Monday

Jabber: dns-operations@conference.dns-oarc.net

Twitter hashtag: #OARC26

Sponsors: We have various sponsor opportunities for OARC 26, and OARC 27.

If your organization is interested in sponsoring OARC workshops, please see our Sponsor Benefits or e-mail sponsor@dns-oarc.net for more information.

Slides

• OARC26-Static-Slides-d2an.pdf
• OARC26-Static-Slides.pdf
• OARC26-Static-Slides.pdf_day2

Sunday, 14 May

08:30 → 09:30 Registration (OARC Members only)

For pre-registered OARC Members to pick up their badges or for OARC members to register.

Non-member registration starts at 11:00 CEST.

09:30 → 11:00 OARC Business

Convener: Mr Keith Mitchell (DNS-OARC)

09:30 Introduction and Welcome

Speaker: Paul Ebersman (Comcast)

09:35 OARC Status Report

Speaker: Mr Keith Mitchell (DNS-OARC)

Slides OARC26-status.odp OARC26-status.pdf

10:15 OARC Engineering Report

Speakers: Mr Jerry Lundström (DNS-OARC), Mr Keith Mitchell (DNS-OARC)

Slides OARC26-DNS-OARC-Software.pdf OARC26_Systems_Update.pdf

10:55 Recognition Announcements

Speaker: Mr Keith Mitchell (DNS-OARC)

Slides OARC26-day2.odp OARC26-day2.pdf

11:00 → 11:30 Morning Coffee Break & General Registration (Members & Non-Members)

11:30 → 12:30 Public Workshop

11:30 Scaling up: How we made millions of domains happier

Cloudflare hosts managed DNS infrastructure for over 5 million zones. In 2016 we began work on re-building a core part of our DNS Nameserver (rrDNS) and data provisioning software to better handle the scale as well as to improve reliability and performance, and pave the way for new features. DNS operations and systems are not immune from scaling bumps; things that work great for 100K domains may not work well for 10M. There are many different ways to represent and distribute zone information, we will talk about how we gained significant improvements by careful design. Wwe’ll give a brief overview of Cloudflare’s DNS infrastructure, explain why we chose to re-write certain aspects from the ground up, and show how we seamlessly migrated customers with negligible impact over a period of weeks. We made extensive use of a new testing infrastructure that we call DNS-mirror

Speaker: Mr Pavel Odintsov (Cloudflare)

Slides RRDNS_-_How_we_made_over_5_million_domains_happier_v6.key RRDNS_-_How_we_made_over_5_million_domains_happier_v6.pdf

12:00 Does parent delegation TTL matter?

I would like to study and present the effect of parent (TLD) zone TTL changes based on behavior of different DNS resolvers implementations and how they handle the delegation NS TTLs. The presentation will include different scenarios where TTL matter, might matter, might not matter, or doesn't matter at all related to the DNS resolver implementations.

Speaker: Mr Ondrej Sury (CZ.NIC)

Slides Does_parent_delegation_NS_TTL_matter-.pdf

12:30 → 14:00 Lunch

14:00 → 16:00 Public Workshop: Measurement

14:00 Scoring the Root Server System

There are 12 different root server anycast constellations, and they all serve the same root zone data. But do they all serve this data the same way? In particular, when the response size is large do all these root server systems respond in the same manner? This is a report on the different forms of responses that were observed when the root servers were coerced into offering a larger than normal response. To illustrate the differences in behaviour that were observed, the root servers are scored on a six star scale and the report will show the results.

Speaker: Mr Geoff Huston (APNIC)

Slides 2017-05-14-scoring-roots.pdf

14:30 DNS over TCP as seen from the authoritative servers

The vast majority of DNS traffic happens over UDP. We can pretty much predict how the resolvers will spread the load, deal with timeouts and server failures. But what will happen if we force the clients to use TCP? Is TCP a first-class citizen in modern resolvers or is it only a fallback mechanism. And can the resolvers use established TCP connections effectively? In this talk, I will try to answer some of these questions based on the analysis of real-world traffic patterns we see in the NS1's DNS network.

Speaker: Jan Včelák (NS1)

Slides Git repository with used tools and scripts Slides

15:00 On a software-based approach to generate and detect flooding attacks against DNS infrastructure

In this presentation we show our ongoing work to develop a testbed --based on software and commodity hardware-- to research on flooding attacks against DNS infrastructure. We have currently developed two prototype components: a flooding DNS query generator, able to saturate 10GbE links with 11Mrps, and an online detector of overabundant queried domains at reception. Relying on DPDK and libmoon (a LuaJIT framework for DPDK), these two tools run on commodity hardware, while optimizing the number of packets that we can handle at transmission and reception. Both generation and reception tools run Lua scripts, achieving a high level of flexibility. In this presentation we show some lessons we are learning, we compare the generator against other available tools, and present some unexpected results. For example, how a slower software query generator has a stronger impact on a Bind server than our current flooding tool (650Krps versus 10Mrps). We also describe how we count the number of queries per domain at reception under 11Mrps traffic, with reduced packet losses. Given the high number of possible elements to analyse from the DNS messages (IP addresses, random qnames) we make use of statistical tools, mainly CountMin-Sketch, to restrict the use of memory space. This tool can trigger an alarm when a domain exceeds a threshold of queries per a small interval of time. In this presentation we also look for feedback from the DNS-OARC community about possible strategies to use this tools to countermeasure flooding attacks.

Speaker: Santiago Ruano Rincón (IMT Atlantique)

Slides demo-gx-rx-count-domain.mp4 DNS-software-based-testbed-10GbE-OARC.pdf

15:30 HyperLogLog inspired three-minute scan of DNSSEC delegations worldwide

Using techniques inspired by the HyperLogLog counting algorithm, it has proven possible to rapidly measure the number of DNSSEC-signed delegations worldwide for both NSEC and NSEC3 zones, using around 4096 queries per zone. In this presentation, I will briefly describe HyperLogLog & then how this maps to NSEC names and NSEC3 hashes. I will also discuss how reliable results from measuring DNSSEC penetration this way can be in theory, and outline how theory matches reality. Finally, I'll run a fresh scan of the internet during the last few slides so we can see the state of DNSSEC penetration "right now", and offer some insights on particularly interesting zones. Compared to the existing 'hypernsec3' article posted earlier, this presentation adds discussion of the precision and accuracy of this technique, plus I will present fresh numbers. References: https://ds9a.nl/hypernsec3 and https://powerdns.org/dnssec-stats/

Speaker: Mr Bert Hubert (PowerDNS)

Slides hyperloglog_(1).pdf

16:00 → 16:30 Afternoon Coffee Break

16:30 → 18:15 Public Workshop: Testing

16:30 DNS Violations

The DNS Violations effort has been kicked off few days ago. In this presentation, I am going to cover the most common types of DNS protocol violations with real world examples, recommendations, workarounds. I would also like to initiate a discussion about possible way how to move forward and whether we need and want to take a stand.

Speaker: Mr Ondrej Sury (CZ.NIC)

Slides The_DNS_Violations_Project.pdf

17:00 ISC's Performance Lab System

I will describe (and hopefully demonstrate) ISC's Performance Lab System, which we intend to release as Open Source by the time of the workshop. This system performs continuous builds and tests of multiple configurations of BIND9 and other DNS software for the purpose of tracking long term trends, identifying performance regressions, and for testing the effects on performance of experimental features. We'll present some of the results we've obtained and describe what changes have been made to BIND9 as a result of those findings. We'll also discuss some of the issues we've encountered with generating consistent results (some of which are still to be resolved).

Speaker: Mr Ray Bellis (Internet Systems Consortium, Inc.)

Slides bellis-oarc-perflab.pdf

17:30 Analysis of DNS Server Software Provisioning Performance

While metrics comparing the query performance of various DNS software are readily available, similar metrics comparing provisioning performance are not as easily found. Over the last year CIRA’s secondary managed DNS platform, D-Zone, experienced large growth with the potential for even more expansion in the next year. Accordingly, we are required to reexamine our current DNS implementation to ensure that our provisioning process will continue to perform at an increased scale. However, there is a scarcity of available data focused on DNS software provisioning performance, which led us to investigate ourselves. As part of our assessment, we have recently performed testing to examine the provisioning performance of ISC Bind which compared the efficiency of manually provisioning zones via config files to dynamically provisioning zones using rndc commands (addzone, delzone, modzone). We also plan to run similar tests using other name server software - Knot and NSD. This presentation will feature an overview of the testing methodology, a summary of the findings, and proposals for future work.

Speaker: Evan Thompson (CIRA)

Slides oarc-prov-performance.pdf oarc-prov-performance.pptx

18:00 Testing DNS resolvers (as) in real world

All DNS resolver vendors face the same question: Is the new version going to upset users? This is a very hard question to answer because DNS resolvers have many use-cases and have to deal with variations in DNS protocol implementation. Opinions on best practices in software testing vary... but from the functional perspective the most important criteria is if users are able to resolve names all the names they want. The phone will be ringing if any of Alexa top 1 million web sites does not resolve correctly. In this talk we will explore how the Knot Resolver team is going to automatically generate tests for most visited sites in the world and what are the possible paths forward. Thanks to the fact that the test tool called Deckard [1] works with multiple DNS resolvers it is possible to re-use the same tests even for other projects.

Speaker: Mr Petr Špaček (CZ.NIC)

Slides presentation_en.odp presentation_en.pdf

19:00 → 21:30 Social Event

https://oarc26-social.eventbrite.com/

Monday, 15 May

08:30 → 09:00 Registration

09:00 → 10:45 Public Workshop: DNSSEC

09:00 Root zone KSK rollover update

ICANN would like to provide another update on the progress of the root zone KSK rollover. Since the rollover is scheduled for October 11, 2017, this DNS-OARC workshop could be the last one before the rollover takes place in the fall, so we would appreciate one more chance to reach the important segment of the DNS operational community that attends DNS-OARC workshops. Recent developments to share include the new KSK having been generated and the availability starting on March 1, 2017, of a RFC 5011 test bed operating in real time designed for operators to test the ability of their infrastructure to properly roll a trust anchor using the automated protocol. The presentation would be similar to one we delivered at NANOG69 in Washington, D.C., in early February. Those slides are attached for reference, but we would shorten and tighten the material to be more suitable for the more DNS-clueful audience at the workshop. I'm submitting this proposal for the main track, but since our main goal is to remind everyone of the rollover and keep it top of mind, we would also be willing to give a lightning talk. Thanks! Matt

Speaker: Matt Larson (ICANN)

Slides Root_KSK_rollover_DNS-OARC-Larson.pdf

09:15 Who's Asking?

The forthcoming roll of the Root Zone KSK has prompted some studies of the behaviour of resolvers that ask questions of the root. Some of these studies use direct experimentation, where a large number of end users are given a DNS name to resolve in order to understand the behaviour of the DNS recursive resolvers that they use. The DNS responses they are given are intended to mimic the behaviour of the Root Zone servers during the KSK roll, such as replicating the response size during certain phases of the KSK roll, or mimicking the particular truncation and fragmentation behaviours of individual root server behaviours. As we can't experiment with the root zone the common approach is to manipulate a test zone and test that. The assumption here is that the set of resolvers that ask an authoritative name server is the same as the set of name servers that ask the root. But are these two sets of visible resolvers the same? What can we assume from queries presentation to an authoritative name server that would relate to queries to the root servers? We present the results of a study that compares these two sets of resolvers and report on the degree of correlation.

Speaker: Mr Geoff Huston (APNIC)

Slides 2017-05-14-whos-asking.pdf

09:45 NSEC5: Updated specification and implementation results

NSEC5 is a proposed enhancement to DNSSEC that provably prevents zone enumeration. It does this by replacing the hashes used in NSEC3 with hashes computed by a verifiable random function (VRF), and requiring authoritative servers to perform a small amount of online cryptography for negative responses. This talk will give an overview of the latest NSEC5 protocol specification, and describe the results of implementing it and evaluating its performance.

Speakers: David Lawrence (Akamai Technologies), Jan Včelák (NS1), Shumon Huque (Salesforce)

Slides nsec5.pdf nsec5.pptx

10:15 What DNS Admins Should Know About Post-Quantum Cryptography

There is a lot of talk about the need for post-quantum cryptography (PQC) due to the possibility that quantum computers will be able to break the current cryptography in coming decades. If it becomes possible to build massive quantum computers, all cryptographic protocols will probably move to using PQC algorithms. It is expected that PQC algorithms for signatures use keys and/or signatures that are many times larger than those for RSA 2048. This will make DNSSEC-signed responses so much larger that TCP will probably be used for nearly all DNSSEC using PQC. Signing times might also increase by an order of magnitude. This presentation covers the current state of quantum computing, guesses at timelines leading to the need for PQC, and an overview of the size requirement for the PQC algorithms that might be chosen.

Speaker: Paul Hoffman (ICANN)

Slides dns-oarc-27-pqc-v3.pdf

10:30 Faking EDNS Key Tag

EDNS Key Tag promises to provide much-needed information about the DNSSEC configuration of recursive resolvers. Sadly, this technology is not yet standardized or implemented. Luckily, we can fake it in a useful way. This presentation very briefly covers the EDNS Key Tag as well as a hack built to provide EDNS Key Tag functionality for systems that do not support it. Also, we learn that awk is amazingly useful and cool. Yes, awk.

Speaker: Shane Kerr (Oracle / Dyn)

Slides 2017-05-14-dns-oarc-edns-key-tag.pdf

10:45 → 11:15 Morning Coffee Break

11:15 → 12:45 Public Workshop: Data Analysis

11:15 How do we analyze over O(100B) DNS requests daily

Cloudflare operates multiple DNS services in over 100 data centers around the globe, which makes troubleshooting with unstructured logs or packet captures impractical due to its storage and computational costs. In the first part of this talk we’ll go over our current data analytics architecture and how we got there, after a few false starts. This will cover logging infrastructure, that securely transports structured logs from all data centers, to an OLAP system built with ClickHouse open source RDMS. ClickHouse enables us and our customers to explore the the dataset in real time to get operational insights. Due to many of the optimizations built into ClickHouse we are able to store the data for a long time allowing us to look events is perspective and at historical trends. In the second part of the talk we will demonstrate some of the capabilities of the system, both on the granular level: track down problems with a particular record; and high level experiment with the dataset to show how it’s possible to track resolver affinity.

Speaker: Mr Ólafur Guðmundsson (CloudFlare)

Slides 2017-05-OARC-CF-DNSAnalytics.key 2017-05-OARC-CF-DNSAnalytics.pdf

11:45 dnstap: Use Cases and Performance Analysis

This talk will provide updates on dnstap, including the latest code developments and operational use cases. It will detail the results of tests that compare performance characteristics of a dnstap enabled pDNS sensor versus those of a BPF pDNS sensor.

Speaker: Merike Kaeo (Farsight Security)

Slides dnstap-DNSOARC-2017.pdf

12:15 DNS Traffic Sampling in dnscap

Many DNS operators, particularly those of high volume authoritative servers (such as TLD operators) perform operational monitoring of incoming (and outgoing) DNS query load. Often, this entails capturing (and subsequently storing and analyzing) the query/response stream. With DNS query rates (and hence traffic) increasing year by year, operators face the challenge that capture, transport and storage of the full message stream exceeds their operational capacity, and operating the capture infrastructure is more resource intensive than operating the actual DNS service. Furthermore, operators of DNS outsourcing solutions (such as anycast operators) experience similar challenges, at magnitudes worse than operators of a single zone. An obvious solution to this is to limit traffic capture to a certain subset of the messages received and sent. For example, some operators perform full capture for a few minutes each hour only, but this limits the "visibility" of traffic to a very small subset of continues time. In statistics, the problem of choosing a certain subset out of a large set of observations is very frequent, and called "Sampling". The most commonly used methods for Sampling are "random", "systematic", and "stratified", each of them involving a different algorithm to identify the observations to be selected. Together with SIDN, nic.at runs a project with a student to investigate the properties of the invidual sampling methodologies with regards to DNS traffic. While research around this Master Thesis paper is still in progress, nic.at decided to also kick off a project to implement DNS sampling in practice in a widely available tool. This talk explores the reasoning, current status, and result of adding systematic sampling to "dnscap", a widely used DNS traffic capture utility. The talk also explores how damage to figures which are badly affected by sampling (such as the total number of unique clients and total number of unique QNAMES) can be offset by adding an aggregating data structure right into the capturing tool. This aggregating data structure is called "HyperLogLog", allows estimation of the cardinality of items in set, and does this in a memory constant way. No matter how many elements are inserted in the structure, the amount of memory required stays constant, eg. at about 12kB in the presented prototype implementation. The obvious practical application in DNS for a "HyperLogLog" structure is therefore to provide an estimation of the cardinality of client IP addresses and QNAMEs, without the need to store and analyze the full data. One of the most amazing aspects of the HyperLogLog structure is that given two servers A and B, each of the servers could create an individual HyperLogLog structure, and the “summary” of the cardinality of the union of queries from A and B can simply be calculated by creating the union of the HyperLogLog structures from both servers – a property that fits perfectly to the DNS structure of highly parallel installations in anycast networks. So, instead of transferring the whole list of unique names for aggregation, each nameserver would just need to dump their own HyperLogLog structure (few kB’s) to a central location, at which the final aggregation could take place very efficiently. A "proof-of-concept" level implementation of “HyperLogLog” in dnscap exists, and is presented during the talk.

Speaker: Mr Alexander Mayrhofer (nic.at GmbH)

Slides DNS-OARC26-Mayrhofer-DNS-Sampling-v4.pdf

12:45 → 14:15 Lunch

13:30 → 14:00 PGP Signing Session

Speaker: Matthew Pounsett (Rightside)

notes Keyring from OARC 26 PGP Key Party

14:15 → 16:15 Public Workshop: Security and Privacy

14:15 Zone Poisoning: The How and Where of Non-Secure DNS Dynamic Updates

Domain names are a critical resource for legitimate users, but also for criminals. This has led to a variety of attacks on the underlying technology, the Domain Name System (DNS) infrastructure. Registrars have been hacked, attackers have set up malicious domain name resolution services and DNS caches have been poisoned. What most attacks share in common is that they compromise the resolution path somewhere between the user and the authoritative name server for a domain. One attack vector has been overlooked so far, namely using poorly configured name servers to manipulate domain name records at the authoritative end of the path: the zone file for the domain. We call this attack ‘zone poisoning’. The attack is as simple as sending a single RFC compliant DNS dynamic update packet to a misconfigured server. In the simplest version of an attack, a miscreant could replace an existing A or MX resource record in a zone file of an authoritative server and point the domain name to an IP address under control of an attacker. We present the first measurement study of the vulnerability. To assess the potential impact of non-secure dynamic updates, we scanned a random sample of 2.9 million domains worldwide and the Alexa top 1 million domains. We find that among the vulnerable domains are governments, health care providers and banks, demonstrating that the threat impacts important services. We have also issued notifications for DNS service providers, website owners and network operators suffering from non-secure DNS dynamic updates to assess which mechanisms are more reachable and effective at remediating the vulnerability. We have systematically assessed the effectiveness of communication channels and notifications with demonstrative content where recipients of the notifications can identify an existence of the vulnerability from a given external link versus standard vulnerability notifications. We monitored name servers and domain names for several weeks to determine their rate of remediation. Via our study of the zone poisoning attack and subsequent notifications to affected parties, we aim to improve the security of the DNS ecosystem.

Speaker: Dr Maciej Korczynski (Deflt University of Technology)

Slides non-secure-dns.pdf

14:45 Survey of DNS abuse types from a TLD point of view

Please see paper at[1] ,and blogpost at [2] But in short, this is a concise survey paper on the forms of DNS abuse and their relation with TLD operators. We show how we can use the datasets we have in hand to detect these sorts of abuse, and how each of them have different business models that leave distinct traces on our datasets. IMHO, I think other TLD operators may benefit from that. Please keep in mind that this is a short paper (6 pages) to be presented at the DISSEC2017 workshop, co-hosted with IEEE IM 2017. [1] https://www.sidnlabs.nl/downloads/papers-reports/dissect2017.pdf [2] https://www.sidnlabs.nl/a/weblog/survey-of-dns-abuse-types?language_id=2

Speaker: Dr Giovane Moura (SIDN Labs)

Slides moura-oarc26.pdf

15:00 The Dark Side of the DNS

1. **Data exfiltration using the DNS** A. Multigrain malware, and other examples of the use of DNS for data exfiltration 1. Detecting subdomain-type data exfiltration through statistical analysis of subdomain lengths B. Use of DNS 0x20 / XQID / IDN as a covert channel 1. Cisco Talos stats on malware’s use of mixed-case, XQID, and other queries C. Passive DNS "dead drops" 2. **Malware C2 using the DNS** A. DNSMessenger malware and bi-directional command and control communication using DNS TXT records 3. **Bulletproof hosting** A. Malicious use of the .bit TLD (NameCoin) B. Malware using Tor2Web

Speaker: Mr Jaeson Schultz (Cisco Systems)

Slides TheDarkSideoftheDNS-OARC-May2017.pdf TheDarkSideoftheDNS-OARC-May2017.pptx

15:30 dnsprivacy.net - A project to support deployment of DNS-over-TLS services

The DPRIVE Working Group has recently produced several standards relating to DNS-over-TLS as a method for encrypting Stub to recursive communications. Whilst there are several implementations available, deployment is still in the early stages. Several experiment DNS-over-TLS servers have been running since 2016 and the dnsprivacy.net project is aiming to - Increase DNS-over-TLS deployment during 2017 - Develop tools to ease deployment and monitoring - Perform research into the practicalities of delivering DNS-over-TLS services - Raise awareness of DNS Privacy with operators and end users - Produce a Best Current Practices guide for operators. This talk will provide an update on the status of the dnsprivacy.net project and outline in detail the short and medium term goals with the aim of engaging operators in these activities.

Speaker: Dr Sara Dickinson (Sinodun IT)

Slides OARC26_dnsprivacy.net.pdf

15:45 The Importance of Being an Earnest stub

Many transactions that need to be trustworthy, and possibly encrypted, start with a DNS query. If we consider security from the ground-up, we need to include end users DNS transactions with resolvers in the security realm. The minimal step is DNSSEC where the received data can be verified and validated to be correct and authentic. But if we want to take security and privacy a step further, also the recursive resolver needs to be authenticated and communication with it encrypted. These requirements put higher demands, on the capabilities and also on the role and responsibilities of stub resolvers, and changes the relationship with and the requirements on the recursive resolvers they use. The first/last mile is changing. In this presentation we discuss the idea of security from the ground-up, focussing on the versatile stub resolver as designed in the getdns project. Current challenges and solutions to DNSSEC roadblocks are presented, and ideas/design of future developments are outlined.

Speaker: Mr Willem Toorop (NLnet Labs)

Slides an-earnest-stub.odp an-earnest-stub.pdf

16:15 → 16:35 Afternoon Coffee Break

16:35 → 17:35 Lightning Talks

16:35 DNS Not Over UDP/TCP, mid-2017 Update

Speaker: Paul Hoffman (ICANN)

Slides dns-over-foo-oarc.pdf

16:45 DNS-over-QUIC

Speaker: Dr Sara Dickinson (Sinodun IT)

Slides OARC26_DNS-over-QUIC.pdf

16:55 ECS Support Detection and Birthday Attack Hardening

Speaker: Ralph Dolmans (NLnet Labs)

Slides unbound_ecs_oarc26.pdf

17:05 ANAME: Address-specific DNS Name Redirection

Speaker: Mr Peter van Dijk (PowerDNS)

Slides pvdijk-aname-lightning.pdf

17:15 dnsdist: two years later

Speaker: Pieter Lexis (PowerDNS)

Slides PLexis-dnsdist-2yrs-later.pdf