dnstap: high speed DNS server event replication without packet capture

Robert Edmonds ()

13 June 2013

Slides and code

http://dnstap.info/

Introduction

The DNS protocol is really complicated.

Extracting event information like client queries and cache updates from a running DNS server is hard.

Let’s build something flexible that can replace or supplement existing event monitoring technology.

Contents

  1. DNS overview
  2. Existing monitoring technologies
    1. Query logging
    2. Passive DNS replication
  3. New technology: dnstap
  4. Getting the code
  5. Future plans

DNS overview

Query logging

Query logging

Log information about queries:

Usually syslog based.

Core idea: the query log is generated internally by the DNS server in the normal course of request processing.

Query logging: BIND4

#ifdef QRYLOG
     if (qrylog) {
             syslog(LOG_INFO, "XX /%s/%s/%s",
                    inet_ntoa(from->sin_addr),
                    (dname[0] == '\0') ?"." :dname,
                    p_type(type));
     }
#endif /*QRYLOG*/

– BIND 4.9.2 named/ns_req.c, circa 1994

Query logging: Unbound

log-queries: <yes or no>
    Prints one line per query to the log, with the log timestamp
    and IP address, name, type and class.  Default is no.
    Note that it takes time to print these lines which makes the
    server (significantly) slower. Odd (nonprintable) characters
    in names are printed as '?'.

– Unbound config file

    if(worker->env.cfg->log_queries) {
        char ip[128];
        addr_to_str(&repinfo->addr, repinfo->addrlen, ip, sizeof(ip));
        log_nametypeclass(0, ip, qinfo.qname, qinfo.qtype, qinfo.qclass);
    }

– Unbound daemon/worker.c

Query logging: BIND9

void
ns_client_logv(ns_client_t *client, isc_logcategory_t *category,
           isc_logmodule_t *module, int level, const char *fmt, va_list ap)
{
    char msgbuf[4096];
    char peerbuf[ISC_SOCKADDR_FORMATSIZE];
    char signerbuf[DNS_NAME_FORMATSIZE], qnamebuf[DNS_NAME_FORMATSIZE];
    const char *viewname = "";
    const char *sep1 = "", *sep2 = "", *sep3 = "", *sep4 = "";
    const char *signer = "", *qname = "";
    dns_name_t *q = NULL;

    vsnprintf(msgbuf, sizeof(msgbuf), fmt, ap);

    ns_client_name(client, peerbuf, sizeof(peerbuf));

    if (client->signer != NULL) {
        dns_name_format(client->signer, signerbuf, sizeof(signerbuf));
        sep1 = "/key ";
        signer = signerbuf;
    }

    q = client->query.origqname != NULL
        ? client->query.origqname : client->query.qname;
    if (q != NULL) {
        dns_name_format(q, qnamebuf, sizeof(qnamebuf));
        sep2 = " (";
        sep3 = ")";
        qname = qnamebuf;
    }

    if (client->view != NULL && strcmp(client->view->name, "_bind") != 0 &&
        strcmp(client->view->name, "_default") != 0) {
        sep4 = ": view ";
        viewname = client->view->name;
    }

    isc_log_write(ns_g_lctx, category, module, level,
              "client %s%s%s%s%s%s%s%s: %s",
              peerbuf, sep1, signer, sep2, qname, sep3,
              sep4, viewname, msgbuf);
}

– BIND9 bin/named/query.c

Dealing with DNS text logs

Jun 09 20:24:41 unbound[30101:0] info: start of service (unbound 1.4.21).
Jun 09 20:24:43 unbound[30101:0] info: 127.0.0.1 www.example.com. A IN
Jun 09 20:24:50 unbound[30101:0] info: 127.0.0.1 www.isc.org. A IN
Jun 09 20:24:54 unbound[30101:0] info: 127.0.0.1 www.example.com. AAAA IN
Jun 09 20:24:57 unbound[30101:0] info: 127.0.0.1 www.isc.org. AAAA IN
Jun 10 02:48:43 zappa named[6666]: client 70.89.251.89#37412 (jj.mycre.ws):
    query: jj.mycre.ws IN TXT -E (149.20.54.65)
Jun 10 02:48:44 zappa named[6666]: client 70.89.251.89#58265 (jj.mycre.ws):
    query: jj.mycre.ws IN TXT -ET (149.20.54.65)

Protocol elements have been converted to text format.

How to parse? (Regular expressions? Now you have two problems…)

Different DNS servers use their own formats.

Google Public DNS

Passive DNS replication

Passive DNS replication

Log information about zone content:

Packet capture based.

Core idea: pull records out of binary DNS response messages.

Passive DNS replication: BFK

Passive DNS replication: ISC

Dealing with DNS packet captures

14:48:26.002226 IP 69.94.222.154.32975 > 198.41.0.4.53: 64443% [1au] NS? . (28)
14:48:26.026313 IP 198.41.0.4.53 > 69.94.222.154.32975: 64443*- 13/0/20 NS
K.ROOT-SERVERS.NET., NS L.ROOT-SERVERS.NET., NS M.ROOT-SERVERS.NET., NS
A.ROOT-SERVERS.NET., NS B.ROOT-SERVERS.NET., NS C.ROOT-SERVERS.NET., NS
D.ROOT-SERVERS.NET., NS E.ROOT-SERVERS.NET., NS F.ROOT-SERVERS.NET., NS
G.ROOT-SERVERS.NET., NS H.ROOT-SERVERS.NET., NS I.ROOT-SERVERS.NET., NS
J.ROOT-SERVERS.NET. (615)

DNS messages are encoded into packets.

Query messages fit into one packet.

Response messages sometimes require multiple packets.

Dealing with DNS packet captures

Dealing with DNS packet captures

Need to:

This is hard. The networking stack and DNS server are already doing these things. Why bother?

Introducing dnstap

  1. Query logging does not imply text format logging.

  2. Passive DNS replication does not imply packet capture.

Introducing dnstap

Getting the code

Website:

http://dnstap.info

Open source code repositories are located on GitHub:

https://github.com/dnstap

Getting the code: dnstap.pb

This is the Protocol Buffers schema defining the layout of dnstap payloads.

Use this if you want to process dnstap payloads in your programming language of choice.

$ git clone git://github.com/dnstap/dnstap.pb.git

Here’s a direct link to the schema.

You don’t need this unless you’re writing code that needs to decode dnstap payloads.

dnstap.pb schema (1/2)

message Dnstap {
    optional bytes      identity;
    optional bytes      version;
    enum Type {
        MESSAGE;
    }
    required Type       type;
    optional Message    message;
}

enum SocketFamily {
    INET;
    INET6;
}

enum SocketProtocol {
    UDP;
    TCP;
}

dnstap.pb schema (2/2)

message Message {
    enum Type {
        AUTH_QUERY;
        AUTH_RESPONSE;
        RESOLVER_QUERY;
        RESOLVER_RESPONSE;
        CLIENT_QUERY;
        CLIENT_RESPONSE;
        FORWARDER_QUERY;
        FORWARDER_RESPONSE;
    }
    required Type               type;
    optional SocketFamily       socket_family;
    optional SocketProtocol     socket_protocol;
    optional bytes              query_address;
    optional bytes              response_address;
    optional uint32             query_port;
    optional uint32             response_port;
    optional uint32             message_id;
    optional bytes              query_name;
    optional uint32             query_type;
    optional uint32             query_class;
    optional uint64             query_time_sec;
    optional fixed32            query_time_nsec;
    optional bytes              query_message;
    optional bytes              query_zone;
    optional uint64             response_time_sec;
    optional fixed32            response_time_nsec;
    optional bytes              response_message;
}

Getting the code: dnstap

This is the C library for adding dnstap support to your DNS server.

# apt-get install protobuf-c-compiler
# git clone git://github.com/dnstap/dnstap.git
# cd dnstap && ./autogen.sh && ./configure && make && make install

You need this if you’re a DNS programmer adding dnstap support to your nameserver, or if you’re a sysadmin compiling a nameserver with dnstap support.

Getting the code: unbound/dnstap

This is NLNet Labs’ Unbound DNS server, patched to generate dnstap payloads.

$ git clone git@github.com:dnstap/unbound.git
$ cd unbound && ./configure --enable-dnstap && make && make install

Add to unbound.conf:

server:
    dnstap-enable: yes
    dnstap-socket-path: "/var/run/unbound/dnstap.sock"
    dnstap-send-identity: yes
    dnstap-send-version: yes
    dnstap-log-resolver-response-messages: yes
    dnstap-log-client-query-messages: yes

Start Unbound and it will begin generating dnstap/Message payloads on the dnstap-socket-path.

You also need a capture tool listening on the other end of the dnstap socket.

Getting the code: golang-dnstap

This is the Go language library for working with dnstap sockets, log files, and message payloads. It also includes a command-line capture tool.

# apt-get install golang
$ go get -u github.com/dnstap/golang-dnstap/dnstap

Running dnstap

Example output from the dnstap command-line tool with a dnstap-enabled Unbound recursive DNS server connected.

$ dnstap -s /tmp/dnstap.sock
dnstap: opened input socket: /tmp/dnstap.sock
23:28:56.955950 CQ 127.0.0.1 UDP "www.google.com." IN A
23:28:56.968771 RR 192.58.128.30 UDP "." IN NS
23:28:57.020234 RR 2001:500:2d::d UDP "g.root-servers.net." IN AAAA
23:28:57.025925 RR 2001:500:1::803f:235 UDP "b.root-servers.net." IN AAAA
23:28:57.043608 RR 2001:7fd::1 UDP "c.root-servers.net." IN AAAA
23:28:57.096600 RR 202.12.27.33 UDP "www.google.com." IN A
23:28:57.099600 RR 202.12.27.33 UDP "e.root-servers.net." IN AAAA
23:28:57.115804 RR 192.203.230.10 UDP "i.gtld-servers.net." IN AAAA
23:28:57.117032 RR 192.58.128.30 UDP "c.gtld-servers.net." IN AAAA
23:28:57.149958 RR 192.33.14.30 UDP "j.gtld-servers.net." IN AAAA
23:28:57.164234 RR 2001:503:a83e::2:30 UDP "c.gtld-servers.net." IN AAAA
23:28:57.180134 RR 2001:503:83eb::2:31 UDP "j.gtld-servers.net." IN AAAA
23:28:57.184770 RR 192.35.51.30 UDP "www.google.com." IN A
23:28:57.212179 RR 216.239.38.10 UDP "www.google.com." IN A
23:28:57.219282 RR 216.239.36.10 UDP "ns2.google.com." IN AAAA
23:28:57.220253 RR 216.239.38.10 UDP "ns3.google.com." IN AAAA
23:28:57.222184 RR 192.31.80.30 UDP "i.gtld-servers.net." IN AAAA
23:28:57.223815 RR 192.26.92.31 UDP "e2.gtld-servers.net." IN AAAA
23:28:57.225247 RR 192.35.51.30 UDP "e.gtld-servers.net." IN AAAA
23:28:57.226564 RR 192.36.148.17 UDP "k.gtld-servers.net." IN AAAA
23:28:57.226806 RR 192.35.51.30 UDP "g.gtld-servers.net." IN AAAA
23:28:57.227977 RR 216.239.34.10 UDP "ns1.google.com." IN AAAA
23:28:57.229058 RR 216.239.34.10 UDP "ns4.google.com." IN AAAA
23:28:57.244416 RR 192.43.172.30 UDP "l.gtld-servers.net." IN AAAA
23:28:57.253047 RR 2001:503:a83e::2:30 UDP "k.gtld-servers.net." IN AAAA
23:28:57.254144 RR 2001:503:a83e::2:31 UDP "i.gtld-servers.net." IN AAAA
23:28:57.258091 RR 192.42.93.31 UDP "g2.gtld-servers.net." IN AAAA
23:28:57.259196 RR 192.35.51.31 UDP "d2.gtld-servers.net." IN AAAA
23:28:57.260140 RR 192.42.93.31 UDP "l2.gtld-servers.net." IN AAAA
23:28:57.261115 RR 192.31.80.31 UDP "c.gtld-servers.net." IN AAAA
23:28:57.262082 RR 2001:503:a83e::2:31 UDP "h2.gtld-servers.net." IN AAAA
23:28:57.311686 RR 192.48.79.30 UDP "h.gtld-servers.net." IN AAAA
23:28:57.313437 RR 192.26.92.31 UDP "k.gtld-servers.net." IN AAAA
23:28:57.315933 RR 192.35.51.31 UDP "e.gtld-servers.net." IN AAAA
23:28:57.317539 RR 192.31.80.31 UDP "g.gtld-servers.net." IN AAAA
23:28:57.336650 RR 192.35.51.31 UDP "l.gtld-servers.net." IN AAAA
23:28:57.339383 RR 2001:503:83eb::2:31 UDP "h.gtld-servers.net." IN AAAA
23:28:57.342427 RR 192.35.51.31 UDP "f2.gtld-servers.net." IN AAAA

Future plans

Add support for more DNS servers.

Rigorous testing.

Documentation.

Benchmarking.

Write a specification.

Develop more tools that can consume dnstap payload data.

Future plans

Define more event types like:

Add more features to the dnstap command-line tool like:

Summary

Examined existing syslog and pcap-based DNS monitoring technologies.

Introduced new dnstap technology that combines aspects of both of these traditional approaches.

Demonstrated how to download and use the released code.