Knowledge Base ISC Main Website Ask a Question/Contact ISC
Tuning your BIND configuration effectively for zone transfers (particularly with many frequently-updated zones)
Author: Cathy Almond Reference Number: AA-00726 Views: 25689 Created: 2012-07-25 15:15 Last Updated: 2015-10-30 12:28 0 Rating/ Voters

Zone transfers - AXFR and IXFR

When a master nameserver is updated (irrespective of the mechanism through which this happens), the working contents of the zone held in memory that have changed need to be transferred to the other servers that are authoritative for that zone (the slave servers).  This process is called 'zone transfer'.  

There are two types of zone transfer - full (AXFR) and incremental (IXFR).  A full zone transfer is of the entire zones.  An Incremental zone transfer is a list of changes that, when applied, bring the slave zone up to date.  The use of incremental zone transfers requires that both the provider and the recipient are maintaining a separate journal file of changes for each zone.  The default is that slaves will request IXFR (unless otherwise configured) but they will always handle being given an AXFR instead, if no IXFR is available.

The critical component that controls the decision on whether or not the zone has changed is the serial number that is maintained in the zone's SOA record (although when a master reloads, it does also check the timestamp on the zone data file first, before checking the zone's SOA).   BIND assumes that if the serial number remains the same, then the zone is unchanged.

Zone transfers are always initiated by the slave server (that is, they are 'requested' or 'pulled').  Whilst there are mechanisms in place to alert slaves that they should check to see if a zone transfer is needed, the transfer itself is always started by the slave.

Zone notifies and refreshes

Apart from when a server is restarted, or when rndc commands are used to force a zone transfer, there are two mechanisms that cause slave servers to check whether or not their copy of a zone is current.  The process of checking and updating is called a zone refresh:

  • Refresh timer expiry:
    Each zone's SOA record holds a refresh timer that all slave servers receiving a copy of the zone should use.  The refresh timer tells a slave server how often it should ask one of the master servers to which it has been configured to refer for an SOA record.  This is then compared to the SOA the slave is holding - if it is the same, there is nothing to be done (and the slave waits until the next refresh interval expiry).  There is also a retry timer which is applied if the refresh attempt fails (none of the master servers could be contacted).  And finally, there's an expire timer - if the slave has been unable to contact another master for this period, then it stops serving data from that zone.  (Note that restarting the slave will reset any 'expired' zones, so if they have a copy of the zone backed up to file, they will then resume serving the expired zone.)
  • Notify received:
    When an authoritative (master or slave) server updates a zone, it can send out notifications to other servers that it has changed.  This causes the recipients to set their 'next refresh' time to 'now' and to queue a zone refresh.
Zone notifies and refreshes when restarting the named daemon

When a server is restarted, it sends out notifications for the zones that it has loaded (in case they have been updated while it was not running).  Similarly, when loading zones that are slaved, a newly-restarted named will initiate a refresh for each zone (using jittered time intervals so that all refreshes are not initiated concurrently).

Notification and refresh options unrelated to tuning/performance

There are a number of options that can be used to control (both on masters and slaves) which servers are notified when a zone is updated and which master servers should be queried by slaves when they are refreshing as well as handling the security implications of both via ACLs and TSIG.  This article assumes that the reader is familiar with all of these and that the zone transfer mechanism is both secure and functioning as expected, but needs some attention to tuning.

Potential problems

Where servers handle large numbers of zone transfers - either because they have zones that are highly dynamic (many and frequent changes) or are simply serving a very large number of zones, then some consideration needs to be given to tuning to avoid:

  • Delays in propagation of zone updates (some authoritative servers providing old data for several minutes or hours after the zone has been updated)
  • Refresh or transfer failures
  • Servers non-responsive to clients (too busy handling zone transfers or network traffic peaks are causing dropped/lost/failed queries).

The last problem on the list is the one that historically has led to tuning options that 'throttle' zone transfers, but higher-speed hardware and networks developed in recent years mean that most servers should be able to handle a significant rate of zone maintenance activity in parallel with servicing client queries.

Configuration options for tuning zone transfers that are set too low may therefore cause other issues for zone administrators.  Some commonly-reported log messages are detailed below, along with the explanation of what is happening to cause them to be logged - but note that this is not a complete list!  Some messages are logged from category general, and others from categories xfer-in and xfer-out.  On the server providing the zone transfers you might also see some relevant messages logged from category client.

Commonly-reported Logfile Messages

16-Nov-2011 16:31:07.044 xfer-in: error: transfer of 'testzone.example.com/IN' from 192.0.2.1#53: failed to connect: timed out

This means that a zone transfer was started but failed.  Sometimes this is due to the TCP connection between the master and slave being reset but it can also be due to the master being inaccessible from the slave due to routing or firewall issues.  The outcome is doubly bad:

  • The zone will not be refreshed again until the retry timer period or another notification is received
  • The master server is temporarily cached as 'unreachable' and won't be retried for future transfers until at least 10 minutes has passed since it was last discovered to be unreachable or until a notify is received from it.
    Retrying of a cached 'unreachable' master server upon receipt of a notify was implemented from versions 9.6-ESV-R6, 9.7.5, 9.8.2 and 9.9.0 onwards:

    Master servers that had previously been marked as unreachable because of failed zone transfer attempts will now be removed from the "unreachable" list (i.e. considered reachable again) if the slave receives a NOTIFY message from them. [RT #25960]

    See also: What does named log message "deleted from unreachable cache" mean?

Note also that when attempting a zone refresh there is a fallback from UDP to TCP on the SOA query.  If there is no response to the UDP SOA queries from any server, the final attempt is to retrieve the SOA via TCP.  It's TCP connection failures that cause the server to be added to the unreachable cache - either the last attempt at an SOA query, or, if the SOA query was successful via UDP,  the first attempt at a zone transfer.

17-Nov-2011 20:46:51.807 general: info: zone testzone.com/IN: refresh: skipping zone transfer as master 192.0.2.1#53 (source 192.0.2.4#0) is unreachable (cached)

Here we have an indication that a zone refresh attempted earlier has failed and that this information was cached.  There are several similar error messages, but when the reason for the zone refresh not taking place is that the master's IP address is already in the unreachable cache, then the text 'unreachable (cached)' is always included.

17-Nov-2011 21:50:14.762 xfer-in: info: transfer of 'testzone.com/IN' from 192.0.2.1#53: connected using 192.0.2.4#47296
17-Nov-2011 21:50:14.762 xfer-in: error: transfer of 'testzone.com/IN' from 192.0.2.1#53: failed while receiving responses: connection reset
17-Nov-2011 21:50:14.762 xfer-in: info: transfer of 'testzone.com/IN' from 192.0.2.1#53: Transfer completed: 0 messages, 0 records, 0 bytes, 0.001 secs (0 bytes/sec)

Above we can see that a zone transfer was started, but failed due to the TCP connection to the master server being reset.  Typically this will be due to the tcp-listen-queue not being increased in line with other tuning options.

  • Note that this error has been logged from category xfer-in - this is an example of more detailed information relating to an error logged in the general logging category (as seen above).
  • Since the zone transfer operation was started, it also logs at completion, but you will observe that the statistics indicate that no records were transferred.
  • There are some variations that may be seen for a connection reset, depending at which point in the life of the TCP connection that the reset was received.  Some variations are:
    17-Nov-2011 21:50:15.712 xfer-in: error: transfer of 'testzone.com/IN' from 192.0.2.1#53: failed sending request length prefix: connection reset

    17-Nov-2011 21:50:28.394 xfer-in: error: transfer of 'testzone.com/IN' from 192.0.2.1#53: failed sending request data: connection reset

    The significant portion of the message from each of the above is 'connection reset'.

13-Jan-2012 18:45:51.036 client: warning: client 192.0.2.4#42229: no more TCP clients: quota reached.

This is another problem relating to TCP tuning - in this case a warning logged on the master server which should be providing the zone update.  There is a tunable limit (tcp-clients) on the number of TCP 'clients' that can be handled concurrently - this is shared between client queries that are received over TCP and zone transfers.  This limit has been reached - which can be another reason for a zone transfer to see the a connection reset error logged from category xfer-in

The logged error is the same, irrespective of whether it is a TCP connection for a zone transfer, or a client query that has been dropped/reset.

The error is generated because the limit tcp-clients has been reached.  Named is unable to discriminate between types of tcp client connection but it should be clear from the matching log entries on the slave server that this failed connection was for a zone transfer. 


13-Jan-2012 18:45:01.871 general: info: zone testzone.com/IN: notify from 192.0.2.1#62160: refresh in progress, refresh check queued

A notify was received, but the zone being notified was already in the process of being refreshed or is waiting to be refreshed, so the check is queued and will be processed later.

14-Jan-2012 10:34:06.054 general: info: zone testzone.com/IN: notify from 192.0.2.1#20016: zone is up to date

A notify was received but upon checking, the slave zone SOA serial number is already the same as on the master it checked, therefore no zone transfer is required.

13-Jan-2012 18:45:38.702 xfer-in: info: zone testzone.com/IN: zone transfer deferred due to quota

This error is logged on a slave server and it indicates that it has reached one of its own configured limits on concurrent inbound transfers (transfers-in or transfers-per-ns set globally, or for a specific server statement, the transfers option) - which is not necessarily a bad thing depending on your bandwidth and chosen tuning options.  The transfer is deferred (queued to be retried later).

02-Aug-2012 07:58:20.601 general: info: master 192.0.2.4#53 (source 192.0.2.8#0) deleted from unreachable cache

BIND maintains a cache of unreachable masters to which it refers when handling a zone refresh.  If a zone refresh fails with a specific master (either during the query for the SOA or after querying and while attempting a subsequent zone transfer), then this master is cached as 'unreachable' for 10 minutes.  As of versions 9.6-ESV-R6, 9.7.5, 9.8.2 and 9.9.0 onwards BIND implements an earlier removal of a master server from the unreachable cache if a notify is received from it.   (Note that receipt of a notify (which is a UDP packet travelling from master to slave) doesn't guarantee that the master will be reachable from the slave, but it does ensure quicker recovery in the situation where a master was temporarily unavailable, for example for a reboot.)

Options for tuning Masters

When tuning the behaviour of the master, there are several factors that you can control:

  • The rate of notifications of changes to slave servers (serial-query-rate and notify-delay)
  • Limits on concurrent zone transfers (transfers-out, tcp-clients, tcp-listen-queue, reserved-sockets)
  • Efficiency/management options (max-transfer-time-out, max-transfer-idle-out, transfer-format)

The most important options to focus on are transfers-out, serial-query-rate, tcp-clients and tcp-listen-queue.

transfers-out is the maximum number of concurrent zone transfers outbound that will be permitted - the default is 2.  If you make this value too large on a master server with many zones that are frequently updated, you may find that your server is too busy handling zone transfers to handle queries effectively.  It will also need to be able to handle the volume of concurrent TCP client connections (see tcp-clients) for zone transfers.

tcp-clients controls the maximum number of inbound TCP connections to this server.  Inbound TCP connections includes both zone transfers and TCP-based queries.  The default is 100.  You can monitor the count of tcp clients using rndc status (provides a one-time snapshot of the counters - you can repeat the command, collect the statistics and look at trends).

tcp-listen-queue is a subtle control setting (although not applicable to all OS environments).  When there is a high rate of inbound TCP connections, it controls how many connections can be queued before they are accepted by the application.  If named has already reached both the limit on concurrent zone transfers, and the limit specified by tcp-listen-queue, then any new inbound TCP connections will be dropped.  If you're expecting a high rate of zone transfers or that zone transfer requests will be competing for master server resources, then you should increase this configuration option, whose default is 3 (increased to 10 from BIND 9.10, 9.9.4, 9.8.6 and 9.6-ESV-R10).  

tcp-listen-queue does not directly control the number of pending tcp connections

A value of 10 does not imply a strict limit of 10 queued TCP connections - the impact of changing this configuration setting will be OS-dependent.  Larger values for tcp-listen queue will permit more pending tcp connections, which may be needed where there is a high rate of TCP-based traffic.  Since TCP performance is tightly bound to the kernel new connection-handling code, increasing this value will also have no impact if the kernel itself cannot handle the rate of inbound connections.   For most production server environments the new default value of 10 should be adequate.  A value of 0 may also be used; on most platforms this sets the listen queue length to a system-defined default value (SOMAXCONN).   Please see the man page for listen(2) for more details on how TCP connection backlog is handled).

serial-query-rate (default 20) is a rate-limiter, that is used to control (depending on which version of BIND is running) both the rate of notifies and of zone refresh (SOA queries).  Although the limit is expressed as a per-second rate, it is the actions that are being limited, not the network packets needed to complete an action.  So for example, in the case of a zone refresh, there may multiple queries sent as named retries if no response is received, or tries different authoritative servers.  You are therefore recommended to increase the setting gradually to a level that meets your production environment's requirements.  The effects of this setting are usually most apparently immediately following a restart of server that is authoritative for a large number of zones.

Note that older versions of BIND managed notifies and SOA refresh queries in a single queue (which sometimes caused problems for slaves that are also masters for other servers). To ensure that notifies and refreshes were not competing with each other, in BIND versions 9.6-ESV-R11, 9.8.7, 9.9.5 and 9.10.0 we introduced independent queues - both still with serial-query-rate controlling them.

From BIND 9.9.7-S1 (and this change will also be found in BIND 9.11.1) there are three separate rate-limiting controls: serial-query-rate; notify-rate and startup-notify-rate.

For more information on rate-limiting notifications and SOA refresh queries, please read: serial-query-rate, notify-rate and startup-notify-rate: how they impact zone transfers in different versions of BIND

Setting notification rates too high may cause operational problems

When you send a notify, you are effectively inviting your slave servers to send you an SOA query for the zone, and perhaps as a result to initiate a zone transfer.  If the notification rate is significantly increased, then when rebooting a master server, you will see a large amount of traffic as notifications are sent for all zones to all slave servers (or as configured for notification destinations), followed by SOA queries being received in response.

notify-delay goes hand-in-hand with overall rate-limiting but is used to configure what happens for notifications for a single zone.  The default is 5 seconds - which means that even if the zone is being updated constantly, there will not be a storm of notifications being sent to slave servers.  Instead, the notifies are deferred until 5 seconds has elapsed.   It would be unusual to change this configuration setting, but it might be appropriate in some situations.

Logging of notifications occurs as they are queued for sending, not when they are transmitted

When named logs notifies, it indicates that they have been queued to be sent - but any rate limiting has not yet been applied.  Where there has been a server restart, or where there is a high rate of zone updates, the notifies that are logged may not be sent immediately.  You will be able to see what is really happening from a network trace or in the logs of notifies received on the slave servers.

reserved-sockets controls how many file descriptions (of the maximum number of open files that named can have) are reserved for TCP and for standard files (opened for logging, and management of zone data on disk etcetera).  The default is 512.  It would be very unusual to change this setting.  In order to ensure that named isn't completely broken by inappropriate settings, the minimum value is 128 and the maximum value is 128 less than maxsockets.  Setting this too large will limit the number of UDP sockets available and may cause recursive resolution of client queries to fail.

max-transfer-time-out is used to terminate outbound zone transfers that have not completed by a set time limit.  The default is 2 hours, which should be adequate for most scenarios.

max-transfer-idle-out is used to terminate outbound zone transfers that are no longer making any progress.  The default is 1 hour.

If your network connectivity is unreliable and you are observing high numbers of 'stalled' zone transfers, then consider reducing both of the above max-transfer-*-out settings.  You might also want to tune some of your kernel parameters for TCP too.

transfer-format is used either globally or per server (using the server statement) to control whether zone transfers send one record (one-answer) or many records (many-answers) per DNS message.  Clearly many-answers is more efficient, but at the time that many-answers format was introduced, adoption was not universal and some older servers could not handle this type of zone transfer.  Nowadays it would be unusual to find any servers that don't support many-answers format, but the option may have persisted from old configuration files.  If zone transfers are slow, it would be worth checking that your master server is not still using one-answer format.

Options for tuning Slaves

Note that slave servers may also serve zone transfers

When you are designing your zone data propagation strategy, you may opt to have slave servers notify and serve updates to other slave servers.  In this case, don't forget to calculate and tune for both sets of activities.

 When tuning the behaviour of a slave server, you can control:

  • The rate of inbound zone transfers on a per server basis, as well as in aggregate (transfers-in, transfers-per-ns, server option 'transfers')
  • The rate at which refresh (SOA) queries are sent (serial-query-rate, min-refresh-time, max-refresh-time, min-retry-time, max-retry-time)
  • Efficiency/management options: (try-tcp-refresh, max-transfer-time-in, max-transfer-idle-in)

The most important options on which to focus are transfers-in and transfers-per-ns.

transfers-in is the maximum number of concurrent zone transfers inbound that will be permitted - the default is 10.  If you make this value too large on a slave  server with many zones that are frequently updated, you may find that your server is too busy handling zone transfers to handle queries effectively.  Depending also on your slave zone configuration and zone data propagation strategy, each inbound zone transfer completion may cause onward notifications to other servers along with inbound SOA queries that also increase the workload.

transfers-per-ns is a per-server limit on concurrent zone transfers inbound.  It's a 'good behaviour' configuration setting, so that even though the overall maximum number of inbound zone transfers is larger, named is preventing itself from hitting any one master server too hard. 

The server phrase transfers can be used to override the global transfers-per-ns setting for a specific server.  If the per-server transfer rate is too high for your master servers, you may find that attempted zone transfers are rejected and the server is added to the unreachable masters list (for up to 10 minutes).

serial-query-rate is used to tune the rate at which SOA refresh queries (for slave zones) are sent to the listed masters.  Depending on the version of BIND being used it may also be controlling the rate at which notifies are sent.  Although the limit is expressed as a per-second rate, it is the actions that are being limited, not the network packets needed to complete an action.  So for example, in the case of a zone refresh, there may multiple queries sent as named retries if no response is received, or tries different authoritative servers.   You are therefore recommended to increase the setting gradually to a level that meets your production environment's requirements.  The effects of this setting are usually most apparently immediately following the restart of slave server since this sets effectively sets the refresh time to 'now' (with some jitter) on all zones, even if they are initially loaded and being served from disk backups of the zone data.  

Note that older versions of BIND managed notifies and SOA refresh queries in a single queue (which sometimes caused problems for slaves that are also masters for other servers). To ensure that notifies and refreshes were not competing with each other, in BIND versions 9.6-ESV-R11, 9.8.7, 9.9.5 and 9.10.0 we introduced independent queues - both still with serial-query-rate controlling them.

From BIND 9.9.7-S1 (and this change will also be found in BIND 9.11.1) there are three separate rate-limiting controls: serial-query-rate; notify-rate and startup-notify-rate.

For more information on rate-limiting notifications and SOA refresh queries, please read: serial-query-rate, notify-rate and startup-notify-rate: how they impact zone transfers in different versions of BIND

Setting serial-query-rate too high may cause operational problems

If the serial query rate is too high on a slave server with a large number of zones, then when it is restarted it will send a high initial rate of SOA queries to the configured masters for its slaved zones.  This may have more impact on the masters than the slave itself, although the slave will have to handle both outbound queries and inbound replies.

Notifies are queued ahead of SOA refresh queries

When a slave server is restarted and after loading any zones from available disk files, it will send the notifies (if configured to do so) ahead of checking with the other authoritative servers that it has the most up to date version of the zone(s) loaded.  Since serial-query-rate is used as the throttle for both of these operations, it can mean that servers with a very large number of zones won't commence their zone refreshes until all the notifies have been sent.

Zone refresh times are usually controlled by the master server for a zone which provides them to its slaves in the SOA record as refresh and retry intervals and zone expire interval (see above).  Busy slave servers may want to have some control over their own behaviour however, so the following settings can be used to set boundaries on the refresh and retry settings received in SOA records.  The master server's expire time however is always observed.   It would be unusual, although not unreasonable to reconfigure these limits, although bear in mind that frequent refreshes will use both network bandwidth and server processing resources.

min-refresh-time is used to set a lower boundary on the rate at which a zone refresh is scheduled for any one slave zone.  The default is 300 seconds (5 minutes).  Even if the master server requests a more frequent refresh interval in the SOA record, named will override this.

max-refresh-time (default 4 weeks) is used to ensure that slaved zones are periodically refreshed from the master, even if the master server has tried to specify otherwise.

min-retry-time (500 seconds) and max-retry-time (2 weeks) are used similarly to limit the range of retry timer values that may be received in an SOA record for a zone.

max-transfer-time-in is used to terminate inbound zone transfers that have not completed by a set time limit.  The default is 2 hours, which should be adequate for most scenarios.

max-transfer-idle-in is used to terminate inbound zone transfers that are no longer making any progress.  The default is 1 hour. 

try-tcp-refresh is a compatibility setting that defaults to yes.  When set, if all prior attempts to obtain the SOA record over UDP have failed, then before giving up, there will be one more try, this time using TCP.  It would be unusual to change this setting.

Some recent/relevant code changes:

During an rndc reconfig, defer pausing the server until after parsing the named.conf file.   Implemented from 9.6-ESV-R6, 9.7.5, 9.8.2 and 9.9.0 onwards:

This change can reduce the time when a server is unavailable during "rndc reconfig" for servers with large and complex configurations.  This is achieved by completing the parsing of the configuration files in entirety before entering the exclusive phase.  (Note that it does not reduce the total time spent in "rndc reconfig", and it has no measurable impact on server initial start-up times.) [RT  #21373]

The parsing of the configuration file can take a significant amount of time, but it's not necessary to prevent named from responding to queries until the point is reached where changes are actually being made.

Speed up config reload for installations with many zones.  Implemented from 9.6-ESV-R6, 9.7.5, 9.8.2 and 9.9.0 onwards:

Improves initial start-up and server reload time by increasing the default size of the hash table the configuration parser uses to keep track of loaded zones and allowing it to grow dynamically to better handle systems with large numbers of zones.  [RT #26523]

This change makes a significant difference to the speed at which a server load or reload can take - particularly where the number of individual zones is of the order of 100K or greater.

Retrying of a cached 'unreachable' master server upon receipt of a notify.  Implemented from versions 9.6-ESV-R6, 9.7.5, 9.8.2 and 9.9.0 onwards:

Master servers that had previously been marked as unreachable because of failed zone transfer attempts will now be removed from the "unreachable" list (i.e. considered reachable again) if the slave receives a NOTIFY message from them. [RT #25960]

If a zone refresh fails with a specific master (either during the query for the SOA or after querying and while attempting a subsequent zone transfer), then this master is cached as 'unreachable' for 10 minutes.  The change above removes the server from the unreachable list if a notify is received from it.  Note that receipt of a notify (which is a UDP packet travelling from master to slave) doesn't guarantee that the master will be reachable from the slave, but it does ensure quicker recovery in the situation where a master was temporarily unavailable, for example for a reboot.

Prevention of "deleted from unreachable cache" messages.  Implemented from versions 9.6-ESV-R9, 9.8.5, 9.9.3 and 9.10 onwards:

Silences spurious "deleted from unreachable cache" messages. [RT #30501]

If you are running BIND 9.6-ESV-R6, 9.7.5, 9.8.2, 9.9.0 or newer, named may mistakenly find and report on older unreachable cache entries that are already "deleted".  If there is no evidence of ongoing problems (from further inspection of logging category xfer-in) then you can safely ignore these messages. They indicate that there has been a connectivity issue at some point in the past, but which is now cleared.  The spurious messages should no longer be seen from BIND 9.6-ESV-R9, 9.8.5, 9.9.3 and 9.10 onwards.


© 2001-2016 Internet Systems Consortium

Please help us to improve the content of our knowledge base by letting us know below how we can improve this article.

If you have a technical question or problem on which you'd like help, please don't submit it here as article feedback.

For assistance with problems and questions for which you have not been able to find an answer in our Knowledge Base, we recommend searching our community mailing list archives and/or posting your question there (you will need to register there first for your posts to be accepted). The bind-users and the dhcp-users lists particularly have a long-standing and active membership.

ISC relies on the financial support of the community to fund the development of its open source software products. If you would like to support future product evolution and maintenance as well having peace of mind knowing that our team of experts are poised to provide you with individual technical assistance whenever you call upon them, then please consider our Professional Subscription Support services - details can be found on our main website.

Feedback
  • There is no feedback for this article
Info Submit Feedback on this Article
Nickname: Your Email: Subject: Comment:
Enter the code below:
Quick Jump Menu