Log file DNS resolver with disc cache

Description

Cachelogres is like Apache's logres but with a disc cache which caches look up results.
When a lookup fails logres will try over and over again. Possibly with a long timeout for each try. Sometimes taking more than 24 hours to process a log. Cachelogres on the other hand, also caches negative results. The positive (a host name was found) and negative (no host name was found) TTLs can be set on the command line and default to four and one weeks resp.

Just like logres cachelogres expects the first string in a line of text to be an IP address followed by a space. It reads from standard input and writes to standard output, replacing this IP address by a host name when it can be found. To make sure that the host name is correct, it will do an AAAA or A look up after the PTR look up;

10.20.30.40	→	example.org
example.org	→	10.20.30.40

This is like the logres '-c' option.

Files

Cache file format

       0        1        2        3        4        5        6        7
     ┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
   0 │ IP Address                                                            │
     ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
   8 │                                                                       │
     ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
  16 │ Host name                                                             │
     ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
     ≈                                                                       ≈
     ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
 264 │                                                                  0    │
     ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
 272 │ Epoch                                                                 │
     └────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

Each entry is 280 bytes:

16 Byte IP address. IPv4 Addresses are stored as IPv4-mapped IPv6 address.
256 Byte host name. NULL Terminated string (with zero padding). Max 255 bytes.
The maximum allowed host name size is 253 bytes. So this adds a little margin.
8 Byte epoch.

Files

hash0, hash1, hash2, ... hashe, hashf

1 to 16 Sparse files.
(The lower 16 to 23 bits of) a 24-Bit CRC of the IP address is used as a hash. The hash is multiplied by the record size (280 bytes) and this is used as a file offset (number of bytes from start of file).
hash0 is tried first. In case of a hash clash (an other IP address having the same hash) the next hash file is used.
If all of the hash files clash the lin file is used.

lin

Regular file.
Used for hash clashes in the above files.
File size: 0 or more times 280 bytes.
This file uses a linear search (which is relatively slow).

Cache lay-out

The number of hash files can be 1 to 16.
The maximum number of entries per hash file can be 64k to 16M.

Hash file size
Hash Bits	Entries	Max Hash	Hash file size
16	64k	0x00FFFF	17920 kB
17	128k	0x01FFFF	35 MB
18	256k	0x03FFFF	70 MB
19	512k	0x07FFFF	140 MB
20	1M	0x0FFFFF	280 MB
21	2M	0x1FFFFF	560 MB
22	4M	0x3FFFFF	1120 MB
23	8M	0x7FFFFF	2240 MB
24	16M	0xFFFFFF	4480 MB

If the lin file gets too big:

Dump the cache to text.
Remove the old cache.
Generate a new cache from the text dump, increasing the number of hash files, their size or both.

Download

Download cachelogres.tar.gz