Asterisk name and location lookup AGI

Display name or location ('Lc') of caller on your IP / VoIP or soft SIP phones;



Current version or tar: 2024-09-03 09:40:27 UTC

Description

Use the Asterisk Gateway Interface to lookup telephone numbers and put the callers- name or location in the display-name;
Whenever '${CALLERID(NUM)}' (telephone number) is a number and '${CALLERID(NAME)}' (display-name) is empty, this AGI will try to lookup the name in a database.
If this fails it will try a web lookup. (The current version supports configuring just one website. I'm working on a multi website version though. If you know any reverse lookup websites, please let me know.)
If this also fails, the software will try to lookup the area- and then the country code. The software comes with more than 20000 area codes from all over the world. This is vastly superior to IP address based geolocation. Geolocation databases are notoriously inaccurate and outdated.
If you don't know who is calling you, at least you know where they're calling from. Locations are prefixed with the string 'Lc ';

Note: Asterisk can be made to block anonymous calls, making it unlikely not to have a valid telephone number.
Note: If the software can't even find the country, there is probably something wrong with your setup or configuration.
Note: If the telephone number is in the phones' internal address book, the phone may display the address book info instead.
You can test the area and country lookup here.

AGI Basics

Asterisk sends the following information to the lookup program;

agi_network:
agi_request:
agi_channel:
agi_language:
agi_type:
agi_uniqueid:
agi_version:
agi_callerid:
agi_calleridname:
agi_callingpres:
agi_callingani2:
agi_callington:
agi_callingtns:
agi_dnid:
agi_rdnis:
agi_context:
agi_extension:
agi_priority:
agi_enhanced:
agi_accountcode:
agi_threadid:

Followed by an empty line.

The fields that we are interested in are 'agi_callerid' (telephone number) and 'agi_calleridname' (display-name). If calleridname is empty and callerid is a number the AGI software will try to do a name lookup, and if successful send this to Asterisk;

SET VARIABLE NAME "John Doe"

If all goes well Asterisk will then respond with;

200 result=1

Implementation

Stdin - Stdout

The stdin - stdout AGI interface of some Asterisk versions may be a bit buggy, so the most of the examples below use Fast AGI (AGI over TCP/IP) instead;
Xinetd is used as a server: The lookup program communicates with Xinetd and Xinetd with Asterisk. It is also possible to use a systemd socket instead of (x)inetd.

Below various ways that the communication between Asterisk and this lookup AGI can be implemented. Emphasis is on the second method;

Errors are logged to syslog.
Note: If you already using port 4573, just set this AGI to an other port. For instance port 4574
Note: In the setup described here, this AGI runs as user asterisk. It is possible to run it as an other (system) user.
Note: It may be possible to use this AGI with FreePBX. I don't use FreePBX myself, so you're on your own here.

Daemon

The functionality of this AGI software can also be achieved by means of a standalone network daemon.
There is a daemon version of this software called 'Asterisk name and location lookup AGI Daemon' or 'namelookup.agid' for short. It's based on this software and relies on the utilities and documentation that come with this software. The daemon version is meant for systems that are very busy indeed!
Note: The daemon implementation is highly experimental!

Files

Phone books

Note that a telephone number is NOT a number but a character string;
The numbers 1, 01 and 001 are identical.
Telephone numbers 1, 01 and 001, on the other hand, are completely different things! Leading zeros do count. And therefore telephone numbers aren't actually numbers!

Default file format

These are the following files;

numbers.db

Contains names and telephone numbers of your contacts.

areas.db

Contains area codes and names of cities, counties or provinces.

regions.db

Contains country codes.

areas.db, numbers.db and regions.db are binary files. Records are 56 bytes (or 96 bytes when compiled with 'make wide');

Number:

19 byte US-ASCII NULL terminated string (20 including terminating NULL).

Number length:

4 byte signed integer.

Name:

31 byte UTF-8 NULL terminated string (32 including terminating NULL).
Note: When compiled with 'make wide' this is 71 bytes (72 including terminating NULL).

Records are fixed length with zero padding.
Number length is zero in numbers.db and the length of the number in areas.db and regions.db.
Sort order is Number.
More about this file format below.

Compact file format

With compact files numbers and names or locations are split into separate files:

numbers.lst

Contains the names of your contacts as UTF-8 NULL terminated strings without any padding. So records are variable length, which is more efficient.

numbers.idx

Index to numbers.lst: Contains the numbers, their length and position (offset in bytes) of the corresponding name in 'numbers.lst'. It also contains the length of the name, excluding the terminating NULL.

areas.lst

Contains cities, counties or provinces as UTF-8 NULL terminated strings.

areas.idx

Index to areas.lst

regions.lst

Contains countries as UTF-8 NULL terminated strings.

regions.idx

Index to regions.lst

Number length is zero in 'numbers.idx'. And the length of the number in 'areas.idx' and regions.idx.
Sort order is Number.

In 'numbers.idx', 'araeas.idx' and 'regions.idx' records have a fixed length of 32 bytes with zero padding;

Number

19 byte US-ASCII null terminated string (20 including terminating NULL).

Number length

4 byte signed integer.

Name offset

4 byte signed integer.
Position of name in bytes from start of .lst file.

Name length

4 byte signed integer.
Length of name or location, excluding the terminating NULL.

More about this file format below.

Field sizes

Numbers are never more then 15 digits. And most phones can't display much more then 20 chars. So these sizes should be plenty, except when using non-Latin scripts;
With UTF-8 a character may be more then one byte. E.G. Two bytes for Greek and Cyrillic. And three bytes for Asian scripts. When compiled with 'make wide', the maximum name or location size is 71 instead of 31 bytes. This makes the total .db record size 96 bytes.
Note that the use of 32-bit signed integers limits the file sizes to 2 GB.
More about the phone book below.

Config files

agi-namelookup.conf

If web lookups are disabled, this is set in /etc/namelookup-agi/agi-namelookup.conf;

weblookup=off

If compact files are enabled, this is set in /etc/namelookup-agi/agi-namelookup.conf;

compact=on

If agi-namelookup.conf doesn't exist, web lookups default to on and compact files to off.

agi-nameblock.db

agi-nameblock.db lists names which are considered invalid. When '${CALLERID(NAME)}' (display-name) is set to one of these the software will consider this field empty and do a lookup anyway.
agi-nameblock.db is a binary file. Records are 32 bytes. Each record is a lower case 31 byte UTF-8 NULL terminated string with zero padding.
Sort order is unsigned ascending byte value.
If the file agi-nameblock.db doesn't exist, the list defaults to 'anonymous', 'privacy manager' and 'unknown'.
Note: Matches are not case sensitive. So if 'anonymous' is blocked, 'Anonymous' and 'ANONYMOUS' are also blocked.
Note: When compiled with 'make wide', the record size is 72 instead of 32 bytes.
More about the block list below.

Log file

namelookup.log
Normally these are five fields separated by tabs;

ISO date<Tab>Lookup type<Tab>Number<Tab>Name<Tab>Asterisk result code<Lf>

Lookup types

ArLk	Area code (city, county or province) lookup
DbLk	Database lookup
RgLk	Region (country) lookup
WbLk	Web lookup

Only successful lookups are logged.

Debug (-d) logging

debug is for stdin - stdout tests. Here the last field is replaced by 'Debug'. And the input from the test is also written to the log file.
Logging is done to the default directory. It also uses config- and database files in the default directory.

Syslog

The software logs to syslog too;

Syslog entries

Log entry	Meaning
namelookup.agi[PID]: connect from localhost (127.0.0.1)	Connect to inetd.
namelookup[PID]: Version: Version Number	Start of namelookup.agi
namelookup[PID]: Lookup nr: Number	Telephone number to be lookup up.
namelookup[PID]: Name from VP: Name	Name set by VoIP provider.
namelookup[PID]: Invalid: Name	Name set by VoIP provider is not a valid name.
namelookup[PID]: dblookup: Name	Name or area found in database.
namelookup[PID]: weblookup: Name	Name found on web.

Plus various errors that may occur.
'Timeout expired' usually means that the website lookup takes too long. Other errors are more serious.

Installation

Required

• (g)cc
• make
• For Fast AGI: (x)inetd (or systemd)
• For Fast AGI: tcpd (or systemd)
• For web lookups: Curl downloader
• For agiconf:
  • libncursesw
  • Ncursesw C header files: libncursesw-dev
  Note that these are the 'wide' (W) variety. Within this context 'wide' is not the same thing as 'make wide';
  'wide' ncurses supports wide characters. These are 16- or 32-bits instead of 8-bits.
  Note: Packages may contain version numbers, E.G.: libncursesw5.
• Recommended: xxd

Download and install

• Download: namelookup.tar.gz
• Extract with: tar xvfz namelookup.tar.gz
• cd namelookup/
• Run make. This will compile the binaries;

~$ make
cc -O2 -Wall  -o agiblk2db block2db.c
cc -O2 -Wall  -o agiconf agiconf.c -lncursesw
cc -O2 -Wall  -o agidx2ph idx2ph.c
cc -O2 -Wall  -o aginum2db num2db.c
cc -O2 -Wall  -o agiph2idx ph2idx.c
cc -O2 -Wall -o csv2tsv csv2tsv.c
cc -O2 -Wall -o namelookup namelookup.c
cc -O2 -Wall  -o namelookup.agi agi-namelookup.c

Alternatively you can do 'make wide' instead. This sets a define ('-D') 'ANL_WITH_WIDE_NAMES'. This will increase the maximum name- or location size from 31 to 71 bytes, which makes the software more suitable for non-Latin scripts;

~$ make wide
cc -O2 -Wall -DANL_WITH_WIDE_NAMES -o agiblk2db block2db.c
cc -O2 -Wall -DANL_WITH_WIDE_NAMES -o agiconf agiconf.c -lncursesw
cc -O2 -Wall -DANL_WITH_WIDE_NAMES -o agidx2ph idx2ph.c
cc -O2 -Wall -DANL_WITH_WIDE_NAMES -o aginum2db num2db.c
cc -O2 -Wall -DANL_WITH_WIDE_NAMES -o agiph2idx ph2idx.c
cc -O2 -Wall -o csv2tsv csv2tsv.c
cc -O2 -Wall -o namelookup namelookup.c
cc -O2 -Wall -DANL_WITH_WIDE_NAMES -o namelookup.agi agi-namelookup.c

Next, as root run 'make install'. This runs install.sh

• This will copy and strip;
  namelookup.agi to /usr/local/sbin/
  agiblk2db to /usr/local/bin/
  agiconf to /usr/local/bin/
  aginum2db to /usr/local/bin/
  csv2tsv to /usr/local/bin/
  namelookup to /usr/local/bin/
• Create dir /var/local/log/asterisk/
  and chown it to asterisk:asterisk
• Copy doc/* to /usr/local/share/doc/namelookup/
• Copy and gzip;
  man/agiconf.1 to /usr/local/share/man/man1/
  man/agiblk2db.1 to /usr/local/share/man/man1/
  man/aginum2db.1 to /usr/local/share/man/man1/
  man/csv2tsv.1 to /usr/local/share/man/man1/
  man/namelookup.agi.8 to /usr/local/share/man/man8/
  man/agi-nameblock.db.5 to /usr/local/share/man/man5/
  man/agi-namelookup.conf.5 to /usr/local/share/man/man5/
  man/confagi.conf.5 to /usr/local/share/man/man5/
• Copy;
  data/*.tsv to /usr/local/share/phonebook/
  utils/*.sh to /usr/local/share/phonebook/utils/
• Copy if present;
  agi-namelookup.conf to /etc/namelookup-agi/
  agi-nameblock.db to /etc/namelookup-agi/
  weblookup.conf to /etc/namelookup-agi/
  areas.db to /var/local/lib/phonebook/
  regions.db to /var/local/lib/phonebook/
  numbers.db to /var/local/lib/phonebook/
  chmod 640 /var/local/lib/phonebook/numbers.db
  chown root:asterisk /var/local/lib/phonebook/numbers.db
• Start agiconf if you want to.
  Agiconf is an ncurses program which generates agi-namelookup.conf, agi-nameblock.db, numbers.db, areas.db and regions.db. It can also generate and install compact files.

Target directories are created if they do no exist.
If you re-run install later, only newer versions of the above files are installed.

Configuration

Agiconf

Agiconf is a ncurses configuration tool. It can generate agi-namelookup.conf, and agi-nameblock.db, numbers.db, areas.db, and regions.db (or their compact file format equivalents). It's called from the install script, but you can (re)run it later if you want.

The main menu has the following items;

1. Select Country
2. Configuration
3. Convert Blocklist
4. Convert Phonebook
5. Select Areas
6. Select Regions

Select Country

This sets trunk prefix and international call prefix;

Configuration

If you are not interested in web lookups, you can disable them here.
You can also choose to use compact files.

Convert Blocklist

Convert agi-nameblock.txt to agi-nameblock.db.
The software first looks for agi-nameblock.txt in /etc/namelookup-agi/, then the default directory and then in your home directory.
Note: You can do this conversion later if you want.

Convert Phonebook

Convert numbers.tsv to numbers.db or numbers.idx and numbers.lst
The software first looks for numbers.tsv in /etc/namelookup-agi/, then the default directory and then in your home directory.
Note: You can do this conversion later if you want.

Select Areas

The default behaviour is to just lookup countries. If you want to lookup cities and areas as well you can enable this on a per country basis. Or all if you like and then select language and resolution on a per country basis.
So instead of 'Germany' it might say 'DE Düsseldorf'.



Note: With area code lookups enabled, the software will display the two letter ISO country code instead of the country name.
Note: If the software can't find the area code, it will display the country name instead of the city or area name.

Select Regions

This has an option called 'With federal states'. With this enabled, instead of 'USA' or 'Canada', it will display the federal state as well. E.G.: 'US New Jersey' or 'CA Manitoba'.

Agiconf HTML man page here.

Fast AGI

This is AGI over TCP/IP.
Note: If you are already using port 4573, set this to an other port.

Inetd

In /etc/services, under local services, add:

agi		4573/tcp			# Asterisk Gateway Interface

In /etc/inetd.conf, add;

agi	stream	tcp	nowait	asterisk	/usr/sbin/tcpd	/usr/local/sbin/namelookup.agi

Or, if you want an other user then 'asterisk', E.G. 'namelookup';
Note: This is not supported by the install script.

agi	stream	tcp	nowait	namelookup	/usr/sbin/tcpd	/usr/local/sbin/namelookup.agi

Restart inetd. E.G.:

~# /etc/init.d/xinetd reload

Or the systemd equivalent thereof;

~# systemctl reload xinetd

Systemd socket

You can use a systemd socket instead of (x)inetd.
Hypothetically, the following should work.
I don't use systemd myself, so if this doesn't work, you are on your own!

In /etc/services, under local services, add:

agi		4573/tcp			# Asterisk Gateway Interface

In /etc/systemd/system/agi@.service;

[Unit]
Description=AGI service
After=network.target agi.socket
Requires=agi.socket

[Service]
Type=simple
TimeoutStopSec=10
User=asterisk
Group=asterisk
StandardInput=socket
StandardOutput=socket
StandardError=journal
ExecStart=/usr/local/sbin/namelookup.agi
Restart=no

[Install]
WantedBy=default.target

Replace 'asterisk' by 'namelookup' if you want to run as user namelookup.
Note: This is not supported by the install script.

In /etc/systemd/system/agi.socket;

[Unit]
Description=AGI socket
PartOf=agi.service

[Socket]
ListenStream=4573
Accept=true
ReusePort=true
Writable=true

[Install]
WantedBy=sockets.target

Next;

~# systemctl reload
~# systemctl enable agi.socket
~# systemctl start agi.socket

Check with;

~# systemctl status agi.socket

Note: This is not fully tested!

Check for listening socket

Agi should now show up in a netstat;

~$ netstat -a | grep agi
tcp6       0      0 [::]:agi                [::]:*                  LISTEN

Or;

~$ netstat -an | grep 4573
tcp6       0      0 :::4573                 :::*                    LISTEN

You can further test things with telnet. As shown with the location lookup example below;

~$ telnet localhost agi
Trying ::1...
Connected to localhost.
Escape character is '^]'.
agi_callerid: 0207654321

SET VARIABLE NAME "Lc Amsterdam"
Connection closed by foreign host.

You need to cut and paste ('agi_callerid: 0207654321') quickly because the program has a timeout of just a few seconds. And don't forget the blank line (an extra <Enter>). And after the AGI response, press <Enter> again. This will stop the program.
Note: The above example is for within the Netherlands. Outside the Netherlands the same lookup would include the NL country code (31): agi_callerid: 0031207654321 (assuming the international call prefix is '00').

Asterisk

If you have got all of the above to work, you can start configuring Asterisk.
In /etc/asterisk/extensions.conf: Just before the internal dial command that dials your phone(s) on incoming calls:

	same => n,Set(NAME=${CALLERID(NAME)})
	same => n,Set(AGISIGHUP=no)
	same => n,AGI(agi://127.0.0.1)
	same => n,Set(CALLERID(NAME)=${NAME})

Variable NAME is set to the Caller-Id name.
'Set(AGISIGHUP=no)' keeps Asterisk from sending a 'HANGUP'.
If the lookup is successful, the AGI will send the string 'SET VARIABLE NAME' followed by the name.
This value is then assigned to Caller-Id name.
Note: You can, of course, use a hostname instead of an IP address. And IPv6 is supported as well.

Note: In a non-FastAGI (non-TCP/IP), stdin - stdout version this would have been;

	same => n,Set(NAME=${CALLERID(NAME)})
	same => n,Set(AGISIGHUP=no)
	same => n,AGI(/usr/local/sbin/namelookup.agi)
	same => n,Set(CALLERID(NAME)=${NAME})

Next, reload extensions.conf:

asterisk -rx "dialplan reload"

With non standard port

An example with port 4574;

	same => n,Set(NAME=${CALLERID(NAME)})
	same => n,Set(AGISIGHUP=no)
	same => n,AGI(agi://127.0.0.1:4574)
	same => n,Set(CALLERID(NAME)=${NAME})

Use this if you're already using port 4573.

With privacy manager

If you want to, you can combine this with the Privacy Manager;

exten => Your_Extention,1,GotoIf($["${CALLERID(num)}" = "anonymous"]?anon:nona)
	same => n(anon),Set(CALLERID(num)=)
	same => n(nona),Answer()
	same => n,PrivacyManager()
	same => n,GotoIf($["${PRIVACYMGRSTATUS}" = "FAILED"]?pmfail:num)
	same => n(num),Set(NAME=${CALLERID(NAME)})
	same => n,Set(AGISIGHUP=no)
	same => n,AGI(agi://127.0.0.1)
	same => n,Set(CALLERID(NAME)=${NAME})
	same => n,Dial(Your_Internal_Dial_Command)
	same => n,Hangup()
	same => n(pmfail),Playback(im-sorry)
	same => n,Playback(vm-goodbye)
	same => n,Hangup()

With block anonymous calls

Or block anonymous calls completely.

exten => Your_Extention,1,GotoIf($["${CALLERID(num)}" = "anonymous"]?anon:nona)
	same => n(nona),GotoIf($["${CALLERID(num)}" = ""]?anon:num)
	same => n(num),Set(NAME=${CALLERID(NAME)})
	same => n,Set(AGISIGHUP=no)
	same => n,AGI(agi://127.0.0.1)
	same => n,Set(CALLERID(NAME)=${NAME})
	same => n,Dial(Your_Internal_Dial_Command)
	same => n,Hangup()
	same => n(anon),Answer(500)
	same => n,PlayBack(/usr/local/share/asterisk/mysounds/anonymous)
	same => n,Hangup()

Note: '/usr/local/share/asterisk/mysounds/anonymous' is my own file.
See: Blocking anonymous calls for more info.

With Blacklist

And this adds a blacklist;

exten => Your_Extention,1,GotoIf($["${CALLERID(num)}" = "anonymous"]?anon:nona)
	same => n(nona),GotoIf($["${CALLERID(num)}" = ""]?anon:num)
	same => n(num),GotoIf($[${BLACKLIST()} = 1]?blck:nobl)
	same => n(nobl),Set(NAME=${CALLERID(name)})
	same => n,Set(AGISIGHUP=no)
	same => n,AGI(agi://127.0.0.1)
	same => n,Set(CALLERID(NAME)=${NAME})
	same => n,Dial(Your_Internal_Dial_Command)
	same => n,Hangup()
	same => n(anon),Answer(500)
	same => n,PlayBack(/usr/local/share/asterisk/mysounds/anonymous)
	same => n(blck),Hangup()

See: Blocking blacklisted numbers for more info.

Security

Firewall port 4573.
Limit access in /etc/hosts.allow and /etc/hosts.deny.

Log file rotation

In /etc/logrotate.d/asterisk-namelookup;

/var/local/log/asterisk/namelookup.log {
        monthly
        missingok
        rotate 4
        compress
        delaycompress
        notifempty
        create 660 asterisk asterisk
}

Modify to suit your needs.

Block list

If your VOIP provider set a name for you, there is no need to do a lookup. However, if this name is invalid the software needs to do a lookup anyway. The file 'agi-nameblock.db' lists names that are considered invalid.
agi-nameblock.db is a binary file. It can be derived from 'agi-nameblock.txt'. The script 'block2db.sh' sorts agi-nameblock.txt to alphabetical order and uses 'agiblk2db' to convert the data to agi-nameblock.db.
agi-nameblock.txt is an UTF-8 text file. It contains one invalid name per line.
Example;

anonymous
privacy manager
unknown

Maximum length is 31 bytes (32 including newline). Except when compiled with 'make wide', in which case the maximum length is 71 bytes (72 including newline).
Note: If you use block2db.sh to do the conversion, all entries in agi-nameblock.txt need to be lower case.

Alternatively, you can use agiconf to do the conversion. Agiconf will also convert any upper case to lower case. This includes non-ASCII; It will convert 'Ö' to 'ö' and 'Ω' to 'ω'.
Note: Agiconv will only convert non-ASCII to lower case in an UTF-8 environment!
I recommend strongly against using Agiconv in a non UTF-8 environment. Agiconv was never tested in a non UTF-8 environment on a big-endian machine and may not work under such conditions!

If agi-nameblock.db does not exist, the list of invalid names defaults to 'anonymous', 'privacy manager' and 'unknown'.

Phone book

Phone book format

Phone books come in all sorts of formats. There are two fields they always have in common: Name and Telephone number.
I use a phone book with the following format as a base for all other formats:

Name <Tab> Telephone number <Line Feed>

This I then convert to whatever format is required for soft phones and SIP-phones.

Default phone book file format

The script 'ph2num.sh' converts the file 'phonebook.txt' with this format to;

Telephone number <Tab> Name <Line Feed>

And saves this as 'numbers.tsv'. It also sets the sort order to Telephone number.
It then uses 'aginum2db' to convert this to binary format.
The binary phone book is saved as 'numbers.db'.
Below the file format.

Byte  0       1        2        3        4        5        6        7
   ┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
 0 │ Number                                                                │  7
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
 8 │                                                                       │ 15
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
16 │                            0      │ Number length                     │ 23
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
24 │ Name                                                                  │ 31
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
32 │                                                                       │ 39
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
40 │                                                                       │ 47
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
48 │                                                                0      │ 55
   └────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘
    48       49       50       51       52       53       54       55

Records have a fixed length of 56 bytes with zero padding. Except when compiled with 'make wide', in which case it's 96 bytes;

Byte  0       1        2        3        4        5        6        7
   ┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
 0 │ Number                                                                │  7
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
 8 │                                                                       │ 15
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
16 │                            0      │ Number length                     │ 23
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
24 │ Name                                                                  │ 31
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
32 │                                                                       │ 39
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
40 │                                                                       │ 47
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
48 │                                                                       │ 55
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
56 │                                                                       │ 63
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
64 │                                                                       │ 71
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
72 │                                                                       │ 79
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
80 │                                                                       │ 87
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
88 │                                                                0      │ 95
   └────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘
    88       89       90       91       92       93       94       95

Number and name are NULL-terminated character strings. Number length is a 32-bit signed integer.
Number length is zero in 'numbers.db'. And the length of the number in 'areas.db' and 'regions.db'.
Note: When number length is zero, namelookup.agi will try to match the complete telephone number. And when number length is not zero, namelookup.agi will only try to match the first number-length number of digits of the telephone number.

Compact phone book file format

With compact files numbers and names are split into separate files;

Numbers	numbers.lst	numbers.idx
Areas	areas.lst	areas.idx
Regions	regions.lst	regions.idx

.lst files are UTF-8 NULL terminated strings without any padding.
Below the .idx format;

Byte  0       1        2        3        4        5        6        7
   ┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
 0 │ Number                                                                │  7
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
 8 │                                                                       │ 15
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
16 │                            0      │ Number length                     │ 23
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
24 │ Name offset                       │ Name length                       │ 31
   └────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

Records are fixed length (32 bytes) with zero padding.
Number is a NULL-terminated character string. The other fields are 32-bit signed integers.
Number length is zero in 'numbers.idx'. And the length of the number in 'areas.idx' and 'regions.idx'.
Note: When number length is zero, namelookup.agi will try to match the complete telephone number. And when number length is not zero, namelookup.agi will only try to match the first number-length number of digits of the telephone number.
'agiph2idx' converts the default format to compact. 'agidx2ph' converts the compact format to default. And agiconf can generate and install both formats.

Phone book contents

Normally a phone book is for dialling out. But this phone book is meant for recognising incoming phone calls.
A dial-out phone book may contain toll-free numbers.
A dial-in phone book needs to contain the phone numbers by which companies identify them selves instead.

Phone numbers are just digits: 0123456789. So no spaces, brackets, pluses or hyphens. Format is 'trunk_prefix area_code subscribers_number' or 'international_call_prefix country_code area_code subscribers_number'.
Examples:

Most countries

General phone number

Trunk prefix:	0
International call prefix:	00
Country code:	12
Area code:	34
Subscribers NR:	56789

Phone number is: 03456789 or: 00123456789
Assuming of course that the trunk prefix is '0' and the international call prefix is '00'.

USA

Some places use a different trunk prefix or international call prefix. In the USA for instance, it's '1' resp '011';

USA phone number

Trunk prefix:	1
International call prefix:	011
Country code:	12
Area code:	34
Subscribers NR:	56789

Phone number is: 13456789 or: 011123456789

Other properties

Names are UTF-8 text.
Sort order is ALPHABETIC sort of phone numbers.
Phone numbers have to be UNIQUE: They may occur only ONCE! The same name may occur multiple times though;
A phone number has one name.
A person may have more then one number.

Phone book installation

cp numbers.db /var/local/lib/phonebook/
cd /var/local/lib/phonebook/
chmod 640 numbers.db
chown root:asterisk numbers.db

Alternatively you can use agiconf. If you want to agiconf converts numbers.tsv to numbers.db, sorts to alphabetical order and installs the file with the right permissions.
Note: Generally speaking, Agiconf will insert the right trunk- and international call prefix. An exception is the conversion from 'numbers.tsv' to 'numbers.db'. Here the correct trunk- and international call prefixes should already be in the TSV file!
A list of countries their trunk prefix and international call prefix here.
Most organisations list their phone number as;
+ Country_code (trunk_prefix) area_code subscribers_number
However, according to ITU standard E.123 this should be;
+ Country_code area_code subscribers_number
And some list their number as;
(trunk_prefix area_code) subscribers_number.
Which can be very confusing indeed. Don't put an area code in brackets!

Right and Wrong way to spell telephone numbers

Right ✓	+31 20 7654321	Trunk prefix is missing though
Right ✓	+31 (0) 20 7654321	Trunk prefix in brackets
Wrong!	(020) 7654321	Missing country code!

Always include your country code!
If you do include a trunk prefix, put it in brackets! Telephone number formats are different all over the world. So most people don't know what your trunk prefix is.

Note: The software uses telephone numbers consisting of nothing but digits.
Note: 'numbers.tsv' should use the format that is customary in your country. With (if applicable) trunk prefixes for national numbers and country codes for foreign numbers.

Web lookups

For unrecognised numbers the program does a web lookup. If successful, the result gets saved in 'namelookup.log'. These are marked with the string WbLk (Web Lookup). You can add these to your phone book.

Web look up configuration

The configuration for web look ups is in '/etc/namelookup-agi/weblookup.conf'.
It consists of a number of variable=value pairs, one per line;

weblookup.conf

Variable	Type	Method	Value
preget	String	GET	String before number
		POST	URL
pstget	String	GET	String after number
prepost	String	POST	String before number
pstpost	String	POST	String after number
prename	String	Both	String before name
pstname	String	Both	String after name
atlin	Integer	Both	At or after Line
atcol	Integer	Both	At or after Column

When not specified, strings default to empty and integers to zero.
Note: preget and pstname are required, the is rest opional.
Note: 'weblookup.conf' is not generated by agiconf.

GET Config

preget

String before number

pstget

String after number

POST Config

preget

Website URL

prepost

String before number

pstpost

String after number

Find name in web page

prename

String before name

pstname

String after name

atlin

Look for name at or after this line

atcol

Look for name at or after this column

GET Example

Lookup 0207654321 on a now defunct Dutch website;

http://www.nummerid.com/result.php?input_telefoon=0207654321&submit_telefoon.x=52&submit_telefoon.y=18&input_naam=&input_adres=&input_postcode=

The string before the number is;

http://www.nummerid.com/result.php?input_telefoon=

The string after the number is;

&submit_telefoon.x=52&submit_telefoon.y=18&input_naam=&input_adres=&input_postcode=

This makes 'preget' and 'pstget';

preget=http://www.nummerid.com/result.php?input_telefoon=
pstget=&submit_telefoon.x=52&submit_telefoon.y=18&input_naam=&input_adres=&input_postcode=

The software will insert the telephone number between these two.

You can use (double or single) quotes if you like, but you don't have to;

preget="http://www.nummerid.com/result.php?input_telefoon="

Or;

preget='http://www.nummerid.com/result.php?input_telefoon='

If you do use quotes make sure that you the SAME type of quotes on BOTH beginning and end. Any other quotes will be considered part of the config string!
Note: You can't use quotes with 'atlin' and 'atcol'.

You can use hash for remarks;

# Some remark

But it has to be on the start of the line. Any other hash will be considered part of a config statement!

POST

Post works the same way. Just put the strings in 'prepost' and 'pstpost'. You need to put the URL in 'preget' though;

preget=http://www.nummerid.com/

When 'prepost' and 'pstpost' are not empty, the software will consider 'preget' the URL.
Note: The Curl -d option is used for POST. This means that the Content-Type is 'application/x-www-form-urlencoded'. So you may have to percent-encode some of the characters in the prepost and pstpost strings!

Name

'prename' and 'pstname' are strings before and after the name. Example:

<Some_Tag>John Doe<Other_tag>

prename=<Some_Tag>
pstname=<Other_tag>

This has to be an EXACT copy of the HTML source; This match is case sensitive! An exception is white space other than space. You need to replace linefeed, carriage return and tab by space. So that's two spaces for a carriage return - linefeed pair.
The strings should be long enough to uniquely identify the name. The maximum length is 255 bytes.
Note: With UTF-8 a character may be more than one byte. If you need the column position, it's in bytes, not characters!

Atlin and Atcol

'atlin' means at line.
'atcol' means at column.
Setting these to a non-zero value will disable name-lookup until at least line count 'atlin' and column count 'atcol' are reached.
And when 'prename' is empty, 'atlin' and 'atcol' indicate the first byte of the name.
Example:

atlin=172
atcol=25

Note: You can't use quotes with 'atlin' and 'atcol'.
Note: If you set to 'atlin' to zero and 'atcol' to a non-zero value, the software will count total number of bytes instead of lines and bytes from start of line.

You can use just 'atlin' and 'atcol' or combine these with 'prename' and 'pstname'.
A lot of web designers like to put strings surrounded by spaces in table cells (<td> ... </td>). In which case you can use a bunch of spaces for 'prename' and 'pstname'.
For instance;

prename="   "
pstname="   "

The number of spaces may actually vary. To compensate for this, leading spaces are removed from the name. So you can put a little margin in 'atcol'. Trailing spaces are also removed. (Spaces in names are, of course, preserved.)
Note: Lines and columns start numbering one, not zero.
Note: The column count is in fact bytes, not characters!
Tip: Some text editors show the cursor position in both bytes and characters on the status line.

Minimum required configuration

When 'prename' is empty and 'atlin' and 'atcol' are both one or smaller, the software will look for the name at the begin of the file.
It always needs a postname though. Without postname it will consider the name not found.

Dump HTML Source

You can dump the HTML source to the log file with the '-D' option.
The source is enclosed between;

 --- Start HTML Dump --- 

and;

 --- End HTML Dump --- 

Both start with a space and end with a space and a linefeed.

Curl configuration

You can set proxy, noproxy and user-agent here. In /etc/namelookup-agi/.curlrc;

proxy=http://proxy.example.org:8080
noproxy=localhost
user-agent="Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"

Note: '.curlrc' is not generated by agiconf.

Disable Web lookups

If you want to you can DISABLE web lookups; Create a file /etc/namelookup-agi/agi-namelookup.conf with the following line;

weblookup=off

Agiconf can do this for you.

Area and country code lookups

If the number is not in the phone book, web lookups are disabled or the web lookup fails and area code lookups are enabled (for this country), the software will try to do a area code lookup.

Format is 'trunk_prefix area_code' or 'international_call_prefix country_code area_code'.
Example:

Trunk prefix:	0
International call prefix:	00
Country code:	12
Area code:	34

Area code is: 034 or: 001234

Use the first format for area codes in your own country and the second for foreign area codes.
Assuming of course that the trunk prefix is '0' and the international call prefix is '00'.
Note: Agiconf can set the correct trunk- and international call prefix for you.
For more information see: Asterisk location lookup AGI country and area code files

Country code lookups

If the number hasn't been found yet, the software will do a country lookup.

Format is 'international_call_prefix country_code'.
Example:

International call prefix:	00
Country code:	12

Format is: 0012
Assuming that the international call prefix is '00'.
Note: Agiconf can set the correct international call prefix for you.

Options

Namelookup.agi options;

namelookup.agi [-d] [-D] [-h] [-l altlog ] [-u althome ] [-v]

-d

Enable debug.
With debug the program reads from stdin and logs to the default directory. It also uses config- and database files in the default directory.

-D

Dump data.
This dumps the data from the HTML page downloaded by curl to the logfile.
This also enables debug.

-h

Print help and exit.

-l Alternate log file

Use this log file instead of default (/var/local/log/asterisk/namelookup.log).

-u Alternate home directory

Use this home directory instead of default (/var/local/log/asterisk/).
Note: This has become superfluous; The old version used w3m, which needs a home directory. Curl on the other hand, doesn't need one.

-v

Print version and exit.

You could create a system user 'namelookup' with home directory '/var/local/lib/namelookup/' and log directory '/var/local/log/namelookup/' and then use these instead of the defaults;

namelookup.agi -l /var/local/log/namelookup/namelookup.log -u /var/local/lib/namelookup

The install script doesn't support this, so it's a bit more DIY.
Note: If you don't use FastAGI but the stdin - stdout interface instead, the AGI process owner is always the same as the Asterisk process owner!