Asterisk name and location lookup AGI

Description

Use the Asterisk Gateway Interface to lookup telephone numbers and put the subscribers name in ${CALLERID(NAME)};
Whenever ${CALLERID(NUM)} is a number and ${CALLERID(NAME)} is empty, this AGI lookup will try to lookup the name in a database.
If this fails it will try a web lookup (the website is for Dutch telephone numbers, so you need to change the source if you want lookups in other countries).
If this also fails, the software will try to lookup the area- and then the country code;
If you don't know who is calling you, at least you know where they're calling from. Locations are prefixed with the string 'Lc '.

AGI Basics

Asterisk sends the following information to the lookup program; 

agi_network:
agi_request:
agi_channel:
agi_language:
agi_type:
agi_uniqueid:
agi_version:
agi_callerid:
agi_calleridname:
agi_callingpres:
agi_callingani2:
agi_callington:
agi_callingtns:
agi_dnid:
agi_rdnis:
agi_context:
agi_extension:
agi_priority:
agi_enhanced:
agi_accountcode:
agi_threadid:

Followed by an empty line.

The fields that we are interested in are 'agi_callerid' and 'agi_calleridname'. If calleridname is empty and callerid is a number the AGI software will try to do a name lookup, and if successful send this to Asterisk; 

SET VARIABLE NAME "John Doe"

If all goes well Asterisk will then respond with; 

200 result=1

Implementation

The stdin - stdout AGI of my Asterisk is a bit buggy, so I use Fast AGI (AGI over TCP/IP) instead. Xinetd is used as a server: The lookup program communicates with Xinetd and Xinetd with Asterisk (It shouldn't be too hard to write a standalone daemon though);

Asterisk name lookup Fast AGI

Errors are logged to syslog.

Files

Phone book

These are the following files;

numbers.db
Contains names and phone numbers of your contacts.
areas.db
Contains area codes and names of cities or regions.
regions.db
Contains country codes.

areas.db, numbers.db and regions.db are binary files. Records are 56 bytes;

Number:
19 byte null terminated string (20 including terminating NULL).
Length:
4 byte signed integer.
Name:
31 byte null terminated string (32 including terminating NULL).

Records are fixed length with zero padding.
Length is zero in numbers.db and the length of the number in areas.db and regions.db.

Numbers are never more then 15 digits. And most phones can't display much more then 20 chars. So these sizes should be plenty.
More about the phone book below.

Config file

If web lookups are disabled, this is set in /etc/agi-namelookup.conf.

Log file

namelookup.log
Normally these are five fields separated by tabs; 

ISO date<Tab>Lookup type<Tab>Number<Tab>Name<Tab>Asterisk result code<Lf>
Lookup types
ArLkArea code (city) lookup
DbLkDatabase lookup
RgLkRegion (country) lookup
WbLkWeb lookup

Only successful lookups are logged.

Debug (-d) logging

debug is for stdin - stdout tests. Here the last field is replaced by 'Debug'. And the input from the test is also written to the log file.
Logging is done to the default directory. It also uses config- and database files in the default directory.

Syslog

The software logs to syslog too;

Syslog entries
Log entryMeaning
namelookup.agi[PID]: connect from localhost (127.0.0.1) Connect to inetd
namelookup[PID]: Version: Version Number Start of namelookup.agi
namelookup[PID]: Lookup nr: Number Telephone number to be lookup up
namelookup[PID]: Name from VP: Name Name set by VOIP provider
namelookup[PID]: dblookup: Name Name or area found in database
namelookup[PID]: weblookup: Name Name found on web

Plus various errors that may occur.
'Timeout expired' usually means that the website lookup takes too long. Other errors are more serious.

Installation

Required

Download and install

Note: The compile of agiconf may 'complain' with errors;
'output may be truncated copying 2 bytes from a string of length 4095'
'output may be truncated copying 7 bytes from a string of length 4087'
This is OK. The software is supposed to do just that.

Next, as root run 'make install'. This runs install.sh

Target directories are created if they do no exist.
If you re-run install later, only newer versions of the above files are installed.

Configuration

Agiconf

Agiconf is a ncurses configuration tool. It can generate numbers.db, areas.db and regions.db. It's called from the install script, but you can (re)run it later if you want.

The main menu has the following items;

  1. Select Country
  2. Web Lookups
  3. Convert Phonebook
  4. Select Areas
  5. Select Regions

Select Country

This sets trunk prefix and international call prefix.

Web Lookups

If you are not interested in Dutch web lookups, you can disable them here.

Convert Phonebook

Convert numbers.tsv to numbers.db.
The software first looks for numbers.tsv in the default directory and then in your home directory.
Note: You can do this conversion later if you want.

Select Areas

The default behaviour is to just lookup countries. If you want to lookup cities and regions as well you can enable this on a per country basis or all if you like. So instead of 'Germany' it might say 'DE Düsseldorf'.
Note: With area code lookups enabled, the software will display the two letter ISO country code instead of the country name.
Note: If the software can't find the area code, it will display the country name instead of the city or region name.

Select Regions

This has an option called 'With federal states'. With this enabled, instead of 'USA' or 'Canada', it will display the federal state as well. E.G.: 'US New Jersey' or 'CA Manitoba'.

Agiconf HTML man page here.

Inetd

In /etc/services, under local services, add: 

agi		4573/tcp			# Asterisk Gateway Interface

In /etc/inetd.conf, add; 

agi	stream	tcp	nowait	asterisk	/usr/sbin/tcpd	/usr/local/sbin/namelookup.agi

Restart inetd. E.G.: 

~# /etc/init.d/xinetd reload

agi should now show up in a netstat; 

~$ netstat -a | grep agi
tcp6       0      0 [::]:agi                [::]:*                  LISTEN

You can further test things with telnet. As shown with the location lookup example below; 

~$ telnet localhost agi
Trying ::1...
Connected to localhost.
Escape character is '^]'.
agi_callerid: 0201234567

SET VARIABLE NAME "Lc Amsterdam"
Connection closed by foreign host.

You need to cut and paste ('agi_callerid: 0201234567') quickly because the program has a timeout of just a few seconds. And don't forget the blank line (an extra <Enter>). And after the AGI response, press <Enter> again. This will stop the program.
Note: The above example is for within the Netherlands. Outside the Netherlands the same lookup would include the NL country code (31): agi_callerid: 0031201234567 (assuming the international call prefix is '00').

Asterisk

If you have got all of the above to work, you can start configuring Asterisk.
In /etc/asterisk/extensions.conf: Just before the internal dial command that dials your phone(s) on incoming calls: 

	same => n,Set(NAME=${CALLERID(NAME)})
	same => n,Set(AGISIGHUP=no)
	same => n,AGI(agi://127.0.0.1)
	same => n,Set(CALLERID(NAME)=${NAME})

Variable NAME is set to the Caller-Id name.
'Set(AGISIGHUP=no)' keeps Asterisk from sending a 'HANGUP'.
If the lookup is successful, the AGI will send the string 'SET VARIABLE NAME' followed by the name.
This value is then assigned to Caller-Id name.

If you want to, you can combine this with the Privacy Manager; 

exten => Your_Extention,1,Answer()
	same => n,PrivacyManager()
	same => n,GotoIf($["${PRIVACYMGRSTATUS}" = "FAILED"]?pmfail:num)
	same => n(num),Set(NAME=${CALLERID(NAME)})
	same => n,Set(AGISIGHUP=no)
	same => n,AGI(agi://127.0.0.1)
	same => n,Set(CALLERID(NAME)=${NAME})
	same => n,Dial(Your_Internal_Dial_Command)
	same => n,Hangup()
	same => n(pmfail),Playback(im-sorry)
	same => n,Playback(vm-goodbye)
	same => n,Hangup()

Next, reload extensions.conf: 

asterisk -rx "dialplan reload"

Security

Firewall port 4573.
Limit access in /etc/hosts.allow and /etc/hosts.deny.

Log file rotation

In /etc/logrotate.d/asterisk-namelookup; 

/var/local/log/asterisk/namelookup.log {
        monthly
        missingok
        rotate 4
        compress
        delaycompress
        notifempty
        create 660 asterisk asterisk
}

Modify to suit your needs.

Phone book

Phone book format

Phone books come in all sorts of formats. There are two fields they always have in common: Name and Telephone number.
I use a phone book with the following format as a base for all other formats: 

Name <Tab> Telephone number <Line Feed>

This I then convert to whatever format is required for soft phones and SIP-phones.
The script 'ph2num.sh' converts the file 'phonebook.txt' with this format to; 

Telephone number <Tab> Name <Line Feed>

And saves this as 'numbers.tsv'. It also sets the sort order to Telephone number.
It then uses 'aginum2db' to convert this to binary format.
The binary phone book is saved as 'numbers.db'.
Below the file format. 

     0       1        2        3        4        5        6        7
   ┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
 0 │ Number                                                                │  7
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
 8 │                                                                       │ 15
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
16 │                            0      │ Length                            │ 23
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
24 │ Name                                                                  │ 31
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
32 │                                                                       │ 39
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
40 │                                                                       │ 47
   ├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
48 │                                                                0      │ 55
   └────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

Length is always zero in 'numbers.db'. And the length of the number in 'areas.db' and regions.db.
The software uses the length field to distinguish between 'numbers.db' and the other files.

Phone book contents

Normally a phone book is for dialling out. But this phone book is meant for recognising incoming phone calls.
A dial-out phone book may contain toll-free numbers.
A dial-in phone book needs to contain the phone numbers by which companies identify them selves instead.

Phone numbers are just digits: 0123456789. So no spaces, brackets, pluses or hyphens. Format is '0 area_code subscribers_number' or '00 country_code area_code subscribers_number'.
Example:

Country code: 12
Area code: 34
Subscribers NR: 56789

Phone number is: 03456789 or: 00123456789
Assuming of course that the trunk prefix is '0' and the international call prefix is '00'. Which is usually the case.

Names are UTF-8 text.

Sort order is ALPHABETIC sort of phone numbers, using the C.UTF-8 locale. So that's unsigned ascending byte value. Phone numbers have to be UNIQUE: They may occur only ONCE! The same name may occur multiple times though.
A phone number has one name.
A person may have more then one number.

Phone book installation

cp numbers.db /var/local/lib/phonebook/
cd /var/local/lib/phonebook/
chmod 640 numbers.db
chown root:asterisk numbers.db
Agiconf can do this for you.

Web lookups

For unrecognised numbers the program does a web lookup. If successful, the result gets saved in 'namelookup.log'. These are marked with the string WbLk (Web Lookup). You can add these to your phone book.

W3m configuration

W3m can be configured globally be editing /etc/w3m/config Alternatively, copy this file to /var/local/log/asterisk/.w3m/ and then edit this copy.

The following are the charset settings; 

display_charset UTF-8
document_charset UTF-8
system_charset UTF-8

And these are the proxy settings; 

use_proxy 1
http_proxy http://proxy.example.org:1234
https_proxy http://proxy.example.org:1234
gopher_proxy http://proxy.example.org:1234
ftp_proxy http://proxy.example.org:1234
no_proxy localhost,example.org

Disable Web lookups

If you want to you can DISABLE web lookups; Create a file /etc/agi-namelookup.conf with the following line; 

weblookup=off

Agiconf can do this for you.

Area and country code lookups

Format is '0 area_code' or '00 country_code area_code'.
Example:

Country code: 12
Area code: 34

Area code is: 034 or: 001234

Use the first format for area codes in your own country and the second for foreign area codes.
Assuming of course that the trunk prefix is '0' and the international call prefix is '00'. Which is usually the case.

For more information see: Asterisk location lookup AGI country and area codes