Freedom Internet Fibre optic link configuration

Introduction

This document describes using a Linux computer instead of a FritzBox as internet router.

Note: The default configuration files and scripts that come with most software packages assume that there already is a internet link up and running; In other words a router somewhere on the LAN.
This however, is about a DIY router. A setup that initiates and then maintains an internet link. This requires some tweaking of config files and scripts. Depending on the desired setup, you may or may not want to run some or all of the scripts in '/etc/ppp/ip-up.d/', '/etc/ppp/ip-down.d/' and /etc/wide-dhcpv6/, modify them, replace them or add your own.

Setup

Me

I have various Debian GNU Linux PCs hooked up to a gigabit Ethernet network.
In my setup, everything is static. There is no DHCP (at least not for NIC IP address configuration purposes) or routing daemon. Everything is configured by means of static config files; The resolvconf package is not installed: I run my own nameserver so /etc/resolv.conf points to my own server! And network-manager isn't installed either.
Freedom Internet emailed me my IPv4 address and IPv6 /48 prefix (both are static), which I then cut and paste to the various config files (convert upper case hex digits to lower case first).
I don't have systemd; I use a classic SysV boot (actually insserv).
The Ethernet interface for my LAN is 'eth0'. The link to my NTU (fibre media converter) uses 'eth1'. Both are gigabit Ethernet;

                                               ┌────────┐
                        ┌─────┐           eth1 │ Linux  │ eth0
  FttH <─>──────────────┤ NTU ├────────────────┤ Server ├────────────────<─> LAN
         Gbit           └─────┘ Gbit           │        │ Gbit               Other computers,
         Fibre Optic            Ethernet       └────────┘ Ethernet           SIP phone
         Cable                                                               and printer

I do not use the 'FritzBox'; The NTU (fibre media converter) is directly connected to the Linux server!
The Linux server acts as a internet router; It contains a firewall, a VoIP PBX, webserver, nameserver, mailserver, timeserver and a web proxy server.
Only a part of the IP addresses in the IPv6 /48 are used; I use some the first 256 addresses (/120) in the 2nd /64 range. In other to avoid NDP exhaustion attacks, the rest are firewalled; Results here.
Note: Newer systems may use different ethernet interface names such as enp3s0 and enp5s0. You may need to modify the stuff below accordingly.

Freedom Internet

For fibre optic- and VDSL(2) links, Freedom Internet uses PPPoE in IEEE 802.1Q VLAN 6.
Baby jumbo / rfc4638 is supported. So you can use an IP MTU of 1500, provided your NIC supports a MTU > 1500;

   ┌───────────────────────┐
   │    +VLAN hdr 1512     │
   │ ┌───────────────────┐ │
   │ │  PPPoE MTU 1508   │ │
   │ │ ┌───────────────┐ │ │
   │ │ │ IP   MTU 1500 │ │ │
   │ │ └───────────────┘ │ │
   │ └───────────────────┘ │
   └───────────────────────┘

Otherwise you need to reduce your IP MTU;

   ┌───────────────────────┐
   │    +VLAN hdr 1500     │
   │ ┌───────────────────┐ │
   │ │  PPPoE MTU 1496   │ │
   │ │ ┌───────────────┐ │ │
   │ │ │ IP   MTU 1488 │ │ │
   │ │ └───────────────┘ │ │
   │ └───────────────────┘ │
   └───────────────────────┘

To get Freedom Internet to route the IPv6 /48 to you, you need to do a DHCPv6 Prefix Delegation request. To this end the Debian package 'Wide-DHCPv6-Client' is installed.

Required packages

'Quirks'

PPPD and IPv6

  1. For IPv6 the PPPD uses IPv6 link-local addresses for the PPP link ends. Which actually makes sense, but may seem a bit odd at first.
  2. These addresses may be different each time the PPP link comes up.
  3. It does not necessarily set an IPv6 route the the remote end.
  4. It does not necessarily set an IPv6 default route, even if the config file says it should!

Wide-DHCPv6-Client

  1. dhcp6c won't start if the PPP link isn't up.
  2. dhcp6c is hard to kill.
  3. You need to remove the DUID file (/var/lib/dhcpv6/dhcp6c_duid) before a (re)start.
  4. dhcp6c doesn't necessarily know where to send config requests to. You need an IPv6 route to the outside world (IPv6 default route) for dhcp6c to work. Without this it will complain: 'client6_send: transmit failed: Network is unreachable'.

The DUID (DHCP Unique Identifier) contains the media type, a mac address and a timestamp in seconds since 00:00:00 Jan 1st 2000 UTC.

Config

Please note that I don't use systemd, which requires a different configuration!

Ethernet

Speed

The network card connected to the NTU needs to support 1 Gbps.

Modules

You may need to load the '8021q' module for VLAN. Just put it in /etc/modules or /etc/modules-load.d/local.conf;

8021q

This will load the module on boot.
You can also load the module manually with modeprobe;

~# modprobe 8021q

/etc/network/interfaces;

Note: This is pre Debian Stretch (9).
Newer stuff below.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
	address 192.168.1.1
	netmask 255.255.255.0
	network 192.168.1.0
	broadcast 192.168.1.255

# IPv6
iface eth0 inet6 static
	address 2001:db8:1234:1::1
	netmask 64

# Fibre optic link
auto eth1
iface eth1 inet manual
	mtu 1508

# VLAN 6
# VLAN number is derived from interface name
auto eth1.6
iface eth1.6 inet manual
	pre-down ( poff provider ) &
	up ip link set eth1.6 up
	mtu 1508
	down ifconfig eth1.6 down
	post-up ( pon provider ) &

If you stick to the 'ethx.y' naming scheme, where 'x' and 'y' are decimal numbers, you don't need a 'vlan-raw-device ethx' statement.
'manual' means that it doesn't get configured (IP address assigned to it) right now (or ever).
You need the 'up ip link set eth1.6 up' statement to actually make the interface go up. Otherwise it just gets configured without going up. If you want to check this: 'ifconfig -a' shows all interfaces, 'ifconfig' just the ones that are actually up.
'post-up ( pon provider ) &' actually starts the PPPD.

Note: One might think that the eth1 MTU needs to be 1512 (1500 + 8 + 4).
However, the eight extra bytes for PPPoE are part of the Ethernet payload and therefore get added to the MTU (1500 + 8 = 1508).
The four extra bytes for VLAN on the other hand, are part if the Ethernet header, not the payload (See IEEE 802.1Q).

Debian Stretch (9) and newer (2017+)

The processing of '/etc/network/interfaces' has become real fussy in Stretch, to the extent that I consider it broken. For instance, editing the file and then doing a '( ifdown eth0 ; sleep 1 ; ifup eth0 ) &' doesn't work anymore. What does work is changing things on the commandline or putting stuff in a script.
The setup above doesn't work on Debian Stretch either. Below a fix:

auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.1.1
	netmask 255.255.255.0
	network 192.168.1.0
	broadcast 192.168.1.255

# IPv6
iface eth0 inet6 static
	address 2001:db8:1234:1::1
	netmask 64
	pre-down /usr/local/sbin/wan-link stop
	post-up ( /usr/local/sbin/wan-link start )

Which calls '/usr/local/sbin/wan-link';

#!/bin/bash

check_syslog()
{
	CNT=0
	while [ ${CNT} -lt 10 ]
	do
		# Wait for syslogd
		if ( netstat -an | grep ":514" > /dev/null )
		then
			break
		fi
		sleep 1
		let CNT+=1
	done
}

case "$1" in
	start)
		check_syslog
		ifconfig eth1 mtu 1508 up
		vconfig add eth1 6
		ifconfig eth1.6 mtu 1508 up
		pon provider
		;;
	stop)
		poff provider
		ifconfig eth1.6 down
		ifconfig eth1 down
		;;
	*)
		echo "usage: wan-link {start|stop}"
		;;
esac

To make sure everything is logged, the scripts waits for the syslogdaemon.
Note: the 'check_syslog()' function expects a syslog daemon listening on port 514 (syslog)! If your syslogd doesn't, you have to check in some other way. A '( netstat -anp | grep syslog | grep "dev/log" > /dev/null )' might work.

Use IP

If you want to use 'ip' instead of 'ifconfig' and 'vconfig', the following will probably work;

ip link set dev eth1 mtu 1508
ip link set dev eth1 up
ip link add link eth1 name eth1.6 type vlan id 6
ip link set dev eth1.6 mtu 1508
ip link set dev eth1.6 up

Note: This hasn't been tested!

PPP

/etc/ppp/peers/provider

ifname wan
noipdefault
+ipv6 ipv6cp-use-ipaddr
defaultroute
connect /bin/true
noauth
persist
lcp-echo-interval 10
maxfail 0
debug
holdoff 10
mtu 1500
mru 1500
noaccomp
default-asyncmap
plugin rp-pppoe.so eth1.6
user "fake@freedom.nl"

'ifname wan' provides the PPP interface with an unique name.
If your network card doesn't support an MTU > 1500, you need to set the mtu and mru in 'provider' to 1488;

mtu 1488
mru 1488

You need the appropriate entries in /etc/ppp/pap-secrets as well.

DHCPv6

/etc/default/wide-dhcpv6-client

# Defaults for dhcpv6 client initscript
# Used by /etc/init.d/wide-dhcpv6-client

# Interfaces on which the client should send DHCPv6 requests and listen to
# answers. If empty, the client is deactivated.
INTERFACES="wan"

Interface 'wan' only exists after the link had gone up. The boot process might complain about that.

/etc/wide-dhcpv6/dhcp6c.conf

profile default
{
  script "/etc/wide-dhcpv6/dhcp6c-script";
};

interface wan {
    send ia-pd 0;
};

id-assoc pd 0 {
   prefix-interface eth1 {
       sla-len 16;
       sla-id 0;
       ifid 1;
   };
};

'sla' means 'Site-Level Aggregation identifier'. In a prefix '2001:db8:1234::/48' it's the '1234' bit. These are four hex digits, so it's 16 bits long (sla-len).
It insists on assigning an IP address to an interface. If you don't want this, keep the 'id-assoc pd' section in 'dhcp6c.conf' empty;

id-assoc pd 0 {
};

You can replace 'dhcp6c-script' with a script which doesn't do anything at all;

#!/bin/bash
exit 0

Note: The original 'dhcp6c-script' messes with '/etc/resolv.conf', which you don't want if you run your own nameserver!

Scripts

/etc/ppp/ipv6-up.d/add-rem-rt

#!/bin/sh

# Sets route to remote IPv6 address of PPP link
if ! ( ip -6 route | grep -q "${PPP_REMOTE}" )
then
	ip -6 route add "${PPP_REMOTE}" dev "${PPP_IFACE}"
fi

# Set IPv6 default route 
if ! ( ip -6 route | grep -q default )
then
	ip -6 route add default dev wan
fi

Called from /etc/ppp/ip-up

I don't use the regular scripts in '/etc/ppp/ip-up.d/' and '/etc/ppp/ip-down.d/'. I modified '/etc/ppp/ip-up' instead. You may need to modify the stuff below to suit your needs.

# Take care of the (default) route(s)
case $PPP_LOCAL in
	"Your_Freedom_Internet_IPv4_address")
	# Freedom iface
	# Store remote IPv4 address if you want to; '/run/ppp/' needs to exist.
	echo $5 > "/run/ppp/wan-rem-ip"
	chmod a+r "/run/ppp/wan-rem-ip"
	# Restart NTPD
	/etc/init.d/ntp restart 2>&1 | mail -s "NTPD restarted" root@Your_Domain
	# IPv6 stuff
	# Make sure link works
	ping -c5 $PPP_REMOTE
	# Default route
	if ! ( ip -6 route | grep -q default )
	then
                ip -6 route add default dev wan
	fi
	# Restart DHCPv6
	( /etc/init.d/wide-dhcpv6-client stop ) &
	sleep 2
	killall dhcp6c
	sleep 2
	killall -9 dhcp6c
	sleep 2
	rm /var/lib/dhcpv6/dhcp6c_duid
	sleep 2
	/etc/init.d/wide-dhcpv6-client start

	;;
esac

Notes

IPv6 uses IPv6 link-local (fe80::/10) - and IPv6 multicast addresses (ff02::/16) for internal housekeeping and configuration. If you firewall these too heavy, some things will not work.
It can be handy to enable debugging. In '/etc/init.d/wide-dhcpv6-client' add '-D' (shown red);

case "$1" in
	start)
		log_daemon_msg "Starting $DESC" "$NAME"
		start-stop-daemon --start --quiet --pidfile $DHCP6CPID \
			--oknodo --exec $DHCP6CBIN -- -D -Pdefault $INTERFACES
		sleep 2

With Debian Stretch you can set the debug level in /etc/default/wide-dhcpv6-client (shown red);

# Defaults for dhcpv6 client initscript
# Used by /etc/init.d/wide-dhcpv6-client

# Interfaces on which the client should send DHCPv6 requests and listen to
# answers. If empty, the client is deactivated.
INTERFACES="wan"

# Verbose level for syslog. Default is 0 (0: minimal; 1: info; 2: debug)
VERBOSE=2

If things don't work, have a look at '/var/log/syslog'.

Check the MTU with ping

With these you can check your MTU.
'-s' sets the payload size.
The '-M do' bit disables fragmentation.

IPv4

An IPv4 ping header is 28 bytes (1472 + 28 = 1500);

~$ ping4 -s 1472 -M do Some_Host

IPv6

An IPv6 ping header is 48 bytes (1452 + 48 = 1500);

~$ ping -s 1452 -M do Some_Host

Additional notes on booting

A boot might 'hang' with considerable timeouts on the following;

  1. All interfaces need to be up.
  2. All NFS volumes need to be mounted.

The 1st I fixed by calling the PPP-link from /etc/network/interfaces instead of putting the PPP interface directly in /etc/network/interfaces. If the PPP-link doesn't come up, all the services are still started. Alternatively you might want to try 'allow-hotplug' instead of auto for the wan interface (This is based on experiences from the 1990-ies. Things might be different now).

The 2nd I fixed by using IP addresses instead of hostnames for NFS mounts in /etc/fstab: If the nameserver is down, the remote discs are still mounted. I also mount in the background (bg option in /etc/fstab). This way if a remote disc can't be mounted, the boot still proceeds.

Debug examples

I my current setip, 'enp5s0' (was eth0) is the LAN interface, 'enp3s0' (was eth1) the link to the NTU. 'enp3s0.6' is the VLAN 6 interface.

Tcpdump

This shows the beginning of the PPoE negotiation. The first line is my system sending data to the ethernet broadcast address, 'ff:ff:ff:ff:ff:ff'.

root@sput:~#  tcpdump -n -e -vvv -s1500 -i enp3s0.6 -U
tcpdump: listening on enp3s0.6, link-type EN10MB (Ethernet), capture size 1500 bytes
13:49:00.367689 Local_Ethrn_Addrs > ff:ff:ff:ff:ff:ff, ethertype PPPoE D (0x8863), length 30: PPPoE PADI [Service-Name] [PPP-Max-Payload 0x05DC]
13:49:00.376422 Remote_Ethr_Addrs > Local_Ethrn_Addrs, ethertype PPPoE D (0x8863), length 63: PPPoE PADO [PPP-Max-Payload 0x05DC] [AC-Name "asd-dc2-rtr1.cambrium.net"] [Service-Name] [EOL]
13:49:00.376559 Local_Ethrn_Addrs > Remote_Ethr_Addrs, ethertype PPPoE D (0x8863), length 30: PPPoE PADR [Service-Name] [PPP-Max-Payload 0x05DC]
13:49:00.396705 Remote_Ethr_Addrs > Local_Ethrn_Addrs, ethertype PPPoE D (0x8863), length 63: PPPoE PADS [ses 0x29] [Service-Name] [PPP-Max-Payload 0x05DC] [AC-Name "asd-dc2-rtr1.cambrium.net"] [EOL]
13:49:00.411982 Local_Ethrn_Addrs > Remote_Ethr_Addrs, ethertype PPPoE S (0x8864), length 32: PPPoE  [ses 0x29] LCP (0xc021), length 12: LCP, Conf-Request (0x01), id 1, length 12

Debug in syslog

This is from /var/log/syslog with PPP debug on. It shows the first bit of the PPPoE negotiation. The rest is no different from any other PPP link setup.

Nov 21 13:47:40 sput pppd[12799]: pppd 2.4.7 started by root, uid 0
Nov 21 13:47:40 sput pppd[12799]: Send PPPOE Discovery V1T1 PADI session 0x0 length 10
Nov 21 13:47:40 sput pppd[12799]:  dst ff:ff:ff:ff:ff:ff  src Local_Ethrn_Addrs
Nov 21 13:47:40 sput pppd[12799]:  [service-name] [PPP-max-payload  05 dc]
Nov 21 13:47:40 sput pppd[12799]: Recv PPPOE Discovery V1T1 PADO session 0x0 length 43
Nov 21 13:47:40 sput pppd[12799]:  dst Local_Ethrn_Addrs  src Remote_Ethr_Addrs
Nov 21 13:47:40 sput pppd[12799]:  [PPP-max-payload  05 dc] [AC-name asd-dc2-rtr1.cambrium.net] [service-name] [end-of-list]
Nov 21 13:47:40 sput pppd[12799]: Send PPPOE Discovery V1T1 PADR session 0x0 length 10
Nov 21 13:47:40 sput pppd[12799]:  dst Remote_Ethr_Addrs  src Local_Ethrn_Addrs
Nov 21 13:47:40 sput pppd[12799]:  [service-name] [PPP-max-payload  05 dc]
Nov 21 13:47:40 sput pppd[12799]: Recv PPPOE Discovery V1T1 PADS session 0x2008 length 43
Nov 21 13:47:40 sput pppd[12799]:  dst Local_Ethrn_Addrs  src Remote_Ethr_Addrs
Nov 21 13:47:40 sput pppd[12799]:  [service-name] [PPP-max-payload  05 dc] [AC-name asd-dc2-rtr1.cambrium.net] [end-of-list]
Nov 21 13:47:40 sput pppd[12799]: PADS: Service-Name: ''
Nov 21 13:47:40 sput pppd[12799]: PPP session is 8200
Nov 21 13:47:40 sput pppd[12799]: Connected to Remote_Ethr_Addrs via interface enp3s0.6
Nov 21 13:47:40 sput pppd[12799]: using channel 5
Nov 21 13:47:40 sput kernel: [4829498.873711] wan: renamed from ppp0
Nov 21 13:47:40 sput pppd[12799]: Renamed interface ppp0 to wan
Nov 21 13:47:40 sput pppd[12799]: Using interface wan
Nov 21 13:47:40 sput pppd[12799]: Connect: wan <--> enp3s0.6