softflow is a great opensource Netflow exporter that is support on OperWrt :) This allows us to sample the flows running through our OpenWrt router, export them to a netflow analyzer (more on those in a future post) and "see" our traffic. Netflow is much richer in information than just SNMP and allows for detailed analysis of individual traffic flows.

Install softflowd

SSH to the router, update package lists and install with opkg install softflowd

BusyBox v1.30.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.2, r10947-65030d81f3
 -----------------------------------------------------
root@Them00n:~# opkg update
Downloading http://downloads.openwrt.org/releases/19.07.2/targets/lantiq/xrx200/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_core
Downloading http://downloads.openwrt.org/releases/19.07.2/targets/lantiq/xrx200/packages/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.2/targets/lantiq/xrx200/kmods/4.14.171-1-0f59e90218b95a909e229a713d3da157/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_kmods
Downloading http://downloads.openwrt.org/releases/19.07.2/targets/lantiq/xrx200/kmods/4.14.171-1-0f59e90218b95a909e229a713d3da157/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/base/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_base
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/base/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/luci/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_luci
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/luci/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_packages
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/packages/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/routing/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_routing
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/routing/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/telephony/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_telephony
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/telephony/Packages.sig
Signature check passed.

root@Them00n:~# opkg install softflowd
Installing softflowd (0.9.9-2) to root...
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/packages/softflowd_0.9.9-2_mips_24kc.ipk
Configuring softflowd.
root@Them00n:~#

Configue softflowd

softflowd by default reads its config from the file /etc/config/softflowd. As with most applications on OpenWrt we can either manually edit this file or use UCI. We'll look at the options for both below but just use the option you prefer (I prefer UCI since it does some sanity checking).
You can add multiple interfaces to be monitored and you can also monitor any interface that appears in the output of ifconfig. I have two interfaces to monitor so the examples below include two, just modify the examples to suit your own setup.
Some details you'll need to know before you can configure softflowd though:

  • Interface(s) to monitor - (e.g pick from ifconfig)
  • Netflow collector IP address and port(s)
  • Netflow collector netflow version support - If you're not sure I'd suggest trying v9 these days
  • Check CPU load before and after enabling softflow! (This has a minimal impact on my systems but this will add some load)
  • The sampling rate you would like to set - Default is 1 in 100 packets and I'd suggest leaving it around that level

Configure using UCI

To configure softflowd using uci just replace your collector IP and interfaces below. Also remove the references to softflow[1] if you dont have more than one interface.

uci set softflowd.@softflowd[0].enabled='1'
uci set softflowd.@softflowd[0].interface='pppoe-wan'
uci set softflowd.@softflowd[0].timeout='maxlife=60'
uci set softflowd.@softflowd[0].max_flows='8192'
uci set softflowd.@softflowd[0].host_port='10.1.1.1:9995'
uci set softflowd.@softflowd[0].pid_file='/var/run/softflowd.pid'
uci set softflowd.@softflowd[0].control_socket='/var/run/softflowd.ctl'
uci set softflowd.@softflowd[0].export_version='9'
uci set softflowd.@softflowd[0].tracking_level='full'
uci set softflowd.@softflowd[0].track_ipv6='1'
uci set softflowd.@softflowd[0].sampling_rate='100'

If you need more than one interface monitored just add multiple versions of the below and modify the [1] accordingly. So my second interface:

uci add softflowd softflowd
uci set softflowd.@softflowd[1].enabled='1'
uci set softflowd.@softflowd[1].interface='br-iot'
uci set softflowd.@softflowd[1].timeout='maxlife=60'
uci set softflowd.@softflowd[1].max_flows='8192'
uci set softflowd.@softflowd[1].host_port='10.1.1.1:9996'
uci set softflowd.@softflowd[1].pid_file='/var/run/softflowd.pid'
uci set softflowd.@softflowd[1].control_socket='/var/run/softflowd.ctl'
uci set softflowd.@softflowd[1].export_version='9'
uci set softflowd.@softflowd[1].tracking_level='full'
uci set softflowd.@softflowd[1].track_ipv6='1'
uci set softflowd.@softflowd[1].sampling_rate='100'

Commit the changes using uci commit and restart the service using /etc/init.d/softflowd restart. Config done :) See below for how to confirm its working.

Configured editing config file

To do this simply edit /etc/config/softflowd directly and edit it similar to the below:

config softflowd
        option enabled '1'
        option interface 'pppoe-wan'
        option timeout 'maxlife=60'
        option max_flows '8192'
        option host_port '10.1.1.1:9995'
        option pid_file '/var/run/softflowd.pid'
        option control_socket '/var/run/softflowd.ctl'
        option export_version '9'
        option tracking_level 'full'
        option track_ipv6 '1'
        option sampling_rate '100'

config softflowd
        option enabled '1'
        option interface 'br-iot'
        option timeout 'maxlife=60'
        option max_flows '8192'
        option host_port '10.1.1.1:9996'
        option pid_file '/var/run/softflowd.pid'
        option control_socket '/var/run/softflowd.ctl'
        option export_version '9'
        option tracking_level 'full'
        option track_ipv6 '1'
        option sampling_rate '100'

Save the file and restart the service using /etc/init.d/softflowd restart. Config done :) See below for how to confirm its working.

Verify it works

softflowd has a sister application called softflowctl we can use to analyze the currently cached flows in softflowd. Just run softflowctl dump-flows.

root@Them00n:~# softflowctl dump-flows
softflowd[11100]: Dumping flow data:
ACTIVE seq:24 [1.2.3.4]:53592 <> [10.11.12.13]:443 proto:6 octets>:208 packets>:4 octets<:825 packets<:2 start:2020-04-05T09:38:39.821 finish:2020-04-05T09:39:37.483 tcp>:10 tcp<:18 flowlabel>:00000000 flowlabel<:00000000
EXPIRY EVENT for flow 24 in 1 seconds

ACTIVE seq:25 [1.2.3.4]:47916 <> [6.7.8.9]:443 proto:6 octets>:104 packets>:2 octets<:4476 packets<:3 start:2020-04-05T09:38:46.596 finish:2020-04-05T09:38:46.711 tcp>:10 tcp<:10 flowlabel>:00000000 flowlabel<:00000000
EXPIRY EVENT for flow 25 in 8 seconds

ACTIVE seq:26 [1.2.3.4]:47768 <> [6.7.8.9]:443 proto:6 octets>:936 packets>:18 octets<:88028 packets<:59 start:2020-04-05T09:38:49.294 finish:2020-04-05T09:39:37.531 tcp>:10 tcp<:10 flowlabel>:00000000 flowlabel<:00000000
EXPIRY EVENT for flow 26 in 11 seconds

ACTIVE seq:27 [1.2.3.4]:59374 <> [14.15.16.17]:80 proto:6 octets>:0 packets>:0 octets<:52 packets<:1 start:2020-04-05T09:39:09.036 finish:2020-04-05T09:39:09.036 tcp>:00 tcp<:10 flowlabel>:00000000 flowlabel<:00000000
EXPIRY EVENT for flow 27 in 31 seconds

Netflow version 9 sends a template every so often (I think ~ every 30mins or so) to describe the structured of the exported flows. If your analyzer hasnt received this you can force softflowd to include the template in the next update with softflowctl send-template.

root@Them00n:~# softflowctl send-template
softflowd[11100]: Template will be sent at next flow export

You can also view the statistics of the application using softflowctl statistics

root@Them00n:~# softflowctl statistics
softflowd[11100]: Accumulated statistics since 2020-04-05T09:35:42 UTC:
Number of active flows: 10
Packets processed: 4896
Packets non-sampled: 484646
Fragments: 0
Ignored packets: 0 (0 non-IP, 0 too short)
Flows expired: 567 (0 forced)
Flows exported: 593 in 133 packets (18 failures)
Packets received by libpcap: 489624
Packets dropped by libpcap: 0
Packets dropped by interface: 0

Expired flow statistics:  minimum       average       maximum
  Flow bytes:                  36          9768        119091
  Flow packets:                 1             9           105
  Duration:                  0.00s        15.61s        81.51s

Expired flow reasons:
       tcp =       398   tcp.rst =         5   tcp.fin =         0
       udp =        44      icmp =         9   general =         0
   maxlife =       111
over 2 GiB =         0
  maxflows =         0
   flushed =         0

Per-protocol statistics:     Octets      Packets   Avg Life    Max Life
           icmp (1):            686            9       0.00s       0.00s
            tcp (6):        5473973         4746      17.06s      81.51s
           udp (17):          63547           91       1.82s      48.37s
root@Them00n:~#

The final (and as ever best) proof we're actually exporting flows can be done with tcpdump. If its not installed already, install it and listen for the flows being exported:

root@Them00n:~# opkg install tcpdump 
Installing tcpdump (4.9.3-1) to root...
Downloading http://downloads.openwrt.org/releases/19.07.2/packages/mips_24kc/base/tcpdump_4.9.3-1_mips_24kc.ipk
Configuring tcpdump.
root@Them00n:~# tcpdump -i br-lan port 9995 or port 9996
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
10:31:06.421755 IP Netflow.home.45100 > OpenWrt.home.9995: UDP, length 264
10:31:26.224629 IP Netflow.home.45100 > OpenWrt.home.9995: UDP, length 168
10:32:01.154466 IP Netflow.home.45100 > OpenWrt.home.9995: UDP, length 72
10:32:01.206544 IP Netflow.home.59400 > OpenWrt.home.9996: UDP, length 328
10:32:10.522646 IP Netflow.home.45100 > OpenWrt.home.9995: UDP, length 216

There we go job done :)

m00nie

More detailed info can be found on these links: