junkview project

summary

A suite of awk and bash scripts to analyse iptables firewall log files and summarise data in a plaintext presentation. See: junkshow to view a summary of this machine's firewall activity.

See the junkview page for source code. The junkview suite uses bash and gawk — nothing to compile, no tricky dependencies to resolve. Requires GNU/Linux system with iptables firewall. Developed on a Slackware-10.2 based system.

why junkview?

Why yet another log file analyser? Couple years ago I switched from dialup to 24/7 ADSL Internet connection. As part of analysing firewall activity I would write ad hoc filters on the command line with grep, cut, sort, uniq and friends. After some time this paled, rewrite log analyser in awk and present the output in a simple plaintext form. The reasons for plaintext? Ability to copy / paste IPs from the report, and it is a 'low rent' solution suitable for running on an older box, development system is a pentium-mmx/233 with 128MB memory.

iptables integration: junkview is intended for use with iptables firewall, the firewall logging mechanism is used to transfer information about firewall activity to junkview. Therefore, junkview presents information in context of the user's own system.
plaintext output: junkview produces a plaintext output format restricted to 80 columns. The output may be included in web pages or emailed.
IP allocation blocks: junkview groups offending IP addresses by network within the allocated blocks, as well as detect IP addresses from unallocated blocks.
smart summary: junkview summarises the log information, producing a concise report highlighting the most frequent attacks. Summarising over longer periods of time may highlight stealthy attacks such as those IPs probing at a slow rate or portknocking.

linking iptables to junkview

The development firewall runs linux-kernel 2.4.latest and junkview analyses iptables via syslogd and /var/log/messages. When writing iptables rules one uses a common --log-prefix for messages to be reported by junkview. For example, from /etc/rc.d/rc.firewall, to log events that were not explicitly handled:

...
PREFIX="JLE"                    # prefix for Junkview Log Entry
LOGGED="LOG --log-level info --log-prefix " # -> /var/log/messages
...
	iptables -A INPUT -p all -j $LOGGED "$PREFIX:inp:drop policy "

This results in an event recorded in /var/log/messages looking like (IPs masked to protect the guilty parties ;)

Jun 15 08:19:28 deltree kernel: JLE:inp:drop policy IN=ppp0 OUT= MAC= 
  SRC=XXX.XX.XX.XXX DST=XXX.XXX.XXX.XXX LEN=931 TOS=0x00 PREC=0x20 TTL=48
  ID=0 DF PROTO=UDP SPT=55658 DPT=1027 LEN=911

prefix first word

The colon separated fields within the first prefix word are part of new development and may be ignored for the present discussion. Choose a prefix not used by any other kernel process, discover reserved words words with something like:

~# grep -v JLE /var/log/messages|awk '{print $5}'\
	|sed 's/\[[0-9]*\]//;s/://g'|sort|uniq
dnsmasq
kernel
mountd
ntpd
pppd
pppoe
pppoe-connect
sshd
syslogd

prefix second word

The second word is used as an index to a lookup table for the Classify junk summary. The lookup table:

$ cat data/classify-text
junkview classify 1.00 -- for deltree
  calm_rpt:drop    web traffic calming
  deny_all:drop    explicit deny access
  deny_tcp:reject  explicit deny access
    policy:drop    junk, other
  sshd_new:drop    ssh login attempt, new ip
  sshd_rpt:drop    ssh login attempt, repeat
 tcpreject:reject  junk, tcp
  wbot_rpt:drop    web crawler calming

produces a junkview summary like this:

Classify junk:
   342  drop    junk, other
   210  reject  junk, tcp
    26  drop    web crawler calming
    23  drop    explicit deny access
    18  drop    web traffic calming
     1  drop    ssh login attempt, new ip

project status

2006-09-10

Project history for last several months is on the download page.

2006-05-31

Very minor changes during the last several weeks. One area of change is the logging --log-prefix format — several methods have been trialled in order to meet demands of reporting, as well as allow for future extension of junkview to real-time monitoring.

2006-04-07

Standalone operation is back! Plus, if you tell junkview to use ip2c-server, and the server is not running, junkview will switch to standalone operation. Loading the data files takes about ten seconds on the test box.

Check new layout of /etc/junkview.conf as this one file controls much junkview and the database update scripts.

Grab the latest tarball from the archive.

install file tree

source tree defined (updated 2006-09-10):

.
|-- README                              # junkview project information
|-- bin
|   |-- ccfind                          # ip2cn-server command line client
|   |-- ip2cn-server                    # optional junkview database server
|   |-- junkshow                        # example: cron job script
|   |-- junkview                        # junkview log file analysis program
|   |-- junkview-update-database        # download/process data source file
|   `-- pre-filter                      # multiple logfile record streamer
|-- data
|   |-- classify-text                   # example: classify text lookup table
|   |-- ip2c-data                       # processed ip2c database
|   `-- ip2c-names                      # hand crafted country names lookup
|-- etc
|   `-- junkview.conf                   # junkview configuration defaults
|-- install                             # junkview suite installer script
`-- rc.d
    `-- rc.junkview                     # ip2cn-server start|stop|restart|reload

about the author

Grant is a retired electronics design engineer who spent much of his career working with 8-bit microcontrollers, both hardware and software design. Real-time operating system for event driven 24/7 processing, mainly for security monitoring equipment, also industrial measurement and control systems for diverse applications.

Grant completed B.Computing from La Trobe University, Bendigo in 2001, he has been working with GNU/Linux since 1997.