How can I analyze line rate taking log file as input on a Linux system? How do I find the IP flooding my Apache/Nginx/Lighttpd web-server on a Debian or Ubuntu Linux?
You need to use a tool called logtop. It is a system administrator tool to analyze line rate taking log file as input. It reads on stdin and print a constantly updated result displaying, in columns in the following format:
Line number, count, frequency, and the actual line
How do install logtop on a Debian or Ubuntu based system?
Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: logtop 0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded. Need to get 15.7 kB of archives. After this operation, 81.9 kB of additional disk space will be used. Get:1 http://mirrors.service.networklayer.com/ubuntu/ precise/universe logtop amd64 0.3-1 [15.7 kB] Fetched 15.7 kB in 0s (0 B/s) Selecting previously unselected package logtop. (Reading database ... 114954 files and directories currently installed.) Unpacking logtop (from .../logtop_0.3-1_amd64.deb) ... Processing triggers for man-db ... Setting up logtop (0.3-1) ...
The syntax is as follows:
logtop [OPTIONS] [FILE] command | logtop command1 | filter | logtop command1 | filter | logtop [options] [file]
Here are some common examples of logtop.
Show the IP address flooding your LAMP server
Type the following command:
tail -f www.cyberciti.biz_access.log | cut -d' ' -f1 | logtop
See squid cache HIT and MISS log
tail -f cache.log | grep -o "HIT|MISS" | logtop
To see realtime hit / miss ratio on some caching software log file, enter:
tail -f access.log | cut -d' ' -f1 | logtop -s 20000
The -s option set logtop to work with the maximum of K lines instead of 10000.