Difference between revisions of "HOWTO use geoiplookup"

From Fail2ban
Jump to: navigation, search
m (Reverted edits by Henryut (talk) to last revision by 90.231.68.5)
 
(33 intermediate revisions by 21 users not shown)
Line 1: Line 1:
 
== Geolocalization of banned IPs ==
 
== Geolocalization of banned IPs ==
You may be interested in a quick summary of the countries where the attacks come from. This document explains how to find these information.  
+
You may be interested in a quick summary of the countries where the attacks come from. This document explains how to find these information.
  
 
== Requirements==
 
== Requirements==
Line 23: Line 23:
  
 
== Script ==
 
== Script ==
This small script will extract the banned IPs from fail2ban.log. It looks for lines such as "..... Ban 192.168.1.1", extracts the IP and runs geoiplookup. You may have to change the hardcoded paths in the script depending on your configuration.
+
This small script will extract the banned IPs from fail2ban.log. It looks for lines such as "..... Ban 192.168.1.1", extracts the IP and runs geoiplookup. You may have to change the hardcoded <span class="plainlinks">[http://thebeginnerslens.com/top-iphone-camera-apps-of-the-month <span style="color:black;font-weight:normal; text-decoration:none!important; background:none!important; text-decoration:none;">iphone camera apps</span>] paths in the script depending on your <span class="plainlinks">[http://xTiburon.com <span style="color:black;font-weight:normal; text-decoration:none!important; background:none!important; text-decoration:none;">business blog</span>] configuration.
 +
 
 +
 
  
 
<pre>
 
<pre>
Line 42: Line 44:
 
</pre>
 
</pre>
  
There is a package of GeoIP bindings for Python available as well.  The following script performs a similar function using those bindings, plus it works on Fedora and any other distro where fail2ban outputs by the <span class="plainlinks">[http://c-c.com.au/ <span style="color:black;font-weight:normal; text-decoration:none!important; background:none!important; text-decoration:none;">Brisbane Website Designers</span>] to syslog:
+
There is a package of GeoIP bindings for Python available as well.  The following script performs a similar function using those bindings, plus it works on Fedora and any other distro where fail2ban outputs to syslog:
 
<pre>
 
<pre>
 
<nowiki>
 
<nowiki>
Line 97: Line 99:
 
This will log output such as
 
This will log output such as
 
  2008-04-27 20:18:03,109 fail2ban.actions: WARNING [ssh] Ban 256.256.256.256 US, United States
 
  2008-04-27 20:18:03,109 fail2ban.actions: WARNING [ssh] Ban 256.256.256.256 US, United States
If you only want the two character country code, change the line to
+
If you only want the two character country code, change the line to  
  logSys.warn("[%s] Ban %s %s" % (self.jail.getName(), aInfo["ip"], commands.getstatusoutput('geoiplookup ' + aInfo["ip"])[1][23:'''25''']))
+
  logSys.warn("[%s] Ban <span class="plainlinks">[http://www.smslana.eu <span style="color:black;font-weight:normal; text-decoration:none!important; background:none!important; text-decoration:none;">Smslån blogg</span>] %s %s" % (self.jail.getName(), aInfo["ip"], commands.getstatusoutput('geoiplookup ' + aInfo["ip"])[1][23:'''25''']))
  
  
Line 104: Line 106:
 
For advanced results, you may be interested in :  
 
For advanced results, you may be interested in :  
  
* http://www.geobytes.com/
+
* http://www.maxmind.com/
 +
 
 +
Other databases City/Country/ASNum/CountryV6/CityV6/...
 +
 
 +
* http://geolite.maxmind.com/download/geoip/database/
  
 
== Updated GeoIP database ==
 
== Updated GeoIP database ==
As the geoiplookup database will be pretty outdated (the current included version are from 20060501 in stable debian) you might want to update it regularly as IP assignment changes. One way of doing that is to use a crontab entry that downloads the updated version from maxmind and untar's it to the correct position. I use something like this [http://www.autoinsurancequoteseasy.com/ <span style="color:black;font-weight:normal; text-decoration:none!important; background:none!important; text-decoration:none;">quote</span>]
+
As the geoiplookup database will be pretty outdated (the current included version are from 20060501 in stable debian) you might want to update it regularly as IP assignment changes. One way of doing that is to use a crontab
 
  wget -q http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoIP.dat.new && mv /usr/share/GeoIP/GeoIP.dat.new /usr/share/GeoIP/GeoIP.dat
 
  wget -q http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoIP.dat.new && mv /usr/share/GeoIP/GeoIP.dat.new /usr/share/GeoIP/GeoIP.dat
 +
===Update To URLS===
 +
The urls seem to have moved somewhat.
 +
<pre>
 +
wget -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoCity.dat.new && mv /usr/share/GeoIP/GeoIP/GeoCity.dat.new /usr/share/GeoIP/GeoCity.dat</pre>
 +
 +
<pre>wget -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoIP.dat.new && mv /usr/share/GeoIP/GeoIP.dat.new /usr/share/GeoIP/GeoIP.dat</pre>
 +
  
 
And in /etc/crontab it could look like this:
 
And in /etc/crontab it could look like this:
 
  @monthly root sleep $[$RANDOM/1024]; wget -q http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoIP.dat.new && mv /usr/share/GeoIP/GeoIP.dat.new /usr/share/GeoIP/GeoIP.dat
 
  @monthly root sleep $[$RANDOM/1024]; wget -q http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoIP.dat.new && mv /usr/share/GeoIP/GeoIP.dat.new /usr/share/GeoIP/GeoIP.dat
 +
  
 
The above command means:
 
The above command means:
Line 117: Line 131:
 
  root = as user root
 
  root = as user root
 
  sleep $[$RANDOM/1024]; = sleep for a random time (so all you guys don't DDoS maxminds server every month at the same time)
 
  sleep $[$RANDOM/1024]; = sleep for a random time (so all you guys don't DDoS maxminds server every month at the same time)
  wget ... | gunzip ... && ... = wget and pipe to gunzip, if that succeeds, move the file to the correct place [http://www.vasundharafertility.com/pregnancy.php high risk pregnancy]all about it
+
  wget ... <span class="plainlinks">[http://www.teethwhiteningathomev.com <span style="color:black;font-weight:normal; text-decoration:none!important; background:none!important; text-decoration:none;">Teeth whitening</span>] | gunzip ... && ... = wget and pipe to gunzip, if that succeeds, move the file to the correct place
[http://www.diamondlinks.net link building service]
+
 
* [http://www.thegallbladdersymptoms.com/ Gallbladder symptoms]
+
===Update To CRON===
* [http://www.pacquiaomosleyfight.com/ pacquiao vs marquez]
+
Add this script to cron.monthly
* <span class="plainlinks">[http://www.gallbladderdetox.com<span style="color:black;font-weight:normal; text-decoration:none!important; background:none!important; text-decoration:none;">gallbladder removal</span>]
+
 
* [http://www.monarch-garments.com/embroideredshirts.asp embroidered shirts]
+
<pre>#!/bin/bash
* [http://www.khanvsjudahlive.co.cc/ khan vs judah live streaming]
+
wget -q http://geolite.maxmind.com/download/geoip/database/Ge
* [http://www.blogcommentinglinkbuilding.com blog commenting service]
+
oLiteCity.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoCity.dat.
 +
new && mv /usr/share/GeoIP/GeoCity.dat.new /usr/share/GeoIP/G
 +
eoCity.dat
 +
wget -q http://geolite.maxmind.com/download/geoip/database/Ge
 +
oLiteCountry/GeoIP.dat.gz -O - |gunzip > /usr/share/GeoIP/Geo
 +
IP.dat.new && mv /usr/share/GeoIP/GeoIP.dat.new /usr/share/Ge
 +
oIP/GeoIP.dat</pre>

Latest revision as of 10:52, 6 November 2011

Geolocalization of banned IPs

You may be interested in a quick summary of the countries where the attacks come from. This document explains how to find these information.

Requirements

In Gentoo, the needed package is the following :

dev-libs/geoip
     Latest version available: 1.3.14
     Latest version installed: [ Not Installed ]
     Size of downloaded files: 1,984 kB
     Homepage:    http://www.maxmind.com/geoip/api/c.shtml
     Description: easily lookup countries by IP addresses, even when Reverse DNS entries don't exist
     License:     GPL-2

This will install "geoiplookup" and "geoipupdate" to update the database (you need a license id to get a new db)

In Debian or Ubuntu, one can simple do apt-get install geoip-bin

In Fedora, you can install with this command:
pkcon install GeoIP

Script

This small script will extract the banned IPs from fail2ban.log. It looks for lines such as "..... Ban 192.168.1.1", extracts the IP and runs geoiplookup. You may have to change the hardcoded iphone camera apps paths in the script depending on your business blog configuration.



# Fail2BanGeo.py
import os
import re
f = open('/var/log/fail2ban.log', 'r')    
pattern = r".*?Ban\s*?((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))$"
p = re.compile(pattern)
for i in f:
        m = p.match(i)
        if m:
                ip = m.group(1)
                file = os.popen('geoiplookup %s' % ip)
                print file.read()

There is a package of GeoIP bindings for Python available as well. The following script performs a similar function using those bindings, plus it works on Fedora and any other distro where fail2ban outputs to syslog:


#!/usr/bin/env python
# Fail2BanGeo.py, improved
import os

import GeoIP
import pyparsing as pp

log_path = '/var/log/fail2ban.log'
if os.path.exists(log_path):
    log = open(log_path, 'r')
else:
    log = open('/var/log/messages', 'r')

geo = GeoIP.new(GeoIP.GEOIP_MEMORY_CACHE)
octet = pp.Word(pp.nums, min=1, max=3)
ip_matcher = pp.Combine(octet + ('.' + octet) * 3)

for line in log:
    if 'fail2ban' in line and 'Ban' in line:
        match = ip_matcher.searchString(line)
        if match:
            ip = match.pop()[0]
            code = geo.country_code_by_addr(ip)
            name = geo.country_name_by_addr(ip)
            if not code or not name:
                print "No GeoIP info for IP %s." % ip
            else:
                print "GeoIP info for %s:\t%s, %s" % (ip, code, name)

Output

myserver # python fail2bangeo.py
GeoIP Country Edition: CI, Cote D'Ivoire

GeoIP Country Edition: FR, France

GeoIP Country Edition: CN, China

GeoIP Country Edition: KO, South Korea

GeoIP Country Edition: VN, Vietnam

Logging

You can also change the fail2ban script to write the country code to the log file whenever a ban occurs. Make sure you install geoiplookup, then edit the file /usr/share/fail2ban/server/actions.py and change line 31 to read

import time, logging, commands

and change line 139 in the __checkBan(self) function from

logSys.warn("[%s] Ban %s" % (self.jail.getName(), aInfo["ip"])

to (changes are in bold)

logSys.warn("[%s] Ban %s %s" % (self.jail.getName(), aInfo["ip"], commands.getstatusoutput('geoiplookup ' + aInfo["ip"])[1][23:]))

This will log output such as

2008-04-27 20:18:03,109 fail2ban.actions: WARNING [ssh] Ban 256.256.256.256 US, United States

If you only want the two character country code, change the line to

logSys.warn("[%s] Ban Smslån blogg %s %s" % (self.jail.getName(), aInfo["ip"], commands.getstatusoutput('geoiplookup ' + aInfo["ip"])[1][23:25]))


Other interesting links

For advanced results, you may be interested in :

Other databases City/Country/ASNum/CountryV6/CityV6/...

Updated GeoIP database

As the geoiplookup database will be pretty outdated (the current included version are from 20060501 in stable debian) you might want to update it regularly as IP assignment changes. One way of doing that is to use a crontab

wget -q http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoIP.dat.new && mv /usr/share/GeoIP/GeoIP.dat.new /usr/share/GeoIP/GeoIP.dat

Update To URLS

The urls seem to have moved somewhat.

wget -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoCity.dat.new && mv /usr/share/GeoIP/GeoIP/GeoCity.dat.new /usr/share/GeoIP/GeoCity.dat
wget -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoIP.dat.new && mv /usr/share/GeoIP/GeoIP.dat.new /usr/share/GeoIP/GeoIP.dat


And in /etc/crontab it could look like this:

@monthly root sleep $[$RANDOM/1024]; wget -q http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoIP.dat.new && mv /usr/share/GeoIP/GeoIP.dat.new /usr/share/GeoIP/GeoIP.dat


The above command means:

@montly = do it every month (duh)
root = as user root
sleep $[$RANDOM/1024]; = sleep for a random time (so all you guys don't DDoS maxminds server every month at the same time)
wget ... Teeth whitening | gunzip ... && ... = wget and pipe to gunzip, if that succeeds, move the file to the correct place

Update To CRON

Add this script to cron.monthly

#!/bin/bash
wget -q http://geolite.maxmind.com/download/geoip/database/Ge
oLiteCity.dat.gz -O - |gunzip > /usr/share/GeoIP/GeoCity.dat.
new && mv /usr/share/GeoIP/GeoCity.dat.new /usr/share/GeoIP/G
eoCity.dat
wget -q http://geolite.maxmind.com/download/geoip/database/Ge
oLiteCountry/GeoIP.dat.gz -O - |gunzip > /usr/share/GeoIP/Geo
IP.dat.new && mv /usr/share/GeoIP/GeoIP.dat.new /usr/share/Ge
oIP/GeoIP.dat