8000 dnsbl ipset.sh · firehol/firehol Wiki · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

dnsbl ipset.sh

Jump to bottom Edit New page
Gopinaath edited this page Dec 10, 2022 · 24 revisions

dnsbl-ipset.sh

The tool dnsbl-ipset.sh is part of FireHOL v3. You can find it at the contrib directory of the distribution.

If you are looking for maintained IP blacklists/blocklists, we have a large collection of IP lists tracking abuse, attacks, malware, botnets, command and control hosts, open proxies, anonymizers, etc. at FireHOL IP Lists.

This tool tries to figure out a blocklist (a list of IPs that should be blocked at the firewall level) using DNSBLs.

As you may know DNSBLs are used for detecting spam messages. How DNSBLs can help at a firewall level?

Well, it turns out that email spammers are using the same IPs with attackers. Spammers and attackers use open proxies, worms, control and command hosts, etc to send spam and attack other networks. There is a big overlap.

Of course, detecting bad guys using DNSBLs introduces the same problem with spam email: there will be false positives, and since we are going to apply this at a firewall level, where we cannot pattern match the content, these false positive might be a lot.

Then why do we really need this?

Consider this example:

Your web servers are under attack. Attackers are coming from thousands or random IPs all over the internet. They are doing normal web requests, like your legit users/customers, but a lot more. You normally have 20.000 posts per day, but now you have 500.000 ! Your systems are filling up with fake data / orders / comments / posts / transactions. What can you do? How can you tell which are the good ones and which are the bad ones?

To solve this problem you will go through a series of steps:

  1. You will examine if the attackers are using just a few IPs. You will be disappointed when you will realize that each of the fake posts was submitted from an IP that has been used just once today and once yesterday. So you cannot use the rate of requests per IP to detect them.

  2. Next, you will search for available blocklists on the net, hoping that these blocklists will match the attackers IPs. You will use FireHOL's update-ipsets.sh to activate most or all supported ipsets. You will be disappointed again, when you will realize that these blocklists managed to stop only 50% to 80% percent of the attack.

What is next?

A better captcha? This is bypassable (see the Xrumer software and the dozens of captcha decoding companies).

A better application? Probably. If you analyze the data the attackers submit and your web server logs, you may find a few patterns the attackers use, but your legit users do not. You should do this analysis and if you are lucky enough, you may find some patterns easily. A solution however may need some time to be developed and deployed.

This attack is sustained. It will destroy your service if it continues for a long time.

Normally, the attacker will stop when he has no way to execute the attack. If you could somehow find the IPs he is using and block them, after a few days he will stop.

This is where dnsbl-ipset.sh comes into the picture.

Instead of shutting down your service to find how to defend yourself, you can use this tool to generate a blocklist that most probably will detect all the bad guys. Unfortunately it will include a few (or a lot) legit users in it, depending on how aggressive you want to be.

How it works

This program will tail the iptables kernel log, extract SRC/DST IP addresses from it, do DNSBL lookups for them, score them and according to this score add them to an ipset. This ipset may be used by a firewall rule to block further access or just redirect the users to a different page instructing them how to contact you for unblocking them.

So, initially an attacker will get access, but after a few seconds (after 5-10 seconds according to my tests), his IP might be blocked.

dnsbl-ipset.sh itself just manipulates ipsets. It does not alter your firewall - it does not generate or execute iptables statements. It is up to you to decide how to use the ipset it fills with IPs.

How to use it

First, install adnshost which is part of the adns or adns-tools package.

Next, dnsbl-ipset.sh needs a log to search for IPs and an ipset to list the bad guys.

This what I add in firehol.conf:

   # create the dnsbl ipset
   ipset4 create dnsbl hash:ip timeout $[86400 * 7] maxelem 500000 prevent_reset_on_restart comment

   # create a new action called AUDIT_ACCEPT
   # that will log and accept packets
   action4 AUDIT_ACCEPT \
        action ACCEPT state NEW log "AUDIT ACCEPT" \
        next action ACCEPT

   # ... then, later in firehol.conf ...
   server http AUDIT_ACCEPT

The above will not block anything. We just created an ipset which we did not use anywhere and an action that logs a line for each socket it accepts (it does not log every packet - only the first packet of every connection is logged).

Before running dnsbl-ipset.sh we need to add one or more whitelist ipsets. By default the scripts looks for whitelist ipsets named bogons, fullbogons and whitelist. The first 2 can be created by update-ipsets.sh, so please use it. In whitelist you can put your own IP address range. If you don't want to exclude this or you already have another, please edit dnsbl-ipset.conf and set it properly. dnsbl-ipset.conf will be created automatically if it does not exist:

# ./dnsbl-ipset.sh
Generated default config file '/etc/firehol/dnsbl-ipset.conf'.
Please run me again to execute..."

Ok. Now edit /etc/firehol/dnsbl-ipset.conf and set the options according to your preference. The important one is the location of the iptables log in your system.

Now run it again. It will not harm. It will generate a number of DNS requests and depending on their replies it will add a few IPs to the dnsbl ipset which we didn't use to block or accept anything.

Let's see it:

Using ulogd iptables log: /var/log/ulogd/ulogd_syslogemu.log
Searching for: AUDIT

Please wait some time... pipes are filling up... (this is not a joke)
 >  IP         182.70.68.149
 >  IP         124.168.43.200
 >  IP         112.203.172.174
 >  IP         109.246.110.247
 >  IP         59.115.52.206
 -  CLEAN      182.70.68.149    # score -200 from 1 list: -200/127.0.0.11/zen.spamhaus.org
 >  IP         107.15.148.16
 -  CLEAN      124.168.43.200   # score -400 from 2 lists: -200/127.0.0.11/zen.spamhaus.org -200/127.0.0.10/dnsbl.sorbs.net
 -  CLEAN      112.203.172.174  # score -200 from 1 list: -200/127.0.0.10/zen.spamhaus.org
 -  CLEAN      109.246.110.247  # not matched by any list
 >  IP         41.56.225.78
 -  CLEAN      201.226.156.67   # not matched by any list
 -  CLEAN      1.34.184.78      # not matched by any list
 >  IP         83.27.172.27
 -  CLEAN      59.115.52.206    # score -555 from 4 lists: -200/127.0.0.10/dnsbl.sorbs.net -200/127.0.0.11/zen.spamhaus.org -200/127.0.0.36/all.spamrats.com 45/127.0.0.2/b.barracudacentral.org
 +  BLACKLIST  183.252.52.160   # score 365 from 15 lists: 25/127.0.0.2/dnsbl.justspam.org 35/127.0.0.2/all.s5h.net 25/127.0.0.2/rbl.megarbl.net 45/127.0.0.2/z.mailspike.net 100/127.0.0.3/zen.spamhaus.org 45/127.0.0.4/zen.spamhaus.org -200/127.0.0.11/zen.spamhaus.org 45/127.0.0.37/all.spamrats.com 25/127.0.0.2/bl.spamcop.net 25/127.0.0.2/db.wpbl.info 25/127.0.0.6/spam.dnsbl.sorbs.net 25/127.0.0.6/dnsbl.sorbs.net 45/127.0.0.2/b.barracudacentral.org 100/127.0.0.2/hostkarma.junkemailfilter.com 0/127.0.1.1/hostkarma.junkemailfilter.com
 >  IP         46.190.16.176
 -  CLEAN      92.241.52.245    # score -200 from 1 list: -200/127.0.0.11/zen.spamhaus.org

As you can see it started detecting in just seconds. The IP 183.252.52.160 is marked as BLACKLIST with a score of 365, which comes from 15 DNSBLs:

score response DNSBL
25 127.0.0.2 dnsbl.justspam.org
35 127.0.0.2 all.s5h.net
25 127.0.0.2 rbl.megarbl.net
45 127.0.0.2 z.mailspike.net
100 127.0.0.3 zen.spamhaus.org
45 127.0.0.4 zen.spamhaus.org
-200 127.0.0.11 zen.spamhaus.org
45 127.0.0.37 all.spamrats.com
25 127.0.0.2 bl.spamcop.net
25 127.0.0.2 db.wpbl.info
25 127.0.0.6 spam.dnsbl.sorbs.net
25 127.0.0.6 dnsbl.sorbs.net
45 127.0.0.2 b.barracudacentral.org
100 127.0.0.2 hostkarma.junkemailfilter.com
0 127.0.1.1 hostkarma.junkemailfilter.com

dnsbl-ipset.sh gave a different score to each DNSBL response. All the scores for every match are configured in its config file. For example:

  • 127.0.0.2 from z.mailspike.net tells us that this IP participated in spam mail wave in the last 48 hours, so it got 45
  • 127.0.0.3 from zen.spamhaus.org says this is permanent spammer, so it got 100
  • 127.0.0.4 from zen.spamhaus.org is an indication of exploit, or virus, so it got 45
  • 127.0.0.11 from zen.spamhaus.org is an indication of dynamic IP user, so it got -200
  • 127.0.0.2 from hostkarma.junkemailfilter.com is an indication for very bad IP reputation, so it got 100
  • 127.0.0.2 from b.barracudacentral.org is also bad reputation, so it got 45
  • etc.

The tool will blacklist an IP when the sum of all scores of all DNSBLs that matched an IP, is above 100 (this is also configured in the config file).

Let's see the ipset in memory:

# ipset list dnsbl
Name: dnsbl
Type: hash:ip
Revision: 2
Header: family inet hashsize 4096 maxelem 500000 timeout 604800 comment
Size in memory: 249528
References: 4
Members:
183.252.52.160 timeout 602095 comment "score 365 from 15 lists: 25/127.0.0.2/dnsbl.justspam.org 35/127.0.0.2/all.s5h.net 25/127.0.0.2/rbl.megarbl.net 45/127.0.0.2/z.mailspike.net 100/127.0.0.3/zen.spamhaus.org 45/127.0.0.4/zen.spamhaus.org -200/127.0.0.11/zen.spamhaus.org 45/127.0.0.37/all.spa"

So, even the ipset in memory has all the info to find out why an IP was blacklisted.

In the configuration file you can set the scores you would like each DNSBL response to have and information for the meaning of each response.

These are the defaults for the current version (they have different values compared to the example above, because they come from a more recent version of the tool):

IGNORE="0"              # do not take into account this result
PROXY="1000"            # a verified open proxy
EXPLOIT="100"           # a verified exploit
SPAM="15"               # the host is known to send spam
SPAMPRO="100"           # a professional spammer
SPAMWAVE="200"          # participated in a recent spam wave
DYNAMICIP="-500"        # a dynamic IP
BADKARMA="300"          # the host is known to have bad karma
GOODKARMA="-500"        # the host is known to have good karma

The default scoring that comes with the tool favours dynamic IP users. This means that with the default configuration the tool is trying not to block end users, even if a few attackers are left running.

The biggest problem with the false positives is dynamic IP users, where a user with an exploit disconnected and a legit user got the IP he had. The big DNSBLs include hints about this, but their lists are not complete. They know a few ISPs, they don't know others.

The right procedure of using dnsbl-ipset.sh, is this:

  1. Run it to just monitor what would have been blacklisted with the current scores on your traffic
  2. Pick the blacklisted IPs and whois them - this will give you an idea of what is blacklisted. The config has such a whois line commented. Just uncomment it and dnsbl-ipset.sh will also whois all the IPs it blacklists
  3. Adapt the scores, possibly whitelist IP ranges you know are good
  4. Run it with the flush parameter to flush the dnsbl and dnsbl_cache ipsets (so that it will start fresh) and
  5. Run it again.
  6. Repeat this process until you get satisfactory results.

Once you have good results, it is up to you to decide what are you going to do with the bad IPs in the dnsbl ipset. You can blacklist them using the FireHOL's blacklist helper, or you can dnat them to a different web server, or you can just exclude them from the server statement.

Features

  1. Customizable scoring system. You can set the score for each response of a DNSBL.
  2. You can add or remove DNSBLs.
  3. It will try to be nice with the DNSBLs by rate-limiting the requests it does.
  4. It will throttle (stall) if the DNS is taking too long to respond, to avoid occupying too much resources.
  5. The IPs in the dnsbl ipset will have a comment stating the score and the DNSBLs that matched it.
  6. The time each IP will be re-checked is configurable (by default a day)
  7. The time each IP will be blacklisted is also configurable (by default a week)
  8. It can run anything you like for each IP to be blacklisted, so that you can make additional checks and decide if you really want this IP blacklisted or not.
  9. It generates logs for all IPs checked, all the matches found, all the clean and all the bad IPs.

You may be afraid that this tool will generate tons of DNS queries. It will when it starts. But the tool is smart enough to cache what it has already checked, so the rate of DNS queries will drop in just a few minutes.

There is more detailed documentation about its operation, at the top of the script.

Important

Please make sure you don't do more than 100.000 DNSBL queries per day. If you do, most DNSBL's will block you. The tool counts them and reports them to you.

Add a custom footer

FireHOL installation

FireQOS

Link Balancer - routing tables with inheritance, multiple balancing gateways, routing rules

FireHOL & iptables marks

FireHOL & ipsets

FireHOL & SYNPROXY (DDoS mitigation)

FireHOL with basic IDS - just with plain iptables and ipsets

Clone this wiki locally
0