3. Installing and Configuring Necessary Software

Here, I will explain how to install the necessary software so that we can limit and test the bandwidth usage.

3.1. Installing Squid with the delay pools feature

As I mentioned before, Squid has a feature called delay pools, which allows us to control download bandwidth. Unfortunately, in most distributions, Squid is shipped without that feature.

So if you have Squid already installed, I must disappoint you -- you need to uninstall it and do it once again with delay pools enabled in the way I explain below.

  1. To get maximum performance from our Squid proxy, it's best to create a separate partition for its cache, called /cache/. Its size should be about 300 megabytes, depending on our needs.

    If you don't know how to make a separate partition, you can create the /cache/ directory on a main partition, but Squid performance can suffer a bit.

  2. We add a safe 'squid' user:

    # useradd -d /cache/ -r -s /dev/null squid >/dev/null 2>&1

    No one can log in as squid, including root.

  3. We download Squid sources from http://www.squid-cache.org

    When I was writing this HOWTO, the latest version was Squid 2.4 stable 1:

    http://www.squid-cache.org/Versions/v2/2.4/squid-2.4.STABLE1-src.tar.gz

  4. We unpack everything to /var/tmp:

  5. # tar xzpf squid-2.4.STABLE1-src.tar.gz

  6. We compile and install Squid (everthing is in one line):

    # ./configure --prefix=/opt/squid --exec-prefix=/opt/squid --enable-delay-pools --enable-cache-digests --enable-poll --disable-ident-lookups --enable-truncate --enable-removal-policies

    # make all

    # make install

3.2. Configuring Squid to use the delay pools feature

  1. Configure our squid.conf file (located under /opt/squid/etc/squid.conf):

    #squid.conf
    #Every option in this file is very well documented in the original squid.conf file
    #and on http://www.visolve.com/squidman/Configuration%20Guide.html
    
    #
    #The ports our Squid will listen on
    http_port 8080
    icp_port 3130
    #cgi-bins will not be cached
    acl QUERY urlpath_regex cgi-bin \?
    no_cache deny QUERY
    #Memory the Squid will use. Well, Squid will use far more than that.
    cache_mem 16 MB
    #250 means that Squid will use 250 megabytes of disk space
    cache_dir ufs /cache 250 16 256
    redirect_rewrites_host_header off
    cache_replacement_policy GDSF
    acl localnet src 192.168.1.0/255.255.255.0
    acl localhost src 127.0.0.1/255.255.255.255
    acl Safe_ports port 80 443 210 119 70 20 21 1025-65535
    acl CONNECT method CONNECT
    acl all src 0.0.0.0/0.0.0.0
    http_access allow localnet
    http_access allow localhost
    http_access deny !Safe_ports
    http_access deny CONNECT
    http_access deny all
    maximum_object_size 3000 KB
    store_avg_object_size 50 KB
     
    #all our LAN users will be seen by external servers
    #as if they all use Mozilla on Linux :)
    anonymize_headers deny User-Agent
    fake_user_agent Mozilla/5.0 (X11; U; Linux 2.4.4 i686)
     
    #To make our connection even faster, we put a line similar
    #to the one below. Don't forget to change the server to your closest!
    #Measure pings, traceroutes and so on.
    #Make sure that http and icp ports are correct
    #cache_peer w3cache.icm.edu.pl parent 8080 3130 no-digest default
     
    #This is useful when we want to use the Cache Manager
    #copy cachemgr.cgi to cgi-bin of your www server
    cache_mgr your@email
    cachemgr_passwd secret_password all
     
    #This is a name of a user our Squid will work as
    cache_effective_user squid
    cache_effective_group squid
     
    log_icp_queries off
    buffered_logs on
     
     
    #####DELAY POOLS
    #This is the most important part for shaping incoming traffic with Squid
    #For detailed description see squid.conf file or docs at http://www.squid-cache.org
     
    #We don't want to limit downloads on our local network
    acl magic_words1 url_regex -i 192.168
     
    #We want to limit downloads of these type of files
    #Put this all in one line
    acl magic_words2 url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt
    .ram .rm .iso .raw .wav
    #We don't block .html, .gif, .jpg and similar files, because they
    #generally don't consume much bandwidth
     
    #We have two different delay_pools
    delay_pools 2
     
    #First delay pool
    #W don't want to delay our local traffic
    #There are three pool classes; here we will deal only with the second
    delay_class 1 2
     
    #-1/-1 mean that there are no limits
    delay_parameters 1 -1/-1 -1/-1
     
    #magic_words1: 192.168
    delay_access 1 allow magic_words1
     
    #Second delay pool
    #we want to delay downloading files mentioned in magic_words2
    delay_class 2 2
     
    #The numbers here are values in bytes;
    #we must remember that Squid doesn't consider start/stop bits
    #5000/150000 are values for the whole network
    #5000/120000 are values for the single IP
    #after downloaded files exceed about 150000 bytes,
    #(or even twice or three times as much)
    #they will continue to download at about 5000 bytes/s
     
    delay_parameters 2 5000/150000 5000/120000
    delay_access 2 allow magic_words2
     
    #EOF

    OK, when we have configured everything, we must make sure everything under /opt/squid and /cache directories belongs to user 'squid'.

    # chown -R squid:squid /opt/squid/

    # chown -R squid:squid /cache/

    or

    # chown -R squid.squid /opt/squid/

    # chown -R squid.squid /cache/

    Now everything is ready to run Squid. When we do it for the first time, we have to create its cache directories:

    # /opt/squid/usr/bin/squid -z

    We run Squid and check if everything is working. A good tool to do that is IPTraf; you can find it on http://freshmeat.net. Make sure you have set the appropriate proxy in your web browsers (192.168.1.1, port 8080 in our example):

    # /opt/squid/usr/bin/squid

    If everything is working, we add /opt/squid/usr/bin/squid line to the end of our initializing scripts. Usually, it can be /etc/rc.d/rc.local.

    Other helpful options in Squid may be:

    # /opt/squid/usr/bin/squid -k reconfigure (it reconfigures Squid if we made any changes in its squid.conf file)

    # /opt/squid/usr/bin/squid -help :) self-explanatory

    You can also copy cachemgr.cgi to the cgi-bin directory of your WWW server.

3.3. Solving remaining problems

OK, we have installed Squid and configured it to use delay pools. I bet nobody wants to be restricted, especially our clever LAN users. They will likely try to avoid our limitations, just to download their favourite mp3s a little faster (and thus causing your headache).

I assume that you use IP-masquerade on your LAN so that your users can use IRC, ICQ, e-mail, etc. That's OK, but we must make sure that our LAN users will use our delay pooled Squid to access web pages and use ftp.

We can solve most of these problems by using ipchains (Linux 2.2.x kernels) or iptables (Linux 2.4.x kernels).

3.3.1. Linux 2.2.x kernels (ipchains)

We must make sure that nobody will try to cheat and use a proxy server other than ours. Public proxies usually run on 3128 and 8080 ports:

/sbin/ipchains -A input -s ! 192.168.1.1 -d ! 192.168.1.1 3128 -p TCP -j REJECT

/sbin/ipchains -A input -s ! 192.168.1.1 -d ! 192.168.1.1 8080 -p TCP -j REJECT

We must also make sure that nobody will try to cheat and connect to the internet directly (IP-masquerade) to download web pages:

/sbin/ipchains -A input -s ! 192.168.1.1 -d ! 192.168.1.1 http -p TCP -j REDIRECT 8080

/sbin/ipchains -A input -s ! 192.168.1.1 -d ! 192.168.1.1 https -p TCP -j REDIRECT 8080

If everything is working, we add these lines to the end of our initializing scripts. Usually, it can be /etc/rc.d/rc.local.

We might think to block ftp traffic (ports 20 and 21) to force our LAN users to use Squid, but it's not a good idea for at least two reasons:

  • Squid is a http proxy with ftp support, not a real ftp proxy. It can download from ftp, it can also upload to some ftp, but it can't delete/change name of files on remote ftp servers.

    When we block ports 20 and 21, we won't be able to delete/change name of files on remote ftp servers.

  • IE5.5 has a bug -- it doesn't use a proxy to retrieve the ftp directory. Instead it connects directly via IP-masquerade.

    When we block ports 20 and 21, we won't be able to browse through ftp directories, using IE5.5.

So, we will block excessive ftp downloads using other methods. We will deal with it in chapter 4.

3.3.2. Linux 2.4.x kernels (iptables)

We must make sure that nobody will try to cheat and use a proxy server other than ours. Public proxies usually run on 3128 and 8080 ports:

/sbin/iptables -A FORWARD -s ! 192.168.1.1 -d ! 192.168.1.1 --dport 3128 -p TCP -j DROP

/sbin/iptables -A FORWARD -s ! 192.168.1.1 -d ! 192.168.1.1 --dport 8080 -p TCP -j DROP

We must also make sure that nobody will try to cheat and connect to the internet directly (IP-masquerade) to download web pages:

/sbin/iptables -A FORWARD -s ! 192.168.1.1 -d ! 192.168.1.1 --dport 80 -p TCP -j DROP

/sbin/iptables -A FORWARD -s ! 192.168.1.1 -d ! 192.168.1.1 --dport 443 -p TCP -j DROP

If everything is working, we add these lines to the end of our initializing scripts. Usually, it can be /etc/rc.d/rc.local.

We might think to block ftp traffic (ports 20 and 21) to force our LAN users to use Squid, but it's not a good idea for at least two reasons:

  • Squid is a http proxy with ftp support, not a real ftp proxy. It can download from ftp, it can also upload to some ftp, but it can't delete/change name of files on remote ftp servers.

    When we block ports 20 and 21, we won't be able to delete/change name of files on remote ftp servers.

  • IE5.5 has a bug -- it doesn't use a proxy to retrieve the ftp directory. Instead it connects directly via IP-masquerade.

    When we block ports 20 and 21, our LAN users won't be able to browse through ftp directories, using IE5.5.

So, we will block excessive ftp downloads using other methods. We will deal with it in chapter 4.