LibmodSecurity is a WAF (Web Application Firewall), which allows us to detect certain types of attacks based on predefined rules. Through these signatures, we can detect SQL injections, XSS, LFI, RFI, and we can also have specific rules for certain software such as Wordpress, cPanel, osCommerce, Joomla. Additionally, we can load rules from the OWASP project or even our own.
Until version 2.X, the project was called modsecurity, which depended heavily on Apache and adaptations to other servers were mere wrappers around the native operation mode of Apache, which resulted in a performance penalty on servers that are not Apache. We can find a more detailed analysis at this link.
Starting from version 3, it is no longer a module but a library, and each web server accesses it through a connector in the form of a module. We will use the connectors for Apache and Nginx:
https://github.com/SpiderLabs/ModSecurity
https://github.com/SpiderLabs/ModSecurity-nginx
https://github.com/SpiderLabs/ModSecurity-apache
Version 2.X is still maintained, but it is not recommended to use it:
https://github.com/SpiderLabs/ModSecurity/tree/v2/master
NOTE: If we use Nginx as a balancer, it can be installed at this common point and would protect the rest of the web servers.
This manual consists of several parts:
- Compilation and installation of PHP
- Compilation and installation of Apache + PHP
- Compilation and installation of Nginx + PHP
- Test code
- Compilation and installation of MySQL
- PHP tests and injection
- Compilation and installation of libmodsecurity
- Apache connector
- Nginx connector
- libmodsecurity configuration
- Apache + libmodsecurity configuration
- Nginx + libmodsecurity configuration
- Injection test2
- Concurrency
- Custom config for having two simultaneous web servers
- Integration with Telegram
- DEBUG
- Performance
Compilation and installation of PHP
We start by compiling and installing PHP, first we define the PHP version:
PHP_TARGETS="php7-3"
PHP_INI_VERSION="production"
We define the use flags we want in PHP:
dev-lang/php apache2 berkdb bzip2 cli crypt ctype curl curlwrappers exif fileinfo filter ftp gd gdbm hash iconv imap intl json mysql mysqli nls odbc pdo phar posix readline session simplexml soap sockets sqlite3 ssl sysvipc threads tokenizer unicode xml xmlreader xmlrpc xmlwriter zip zlib threads fpm cgi truetype bcmath
We add the fpm use flag to eselect-php:
We compile PHP:
We configure the php-fpm pool:
[www]
user = nobody
group = nobody
listen = 127.0.0.1:9000
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
We start PHP-FPM and add it to the startup:
rc-update add php-fpm default
We select the PHP version we want to use:
[1] php7.2 *
[2] php7.3
We restart FPM:
Compilation and installation of Apache + PHP
Now we compile Apache with support for PHP-FPM:
APACHE2_MPMS="worker"
APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_core authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias proxy socache_shmcb proxy_fcgi authn_core unixd proxy_http proxy_http2"
www-servers/apache threads
We configure Apache to load the necessary modules to run PHP through FPM:
APACHE2_OPTS="-D LANGUAGE -D PROXY -D CGI"
We create the Apache vhost, which will listen on port 8000:
Listen 8000
<VirtualHost *:8000>
ServerAdmin sys@alfaexploit.com
DocumentRoot /var/www/modsecurityTest/
ServerName modsecurityTest.alfaexploit.com
ErrorLog /var/log/apache2/modsecurityTest.error_log
CustomLog /var/log/apache2/modsecurityTest.access_log combined
DirectoryIndex index.php
AddHandler application/x-httpd-php .php .php5 .phtml
AddHandler application/x-httpd-php-source .phps
<FilesMatch ".php$">
SetHandler "proxy:fcgi://127.0.0.1:9000/"
</FilesMatch>
<Directory "/var/www/modsecurityTest/">
Options -Indexes +FollowSymLinks +ExecCGI
AllowOverride None
Require all granted
AllowOverride All
</Directory>
</VirtualHost>
We create the log files for our web project:
touch /var/log/apache2/modsecurityTest.access_log
We restart Apache:
Compilation and installation of Nginx + PHP
Now we compile Nginx:
We create the Nginx vhost, which will listen on port 8001:
user nginx nginx;
worker_processes 1;
error_log /var/log/nginx/error_log info;
events {
worker_connections 1024;
use epoll;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main
'$remote_addr - $remote_user [$time_local] '
'"$request" $status $bytes_sent '
'"$http_referer" "$http_user_agent" '
'"$gzip_ratio"';
client_header_timeout 10m;
client_body_timeout 10m;
send_timeout 10m;
connection_pool_size 256;
client_header_buffer_size 1k;
large_client_header_buffers 4 2k;
request_pool_size 4k;
gzip off;
output_buffers 1 32k;
postpone_output 1460;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 75 20;
ignore_invalid_headers on;
index index.php;
server {
listen 0.0.0.0:8001;
server_name modsecurityTest.alfaexploit.com;
access_log /var/log/nginx/modsecurityTest.access_log main;
error_log /var/log/nginx/modsecurityTest.error_log info;
root /var/www/modsecurityTest/;
location ~* \.php$ {
fastcgi_index index.php;
fastcgi_pass 127.0.0.1:9000;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
}
}
}
We restart Nginx:
Test code
To perform the tests, we will program a small web application vulnerable to SQL injections:
vi /var/www/modsecurityTest/index.php
<html>
<body>
<?php
if(isset($_POST['login']))
{
$username = $_POST['username'];
$password = $_POST['password'];
$con = mysqli_connect('localhost','root','password','sample');
$result = mysqli_query($con, "SELECT * FROM `users` WHERE username='$username' AND password='$password'");
if(mysqli_num_rows($result) == 0)
echo 'Invalid username or password';
else
echo '<h1>Logged in</h1><p>A Secret for you....</p>';
}
else
{
?>
<form action="" method="post">
Username: <input type="text" name="username"/><br />
Password: <input type="password" name="password"/><br />
<input type="submit" name="login" value="Login"/>
</form>
<?php
}
?>
</body>
</html>
Compilation and installation of MySQL
We compile MySQL:
emerge --config =dev-db/mysql-5.7.27-r1
/etc/init.d/mysql start
rc-update add mysql default
We create the database accessed by the web application:
create database sample;
connect sample;
create table users(username VARCHAR(100),password VARCHAR(100));
insert into users values('jesin','pwd');
insert into users values('alice','secret');
quit;
PHP tests and injection
We check that PHP execution works correctly:
We access the corresponding URLs of Apache/Nginx:
http://modsecuritytest.alfaexploit.com:8000/info.php
http://modsecuritytest.alfaexploit.com:8001/info.php
Once verified, we delete the phpinfo file:
We verify that the app works correctly on both servers:
http://modsecurityTest.alfaexploit.com:PORT
Username: jesin
Password: pwd
Logged in
A Secret for you....
We verify that the application is vulnerable to SQL injections, if we enter as a user:
Username: ' or true --
Logged in
A Secret for you....
Compilation and installation of libmodsecurity
We compile libmodsecurity, which is not available in the repositories of any distro, it must be compiled from the sources unless
Nginx-Plus
is used. First, we compile a json dependency:
Now the library itself:
git clone –depth 1 -b v3/master –single-branch https://github.com/SpiderLabs/ModSecurity/
cd ModSecurity
git submodule init
git submodule update
./build.sh
./configure –prefix=/ –with-yajl –enable-standalone-module
make -j4
make install
ln -s /include/modsecurity /usr/include/modsecurity
Apache Connector
We compile the connector for Apache:
git clone –depth 1 https://github.com/SpiderLabs/ModSecurity-apache.git
cd ModSecurity-apache
./autogen.sh
./configure
make -j4
make install
We verify that the module exists:
-rwxr-xr-x 1 root root 29488 nov 4 13:01 /usr/lib64/apache2/modules/mod_security3.so
We load the module:
LoadModule security3_module modules/mod_security3.so
We restart Apache and verify that the module has been loaded:
/etc/init.d/apache2 modules
security3_module (shared)
Nginx Connector
We compile the connector for Nginx:
git clone –depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git
We recompile Nginx indicating where the
external module
is located:
NGINX_ADD_MODULES="/usr/src/ModSecurity-nginx"
We check that the module has been loaded:
http_v2_module
http_realip_module
http_ssl_module
stream_access_module
stream_geo_module
stream_limit_conn_module
stream_map_module
stream_return_module
stream_split_clients_module
stream_upstream_hash_module
stream_upstream_least_conn_module
stream_upstream_zone_module
mail_imap_module
mail_pop3_module
mail_smtp_module
module=/usr/src/ModSecurity
libmodsecurity Configuration
We configure modsec starting from a base configuration:
cd /etc/modsec
wget https://raw.githubusercontent.com/SpiderLabs/ModSecurity/v3/master/modsecurity.conf-recommended
mv modsecurity.conf-recommended modsecurity.conf
vi /etc/modsec/modsecurity.conf
SecRuleEngine On
SecAuditLogFormat json
SecAuditEngine RelevantOnly
SecAuditLog /var/log/modsec_audit.log
There is a file necessary for the operation of libmodsecurity but for some reason it is not installed when doing the make install, we download it manually:
We download the OWASP rules:
git clone https://github.com/SpiderLabs/owasp-modsecurity-crs.git
cd owasp-modsecurity-crs
cp crs-setup.conf.example crs-setup.conf
We tell modsec to load these rules:
# Include the recommended configuration
Include /etc/modsec/modsecurity.conf
# OWASP CRS v3 rules
Include /usr/local/owasp-modsecurity-crs/crs-setup.conf
Include /usr/local/owasp-modsecurity-crs/rules/*.conf
Apache + libmodsecurity Configuration
We configure Apache to use libmodsecurity:
Listen 8000
<VirtualHost *:8000>
ServerAdmin sys@alfaexploit.com
DocumentRoot /var/www/modsecurityTest/
ServerName modsecurityTest.alfaexploit.com
ErrorLog /var/log/apache2/modsecurityTest.error_log
CustomLog /var/log/apache2/modsecurityTest.access_log combined
DirectoryIndex index.php
modsecurity on
modsecurity_rules_file /etc/modsec/main.conf
AddHandler application/x-httpd-php .php .php5 .phtml
AddHandler application/x-httpd-php-source .phps
<FilesMatch ".php$">
SetHandler "proxy:fcgi://127.0.0.1:9000/"
</FilesMatch>
<Directory "/var/www/modsecurityTest/">
Options -Indexes +FollowSymLinks +ExecCGI
AllowOverride None
Require all granted
AllowOverride All
</Directory>
</VirtualHost>
NOTE: If we want to do it globally.
modsecurity on
modsecurity_rules_file /etc/modsec/main.conf
NOTE: If it is a production server, it is preferable to use only detection first, in order to adapt the rules to our particular scenario.
SecRuleEngine DetectionOnly
We restart Apache:
Nginx + libmodsecurity Configuration
We configure Nginx to use libmodsecurity:
user nginx nginx;
worker_processes 1;
error_log /var/log/nginx/error_log info;
events {
worker_connections 1024;
use epoll;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main
'$remote_addr - $remote_user [$time_local] '
'"$request" $status $bytes_sent '
'"$http_referer" "$http_user_agent" '
'"$gzip_ratio"';
client_header_timeout 10m;
client_body_timeout 10m;
send_timeout 10m;
connection_pool_size 256;
client_header_buffer_size 1k;
large_client_header_buffers 4 2k;
request_pool_size 4k;
gzip off;
output_buffers 1 32k;
postpone_output 1460;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 75 20;
ignore_invalid_headers on;
index index.php;
server {
listen 0.0.0.0:8001;
server_name modsecurityTest.alfaexploit.com;
modsecurity on;
modsecurity_rules_file /etc/modsec/main.conf;
access_log /var/log/nginx/modsecurityTest.access_log main;
error_log /var/log/nginx/modsecurityTest.error_log info;
root /var/www/modsecurityTest/;
location ~* \.php$ {
fastcgi_index index.php;
fastcgi_pass 127.0.0.1:9000;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
}
}
}
We restart Nginx:
Injection Test2
If we try to log in using SQL injection, we get:
Forbidden
You don't have permission to access this resource.
In the logs, we can see the alerts:
Concurrency
If we are going to generate many entries in the log, it is preferable to write the entries in concurrent format. This way, they will not be logged in a text file in a serialized way, but in different files in parallel:
#SecAuditLogType Serial
SecAuditLogType Concurrent
SecAuditLog /var/log/modsec_audit.log
SecAuditLogStorageDir /opt/modsecurity/var/audit
Custom Configuration for Having Two Simultaneous Web Servers
As we have two web servers, we need to give access to both users so that they are able to generate log files:
gpasswd -a apache httpservers
gpasswd -a nginx httpservers
We create the directory where the logs will be stored:
chown -R root:httpservers /opt/modsecurity/var/audit/
chmod 775 /opt/modsecurity/var/audit/
We restart both servers to apply the new configuration:
As soon as we trigger a rule, we can see how new directories are generated:
total 12
drwxrwxr-x 3 root httpservers 4096 nov 4 13:37 .
drwxr-xr-x 4 root root 4096 nov 4 13:28 ..
drwxr-x--- 3 apache apache 4096 nov 4 13:37 20191104
NOTE: In this example, Modsecurity is shared by Apache and Nginx. Each server runs with its corresponding user. Therefore, when Modsecurity generates logs, it will do so with that user. In our example, this creates problems because the first one to generate the directory of the day can write in it, denying access to the second one.
drwxr-x--- 3 apache apache 4096 nov 4 18:56 20191104
drwxr-x--- 3 nginx nginx 4096 nov 4 18:58 20191104
To test both servers, we can delete the log files, perform tests with one server, delete the logs again, and perform tests with the second server. An alternative would be to have two configurations of /etc/modsec/modsecurity.conf, each with a different SecAuditLogStorageDir.
Integration with Telegram
An interesting way to receive notifications is via Telegram. To do this, we must create a bot and add it to a group where we will send the alerts generated by our log parser.
To store the number of attacks per IP, we will use Redis.
We configure the service to require authentication:
requirepass XXXXXXXXXXXXX
We start the service and add it to the startup:
rc-update add redis default
If we look at the files generated by libmodsecurity, we can see that the message field is null except for the rule that has matched. Based on this, we can develop our parser:
""
""
"SQL Injection Attack Detected via libinjection"
""
""
According to comments found on the Internet, even if the rules have the nolog option enabled, they generate log entries with a null message:
If you google the problem, you will see that it is a common bug
We install the necessary Python libraries:
pip install redis –user
pip install pyinotify –user
We program the parser to block attackers if 5 or more attacks are performed within 60 seconds:
#!/usr/bin/python
import pyinotify
import os
import json
import requests
import redis
import iptc
apikey = "XXXXX:YYYYYYYYYYYYYYYYYYYYYY"
telegramurl = "https://api.telegram.org/bot{}/sendMessage".format(apikey)
userid = "ZZZZZZZZZ"
try:
redisconnection = redis.Redis(host="127.0.0.1", port=6379, db=0, password='XXXXXXXXXXXXXX')
redisconnection.ping()
except:
print '++ ERROR: Cant connect to redis server'
quit()
class CommitFunction(pyinotify.ProcessEvent):
def process_default(self, event):
fullpath = event.path + '/' + event.name
if os.path.isfile(fullpath):
print '------------------------------------------------'
print "Processing: " + str(fullpath)
with open(fullpath) as fp:
for line in fp:
try:
rawdata = json.loads(line)
except:
continue
for messageline in rawdata['transaction']['messages']:
message = messageline['message']
data = messageline['details']['data']
# Delete not matched rules messages and anomaly score checks
if message != "" and data != "":
try:
timestamp = rawdata['transaction']['time_stamp']
except:
timestamp = 'NULL'
try:
attacker = rawdata['transaction']['request']['headers']['X-Forwarded-For']
except:
attacker = rawdata['transaction']['client_ip']
#attacker = 'NULL'
try:
useragent = rawdata['transaction']['request']['headers']['User-Agent']
except:
useragent = 'NULL'
try:
host = rawdata['transaction']['request']['headers']['Host']
except:
host = 'NULL'
try:
url = rawdata['transaction']['request']['uri']
except:
url = 'NULL'
try:
method = rawdata['transaction']['request']['method']
except:
method = 'NULL'
try:
payload = messageline['details']['data']
except:
payload = 'NULL'
print '>> Timestamp: ' + str(timestamp)
print 'Attacker: ' + str(attacker)
print 'UserAgent: ' + str(useragent)
print 'Message: ' + str(message)
print 'Host: ' + str(host)
print 'URL: ' + str(url)
print 'Method: ' + str(method)
print 'Payload: ' + str(payload)
print 'Checking redis IP: ' + str(attacker)
if redisconnection.get(attacker):
rediscounter = redisconnection.get(attacker)
#print 'rediscounter: ' + str(rediscounter)
if int(rediscounter) >= 5:
blacklistpath = '/etc/.blacklists/modsecurity.list'
if os.path.isfile(blacklistpath):
# Check if attacker is in blacklist
attackerfound = 0
print 'Checking if attacker is in blacklist'
with open(blacklistpath, "r") as blacklistcontent:
for blacklistline in blacklistcontent:
blacklistline = blacklistline.strip('\n')
print 'Comparing: ' + str(blacklistline) + ' against ' + str(attacker)
if blacklistline == attacker:
print 'Matched'
attackerfound = 1
break
# Add attacker to blacklist
if attackerfound == 0:
blacklistfile = open(blacklistpath,"a")
blacklistfile.write(attacker+'\n')
blacklistfile.close()
print 'Writting attacker ip to blacklist: ' + str(attacker)
else:
blacklistfile = open(blacklistpath,"w+")
blacklistfile.write(attacker+'\n')
blacklistfile.close()
# Check if attacker is in running iptables rules
print 'Checking in running iptables rules'
iptablesdata = iptc.easy.dump_table('filter', ipv6=False)
attackerfoundiptables = 0
for rule in iptablesdata['INPUT']:
if 'src' in rule and 'target' in rule:
if rule['src'] == attacker+'/32' and rule['target'] == 'DROP':
attackerfoundiptables = 1
print 'Matched'
break
# Add attacker to running iptables rules
if attackerfoundiptables == 0:
print 'Inserting in running iptables rules'
rule = iptc.Rule()
rule.src = attacker
rule.create_target("DROP")
rule.target = iptc.Target(rule, "DROP")
chain = iptc.Chain(iptc.Table(iptc.Table.FILTER), "INPUT")
chain.insert_rule(rule)
msg = 'Banning time for: ' + str(attacker)
data = {"chat_id":userid, "text":msg}
try:
r = requests.post(telegramurl,json=data)
except:
print '++ Error sending telegram message'
continue
print 'Sending telegram alert'
msg = 'Date: ' + str(timestamp) + '\nAttacker: ' + str(attacker) + '\nUserAgent: ' + str(useragent) + '\nHost: ' + str(host) + '\nUrl: ' + str(url) + '\nAlert: ' + str(message) + '\nPayload: ' + str(payload)
data = {"chat_id":userid, "text":msg}
try:
r = requests.post(telegramurl,json=data)
except:
print '++ Error sending telegram message'
continue
else:
print 'Incrementing redis key value for IP: ' + str(attacker)
redisconnection.incr(attacker)
else:
print 'Creating redis key for IP: ' + str(attacker)
redisconnection.incr(attacker)
redisconnection.expire(attacker, 60)
wm = pyinotify.WatchManager()
notifier = pyinotify.Notifier(wm)
wm.add_watch('/opt/modsecurity/var/audit/', pyinotify.IN_CREATE, rec=True, auto_add=True, proc_fun=CommitFunction())
notifier.loop(daemonize=False, callback=None)
We assign the necessary permissions:
We can test the parser using several tools:
python sqlmap.py -u http://modsecuritytest.alfaexploit.com –data=“username=&password=”
perl nikto.pl -host http://modsecuritytest.alfaexploit.com:8001
ZAP
Below we can see the different notifications received on Telegram:
To auto-start the parser, we create the script:
/usr/bin/python /usr/local/modsecurityNotifier.py &
We start it manually:
NOTE: If we compile iptables with the static-libs use flag, the python script will fail for some reason:
iptc.errors.XTablesError: can't find target DROP
If there is any functionality in our web application that requires slightly unusual behavior, we can whitelist it so that the rules do not trigger when it comes to this section:
vi /usr/local/owasp-modsecurity-crs/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
SecRule REQUEST_URI "@beginsWith /strangefunctionality" \
"id:1001,\
phase:1,\
pass,\
nolog,\
ctl:ruleEngine=Off"
We reload the configuration by restarting the web servers:
If, on the other hand, we want to disable a specific rule, we will use the SecRuleRemoveById directive.
Note: This directive must be specified after the rule to be disabled.
In my case, I was having problems with the rule:
CGI source code leakage
"ruleId": "950140",
"file": "/usr/local/owasp-modsecurity-crs/rules/RESPONSE-950-DATA-LEAKAGES.conf",
"lineNumber": "66",
When displaying code on the web, the alarm is triggered because it thinks that the source code of the web itself is being filtered:
# Include the recommended configuration
Include /usr/local/etc/modsec/modsecurity.conf
# OWASP CRS v3 rules
Include /usr/local/owasp-modsecurity-crs/crs-setup.conf
Include /usr/local/owasp-modsecurity-crs/rules/*.conf
# Disabled rules
Include /usr/local/etc/modsec/disabledRules.conf
SecRuleRemoveById 950140
DEBUG
If we want to visually see the requests to include other fields in the notifications, we can use this website:
http://jsonviewer.stack.hu/
We can also increase the debug level:
SecDebugLog /opt/modsecurity/var/log/debug.log
SecDebugLogLevel 9
touch /opt/modsecurity/var/log/debug.log
Restart the web servers:
Check the debug logs:
The possible debug levels are:
- 0 No logging
- 1 Errors (e.g., fatal processing errors, blocked transactions)
- 2 Warnings (e.g., non-blocking rule matches)
- 3 Notices (e.g., non-fatal processing errors)
- 4 Informational
- 5 Detailed
- 9 Everything!
For production environments, we can generate only logs but not block requests until the rules are polished:
SecRuleEngine DetectionOnly
Another option is to analyze only a percentage of the requests: Adjust the percentage of requests that are funnelled into the Core Rules by setting TX.sampling_percentage below. The default is 100, meaning that every request gets checked by the CRS.
SecAction "id:900400,\
phase:1,\
pass,\
nolog,\
setvar:tx.sampling_percentage=10"
Performance
Finally, I leave a link to an article on performance optimization:
https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/modsecurity-performance-recommendations/