HaProxy is an HTTP/HTTPS traffic and TCP connection balancer, but it is mainly focused on HTTP/HTTPS traffic. It is a very flexible software due to its large number of options and configuration parameters. In this example, we will see how to discriminate traffic using ACLs and how to balance HTTP/HTTPS traffic.
We install HaProxy:
HaProxy uses Syslog as a logging system, therefore we must configure it. In my case, I will use the local0 facility for HaProxy logs:
vi /usr/local/etc/syslog.d/haproxy.conf
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log
We generate the log files:
touch /var/log/haproxy-admin.log
If it is a jail, we must take into account that it is not able to access /dev/log, so we will have to configure HaProxy to send the logs over the network to the jail’s IP.
Syslogd comes with remote logging disabled by default, to enable it we change the startup flags:
We start the service:
service syslogd restart
In the defaults section of the HaProxy configuration, we leave the log options commented out, so they are prepared in case of debugging needs.
The rest of the configuration consists of filtering by domain name and sending it to the appropriate backend. If accessed via HTTP, it will redirect to HTTPS unless it is LetsEncrypt trying to renew the certificate.
HaProxy has been configured in passthrough mode, so each website is responsible for renewing its own certificate. Passthrough mode is the safest because data is never transmitted in plain text.
It is worth noting that when we send traffic from HaProxy to web servers, they will only see the HaProxy’s IP address as the source, not the real client’s IP address. To avoid this, we will make HaProxy add a header called X-Forwarded-For for HTTP and use ProxyProtocol for HTTPS. With a little additional configuration on the web server, it will be able to determine the correct IP address.
global
daemon
maxconn 5000
log 192.168.69.11:514 local0
user nobody
group nobody
stats socket /var/run/haproxy.sock user nobody group nobody mode 660 level admin
ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
timeout connect 10s
timeout client 30s
timeout server 30s
#log global
mode http
#option httplog
listen stats
bind *:8404
stats enable
stats uri /stats
stats refresh 5s
stats auth kr0m:PASSWORD
frontend HTTP
bind :80
option forwardfor
acl letsencrypt path_beg /.well-known/acme-challenge/
http-request redirect scheme https unless letsencrypt
acl rxwod hdr(host) -i rxwod.alfaexploit.com
acl drwho hdr(host) -i mail.alfaexploit.com
acl alfaexploit hdr(host) -i alfaexploit.com
acl www_alfaexploit hdr(host) -i www.alfaexploit.com
http-request deny if !rxwod !drwho !alfaexploit !www_alfaexploit
use_backend rxwod if rxwod
use_backend drwho if drwho
use_backend alfaexploit if alfaexploit
use_backend alfaexploit if www_alfaexploit
frontend HTTP-SSL
bind :443
mode tcp
acl rxwod_ssl req.ssl_sni -i rxwod.alfaexploit.com
acl drwho_ssl req.ssl_sni -i mail.alfaexploit.com
acl alfaexploit_ssl req.ssl_sni -i alfaexploit.com
acl www_alfaexploit_ssl req.ssl_sni -i www.alfaexploit.com
tcp-request inspect-delay 2s
tcp-request content reject if !rxwod_ssl !drwho_ssl !alfaexploit_ssl !www_alfaexploit_ssl
use_backend rxwod_ssl if rxwod_ssl
use_backend drwho_ssl if drwho_ssl
use_backend alfaexploit_ssl if alfaexploit_ssl
use_backend alfaexploit_ssl if www_alfaexploit_ssl
backend rxwod
option httpchk GET /
server rxWod 192.168.69.10:80 check
backend drwho
option httpchk GET /
server drwho 192.168.69.6:80 check
backend alfaexploit
option httpchk GET /
server mistery 192.168.69.3:80 check
backend rxwod_ssl
mode tcp
option ssl-hello-chk
server rxWod 192.168.69.10:443 check sni req.ssl_sni send-proxy-v2
backend drwho_ssl
mode tcp
option ssl-hello-chk
server drwho 192.168.69.6:443 check sni req.ssl_sni send-proxy-v2
backend alfaexploit_ssl
mode tcp
option ssl-hello-chk
server mistery 192.168.69.3:443 check sni req.ssl_sni send-proxy-v2
Start the service:
service haproxy start
A good way of showing the HaProxy state is using hatop tool:
We execute it indicating the stats socket:
Another way is accessing the web interface,
http://192.168.69.11:8404/stats
Here is an example configuration for Nginx to correctly read the client’s IP address.
We assign the $proxy_protocol_addr variable the value readed from X-Forwarded-For and we define a new log format called proxyprotocollogformat which will use that variable.
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
log_format proxyprotocollogformat
'$proxy_protocol_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
Then we include the vhost configuration file, alfaexploit:
include alfaexploit.conf;
The file content should be like this:
worker_processes 1;
load_module /usr/local/libexec/nginx/ngx_http_modsecurity_module.so;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
log_format proxyprotocollogformat
'$proxy_protocol_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
include alfaexploit.conf;
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name localhost;
location / {
root /usr/local/www/nginx;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/local/www/nginx-dist;
}
}
}
The vhost configuration file consists of two parts, HTTP and HTTPS configuration.
In the first part, we will indicate that we should only trust 192.168.69.11 (HaProxy IP) when obtaining the data from the request’s source IP, and we will obtain this information by reading the X-Forwarded-For header.
set_real_ip_from 192.168.69.11;
real_ip_header X-Forwarded-For;
In the second part, we will enable ProxyProtocol in the listen statement, indicating that we should only trust 192.168.69.11 (HaProxy IP) when obtaining the data from the request’s source IP, but this time we will obtain this information through ProxyProtocol.
listen 443 ssl default_server proxy_protocol;
set_real_ip_from 192.168.69.11;
real_ip_header proxy_protocol;
server {
listen 80 default_server;
server_name alfaexploit.com www.alfaexploit.com;
set_real_ip_from 192.168.69.11;
real_ip_header X-Forwarded-For;
root /usr/local/www/nginx/alfaexploit/public;
modsecurity on;
modsecurity_rules_file /usr/local/etc/modsec/main.conf;
location / {
try_files $uri /index.php$is_args$args;
}
location ~ ^/index\.php(/|$) {
fastcgi_pass unix:/var/run/php-fpm.sock;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
fastcgi_param DOCUMENT_ROOT $realpath_root;
internal;
}
location ~ \.php$ {
return 404;
}
}
server {
listen 443 ssl default_server proxy_protocol;
server_name alfaexploit.com www.alfaexploit.com;
set_real_ip_from 192.168.69.11;
real_ip_header proxy_protocol;
root /usr/local/www/nginx/alfaexploit/public;
modsecurity on;
modsecurity_rules_file /usr/local/etc/modsec/main.conf;
location / {
try_files $uri /index.php$is_args$args;
}
location ~ ^/index\.php(/|$) {
fastcgi_pass unix:/var/run/php-fpm.sock;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
fastcgi_param DOCUMENT_ROOT $realpath_root;
internal;
}
location ~ \.php$ {
return 404;
}
ssl_certificate "/usr/local/etc/ssl/alfaexploit.com/fullchain.cer";
ssl_certificate_key "/usr/local/etc/ssl/alfaexploit.com/alfaexploit.com.key";
}
NOTE: We could also use Proxy Protocol for HTTP, but ProxyProtocol is a binary protocol, so it is more complicated to debug in case of problems.
As a final step, we must change the DNS of all our domains to point to the HaProxy IP, and if we use IPFW , configure the filtering rules so that access to the web servers is only through HaProxy.
The IPFW configuration should be very similar to the following if we use jails or iocage :
#!/bin/sh
# Flush out the list before we begin.
ipfw -q -f flush
# Set rules command prefix
cmd="ipfw -q add"
wanif="nfe0"
# No restrictions on Loopback Interface
$cmd 00010 allow all from any to any via lo0
# -----------------------------------------------------
# Generic traffic Host + Jails
# Allow access to public DNS
# DNS TCP
$cmd 00100 allow tcp from me to 8.8.8.8 53 out via $wanif
$cmd 00100 allow tcp from 8.8.8.8 53 to me in via $wanif
$cmd 00100 allow tcp from me to 8.8.4.4 53 out via $wanif
$cmd 00100 allow tcp from 8.8.4.4 53 to me in via $wanif
# DNS UDP
$cmd 00101 allow udp from me to 8.8.8.8 53 out via $wanif
$cmd 00101 allow udp from 8.8.8.8 53 to me in via $wanif
$cmd 00101 allow udp from me to 8.8.4.4 53 out via $wanif
$cmd 00101 allow udp from 8.8.4.4 53 to me in via $wanif
# Allow HTTP
$cmd 00200 allow tcp from me to any 80 out via $wanif
$cmd 00200 allow tcp from any 80 to me in via $wanif
# Allow HTTPS
$cmd 00300 allow tcp from me to any 443 out via $wanif
$cmd 00300 allow tcp from any 443 to me in via $wanif
# Allow SMTP
$cmd 00400 allow tcp from me to any 25 out via $wanif
$cmd 00400 allow tcp from any 25 to me in via $wanif
$cmd 00401 allow tcp from me to any 587 out via $wanif
$cmd 00401 allow tcp from any 587 to me in via $wanif
# Allow ping
$cmd 00500 allow icmp from me to any out via $wanif
$cmd 00500 allow icmp from any to me in via $wanif
# Allow NTP
$cmd 00600 allow udp from me to any 123 out via $wanif
$cmd 00600 allow udp from any 123 to me in via $wanif
# Allow SSH
# Outbound
$cmd 00700 allow tcp from me to any 22 out via $wanif
$cmd 00700 allow tcp from any 22 to me in via $wanif
# Inbound
$cmd 00701 allow tcp from any to me 22 in via $wanif
$cmd 00701 allow tcp from me 22 to any out via $wanif
# -----------------------------------------------------
# Jails services specific traffic
# HAProxy Stats: 8404
$cmd 00800 allow tcp from 192.168.69.4 to 192.168.69.11 8404 in via $wanif
$cmd 00800 allow tcp from 192.168.69.11 8404 to 192.168.69.4 out via $wanif
# HTTP + HTTPS: INET - HAProxy
$cmd 00801 allow tcp from any to 192.168.69.11 80 in via $wanif
$cmd 00801 allow tcp from 192.168.69.11 80 to any out via $wanif
$cmd 00802 allow tcp from any to 192.168.69.11 443 in via $wanif
$cmd 00802 allow tcp from 192.168.69.11 443 to any out via $wanif
# SMTP
$cmd 00900 allow tcp from any to 192.168.69.6 25 in via $wanif
$cmd 00900 allow tcp from 192.168.69.6 25 to any out via $wanif
# Deny and log all other connections
$cmd 60000 deny log all from any to any