This page looks best with JavaScript enabled

Dynamic HAProxy Configuration through DNS

 ·  🎃 kr0m

The easiest way to have a DNS entry for each web server is to use a Consul server since it will automatically generate these entries for us. In this previous article , we learned how to configure HAProxy to reconfigure based on the servers registered in Consul services. The difference between doing it based on services or DNS entries are:

  • Service: We use DataPlaneAPI for configuration rewriting, which allows us absolute flexibility, being able to increase or decrease the pool of backend servers at will.
  • DNS: HAProxy configuration is much simpler without the need for third-party applications (DataPlaneAPI), but the problem is that the number of backend servers is semi-static since the maximum number of servers will be defined in the HAProxy configuration.

For greater clarity, the article has been divided into several parts:

INTRODUCTION:

The scenario consists of a Consul server that will act as a DNS, an HAProxy, and some Nginx servers, all of them jails created using IOCage .

CONSUL: NYNEX-King
HAPROXY: BlindSnipper
NGINX: PeanutBrain00/01/02/03/04/05/06/07/09/10/11

+-----+---------------+-------+--------------+---------------+
| JID |     NAME      | STATE |   RELEASE    |      IP4      |
+=====+===============+=======+==============+===============+
| 39  | BlindSnipper  | up    | 13.1-RELEASE | 192.168.69.51 |
+-----+---------------+-------+--------------+---------------+
| 36  | NYNEX-King    | up    | 13.1-RELEASE | 192.168.69.50 |
+-----+---------------+-------+--------------+---------------+
| 42  | PeanutBrain00 | up    | 13.1-RELEASE | 192.168.69.52 |
+-----+---------------+-------+--------------+---------------+
| 45  | PeanutBrain01 | up    | 13.1-RELEASE | 192.168.69.53 |
+-----+---------------+-------+--------------+---------------+
| 50  | PeanutBrain02 | up    | 13.1-RELEASE | 192.168.69.54 |
+-----+---------------+-------+--------------+---------------+
| 53  | PeanutBrain03 | up    | 13.1-RELEASE | 192.168.69.55 |
+-----+---------------+-------+--------------+---------------+
| 56  | PeanutBrain04 | up    | 13.1-RELEASE | 192.168.69.56 |
+-----+---------------+-------+--------------+---------------+
| 59  | PeanutBrain05 | up    | 13.1-RELEASE | 192.168.69.57 |
+-----+---------------+-------+--------------+---------------+
| 63  | PeanutBrain06 | up    | 13.1-RELEASE | 192.168.69.58 |
+-----+---------------+-------+--------------+---------------+
| 66  | PeanutBrain07 | up    | 13.1-RELEASE | 192.168.69.59 |
+-----+---------------+-------+--------------+---------------+
| 69  | PeanutBrain08 | up    | 13.1-RELEASE | 192.168.69.60 |
+-----+---------------+-------+--------------+---------------+
| 72  | PeanutBrain09 | up    | 13.1-RELEASE | 192.168.69.61 |
+-----+---------------+-------+--------------+---------------+
| 75  | PeanutBrain10 | up    | 13.1-RELEASE | 192.168.69.62 |
+-----+---------------+-------+--------------+---------------+
| 78  | PeanutBrain11 | up    | 13.1-RELEASE | 192.168.69.63 |
+-----+---------------+-------+--------------+---------------+

CONSUL SERVER:

We install Consul:

pkg install consul

We start Consul so that it generates the configuration directory itself:

sysrc consul_enable=YES
service consul start

We perform the base configuration:

vi /usr/local/etc/consul.d/consul.hcl

datacenter = "AlfaExploitDC01"
server = true
data_dir = "/var/db/consul"
bind_addr = "192.168.69.50"
client_addr = "192.168.69.50"
bootstrap = true
bootstrap_expect = 1

ui_config {
    enabled = true
}

enable_syslog = true
log_level = "INFO"

Restart Consul:

service consul restart

Access the web interface, we will only see one registered service, Consul’s own:
http://192.168.69.50:8500

HAPROXY:

Install the necessary software:

pkg install haproxy

Make sure the syslogd service is network-bound, check the configuration flags, if the -ss parameter appears, we must reconfigure it:

sysrc syslogd_flags

syslogd_flags: -c -ss

Reconfigure and restart the service:

sysrc syslogd_flags="-c"
service syslogd restart

Configure HAProxy:

vi /usr/local/etc/haproxy.conf

global
    daemon
    maxconn 5000
    log 192.168.69.51:514 local0
    user nobody
    group nobody
    stats socket /var/run/haproxy.sock user nobody group nobody mode 660 level admin

    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    
defaults
    timeout connect 10s
    timeout client 30s
    timeout server 30s
    log global
    mode http
    option httplog

listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 5s
    stats auth kr0m:PASSWORD

frontend http_front
   bind *:80
   default_backend http_back

backend http_back
    balance roundrobin
    server-template mywebapp 1-10 _web._tcp.service.consul resolvers consul resolve-opts allow-dup-ip resolve-prefer ipv4 check

resolvers consul
    nameserver consul 192.168.69.50:8600
    accepted_payload_size 8192
    hold valid 5s

The most important line is server-template, where we indicate how the backend should be generated:

  • mywebapp: Name of the web app, the servers will be named based on this parameter.
  • 1-10: Number of servers that will be in the backend, as we see it is a static value.
  • _web._tcp.service.consul: DNS entry to query to obtain the web servers of the backend: _SERVICE_NAME._tcp.service.consul
  • resolvers consul: Which resolver to use to perform the DNS query.
  • resolve-opts allow-dup-ip resolve-prefer ipv4 check: Additional options on DNS resolution.

Assign the correct permissions to the file:

chmod 660 /usr/local/etc/haproxy.conf

Manually check that the configuration is correct:

haproxy -c -f /usr/local/etc/haproxy.conf

[NOTICE]   (83382) : haproxy version is 2.6.1-f6ca66d
[NOTICE]   (83382) : path to executable is /usr/local/sbin/haproxy
[WARNING]  (83382) : config : ca-file: 0 CA were loaded from '@system-ca'
Warnings were found.
Configuration file is valid

Start the service:

sysrc haproxy_enable=YES
service haproxy start

NGINX:

Install Consul and Nginx on the web servers:

pkg install consul nginx

We start Nginx:

sysrc nginx_enable=YES
service nginx start

We start Consul so that it generates the configuration directory itself:

sysrc consul_enable=YES
service consul start

In the Consul configuration, we define the IP to bind to and the IP of the consul-server (192.168.69.50):

vi /usr/local/etc/consul.d/consul.hcl

datacenter = "AlfaExploitDC01"
server = false
data_dir = "/var/db/consul"
bind_addr = "192.168.69.XX"
retry_join = ["192.168.69.50"]
enable_syslog = true
log_level = "INFO"

We restart the service:

service consul restart

We register the server as a web server:

vi web.json

{
  "service": {
    "name": "web",
    "port": 80
  }
}
consul services register ./web.json
Node name "PeanutBrain00.alfaexploit.com" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
Registered service: web

NOTE: For now, we are only going to register the node 00-08, 9 in total, which includes the IPs from 192.168.69.52-60.

We can manually check the DNS entries:

dig @192.168.69.50 -p 8600 _web._tcp.service.consul

; <<>> DiG 9.18.5 <<>> @192.168.69.50 -p 8600 _web._tcp.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41802
;; flags: qr aa rd; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_web._tcp.service.consul.	IN	A

;; ANSWER SECTION:
_web._tcp.service.consul. 0	IN	A	192.168.69.52
_web._tcp.service.consul. 0	IN	A	192.168.69.55
_web._tcp.service.consul. 0	IN	A	192.168.69.58
_web._tcp.service.consul. 0	IN	A	192.168.69.56
_web._tcp.service.consul. 0	IN	A	192.168.69.59
_web._tcp.service.consul. 0	IN	A	192.168.69.54
_web._tcp.service.consul. 0	IN	A	192.168.69.60
_web._tcp.service.consul. 0	IN	A	192.168.69.53
_web._tcp.service.consul. 0	IN	A	192.168.69.57

;; Query time: 6 msec
;; SERVER: 192.168.69.50#8600(192.168.69.50) (UDP)
;; WHEN: Tue Sep 13 09:18:35 CEST 2022
;; MSG SIZE  rcvd: 197

If we access the Consul server interface again, we can see the instances registered in the web service:
http://192.168.69.50:8500/ui/alfaexploitdc01/services

We can see how the servers are added to HAProxy, and if there are less than 10 servers registered in DNS, the excess is disabled:

We register the rest of the web servers in Consul: 192.168.69.61-63 and check the DNS entries:

dig @192.168.69.50 -p 8600 _web._tcp.service.consul

; <<>> DiG 9.18.5 <<>> @192.168.69.50 -p 8600 _web._tcp.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65031
;; flags: qr aa rd; QUERY: 1, ANSWER: 12, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_web._tcp.service.consul.	IN	A

;; ANSWER SECTION:
_web._tcp.service.consul. 0	IN	A	192.168.69.56
_web._tcp.service.consul. 0	IN	A	192.168.69.62
_web._tcp.service.consul. 0	IN	A	192.168.69.53
_web._tcp.service.consul. 0	IN	A	192.168.69.60
_web._tcp.service.consul. 0	IN	A	192.168.69.52
_web._tcp.service.consul. 0	IN	A	192.168.69.63
_web._tcp.service.consul. 0	IN	A	192.168.69.58
_web._tcp.service.consul. 0	IN	A	192.168.69.61
_web._tcp.service.consul. 0	IN	A	192.168.69.57
_web._tcp.service.consul. 0	IN	A	192.168.69.55
_web._tcp.service.consul. 0	IN	A	192.168.69.54
_web._tcp.service.consul. 0	IN	A	192.168.69.59

;; Query time: 1 msec
;; SERVER: 192.168.69.50#8600(192.168.69.50) (UDP)
;; WHEN: Tue Sep 13 09:25:01 CEST 2022
;; MSG SIZE  rcvd: 245

Despite having 12 active web servers, HAProxy only uses 10 of them since we statically assigned that value in the configuration. This could be mitigated by configuring a very high number of servers that we know we will never reach or by using DataPlaneAPI :

The servers used are internally decided by HAProxy without a way to know their IPs.

TROUBLESHOOTING:

We can check the HAProxy logs:

tail -f /var/log/messages

Start HAProxy in debug mode from the RC script:

cp /usr/local/etc/rc.d/haproxy /usr/local/etc/rc.d/haproxy.ori

vi /usr/local/etc/rc.d/haproxy
: ${haproxy_flags:="-d -V -f ${haproxy_config} -p ${pidfile}"}

Start HAProxy manually:

haproxy -d -V -f /usr/local/etc/haproxy.conf

Manually check DNS entries:

dig @192.168.69.50 -p 8600 _web._tcp.service.consul

If you liked the article, you can treat me to a RedBull here