Consul is a software that offers very interesting functionalities, especially in dynamic environments with many changes. These functionalities are:

• Service registration: Groups of servers grouped by the service offered.
• K/V database: Centralization of parameters of our applications.
• Access control: Definition of access to registered services.

In this article, we will only use the service registration to dynamically reconfigure an HAProxy and thus add or remove servers from the backends according to their status. The rewriting of the HAProxy configuration will be possible thanks to a tool called DataPlaneAPI .

The article is composed of several parts:

• Introduction
• Consul server
• DataPlaneAPI
• HAProxy
• Nginx
• Troubleshooting

INTRODUCTION:

The scenario consists of a Consul server, an HAProxy, and some Nginx servers, all of them jails created using IOCage .

CONSUL: NYNEX-King
HAPROXY: BlindSnipper
NGINX: PeanutBrain00/01/02/03/04/05/06/07/09/10/11

+-----+---------------+-------+--------------+---------------+
| JID |     NAME      | STATE |   RELEASE    |      IP4      |
+=====+===============+=======+==============+===============+
| 39  | BlindSnipper  | up    | 13.1-RELEASE | 192.168.69.51 |
+-----+---------------+-------+--------------+---------------+
| 36  | NYNEX-King    | up    | 13.1-RELEASE | 192.168.69.50 |
+-----+---------------+-------+--------------+---------------+
| 42  | PeanutBrain00 | up    | 13.1-RELEASE | 192.168.69.52 |
+-----+---------------+-------+--------------+---------------+
| 45  | PeanutBrain01 | up    | 13.1-RELEASE | 192.168.69.53 |
+-----+---------------+-------+--------------+---------------+
| 50  | PeanutBrain02 | up    | 13.1-RELEASE | 192.168.69.54 |
+-----+---------------+-------+--------------+---------------+
| 53  | PeanutBrain03 | up    | 13.1-RELEASE | 192.168.69.55 |
+-----+---------------+-------+--------------+---------------+
| 56  | PeanutBrain04 | up    | 13.1-RELEASE | 192.168.69.56 |
+-----+---------------+-------+--------------+---------------+
| 59  | PeanutBrain05 | up    | 13.1-RELEASE | 192.168.69.57 |
+-----+---------------+-------+--------------+---------------+
| 63  | PeanutBrain06 | up    | 13.1-RELEASE | 192.168.69.58 |
+-----+---------------+-------+--------------+---------------+
| 66  | PeanutBrain07 | up    | 13.1-RELEASE | 192.168.69.59 |
+-----+---------------+-------+--------------+---------------+
| 69  | PeanutBrain08 | up    | 13.1-RELEASE | 192.168.69.60 |
+-----+---------------+-------+--------------+---------------+
| 72  | PeanutBrain09 | up    | 13.1-RELEASE | 192.168.69.61 |
+-----+---------------+-------+--------------+---------------+
| 75  | PeanutBrain10 | up    | 13.1-RELEASE | 192.168.69.62 |
+-----+---------------+-------+--------------+---------------+
| 78  | PeanutBrain11 | up    | 13.1-RELEASE | 192.168.69.63 |
+-----+---------------+-------+--------------+---------------+

CONSUL SERVER:

We install Consul:

We start Consul so that it generates the configuration directory itself:

We perform the base configuration:

datacenter = "AlfaExploitDC01"
server = true
data_dir = "/var/db/consul"
bind_addr = "192.168.69.50"
client_addr = "192.168.69.50"
bootstrap = true
bootstrap_expect = 1

ui_config {
    enabled = true
}

enable_syslog = true
log_level = "INFO"

We restart Consul:

When we access the web interface, we will only see one registered service, Consul’s own:
http://192.168.69.50:8500

DATAPLANEAPI:

To be able to reconfigure HAProxy dynamically, we will use Data Plane API . This allows two service discovery backends , AWS and Consul.

To use Data Plane API, a version of HAProxy >= 1.9.0 is required, which is not a problem in new installations since FreeBSD installs 2.6.1.

We install the necessary software for the compilation of DataPlaneAPI since it is not distributed in binary form. We also install jq to be able to view the json output more comfortably:

We proceed with its compilation:

We can see the binary:

-rwxr-xr-x  1 root  wheel  36904680 Sep  7 19:23 build/dataplaneapi

We copy it to the system and assign the correct permissions:

In the Data Plane API configuration, we define the IP/port binding as well as some parameters such as the user/password to access the Data Plane API API.

config_version: 2
name: haproxy0
dataplaneapi:
  host: 192.168.69.51
  port: 5555
  user:
  - name: kr0m
    password: PASSWORD
    insecure: true

In most configurations, we will see that HAProxy has been configured with the program option . This is done so that HAProxy itself manages DataPlaneAPI, but this is not actually necessary since each process can function independently, with HAProxy serving requests and DataPlaeAPI rewriting the configuration. Additionally, in FreeBSD, this is problematic since the HAProxy RC script gets confused and loses track of the HAProxy PID, causing errors when stopping/restarting the service. Therefore, we will manage each service independently.

We generate the RC script for DataPlaneAPI:

#!/bin/sh
#
# $FreeBSD$
#

# PROVIDE: dataplaneapi

. /etc/rc.subr

name="dataplaneapi"
command="/usr/local/bin/dataplaneapi"
desc="Data Plane API: HAProxy reconfigurator"
rcvar="dataplaneapi_enable"

start_cmd="dataplaneapi_start"
stop_cmd="dataplaneapi_stop"
status_cmd="dataplaneapi_status"

extra_commands="status"

: ${dataplaneapi_config:="/usr/local/etc/dataplaneapi.yaml"}
: ${dataplaneapi_haproxyconfig:="/usr/local/etc/haproxy.conf"}
: ${dataplaneapi_haproxybin:="/usr/local/sbin/haproxy"}
: ${dataplaneapi_haproxyreloadcmd:="/usr/sbin/service haproxy reload"}
: ${dataplaneapi_haproxyrestartcmd:="/usr/sbin/service haproxy restart"}
: ${dataplaneapi_logfile:="/var/log/dataplaneapi.log"}
: ${dataplaneapi_loglevel:="warning"}

dataplaneapi_start()
{
	if [ -f /var/run/${name}.pid ]; then
		echo "${name} running with PID: $(cat /var/run/${name}.pid)"
		echo "Stop it before starting it"
		exit
	else
		echo "Starting service: ${name}"
		/usr/sbin/daemon -S -p /var/run/${name}.pid -u root ${command} -f ${dataplaneapi_config} --config-file ${dataplaneapi_haproxyconfig} --haproxy-bin ${dataplaneapi_haproxybin} --reload-cmd "${dataplaneapi_haproxyreloadcmd}" --restart-cmd "${dataplaneapi_haproxyrestartcmd}" --log-to file --log-file ${dataplaneapi_logfile} --log-level ${dataplaneapi_loglevel} --show-system-info
	fi
}

dataplaneapi_stop()
{
	if [ -f /var/run/${name}.pid ]; then
        	echo "Stopping service: ${name}"
	        kill -s INT $(cat /var/run/${name}.pid)
	        sleep 3
	else
        	echo "It appears ${name} is not running."
	fi
}

dataplaneapi_status()
{
	if [ -f /var/run/${name}.pid ]; then
        	echo "${name} running with PID: $(cat /var/run/${name}.pid)"
	else
        	echo "It appears ${name} is not running."
	fi
}

load_rc_config $name
run_rc_command $*

We assign the necessary permissions:

We start the service enabling the log to debug possible problems:

Data Plane API has its interface, we access it to check that it works correctly:
http://192.168.69.51:5555/v2/docs

kr0m/PASSWORD

HAPROXY:

We install the necessary software:

We make sure that the syslogd service is bound by network, we check the configuration flags, if the -ss parameter appears, we must reconfigure it:

syslogd_flags: -c -ss

We reconfigure it and restart the service:

We configure HAProxy:

global
    daemon
    maxconn 5000
    log 192.168.69.51:514 local0
    user nobody
    group nobody
    stats socket /var/run/haproxy.sock user nobody group nobody mode 660 level admin

    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    
defaults
    timeout connect 10s
    timeout client 30s
    timeout server 30s
    log global
    mode http
    option httplog

listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 5s
    stats auth kr0m:PASSWORD

We assign the correct permissions to the file:

We manually check that the configuration is correct:

[NOTICE]   (83382) : haproxy version is 2.6.1-f6ca66d
[NOTICE]   (83382) : path to executable is /usr/local/sbin/haproxy
[WARNING]  (83382) : config : ca-file: 0 CA were loaded from '@system-ca'
Warnings were found.
Configuration file is valid

We start the service:

We indicate to DataPlaneAPI that it must query the service called web from Consul to obtain the backend servers:
https://github.com/haproxytech/dataplaneapi/blob/master/discovery/CONSUL.md
http://192.168.69.51:5555/v2/docs#tag/ServiceDiscovery/operation/createConsul

curl -u kr0m:PASSWORD \
       -H 'Content-Type: application/json' \
       -d '{
             "address": "192.168.69.50",
             "port": 8500,
             "enabled": true,
             "retry_timeout": 10,
             "server_slots_growth_increment": 1,
             "server_slots_growth_type": "linear",
             "service_allowlist": ["web"]
           }' http://192.168.69.51:5555/v2/service_discovery/consul | jq

{
  "address": "192.168.69.50",
  "enabled": true,
  "id": "d36476d7-3bea-474e-a4b6-0f39c3878759",
  "port": 8500,
  "retry_timeout": 10,
  "server_slots_base": 10,
  "server_slots_growth_increment": 10,
  "server_slots_growth_type": "linear",
  "service-blacklist": null,
  "service-whitelist": null,
  "service_allowlist": [
    "web"
  ],
  "service_denylist": null
}

We consult the registered Consul servers:

{
  "data": [
    {
      "address": "192.168.69.50",
      "enabled": true,
      "id": "d36476d7-3bea-474e-a4b6-0f39c3878759",
      "port": 8500,
      "retry_timeout": 10,
      "server_slots_base": 10,
      "server_slots_growth_increment": 10,
      "server_slots_growth_type": "linear",
      "service-blacklist": null,
      "service-whitelist": null,
      "service_allowlist": [
        "web"
      ],
      "service_denylist": null
    }
  ]
}

We can also consult the data of one of the Consul servers:

Or delete one of the Consul servers:

If we consult the DataPlaneAPI configuration file, we will see that it has changed:

config_version: 2
name: haproxy0
mode: single
status: ""
dataplaneapi:
  host: 192.168.69.51
  port: 5555
  user:
  - name: kr0m
    insecure: true
    password: PASSWORD
  advertised:
    api_address: ""
    api_port: 0
service_discovery:
  consuls:
  - address: 192.168.69.50
    description: ""
    enabled: true
    id: d36476d7-3bea-474e-a4b6-0f39c3878759
    name: ""
    namespace: ""
    port: 8500
    retrytimeout: 10
    serverslotsbase: 10
    serverslotsgrowthincrement: 10
    serverslotsgrowthtype: linear
    serviceblacklist: []
    servicewhitelist: []
    serviceallowlist:
    - web
    servicedenylist: []
    token: ""

NGINX:

We install Consul and Nginx on the web servers:

We start Nginx:

We start Consul so that it generates the configuration directory itself:

In the Consul configuration, we define the IP to bind to and the IP of the consul-server (192.168.69.50):

datacenter = "AlfaExploitDC01"
server = false
data_dir = "/var/db/consul"
bind_addr = "192.168.69.XX"
retry_join = ["192.168.69.50"]
enable_syslog = true
log_level = "INFO"

We restart the service:

We register the server as a web server:

{
  "service": {
    "name": "web",
    "port": 80
  }
}

Node name "PeanutBrain00.alfaexploit.com" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
Registered service: web

NOTE: For now, we will only register the node 00-08, 9 in total.

If we access the Consul server interface again, we can see the instances registered in the web service:
http://192.168.69.50:8500/ui/alfaexploitdc01/services

We can check the status of each instance:

DataPlaneAPI will have detected the registered servers and we will see that HAProxy has been reconfigured with a backend with the web nodes:
http://192.168.69.51:8404/stats

We can see how the backend has servers in maintenance, this is due to a limitation of DataPlaneAPI, the minimum number of servers in the backend will always be 10, if we have less than 10 instances registered in Consul, the remaining ones up to 10 will be servers in maintenance, remember that we have registered in Consul the web nodes 00-08, 9 in total.

According to the DataPlaneAPI documentation, it allows indicating the server_slots_base parameter, which is the number of servers with which to generate the backend:
http://192.168.69.51:5555/v2/docs#tag/ServiceDiscovery/operation/createConsul

But the software imposes a minimum of 10 servers to avoid unnecessary service restarts, in this open issue to the developer, you can see how he kindly responds to my question:
https://github.com/haproxytech/dataplaneapi/issues/265

We register a frontend with the backend generated by DataPlaneAPI:

consul-backend-<service-ip>-<service-port>-<service-name>
consul-backend-192.168.69.50-8500-web

We can see the documentation at the URL:
http://192.168.69.51:5555/v2/docs#tag/Frontend/operation/createFrontend

We obtain the current version of the configuration:

{"_version":2,"data":[]}

We register the frontend:

curl -u kr0m:PASSWORD \
       -H 'Content-Type: application/json' \
       -d '{
            "default_backend": "consul-backend-192.168.69.50-8500-web",
            "mode": "http",
            "name": "consul-frontend"
            }' 'http://192.168.69.51:5555/v2/services/haproxy/configuration/frontends?version=2' | jq

{
  "default_backend": "consul-backend-192.168.69.50-8500-web",
  "mode": "http",
  "name": "consul-frontend"
}

We check that it has been registered correctly:

{"_version":3,"data":[{"default_backend":"consul-backend-192.168.69.50-8500-web","mode":"http","name":"consul-frontend"}]}

Bind the frontend to port 80:

curl -u kr0m:PASSWORD \
       -H 'Content-Type: application/json' \
       -d '{
            "address": "192.168.69.51",
            "port": 80
       }' 'http://192.168.69.51:5555/v2/services/haproxy/configuration/binds?version=3&frontend=consul-frontend' | jq

{
  "address": "192.168.69.51",
  "port": 80
}

Verify that HAProxy is indeed bound to port 80 at the indicated IP:

USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS       PATH STATE   CONN STATE  
nobody   haproxy    29646 5  tcp4   192.168.69.51:80      *:*                                LISTEN

Now we register a new web server in Consul and we can see how we have reached 10 backend servers and 10 more have been added:

If we were to add servers, the slots would be filled up to the next 10, expanding the pool by 10 more. On the other hand, if we unregister them from Consul, when we reach 9 servers, the backend will be left with 10 servers, one of them in maintenance.

TROUBLESHOOTING:

We can check the HAProxy logs:

Start HAProxy in debug mode from the RC script:

: ${haproxy_flags:="-d -V -f ${haproxy_config} -p ${pidfile}"}

Start DataPlaneAPI in debug mode:

Manually start HAProxy:

Query information through the DataPlaneAPI API:

{
  "api": {
    "build_date": "0001-01-01T00:00:00.000Z",
    "version": " "
  },
  "system": {
    "cpu_info": {
      "num_cpus": 2
    },
    "hostname": "BlindSnipper.alfaexploit.com",
    "mem_info": {
      "dataplaneapi_memory": 53365520,
      "free_memory": 408793088,
      "total_memory": 8299794432
    },
    "os_string": "FreeBSD 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 #19 releng/13.1-n250155-514a191356c-dirty: Thu Sep  1 13:01:03 CEST 2022     root@MightyMax.alfaexploit.com:/usr/obj/usr/src/amd64.amd64/sys/KR0M-MINIMAL ",
    "time": 1662710775,
    "uptime": 539432
  }
}

[
  {
    "info": {
      "active_peers": 0,
      "busy_polling": 0,
      "bytes_out_rate": 0,
      "compress_bps_in": 0,
      "compress_bps_out": 0,
      "compress_bps_rate_lim": 0,
      "conn_rate": 0,
      "conn_rate_limit": 0,
      "connected_peers": 0,
      "cum_conns": 1997,
      "cum_req": 463,
      "cum_ssl_conns": 0,
      "curr_conns": 1,
      "curr_ssl_conns": 0,
      "dropped_logs": 0,
      "failed_resolutions": 0,
      "hard_max_conn": 5000,
      "idle_pct": 100,
      "jobs": 6,
      "listeners": 4,
      "max_conn": 5000,
      "max_conn_rate": 1,
      "max_pipes": 0,
      "max_sess_rate": 1,
      "max_sock": 10032,
      "max_ssl_conns": 0,
      "max_ssl_rate": 0,
      "max_zlib_mem_usage": 0,
      "mem_max_mb": 0,
      "nbthread": 2,
      "node": "BlindSnipper.alfaexploit.com",
      "pid": 31170,
      "pipes_free": 0,
      "pipes_used": 0,
      "pool_alloc_mb": 0,
      "pool_failed": 0,
      "pool_used_mb": 0,
      "process_num": 1,
      "processes": 1,
      "release_date": "2022-06-21",
      "run_queue": 0,
      "sess_rate": 0,
      "sess_rate_limit": 0,
      "ssl_backend_key_rate": 0,
      "ssl_backend_max_key_rate": 0,
      "ssl_cache_lookups": 0,
      "ssl_cache_misses": 0,
      "ssl_frontend_key_rate": 0,
      "ssl_frontend_max_key_rate": 0,
      "ssl_frontend_session_reuse": 0,
      "ssl_rate": 0,
      "ssl_rate_limit": 0,
      "stopping": 0,
      "tasks": 17,
      "total_bytes_out": 17103080,
      "ulimit_n": 10032,
      "unstoppable": 1,
      "uptime": 1957,
      "version": "2.6.1-f6ca66d",
      "zlib_mem_usage": 0
    },
    "runtimeAPI": "/var/run/haproxy.sock"
  }
]