PMM2 Scraping exporter through Socks proxy

Prometheus allows scraping exporters through a Socks proxy, which can be useful in different scenarios, for example when dealing with a highly restricted environment where direct access from the PMM server is not allowed, or when we do not want the final server to know the IP of our PMM server for some reason, the Socks server would be masking the IP of our PMM server.

There are many Socks proxies available, but in this case we will use Srelay as it is very easy to install and does not require any configuration.

We compile and install Srelay:

git clone https://git.code.sf.net/p/socks-relay/srelay socks-relay-srelay
cd socks-relay-srelay
./configure
make
cp srelay /usr/local/bin/

We start it:

srelay

We check that it is working and listening:

ps aux|grep srelay

nobody   20373  0.0  0.0 527204  1756 ?        Ssl  11:32   0:00 srelay

netstat -nputa|grep LISTEN|grep srelay

tcp        0      0 0.0.0.0:1080            0.0.0.0:*               LISTEN      12653/srelay          
tcp6       0      0 :::1080                 :::*                    LISTEN      12653/srelay

We configure firewall rules to not leave the Socks server open:

iptables -I INPUT 1 -p tcp --dport 1080 -j DROP
iptables -I INPUT 1 -s PMM_SERVER_IP -p tcp --dport 1080 -j ACCEPT

We save the rules and indicate that they should be loaded at startup:

/etc/init.d/iptables save
rc-update add iptables default

We daemonize the process:

vi /etc/local.d/srelay.start

/usr/local/bin/srelay

We assign the necessary permissions:

chmod 700 /etc/local.d/srelay.start

The Prometheus configuration to monitor a php-fpm through a Socks proxy would be as follows:

vi prometheusConf/prometheus.base.yml

scrape_configs:
- job_name: fpm-status
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - SERVER:8080
  basic_auth:
    username: admin
    password: PASSWORD
  proxy_url: socks5://SRELAY_SERVER_IP:1080

Now the PMM scrapes will go through our Socks server to access the exporter metrics. We can check on the php-fpm server how the requests come from the Socks server:

tcpdump -ni eth0 port 8080

06:48:17.337735 IP SRELAY_SERVER_IP.56866 > SERVER.8080: Flags [P.], seq 4290895866:4290896151, ack 2322227802, win 501, options [nop,nop,TS val 1415235347 ecr 4174654489], length 285: HTTP: GET /metrics HTTP/1.1  
06:48:17.337772 IP SERVER.8080 > SRELAY_SERVER_IP.56866: Flags [.], ack 285, win 505, options [nop,nop,TS val 4174714400 ecr 1415235347], length 0  
06:48:17.341789 IP SERVER.8080 > SRELAY_SERVER_IP.56866: Flags [P.], seq 1:696, ack 285, win 505, options [nop,nop,TS val 4174714404 ecr 1415235347], length 695: HTTP: HTTP/1.1 200 OK  
06:48:17.341870 IP SRELAY_SERVER_IP.56866 > SERVER.8080: Flags [.], ack 696, win 501, options [nop,nop,TS val 1415235351 ecr 4174714404], length 0

One way to manually access the metrics through the Socks proxy is by using Curl:

curl –user admin:PASSWORD http://SERVER:8080/metrics -x socks5h://SRELAY_SERVER_IP:1080