PMM2 Organizations in Grafana

PMM2 provides us with a complete monitoring solution for our servers, but there are times when this information needs to be accessed by an external team. Grafana solves this problem through organizations, so we can have specific dashboards per organization.

We are going to use a tool called wizzy as it makes it much easier for us to make copies of dashboards and then load them. It is a tool written in NodeJs so we install npm, in my case under FreeBSD:

pkg install npm
npm install -g wizzy

We create a directory where we will store the dashboards and all the information obtained from Grafana:

mkdir pmm2
cd pmm2/

We initialize the Git repository and configure the connection to the Grafana server:

git init
wizzy init
wizzy set grafana url http://grafana.alfaexploit.com/graph
wizzy set grafana username admin
wizzy set grafana password PASSWORD
wizzy status

We show the current organizations:

wizzy show orgs

Output:
[
  {
    "id": 1,
    "name": "Main Org."
  }
]

We make sure we are in the main org:

wizzy switch org 1

{
  "message": "Active organization changed"
}
✔ Org switched to 1 successfully.

We download the configured datasources:

wizzy import datasources

✔ Total datasources imported: 3
✔ Datasources import successful.

If we only want one:

wizzy import datasource DATASOURCE_NAME

We consult the datasources we have downloaded:

wizzy summarize datasources

Output:
┌──────────────────────────────┬──────────────────────────────┐
│ Datasource Name              │ Datasource Type              │
├──────────────────────────────┼──────────────────────────────┤
│ ClickHouse                   │ vertamedia-clickhouse-datas… │
├──────────────────────────────┼──────────────────────────────┤
│ PostgreSQL                   │ postgres                     │
├──────────────────────────────┼──────────────────────────────┤
│ Prometheus                   │ prometheus                   │
└──────────────────────────────┴──────────────────────────────┘
✔ Total datasources: 3
✔ Showed datasources summary successfully.

We list the available dashboards:

wizzy list dashboards

Output:
┌──────────────────────────────┬──────────────────────────────┬──────────────────────────────┬──────────────────────────────┬──────────────────────────────┐
│ Title                        │ Slug                         │ ID / UID                     │ Type                         │ folderTitle                  │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Insight                      │ insight                      │ 3 / QxVaaQuWk                │ dash-folder                  │                              │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MongoDB                      │ mongodb                      │ 2 / mxVaawXWz                │ dash-folder                  │                              │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL                        │ mysql                        │ 5 / vBV-aQXZz                │ dash-folder                  │                              │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ OS                           │ os                           │ 7 / QL4a-wXWk                │ dash-folder                  │                              │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PMM                          │ pmm                          │ 6 / kY4a-QXWz                │ dash-folder                  │                              │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PostgreSQL                   │ postgresql                   │ 1 / G14aawuZk                │ dash-folder                  │                              │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Query Analytics              │ query-analytics              │ 4 / A-V-awXZk                │ dash-folder                  │                              │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Advanced Data Exploration    │ advanced-data-exploration    │ 71 / 1oz9QMHmk               │ dash-db                      │ Insight                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ CPU Utilization Details      │ cpu-utilization-details      │ 87 / node-cpu                │ dash-db                      │ OS                           │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Disk Details                 │ disk-details                 │ 111 / node-disk              │ dash-db                      │ OS                           │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Home Dashboard               │ home-dashboard               │ 66 / pmm-home                │ dash-db                      │ Insight                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Memory Details               │ memory-details               │ 103 / node-memory            │ dash-db                      │ OS                           │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MongoDB Cluster Summary      │ mongodb-cluster-summary      │ 95 / mongodb-cluster-summary │ dash-db                      │ MongoDB                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MongoDB InMemory Details     │ mongodb-inmemory-details     │ 85 / mongodb-inmemory        │ dash-db                      │ MongoDB                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MongoDB Instance Summary     │ mongodb-instance-summary     │ 84 / mongodb-instance-summa… │ dash-db                      │ MongoDB                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MongoDB Instances Compare    │ mongodb-instances-compare    │ 109 / mongodb-instance-comp… │ dash-db                      │ MongoDB                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MongoDB Instances Overview   │ mongodb-instances-overview   │ 106 / mongodb-instance-over… │ dash-db                      │ MongoDB                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MongoDB MMAPv1 Details       │ mongodb-mmapv1-details       │ 72 / mongodb-mmapv1          │ dash-db                      │ MongoDB                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MongoDB ReplSet Summary      │ mongodb-replset-summary      │ 91 / mongodb-replicaset-sum… │ dash-db                      │ MongoDB                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MongoDB RocksDB Details      │ mongodb-rocksdb-details      │ 90 / mongodb-rocksdb         │ dash-db                      │ MongoDB                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MongoDB WiredTiger Details   │ mongodb-wiredtiger-details   │ 80 / mongodb-wiredtiger      │ dash-db                      │ MongoDB                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL Amazon Aurora Details  │ mysql-amazon-aurora-details  │ 70 / mysql-amazonaurora      │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL Command/Handler Count… │ mysql-command-handler-count… │ 68 / mysql-commandhandler-c… │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL InnoDB Compression De… │ mysql-innodb-compression-de… │ 67 / mysql-innodb-compressi… │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL InnoDB Details         │ mysql-innodb-details         │ 94 / mysql-innodb            │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL Instance Summary       │ mysql-instance-summary       │ 77 / mysql-instance-summary  │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL Instances Compare      │ mysql-instances-compare      │ 108 / mysql-instance-compare │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL Instances Overview     │ mysql-instances-overview     │ 105 / mysql-instance-overvi… │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL MyISAM/Aria Details    │ mysql-myisam-aria-details    │ 89 / mysql-myisamaria        │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL MyRocks Details        │ mysql-myrocks-details        │ 75 / mysql-myrocks           │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL Performance Schema De… │ mysql-performance-schema-de… │ 82 / mysql-performance-sche… │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL Query Response Time D… │ mysql-query-response-time-d… │ 76 / mysql-queryresponsetime │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL Replication Summary    │ mysql-replication-summary    │ 98 / mysql-replicaset-summa… │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL Table Details          │ mysql-table-details          │ 96 / mysql-table             │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL TokuDB Details         │ mysql-tokudb-details         │ 93 / mysql-tokudb            │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL User Details           │ mysql-user-details           │ 86 / mysql-user              │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ MySQL Wait Event Analyses D… │ mysql-wait-event-analyses-d… │ 83 / mysql-waitevents-analy… │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Network Details              │ network-details              │ 99 / node-network            │ dash-db                      │ OS                           │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Node Summary                 │ node-summary                 │ 88 / node-instance-summary   │ dash-db                      │ OS                           │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Node Temperature Details     │ node-temperature-details     │ 112 / node-temp              │ dash-db                      │ OS                           │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Nodes Compare                │ nodes-compare                │ 100 / node-instance-compare  │ dash-db                      │ OS                           │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Nodes Overview               │ nodes-overview               │ 102 / node-instance-overview │ dash-db                      │ OS                           │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ NUMA Details                 │ numa-details                 │ 97 / node-memory-numa        │ dash-db                      │ OS                           │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PMM Add Instance             │ pmm-add-instance             │ 65 / pmm-add-instance        │ dash-db                      │ PMM                          │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PMM Inventory                │ pmm-inventory                │ 63 / pmm-inventory           │ dash-db                      │ PMM                          │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PMM Query Analytics          │ pmm-query-analytics          │ 62 / pmm-qan                 │ dash-db                      │ Query Analytics              │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PMM Settings                 │ pmm-settings                 │ 64 / pmm-settings            │ dash-db                      │ PMM                          │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PostgreSQL Instance Summary  │ postgresql-instance-summary  │ 101 / postgresql-instance-s… │ dash-db                      │ PostgreSQL                   │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PostgreSQL Instances Compare │ postgresql-instances-compare │ 110 / postgresql-instance-c… │ dash-db                      │ PostgreSQL                   │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PostgreSQL Instances Overvi… │ postgresql-instances-overvi… │ 107 / postgresql-instance-o… │ dash-db                      │ PostgreSQL                   │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Processes Details            │ processes-details            │ 104 / node-cpu-process       │ dash-db                      │ OS                           │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Prometheus                   │ prometheus                   │ 81 / prometheus              │ dash-db                      │ Insight                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Prometheus Exporter Status   │ prometheus-exporter-status   │ 69 / prometheus-status       │ dash-db                      │ Insight                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ Prometheus Exporters Overvi… │ prometheus-exporters-overvi… │ 79 / prometheus-overview     │ dash-db                      │ Insight                      │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ ProxySQL Instance Summary    │ proxysql-instance-summary    │ 78 / proxysql-instance-summ… │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PXC/Galera Cluster Summary   │ pxc-galera-cluster-summary   │ 92 / pxc-cluster-summary     │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PXC/Galera Node Summary      │ pxc-galera-node-summary      │ 73 / pxc-node-summary        │ dash-db                      │ MySQL                        │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ PXC/Galera Nodes Compare     │ pxc-galera-nodes-compare     │ 74 / pxc-nodes-compare       │ dash-db                      │ MySQL                        │
└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘
✔ Total dashboards: 58
✔ Displayed dashboards list successfully.

We download the dashboards locally:

wizzy import dashboards

Importing 51 dashboards:
✔ Dashboard advanced-data-exploration imported successfully.
✔ Dashboard cpu-utilization-details imported successfully.
✔ Dashboard disk-details imported successfully.
✔ Dashboard home-dashboard imported successfully.
✔ Dashboard memory-details imported successfully.
✔ Dashboard mongodb-cluster-summary imported successfully.
✔ Dashboard mongodb-inmemory-details imported successfully.
✔ Dashboard mongodb-instance-summary imported successfully.
✔ Dashboard mongodb-instances-compare imported successfully.
✔ Dashboard mongodb-instances-overview imported successfully.
✔ Dashboard mongodb-mmapv1-details imported successfully.
✔ Dashboard mongodb-replset-summary imported successfully.
✔ Dashboard mongodb-rocksdb-details imported successfully.
✔ Dashboard mongodb-wiredtiger-details imported successfully.
✔ Dashboard mysql-amazon-aurora-details imported successfully.
✔ Dashboard mysql-command-handler-counters-compare imported successfully.
✔ Dashboard mysql-innodb-compression-details imported successfully.
✔ Dashboard mysql-innodb-details imported successfully.
✔ Dashboard mysql-instance-summary imported successfully.
✔ Dashboard mysql-instances-compare imported successfully.
✔ Dashboard mysql-instances-overview imported successfully.
✔ Dashboard mysql-myisam-aria-details imported successfully.
✔ Dashboard mysql-myrocks-details imported successfully.
✔ Dashboard mysql-performance-schema-details imported successfully.
✔ Dashboard mysql-query-response-time-details imported successfully.
✔ Dashboard mysql-replication-summary imported successfully.
✔ Dashboard mysql-table-details imported successfully.
✔ Dashboard mysql-tokudb-details imported successfully.
✔ Dashboard mysql-user-details imported successfully.
✔ Dashboard mysql-wait-event-analyses-details imported successfully.
✔ Dashboard network-details imported successfully.
✔ Dashboard node-summary imported successfully.
✔ Dashboard node-temperature-details imported successfully.
✔ Dashboard nodes-compare imported successfully.
✔ Dashboard nodes-overview imported successfully.
✔ Dashboard numa-details imported successfully.
✔ Dashboard pmm-add-instance imported successfully.
✔ Dashboard pmm-inventory imported successfully.
✔ Dashboard pmm-query-analytics imported successfully.
✔ Dashboard pmm-settings imported successfully.
✔ Dashboard postgresql-instance-summary imported successfully.
✔ Dashboard postgresql-instances-compare imported successfully.
✔ Dashboard postgresql-instances-overview imported successfully.
✔ Dashboard processes-details imported successfully.
✔ Dashboard prometheus imported successfully.
✔ Dashboard prometheus-exporter-status imported successfully.
✔ Dashboard prometheus-exporters-overview imported successfully.
✔ Dashboard proxysql-instance-summary imported successfully.
✔ Dashboard pxc-galera-cluster-summary imported successfully.
✔ Dashboard pxc-galera-node-summary imported successfully.
✔ Dashboard pxc-galera-nodes-compare imported successfully.
✔ 51 dashboards imported successfully.

If we only want one:

wizzy import dashboard DASHBOARD_SLUG

We create a new organization:

wizzy create org alfaexploit

Output:
{
  "message": "Organization created",
  "orgId": 14
}
✔ Created Grafana org alfaexploit successfully.

We show the organizations again:

wizzy show orgs

Output:
[
  {
    "id": 1,
    "name": "Main Org."
  },
  {
    "id": 14,
    "name": "alfaexploit"
  }
]

We switch organizations to upload the datasources/dashboards:

wizzy switch org 14

Output:
{
  "message": "Active organization changed"
}
✔ Org switched to 14 successfully.

We add the datasource:

wizzy export datasource Prometheus

Datasource does not exists in Grafana.
Trying to create a new datasource.
Output:
{
  "datasource": {
    "id": 11,
    "orgId": 14,
    "name": "Prometheus",
    "type": "prometheus",
    "typeLogoUrl": "",
    "access": "proxy",
    "url": "http://127.0.0.1:9090/prometheus/",
    "password": "",
    "user": "",
    "database": "",
    "basicAuth": false,
    "basicAuthUser": "",
    "basicAuthPassword": "",
    "withCredentials": false,
    "isDefault": true,
    "jsonData": {
      "httpMethod": "POST",
      "keepCookies": [],
      "timeInterval": "1s"
    },
    "secureJsonFields": {},
    "version": 1,
    "readOnly": false
  },
  "id": 11,
  "message": "Datasource added",
  "name": "Prometheus"
}
✔ Datasource Prometheus export successful.

In my case, I am only interested in a few dashboards, before loading them, we remove the links from the query analytics and others since we do not want anyone to have access:

for DASH in OS/cpu-utilization-details OS/memory-details OS/disk-details MySQL/mysql-instance-summary MongoDB/mongodb-instance-summary; do
  jq 'del(.links)' dashboards/$DASH.json > /tmp/grafana.json
  mv /tmp/grafana.json dashboards/$DASH.json
done

We will have to modify the queries of the dashboards so that only the servers we want to be visible in that organization appear in the drop-downs. Depending on the dashboard, we will have to modify it in one way or another. In my case, I only want the servers called kr0mtest* to be visible:

vi dashboards/OS/cpu-utilization-details.json

"label_values({__name__=~\"node_boot_time_seconds|rdsosmetrics_timestamp\"}, node_name)"
"label_values({__name__=~\"node_boot_time_seconds|rdsosmetrics_timestamp\", node_name=~\"kr0mtest.*\"}, node_name)"

vi dashboards/OS/memory-details.json

"label_values(up{node_type=~\"generic|remote_rds|container|remote\"}, node_name)"
"label_values(up{node_type=~\"generic|remote_rds|container|remote\", node_name=~\"kr0mtest.*\"}, node_name)"

vi dashboards/OS/disk-details.json

"label_values({__name__=~\"node_disk_reads_completed_total|aws_rds_disk_queue_depth_average|rdsosmetrics_diskIO_readLatency\"}, node_name)")
"label_values({__name__=~\"node_disk_reads_completed_total|aws_rds_disk_queue_depth_average|rdsosmetrics_diskIO_readLatency\", node_name=~\"kr0mtest.*\"}, node_name)"

vi dashboards/MySQL/mysql-instance-summary.json

"label_values(mysql_up{node_name=~\"$node_name\"}, node_name)"
"label_values(mysql_up{node_name=~\"$node_name\", node_name=~\"kr0mtest.*\"}, node_name)"

vi dashboards/MongoDB/mongodb-instance-summary.json

"label_values(mongodb_up{node_name=~\"$node_name\"}, node_name)"
"label_values(mongodb_up{node_name=~\"$node_name\", node_name=~\"kr0mtest.*\"}, node_name)"

We upload the dashboards:

wizzy export dashboard DASHBOARD_SLUG
wizzy export dashboard OS/cpu-utilization-details
wizzy export dashboard OS/memory-details
wizzy export dashboard OS/disk-details
wizzy export dashboard MySQL/mysql-instance-summary
wizzy export dashboard MongoDB/mongodb-instance-summary

Wizzy does not allow user management, so we will create them manually. We could use the API, but since we are going to create few users, I don’t consider it necessary to generate an API key: