Grafana
High Availability in Practice
### Database Migration
By default, Grafana is configured with **Sqlite** database, based on a local file.
We need to migrate it to **PostgreSQL** to enable remote access.
```bash
# 1. install PostgreSQL
(left out)
# 2. create user and db for grafana
su - postgres
psql
create user grafana with password 'grafana@{your-password}';
create database grafanadb with owner grafana;
# now connection string is
# postgres://grafana:grafana@{your-password}@{database-ip}:5432/grafanadb
# 3. create initial tables in grafanadb
# 3.1 stop grafana
service grafana-server stop
vi /etc/grafana/grafana.ini
# configure the following line
[database]
type = postgres
url = {connection_string}
ssl_mode = disable
# end of configuration
# 3.2. restart grafana-server
service grafana-server start
systemctl status grafana-server.service --no-pager --full
# 4. migrate data from sqlite to postgresql
# 4.1 download and install grafana-migrate tool
# download
https://github.com/wbh1/grafana-sqlite-to-postgres/releases/download/v2.0.1/grafana-migrate_linux_amd64-v2.0.1
# and then scp it to ~ and copy to /usr/local/bin/grafana-migrate
(left out)
# 4.2 run grafana-migrate
chmod +x /usr/local/bin/grafana-migrate
grafana-migrate /var/lib/grafana/grafana.db {connection_string}
# NOTICE
# if you see some outputs like `pq: relation "org" does not exist`
# it means that grafana is creating the tables.
# most likely because grafana is not able to connect to postgresql
# check log /var/log/grafana/grafana.log for more details
```
### Disable Built-in SSL
Considering that we use nginx to configure certificates, the built-in SSL is not necessary.
But note that `root_url` should be set as an `https` url, because this is for link generation.
```ini
# /etc/grafana/grafana.ini
[server]
# protocol = https
# http_port = 443
domain = example.sub.domain.microsoftapp.net
root_url = https://example.sub.domain.microsoftapp.net:443
cert_file = /etc/grafana/certs/demo-site.crt
cert_key = /etc/grafana/certs/demo-site.rsa
```
### Configure AAD
Actually, no changes from the old configuration.
```ini
[auth.azuread]
name = Azure AD
enabled = true
allow_sign_up = true
client_id = ***
client_secret = ***
scopes = openid email profile
auth_url = https://login.microsoftonline.com/****/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/****/oauth2/v2.0/token
allowed_domains = microsoft.com microsoft.onmicrosoft.com
allowed_groups = *** *** ***
```
### Prepare for certificates
Upload the certificates to the server.
We will use the following certificate and key:
```bash
/etc/grafana/certs/demo-site.crt
/etc/grafana/certs/demo-site.rsa
```
### Configure Load Balancer
Any reverse proxy is OK. Take Nginx as an example.
```bash
vi /etc/nginx/sites-available/grafana-ha
```
Configurations
```nginx
# == Grafana start ==
# this is required to proxy Grafana Live WebSocket connections.
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream grafana {
# !!! when a new grafana instance is created,
# please update the following lines
server localhost:3000;
}
server {
# listen 80;
listen 443 ssl;
server_name grafana-int.demo-site.microsoftapp.net;
ssl_certificate /etc/grafana/certs/demo-site.crt;
ssl_certificate_key /etc/grafana/certs/demo-site.rsa;
location / {
proxy_set_header Host $http_host;
proxy_pass http://grafana;
}
# Proxy Grafana Live WebSocket connections.
location /api/live/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $http_host;
proxy_pass http://grafana;
}
}
# == Grafana end ==
```
### Configure HSTS
If we use AAD, we must enable HSTS to ensure HTTPS access.
```nginx
server {
listen 80;
listen [::]:80;
server_name grafana-int.demo-site.microsoftapp.net;
return 301 https://$server_name$request_uri;
}
```
Enable new site and check status
```bash
ln -s /etc/nginx/sites-available/grafana-ha /etc/nginx/sites-enabled/grafana-ha
nginx -t && service nginx restart && service nginx status
```
### Install new Grafana instance
Make sure all instances are of same version.
```bash
# view the current version
curl https://example.sub.domain.microsoftapp.net/api/health
# start sample output
{
"commit": "29e75ad97b",
"database": "ok",
"version": "7.4.2"
}
# end of sample output
```
Be aware that what plugins are installed:
```bash
sudo ls /var/lib/grafana/plugins/
# sample output
agenty-flowcharting-panel auxmoney-waterfall-panel bilibala-echarts-panel briangann-datatable-panel
gapit-htmlgraphics-panel grafana-groupedbarchart-panel grafana-piechart-panel grafana-worldmap-panel
isaozler-paretochart-panel marcusolsson-treemap-panel
```
### Install new Grafana instance
Download and Install Grafana
```bash
# 1. download install
sudo apt-get install -y adduser libfontconfig1
# NOTICE: VM of Azure may have no access to outside internet.
# In this case, you need to download manually and scp it to VM.
wget https://dl.grafana.com/oss/release/grafana_7.4.2_amd64.deb
sudo dpkg -i grafana_7.4.2_amd64.deb
# 2. configure grafana
sudo vi /etc/grafana/grafana.ini
# make sure ALL instances are configured exactly the same
(left out)
```
+ Copy aforementioned **grafana.ini** to this new instances. (details omitted)
+ copy **plugins** to this new instances. (details omitted)
### All done.
Now visit the server name provided in nginx configuration, and check if it is working.
#### Validate if each node works
```bash
curl https://{node-ip}:{node-port}/api/health
```
Or comment out the not corresponding upstreams in nginx configuration
to check if specific node is working
```nginx
upstream grafana {
# server 1.2.3.4:3000;
server 1.2.3.5:3000;
# server 1.2.3.6:3000;
}
```