Loki Log Management

The following is a guide outlining the steps to setup Loki for log management of a Tangle node. If you do not have Tangle node setup yet, please review the Tangle Node Quickstart setup guide here.

In this guide we will configure the following modules to scrape metrics from the running Tangle node.

Loki provides log aggregation system and metrics. Download (opens in a new tab)
Promtail is the agent responsible for gathering logs, and sending them to Loki.Download (opens in a new tab)

Let's first start by downloading the latest releases of the above mentioned modules (Loki, Promtail download pages).

This guide assumes the user has root access to the machine running the Tangle node, and following the below steps inside that machine.

1. Download Loki

AMD version:

AMD

curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.0/loki-darwin-amd64.zip"

ARM version:

ARM

curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.0/loki-darwin-arm64.zip"

2. Download Promtail

AMD version:

AMD

curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.0/promtail-darwin-amd64.zip"

ARM version:

ARM

curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.0/promtail-darwin-arm64.zip"

3. Extract the Downloaded Files:

unzip

unzip "loki-linux-amd64.zip" &&
unzip "promtail-linux-amd64.zip"

4. Copy the Extracted Files into /usr/local/bin:

sudo cp loki-linux-amd64 /usr/local/bin/ &&
sudo cp promtail-linux-amd64 /usr/local/bin/

5. Create Dedicated Users:

Now we want to create dedicated users for each of the modules we have installed:

useradd

sudo useradd --no-create-home --shell /usr/sbin/nologin loki &&
sudo useradd --no-create-home --shell /usr/sbin/nologin promtail

6. Create Directories for loki, and promtail:

mkdir

sudo mkdir /etc/loki &&
sudo mkdir /etc/promtail

7. Change the Ownership for all Directories:

We need to give our user permissions to access these directories:

chown

sudo chown loki:loki /usr/local/bin/loki-linux-amd64 &&
sudo chown promtail:promtail /usr/local/bin/promtail-linux-amd64

9. Finally, let's clean up these directories:

rm -rf ./loki-linux-amd64* &&
rm -rf ./promtail-linux-amd64*

The next series of steps will be configuring each service.

Configuration

For implementation examples, refer to our GitHub. (opens in a new tab).

Loki

Loki's configuration details what ports to listen to, how to store the logs, and other configuration options. There are many other config options for Loki, read more. (opens in a new tab)

Let's create the file:

nano

sudo touch /etc/loki/config.yml
sudo nano /etc/loki/config.yml

config.yaml

auth_enabled: false
 
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
 
ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0
 
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h
 
 
storage_config:
  boltdb:
    directory: /data/loki/index
 
  filesystem:
    directory: /data/loki/chunks
 
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
 
chunk_store_config:
  max_look_back_period: 0s
 
table_manager:
  retention_deletes_enabled: false
  retention_period: 0

Promtail

The Promtail configuration details what logs to send to Loki. In the below configuration we are indicating to send the logs to Loki from the /var/log/dkg directory. This directory can be changed based on what logs you want to pick up. There are many other config options for Promtail, refer to the Promtail documentation (opens in a new tab)

Let's create the file:

nano

sudo touch /etc/promtail/config.yml
sudo nano /etc/promtail/config.yml

config.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0
 
positions:
  filename: /data/loki/positions.yaml
 
clients:
  - url: http://localhost:3100/loki/api/v1/push
 
scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/dkg/*log

Service Setup

Loki

Create and open the Loki service file:

loki.service

sudo tee /etc/systemd/system/loki.service > /dev/null << EOF
[Unit]
  Description=Loki Service
  Wants=network-online.target
  After=network-online.target
 
[Service]
  User=loki
  Group=loki
  Type=simple
  ExecStart=/usr/local/bin/loki-linux-amd64 -config.file /etc/loki/config.yml
 
[Install]
WantedBy=multi-user.target
EOF

Promtail

Create and open the Promtail service file:

promtail.service

sudo tee /etc/systemd/system/promtail.service > /dev/null << EOF
[Unit]
  Description=Promtail Service
  Wants=network-online.target
  After=network-online.target
 
[Service]
  User=promtail
  Group=promtail
  Type=simple
  ExecStart=/usr/local/bin/promtail-linux-amd64 -config.file /etc/promtail/config.yml
 
[Install]
WantedBy=multi-user.target
EOF

Great! You have now configured all the services needed to run Loki.

Starting the Services

Launch a daemon reload to take the services into account in systemd:

daemon-reload

sudo systemctl daemon-reload

Next, we will want to start each service:

start service

sudo systemctl start loki.service &&
sudo systemctl start promtail.service

And check that they are working fine, one by one:

loki:

status

systemctl status loki.service

promtail:

status

systemctl status promtail.service

If everything is working adequately, activate the services!

enable

sudo systemctl enable loki.service &&
sudo systemctl enable promtail.service

Amazing! You have now successfully configured Loki for log management. Check out the Grafana documentation to create a Loki log dashboard!

Grafana Dashboard