Analytics with Grafana

Overview

Log Writing : Documents each request and associated metadata

Morgan (HTTP Logger Middleware)
            ↓
fs (Writes Morgan logs to .log file)
            ↓
Alloy (Scrapes metrics and logs, sends to Prometheus and Loki)
            ↓

Logs

Loki (Stores logs)

Metrics

Prometheus (Stores metrics)

Traces

Otel-collector (Scrapes trace data, sends to Tempo)
            ↓
Tempo (Stores trace data, queried by Grafana)

Visualisation of Data : Logs and metrics

                            ↓↓
Grafana (Dashboard and Graphs - Reads Prometheus and Loki)

---

Component Responsibilities

1. Morgan

• Middleware
• Captures request/response metadata (status codes, request time, IP address).
• Confugred in JSON format.

2. fs (Node.js file system logging)

• Local Persistence
• Ensures Morgan outputs are written to rotating log files stored locally.
• Enures that logs persist even if downstream systems are temporarily unavailable (like Alloy/Loki)

3. Alloy (Log and Metric collector)

• Unified Agent/Collector
• Otel (Open-Telemetry)-based
• Scrapes local logs and metrics.
• Sends logs to Loki for storage.
• Sends metrics to Prometheus for storage.

4. Loki

• Horizontal scalable log aggregation system.
• Stores logs indexed by labels
• Allows fast quering of logs.
• Acts as data-source for Grafana.

5. Prometheus

• Stores metrics in time-series database.
• Provides API for querying in Grafana using the PromQL language.
• Acts as data-source for Grafana.

6. Otel-Collector

• Data processing agent.
• Alloy is a fine-tuned version of Otel-collector.
• Processes trace data.
• Exports data to Tempo

7. Tempo

• Trace storage for backend.
• Ingests trace data (spans).
• Stores spans for optimised look-up.
• Acts as data-source for Grafana.

8. Grafana

• Unified portal for Loki (logs), Prometheus (metrics) and Tempo (traces).
• Provides custom dashboards for real-time monitoring.

Grafana must be downloaded separately and works on localhost:3000.

Prometheus metrics are queried in Grafana using PromQL, a query language designed for the monitoring of time series data. Names for each metric collected are set manually, and can be found in server.js in the Gateway repo,

// Counter for API requests
const httpRequestCounter = new client.Counter({
  name: 'api_gateway_requests_total_new',
  help: 'Total number of requests to API endpoints',
  labelNames: ['method', 'endpoint', 'status_code']
});
register.registerMetric(httpRequestCounter);

PromQL line to graph sum of total requests:

sum(rate(api_gateway_requests_total_new[1m]))

Custom dashboard configurations are read as JSON files - the current Grafana dashboard is stored as JSON in the API Gateway repo [Grafana -> Dashboard]

---

Summary

Component	Function	Output
Morgan	Captures API Requests	Structured log line
fs	Writes Morgan data to a local file	Persistent local log storage
Otel-collector	Scrapes and ships Traces data	Sends to Tempo
Alloy	Scrapes and ships logs and metrics	Sends to Loki and Prometheus
Tempo	Stores trace data	Traces
Loki	Stores logs	Queryable log data
Prometheus	Stores metrics	Time-series metrics
Grafana	Visualisation of logs, metrics and traces	Dashboards