Skip to content
BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
.github
.github
 
 
helm
helm
 
 
infra
infra
 
 
middleware
middleware
 
 
.env.example
.env.example
 
 
.prettierignore
.prettierignore
 
 
LICENSE
LICENSE
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
docker-compose.yml
docker-compose.yml
 
 
start.sh
start.sh
 
 

Repository files navigation

Vantage

Observability Stack

Centralized structured logging, distributed tracing, metrics aggregation, and intelligent alerting - deployable as Docker Compose or a single Helm chart.

Architecture

                        ┌──────────────────────────────────────────────────────┐
                        │               Your Services                          │
                        │  (Python + vantage-py)  (Node.js + vantage-node)     │
                        └───────┬──────────────────────┬───────────────────────┘
                                │ OTLP (traces/metrics) │ Fluentd (logs)
                    ┌───────────▼───────────┐  ┌────────▼──────────┐
                    │   OTel Collector      │  │      Fluentd      │
                    │   :4317 (gRPC)        │  │   :24224 (TCP)    │
                    │   :4318 (HTTP)        │  │   :9880  (HTTP)   │
                    └──────┬────────┬───────┘  └────────┬──────────┘
                           │        │                    │
              ┌────────────▼─┐  ┌───▼──────────┐  ┌─────▼──────────────┐
              │   Jaeger     │  │  Prometheus  │  │  Elasticsearch     │
              │   :16686 UI  │  │  :9090       │  │  :9200             │
              │   Traces     │  │  Metrics     │  │  Logs              │
              └──────────────┘  └───┬──────────┘  └──────┬─────────────┘
                                    │                      │
                               ┌────▼──────────────────────▼───┐
                               │           Grafana  :3000      │
                               │  Dashboards · Alerts · Explore│
                               └───────────────────────────────┘
                                    │
                               ┌────▼──────────────┐
                               │   Alertmanager    │
                               │   :9093           │
                               │   Slack · Email   │
                               └───────────────────┘

Quick Start - Docker Compose

Prerequisites

  • Docker ≥ 24 + Docker Compose v2
  • 4 GB RAM minimum (Elasticsearch needs headroom)

1. Configure

git clone https://github.com/quantsingularity/Vantage.git
cd Vantage
cp .env.example .env
# Edit .env — set SLACK_WEBHOOK_URL and ALERT_EMAIL for alerting

2. Start the full stack

docker compose up -d

Allow ~60 seconds for Elasticsearch to become healthy, then verify:

Service URL Credentials
Grafana http://localhost:3000 admin / vantage_grafana
Kibana http://localhost:5601 (no auth)
Jaeger UI http://localhost:16686 (no auth)
Prometheus http://localhost:9090 (no auth)
Alertmanager http://localhost:9093 (no auth)

3. View pre-built dashboards

Open Grafana → Dashboards → Vantage folder:

  • API Latency & Error Rates - p50/p95/p99 per service + error rate
  • Infrastructure & Business Metrics - Kafka lag, DB query times, disk, custom events

Middleware Integration

Python - vantage-py

Install:

pip install -e middleware/python/[all]

Three lines of config (FastAPI):

from vantage_py import Vantage

vantage = Vantage(
    service_name  = "payment-service",
    environment   = "production",
    otel_endpoint = "http://otel-collector:4317",
)
vantage.instrument(app)   # your FastAPI/Starlette instance

From environment variables:

export VANTAGE_SERVICE_NAME=payment-service
export VANTAGE_ENVIRONMENT=production
export VANTAGE_OTEL_ENDPOINT=http://otel-collector:4317
export VANTAGE_FLUENTD_HOST=fluentd
vantage = Vantage.from_env()
vantage.instrument(app)

What gets auto-instrumented:

  • ✅ Distributed traces (every request gets a trace/span ID)
  • ✅ Prometheus metrics on /metrics (http_requests_total, http_request_duration_seconds, etc.)
  • ✅ Structured JSON logs with trace/span ID injected for correlation
  • ✅ Logs forwarded to Fluentd → Elasticsearch → Kibana
  • ✅ SQLAlchemy query tracing (pass db_engine=engine)
  • ✅ Outgoing httpx request tracing

Manual instrumentation:

logger = vantage.get_logger(__name__)
tracer = vantage.get_tracer()

# Structured logging — trace_id injected automatically
logger.info("Payment processed", extra={"amount": 250.0, "user_id": "u_123"})

# Manual span
with tracer.start_as_current_span("fraud-check") as span:
    span.set_attribute("transaction.id", txn_id)
    result = run_fraud_check(txn_id)

# Business event tracking → Prometheus + Grafana
vantage.track_business_event("payment_processed", status="success", value=250.0, metric_name="payment_amount")

Node.js - vantage-node

Install:

cd middleware/node && npm install

Three lines of config (Express):

const { Vantage } = require("vantage-node");

const vantage = new Vantage({
  serviceName: "payment-service",
  environment: "production",
  otelEndpoint: "http://otel-collector:4317",
});
vantage.instrument(app); // your Express instance

From environment variables:

const vantage = Vantage.fromEnv(); // reads VANTAGE_* env vars
vantage.instrument(app);

What gets auto-instrumented:

  • ✅ Distributed traces via @opentelemetry/instrumentation-express
  • ✅ Prometheus metrics on /metrics (request rate, latency, error rate)
  • ✅ Default Node.js metrics (heap, GC, event loop lag)
  • ✅ Structured Winston JSON logs with trace ID injection
  • ✅ Fluentd log forwarding
  • ✅ /health endpoint

Manual instrumentation:

const logger = vantage.getLogger("payments");
const tracer = vantage.getTracer();

// Structured log with trace correlation
logger.info("Payment processed", { amount: 250.0, userId: "u_123" });

// Manual span
tracer.startActiveSpan("fraud-check", (span) => {
  span.setAttribute("transaction.id", txnId);
  const result = runFraudCheck(txnId);
  span.end();
  return result;
});

// Business event
vantage.trackBusinessEvent(
  "payment_processed",
  "success",
  250.0,
  "payment_amount",
);

Alerting Rules

Four alert categories are pre-configured in Prometheus Alertmanager:

Alert Condition Severity Channel
ServiceDown up == 0 for 1 min 🔴 critical #vantage-critical + email
HighP99Latency p99 > 500ms for 5 min 🟡 warning #vantage-alerts
CriticalP99Latency p99 > 2s for 3 min 🔴 critical #vantage-critical
HighErrorRate 5xx rate > 1% for 5 min 🟡 warning #vantage-alerts
CriticalErrorRate 5xx rate > 5% for 3 min 🔴 critical #vantage-critical
DiskUsageHigh disk > 80% for 10 min 🟡 warning #platform-alerts
DiskUsageCritical disk > 95% for 5 min 🔴 critical #vantage-critical
KafkaConsumerLagHigh lag > 10k for 10 min 🟡 warning #data-eng-alerts

Configure Slack webhook in .env:

SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/HERE
ALERT_EMAIL=alerts@yourcompany.com

Prometheus Metrics Reference

All instrumented services expose these metrics on /metrics:

Metric Type Description
http_requests_total Counter Total requests by method/route/status/service
http_request_duration_seconds Histogram Request latency (p50/p95/p99 in Grafana)
http_requests_in_flight Gauge Active concurrent requests
http_request_size_bytes Histogram Request body sizes
db_query_duration_seconds Histogram DB query latency by operation
db_errors_total Counter DB errors by operation
vantage_business_events_total Counter Custom business events
vantage_business_value Histogram Business metric values (e.g. amounts)
nodejs_* Various Node.js runtime metrics (heap, GC, event loop)

Kibana Log Search

Logs arrive in Elasticsearch with index pattern vantage-logs-YYYY.MM.DD.

Create index pattern in Kibana:

  1. Go to Stack Management → Index Patterns
  2. Create pattern: vantage-logs-*
  3. Time field: @timestamp

Useful KQL queries:

# All errors from a service
service: "payment-service" AND level: "error"

# Trace-correlated logs (paste trace_id from Jaeger)
trace_id: "4bf92f3577b34da6a3ce929d0e0e4736"

# Slow requests
http.duration_ms > 500

# Failed payments
event_type: "payment_processed" AND status: "failed"

Helm Deployment

Install dependencies

cd helm
helm dependency update

Deploy to Kubernetes

helm install vantage . \
  --namespace observability \
  --create-namespace \
  --set global.environment=production \
  --set grafana.adminPassword=your-secure-password \
  --set prometheus.alertmanager.config.receivers[0].slack_configs[0].api_url=https://hooks.slack.com/...

Upgrade

helm upgrade vantage . --namespace observability -f production-values.yaml

Uninstall

helm uninstall vantage --namespace observability

Folder Structure

Vantage/
├── Makefile                    # make up / test / open-grafana / helm-install ...
├── docker-compose.yml          # Full 10-service local stack
├── .env.example                # All VANTAGE_* config variables
│
├── infra/
│   ├── elasticsearch/
│   │   └── bootstrap.sh        # ILM policy + index templates (run once)
│   ├── kibana/
│   │   └── setup.sh            # Saved searches: Errors, Slow, Trace correlation
│   ├── fluentd/
│   │   ├── Dockerfile          # Fluentd + elasticsearch + record-modifier plugins
│   │   └── fluent.conf         # Routes: vantage.** → ES, catch-all → ES-misc
│   ├── otel/
│   │   └── config.yml          # Receivers (OTLP gRPC+HTTP) → Jaeger + Prometheus
│   ├── prometheus/
│   │   ├── prometheus.yml      # Scrape configs for all services
│   │   └── rules/
│   │       └── alerts.yml      # 10 alert rules across 5 categories
│   ├── alertmanager/
│   │   └── alertmanager.yml    # Routing → Slack (default/critical/platform/data) + email
│   └── grafana/
│       └── provisioning/
│           ├── datasources/
│           │   └── datasources.yml     # Prometheus + Jaeger + Elasticsearch
│           └── dashboards/
│               ├── dashboard.yml
│               ├── api_latency.json    # Request rate, error %, p50/p95/p99
│               ├── infra_metrics.json  # Kafka lag, DB, disk, Redis, ES, business events
│               └── tracing_overview.json  # OTel collector spans, queue, Jaeger link
│
├── middleware/
│   ├── python/                 # vantage-py — pip-installable
│   │   ├── setup.py
│   │   ├── pyproject.toml
│   │   ├── requirements.txt
│   │   ├── Dockerfile.demo
│   │   ├── demo_service.py     # FastAPI demo (3-line integration)
│   │   ├── tests/
│   │   │   ├── conftest.py
│   │   │   └── test_middleware.py
│   │   └── src/vantage_py/
│   │       ├── __init__.py     # Public: Vantage, get_logger, track_business_event, trace_db
│   │       ├── core.py         # Vantage class — OTel SDK + FastAPI/Flask middleware
│   │       ├── config.py       # VantageConfig (constructor + from_env())
│   │       ├── logger.py       # VantageFormatter, Fluentd handler, trace ID injection
│   │       ├── metrics.py      # Prometheus counters/histograms + track_business_event()
│   │       └── db_tracer.py    # trace_db(), atrace_db(), @trace_query decorator
│   │
│   └── node/                   # vantage-node — npm package
│       ├── package.json
│       ├── README.md
│       ├── Dockerfile.demo
│       ├── tests/
│       │   └── vantage.test.js
│       └── src/
│           ├── index.js        # Public API exports
│           ├── index.d.ts      # TypeScript type declarations
│           ├── config.js       # VantageConfig (constructor + fromEnv())
│           ├── logger.js       # Winston + FluentdTransport + trace injection
│           ├── metrics.js      # prom-client counters/histograms + trackBusinessEvent()
│           ├── vantage.js      # Vantage class — Express middleware + OTel SDK
│           ├── db_tracer.js    # traceDb() + traceDbFn() helpers
│           └── demo.js         # Express demo (3-line integration)
│
└── helm/
    ├── Chart.yaml              # Chart metadata + sub-chart dependencies
    ├── values.yaml             # Full configurable defaults
    └── templates/
        ├── _helpers.tpl
        ├── otel-collector.yaml  # Deployment + Service + ConfigMap
        ├── fluentd.yaml         # Deployment + Service + ConfigMap
        ├── prometheus-rules.yaml # All 10 alert rules as ConfigMap
        ├── ingress.yaml         # Optional ingress for Grafana/Kibana/Jaeger
        └── NOTES.txt            # Post-install instructions

Connecting to Nexon (0.1)

Vantage is designed to instrument Nexon services automatically. In each Nexon service:

# In nexon/feature-store/src/main.py
from vantage_py import Vantage

vantage = Vantage.from_env()
vantage.instrument(app, db_engine=engine)

Set these env vars in the Nexon docker-compose.yml:

environment:
  - VANTAGE_SERVICE_NAME=nexon-feature-store
  - VANTAGE_ENVIRONMENT=production
  - VANTAGE_OTEL_ENDPOINT=http://vantage-otel-collector:4317
  - VANTAGE_FLUENTD_HOST=vantage-fluentd

Jaeger will automatically correlate traces across all Nexon services.

Running Tests

# Python middleware
cd middleware/python
pip install pytest -e .[all]
pytest tests/ -v

# Node.js middleware
cd middleware/node
npm install
npm test

About

Observability stack with OpenTelemetry, Prometheus, Grafana, Jaeger, and ELK, plus Python and Node.js middleware SDKs.

Resources

Readme
MIT license
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Contributors

Languages