CLI, dashboard, and metrics

Applies to: baseline cluster observation plus newer development diagnostics. Capacity, scans, compatibility, and sync-checkpoint views need matching newer tools and servers.

Inspect from the command line

Install tools on an operator host that can reach every advertised management address. Standalone nodes do not expose the sharded-cluster management API.

smkv-ctl --seed 10.0.0.11:7381 info
smkv-ctl --seed 10.0.0.11:7381 partitions
smkv-ctl --seed 10.0.0.11:7381 health --json
smkv-ctl --seed 10.0.0.11:7381 health --watch 5

Inspection is read-only. Degraded health returns exit code 2; missing observations are reported as unavailable. Samples across nodes are not atomic: small transient replica lag during writes is not by itself evidence of failure. Investigate sustained lag, missing owners, fenced writes, and capacity rejections.

Primary entries and replica copies are counted separately. Indexed entries may include expired records pending cleanup. Retained log size includes old versions and tombstones. Index memory is an estimate, not measured process RSS.

Start the dashboard

Run the backend on a management host with cluster connectivity:

smkv-web --seed 10.0.0.11:7381 \
  --metrics n1=10.0.0.11:9108 \
  --metrics n2=10.0.0.12:9108 \
  --metrics n3=10.0.0.13:9108

It listens on 127.0.0.1:8080 by default. For a remote management host:

ssh -N -L 8080:127.0.0.1:8080 operator@MANAGEMENT_HOST

Open http://localhost:8080. A non-loopback bind requires SMKV_WEB_TOKEN with at least 32 printable ASCII characters, plus an SSH tunnel or trusted HTTPS proxy. Database transport credentials are configured separately with --security-config.

Overview shows live read/write QPS; Activity shows request metrics. Choose a browser refresh interval using the refresh selector. Browser refresh reads cached observations, not a new cluster-wide query. Collection intervals remain independently configured. Graph gaps represent unavailable data; refreshing or reopening the page does not provide durable historical storage. Actionable health badges open further findings.

Expose Prometheus metrics

Start each server with a private --metrics-bind IP:9108. Example Prometheus job:

scrape_configs:
  - job_name: smkv
    scrape_interval: 10s
    static_configs:
      - targets: ['10.0.0.11:9108', '10.0.0.12:9108', '10.0.0.13:9108']

The HTTP /metrics listener reports request outcomes and latency, worker queues, storage, replication, and readiness. It does not inherit database transport TLS/authentication. Limit access to your monitoring network.

Use counter rates for throughput and error rates. Server processing latency is not the same as end-to-end client latency, which also includes network, queuing, and client behavior. Use Prometheus or another monitoring system for persistent history and alerting.

Daily checks

Check capacity headroom, readiness, sustained replica lag, request errors, and p99 latency against your own service objectives. After a restart or topology change, verify all partition owners and both copies before another maintenance step. For a capacity rejection, use smkv-ctl capacity and the dashboard Nodes details. Experimental sync checkpoint status appears under smkv-ctl checkpoints and Activity.