Metrics and the statistics page

A Prometheus endpoint and a live statistics page, both built in and both off until you ask for them. Neither accumulates anything: they are rendered from the balancer's own state at the moment they are read.

Two ways to see what SSLB3 is doing, aimed at different readers. The metrics endpoint is for a scraper and an alert; the statistics page is for a person with a question right now. They report the same state, so they cannot disagree.

Nothing is counted twice

Both are rendered from live state when they are asked, rather than from counters maintained alongside the real ones. It is the difference between a number that is true and a number that was true when something last remembered to update it — and it means there is no second set of state to drift, and nothing to reset.

The metrics endpoint

Its own plaintext listener, separate from the data plane, serving /metrics in the usual exposition format.

sslb.conf

[Prometheus]
Enabled = yes
BindAddress = ::
ListenPort = 9101
Path = /metrics

On Kubernetes it is a chart value, and the container port is already declared:

shell

helm upgrade sslb3 oci://docker.hacking.hu/public/sslb3-ingress \
  --set config.prometheus.enabled=true

What it reports

Thirty-seven metric families, in seven groups:

Group	What you get
Listeners and workers	Every listener with its address and protocol, what each has carried — connections and HTTP requests, split both by the version they arrived as (HTTP/1, HTTP/2, HTTP/3) and by the client’s address family (IPv4, IPv6) — connections per worker, and the configured ceiling. Enough to see the total load, whether it is spread evenly across workers, how much of it has moved to HTTP/3, and how much of it is IPv6.
Serverfarms and backends	Every farm with its balancer and layer 7 module, its backend count and how many are healthy; then each backend with its own health, weight and connection count.
Health checks	For every checked backend, its check type and the address the check connects to — which is not the backend’s own when a management port is used — plus consecutive failures and the seconds since it was last checked. Failures rising while the backend is still up is a backend about to be taken out; an age far beyond the interval is a checker that has stopped, leaving the status frozen.
Session affinity	Sessions held against the configured maximum, plus hits, misses and evictions — the three numbers that say whether affinity is working or quietly falling back to balancing.
Intrusion detection	Sources tracked and sources blocked, with connections and requests refused, blocks issued and observations recorded. The mode is a label, so an alert can tell detecting from enforcing.
Kernel fast path	Which facility is forwarding, whether the last reconcile succeeded, how many services moved, and per listener whether it was eligible and the reason if not — plus the connections the kernel carried, which SSLB3 never saw and which are therefore missing from every other counter.
Build and configuration	The version, and derived facts worth alerting on rather than reading from a config file — such as whether any farm still claims a host at layer 7.
Scripts	Whatever the request script publishes, under sslb3_script_*. For the API gateway: checks passed and failed by contract and operation, refusals by status, contracts loaded and body bytes held. See Metrics from scripts below.

Every family carries HELP text saying what it means and, where it is not obvious, why it is worth alerting on. A metric that needs the documentation open to interpret is a metric nobody will act on at three in the morning.

The ones worth an alert

Most of these describe configuration and are useful as labels to join against. A few are genuinely operational:

promql

# A farm with no healthy backend left.
sslb3_serverfarm_backends_healthy == 0

# A worker thread has died. The process keeps serving on the rest and /healthz
# already reports unhealthy, but this is the line to alert on.
sslb3_workers_alive < sslb3_workers

# The fast path stopped working. Traffic is fine -- SSLB3 serves what the kernel
# is not carrying -- but it is no longer accelerating anything.
sslb3_fastpath_healthy == 0

# A route quietly fell off the native path onto the script path.
sslb3_routes_requests == 0

# Approaching the connection ceiling.
sum(sslb3_worker_connections) / sslb3_max_connections > 0.8

# How much of the traffic HTTP/3 is actually carrying, once it is advertised.
sum(rate(sslb3_listener_requests_total{version="http3"}[5m]))
  / sum(rate(sslb3_listener_requests_total[5m]))

# The IPv6 share of connections. Watch it fall as clients follow Alt-Svc to a
# v4-only HTTP/3 address -- the same people, arriving over a different family.
sum(rate(sslb3_listener_connections_total{family="ipv6"}[5m]))
  / sum(rate(sslb3_listener_connections_total{family=~"ipv4|ipv6"}[5m]))

The controller's own metrics

A separate endpoint answering a different question: not what the balancer is doing, but whether the thing that configures it is still managing to. It is on by default, on port 9105, and needs no configuration.

The failure it exists for is a quiet one. A push the balancer refuses — one Ingress carrying a value it will not take — fails the same way on every pass, while the pod stays Running and Ready and the balancer goes on serving the configuration it already has. Endpoints go stale behind it, and nothing else in the deployment says so.

promql

# The controller has stopped configuring the balancer. The number to alert on:
# -1 means it has never managed a single push.
sslb3_ingress_last_apply_seconds > 300 or sslb3_ingress_last_apply_seconds < 0

# Something is being served from what it last said rather than what it says now,
# and has been for five minutes. A Secret or a namespace it has lost access to
# shows up here, not above. Alert on the age: a count above zero also happens
# when the API server is merely busy, and clears itself.
sslb3_ingress_stale_seconds > 300

# How many objects that is.
sslb3_ingress_stale_objects

# Passes accepted and refused, for the rate rather than the state.
rate(sslb3_ingress_applies_total[5m])
rate(sslb3_ingress_failures_total[5m])

The age matters for another reason. Endpoints are only reused from memory for ten minutes; after that the service is left out and its traffic stops. That is deliberate. Health checks are off under the controller because endpoint membership is the readiness signal, so nothing else would retire a backend — and a pod address kept for longer than that may well have been handed to a pod in another namespace. Certificates and credentials have no such limit: an old one is still right, only late.

Objects the controller refuses also get a Kubernetes Event on themselves, so kubectl describe ingress says why — the controller's log lives in its own namespace, which the person who wrote the Ingress usually cannot read. The same note about the same object is written once an hour at most, and one pass writes no more than twenty; the rest are in the log and get their Event on a later pass.

Metrics from scripts

A role written in Lua knows things about its traffic the engine does not: which contract a request matched, which check refused it. From 0.11.0 a script can say so, through the same endpoint and the same page. It counts in its own tables and every few seconds hands the engine the whole picture with one call, metrics.publish. The scrape exposes it under sslb3_script_*, so a script can never shadow one of the metrics above, and the page draws a panel for it.

The engine does the parts that are easy to get wrong. Every worker publishes separately, and the engine adds them up, taking the maximum where a gauge asks for it. It keeps a counter monotonic across a reload, which rebuilds every VM and so restarts every count. It refuses a publication that would break the scrape or grow without bound — a script labelling by client address gets an error back, not a ten-thousand-series exposition. Nothing is incremented per request on the engine's side, so a script's metrics cost the traffic nothing.

The API gateway is the first script to use it:

promql

# Responses failing their contract, per operation: a backend drifting.
sum by (contract, operation) (
  rate(sslb3_script_apigw_checks_total{check=~"response.*",result="failed"}[5m]))

# What the gateway answered instead of a backend, by status.
sum by (status) (rate(sslb3_script_apigw_refusals_total[5m]))

Writing your own is a few lines; the Lua interface reference has the shape and the limits.

The statistics page

A single self-contained page on its own listener, showing the same state arranged for reading. It answers "what is happening right now" without a query language.

sslb.conf

[Stats]
Enabled = yes
BindAddress = ::
ListenPort = 9102
RefreshInterval = 2

It updates over a WebSocket rather than by reloading. A page that reloads loses your scroll position and whatever you were reading, every couple of seconds — which makes it useless for the thing you actually want it for, which is watching a number move while you change something.

Workers, affinity and detection

Connections per worker, the affinity store’s hit rate, and what the intrusion detector has been doing. Placed at the top, because these are the numbers you check first when something feels wrong.

Listeners

Every socket that is accepting, with its protocol and the farm it is pinned to.

Serverfarms and backends

Each farm with how its traffic actually reaches its backends — parsed, spliced, or carried by the kernel — and every backend beneath it with its status, its health check and its connection count. A backend reads UP, DOWN, or PENDING while its first check is still out; the check shows its type and how it is going against the tolerance, with the address it connects to and the last error on hover.

Kernel fast path

Which forwarder is in use, how many services are eligible, how many are actually being forwarded, and for each one that is not, the reason. Byte columns appear only for a forwarder that measures bytes.

What the script publishes

A panel per script that publishes metrics — the API gateway’s, when it is the request script. A single figure is a tile; a labelled one is a table, busiest rows first, with a label of two to four values such as passed and failed turned into columns.

The page itself is a React application built with Vite and embedded in the balancer binary as one HTML file, so there is nothing to deploy alongside and no request to anywhere else.

Why both are off by default

Because they are unauthenticated, and what they serve is a description of your deployment: every backend address, every farm name, every listener. That belongs somewhere only your scraper and your operators can reach, and the safe default for something you have not thought about yet is off.

Neither is a path on the admin API, deliberately. The two want opposite things — the admin API changes configuration, so it is TLS-only and authenticated and always will be, while a scrape endpoint is read-only and called every few seconds by a robot that usually cannot be handed a credential. Requiring one is how metrics end up not being collected.

Reaching them

On Kubernetes, without exposing either:

shell

kubectl -n sslb3 port-forward deploy/sslb3-ingress 9101:9101   # metrics
kubectl -n sslb3 port-forward deploy/sslb3-ingress 9102:9102   # the page

The chart can also create a ServiceMonitor where the Prometheus Operator is running, so the endpoint is scraped without anything else being able to reach it.

And the log

Separate from both, and worth knowing about: access logging is per connection and off by default. It is also one of the conditions that keeps a service in SSLB3's path rather than the kernel's — a forwarded connection is one SSLB3 never sees, so it is not one it can log.