Local analyzer (experimental)

The local analyzer is an experimental capability that explains what is going wrong with an application and what in its setup is worth improving. Detection rules find incidents and review the setup; a small open-source model served inside your cluster can rephrase the incident explanation, so analysis data stays on infrastructure you run. Everything it finds is listed on the Insights page. It is on by default on a fresh install and not part of the telark flagship surface.

If you do not need it, turn it off. The rest of telark is fully functional without it.

Defaults

The analyzer writes its explanations with a model runtime. The chart installs one (Ollama) by default, and a fresh install needs no dashboard step:

• The analyzer is enabled with the model granite4:350m; automatic analysis stays off.
• With app.ollama.autoPull=true (the default), the analyzer pulls the model right after it starts if the runtime lacks it. The runtime status shows the download progress.
• Air-gapped (app.ollama.autoPull=false), nothing is pulled: load the model on the runtime's volume yourself (see Modes).

To skip the runtime, for example because you point the analyzer at your own endpoint:

helm upgrade telark oci://ghcr.io/telark/charts/telark -n telark \
  --set app.ollama.enabled=false

Settings → Platform → Local analyzer in the dashboard changes the rest:

1. Pick the model. The page shows each model's licence; every model it lists is Apache-2.0.
2. Install model if the runtime does not have it yet.
3. Turn the local analyzer off or on, and optionally analyze automatically on incidents and recoveries.

The settings live on the GlobalConfig custom resource:

spec:
  ai:
    enabled: true           # default on a fresh install
    model: granite4:350m    # default
    autoAnalyze: false      # default

telark writes these defaults only when it creates the GlobalConfig; an upgrade keeps the settings you already have. Changing them requires the Owner level on settings. There is no cloud provider and no API key to configure, ever: the analyzer only talks to a runtime you run.

How a run works

• Triggers. With automatic analysis on, discovery queues a run when it records an incident or a recovery on an application. You can also run one on demand with Analyze in an insight's details panel or an application group header on the Insights page.
• Rules first. The analyzer makes four read-only reads — the application overview, its recent change history, its recent events, and the status of up to three workloads — and detection rules turn them into insights: image pull failures, crash loops, out-of-memory kills, scheduling failures, failing probes, resource pressure, stuck rollouts, and a change followed by degraded health. Each insight also gets a precise cause (for example image not found rather than just image pull). The cards appear in about 1–2 seconds.
• Probe restarts are crash loops. A liveness or startup probe that keeps restarting a container is reported as a crash loop (restarted by its liveness probe), not as a probe failure: the restarts are what takes the workload down. A readiness probe that keeps failing stays a probe failure.
• One short narration. The model then rewrites the title and summary of those cards in plain sentences, from the facts the rules found, in one call with no tools. A rewrite is kept only if it names the workload, keeps every name and number of the rule text and the phrase that names the cause, and adds no number and no symptom of another kind of problem; otherwise, or if the model is not ready yet, the cards keep the rule-written text and the run line reads rules only. The rule-written text is complete on its own: narration is optional polish.
• Setup review. After the run, the analyzer also reviews the application's setup and refreshes its recommendations (see Recommendations). The review does not delay the incident cards and never calls the model.
• Output. At most three insights per run, each tied to the evidence the reads returned. telark code — not the model — owns kinds, severities, ids, timestamps, evidence, and whether an insight is open or resolved. A recovery resolves open insights without a model call.
• Deep mode (opt-in). With --set services.analyzer.env.ANALYZER_MODE=deep, the model investigates on its own through the same four read-only tools, over several steps. It needs a model that supports tool calling and, in practice, a GPU: on a 2 vCPU node one step of qwen3:4b takes over three minutes.
• Live updates. The run line on the Insights page updates as a run progresses, without a page reload.

What an insight tells you

Every card carries the same parts:

• Title and summary — what is wrong and what it means, written by telark from the facts the rules found: for example web is down: image not found, then how many replicas are ready and one factual detail. When the problem began shortly after a recorded change, the summary names that change.
• Why — the likely cause, in a sentence or two.
• What to do — up to four concrete steps. When the incident began after a change, a last step points to the snapshot taken before it, in the application's Snapshots section.
• Details — collapsed by default: the facts behind the card (workload, container, image, exit code, probe, port, limits…), the evidence it links to, the raw event message, first and last seen, and how many runs saw it.

Severity (Info, Warning, Critical) and confidence (Low, Medium, High) are set by the rules. A rule that relies on a heuristic says so with Medium confidence. Secret values are never stored on a card; only names are.

Recommendations

Besides incidents, the analyzer reviews each application's setup with deterministic rules and shows what it finds as recommendations. Reviews read the same cluster objects as incident analysis plus a few namespace lists (see Read-only by construction); they never call the model.

The rules are grouped into families:

• When reviews run. After every analysis run (skipped while other analyses are queued, so it never delays them), and from a background sweep: applications whose configuration changed are reviewed first, then any application not reviewed for two hours, at most 20 applications a minute by default. The sweep runs only while the analyzer is enabled. The application's view shows Setup reviewed … ago, or Not reviewed yet.
• Only on complete reads. A rule fires only when every read it depends on succeeded in full. A failed or truncated read neither creates nor resolves its cards.
• Production. An application is treated as production when one of its namespaces, or the environment of a protection plan that covers it, matches the production pattern (by default names such as prod, production or prd). Production raises the single-replica and missing-budget findings from Info to Warning and enables the protection rules and the digest-pinning check.
• Usage window. The usage rules (near a limit, over- or under-provisioned, running hot without an autoscaler) need at least 12 usage samples spanning 12 hours, taken from the application's metrics at each review. Without metrics-server they stay silent.
• Lifecycle. A recommendation is Open when first found and Updated when its facts change. It resolves when a complete review no longer finds it, and opens again if it comes back. An application keeps its 40 most severe recommendations; dismissed ones don't count toward the 40.
• Dismiss. A recommendation that does not apply can be dismissed. It stays dismissed until its facts change, it resolves, or someone reopens it.

The environment variables that tune reviews are listed in Reference → Environment variables.

Recommendation rules

These rules are available today. Each one names the workload, Service, or application it applies to.

Reliability

Resources

Scaling

Security

Images

Configuration

Networking

Change risk

Protection

Consistency

Not yet available

Checks catalogued but not shipped yet, because they would be noisy or need data the analyzer does not read today, include missing preStop hooks, zone topology spread, measured CPU throttling, vertical autoscaler recommendations, image vulnerability scanning, ConfigMap changes without a rollout, Ingress backends, incidents that follow changes, drift, missing recent snapshots, and namespace-level quotas and priority classes.

The Insights page

Insights in the sidebar, right after Applications, lists the incidents and recommendations of every application. It is served by discovery, so it stays readable while the analyzer is off.

• Two tabs — Incidents and Recommendations, each labelled with how many are open and not dismissed. ?tab=recommendations opens the second one.
• Severity pills above the table filter by severity and show how many insights each severity holds under the other filters.
• Search by application, namespace, subject, or title.
• Filters — on Incidents: severity, state, kind, namespaces, and environment; on Recommendations: severity, family, triage, namespaces, and environment. The environment comes from the protection plans that cover each application; an application no plan names has none. ?app=<namespace>/<name> limits both tabs to one application.
• States — Open, Updated, Resolved, and Stale: an active insight not seen for a day. By default the page shows open, updated, and stale insights that are not dismissed.
• Sort by clicking a column header: insight, severity, application, namespace, kind or family, state, or last seen. 25, 50 or 100 rows a page; narrow screens drop the namespace, state, kind, and environment columns first.
• Group by application, namespace, or category (kind for incidents, family for recommendations). Each group header shows its size and per-severity counts across every page and folds on click; an application header also has Analyze and the last run line. Your browser remembers the choice.
• Select turns on bulk mode: pick rows, then Acknowledge or Dismiss them together. Rows the action does not apply to are skipped and counted.
• Refresh — the page checks for changes every 15 seconds while the browser tab is visible; an unchanged list costs almost nothing.

Clicking a row opens a details panel for that insight: its severity, category, kind and state, the application (linked) and namespace, Why, What to do (a step that names a setting has a copy button), Details (folded), when it was first seen, last seen and resolved, triage, and Analyze this app with the run line. Other findings for this app lists the application's other cards. Move through the list with the arrow buttons, the ↑ and ↓ keys, or k and j; Esc closes it. ?insight=<id> opens the panel directly.

Triage. Acknowledge marks an incident or recommendation as seen; it clears when the card resolves. Dismiss hides a recommendation that does not apply. Reopen clears either one.

Who can do what: the page has its own scope, insights. Viewing it needs the ReadOnly level on insights; Analyze and triage need the Contributor level, and the insights.analyzeinsights.deny and insights.triageinsights.deny rules withhold each one. Bulk mode is offered to the Owner level on insights. Custom roles see nothing here until they are granted insights; see Features → RBAC. Namespaces excluded in GlobalConfig are hidden, as everywhere else.

Modes

Where the model comes from is a chart setting:

• Connected (app.ollama.autoPull=true, the default). The in-cluster runtime downloads a missing model over HTTPS: while the analyzer is enabled, it asks for the model right after it starts and whenever the model changes. Only the runtime has that egress.
• Air-gapped (app.ollama.autoPull=false). The runtime never downloads anything; the network policy drops its outbound HTTPS. Load the model on the runtime's volume yourself, then pick it in Settings. The dashboard hides Install model and says so.
• Your own endpoint (app.ollama.runtimeUrl). Point the analyzer at a server you run that speaks the Ollama API — for example an open-source Ollama on a GPU host — and set app.ollama.enabled=false to skip the in-cluster runtime. No key, no Secret. The facts of each run are sent to that endpoint, so keep it on infrastructure you control, and set ANALYZER_NUM_THREAD to that host's cores.

helm upgrade telark oci://ghcr.io/telark/charts/telark -n telark \
  --set app.ollama.runtimeUrl=http://ollama.gpu.internal:11434

In every mode the models are open-weight and the runtime is open-source: no cloud provider, no API key.

Sizing

The chart README has the full profiles. The times below are per run, derived from benchmarks on 2 vCPU nodes (the default model narrated one insight in 4.9 s once loaded); the first run after a runtime restart adds a few seconds, and a busy node adds more.

Keep ANALYZER_NUM_THREAD equal to the runtime's CPU limit and never above the node's vCPU count; more threads than cores makes narration many times slower. The runtime has a memory request but no memory limit by default, so downloaded models held in the page cache never trigger a false out-of-memory refusal; the chart README describes the Guaranteed alternative.

Read-only by construction

The analyzer has no mutation tool and no write access to workloads. Its role grants only get and list on pods, events, Deployments, StatefulSets, DaemonSets, and ReplicaSets, and, for setup reviews, on four kinds in each application namespace:

• services — Services that select no pods or target a missing port,
• policy/poddisruptionbudgets — missing or blocking disruption budgets,
• autoscaling/horizontalpodautoscalers — autoscaler limits and missing requests,
• networking.k8s.io/networkpolicies — pods no network policy selects.

It also reads the protection plans and plan environments from telark's own API. It never reads the contents of Secrets or ConfigMaps, nodes, metrics, RBAC objects, or logs; rules that look at environment variables store their names, never their values. It respects the namespaces excluded in GlobalConfig.

A network policy admits only the analyzer to the in-cluster model runtime, and allows the runtime outbound HTTPS only while model auto-pull is on.

Failure handling

The analyzer is best-effort. Rule insights do not depend on the model: if the runtime is absent, unreachable, or still downloading, or the narration times out or returns something unusable, the run still finishes with the rule-written text. In deep mode, which needs the model for every step, a missing runtime or model or a timeout records the run as failed on the application's view of the Insights page. Recommendations never depend on the model; a failed read only skips the rules that need it. Either way nothing else is affected: discovery only queues work for it; protection plans, snapshots, and rollback never wait on it.

What this is not

• Not on the landing page as a pillar. It exists for teams that want it, not as a selling point.
• Not an audit feature. Insights are descriptive, not authoritative. Do not feed them into automation that takes action on the cluster.
• Not a remote service. There is no hosted model and no API key. Without a runtime, the analyzer writes rule insights only.
• Not a cloud AI feature. Every model it runs is open-weight and runs on a runtime you operate.

Related

• Reference → CRD reference — GlobalConfig
• Reference → Environment variables
• Reference → API reference
• Features → RBAC
• Getting started → System requirements