Platform Health Specialist

Role definition

Why this specialist exists

The Platform Health Specialist exists because incident response alone is too late. Its job is to continuously build a clear picture of whether the platform is safe, reliable, maintainable, and sensibly provisioned before users or engineers feel pain.

It looks at the clusters and the applications running on them as one operational system. It does not replace the Incident Specialist. It asks a different question: what is likely to hurt us soon if nobody acts?

Read-only GKE/GCP-oriented Preventive Cluster-aware

Initial implementation should focus on Kubernetes and GKE runtime evidence, then enrich that view with GCP dependencies such as MongoDB, Pub/Sub, PostgreSQL, Cloud Storage buckets, IAM, DNS, certificates, networking, and expiry or rotation metadata for operational credentials and licences.

What It Does

Builds a preventive health view per cluster: dev, stage, prd, and management.
Checks workload readiness for production use: probes, resources, rollout health, image hygiene, replicas, disruption tolerance, and ownership labels.
Looks for early reliability risks such as pending pods, node pressure, failing jobs, unstable rollouts, unhealthy system components, certificate expiry, token or licence expiry, DNS issues, storage pressure, and fragile dependencies.
Correlates Kubernetes runtime state with ArgoCD desired state, observability signals, and known platform standards.
Highlights drift from platform conventions before that drift becomes an incident.

What It Does Not Do

It does not automatically change deployments, secrets, RBAC, network policies, IAM, or cloud resources.
It does not show secret values, private keys, token contents, PAT contents, licence keys, or passwords. Only metadata needed for operational renewal should be visible.
It does not replace alerting, incident response, security review, or human ownership.
It does not judge every dev workload as production-critical. Dev and stage need different interpretation than prd.
It does not pretend Kind has the same signal richness as real GKE clusters.
It does not call something unsafe without evidence, severity, scope, and a practical next action.

Why It Matters

Preventive inspection turns platform knowledge into operational leverage. SREs should not discover missing probes, fragile Redis dependencies, exhausted node pools, unmanaged secrets, or risky image sources only after an outage starts.

For management, it provides confidence that the platform is being watched for avoidable failure and waste. For SREs, it reduces surprise, noise, and repetitive manual checking.

Cluster Scope

prd

Highest operational weight. Workload health, dependency health, rollout safety, capacity, security posture, and external GCP dependencies must all be evaluated seriously.

stage

Always-on environment. Good place to catch drift, risky releases, missing dependencies, and near-production configuration mistakes before prd.

dev

Lower workload weight because it may be off outside office hours and contain normal development churn. Still important for cluster health, shared components, security hygiene, and configuration drift.

management

Critical control-plane support area for Platform Radar, ArgoCD, observability, ingress, GitOps, and shared operational tooling. Failures here can reduce visibility or delivery confidence.

How It Works

Collect current runtime facts from the Kubernetes APIs across all configured clusters.
Compare runtime state with expected platform standards and ArgoCD desired state.
Enrich the cluster view with observability data from Prometheus/Thanos, Loki, and later Tempo where useful.
Correlate Kubernetes resources with GCP dependencies and ownership metadata.
Classify findings by severity, confidence, blast radius, affected cluster, affected product, and recommended next action.
Store summaries of confirmed findings and operator feedback so recurring issues become easier to spot.

When It Should Ask For Attention

Immediate attention: production risk, security exposure, capacity exhaustion, unreachable critical dependency, broken GitOps sync, certificate, PAT, key, token, or licence expiry soon, or degraded observability/control-plane components.
Planned attention: overprovisioning, missing ownership labels, absent probes, single replicas for important services, outdated images, or non-standard resource shapes.
Informational: low-risk drift, dev-only churn, documentation mismatch, or signals that need more history before becoming actionable.
No attention: expected dev noise, intentionally stopped environments, already acknowledged maintenance, or signals without enough evidence.

What A Good Platform Health Specialist Needs

Kubernetes And GKE

Pods, deployments, StatefulSets, DaemonSets, Jobs, CronJobs, services, ingresses, gateways, HPA/VPA/PDB, events, nodes, taints, tolerations, requests, limits, probes, PVCs, and service accounts.
GKE node pools, autoscaling settings, version skew, workload identity, cluster upgrades, network policy posture, and system namespace health.

Delivery And Ownership

ArgoCD app/project, sync status, health status, repo path, target revision, image revision, drift, and rollout history.
Product, team, cost-center, environment, criticality, customer, and dependency ownership labels.

Observability

Prometheus/Thanos metrics for resource usage, restarts, saturation, latency, error rates, queue depth, node health, and availability trends.
Loki logs for repeated startup failures, dependency errors, auth errors, image pull issues, and configuration exceptions.
Tempo traces later for cross-service latency and dependency path validation.

GCP Dependencies

MongoDB clusters, Cloud SQL/PostgreSQL, Pub/Sub topics/subscriptions, Cloud Storage buckets, Secret Manager, IAM/service accounts, DNS, certificates, load balancers, NAT, firewall rules, and VPC connectivity.
Billing and provisioning data for resource sizing, cost anomalies, shared platform spend, and overprovisioning signals.

Expiry And Rotation Metadata

Expiry dates, last rotation dates, owner, renewal location, affected cluster/product, and operational impact for PATs, API keys, certificates, service account keys, registry credentials, ArgoCD repo credentials, licences, and other time-bound dependencies.
Secret contents must never be collected or displayed. The useful health signal is whether something important is going to expire, who owns it, and what must be renewed before it causes an outage.

Output It Should Produce

The page should eventually show a cluster-by-cluster health score, but not as a vague traffic light. Each score must be explainable with evidence.

Top risks per cluster.
Impacted products or platform capabilities.
Why this matters now.
Recommended owner or next team.
Confidence and missing context.
Trend: new, recurring, improving, or worsening.

Relationship To Other Pages

The Incident Board answers: what is broken now and how do we fix it? Cost Insight answers: where do we spend money and where can provisioning improve? Platform Health answers: what is likely to become a problem if nobody acts?

Some signals overlap, but the intent is different. This page should be calmer, more preventive, and more trend-oriented than the incident board.

Initial Implementation Direction

Start with read-only Kubernetes/GKE health because those signals are closest to the current Platform Radar. Add GCP resource connectors only when the data can be mapped to clusters, namespaces, products, and owners without guessing.

The Kind lab can validate the UI and basic logic, but production readiness must be judged against real GKE/GCP data.

Expiry & Rotation Watch

This should become a preventive health lane, not a vault. It should show which operational credentials, certificates, PATs, registry tokens, licences, and external access objects need renewal, when they expire, who owns them, and what will break if they are missed.

The page should only use safe metadata. Values stay in Secret Manager, Kubernetes Secrets, GitLab, ArgoCD, vendor portals, or the relevant source system.