The problem and purpose
Deploying an application is only one part of operating a platform. I built this lab to connect the rest of the workflow: repeatable packaging, versioned delivery, reconciliation, metrics, alerts, logs, and scaling.
The application is deliberately small so the platform behavior stays visible. A local cluster gives me an environment to inspect changes and exercise failure scenarios without presenting the results as evidence of enterprise-scale performance or availability.
My responsibility
I own the lab’s design and development. I use AI assistance for test cases, validation, and documentation. The repository’s recorded test results are separate from tests I have personally reviewed and repeated.
One difficult decision was selecting the tool stack. Several CNCF tools cover similar needs. I researched the options, selected tools for the lab, and deployed them to understand how they work together.
Current status
- Recorded validation
- Release delivery, Argo CD configuration recovery, Kyverno policy rejection, logging, and database restoration have repository records. These describe the recorded runs, not a new personal test of every feature.
- Configured checks
- Trivy image scanning and SBOM generation are defined in CI. Successful execution of the current checks still needs confirmation.
- Incomplete API integration
- The platform API needs real PostgreSQL and Redis integration and dependency health checks. Its readiness endpoint currently reports configured dependencies without checking them.
- Next work
- Review the current implementation, repeat key tests, complete the API data integration, expand reliability exercises, and update the roadmap.
Platform architecture
A two-node Kind cluster runs a React frontend and Flask API, packaged with Helm. PostgreSQL uses persistent storage; Redis supports the application. The local ingress routes requests to the frontend and API.
Follow an artifact from source to workload.
01BuildGitHub Actions
Build frontend and API images from a source commit. CI creates artifacts; it does not deploy them to the cluster.
02PublishGHCR
Store the images with commit-specific tags. The registry holds artifacts that Kubernetes can pull.
03PromoteGit + Helm
After both builds succeed, update the Helm image tags in Git. This commit records the version the cluster should run.
04ReconcileArgo CD
Compare live configuration with Git. Apply the Helm configuration and let Kubernetes roll out the workloads.
05OperateKubernetes + observability
Run React, Flask, PostgreSQL, and Redis in a two-node Kind cluster. Use metrics, alerts, and logs to inspect workload behavior.
Conceptual walkthrough of my independent lab. Open the project for source code and validation records.
Observability is a separate platform concern: Prometheus collects metrics, Grafana presents dashboards, and Alertmanager handles alerts. Grafana Alloy collects workload logs and sends them to Loki for querying through Grafana.
CI/CD and GitOps delivery
- Build: GitHub Actions builds frontend and API container images after qualifying changes on the main branch.
- Publish: Images are published to GitHub Container Registry (GHCR), including commit-specific tags.
- Promote: After both image builds succeed, an automated job updates the Helm image tags and commits the desired version back to Git.
- Reconcile: Argo CD applies the desired Helm configuration to Kubernetes, which rolls out the versioned workloads.
- Maintain: Argo CD detects drift and uses automated self-healing to restore the declared configuration.
The repository documents an end-to-end image release and a drift exercise in which a deleted frontend Deployment was recreated by Argo CD. These are bounded lab validations, not a production service-level guarantee.
Release validation
- What was built?
- I built the lab delivery path with GitHub Actions, GHCR, Helm, and Argo CD.
- Why this design?
- Record the desired image version in Git so delivery has a reviewable history.
- What was tested?
- Publish commit-specific images, inspect the Argo CD application and running pods, and request the application through ingress.
- What happened?
- The release record documents delivery of commit-specific images to running Kubernetes workloads.
Independent local lab. Image publication does not depend on the separate validation workflow. Commit-specific tags do not guarantee registry-enforced immutability.
Open the release validationThe publication workflow does not depend on the separate validation workflow. The latest reviewed CI run, dated 5 September 2026, failed during job setup before validation started. Successful image publication does not establish that the scan checks passed.
Image publishing and promotion workflow
Release validation · Self-healing evidence
Reliability and observability
- Health: Liveness and readiness probes make workload health part of Kubernetes operation.
- Metrics and alerts: Prometheus, Grafana, and Alertmanager support inspection of request rate, latency, resource use, and alert conditions.
- Scaling: Horizontal Pod Autoscaling adjusts API replicas against CPU utilization. The repository documents a load exercise and scale-down after traffic stops.
- Logs: Alloy and Loki centralize workload logs so investigation can move between metrics and individual events.
- State: PostgreSQL persistence separates data from application-pod lifecycles; a persistent volume alone is not a backup strategy.
Autoscaling exercise · Logging configuration
Follow the operating response.
Select a scenario. Follow the response and see which component owns each step.
A Deployment is deleted
Git still declares the frontend Deployment. Follow how the controller restores that desired state.
Read the self-healing validationGit declares a resource that is missing in the cluster.
- 01
Drift detected
Argo CD compares the live cluster with Git. The declared Deployment is missing.
Desired: Deployment present / Live: Deployment missingComponent boundary: Argo CD detects configuration drift. Prometheus does not restore Kubernetes resources.
- 02
Configuration restored
With automated self-healing enabled, Argo CD reapplies the declared Deployment.
Desired and live configuration convergeComponent boundary: The Deployment controller creates the ReplicaSet and Pods. The scheduler selects nodes; the kubelet starts containers.
- 03
Workload checked
Inspect the rollout and readiness. A restored manifest alone does not prove that the application can serve traffic.
Check rollout status, ready Pods, and application responseComponent boundary: Self-healing restores declared configuration. It cannot correct an error already committed to Git.
API CPU utilization rises
Follow the CPU-based scaling path. Each controller has a separate job.
Read the autoscaling validationIllustrative counts: 2, 4, then 2. These are not measured test results.
- 01
Demand measured
Metrics Server supplies resource metrics. The HPA compares CPU utilization with its configured target.
CPU utilization relative to container CPU requestsComponent boundary: For this CPU-based HPA, Metrics Server supplies the metrics. Prometheus is a separate monitoring system.
- 02
Replica count changed
The HPA changes the desired replica count within its limits. The workload controller creates more Pods.
Desired replicas increase within configured limitsComponent boundary: The HPA changes replica count. The scheduler places new Pods only when eligible nodes have enough resources.
- 03
Demand falls
After traffic stops, the HPA can reduce replicas according to its scaling policy and stabilization settings.
Observe CPU, replica count, and scale-down behaviorComponent boundary: More replicas use existing cluster capacity. They do not add nodes or guarantee lower latency.
Requests start to fail
Use each observability tool for a specific question. This is an illustrative investigation, not a recorded incident.
Inspect the logging configurationGrafana
Loki
Move from the symptom to the notification, then inspect workload events.
- 01
Find the symptom
Prometheus collects metrics and evaluates alert rules. Grafana displays the metrics so an operator can inspect changes in errors and latency.
Metrics answer: what changed, and when?Component boundary: Prometheus evaluates the rule. Grafana provides the dashboard; it does not collect workload metrics in this flow.
- 02
Route the alert
Prometheus sends firing alerts to Alertmanager. Alertmanager groups, routes, and silences notifications.
Alerts answer: who needs to respond?Component boundary: Alertmanager handles notifications. It does not scrape metrics or evaluate Prometheus alert rules.
- 03
Inspect the events
Grafana Alloy collects workload logs and sends them to Loki. Query the logs through Grafana to inspect events around the failure.
Logs answer: what happened in the workload?Component boundary: A dashboard or notification does not repair a failure. Use the evidence to choose a fix, then verify recovery.
Recover the declared configuration
The recorded exercise deleted the frontend Deployment to test a specific failure: a declared resource missing from the cluster. Argo CD recreated it from Git. This checks configuration recovery; restoring database contents requires a different process.
Self-healing validation
- What was built?
- I own the lab’s GitOps configuration. The repository records an exercise with a deleted frontend Deployment.
- Why this design?
- Use Git as the desired-state record and enable Argo CD self-healing.
- What was tested?
- Delete the frontend Deployment, then check whether Argo CD recreates it from Git.
- What happened?
- The validation record states that Argo CD detected the missing Deployment and recreated it from Git.
The record validates configuration recovery. It does not measure recovery time or production availability.
Open the self-healing validationA backup needs a restore test
The first restore exercise exposed a gap. The tested Velero backup did not capture the Kind local-path database volume. The corrected approach uses Velero for Kubernetes resources and a PostgreSQL logical dump for database contents.
The dump is uploaded to local MinIO. A separate Job restores the database in an isolated namespace. The validation then checks expected tables and application readiness.
Database recovery validation
- What was built?
- The lab combines Velero resource backups with PostgreSQL logical backups stored in local MinIO.
- Why this design?
- The restore exercise showed that the tested volume backup did not capture the Kind local-path database volume. The corrected database recovery path uses pg_dump.
- What was tested?
- Upload a database dump, restore Kubernetes resources into an isolated namespace, restore the database, and check tables and workload readiness.
- What happened?
- The record reports four expected tables restored and application workloads Ready. The drill completed within the 60-minute recovery target.
assessment.alembic_version · assessment.assessments · content.alembic_version · content.definitionsThe record does not provide an exact elapsed recovery time. Local MinIO shares the cluster failure domain, so this does not prove recovery after loss of the entire host.
Read the backup and restore recordSecurity considerations
The repository defines dedicated service accounts, RBAC, non-root container settings, and network policies. Those controls make workload privileges and permitted traffic explicit. NetworkPolicy enforcement depends on the cluster’s network implementation; a manifest alone does not prove isolation.
Policy and logging validation
- What was built?
- The lab includes Kyverno admission policies and application log collection through Alloy and Loki.
- Why this design?
- Check workload settings at admission and collect logs across the platform and application namespaces.
- What was tested?
- Submit a noncompliant pod and query Loki for an application health event.
- What happened?
- Kyverno rejected the test pod. Loki returned an assessment-service health event with workload and request fields.
This validates the recorded admission and logging checks. It does not establish continuous compliance or successful image scanning.
Read the policy and logging recordImage promotion also creates a trust boundary: the publishing workflow can write packages and update deployment configuration. A production adaptation would require deliberate review of workflow permissions, protected branches, digest pinning, secret management, and credential rotation.
Local configuration is not a production security certification. Public exposure, TLS, durable storage, access control, and recovery requirements need their own validation before this pattern is used in a shared environment.
Engineering decisions and tradeoffs
- Kind over a managed cloud cluster
- A repeatable local environment keeps iteration accessible. Two local nodes still share a host and do not demonstrate independent failure domains or cloud availability.
- Helm as the installation contract
- A compact chart groups the workload and configuration. It is easy to inspect, but couples components that a larger platform might version independently.
- Git as the desired-state record
- Image promotion leaves a reviewable history and Argo CD reconciles from it. Direct cluster edits may be reverted, and an incorrect desired state can also be automatically applied.
- Commit-specific image tags
- Tags connect a rollout to a source revision. Registry tags are not inherently immutable; digest references would give stronger artifact identity.
- A complete local observability stack
- Metrics, alerts, and logs can be investigated together. They consume local resources; production logging requires separate capacity, retention, access, and durability decisions.
What I learned
The release and self-healing exercises make the difference between CI and GitOps concrete: publishing an image creates an artifact; changing desired state and reconciling it delivers that artifact.
Autoscaling and self-healing solve different problems. Increasing replicas responds to demand, while reconciliation repairs configuration drift. Neither removes the need to understand stateful dependencies, host capacity, or failure domains.
The architectural lesson is to validate each boundary. A successful rollout, a recreated Deployment, and a scaling exercise each establish something specific. Together they provide useful evidence without claiming production readiness.
The restore drill changed the backup design. A completed backup operation was not enough: I needed to check whether the database contents could be recovered and the application could become Ready.
Explore the evidence
The repository contains the application, Helm chart, GitOps configuration, operational notes, and validation records. The related Impact Explorer adds a separate application workload with its own delivery stage.