Spin up a local dev environment for Flux with Docker and Kubernetes KIND in under five minutes.
- Flux users who want to test Flux configs locally, without having to push changes to a Git repository. Config changes are pushed to a local registry and synced on the cluster by Flux automatically.
- Flux contributors who want to test their changes to Flux controllers locally, without having to push the container images to an external registry.
- Flux maintainers who want to test Flux prereleases on various Kubernetes versions and configurations.
This project spins up a Docker Registry container named kind-registry and a Kubernetes Kind cluster
named flux under the same Docker network. Then it installs Flux and configures it to upgrade itself
from the latest OCI artifact published at ghcr.io/fluxcd/flux-manifests. Before an upgrade, Flux
verifies that the OCI artifacts are signed by the Flux team with Cosign and GitHub OIDC.
| Component | Role | Host |
|---|---|---|
| Kubernetes KIND | Local cluster | Binds to port 80 and 443 |
| Docker Registry | Loc 8000 al registry | Binds to port 5050 |
| Flux | Cluster reconciler | - |
| ingress-nginx | Ingress for *.flux.local |
- |
| cert-manager | Self-signed ingress certs | - |
| metrics-server | Container resource metrics | - |
| kube-prometheus-stack | Prometheus Operator and Grafana | Binds to grafana.flux.local |
| weave-gitops | Flux UI | Binds to ui.flux.local |
| podinfo | Demo app | Binds to podinfo.flux.local |
The Docker registry is exposed on the local machine on localhost:5050 and inside the cluster
on kind-registry:5000. The registry servers two purposes:
- hosts container images e.g.
docker push localhost:5050/podinfo:test1 - hosts OCI artifacts e.g.
flux push artifact oci://localhost:5050/podinfo-manifests:test1
To facilitate ingress access to the Flux UI and any other
application running inside the cluster, the Kubernetes Kind container
binds to port 80 and 443 on localhost.
Ingress is handled by Kubernetes ingress-nginx and self-signed TLS certs
are provided by cert-manager.
To monitor how the deployed applications perform on the cluster, the kube-prometheus-stack and metrics-server Helm charts are installed at bootstrap along with the Flux Grafana dashboards.
To monitor and debug Flux using a Web UI, the Weave GitOps Helm chart is installed at bootstrap.
Start by cloning the repository locally:
git clone https://github.com/stefanprodan/flux-local-dev.git
cd flux-local-devInstall Kubernetes kind, kubectl, flux and other CLI tools with Homebrew:
make toolsThe complete list of tools can be found in the Brewfile.
Note that the minimum required version of Flux is v2.0.0-rc.1.
Start the dev environment with:
make upThe make up command performs the following steps:
- creates the Docker registry container if it's not already running
- creates the Kubernetes Kind cluster if it's not already running
- pushes the Kubernetes manifests as OCI artifacts to the local registry
locahost:5050/flux-cluster-syncis generated fromkubernetes/clusters/locallocahost:5050/flux-infra-syncis generated fromkubernetes/infralocahost:5050/flux-apps-syncis generated fromkubernetes/apps
- installs Flux on the clusters and configures it to self upgrade from
oci://ghcr.io/fluxcd/flux-manifests - waits for Flux to reconcile the cluster addons from
oci://kind-registry:5000/flux-infra-sync - waits for Flux to reconcile the demo apps from
oci://kind-registry:5000/flux-apps-sync
Add the following domains to /etc/hosts:
127.0.0.1 podinfo.flux.local
127.0.0.1 grafana.flux.local
127.0.0.1 ui.flux.localVerify that the NGINX ingress self-signed TLS works:
make checkAccess the Flux UI and Grafana using the username admin and password flux:
- http://ui.flux.local/applications
- http://grafana.flux.local/d/flux-control-plane (username: admin, password: flux)
- http://grafana.flux.local/d/flux-cluster
Access the demo application on http://podinfo.flux.local.
Add a label to the apps namespace definition:
yq eval '.metadata.labels.env="dev"' -i ./kubernetes/apps/namespace.yamlValidate the Kubernetes manifests and Flux custom resources:
make validatePush the changes to the local registry with:
make syncVerify that Flux has reconciled the namespace:
kubectl get ns apps --show-labelsDelete the registry and the Kubernetes cluster with:
make downIn the cue directory you can find an example of how to use cuelang to define and generate Kubernetes resources.
List the CUE generated resources with make cue-ls:
$ make cue-ls
RESOURCE API VERSION
Namespace/cue-apps v1
ServiceAccount/cue-apps/flux-cue-apps v1
Service/cue-apps/podinfo v1
ServiceAccount/cue-apps/podinfo v1
Deployment/cue-apps/podinfo apps/v1
HorizontalPodAutoscaler/cue-apps/podinfo autoscaling/v2beta2
Ingress/cue-apps/podinfo networking.k8s.io/v1
ServiceMonitor/cue-apps/podinfo monitoring.coreos.com/v1
RoleBinding/cue-apps/flux-cue-apps rbac.authorization.k8s.io/v1Push the generated resources to the local registry with make cue-push:
$ make cue-push
► pushing artifact to localhost:5050/flux-cue-apps-sync:local
✔ artifact successfully pushed to localhost:5050/flux-cue-apps-sync@sha256:59676338abbb245a80345713fea24c2686c8c38cbd235691dd0af0fdc00fe116To reconcile the resources on the cluster, add a file called cue-apps.yaml to the kubernetes/cluster/local directory:
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: cue-apps-source
namespace: flux-system
spec:
insecure: true
interval: 1m
provider: generic
ref:
tag: local
url: oci://kind-registry:5000/flux-cue-apps-sync
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cue-apps-sync
namespace: flux-system
spec:
dependsOn:
- name: infra-config
interval: 5m
retryInterval: 30s
timeout: 5m
path: ./
prune: true
sourceRef:
kind: OCIRepository
name: cue-apps-sourceSync the changes on the cluster with:
make syncList the reconciled objects with:
$ flux tree ks cue-apps-sync
Kustomization/flux-system/cue-apps-sync
├── Namespace/cue-apps
├── ServiceAccount/cue-apps/flux-cue-apps
├── ServiceAccount/cue-apps/podinfo
├── RoleBinding/cue-apps/flux-cue-apps
├── Service/cue-apps/podinfo
├── Deployment/cue-apps/podinfo
├── HorizontalPodAutoscaler/cue-apps/podinfo
├── ServiceMonitor/cue-apps/podinfo
└── Ingress/cue-apps/podinfoIf you make changes to the CUE definitions, run make cue-push and Flux will apply the changes on its own.
Assuming you are contributing a change to kustomize-controller, and you want to run a series of end-to-end tests before opening a PR. Most importantly, you want to make sure your changes don't break Flux capability to upgrade itself.
From within the kustomize-controller local clone, run make docker-build to build the controller image.
If your local machine is an Apple M1, set the arch to linux/arm64 and run:
IMG=localhost:5050/kustomize-controller:latest make docker-build docker-push
BUILD_PLATFORMS=linux/arm64 make docker-buildTag and push the image to your local registry:
docker tag fluxcd/kustomize-controller:latest localhost:5050/kustomize-controller:test1
docker push localhost:5050/kustomize-controller:test1From within the flux-local-dev clone, open the kubernetes/clusters/local/flux-system/flux-sync.yaml file
and add an image patch:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: flux-sync
namespace: flux-system
spec:
images:
- name: ghcr.io/fluxcd/kustomize-controller
newName: localhost:5050/kustomize-controller
newTag: test1Sync the changes on the cluster with make sync and verify that the image is being rolled out:
make sync
kubectl -n flux-system get deploy/kustomize-controller --watchFinally, verify that the upgrade was successful with:
$ flux check
✔ kustomize-controller: deployment ready
► localhost:5050/kustomize-controller:test1If you don't want to bump the image tag on every build, you can bypass the local registry and import the image directly in the cluster cache:
export IMG=localhost:5050/kustomize-controller:test1
IMG=${IMG} make docker-build &&
kind import docker-image ${IMG} &&
kubectl delete pod -n flux-system -l app=kustomize-controllerAssuming you are maintainer, and you want to test the Flux controller suite before a release.
From within the flux2 local clone, run make build-dev to build the Flux CLI
binary that embeds the install manifests.
Extract the manifests to a directory with:
mkdir -p flux-vnext
./bin/flux install --components-extra=image-reflector-controller,image-automation-controller \
--export > ./flux-vnext/install.yamlPush the manifests to your local registry:
./bin/flux push artifact oci://localhost:5050/flux:latest --path ./flux-vnext \
--source="$(git config --get remote.origin.url)" \
--revision="$(git rev-parse HEAD)"From within the flux-local-dev clone, open the kubernetes/clusters/local/flux-system/flux-source.yaml file,
change the URL to point to your local registry and enable the insecure flag:
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: flux-source
namespace: flux-system
spec:
url: oci://kind-registry:5000/flux
insecure: trueSync the changes on the cluster with make sync and wait for the new version to rollout:
make sync
flux reconcile ks flux-sync --with-sourceFinally, verify that the upgrade was successful with:
flux check Assuming you are contributing an API change to source-controller, and you want to run a series of end-to-end tests before opening a PR.
From within the source-controller local clone, run make docker-build to build the controller image.
If your local machine is an Apple M1, set the arch to linux/arm64 and run:
IMG=localhost:5050/source-controller \
TAG=oci1 \
BUILD_PLATFORMS=linux/arm64 \
BUILD_ARGS=--load \
make docker-build docker-pushFrom within the source-controller local clone, extract the Flux manifests to a directory:
mkdir -p flux-vnext
flux install --components-extra=image-reflector-controller,image-automation-controller \
--export > ./flux-vnext/install.yamlReplace the CRD with the one from your branch:
export CRD_NAME="ocirepositories.source.toolkit.fluxcd.io"
yq e 'select(.metadata.name != env(CRD_NAME))' -i ./flux-vnext/install.yaml
CRD_BASE_PATH="./config/crd/bases"
CRD_FILE_NAME="source.toolkit.fluxcd.io_ocirepositories.yaml"
cp ${CRD_BASE_PATH}/${CRD_FILE_NAME} ./flux-vnext/Push the manifests to your local registry:
flux push artifact oci://localhost:5050/flux:latest --path ./flux-vnext \
--source="$(git config --get remote.origin.url)" \
--revision="$(git rev-parse HEAD)"From within the flux-local-dev clone, open the kubernetes/clusters/local/flux-system/flux-source.yaml file,
change the URL to point to your local registry and enable the insecure flag:
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: flux-source
namespace: flux-system
spec:
url: oci://kind-registry:5000/flux
insecure: trueOpen the kubernetes/clusters/local/flux-system/flux-sync.yaml file
and patch the controller deployment with the local image:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: flux-sync
namespace: flux-system
spec:
images:
- name: ghcr.io/fluxcd/source-controller
newName: localhost:5050/source-controller
newTag: oci1Sync the changes on the cluster with make sync and wait for the new version to rollout:
make sync
flux reconcile ks flux-sync --with-sourceTest the new feature by adding a Flux resource to the kubernetes/apps/source-test.yaml:
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: podinfo-keyless
namespace: apps
spec:
interval: 5m
url: oci://ghcr.io/stefanprodan/manifests/podinfo
ref:
semver: "*"
verify:
provider: cosignSync the changes on the cluster and see the reconciliation result:
make sync
flux get source oci podinfo-keyless -n apps