This repository demonstrates how security best practices for a Go CLI application & backend service can be implemented on GitHub using GitHub Actions.
It features security best practices such as:
- Git commit signing (gitsign) and verification (chaingurad/enforce).
- Integrity protected SBOM generation (anchore/syft) and vulnerability scanning (anchore/grype) with a verified SBOM.
- SLSA provenance generation (slsa-framework/slsa-github-generator) and verification (slsa-framework/slsa-verifier).
This will protect us from different threats in our software supply chain.
GitHub uses your commit email address to associate commits with your account on GitHub.com, but it does not validate that you own the email address when you push changes. Therefore it is trivial to impersonate another user on GitHub. We simply commit using their email address.
As with any cyber security attack, the first stage is reconnaissance, i.e., information gathering.
- Find GitHub user (@kelseyhightower)
- Open recent activity
- Open a commit
- Add .patch to URL
The second line will read: From: username <email>.
Note
GitHub allows you to protect your privacy using a noreply address from GitHub and automatically block pushing with your personal email address.
Warning
All excercises are intended for educational purposes only. Users are strictly advised not to employ the acquired knowledge for any malicious activities or unauthorized access.
# Set username and email
git config user.name "Kelsey Hightower"
git config user.email "kelsey.hightower@gmail.com"
# Commit as Kelsey
git commit --allow-empty --no-gpg-sign -m "awesome change"
git push origin HEADRevert config & push.
# Use global config values
git config --unset user.name
git config --unset user.email
# Revert commit locally and remote
git reset --hard HEAD~1
git push origin HEAD --force-with-leaseTo mitigate the risk that someone impersonates other users on GitHub, we should sign and verify commits to our repositories, i.e., cryptographically bind the content of the commit to the signer's identity.
git commit signing is supported, since git v1.7.9 (January 2012), and is based on GPG keys.
GPG based commit signing (example).
GitHub provides great documentation on how to sign your commits with GPG.- Generate your key
gpg --full-generate-key
- Add public key to your profile
# List keys gpg --list-secret-keys --keyid-format=long # Export public key gpg --armor --export 3AA5C34371567BD2
- Configure git
# Enable commit signing git config --global commit.gpgsign true # Configure key git config --global user.signingkey 3AA5C34371567BD2
- Sign and push
# Our settings will automatically sign, otherwise use -S git commit -m "awesome change" git push
- Verification
git verify-commit HEAD
Read more about git commit signature verification support on GitHub.
So why isn't everyone signing their git commits using GPG?
GPG based keys come with a set of trade-offs. They do provide a very high level of security (given that no other service providers need to be trusted), if all users have secure processes in place to answer the following questions:
- How do I securely store my master-key and sub-keys?
- How do I make the right keys available on all my machines?
- How do I renew my keys when they expire?
- How do I revoke my key when it leaks?
- Which keys do I trust?
We can improve the UX significantly by placing some trust in an identity provider, which we (probably) do already.
Sigstore enables us to:
- Generate a key and short-lived certificate that is bound to an OpenID Connect identity.
- Generate proof that we own this key at a specific point in time.
- Store the proof in an immutable transparency log for later verification.
For a deep dive watch Life of a Sigstore Signature - Jed Salazar & Zack Newman, Chainguard, from SigstoreCon 2022.
Download and install sigstore/gitsign.
Installation script
VERSION=0.8.0
cd $(mktemp -d)
curl -LO https://github.com/sigstore/gitsign/releases/download/v${VERSION}/gitsign_${VERSION}_linux_amd64
curl -LO https://github.com/sigstore/gitsign/releases/download/v${VERSION}/gitsign-credential-cache_${VERSION}_linux_amd64
sudo install gitsign_${VERSION}_linux_amd64 /usr/local/bin/gitsign
sudo install gitsign-credential-cache_${VERSION}_linux_amd64 /usr/local/bin/gitsign-credential-cache
cd -Configure git to sign using gitsign.
git config commit.gpgsign true # Sign all commits
git config tag.gpgsign true # Sign all tags
git config gpg.x509.program gitsign # Use gitsign for signing
git config gpg.format x509 # gitsign expects x509 argsOptional:
# Static port for OIDC callback. This is helpful when you need to whitelist
# or proxy the callback, e.g., when working with remote dev environments.
git config gitsign.redirecturl http://localhost:39807/auth/callback
# Pre-select GitHub as default OIDC provider.
git config gitsign.connectorid https://github.com/login/oauthNote
Add --global to previous git config commands, so they apply for all repositories.
Verify & inspect:
# Using git (partial verification)
git verify-commit HEAD
# Using gitsign
gitsign verify \
--certificate-identity=datosh18@gmail.com \
--certificate-oidc-issuer=https://github.com/login/oauth
# Show actual signature value
git log --pretty=raw
# Show (partial) signature validation
git log --show-signatureHelpful debug commands
# Check your git config
git config --list --show-origin --show-scope
# Remove a config paramter
git config --unset gitsign.connectorid
# Create unsigned, empty commit
git commit --allow-empty --no-gpg-sign -m "nothing, unsigned"
# Parse git signature
git cat-file commit HEAD | sed -n '/-BEGIN/, /-END/p' | sed 's/^ //g' | sed 's/gpgsig //g' | sed 's/SIGNED MESSAGE/PKCS7/g' | openssl pkcs7 -print -print_certs -textConfigure gitsign credential cache
When doing multiple git commits in a short period of time, it might become annoying to do the OIDC dance for every commit.
The gitsign credential cache binary enables users to re-use the key during its 10 minutes lifetime.
Check the official documentation as the configuration is highly platform dependent.
gitsign-credential-cache &
export GITSIGN_CREDENTIAL_CACHE="$HOME/.cache/sigstore/gitsign/cache.sock"[!WARNING] Users should consider that caching the key introduces a security risk, as the key is exposed via unix sockets.
chainguard-enforce allows us to set policies for which identities can/must sign your code.
spec:
authorities:
- keyless:
identities:
- issuer: https://github.com/login/oauth
subject: datosh18@gmail.com
- key:
kms: https://github.com/web-flow.gpg
- key:
kms: https://github.com/renovate-bot.gpg
Additional we can configure a merge-blocking check to prevent any unsigned commits making it to main.
gittuf/gittuf provides a security layer for Git.
Among other features it allows you to set permissions for repository branches, tags, files, etc. This is much more powerful than git (signature) verification policies supported by the other projects we looked at.
At the same time, gittuf v0.1.0 was release in October 2023, is currently in alpha and therefore NOT intended for production use.
The Dockerfile for the backend service uses a multi-stage build to:
- containerize the build stage
- keep the final production image as small as possible
make build-svc-ociAlternatively, we can also build the service using a local Go toolchain:
make build-svcTo protect the container image from malicious tampering, we want to sign it:
# Note that this process stores public information in the transparency log.
cosign sign calculator-svc
# Verify locally signed image
cosign verify \
--certificate-identity datosh18@gmail.com \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
calculator-svc
# Verify images signed by the pipeline
cosign verify \
--certificate-identity-regexp https://github.com/datosh-org/most-secure-calculator/.github/workflows/calculator-svc.yml.* \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
ghcr.io/datosh-org/most-secure-calculator/calculator-svc@sha256:b2d2de552850cc7f99ebf0d0071357306159242ccf720b5b001ab694220f9547Warning
Depending on how the image was build and pushed, there are still risks to consider, as you might only know the digest of the image, after it was pushed to the registry. sigstore/cosign#2516
K8s deployments will regularly require to pull images from the registry and make them available on the node. As this is an automatic process, the verification should also happen automatically. For this, we can use projects such as Sigstore Kubernetes Policy Controller.
Local Kubernetes in Docker (kind) cluster
The Sigstore Kubernetes policy controller works with any K8s derivate. To test it on a local developer machine projects such as minikube, microk8s or kind work great. Here we show how to stand up a local K8s cluster with kind:
KIND_VERSION=0.20.0
cd $(mktemp -d)
curl -LO https://github.com/kubernetes-sigs/kind/releases/download/v{KIND_VERSION}/kind-linux-amd64
sudo install kind-linux-amd64 /usr/local/bin/kind
rm kind-linux-amd64
cd -We use a simple single-node kind configuration and additionally deploy nginx as a loadblancer.
make kind-upInstall calculator service with no verification:
kubectl apply -f k8s/deployment.yml
curl localhost:80/calculator/add/2/3Install the Sigstore Kubernetes Policy Controller:
helm repo add sigstore https://sigstore.github.io/helm-charts
helm repo update
kubectl create namespace cosign-system
helm install policy-controller -n cosign-system sigstore/policy-controller --devel
kubectl get all -n cosign-systemCreate a namespace, enforce the policy and deploy:
kubectl create namespace secured
# https://docs.sigstore.dev/policy-controller/overview/#configure-policy-controller-admission-controller-for-namespaces
kubectl label namespace secured policy.sigstore.dev/include=true
kubectl apply -f k8s/policy.yml
kubectl apply -f k8s/secure-deployment.yml
curl localhost:80/secure-calculator/add/2/3Software Bill of Materials (SBOMs) are becoming the standard tool to keep track of all ingredients in your artifacts.
Syft is an open source tool to generate a Software Bill of Materials (SBOM) from container images and filesystems.
Installation script
# https://github.com/anchore/syft/releases
SYFT_VERSION=0.98.0
cd $(mktemp -d)
curl -LO https://github.com/anchore/syft/releases/download/v${SYFT_VERSION}/syft_${SYFT_VERSION}_linux_amd64.tar.gz
tar -xzf syft_${SYFT_VERSION}_linux_amd64.tar.gz
sudo install syft /usr/local/bin
rm syft_${SYFT_VERSION}_linux_amd64.tar.gz
cd -
syft versionWe can easily generate an SBOM for any container images:
# table to stdout
syft httpd:2.4.58
# spdx format: https://spdx.dev/
syft httpd:2.4.58 -o spdx-json=spdx.json
# cyclonedx format: https://cyclonedx.org/
syft httpd:2.4.58 -o cyclonedx-json=cyclone.jsonGenerating a list of these ingredients as a distinct build artifact allows us to keep the responsibilities of vulnerability management and application deployment seperate.
Grype is an open source vulnerability scanner that can directly work on SBOMs.
Installation script
# https://github.com/anchore/grype/releases
GRYPE_VERSION=0.73.3
cd $(mktemp -d)
curl -LO https://github.com/anchore/grype/releases/download/v${GRYPE_VERSION}/grype_${GRYPE_VERSION}_linux_amd64.tar.gz
tar -xzf grype_${GRYPE_VERSION}_linux_amd64.tar.gz
sudo install grype /usr/local/bin
rm grype_${GRYPE_VERSION}_linux_amd64.tar.gz
cd -
grype versionWe use the SBOM generated in the previous step to scan for known vulnerabilities:
grype spdx.json
# exit code is 0, even though we have findings.
echo $?
# fail, if there are findings >= threshold.
grype spdx.json --fail-on critical
echo $?
# ignore things without a fix.
grype spdx.json --fail-on critical --only-fixedGrype also offers the option to persist a configuration directly in the repository:
external-sources:
enable: true
maven:
search-upstream-by-sha1: true
base-url: https://search.maven.org/solrsearch/selectThis can also help you to track the state of vulnerabilities directly with your source code:
ignore:
# This is the full set of supported rule fields:
- vulnerability: CVE-2008-4318
fix-state: unknown
# VEX fields apply when Grype reads vex data:
vex-status: not_affected
vex-justification: vulnerable_code_not_present
package:
name: libcurl
version: 1.5.1
type: npm
location: "/usr/local/lib/node_modules/**"
# We can make rules to match just by vulnerability ID:
- vulnerability: CVE-2014-54321As with our applications, we also want to protect our SBOMs from manipulations.
Attackers could:
- remove entries, to prevent us from patching vulnerabilities
- add entries, to harm the reputation of a project
Therefore, we use the same concepts to also sign our SBOM:
Warning
Make sure to install syft>=v0.98.0, as syft attest
was broken before.
syft attest --output spdx-json \
ghcr.io/datosh-org/most-secure-calculator/calculator-svc:please-sign-meThis attestation statement and SBOM is stored in the same OCI registry as our container image, and makes discoverability straight forward:
cosign verify-attestation \
ghcr.io/datosh-org/most-secure-calculator/calculator-svc:please-sign-me \
--certificate-identity=datosh18@gmail.com \
--certificate-oidc-issuer=https://github.com/login/oauth \
--type=spdxjson > spdx.json
# make sure there is only a single attestation associated.
wc -l spdx.json
# Extract SBOM from attestation
cat spdx.json | jq -r '.payload | @base64d | fromjson | .predicate' | grypeTODO: Write a few lines about the problems VEX is solving.
OCI compatible artifacts can be distributed via a breadth of registries. On the other hand, CLI binaries are usually distributed via GitHub release pages.
Provenance is the verifiable information about software artifacts describing where, when and how something was produced. It records information such as:
- a reference to the input source code
- information about the build environment
- output of the build, which can include files distinct from the actual binary
In contrast, classical software signatures only prove that the distributed artifact and the cryptographic private key have been at the same place at the same time.
SLSA-GitHub-Generator is a project that contains free tools to generate and verify SLSA Build Level 3 provenance for native GitHub projects using GitHub Actions.
For the pipeline implementation refer to workflows/calculator-cli.yml.
Once a release was generated we can use SLSA-Verifier to verify both the cryptographic signature, as well as contents of the provenance document, before consuming the binary.
curl -LO https://github.com/datosh-org/most-secure-calculator/releases/download/v0.1.0/calculator
curl -LO https://github.com/datosh-org/most-secure-calculator/releases/download/v0.1.0/calculator.spdx.json
curl -LO https://github.com/datosh-org/most-secure-calculator/releases/download/v0.1.0/calculator.intoto.jsonl
slsa-verifier verify-artifact calculator \
--provenance-path calculator.intoto.jsonl \
--source-uri github.com/datosh-org/most-secure-calculator \
--source-tag v0.1.0
slsa-verifier verify-artifact calculator.spdx.json \
--provenance-path calculator.intoto.jsonl \
--source-uri github.com/datosh-org/most-secure-calculator \
--source-tag v0.1.0
grype calculator.spdx.json
chmod u+x calculator
./calculator 2 3