- Install Go 1.20 or newer
- In the project root, run
go build ./cmd/attacknet - Copy the "attacknet" binary to your PATH or directly invoke it.
- Set up a containerd k8s cluster. (1.26 or older) (todo: recommended resourcing. Also note that auto-scaling can sometimes be too slow, and kurtosis will time out before the nodes for its workload can be provisioned.)
- Authenticate to the cluster for kubectl
- Install chaos-mesh
kubectl create ns chaos-meshhelm repo add chaos-mesh https://charts.chaos-mesh.orghelm install chaos-mesh chaos-mesh/chaos-mesh -n=chaos-mesh --version 2.6.1 --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock --set dashboard.securityMode=false --set bpfki.create=true- To access chaos dashboard, use
kubectl --namespace chaos-mesh port-forward svc/chaos-dashboard 2333
- Install kurtosis locally.
- Run
kurtosis cluster set cloud - If running in digitalocean, edit the kurtosis-config.yml file from
kurtosis config pathand add the following setting under kubernetes-cluster-name:storage-class: "do-block-storage" - In a separate terminal, run
kurtosis engine start - In a separate terminal, run
kurtosis gateway. This process needs to stay alive during all attacknet testing and cannot be started via SDK.
There are three workflows in attacknet:
- Manually creating test suites/network configs
- Automatically creating test suites/network configs using the planner
- Running the test suite
Attacknet is configured using "test suites". These are yaml files found under ./test-suites that define everything
Attacknet needs to genesis a network, test the network, and determine the health of the network.
Test suite configuration is broken into 3 sections:
- Attacknet configuration.
- Harness configuration. This is used to configure the Kurtosis package that will be used to genesis the network.
- Test configuration. This is used to determine which tests should be run against the devnet and how those tests should be configured. As of right now, only the first test in the array is run before exiting.
Here is an annotated test suite configuration that explains what each bit is for:
attacknetConfig:
grafanaPodName: grafana # the name of the pod that grafana will be deployed to.
grafanaPodPort: 3000 # the port grafana is listening to in the pod
waitBeforeInjectionSeconds: 10
# the number of seconds to wait between the genesis of the network and the injection of faults. To wait for finality, use 25 mins (1500 secs)
reuseDevnetBetweenRuns: true # Whether attacknet should skip enclave deletion after the fault concludes. Defaults to false.
existingDevnetNamespace: kt-ethereum # If you don't want to genesis a new network, you can specify an existing namespace that contains a Kurtosis enclave and run tests against it instead. I'm expecting this to only be useful for dev/tool testing. Exclude this parameter for normal operation.
allowPostFaultInspection: true # When set to true, Attacknet will maintain the port-forward connection to Grafana once the fault has concluded to allow the operator to inspect metrics. Default: true
harnessConfig:
networkPackage: github.com/crytic/ethereum-package # The Kurtosis package to deploy to instrument the devnet.
networkConfig: default.yaml # The configuration to use for the Kurtosis package. These live in ./network-configs and are referenced by their filename.
networkType: ethereum # no touchy
# The list of tests to be run. As of right now, the first test is run and the tool terminates. In the future, we will genesis single-use devnets for each test, run the test, and terminate once all the tests are completed and all the enclaves are cleaned up.
testConfig:
tests:
- testName: packetdrop-1 # Name of the test. Used for logging/artifacts.
health:
enableChecks: true # whether health checks should be run after the test concludes
gracePeriod: 2m0s # how long the health checks will attempt to pass before marking the test a failure
planSteps: # the list of steps to facilitate the test, executed in order
- stepType: injectFault # this step injects a fault, the continues to the next step without waiting for the fault to terminate
description: "inject fault"
chaosFaultSpec: # The chaosFaultSpec is basically a pass-thru object for Chaos Mesh fault resources. This means we can support every possible fault out-of-the-box, but slightly complicates generating the configuration. To determine the schema for each fault type, check the Chaos Mesh docs: https://chaos-mesh.org/docs/simulate-network-chaos-on-kubernetes/. One issue with this method is that Attacknet can't verify whether your faultSpec is valid until it tries to create the resource in Kubernetes, and that comes after genesis which takes a long time on its own. If you run into schema validation issues, try creating these objects directly in Kubernetes to hasten the debug cycle.
kind: NetworkChaos
apiVersion: chaos-mesh.org/v1alpha1
spec:
selector:
labelSelectors:
kurtosistech.com/id: cl-1-lighthouse-geth-validator
mode: all
action: loss
duration: 1m
loss:
loss: '10'
correlation: '0'
direction: to
- stepType: waitForFaultCompletion # this step waits for all previous running faults to complete before continuing
description: wait for faults to terminateAttacknet can automatically create test suites based off a pre-defined test plan. This can be used to create large, comprehensive test suites that test against a variety of different client combos.
An example test plan can be found in the planner-configs/ directory
Here's an annotated version:
execution: # list of execution clients that will be used in the network topology
- name: geth
image: ethereum/client-go:latest
- name: reth
image: ghcr.io/paradigmxyz/reth:v0.1.0-alpha.13
consensus: # list of consensus clients that will be used in the network topology
- name: lighthouse
image: sigp/lighthouse:latest
has_sidecar: true
- name: prysm
image: prysmaticlabs/prysm-beacon-chain:latest,prysmaticlabs/prysm-validator:latest
has_sidecar: true
network_params:
num_validator_keys_per_node: 32 # required.
kurtosis_package: "github.com/kurtosis-tech/ethereum-package"
kubernetes_namespace: kt-ethereum
fault_config:
fault_type: ClockSkew # which fault to use. A list of faults currently supported by the planner can be found in pkg/plan/suite/types.go in FaultTypeEnum
target_client: reth # which client to test. this can be an exec client or a consensus client. must show up in the client definitions above.
wait_before_first_test: 300s # how long to wait before running the first test. Set this to 25 minutes to test against a finalized network.
fault_config_dimensions: # the different fault configurations to use when creating tests. At least one config dimension is required.
- skew: -2m # these configs differ for each fault
duration: 1m
grace_period: 1800s # how long to wait for health checks to pass before marking the test as failed
- skew: 2m
duration: 1m
grace_period: 1800s
fault_targeting_dimensions: # Defines how we want to impact the targets. We can inject faults into the client and only the client, or we can inject faults into the node (injects into cl, el, validator)
- MatchingNode
- MatchingClient
fault_attack_size_dimensions: # Defines how many of the matching targets we actually want to attack.
- AttackOneMatching # attacks only one matching target
- AttackMinorityMatching # attacks <33%
- AttackSuperminorityMatching # attacks >33% but <50%
- AttackMajorityMatching # attacks >50% but <66%
- AttackSupermajorityMatching # attacks >66%
- AttackAllMatching # attacks allThe total number of tests generated by a plan is equal to len(fault_config_dimensions) * len(fault_targeting_dimensions) * len(fault_attack_size_dimensions)
You can create a test plan by invoking attacknet plan <suitename> <planner config path>
The suite plan will be written to ./test-suites/plan/<suitename>.yaml
The network config will be written to ./network-configs/plan/<suitename>.yaml
and can be executed by attacknet using attacknet start plan/suitename
Config:
- skew: -2m # how far to skew the clock. can be positive or negative
duration: 1m # how long to skew the clock for
grace_period: 1800s # how long to wait for health checks to pass before marking the test as failedConfig:
- grace_period: 1800s # how long to wait for health checks to pass before marking the test as failedConfig:
- grace_period: 1800s # how long to wait for health checks to pass before marking the test as failed
delay: 1000ms # how long the i/o delay should be
duration: 1m # how long the fault should last
percent: 50 # the percentage of i/o requests impacted.Once you've got your configuration set up, you can run Attacknet:
attacknet start <suitename>
If your suite config is located at ./test-suites/suite.yaml, you would run attacknet start suite. This will
probably be changed.
At this time, health checks will be run in perpetuity once the fault has concluded. Simply ctrl+c to terminate.
Dec 15, 2023 version v0.1 (internal)
- Initial internal release
Jan 11, 2023 version v0.2 (internal)
- Updated to kurtosis v0.86.1
- Updated to Go 1.21
- Grafana port-forwarding has been temporarily disabled
- Introduces multi-step tests. This allows multiple faults and other actions to be composed into a single test.
- Introduces the suite planner. The suite planner allows the user to define a set of testing criteria/dimensions, which the planner turns into a suite containing multiple tests.
- Successful & failed test suites now emit test artifacts summarizing the results of the test.
- Install pre-commit
brew install pre-commitpre-commit install