Code release for An Automated Blackbox Noncompliance Checker for QUIC Server Implementations.
This is a learning-based tester consists of 3 components, the Learner, Mapper and QUT (QUIC server under test).
Its development and testing are performed on Ubuntu 20.04 with Python3.8.
In total, We have identified 55 faults, with 5 CVEs assigned:
The testing process will consume many resources. To ensure OS limits don't interfere, consider doing the following:
- add
ulimit -n 65536to .bashrc - add
ulimit -s 16384to .bashrc - add
vm.max_map_count=100000in/etc/sysctl.confand runsysctl -p - disable whoopsie:
systemctl stop whoopsie
| Directory | Description |
|---|---|
| dockerFile | Docker files to test all 19 QUIC implementations (with all security configurations). |
| findND | Program that can reproduce and record any non-deterministic behaviour detected during learning. |
| quicLearner | Source code for the Learner, and temporary results generated during the testing process. |
| quicMapper | Source code for the Mapper, config file (inOutput.py) for the Learner and Mapper. |
| results | Description of all faults with input sequences for reproducing the fault. |
| scripts | Tools for automated analysis task. |
| secrets | Certificate of CA, server and client; mapperSecrets.log for Wireshark decryption. |
- Install the Learner and Mapper using setup.sh:
./setup.shThe Mapper is built from Aioquic. It receives abstract inputs from the Learner, produces concrete inputs (valid QUIC packets) and sends concrete inputs to the QUIC server. Once it receives concrete outputs (responses) from the QUIC server, it extracts the abstract output and sends the abstract output to the Learner. Note: The Mapper can be used as a stand alone tool, see --sequence option in the help menu.
On Debian/Ubuntu, run the following:
sudo apt install libssl-dev python3-dev
cd quicMapper
python3 -m venv env
source env/bin/activate
python -m pip install -e .
python -m pip install asgiref dnslib httpbin starlette wsproto --no-cache-dir
python -m pip install Werkzeug==2.0.3
python -m pip install "flask<2.2.0"
deactivate
cd ..| Arguments | Description |
|---|---|
| ‑‑sequence | Only test a list of input sequence defined in mapper.py (for manual testing, optional) |
| -s / ‑‑server | Server to be tested (e.g., aioquic, google-quiche, kwik, lsquic, msquic, mvfst, neqo, ngtcp2, picoquic, pquic, quant, quiche, quiche4j, quic-go, quicly, quinn, quiwi, s2n-quic, xquic). |
| ‑‑cipher | Cipher suite to use (e.g., AES_128_GCM_SHA256, AES_256_GCM_SHA384, CHACHA20_POLY1305_SHA256). |
| -p / ‑‑port | Mapper port (e.g., 5544). |
| -t / ‑‑targetPort | QUIC server port (e.g., 4433). |
| -c / ‑‑cert | Client certificate for client authentication (PEM file, optional). |
| -k / ‑‑key | Client key for client authentication (PEM file, optional). |
| ‑‑invalidCertificate | Invalid client certificate for client authentication (PEM file, optional). |
| ‑‑caCert | CA certificate for peer verification (PEM file, optional). |
| ‑‑testRetryTokenExp | Enable to test Retry token expiration (optional). |
| ‑‑second | Seconds to validate the Retry token, use with ‑‑testRetryTokenExp (optional). |
| -r / ‑‑run | Command to run the QUIC server when ‑‑testRetryToken is enabled (e.g., "python3 http3_server.py ‑‑port 4431 ‑‑certificate server-cert.pem ‑‑private-key server-key.pem"). |
| ‑‑secrets | File to store the QUIC secrets (file, optional). The TLS decryption keys for Wireshark are available in secrets/mapperSecrets.log |
| ‑‑log | Store Aioquic library log in file (file, optional). |
| -i / ‑‑input | Dictionary that the Learner will use for fuzing (e.g., B ,BWR, BWCA, BWRCA) (for Learner). |
| ‑‑count | Input sequence count of the Learner (for Learner). |
The Learner is built from LearnLib and AALpy. It generates abstract inputs and receives abstract output. It is responsible for learning and building the models for the tested QUIC server.
On Debian/Ubuntu, run the following:
sudo apt install maven python3-tk
git submodule update --init
git apply quicLearner/learnlib-py4j-example.patch
cd learnlib-py4j-example/java
mvn package
cd ../../quicLearner
mkdir run
python3 -m venv env
source env/bin/activate
python -m pip install pydot pkg-resources py4j==0.10.9.7
deactivate
cd ..| Arguments | Description |
|---|---|
| -s / --server | Server to be tested (e.g., aioquic, google-quiche, kwik, lsquic, msquic, mvsfst, neqo, ngtcp2, picoquic, pquic, quant, quiche, quiche4j, quic-go, quicly, quinn, quiwi, s2n-quic, xquic). |
| -i / ‑‑input | Dictionary that the Learner will use for fuzing (e.g., B ,BWR, BWCA, BWRCA). |
| ‑‑config | Handshake configuration of the server (e.g., B, BWR, BWCA, BWRCA). |
| -p / ‑‑port | Mapper port (e.g., 5544). |
| -t / ‑‑targetPort | QUIC server port (e.g., 4433). |
| -c / ‑‑cert | Client certificate for client authentication (PEM file, optional). |
| -k / ‑‑key | Client key for client authentication (PEM file, optional). |
| ‑‑invalidCertificate | Invalid client certificate for client authentication (PEM file, optional). |
| ‑‑caCert | CA certificate for peer verification (PEM file, optional). |
| ‑‑secrets | File to store the QUIC secrets (file, optional). |
| ‑‑log | Show mapper's log in terminal (optional). |
| ‑‑learnlib | Use LearnLib instead of AALPY. |
| -r / ‑‑run | Command to run the QUIC server (e.g., "python3 http3_server.py ‑‑port 4431 ‑‑certificate server-cert.pem ‑‑private-key server-key.pem"). |
Notes: The Mapper will use the default certificate and private key in secrets.
The testing can be either run inside Docker, or manually built and run. Docker files for testing all 19 QUIC implementations are provided in dockerFile.
cd dockerFile/aioquic/fuzz_aioquic
sudo docker compose build --no-cache
sudo docker compose up fuzz_aioquic_Bmkdir quicServers
cd quicServers
git clone https://github.com/aiortc/aioquic.git aioquic
cd aioquic
git checkout 239f99b8
#### Aioquic virtual environment installation (tested on Ubuntu 20.04)
python3 -m venv env
source env/bin/activate
python -m pip install -e .
python -m pip install asgiref dnslib httpbin starlette wsproto --no-cache-dir
python -m pip install Werkzeug==2.0.3
python -m pip install "flask<2.2.0"
deactivate
cd ../..
# Run
cd quicLearner
# Test Basic (default handshake) configuration with short timeout (use "-i B-l" for long timeout and "-i B" for mixed timeout)
setsid env/bin/python3 learner.py -s aioquic --config B -i B-s -p 3341 -t 4431 --learnlib -r "../quicServers/aioquic/env/bin/python ../quicServers/aioquic/examples/http3_server.py --port 4431 --certificate ../secrets/serverCert/server-cert.pem --private-key ../secrets/serverCert/server-key.pem -l ../secrets/aioquicServer.log" > run/learner_aioquic_B-s_3341.log 2>&1 &
# Test Basic with Retry (handshake with client address validation) configuration with short timeout (use "-i BWR-l" for long timeout and "-i BWR" for mixed timeout)
setsid env/bin/python3 learner.py -s aioquic --config BWR -i BWR-s -p 3342 -t 4432 --learnlib -r "../quicServers/aioquic/env/bin/python ../quicServers/aioquic/examples/http3_server.py --port 4432 --certificate ../secrets/serverCert/server-cert.pem --private-key ../secrets/serverCert/server-key.pem -l ../secrets/aioquicServer.log --retry" > run/learner_aioquic_BWR-s_3342.log 2>&1 &Decryption keys for Wireshark are in secrets/aioquicServer.log or secrets/mapperSecrets.log
The results will be stored in the results/<serverName>Models/<serverName>-<serverConfig>-<inputDictionary>-<N>/ directory after the testing is completed. The testing process typically lasts for several hours, or several days for some configurations.
Files to analyse:
- optimisedLearnedModel.pdf (tools are provided in
scripts/) - stat.txt (for information on server crashes during testing)
Currently, we support 19 QUIC implementations, to add new implementations, in quicMapper/inOutPut.py:
- Add the server name, short timeout and long timeout to the
Serverclass. - Add the server name to the
LISTin bottom of theServerclass.
- Add the server to the
startConfigureMapper()function. - For
alpnProtocols,allAlpnis used in default.
Once these are set, QUICTester is ready to test a new QUIC server. Have fun!
Modify quicMapper/mapper.py and any relevant code (e.g. connection.py).
Then, in quicMapper/inOutPut.py:
- Add the input symbol to
Inputclass. - Add output symbol to
Outputclass (if any). - Create a new dictionary with the new input in
InputDictionaryclass. - Create a string for the new input dictionary and add the new input dictionary to the
args.inputlist in quicLearner/learner.py and quicMapper/mapper.py. - Map the input dictionary string to the input dictionary in quicLearner/learner.py:main().