CLI tool to extract all properties from the Redpanda's source code and generate a JSON output with their definitions.
- Python 3
- A C++ compiler (tested with
gcc)
- After cloning the project, initialize and update the C++ tree-sitter submodule:
git submodule init
git submodule update- Install the
tree_sitterdependency for Python:
pip install -r requirements.txtOptionally, you can use venv to avoid installing this dependency globally:
python -m venv /path/to/your/venv
source /path/to/your/vm/bin/activate
pip install -r requirements.txt- You will also need to download Redpanda's source code:
git clone https://github.com/redpanda-data/redpanda.git <dir>Alternatively, you can run make to generate the file. It will be stored at ./gen/properties-output.json.
./property_extractor.py [-h] --path PATH [--recursive] [--output OUTPUT] --definitions DEFINITIONS [-v]
options:
-h, --help show this help message and exit
--path PATH Path to the Redpanda's source dir to extract the properties
--recursive Scan the path recursively
--output OUTPUT File to store the JSON output. If no file is provided, the JSON will be printed to the standard output
--definitions DEFINITIONS
JSON file with extra type definitions to be used during the output generation
-v, --verbose
Example usage:
./property_extractor.py --recursive --path /path/to/redpanda --definitions definitions.json --output my-output.json
A transformer is an object used to add, remove, or modify the fields of a property. To create a new transformer, you need to create a class with the following methods:
accepts(self, info, file_pair) -> boolwhere:info: Information about the property extracted from the source code. This dictionary contains the following fields:name_in_file: name of the property as declared in the header filedeclaration: the whole line of the property declaration in the header file. This is useful because it contains the C++ type of the propertyparams: list of the params used in the constructor's initializer list to create the property in the implementation file (in that order):- The name of the property
- The description of the property
- The metadata object
- The default value
file_pair: object with two propertiesheaderandimplementationcontaining the path (as string) of the files that the property being processed belongs to
pa 60AC rse(self, property, info, file_pair) -> voidwhere:infoandfile_pairare the same as in theacceptsmethodpropertyis a dictionary where you can operate to modify the fields in the JSON output
To use your new transformer, add it to the list of enabled transformers. To see the existing transformers, check the transformers.py file.
The definitions.json file is merged in the output under the definitions field. Those definitions are used to change the types of the properties in the last stage of the generation. To change the types used in the generation, modify the file or provide another one using the --definitions option.