Add rule-ctl.py script #2068
Add rule-ctl.py script #2068
Conversation
b2fe419 to
d21c482
Compare
|
This is a very, very useful script that makes msc_pyparser much more accessible. I'll try to get @airween to review this PR and provide some feedback, so this can be finished and merged asap. |
|
Meeting decision June (#2074 (comment)): |
|
Very minor point... some lines in multi-line command examples in README.md are missing trailing \ |
|
Running command (as non-root user) results in warnings: |
How did you install the |
|
on ubuntu 20.04 LTS |
|
Hmm... it's interesting. What do you get when you type this command? >>> print(msc_pyparser.__version__)You should get I have cloned theMiddle's repository, and have tried to repdoduce your issue, but I couldn't. |
|
|
Which Python version do you have? I could reproduce your issue on Python3.5 - looks like this version does not install the I have to review it - workaround: try to run as root the script at first time, that will generate the necessary files. |
Looked like warnings were not fatal, script seemed to be producing expected results. As per your workaround, I can confirm running script as root, has no warnings, and results in the creation of: /usr/local/lib/python3.8/dist-packages/parser.out and /usr/local/lib/python3.8/dist-packages/parsetab.py . Subsequently running as non-root then has no warnings. |
|
I've tried adding/removing transformation functions and tags. Might be expected behaviour, but attempts to remove multiple tags seems to silently fail to remove all the tags: whereas it does seem possible to remove multiple transformation functions: also I kind of expected (possibly wrongly/unreasonably) that transformation function removal would occur before appending so that it would be possible to strip transformation functions and re-add them in a specific order: |
|
@theMiddleBlue : We've had a few comments for this welcome script. Can you incorporate these and finish the PR? Would be nice to merge this in the coming days. |
|
I have found a strange behavior in the 132,133c134
< SecRule MATCHED_VARS "@pm =" \
< "capture,\
---
> SecRule MATCHED_VARS "@pm =" "capture,\
(the first action after the operator and its argument goes into same line, not the next one) There are few lines where the line increment statements are commented, eg. here. May be this caused the behavior above. I can take a look it later, if you need. |
|
Meeting decision July: |
|
@airween How did you produce that issue? The simple things I've tried all work. |
|
Hi @theseion, here is how I did it: $ git branch
* add-script-rule-ctl
v3.4/dev
$ python3 rule-ctl.py --config ../../rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf --filter-rule-id ^933.+ --append-tag foo
$ git diff ../../rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
diff --git a/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf b/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
index 49974f1..271b171 100644
--- a/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
+++ b/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
@@ -14,6 +14,7 @@
SecRule TX:EXECUTING_PARANOIA_LEVEL "@lt 1" "id:933011,phase:1,pass,nolog,skipAfter:END-REQUEST-933-APPLICATION-ATTACK-PHP"
+
SecRule TX:EXECUTING_PARANOIA_LEVEL "@lt 1" "id:933012,phase:2,pass,nolog,skipAfter:END-REQUEST-933-APPLICATION-ATTACK-PHP"
#
# -= Paranoia Level 1 (default) =- (apply only when tx.executing_paranoia_level is sufficiently high: 1 or higher)
@@ -58,12 +59,12 @@ SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAME
tag:'paranoia-level/1',\
tag:'OWASP_CRS',\
tag:'capec/1000/152/242',\
+ tag:'foo',\
...
...
@@ -125,12 +126,12 @@ SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAME
tag:'paranoia-level/1',\
tag:'OWASP_CRS',\
tag:'capec/1000/152/242',\
+ tag:'foo',\
ctl:auditLogParts=+E,\
ver:'OWASP_CRS/3.3.0',\
severity:'CRITICAL',\
chain"
- SecRule MATCHED_VARS "@pm =" \
- "capture,\
+ SecRule MATCHED_VARS "@pm =" "capture,\
setvar:'tx.php_injection_score=+%{tx.critical_anomaly_score}',\See the last few lines. The new lines with |
|
@theseion can you confirm (or refute :)) this behavior? I mean after the transformation (eg. append a new tag) the firs action moves into same line as variable and operator are. |
|
Another strange behavior: if there is no any |
|
Sorry for the delay.
Yes, I can confirm that. Only appears to affect chained rules.
The |
only appears in case of chained rules, if there is any non-chained rule before that :)
I think the problem is that the code doesn't follow the number of appended lines (as an offset/shift value), only increments by 1 the lines. I can take a look later. |
|
@theMiddleBlue, @theseion - would you like to review this patch? This solves only the |
|
I'm a bit busy but I'll try to get look at the patch in a couple of days. |
thanks for fixing it! I'll try it asap |
|
The patch works. I've started refactoring the script heavily because I didn't understand how it works. If something useful comes out of that I'll contribute that. |
Let me know if I can help you anything. |
|
There is still work going on for this PR and reviews are happening. So I suggest we skip it for the agenda of the meeting tonight. Feel free to merge if you guys think you are done. |
|
FYI: I have a working implementation. I'm writing tests at the moment, which will take me a while but we'll have a proper test suite for all the actions at the end. |
|
Hey @theseion, the expectation of getting a test suite for this is very nice. Do you have a time line for this? |
|
I'm working on it now. I hope to have it ready by monday. |
|
I've opened a new PR with the refactored version of the script. The PR includes a pytest test suite. |
|
Closing this in favour of #2193 |
This is the first draft of rule-ctl.py. This is the first README:
draft
OWASP CRS Rule Control Script
This script aims to help when a bulk change on configuration files is needed. rule-ctl.py can, for example, change the value of an action on all rules, or can add/remove/rename a tag on each rule in a file, or can add/remove a transformation function only in rules that match range 942100-942190, etc...
Example Usage
There're only two mandatory parameters:
--configand--filter-rule-id.--config set the target config file
--filter-rule-id a regex that matches only rule ids to change
For example, if you want to add a new tag on each rule in file
REQUEST-933-APPLICATION-ATTACK-PHP.confyou can do:python3 util/rule-ctl/rule-ctl.py \ --config rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf \ --filter-rule-id ^933.+ \ --append-tag foo --dryrun--dryrunsends to stdout the result of changes and prevent writing changes on file. It's a good idea to always check all commands with dryrun before overwrite the target configuration file.You can even alphabetically sort tag list while adding new tags:
python3 util/rule-ctl/rule-ctl.py \ --config rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf \ --filter-rule-id ^933.+ \ --append-tag foo --sort-tag --dryrunVariables
--append-variable: Append a variable on the variable list of selected rules--remove-variable: Remove exact matching variable from selected rules--replace-variable: Replace variable on selected rules--replace-variable-name: Replace just variable name part on selected rulesExamples
Replace all
ARGSvariable withARGS_GETeven ifARGS:foopython3 rule-ctl.py --config tests/rules.conf \ --filter-rule-id ^.* \ --replace-variable-name ARGS,ARGS_GET \ --dryrunTags
--append-tag: Append a new tag to the tag list on selected rules--remove-tag: Remove tag from tag list on selected rules--rename-tag: Rename tag on selected rules--sort-tags: Alphabetically sort tag list on selected rulesExamples
Append a new tag
fooand sort tag listpython3 rule-ctl.py --config tests/rules.conf \ --filter-rule-id ^.* \ --append-tag foo \ --sort-tags \ --dryrunTransformation Functions
--append-tfunc: Append a new transformation function on selected rules--remove-tfunc: Remove a transformation function on selected rulesExamples
Append
t:lowercaseto all selected rules (you don't need thet:prefix)python3 rule-ctl.py --config tests/rules.conf \ --filter-rule-id ^.* \ --append-tfunc lowercase \ --dryrunActions
--replace-action: Replace action on selected rules--uncond-replace-action: Unconditional replace action on selected rules--remove-action: remove action from selected rulesExamples
Replace action
severity:CRITICALwithseverity:INFOand set a new message on rule id 125python3 rule-ctl.py --config tests/rules.conf \ --filter-rule-id ^125 \ --replace-action severity:CRITICAL,severity:INFO \ --uncond-replace-action 'msg:this is a new message for rule 125' \ --dryrunCTL
--append-ctl: Append a new ctl action on selected rulesExamples
Remove rule id 1337 on rule 125 by adding ctl:ruleRemoveById=1337. Do it on main rule (skipping chained rules if present)
python3 rule-ctl.py --config tests/rules.conf \ --filter-rule-id ^125 \ --append-ctl ruleRemoveById=1337 \ --skip-chain \ --dryrunOthers
--target-file: Set the target file where changes will be saved (default: use file set by--config)--skip-chain: Skip chained rules--dryrun: Do not write any changes, just output the results--debug: Show debug messages--silent: Used with--dryrunand--debugdoesn't write and shows only debug messages--json: Used with--dryrunreturn the msc_pyparser JSON output instead of ModSecurity file