`curl` and `wget` are matched in User-Agent header #3189

theseion · 2023-04-04T19:40:49Z

The valid user agents for curl and wget are matched by rules 932236 and 932237 in the User-Agent header.

The text was updated successfully, but these errors were encountered:

emphazer · 2023-04-04T20:03:58Z

you are absolutely right. that's why we sadly had to remove this rules for our new deployment. I would really appreciate if we could remove them from our list...

theMiddleBlue · 2023-08-10T07:27:25Z

just as a reminder to address this problem:

we still block curl and wget at PL2

coreruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf

Lines 1220 to 1239 in e36f27e

    
           SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/*|REQUEST_HEADERS:Referer|REQUEST_HEADERS:User-Agent "@rx (?i)(?:(?:^|=)[\s\v]*(?:t[\"'\)\[-\x5c]*(?:(?:(?:\|\||&&)[\s\v]*)?\$[!#\(\*\-0-9\?-@_a-\{]*)?\x5c?i[\"'\)\[-\x5c]*(?:(?:(?:\|\||&&)[\s\v]*)?\$[!#\(\*\-0-9\?-@_a-\{]*)?\x5c?m[\"'\)\[-\x5c]*(?:(?:(?:\|\||&&)[\s\v]*)?\$[!#\(\*\-0-9\?-@_a-\{]*)?\x5c?e|[\$\{]|(?:[\s\v]*\(|!)[\s\v]*|[0-9A-Z_a-z]+=(?:[^\s\v]*|\$(?:.*|.*)|[<>].*|'.*'|\".*\")[\s\v]+)*|(?:t[\"'\)\[-\x5c]*(?:(?:(?:\|\||&&)[\s\v]*)?\$[!#\(\*\-0-9\?-@_a-\{]*)?\x5c?i[\"'\)\[-\x5c]*(?:(?:(?:\|\||&&)[\s\v]*)?\$[!#\(\*\-0-9\?-@_a-\{]*)?\x5c?m[\"'\)\[-\x5c]*(?:(?:(?:\|\||&&)[\s\v]*)?\$[!#\(\*\-0-9\?-@_a-\{]*)?\x5c?e|[\n\r;`\{]|\|\|?|&&?|\$(?:\(\(?|\{)|[<>]\(|\([\s\v]*\))[\s\v]*(?:[\$\{]|(?:[\s\v]*\(|!)[\s\v]*|[0-9A-Z_a-z]+=(?:[^\s\v]*|\$(?:.*|.*)|[<>].*|'.*'|\".*\")[\s\v]+)*)[\s\v]*[\"']*(?:[\"'-\+\--9\?A-\]_a-z\|]+/)?[\"'\x5c]*(?:7z[ar]?|a(?:(?:b|w[ks])[\s\v&\)<>\|]|pt(?:-get)?|r(?:[\s\v&\)<>jp\|]|ch[\s\v<>]|ia2c)|s(?:[\s\v&\)<>h\|]|cii(?:-xfr|85)|pell)|t(?:[\s\v&\)<>\|]|obm)|dduser|getty|l(?:ias|pine)[\s\v<>]|nsible-playbook)|b(?:z(?:z[\s\v&\)<>\|]|c(?:at|mp)|diff|e(?:grep|xe)|f?grep|ip2|less|more)|a(?:s(?:e(?:32|64|nc)|h)|tch[\s\v<>])|pftrace|r(?:eaksw|idge[\s\v<>])|sd(?:cat|iff|tar)|u(?:iltin|n(?:dler[\s\v<>]|zip2)|s(?:ctl|ybox))|yebug)|c(?:[8-9]9|a(?:t[\s\v&\)<>\|]|(?:ncel|psh)[\s\v<>])|c[\s\v&\)<>\|]|mp|p(?:[\s\v&\)<>\|]|an|io|ulimit)|s(?:h|plit|vtool)|u(?:t[\s\v&\)<>\|]|psfilter|rl)|ertbot|h(?:attr|dir[\s\v<>]|eck_(?:by_ssh|cups|log|memory|raid|s(?:sl_cert|tatusfile))|flags|mod|o(?:om|wn)|root)|o(?:(?:b|pro)c|lumn[\s\v<>]|m(?:m(?:and[\s\v<>])?|p(?:oser|ress[\s\v<>]))|w(?:say|think))|r(?:ash[\s\v<>]|ontab))|d(?:[du][\s\v&\)<>\|]|i(?:g|(?:alog|ff)[\s\v<>])|nf|a(?:sh|te)[\s\v<>]|hclient|m(?:esg|idecode|setup)|o(?:as|(?:cker|ne)[\s\v<>]|sbox)|pkg|vips)|e(?:[bd][\s\v&\)<>\|]|n(?:v(?:[\s\v&\)<>\|]|-update)|d(?:if|sw))|qn|x(?:[\s\v&\)<>\|]|ec[\s\v<>]|iftool|p(?:(?:and|(?:ec|or)t)[\s\v<>]|r))|(?:asy_instal|va)l|cho[\s\v<>]|fax|grep|macs|sac)|f(?:c[\s\v&\)<>\|]|i(?:[\s\v&\)<>\|]|le(?:[\s\v<>]|test)|(?:n(?:d|ger)|sh)[\s\v<>])|mt|tp(?:[\s\v&\)<>\|]|stats|who)|acter|(?:etch|lock)[\s\v<>]|grep|o(?:ld[\s\v<>]|reach)|ping|unction)|g(?:c(?:c[^\s\v]|ore)|db|e(?:m[\s\v&\)<>\|]|ni(?:e[\s\v<>]|soimage)|tfacl[\s\v<>])|hci?|i(?:t[\s\v&\)<>\|]|mp[\s\v<>]|nsh)|o[\s\v&\)<>\|]|r(?:c|ep[\s\v<>])|awk|tester|unzip|z(?:cat|exe|ip))|h(?:(?:d|up)[\s\v&\)<>\|]|e(?:ad[\s\v<>]|xdump)|i(?:ghlight|story)[\s\v<>]|ost(?:id|name)|ping3|t(?:digest|passwd))|i(?:d[\s\v&\)<>\|]|p(?:6?tables|config)?|rb|conv|f(?:config|top)|nstall[\s\v<>]|onice|spell)|j(?:js|q|ava[\s\v<>]|exec|o(?:(?:bs|in)[\s\v<>]|urnalctl)|runscript)|k(?:s(?:h|shell)|ill(?:[\s\v<>]|all)|nife[\s\v<>])|l(?:d(?:d?[\s\v&\)<>\|]|config)|[np][\s\v&\)<>\|]|s(?:-F|b_release|cpu|hw|mod|of|pci|usb)?|ua(?:[\s\v&\)<>\|]|(?:la)?tex)|z(?:[\s\v&\)<>\|]|c(?:at|mp)|diff|[e-f]?grep|less|m(?:a|ore))|a(?:st(?:[\s\v<>]|comm|log(?:in)?)|tex[\s\v<>])|ess(?:[\s\v<>]|echo|(?:fil|pip)e)|ftp(?:get)?|(?:inks|ynx)[\s\v<>]|o(?:(?:ca(?:l|te)|ok)[\s\v<>]|g(?:inctl|(?:nam|sav)e))|trace|wp-(?:d(?:ownload|ump)|mirror|request))|m(?:a(?:n[\s\v&\)<>\|]|il(?:[\s\v<>q]|x[\s\v<>])|ke[\s\v<>]|wk)|tr|v[\s\v&\)<>\|]|(?:kdir|utt)[\s\v<>]|locate|o(?:(?:re|unt)[\s\v<>]|squitto)|sg(?:attrib|c(?:at|onv)|filter|merge|uniq)|ysql(?:admin|dump(?:slow)?|hotcopy|show)?)|n(?:c(?:[\s\v&\)<>\|]|\.(?:openbsd|traditional)|at)|e(?:t(?:[\s\v&\)<>\|]|(?:c|st)at|kit-ftp)|ofetch)|l[\s\v&\)<>\|]|m(?:[\s\v&\)<>\|]|ap)|p(?:m[\s\v&\)<>\|]|ing)|a(?:no[\s\v<>]|sm|wk)|ice[\s\v<>]|o(?:de[\s\v<>]|hup)|roff|s(?:enter|lookup|tat))|o(?:d[\s\v&\)<>\|]|ctave[\s\v<>]|nintr|p(?:en(?:ssl|v(?:pn|t))|kg))|p(?:a(?:x[\s\v&\)<>\|]|s(?:swd|te[\s\v<>])|tch[\s\v<>])|d(?:b|f(?:la)?tex)|f(?:[\s\v&\)<>\|]|tp)|g(?:rep)?|hp[\s\v&\)<>\|]|i(?:c(?:o[\s\v<>])?|p[^\s\v]|dstat|gz|ng[\s\v<>])|k(?:g(?:_?info)?|exec|ill)|r(?:y?[\s\v&\)<>\|]|int(?:env|f[\s\v<>]))|s(?:ftp|ql)?|t(?:x|ar(?:diff|grep)?)|xz|er(?:f|l(?:5|sh)?|ms)|opd|ython[^\s\v]|u(?:ppet[\s\v<>]|shd))|r(?:a(?:r[\s\v&\)<>\|]|k(?:e[\s\v<>]|u))|cp[\s\v&\)<>\|]|e(?:d(?:[\s\v&\)<>\|]|carpet[\s\v<>])|v|a(?:delf|lpath)|(?:name|p(?:eat|lace))[\s\v<>]|stic)|m(?:[\s\v&\)<>\|]|dir[\s\v<>]|user)|pm(?:[\s\v&\)<>\|]|db|(?:quer|verif)y)|l(?:ogin|wrap)|nano|oute[\s\v<>]|sync|u(?:by[^\s\v]|n-(?:mailcap|parts))|vi(?:ew|m))|s(?:c(?:p|hed|r(?:een|ipt)[\s\v<>])|e(?:d[\s\v&\)<>\|]|t(?:[\s\v&\)<>\|]|arch|env|facl[\s\v<>]|sid)|ndmail|rvice[\s\v<>])|g|h(?:[\s\v&\)<>\|]|\.distrib|ell|u(?:f|tdown[\s\v<>]))|s(?:[\s\v&\)<>\|]|h(?:[\s\v&\)<>\|]|-key(?:ge|sca)n|pass))|u(?:[\s\v&\)<>\|]|do)|vn|(?:ash|nap|plit)[\s\v<>]|diff|ftp|l(?:eep[\s\v<>]|sh)|mbclient|o(?:cat|elim|(?:rt|urce)[\s\v<>])|qlite3|t(?:art-stop-daemon|dbuf|r(?:ace|ings))|ys(?:ctl|tem(?:ctl|d-resolve)))|t(?:a(?:c|r[\s\v&\)<>\|]|il[\s\v<>f]|sk(?:[\s\v<>]|set))|bl|e(?:e|x[\s\v&\)<>\|]|lnet)|i(?:c[\s\v&\)<>\|]|me(?:(?:out)?[\s\v<>]|datectl))|o(?:p|uch[\s\v<>])|c(?:l?sh|p(?:dump|ing|traceroute))|ftp|mux|r(?:aceroute6?|off)|shark)|u(?:l(?:[\s\v&\)<>\|]|imit[\s\v<>])|n(?:ame|compress|expand|iq|l(?:ink[\s\v<>]|z(?:4|ma))|(?:pig|x)z|rar|s(?:et|hare)[\s\v<>]|z(?:ip|std))|pdate-alternatives|ser(?:(?:ad|mo)d|del)|u(?:de|en)code)|v(?:i(?:[\s\v&\)<>\|]|m(?:[\s\v&\)<>\|]|diff)|ew[\s\v<>]|gr|pw|rsh)|algrind|olatility)|w(?:3m|c|h(?:o(?:ami|is)?|iptail)|a(?:ll|tch)[\s\v<>]|get|i(?:reshark|sh[\s\v<>]))|x(?:(?:x|pa)d|z(?:[\s\v&\)<>\|]|c(?:at|mp)|d(?:ec|iff)|[e-f]?grep|less|more)|args|e(?:la)?tex|mo(?:dmap|re)|term)|y(?:um|arn|elp[\s\v<>])|z(?:ip(?:details)?|s(?:h|oelim|td)|athura|c(?:at|mp)|diff|[e-f]?grep|less|more|run|ypper))" \ 
        
               "id:932236,\ 
        
               phase:2,\ 
        
               block,\ 
        
               capture,\ 
        
               t:none,\ 
        
               msg:'Remote Command Execution: Unix Command Injection (command without evasion)',\ 
        
               logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',\ 
        
               tag:'application-multi',\ 
        
               tag:'language-shell',\ 
        
               tag:'platform-unix',\ 
        
               tag:'attack-rce',\ 
        
               tag:'paranoia-level/2',\ 
        
               tag:'OWASP_CRS',\ 
        
               tag:'capec/1000/152/248/88',\ 
        
               tag:'PCI/6.5.2',\ 
        
               ver:'OWASP_CRS/4.0.0-rc1',\ 
        
               severity:'CRITICAL',\ 
        
               setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',\ 
        
               setvar:'tx.inbound_anomaly_score_pl2=+%{tx.critical_anomaly_score}'"

$ curl -s \
   -H 'x-backend: nginx' \
   -H 'x-crs-paranoia-level: 2' \
   -H 'x-format-output: txt-matched-rules' \
   'https://sandbox.coreruleset.org/'

932236 PL2 Remote Command Execution: Unix Command Injection (command without evasion)
949110 PL1 Inbound Anomaly Score Exceeded (Total Score: 5)

it doesn't make any sense to me to block curl UA as a RCE injection.
thoughts?

franbuehler · 2023-08-10T08:12:09Z

Rule 932236 was introduced in this commit and rule 932237 in this commit to address Shivam Bathla's bypass list.

Maybe we can leave the rules as they are but work with an include-except and add these 2 valid user-agents as exclude list. Should I give it a try or are you already solving this issue?
Update: Ah, I see, there's already a linked PR. Sorry!!

dune73 · 2023-08-10T08:20:34Z

I am a wee bit reluctant to remove this.

We're at PL2 and we're looking at RCE. wget and curl are definitely risky. The problem is, we are treating ARGS and UA with the same rule. As we should, but when it comes to curl and wget this causes FPs on the UA. But if we remove them from this rule, we also remove them from the ARGS check and that would be dangerous. So we have 3 options:

Accept FP at PL2
Remove curl and wget from source list, allowing for RCE attacks via ARGS using curl and wget
Separate Referer and UA check from ARGS check by splitting the rule and then remove curl and wget from new rule

Given we're at PL2, I am reluctantly for solution 1: accepting the FP and let people deal with it via rule exclusions.

theMiddleBlue · 2023-08-10T08:27:40Z

we can't leave this rule as is.

I get where you're coming from about accepting false positives at PL2, but I don't really see the point of matching "curl" and "wget" in the User-Agent as we do with this rule.

Let's face it, anyone running CRS at PL2 in a real-world scenario would probably end up removing this rule anyone. Plus, we're trying to shield the app from things like system($user_agent_string); which is not something we can do IMO.

franbuehler · 2023-08-10T08:47:21Z

Separate Referer and UA check from ARGS check by splitting the rule and then remove curl and wget from new rule

This is what @theseion suggested in PR #3190. However, there are still open change requests. But I think the idea makes sense.

theMiddleBlue · 2023-08-10T08:56:57Z

I totally get what @theseion is suggesting in PR #3190. But as I commented there, the solution makes this rule completely useless since it would introduce an easy bypass like <my payload> #curl. It can't be done like this IMO.

franbuehler · 2023-08-10T09:10:07Z

Ah, and this bypass can't be solved so easy... It's not just changing the !@pmFromFile to another operator or regex or so...
Now, I get it. Sorry. I thought that maybe this bypass issue could be solved.

franbuehler · 2023-08-10T09:22:48Z

Maybe it could be solved if we worked with a generated regex instead of pmFromFile and include-except list (with curl an wget in it) for the new rules 932238 and 932239.

theMiddleBlue · 2023-08-10T09:43:28Z

It could be a good approach! But I'm not a big fan of exceptions to be honest, they often leads to bypasses. Moreover, the problem is not only with curl and wget. I have also matches with the following UA:

Cpanel-HTTP-Client/1.0
python-requests/2.31.0
Python-urllib/3.9
Snap URL Preview Service; bot; snapchat; https://developers.snap.com/robots
MailChimp
w3m/0.5.1
Links (2.3pre1; Linux 2.6.38-8-generic x86_64; 170x48)

just in the last 24h

dune73 · 2023-08-10T11:52:59Z

I agree that the exceptions lead to poor rules (based on rule architecture, MATCHED_VAR_NAME is often broken) and there is a substantial bypass problem.

If you really think automated agents should not trigger alerts on PL2, then I think we need a separate rule for UA (and referrer perhaps) that has the same source, but uses include-except.

RedXanadu · 2023-08-14T12:01:21Z

To add some context: this issue was discussed at the April meeting. (Full discussion here, search for "UA problem goes first".) We agreed in April that this issue needs to be fixed.

I agree with @theMiddleBlue that this rule cannot be left in its current state. Tools like curl, wget etc. are legitimate and many CRS users use them.

I'm not sure the chain rule approach in #3190 is a good idea.

I thought there was a crs-toolchain function to exclude unwanted keywords? Let's use that?

theseion · 2023-08-24T19:13:56Z

The approach in #3190 isn't great, as it opens the door to evasions. include-except would be a viable option but it requires listing all benign user agents, much as I've done in #3190. As @theMiddleBlue pointed out, however, that's still prone to FPs, as long as the list isn't stable. And new UAs will pop up frequently and we'd have to make frequent updates to the list.

I now think that there's no way around a separate rule for the User-Agent header. If we did that then we could use include-except to exclude known user agents that are also in the list of Unix commands. Currently, only wget and curl would be excluded, I think.

Opinions?

dune73 · 2023-09-01T06:40:18Z

Just to make sure I get you correctly: You would use the entire Unix command list for include-except or only a (manual) subset? I presume it's the latter but if it is, then only advantage would be that we know which list to use for the manual review and construction of the include-except list.

Given we are stuck with 2 PRs (#3190 and #3276) touching on these rules and since we have been chewing on this problem for several months, I think the separate rule is the right way to address this.

theseion · 2023-09-04T16:28:44Z

Yes, check all words but exclude curl and wget.

franbuehler · 2023-09-12T06:40:53Z

I will work on this after #3276 is merged.

franbuehler · 2023-09-20T05:44:51Z

Now that the PR is merged, I'll work on this before the feature freeze.

theseion added the ➕ False Positive label Apr 4, 2023

theseion self-assigned this Apr 4, 2023

8000

theseion mentioned this issue Apr 4, 2023

fix: exclude well known user agents from unix commands #3190

Closed

azurit mentioned this issue Apr 19, 2023

Nextcloud cookie values collision with 932250 and 932236 #3197

Closed

dune73 mentioned this issue Sep 4, 2023

Monthly Chat Agenda September 2023 (2023-09-04 and 2023-09-18) #3279

Closed

franbuehler self-assigned this Sep 4, 2023

theseion added the v4 Should go into release v4 label Sep 19, 2023

theseion mentioned this issue Sep 21, 2023

Rule 932260 false positive with "scheduledAt" keyword #3288

Closed

franbuehler mentioned this issue Sep 23, 2023

fix: remove user-agents from rules 932236 and 932237 #3318

Merged

theseion closed this as completed in #3318 Oct 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

`curl` and `wget` are matched in User-Agent header #3189

`curl` and `wget` are matched in User-Agent header #3189

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

curl and wget are matched in User-Agent header #3189

curl and wget are matched in User-Agent header #3189

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

`curl` and `wget` are matched in User-Agent header #3189

`curl` and `wget` are matched in User-Agent header #3189