📚 Documentation 💠 Configuration Hub 💬 Discourse (Forum) 💬 Gitter (Live chat) CrowdSec Hub for parsers, enrichers and scenarios. Foreword This repository stores most of the official parsers/scenarios/collections for crowdsec. The repository is not intended for use as-is, but rather as source of truth for the CrowdSec Hub and cscli. Feel free to use the parsers/scenarios here as a source of inspiration. Testing & Continuous integration cscli provides a hubtest sub-command to help contributors to create tests for parsers and scenarios. View & use existing tests ⚠️ most of cscli hubtest commands are expected to be run from the root directory of the hub. A git clone of this repository is the easier way to work ⚠️ list existing tests cscli hubtest list run a specific test cscli hubtest run [test-name] show current tests coverage cscli hubtest coverage Create your own (parser) test We're going to create the CI tests for the dovecot-parser. Before you start : you will need some actual logs you'd better know if the service logs on its own or via syslog (we're in the later case here) Create a new test ▶ cscli hubtest create dovecot-logs --type syslog Test name : dovecot-logs Test path : .../github/hub/.tests/dovecot-logs Log file : .../github/hub/.tests/dovecot-logs/dovecot-logs.log (please fill it with logs) Parser assertion file : .../github/hub/.tests/dovecot-logs/parser.assert (please fill it with assertion) Scenario assertion file : .../github/hub/.tests/dovecot-logs/parser.assert (please fill it with assertion) Configuration File : .../github/hub/.tests/dovecot-logs/config.yaml (please fill it with parsers, scenarios...) What is relevant here is that every test is composed of : A log file and it's associated type (same type as seen in acquis labels:type) A configuration specifying which parsers and/or scenarios must be enabled for the test A ultimately list of assertions that must be run against the parsers and/or scenarios output Note: You can provide the parsers and scenarios you want in your test with --parsers and --scenarios (you can provide multiple parsers and scenarios) If you want to test only a scenario, you can specify (--ignore-parsers) or set the ignore_parsers to true in the config.yaml Configure your test We need to edit the test configuration to use the relevant parsers : ▶ cat .../github/hub/.tests/dovecot-logs/config.yaml parsers: - crowdsecurity/syslog-logs - crowdsecurity/dovecot-logs scenarios: postoverflows: - "" log_file: dovecot-logs.log log_type: syslog note: the order doesn't matter. If the parser name is in the form author/parser it's from the hub, but relative paths are allowed for non-versioned parsers Now we need to dump some actual logs into the test's log file : ▶ cat > .tests/dovecot-logs/dovecot-logs.log Jan 28 10:16:13 dovecot-box dovecot[7508]: imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=<toto@toto.com>, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=<3650VvK5bdIaW-iK> Sep 8 07:16:29 canyon dovecot: auth-worker(24058): pam(toto,1.1.1.1,<youpi>): pam_authenticate() failed: Authentication failure (password mismatch?) Sep 8 07:46:51 canyon dovecot: auth-worker(24544): pam(toto,1.1.1.1): unknown user Run the test for the first time Now that we have config & logs, let's run it for the first time : ▶ cscli hubtest run dovecot-logs INFO[27-09-2021 06:13:59 PM] Running test 'dovecot-logs' INFO[27-09-2021 06:13:59 PM] parser 'crowdsecurity/dovecot-logs' installed succesfully in runtime environment INFO[27-09-2021 06:13:59 PM] parser 'crowdsecurity/syslog-logs' installed succesfully in runtime environment WARN[27-09-2021 06:14:02 PM] Assert file '/home/bui/github/hub/.tests/dovecot-logs/parser.assert' is empty, generating assertion: results["s00-raw"]["crowdsecurity/syslog-logs"][0].Success == true ... results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Success == true results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["pid"] == "7508" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["timestamp"] == "Jan 28 10:16:13" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_login_result"] == "Disconnected (auth failed, 1 attempts in 6 secs)" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["message"] == "imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=<toto@toto.com>, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=<3650VvK5bdIaW-iK>" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["logsource"] == "syslog" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["program"] == "dovecot" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["protocol"] == "imap" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_local_ip"] == "7.7.7.7" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_remote_ip"] == "4.4.4.4" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_user"] == "toto@toto.com" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_path"] == "dovecot-logs.log" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_type"] == "file" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["log_type"] == "dovecot_logs" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["source_ip"] == "4.4.4.4" ... As our parser.assert is empty, the tool is generating some "suggested" asserts for us. Your careful eye will keep only the ones relevant to the parser you're testing : ▶ cat > .tests/dovecot-logs/parser.assert results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Success == true results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["pid"] == "7508" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["timestamp"] == "Jan 28 10:16:13" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_login_result"] == "Disconnected (auth failed, 1 attempts in 6 secs)" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["message"] == "imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=<toto@toto.com>, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=<3650VvK5bdIaW-iK>" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["logsource"] == "syslog" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["program"] == "dovecot" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["protocol"] == "imap" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_local_ip"] == "7.7.7.7" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_remote_ip"] == "4.4.4.4" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_user"] == "toto@toto.com" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_path"] == "dovecot-logs.log" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_type"] == "file" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["log_type"] == "dovecot_logs" results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["source_ip"] == "4.4.4.4" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Success == true results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["dovecot_login_result"] == "Authentication failure (password mismatch?)" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["logsource"] == "syslog" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["message"] == "auth-worker(24058): pam(toto,1.1.1.1,<youpi>): pam_authenticate() failed: Authentication failure (password mismatch?)" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["program"] == "dovecot" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["timestamp"] == "Sep 8 07:16:29" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["dovecot_remote_ip"] == "1.1.1.1" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["dovecot_user"] == "toto" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["datasource_path"] == "dovecot-logs.log" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["datasource_type"] == "file" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["log_type"] == "dovecot_logs" results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["source_ip"] == "1.1.1.1" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Success == true results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["dovecot_login_result"] == "unknown user" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["program"] == "dovecot" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["dovecot_remote_ip"] == "1.1.1.1" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["dovecot_user"] == "toto" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["logsource"] == "syslog" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["message"] == "auth-worker(24544): pam(toto,1.1.1.1): unknown user" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["timestamp"] == "Sep 8 07:46:51" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["log_type"] == "dovecot_logs" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["source_ip"] == "1.1.1.1" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["datasource_path"] == "dovecot-logs.log" results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["datasource_type"] == "file" Test your newly crafted test ▶ cscli hubtest run dovecot-logs INFO[27-09-2021 06:19:33 PM] Running test 'dovecot-logs' INFO[27-09-2021 06:19:33 PM] parser 'crowdsecurity/syslog-logs' installed succesfully in runtime environment INFO[27-09-2021 06:19:33 PM] parser 'crowdsecurity/dovecot-logs' installed succesfully in runtime environment Test 'dovecot-logs' passed successfully (39 assertions) 🟩 And be amazed. Debug your own (parser) test Things went wrong ? Don't panic When working on a test, you can as well pass expressions directly to hubtest command and see the results : ▶ cscli hubtest eval dovecot-logs -e 'results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed' dovecot_login_result: unknown user dovecot_remote_ip: 1.1.1.1 dovecot_user: toto facility: "" logsource: syslog message: 'auth-worker(24544): pam(toto,1.1.1.1): unknown user' pid: "" priority: "" program: dovecot timestamp: Sep 8 07:46:51 timestamp8601: "" Open your PR yes.