[panw_cortex_xdr] Add event data stream and dashboards of incident and alert #13680
Conversation
…Cloud Storage (GCS) input and add dashbaords for alert and incident data-stream
|
@jamiehynds, @kcreddy, @ShourieG — We've encountered an issue with the System Test for the GCS input type. As suggested by @ShourieG, we'll connect with @bhapas to see if it can be replicated similarly to how we handled it for AWS. |
|
@piyush-elastic @janvi-elastic, can you add system tests configs to the PR to understand the exact failure? |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
@kcreddy, the issue is with the fake-gcs-server.There's no way to assign content-type metadata to files from within the docker env/command arguments. Only using a cURL can you set content-types to a particular uploaded file. This is important because the gcs input has a content-type filter that discards any unsupported content-types passed to it. When using real gcs buckets, this metadata is always present hence there's no issue. So to achieve proper system tests we need to do it via terraform with real buckets like awss3 does (though the reason we do it in s3 is not because of content-type limitations). |
|
@kcreddy I have added system test configs in the PR. |
|
@ShourieG thanks for the details 👍🏼 . |
|
@kcreddy, if you look at the sample event you will see it contains an error complaining about content-type. Since we are only checking for hits there this was ignored. At the time this system test was integrated we did not have any terraform setups so this was the only way. But now since we have it, this would ideally also need to be updated. |
| @@ -727,85 +727,3 @@ rules: | |||
| ] | |||
| } | |||
| } | |||
| - path: /public_api/v1/incidents/get_incidents | |||
There was a problem hiding this comment.
As we are going with adding terraform for gcs input of event data-stream we need to mode deploy from package to data-stream level so we have removed configuration for incident in alert data-stream same for incident data-stream also.
packages/panw_cortex_xdr/data_stream/event/_dev/test/pipeline/test-event.log-expected.json
Show resolved
Hide resolved
packages/panw_cortex_xdr/data_stream/event/_dev/test/pipeline/test-event.log-expected.json
Show resolved
Hide resolved
packages/panw_cortex_xdr/data_stream/event/_dev/test/pipeline/test-event.log-expected.json
Show resolved
Hide resolved
packages/panw_cortex_xdr/data_stream/event/_dev/test/pipeline/test-event.log-expected.json
Outdated
Show resolved
Hide resolved
|
@tomsonpl this PR adds support for ingesting events/telemetry from Cortex XDR. Are there any mappings we need to have in place to ensure alignment with what the Analyzer view or Response actions expect? I'm hoping if we can align on mappings now, it will avoid rework and breaking changes down the line... |
🚀 Benchmarks reportTo see the full report comment with |
| }, | ||
| "user": { | ||
| "id": "S-1-5-18", | ||
| "name": "NT AUTHORITY\\SYSTEM" |
There was a problem hiding this comment.
There are few more fields that can be populated into process.entry_leader.
Looks like remote process hash is being copied into process.hash instead of process.entry_leader.hash. Please change that.
Also you could add process.entry_leader.pid from action_remote_process_os_pid.
There was a problem hiding this comment.
We are unable to find an ecs mapping is available for process.entry_leader.hash* fields.
There was a problem hiding this comment.
@janvi-elastic May I know if you don't add these fields, the pipeline tests are failing?
packages/panw_cortex_xdr/data_stream/event/_dev/deploy/docker/docker-compose.yml
Outdated
Show resolved
Hide resolved
packages/panw_cortex_xdr/data_stream/event/_dev/test/pipeline/test-event.log-expected.json
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
@janvi-elastic, I already did a re-review earlier today. Comments here: #13680 (review)
packages/panw_cortex_xdr/data_stream/event/agent/stream/gcs.yml.hbs
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
I am unable to run the system tests for the event data stream due to
ERROR: could not setup service: Terraform deployer is unhealthy: container (ID: 1d89c3f8ed146fefed077c6f6ce7038fd9ce3a2cb8e1298b11912a826db17f0d) exited with code 1
What is required to allow this to work?
The changes that are required to get the static tests to pass are:
diff --git a/packages/panw_cortex_xdr/data_stream/event/sample_event.json b/packages/panw_cortex_xdr/data_stream/event/sample_event.json
index c053126c4e..3470dc2bdb 100644
--- a/packages/panw_cortex_xdr/data_stream/event/sample_event.json
+++ b/packages/panw_cortex_xdr/data_stream/event/sample_event.json
@@ -173,7 +173,7 @@
"image_path": "C:\\Windows\\System32\\svchost.exe",
"instance_id": "remote-host-01::1234::5678",
"integrity_level": "High",
- "os_pid": "1234",
+ "os_pid": 1234,
"signature_is_embedded": true,
"signature_product": "Microsoft Windows Operating System",
"signature_status": "Valid",
diff --git a/packages/panw_cortex_xdr/docs/README.md b/packages/panw_cortex_xdr/docs/README.md
index 806b986771..f7a73a247e 100644
--- a/packages/panw_cortex_xdr/docs/README.md
+++ b/packages/panw_cortex_xdr/docs/README.md
@@ -923,7 +923,7 @@ An example event for `event` looks as following:
"image_path": "C:\\Windows\\System32\\svchost.exe",
"instance_id": "remote-host-01::1234::5678",
"integrity_level": "High",
- "os_pid": "1234",
+ "os_pid": 1234,
"signature_is_embedded": true,
"signature_product": "Microsoft Windows Operating System",
"signature_status": "Valid",| environment: | ||
| - GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS:-"/tmp/creds.json"} | ||
| - GOOGLE_PROJECT=${GOOGLE_PROJECT:-elastic-12345} | ||
| - GOOGLE_REGION=${GOOGLE_REGION:-US} |
There was a problem hiding this comment.
I don't seem to have access this project: elastic-12345.
@janvi-elastic, are there any additional steps to make it work?
|
|
||
| output "bucket_name" { | ||
| value = google_storage_bucket.bucket.name | ||
| } |
| credentials = var.GOOGLE_CREDENTIALS | ||
| } | ||
|
|
||
| resource "google_storage_bucket" "bucket" { |
There was a problem hiding this comment.
| resource "google_storage_bucket" "bucket" { | |
| resource "google_storage_bucket" "panw_cortex_xdr_event_bucket" { |
| location = "US" | ||
| } | ||
|
|
||
| resource "google_storage_bucket_object" "object" { |
There was a problem hiding this comment.
| resource "google_storage_bucket_object" "object" { | |
| resource "google_storage_bucket_object" "panw_cortex_xdr_event_bucket_object" { |
| source = var.FILE_PATH | ||
| } | ||
|
|
||
| output "bucket_name" { |
There was a problem hiding this comment.
| output "bucket_name" { | |
| output "panw_cortex_xdr_event_bucket_name" { |
| wait_for_data_timeout: 1m | ||
| data_stream: | ||
| vars: | ||
| service_account_key: "{{GOOGLE_APPLICATION_CREDENTIALS}}" |
There was a problem hiding this comment.
GOOGLE_APPLICATION_CREDENTIALS --> Where is this coming from?
There was a problem hiding this comment.
IIRC this environment variable is added via the OIDC plugin.
To be allowed to use this variable in the test case configuration, I think this should be added as part of the environment variables defined in packages/panw_cortex_xdr/data_stream/event/_dev/deploy/tf/env.yml too. Is that right @jsoriano ?
version: '2.3'
services:
terraform:
environment:
- GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}
- GOOGLE_PROJECT=${GOOGLE_PROJECT}
- GOOGLE_REGION=${GOOGLE_REGION:-US}Should that environment variable be used to set GOOGLE_CREDENTIALS from https://github.com/elastic/integrations/pull/13680/files#diff-086fde7ab90e0f846da9a7162a1dbada5d8a0ea0281835d06d529a22105a00d4R26 ?
version: '2.3'
services:
terraform:
environment:
- GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}
- TF_VAR_GOOGLE_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}
- GOOGLE_PROJECT=${GOOGLE_PROJECT}
- GOOGLE_REGION=${GOOGLE_REGION:-US}There was a problem hiding this comment.
According to the docs, it looks like the provider can use GOOGLE_APPLICATION_CREDENTIALS directly, so it shouldn't be needed to pass it as a variable.
There was a problem hiding this comment.
@v1v whitelisted the bucket name and object name in GCP project. Since in the new OIDC way the pkugin writes the credentials and project into the env variables, I assume we can use directly
There was a problem hiding this comment.
@mrodm For testing locally we are manually updating creds in variable.tf & config.yml , unable to fetch it from environment variable.
| @@ -0,0 +1 @@ | |||
| {"_id":"c2c455d9f99375d","action_file_access_time":"2023-09-15T12:00:00Z","action_file_attributes":"readonly","action_file_create_time":"2023-09-01T12:00:00Z","action_file_device_type":"Device Type","action_file_group":"gr123","action_file_group_name":"GroupName","action_file_is_anonymous":true,"action_file_last_writer_actor":"Writer Actor","action_file_md5":"d41d8cd98f00b204e9800998ecf8427e","action_file_mod_time":"2023-09-15T12:00:00Z","action_file_mode":"0640","action_file_name":["MyOffice.BackgroundTasks.dll"],"action_file_operation_flags":"Operation Flags","action_file_owner":"Owner","action_file_owner_name":"OwnerName","action_file_path":["C:\\Program Files\\WindowsApps\\Microsoft.MicrosoftOfficeHub_18.2204.1141.0_x64__8wekyb3d8bbwe\\MyOffice.BackgroundTasks.dll"],"action_file_pe_info":"PE info","action_file_prev_type":"Previous Type","action_file_previous_file_name":"file.dll","action_file_previous_file_path":"C:\\PreviousPath\\file.dll","action_file_sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","action_file_signature_is_embedded":true,"action_file_signature_product":"File Product","action_file_signature_status":"Verified","action_file_signature_vendor":"File Vendor","action_file_size":123456,"action_file_type":"DLL","action_local_ip":["1.128.0.0"],"action_local_port":[8080],"action_module_base_address":"0x7fffb1d00000","action_module_boot_code_integrity":"enabled","action_module_code_integrity":"verified","action_module_file_info":"C:\\Program Files\\ExampleApp\\example.dll","action_module_image_size":"512","action_module_last_writer_actor":"SYSTEM","action_module_md5":"9e107d9d372bb6826bd81d3542a419d6","action_module_other_load_location":"C:\\Windows\\Temp\\example.tmp","action_module_page_protection":"PAGE_EXECUTE_READWRITE","action_module_path":"C:\\Program Files\\ExampleApp\\example.dll","action_module_sha256":"2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824","action_module_signature_is_embedded":true,"action_module_signature_product":"Example App Suite","action_module_signature_status":"Valid","action_module_signature_vendor":"ExampleCorp Inc.","action_module_system_properties":"64-bit, Windows 10, Build 19045","action_network_connection_id":"12345","action_network_creation_time":"2023-10-01T12:00:00Z","action_network_is_server":true,"action_network_protocol":"HTTP","action_network_stats_is_last":true,"action_powered_off":"off","action_process_fds":"File descriptors info","action_process_image_command_line":"C:\\Program Files\\App\\app.exe -arg1","action_process_image_extension":".exe","action_process_image_md5":"d41d8cd98f00b204e9800998ecf8427e","action_process_image_name":"app.exe","action_process_image_path":"C:\\Program Files\\App\\app.exe","action_process_image_sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","action_process_in_txn":false,"action_process_instance_execution_time":"2025-04-16T12:34:56Z","action_process_instance_id":"instance_001","action_process_integrity_level":"High","action_process_last_writer_actor":"Writer Actor","action_process_os_pid":"7846","action_process_pe_load_info":"Paint","action_process_peb":"PEB info","action_process_peb32":"PEB32 info","action_process_privileges":"Privileges info","action_process_scheduled_task_name":"Scheduled Task","action_process_signature_is_embedded":true,"action_process_signature_product":"App Product","action_process_signature_status":["Verified"],"action_process_signature_vendor":"App Vendor","action_process_termination_code":0,"action_process_termination_date":"2023-10-01T12:30:00Z","action_process_token":"Token info","action_process_user_sid":"S-1-5-21-3623811015-3361044348-30300820-1013","action_process_username":"user","action_remote_ip":["1.128.0.9"],"action_remote_port":["80"],"action_remote_process_image_command_line":"\"C:\\Windows\\System32\\svchost.exe\" -k netsvcs","action_remote_process_image_md5":"44d88612fea8a8f36de82e1278abb02f","action_remote_process_image_name":"svchost.exe","action_remote_process_image_path":"C:\\Windows\\System32\\svchost.exe","action_remote_process_image_sha256":"3395856ce81f2b7382dee72602f798b642f14140b2fc0f8c7d65e9b32e0d0b09","action_remote_process_instance_id":"remote-host-01::1234::5678","action_remote_process_integrity_level":"High","action_remote_process_os_pid":"1234","action_remote_process_signature_is_embedded":"true","action_remote_process_signature_product":"Microsoft Windows Operating System","action_remote_process_signature_status":"Valid","action_remote_process_signature_vendor":"Microsoft Corporation","action_remote_process_thread_id":"5678","action_remote_process_thread_start_address":"0x7ffdfb2a1000","action_remote_process_user_sid":"S-1-5-18","action_remote_process_username":"NT AUTHORITY\\SYSTEM","action_socket_type":"TCP","action_total_download":2048,"action_total_upload":1024,"action_user_is_local_session":true,"action_user_session_id":"1C3D4E5F","action_user_status":"Active","action_user_status_sid":"S-1-5-21-3623811015-3361044348-30300820-1013","action_username":"CORP\\jdoe","address_mapping":"192.168.1.10 -> 10.0.0.5","agent_content_version":"v5.12.3-1","agent_hostname":"endpoint-22.corp.local","agent_id":"a8f5f167f44f4964e6c998dee827110c","agent_interface_map":"eth0-192.168.1.10","agent_ip_addresses":["67.43.156.0","67.43.156.2"],"agent_ip_addresses_v6":["fe80::1c3d:2eff:fe4a:6b7a"],"agent_os_sub_type":"Enterprise","agent_os_type":"Windows","agent_version":"3.8.7.0","event_id":"evt-987654321","event_rpc_func_opnum":"45","event_rpc_interface_uuid":"12345678-1234-abcd-ef00-0123456789ab","event_sub_type":"remote-access","event_timestamp":["2025-04-16T09:15:30Z"],"event_type":["process-start"],"event_version":"1.0","host_metadata_domain":"corp.local","host_metadata_hostname":"endpoint-22","host_metadata_interface_map":"Ethernet0-192.168.1.10","os_actor_local_ip":"216.160.83.56","os_actor_local_port":49152,"os_actor_primary_user_sid":"S-1-5-21-3623811015-3361044348-30300820-1013","os_actor_primary_username":"CORP\\jdoe","os_actor_process_command_line":"\"C:\\Program Files\\ExampleApp\\example.exe\" -task run","os_actor_process_image_md5":"5d41402abc4b2a76b9719d911017c592","os_actor_process_image_name":"example.exe","os_actor_process_image_path":"C:\\Program Files\\ExampleApp\\example.exe","os_actor_process_image_sha256":"2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824","os_actor_process_instance_id":"endpoint-22::5678::9012","os_actor_process_logon_id":"0x3e7","os_actor_process_os_pid":"5678","os_actor_process_signature_status":["Valid"],"os_actor_remote_ip":"81.2.69.144","os_actor_thread_thread_id":"9012"} | |||
There was a problem hiding this comment.
Can we add one more event just to make sure multiple events are parsed.
| location = "US" | ||
| } | ||
|
|
||
| resource "google_storage_bucket_object" "object" { |
There was a problem hiding this comment.
Can you add the link to the oblt-infra?
| resource "google_storage_bucket_object" "object" { | |
| # See https://github.com/elastic/oblt-infra/blob/main/conf/resources/repos/integrations/01-gcp-buildkite-oidc.tf | |
| resource "google_storage_bucket_object" "object" { |
| variable "TEST_RUN_ID" { | ||
| default = "detached" | ||
| } | ||
|
|
There was a problem hiding this comment.
Missing variables here to add the required information for labels.
At least these ones:
https://github.com/elastic/elastic-package/blob/364fd63f62e486359d534cebaeecdea8c1fbd43c/test/packages/parallel/gcp/data_stream/compute/_dev/deploy/tf/vars.tf#L1-L26
| @@ -0,0 +1,18 @@ | |||
| provider "google" { | |||
| credentials = var.GOOGLE_CREDENTIALS | |||
There was a problem hiding this comment.
In order to create the required labels in all resources, google provider allows to set some default labels (documentation):
google_default_labels = {
environment = var.ENVIRONMENT
repo = var.REPO
branch = var.BRANCH
build = var.BUILD_ID
created_date = var.CREATED_DATE
}
provider "google" {
credentials = var.GOOGLE_CREDENTIALS
default_labels = local.google_default_labels
}
An example from AWS, that requires to set some labels/tags:
This example also adds the required labels if a VM is created.
| terraform: | ||
| environment: | ||
| - GOOGLE_PROJECT=${GOOGLE_PROJECT} | ||
| - GOOGLE_REGION=${GOOGLE_REGION:-US} |
There was a problem hiding this comment.
If this is going to be used in the terraform code , this should be prefixed by TF_VAR. Some examples can be found here:
There was a problem hiding this comment.
No we are not using GOOGLE_PROJECT in terraform variable instead one can set GOOGLE_PROJECT in environment variable
|
|
||
| resource "google_storage_bucket" "bucket" { | ||
| name = "${var.BUCKET_NAME}-${var.TEST_RUN_ID}" | ||
| location = "US" |
There was a problem hiding this comment.
Should it be provided via a Terraform variable this value (e.g. var.region)?
At least, it looks like it is intended to use a variable (e.g. GOOGLE_REGION) for that in the env.yml file.
What region should we use here @v1v ?
There was a problem hiding this comment.
OIDC is agnostic to the region AFAIK, so you can use the one you prefer.
There was a problem hiding this comment.
The current GCP project used by gcp plugin is US region based. But it is nice to have a variable var.region in case project changes in future.
| } | ||
|
|
||
| resource "google_storage_bucket" "bucket" { | ||
| name = "${var.BUCKET_NAME}-${var.TEST_RUN_ID}" |
There was a problem hiding this comment.
If possible, it would be good to have all the resources prefixed by the same string.
For instance, it is tried to use elastic-package-* in aws resources:
integrations/packages/aws/data_stream/redshift/_dev/deploy/tf/main.tf
Lines 17 to 18 in 92921e6
There was a problem hiding this comment.
It would be nice to have some common prefix, or tag, yes, but not a strong opinion.
There was a problem hiding this comment.
Maybe it is enough with what you mention about default labels in https://github.com/elastic/integrations/pull/13680/files#r2095815079.
| data_stream: | ||
| vars: | ||
| service_account_key: "{{GOOGLE_APPLICATION_CREDENTIALS}}" | ||
| project_id: "{{GOOGLE_PROJECT}}" |
There was a problem hiding this comment.
Probably, this would be GCP_PROJECT or GOOGLE_CLOUD_PROJECT. Which one should it be set here @v1v ?
There was a problem hiding this comment.
| wait_for_data_timeout: 1m | ||
| data_stream: | ||
| vars: | ||
| service_account_key: "{{GOOGLE_APPLICATION_CREDENTIALS}}" |
There was a problem hiding this comment.
IIRC this environment variable is added via the OIDC plugin.
To be allowed to use this variable in the test case configuration, I think this should be added as part of the environment variables defined in packages/panw_cortex_xdr/data_stream/event/_dev/deploy/tf/env.yml too. Is that right @jsoriano ?
version: '2.3'
services:
terraform:
environment:
- GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}
- GOOGLE_PROJECT=${GOOGLE_PROJECT}
- GOOGLE_REGION=${GOOGLE_REGION:-US}Should that environment variable be used to set GOOGLE_CREDENTIALS from https://github.com/elastic/integrations/pull/13680/files#diff-086fde7ab90e0f846da9a7162a1dbada5d8a0ea0281835d06d529a22105a00d4R26 ?
version: '2.3'
services:
terraform:
environment:
- GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}
- TF_VAR_GOOGLE_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}
- GOOGLE_PROJECT=${GOOGLE_PROJECT}
- GOOGLE_REGION=${GOOGLE_REGION:-US}| // If testing using the elastic-siem account then update the default value for below | ||
| // mentioned variable GOOGLE_CREDENTIALS and service_account_key in test-event-config.yml | ||
| // with your actual credentials | ||
| variable "GOOGLE_CREDENTIALS" { |
There was a problem hiding this comment.
This variable is not set in the packages/panw_cortex_xdr/data_stream/event/_dev/deploy/tf/env.yml file
💔 Build Failed
Failed CI StepsHistory
|
|
The system testing on CI shall be blocked until elastic/elastic-package#2606 is fixed. |
@janvi-elastic – Let's proceed with updating the PR by removing the System Test, as discussed with @bhapas. |
@piyush-elastic There was lot more discussed. The discussion/suggestion was to first get the system tests of this package tested manually by the reviewers with steps provided by the PR author. If the reviewers are satisfied with the Package then the system test config may be temporarily removed to unblock the PR and add it when the elastic-package fix is in place. |
|
Unless there is an expectation of an extended wait for the fix to elastic-package and there is an urgent need to get this merged, I would be very reluctant to relax testing. |
|
💚 Build Succeeded
History
|
|
Package panw_cortex_xdr - 2.2.0 containing this change is available at https://epr.elastic.co/package/panw_cortex_xdr/2.2.0/ |
Proposed commit message
This release includes an event data stream for supporting event
forwarding via GCS and associated dashboards and visualizations.
Additionally, dashboards for alert and incident data streams have been
added.
Palo Alto Cortex XDR fields are mapped to their corresponding ECS fields
where possible.
Test samples were derived from documentation.
System testing is not included, but we have manually created a GCS
bucket, uploaded a GZIP-compressed log file as outlined in the Cortex
XDR documentation, and successfully collected the logs.
Checklist
How to test this PR locally
Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/panw_cortex_xdr directory.
Run the following command to run tests.
Related issues
Screenshot