8000 add transformation for destination by morhayn · Pull Request #13460 · deckhouse/deckhouse · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

add transformation for destination #13460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open

add transformation for destination #13460

wants to merge 25 commits into from

Conversation

morhayn
Copy link
Contributor
@morhayn morhayn commented May 15, 2025 •
edited
Loading

Description

Added the ability to specify additional log transformations before sending to the destination.

  • NormalizeLabelKeys - Replace dot to underscore in the lable pod_labels
  • EnsureStructuredMessage - If message is string package it to json
  • DropLabels - Delete labels

Why do we need it, and what problem does it solve?

Needed to provide users with processing of log changes for non standard sending methods

  • NormalizeLableKeys - Solved the problem of sending to elasticsearch via a query broker
  • EnsureStructuredMessage - When choosing to write logs in the json format, the logs of the service itself are write as a string. Example nginx or java
  • DropLabels - Removing unnecessary labels to save space on the log aggregator

Why do we need it in the patch release (if we do)?

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: log-shipper
type: feature
summary: Added the additional log transformations
impact: log-shipper module. Destination-log config
impact_level: low

@morhayn
add transformation for destination
9ae2e51 
Signed-off-by: Sergey Kurkin <sergey.kurkin@flant.com>
@morhayn morhayn added area/monitoring Pull requests that update monitoring modules skip/documentation-validation Skip documentation validation in pull request skip/e2e Skip running e2e test for pass pull request check area/logs Pull requests that update logging modules skip/grafana-dashboard labels May 15, 2025
@github-actions github-actions bot added go Pull requests that update Go code area/api-change Pull requests that api changing and removed area/monitoring Pull requests that update monitoring modules labels May 15, 2025
@vladimirGuryanov vladimirGuryanov added this to the v1.70.0 milestone May 16, 2025
morhayn added 3 commits May 16, 2025 17:02
@morhayn
fix linter warning
6b4333e 
Signed-off-by: Sergey Kurkin <sergey.kurkin@flant.com>
@morhayn
fix linter test
2cbbda9 
Signed-off-by: Sergey Kurkin <sergey.kurkin@flant.com>
@morhayn
fix lint gsi
d3b7efc 
Signed-off-by: Sergey Kurkin <sergey.kurkin@flant.com>
@vladimirGuryanov vladimirGuryanov added e2e/run/yandex-cloud Run e2e tests in Yandex Cloud and removed skip/documentation-validation Skip documentation validation in pull request skip/e2e Skip running e2e test for pass pull request check labels May 20, 2025
@deckhouse-BOaTswain
Copy link
Collaborator
deckhouse-BOaTswain commented May 20, 2025 •
edited
Loading

⚪ e2e: Yandex.Cloud for deckhouse:feature/log-shipper-transformation cancelled.

Workflow details

🔴 e2e: Yandex.Cloud for deckhouse:feature/log-shipper-transformation failed in 1h4m27s.

E2e for yandex-cloud WithoutNAT;containerd;1.30 was failed. Use:

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.30 15131170359 15131170359-1-con-1-30 /sys/deckhouse-oss/install:pr13460 - for abort failed cluster

Workflow details (1 job failed)

Yandex.Cloud-WithoutNAT-Containerd-1.30 - Connection string: ssh redos@158.160.123.43

🔴 e2e: Yandex.Cloud, Containerd, Kubernetes 1.30 failed in 1h3m22s.

@github-actions github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label May 20, 2025
@deckhouse-BOaTswain deckhouse-BOaTswain added the e2e/cluster/failed Pull request contains failed e2e cluster label May 20, 2025
@vladimirGuryanov vladimirGuryanov added e2e/run/yandex-cloud Run e2e tests in Yandex Cloud and removed e2e/cluster/failed Pull request contains failed e2e cluster labels May 20, 2025
@deckhouse-BOaTswain
Copy link
Collaborator
deckhouse-BOaTswain commented May 20, 2025 •
edited
Loading

🔴 e2e: Yandex.Cloud for deckhouse:feature/log-shipper-transformation failed in 1h2m27s.

E2e for yandex-cloud WithoutNAT;containerd;1.30 was failed. Use:

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.30 15132806430 15132806430-1-con-1-30 /sys/deckhouse-oss/install:pr13460 - for abort failed cluster

Workflow details (1 job failed)

Yandex.Cloud-WithoutNAT-Containerd-1.30 - Connection string: ssh redos@158.160.41.197

🔴 e2e: Yandex.Cloud, Containerd, Kubernetes 1.30 failed in 1h1m21s.

@github-actions github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label May 20, 2025
@vladimirGuryanov
Copy link
Contributor

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.30 15131170359 15131170359-1-con-1-30 /sys/deckhouse-oss/install:pr13460

@deckhouse-BOaTswain
Copy link
Collaborator
deckhouse-BOaTswain commented May 20, 2025 •
edited
Loading

🟢 destroy cluster: Yandex.Cloud for refs/heads/main succeeded in 10m30s.

Workflow details

🟢 destroy cluster: Yandex.Cloud, Containerd, Kubernetes 1.30 succeeded in 10m8s.

@vladimirGuryanov vladimirGuryanov added the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label May 20, 2025
@deckhouse-BOaTswain
Copy link
Collaborator
deckhouse-BOaTswain commented May 20, 2025 •
edited
Loading

🔴 e2e: Yandex.Cloud for deckhouse:feature/log-shipper-transformation failed in 20m36s.

E2e for yandex-cloud WithoutNAT;containerd;1.30 was failed. Use:

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.30 15132840930 15132840930-1-con-1-30 /sys/deckhouse-oss/install:pr13460 - for abort failed cluster

Workflow details (1 job failed)

Yandex.Cloud-WithoutNAT-Containerd-1.30 - Connection string: ssh redos@158.160.110.205

🔴 e2e: Yandex.Cloud, Containerd, Kubernetes 1.30 failed in 19m24s.

@github-actions github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label May 20, 2025
@morhayn morhayn marked this pull request as ready for review May 20, 2025 08:42
@morhayn
Copy link
Contributor Author
morhayn commented May 22, 2025

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.30 15180614163 15180614163-1-con-1-30 /sys/deckhouse-oss/install:pr13460

@deckhouse-BOaTswain
Copy link
Collaborator
deckhouse-BOaTswain commented May 22, 2025 •
edited
Loading

🟢 destroy cluster: Yandex.Cloud for refs/heads/main succeeded in 6m18s.

Workflow details

🟢 destroy cluster: Yandex.Cloud, Containerd, Kubernetes 1.30 succeeded in 5m55s.

@morhayn
remove test prints
a99f9ea 
Signed-off-by: Sergey Kurkin <sergey.kurkin@flant.com>
@morhayn morhayn added the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label May 22, 2025
@deckhouse-BOaTswain
Copy link
Collaborator
deckhouse-BOaTswain commented May 22, 2025 •
edited
Loading

⚪ e2e: Yandex.Cloud for deckhouse:feature/log-shipper-transformation cancelled.

E2e for yandex-cloud WithoutNAT;containerd;1.30 was failed. Use:

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.30 15183012301 15183012301-1-con-1-30 /sys/deckhouse-oss/install:pr13460 - for abort failed cluster

Workflow details

Yandex.Cloud-WithoutNAT-Containerd-1.30 - Connection string: ssh redos@158.160.108.63

⚪ e2e: Yandex.Cloud, Containerd, Kubernetes 1.30 cancelled.

@github-actions github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label May 22, 2025
@deckhouse-BOaTswain deckhouse-BOaTswain added the e2e/cluster/failed Pull request contains failed e2e cluster label May 22, 2025
@morhayn morhayn added e2e/run/yandex-cloud Run e2e tests in Yandex Cloud and removed e2e/cluster/failed Pull request contains failed e2e cluster labels May 22, 2025
@deckhouse-BOaTswain
Copy link
Collaborator
deckhouse-BOaTswain commented May 22, 2025 •
edited
Loading

🟢 e2e: Yandex.Cloud for deckhouse:feature/log-shipper-transformation succeeded in 35m37s.

Workflow details

Yandex.Cloud-WithoutNAT-Containerd-1.30 - Connection string: ssh redos@51.250.15.111

🟢 e2e: Yandex.Cloud, Containerd, Kubernetes 1.30 succeeded in 34m28s.

@morhayn
Copy link
Contributor Author
morhayn commented May 22, 2025

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.30 15183012301 15183012301-1-con-1-30 /sys/deckhouse-oss/install:pr13460

@github-actions github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label May 22, 2025
@deckhouse-BOaTswain
Copy link
Collaborator
deckhouse-BOaTswain commented May 22, 2025 •
edited
Loading

🟢 destroy cluster: Yandex.Cloud for refs/heads/main succeeded in 10m37s.

Workflow details

🟢 destroy cluster: Yandex.Cloud, Containerd, Kubernetes 1.30 succeeded in 10m18s.

@deckhouse-BOaTswain deckhouse-BOaTswain added e2e/cluster/failed Pull request contains failed e2e cluster and removed e2e/cluster/failed Pull request contains failed e2e cluster labels May 22, 2025
@morhayn
Copy link
Contributor Author
morhayn commented May 22, 2025

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.30 15182322359 15182322359-1-con-1-30 /sys/deckhouse-oss/install:pr13460

@deckhouse-BOaTswain
Copy link
Collaborator
deckhouse-BOaTswain commented May 22, 2025 •
edited
Loading

🟢 destroy cluster: Yandex.Cloud for refs/heads/main succeeded in 5m35s.

Workflow details

🟢 destroy cluster: Yandex.Cloud, Containerd, Kubernetes 1.30 succeeded in 5m16s.

@morhayn
add minItems to crd for DropLabels
5f0b1d9 
Signed-off-by: Sergey Kurkin <sergey.kurkin@flant.com>
@morhayn morhayn added the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label May 22, 2025
@deckhouse-BOaTswain
Copy link
Collaborator
deckhouse-BOaTswain commented May 22, 2025 •
edited
Loading

🟢 e2e: Yandex.Cloud for deckhouse:feature/log-shipper-transformation succeeded in 56m4s.

Workflow details

Yandex.Cloud-WithoutNAT-Containerd-1.30 - Connection string: ssh redos@89.169.129.147

🟢 e2e: Yandex.Cloud, Containerd, Kubernetes 1.30 succeeded in 34m44s.

@github-actions github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label May 22, 2025
@Steamvis Steamvis modified the milestones: v1.70.0, v1.71.0 May 22, 2025
morhayn added 3 commits May 24, 2025 14:50
@morhayn
fix transformations name in apis
e3ff194 
Signed-off-by: Sergey Kurkin <sergey.kurkin@flant.com>
@morhayn
testing
2eb6e9c 
Signed-off-by: Sergey Kurkin <sergey.kurkin@flant.com>
@morhayn
undo testing date
91a9168 
Signed-off-by: Sergey Kurkin <sergey.kurkin@flant.com>
@MargaretKozlova MargaretKozlova requested a review from maxinsky May 30, 2025 09:34
@maxinsky
Edit parameter descriptions
6f2419b 
Signed-off-by: Max Chervov <max.chervov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@z9r5 z9r5 z9r5 approved these changes

@maxinsky maxinsky maxinsky approved these changes

@nabokihms nabokihms Awaiting requested review from nabokihms nabokihms is a code owner

@vladimirGuryanov vladimirGuryanov Awaiting requested review from vladimirGuryanov vladimirGuryanov is a code owner

@lazovskiy lazovskiy Awaiting requested review from lazovskiy lazovskiy is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@morhayn morhayn

Labels
area/api-change Pull requests that api changing area/logs Pull requests that update logging modules go Pull requests that update Go code skip/documentation-validation Skip documentation validation in pull request
Projects
None yet
Milestone
v1.71.0
Development

Successfully merging this pull request may close these issues.
6 participants
@morhayn @deckhouse-BOaTswain @vladimirGuryanov @z9r5 @maxinsky @Steamvis
0