8000 feat: Refactor to clean up processing and prepare for v2 by benjackwhite · Pull Request #31160 · PostHog/posthog · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: Refactor to clean up processing and prepare for v2 #31160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Apr 15, 2025
Merged

feat: Refactor to clean up processing and prepare for v2 #31160

merged 10 commits into from
Apr 15, 2025

Conversation

benjackwhite
Copy link
Contributor
@benjackwhite benjackwhite commented Apr 12, 2025 •
edited
Loading

Problem

As part of the work here I did some general tidying which is pulled out for this isolated PR

Changes

  • Remove the old batch ingester that is unused
  • Removes some unnecessarily complex code
  • Refactors the methods so that we are ready for the follow up PR
  • Modifies the overflower to move the whole batch. This makes way more sense as we don't need the limiter to be hyper accurate

Does this work well for both Cloud and self-hosted?

How did you test this code?

@benjackwhite benjackwhite requested a review from a team April 12, 2025 17:48
@benjackwhite benjackwhite marked this pull request as ready for review April 12, 2025 17:48
greptile-apps[bot]
greptile-apps bot reviewed Apr 12, 2025
View reviewed changes
Copy link
Contributor
@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Summary

This PR refactors the ingestion system by removing the old batch ingester and simplifying the codebase to prepare for a v2 implementation.

  • Removed plugin-server/src/main/ingestion-queues/batch-processing/each-batch-ingestion.ts file containing unused batch ingestion logic
  • Removed ingestionPartitionKeyOverflowed counter from global metrics and made it local to IngestionConsumer
  • Restructured IngestionConsumer with new methods like redirectEvents, runEventRunnerV1, and getEventPipelineRunnerV1 for better modularity
  • Extracted trackIfNonPersonEventUpdatesPersons as a standalone utility function
  • Updated tests to align with the new structure, removing tests for deprecated functionality

5 file(s) reviewed, 1 comment(s)
Edit PR Review Bot Settings | Greptile

pl
pl previously requested changes Apr 14, 2025
View reviewed changes
@benjackwhite benjackwhite requested review from pl and nickbest-ph April 15, 2025 07:00
@benjackwhite
Merge branch 'master' into feat/tidy-ingester
d1b5db1 
# Conflicts:
#	plugin-server/src/ingestion/ingestion-consumer.ts
@benjackwhite
Fixes
3682a16
@benjackwhite benjackwhite merged commit e1ffe03 into master Apr 15, 2025
94 checks passed
@benjackwhite benjackwhite deleted the feat/tidy-ingester branch April 15, 2025 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments

@meikelmosby meikelmosby meikelmosby approved these changes

@pl pl Awaiting requested review from pl

@nickbest-ph nickbest-ph Awaiting requested review from nickbest-ph

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@benjackwhite @pl @meikelmosby @nickbest-ph
0