Tags: StationA/esx
Tags
[Bugfix] Fixing esx scroll query functionality Minor bugfix to more reliably specify a bounding query for the scroll command
Adds support for partial-upsert index semantics Prior to this commit, esx only supported full document upsert by document ID or new document creation. This had a couple of implications: 1. Create-only operations would never benefit from the improved performance of not having to first check for document existence. This also required documents to have `_id` set, instead of using Elasticsearch's auto-generated IDs. 2. Partial document upserts were impossible previously, which may have caused issues when expecting only a partial update, and instead replacing the entire document
[Bugfix] Record timing for requests in error This fixes a bug where an error (including a timeout!) would return 0 for the request duration.
[Bugfix] Make sure to throttle between retries This fixes an issue whereby an index batch would be retried without calling the throttle in between.
[Perf] Adding CLI flags for more throttle controls This change adds two new flags: * `--throttle-window-size` controls the number of runtime samples to use when estimating the expected duration that the next index operation will take * `--throttle-high-water-mark` controls the percentage of the absolute maximum index duration (the ES timeout) that will be considered internally by the throttle so as to more proactively curb latency spikes
[Bugfix] Don't mutate input batches Because batches can optionally be retried, we need to ensure that they're contents cannot be mutated, so as to prevent spooky side-effects at a distance.
[Perf] Adds a sample-based throttler for indexing This change introduces a dynamic throttling system that samples bulk indexing performance timings in order to intelligently throttle the net ingest speed, thus ensuring a more stable indexer and ES cluster health. For additional control, and optional `--num-retries` option is also exposed to automatically retry batches that fail for any reason, including timeouts.
PreviousNext