8000 [data] support streaming writes for `write_lance` · Issue #54069 · ray-project/ray · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
[data] support streaming writes for write_lance #54069
Open
@j93hahn

Description

@j93hahn

Description

Lance is becoming more and more popular (and arguably significantly better than parquet). It would be awesome if write_lance supports streaming writes like write_parquet does, that way we can process large datasets spanning many terabytes without having to memory about materialization / memory constraints

Use case

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    dataRay Data-related issuesenhancementRequest for new feature and/or capabilityperformancetriageNeeds triage (eg: priority, bug/not-bug, and owning component)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0