10000 Unyek, to re-write the files from a yek file · Issue #64 · bodo-run/yek · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Unyek, to re-write the files from a yek file #64

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AntouanK opened this issue Jan 28, 2025 · 3 comments
Open

Unyek, to re-write the files from a yek file #64

AntouanK opened this issue Jan 28, 2025 · 3 comments

Comments

@AntouanK
Copy link

Not really an issue, but posting in case someone else needs it.

I wanted an ... unyek, to un-serialise the files, so I made one

https://github.com/AntouanK/unyek

@mohsen1
Copy link
Collaborator
mohsen1 commented Jan 29, 2025

https://github.com/codeinred/untree :)

Quick suggestion here write in parallel (I think Deno knows not to overwhelm the OS with too many writes in parallel but worth testing with a huge number to make sure)

@AntouanK
Copy link
Author

There's a pattern :D

Since it awaits for every write, it will happen sequentially, right?

@mohsen1
Copy link
Collaborator
mohsen1 commented Jan 30, 2025

Something like this:

async function unyekFile(yekPath: string): Promise<void> {
  try {
    const content = await Deno.readTextFile(yekPath);
    const files = parseYekContent(content);
    const groupedFiles = groupFilesByBasePath(files);

    const results = await Promise.allSettled(
      groupedFiles
      .map(async ([basePath, fileGroup]) =>
        combineFileParts(fileGroup.parts)
      )
      .map((combinedContent) =>
        writeFileContent(basePath, combinedContent)
      )
    );


    if (results.every((result) => result.status === "fulfilled")) {
      console.log("All files written successfully");
    } else {
      console.error("Failed to write some files");
    }
  } catch (error) {
    console.error(`Error processing yek file: ${error.message}`);
    Deno.exit(1);
  }
}

Note that from v0.14.0 yek is only writing one file so you might not need that. I realized chunking is not really useful for LLMs since even if you feed an LLM with 8K token context 2 8K token messages, the result is no different than just not providing the first (less important) chunk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants
0