WriteFile occassionaly truncates files entirely #137

klauern · 2022-10-27T01:18:49Z

I am trying to write a simple sed -i script using this, but it seems that when I run it on the same path that I am reading it from, occasionally the file is completely truncated instead of written with the piped changes.

func Replace(path, from, to string) error {
	pipe := script.File(path)
	file, err := pipe.Replace(from, to).WriteFile(path)
	if err != nil {
		fmt.Println(path, err)
		return fmt.Errorf("error writing file %s: %w", path, err)
	}
    return nil
}

I can make this work by pulling the write operation out of the pipe:

func Replace(path, from, to string) error {
	pipe := script.File(path)
	file, err := pipe.Replace(from, to).String()
	if err != nil {
		fmt.Println(path, err)
		return fmt.Errorf("error writing file %s: %w", path, err)
	}
	return os.WriteFile(path, []byte(file), 0644)
}

The text was updated successfully, but these errors were encountered:

klauern · 2022-10-27T01:19:59Z

What's frustrating is that this isn't the case all the time, so it does work for a good percentage, so there is some kind of weird race condition that is causing it not to work. I am not running any of these calls in parallel, so I am pretty sure it's how the file is handled in bitfield/script itself.

bitfield · 2022-10-27T09:16:09Z

I'm not sure it's safe to write to a file at the same time as reading from it—or rather, it's safe, but might give you puzzling results, as you've experienced. The reads and writes will be buffered, which means the bits don't necessarily show up on disk when you expect them to.

klauern · 2022-10-27T14:51:40Z

I was thinking that, too. I thought maybe the Wait() command would let me do that, but since it doesn't return a *Pipe, that seemed like a dead-end.

andrew-werdna · 2022-11-07T10:49:17Z

I've encountered the same issue as well. Filters run concurrently as stated in the readme, and it doesn't look like there is an option to enforce consecutive execution. I was trying to do the exact same thing as you @klauern and I solved it the same way too.

bitfield · 2022-11-07T12:13:19Z

Another option that avoids reading the whole file into memory would be to pipe it to a temporary file, and then once the pipe has completed, rename the temporary file to the original file.

zhqqqy · 2025-03-21T03:05:20Z

Another option that avoids reading the whole file into memory would be to pipe it to a temporary file, and then once the pipe has completed, rename the temporary file to the original file.

yes, i used ioutil.TempFile("", "temp.txt") to create a temporary file. Perhaps this could be encapsulated into an internal method to be used with Replace function ? Like sed -i.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

WriteFile occassionaly truncates files entirely #137

WriteFile occassionaly truncates files entirely #137

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WriteFile occassionaly truncates files entirely #137

WriteFile occassionaly truncates files entirely #137

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!