Guard could automatically watch only directories matching `watch` regexps #674

e2 · 2014-11-20T18:35:34Z

While this might be tricky, it may be VERY useful for users to just "watch" the project directory, while only directories matching watch statements (watchers) are watched.

The text was updated successfully, but these errors were encountered:

skizzybiz · 2014-11-20T19:06:58Z

👍

With the current implementation of listen (and its inability to deal with symlinks), this seems like the only sensible way to go.

e2 · 2014-11-20T20:20:09Z

@skizzybiz - it's unfair to say listen has an inability to deal with symlinks, because it simply follows them like every other app. As for filesystem loops, even standard tools like find report them as SERIOUS errors.

So it's VERY reasonable for Listen to behave "no more intelligently" than every other tool following symlinks. The only way Listen could really "deal" with symlinks accurately, is if it implemented it's own filesystem and locked it from being changed.

If a specific scenario doesn't work for you - just provide a specific example, so we can make sure it's correctly handled in the next version of listen/guard.

Take even this example: app/foo symlinks to vendor/foo

What if you have in your Guardfile:
watch('app/foo/bar') and watch('vendor/foo/baz'), then is Guard supposed to listen to app/foo or vendor/foo or both? And what if you have multiple watches on symlinks to files in 'vendor/foo'?

With listen 2.8.0 this would result in an error, because the physical directory would be otherwise watched twice!

What I'm saying is that without specific examples turned into test cases, it's too easy to release a "broken" version which didn't take some (even typical) scenarios.

If you show an attitude of being frustrated (as opposed to be willing to test unreleased branches and provide examples), there is less of an incentive to push features out faster.

Not judging - just showing the perspective of the team who's been working very hard to push new features out quickly without breaking setups for others.

skizzybiz · 2014-11-21T03:13:05Z

Look, there's no need to get defensive, I'm just trying to be helpful. Either guard or listen made a change recently, I'm not sure which -- previously, it worked, and in the latest version it doesn't.

My specific scenario: I have a tmp/ directory in my project, and I have a rake task that, among other things, adds some symlinks inside tmp/ to other folders in the projects. My Guardfile watches some directories that are not in tmp/, but because listen is now scanning my whole project for some reason, it bails out when it resolves the symlinks.

I thought that this feature was about using the 'watch' directives as roots for listen, rather than having it scan the whole project. I suspect there will be other developers that have internal symlinks in their projects, so either this feature, or the linked one in the listen project, will be necessary for them to be able to use the latest version of guard.

skizzybiz · 2014-11-21T03:13:52Z

Let me know if that's not specific enough -- I'm happy to build a sample project that illustrates the problem.

skizzybiz · 2014-11-21T03:37:25Z

Hmm, I can't figure out how to add an attachment, so I'll try describing my setup. Here are the files in the sample project:

$ find .
.
./bin
./Guardfile
./lib
./tmp
./tmp/bin

The tmp/bin file is a symlink to the root-level bin:

$ ls -l tmp
total 8
lrwxr-xr-x  1 joelyou  staff  6 Nov 20 19:28 bin -> ../bin

Notice that find has no problem with it, because it isn't a cycle.

The Guardfile contains the following (which won't do anything useful, since there aren't any ruby files or a spec directory, but I wanted to illustrate how non-cyclical symlinks in the project that are completely unrelated to what I'm trying to watch can still cause problems):

guard :rspec do
  watch(%r|^lib/(.+)\.rb$|)     { |m| "spec/#{m[1]}_spec.rb" }
end

And when I run guard (after installing the latest versions of guard, rspec and guard-rspec), I get an exception:

$ guard
19:35:34 - INFO - Guard is using TerminalTitle to send notifications.
19:35:34 - INFO - Guard::RSpec is running
        ** ERROR: Listen detected a duplicate directory being watched! **

        (This may be due to symlinks pointing to parent directories).

        Duplicate: /Users/joelyou/code/github/guard-674/tmp/bin

        which already is added as: /Users/joelyou/code/github/guard-674/bin

        Listen is refusing to continue, because this may likely result in
        an infinite loop.

I hope that explains my previous comments -- it really does seem like listen isn't doing a very good job of dealing with symlinks.

I appreciate the hard work you're all doing, and I apologize if my tone came off as confrontational.

e2 · 2014-11-21T05:42:37Z

Look, there's no need to get defensive, I'm just trying to be helpful.

I know, I'm defending the honor of everyone who contributed to make Listen as useful as it is today - despite many technical issues way beyond their control which Listen simply can't deal with. I really appreciate all the work they did, and I'm not going to let anyone even suggest anything bad about them.

It just doesn't seem fair to overgeneralize without at least a specific example to dissect (you never know who could be "wrong", so why start with assumptions?).

For example, no one will ever be capable of appreciating the work involved in Listen to support OSX. Neither will anyone appreciate how effective and robust Polling is. And when Dropbox causes issues, the conclusion is "Listen stopped working on OSX".

Hey, I'm guilty of same thing - I get frustrated when something doesn't work, but when I start digging, my initial reaction quickly stops making any sense. And it often turns out I'm the one who screwed up on my side.

Either guard or listen made a change recently, I'm not sure which -- previously, it worked, and in the latest version it doesn't.

If there's a bug like that in the latest release, and it's not a bug in the issues, then I don't know about it (like I broke OSX support on multiple directories for a month until people mentioned it was still broken, albeit in a different way).

My Guardfile watches some directories that are not in tmp/, but because listen is now scanning my whole project for some reason, it bails out when it resolves the symlinks.

The following should help: watch everything except the tmp directory, e.g.

bundle exec guard -w app config lib features spec public # .etc.

If you do that, and your project is still scanning the whole directory, I need output using LISTEN_GEM_DEBUGGING=2 (see README/troubleshooting). (Best to put it in a Gist).

I thought that this feature was about using the 'watch' directives as roots for listen

Yes, this feature would do the same as you can do manually with the -w option right now.

I suspect there will be other developers that have internal symlinks in their projects, so either this feature, or the linked one in the listen project, will be necessary for them to be able to use the latest version of guard.

The solution is the -w option in Guard, or for people using Listen directly - the directories passed to Listen.

Yes, I know watching a whole directory is "convenient" and "expected to work", but this forces Listen to be really intelligent. And that's going to take a lot of work.

I hope that explains my previous comments -- it really does seem like listen isn't doing a very good job of dealing with symlinks.

First, try out the -w option to Guard, because it should work as you expect. Second, the message in Listen may need tweaking to include info about the -w option in Guard. Let me know what you think is best.

I appreciate the hard work you're all doing, and I apologize if my tone came off as confrontational.

No, I should apologize - I should've worded things both more kindly and more clearly.

One note, though: an infinite loop can occur in the backend, anyway (e.g. on Linux) - that's because no only Listen uses recursion without checking symlinks, but the same goes for the backends (rb-inotify in this case). And I'd argue that it isn't rb-inotify's responsibility to deal with symlinks, so that's why currently Listen halts.

So let me know how things go - I really appreciate the feedback by the way, because I doubt you're the only person on the planet with this problem.

skizzybiz · 2014-11-21T16:57:02Z

Thanks for the suggestion about using -w, and that's a useful workaround, but I thought the whole point of having a Guardfile was to provide configuration directives so that you can simply run guard (or bundle exec guard) from the command line, without having to remember which directories you need to watch every time you execute it. In my daily work, I have to switch between a lot of different projects, so I hope you can understand that while the incremental burden is small, it does add up. I can add the command line invocation to my README, but that's not great either. I can add a rake guard task -- is there rake support for guard, or is shell invocation the best option there?

I think it would be helpful to add a feature that allows you to specify watched paths directly in Guardfile -- that would neatly route around the whole watching the entire project problem without requiring (hard to remember and potentially very long) command line flags. Should I file another request for that?

e2 · 2014-11-21T18:05:19Z

Thanks for the suggestion about using -w, and that's a useful workaround

It's actually the recommended method, it just hasn't been made automatic/convenient yet - no one had the insights they have now on how to combine "what the user expects" with "what is possible technically" with "we are looking for maintainers, because as authors we don't have time to work on this anymore" (I'm kind of the exception, and I'm pushing my time to at least deal with bugs and issues).

the whole point of having a Guardfile was to provide configuration directives so that you can simply run guard (or bundle exec guard) from the command line, without having to remember which directories you need to watch every time you execute it.

I took over the project quite recently, and neither anyone really complained about this before (except for performance issues due to basically watching entire projects even then they are editing a single file in app/ or whatever directory). No one complains (constructively), no pull requests - and the result means not implementation.

I'm sure you know no one gets paid for this - I just want to say that at this point you're just as much a contributor as anyone else. Either you ge 8000 t your hands dirty fixing this, or someone else has to. I'm sure you have important projects and stuff, and so have the other contributors working on this - most of the bugs I fix don't even affect me personally (e.g. Windows problems, OSX problems, "slowness", editor problems, etc.).

In my daily work, I have to switch between a lot of different projects, so I hope you can understand that while the incremental burden is small, it does add up.

At some point you may find that the burden becomes so big, you'll realize that contributing towards a solution (e.g. through poking around, asking for questions when you're stuck, taking stabs at other errors, helping people with issue they report, watching the project, etc.) will pay off.

But it probably won't be too soon until what you "loose" adds up to 5-20 hours needed to actually fix something properly, test, respond to feedback, requests, issues, submit pull requests to other repositories depending on backward compatibility, etc.

Yes, the world is a better place if I spend 50 hours improving things, and 600 people save 5 minutes. That's why "constructive complaining" is so important, because then I know if I'm fixing an issue allow 5 people to save 5 minutes, or 5000 people to save 5 minutes.

Not to mention how many issues can be solved/resolved just by spending 5 minutes reading through the docs, the issues, the wiki, the code or even asking a question.

Is 5 minutes of your time really more important than 50 hours of mine? I'm sure you don't think that way - just ask yourself if you're somehow not subconsciously implying that.

I can add the command line invocation to my README, but that's not great either.

Ideally, you could submit a pull request to enable users to define directories in the DSL - just check out the :ignore or :notification methods work. Currently I won't do this, because I'm refactoring and cleaning up the code base for such contributions to be easier and faster for everyone (and I'm currently the only guy both capable and willing to do what it takes to clean up the codebase).

I can add a rake guard task -- is there rake support for guard, or is shell invocation the best option there?

Rake support probably doesn't make sense since guard is interactive, and not a "task". Probably just like it doesn't make sense to run a rails app from a Rakefile.

I think it would be helpful to add a feature that allows you to specify watched paths directly in Guardfile

Yes, this may help - like I said, a pull request for this may be nice (although ideally, detecting this would be automatic, just as you'd prefer). I'm currently working on reorganizing things to deal with the scope bugs (which are constantly reappearing, annoying people who rely on scopes) so I have my hands full for now.

Should I file another request for that?

Sure, that's a good idea.

rmm5t · 2014-11-25T00:12:14Z

~~Honestly, I find this to be the easiest workaround so far. The -w command line option is cumbersome.~~

  # Until listen has better support for recursive symlinks or until Guardfiles
  # can specify a list of watchdirs.
  # See https://github.com/guard/guard/issues/674
  # See https://github.com/guard/listen/pull/273
  # See https://github.com/guard/listen/pull/274
  gem "listen", "~> 2.7.12"

EDIT: Use the new directories declaration that @e2 added to Guardfiles.

e2 · 2014-11-26T02:09:53Z

Since this is very related, the watched directories can now be set in the Guardfile: (https://github.com/guard/guard/releases).

If someone is clever enough, they can access a list of watchers from within the Guardfile using Guard's API, get a list of the top directories for the project and match with the watchers.

This is a bit tricky, because matcher expressions are for files, so you'd have to find all the files in the project, select those matching the regexps, get the dirnames of those paths, uniq(), and call directories() with the result.

e2 mentioned this issue Nov 20, 2014

Feature request: Listen could *really ignore* ignored directories (optimization) guard/listen#274

Closed

e2 added this to the v2.9.99 milestone Nov 20, 2014

rymai changed the title ~~guard could automatically watch only directories matching watch regexps~~ Guard could automatically watch only directories matching watch regexps Dec 2, 2014

e2 added the feature request label Dec 22, 2014

rymai added ✨ Feature Adds a new feature and removed feature request labels Oct 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Guard could automatically watch only directories matching `watch` regexps #674

Guard could automatically watch only directories matching `watch` regexps #674

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Guard could automatically watch only directories matching watch regexps #674

Guard could automatically watch only directories matching watch regexps #674

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Guard could automatically watch only directories matching `watch` regexps #674

Guard could automatically watch only directories matching `watch` regexps #674