Junction content still gets deleted after composer update #5053

ghost · 2016-03-13T01:06:08Z

This is similar to #4955, but happens only on consecutive updates, if the dependency composer.json has been updated. All the junction content is gone after this:

Updating dependencies (including require-dev)
  - Removing test/a (dev-master)

  [RuntimeException]
  Package test/a cannot install to "... src\Test\A" inside its source at "... src\Test\A"

Exception trace:
 () at phar://composer.phar/src/Composer/Downloader/PathDownloader.php:45
 Composer\Downloader\PathDownloader->download() at phar://composer.phar/src/Composer/Downloader/DownloadManager.php:213
 Composer\Downloader\DownloadManager->download() at phar://composer.phar/src/Composer/Downloader/DownloadManager.php:260
 Composer\Downloader\DownloadManager->update() at phar://composer.phar/src/Composer/Installer/LibraryInstaller.php:198
 Composer\Installer\LibraryInstaller->updateCode() at phar://composer.phar/src/Composer/Installer/LibraryInstaller.php:110
 Composer\Installer\LibraryInstaller->update() at phar://composer.phar/src/Composer/Installer/InstallationManager.php:172
 Composer\Installer\InstallationManager->update() at phar://composer.phar/src/Composer/Installer/InstallationManager.php:139
 Composer\Installer\InstallationManager->execute() at phar://composer.phar/src/Composer/Installer.php:592
 Composer\Installer->doInstall() at phar://composer.phar/src/Composer/Installer.php:219
 Composer\Installer->run() at phar://composer.phar/src/Composer/Command/UpdateCommand.php:173
 Composer\Command\UpdateCommand->execute() at phar://composer.phar/vendor/symfony/console/Command/Command.php:259
 Symfony\Component\Console\Command\Command->run() at phar://composer.phar/vendor/symfony/console/Application.php:844
 Symfony\Component\Console\Application->doRunCommand() at phar://composer.phar/vendor/symfony/console/Application.php:192
 Symfony\Component\Console\Application->doRun() at phar://composer.phar/src/Composer/Console/Application.php:166
 Composer\Console\Application->doRun() at phar://composer.phar/vendor/symfony/console/Application.php:123
 Symfony\Component\Console\Application->run() at phar://composer.phar/src/Composer/Console/Application.php:99
 Composer\Console\Application->run() at phar://composer.phar/bin/composer:43
 require() at composer.phar:24

Junction lstat is as follows:

Array
(
    [0] => 4
    [1] => 0
    [2] => 0
    [3] => 1
    [4] => 0
    [5] => 0
    [6] => 4
    [7] => 0
    [8] => 1457828772
    [9] => 1457828772
    [10] => 1457828772
    [11] => -1
    [12] => -1
    [dev] => 4
    [ino] => 0
    [mode] => 0
    [nlink] => 1
    [uid] => 0
    [gid] => 0
    [rdev] => 4
    [size] => 0
    [atime] => 1457828772
    [mtime] => 1457828772
    [ctime] => 1457828772
    [blksize] => -1
    [blocks] => -1
)

And this is the premise: (On Windows 10)

# composer.json
{
    "repositories": [
        {"type": "path", "url": "src/Test/*"}
    ],

    "require": {
        "test/a": "@dev"
    }
}

# src/Test/A/composer.json
{
    "name": "test/a",
    "description": "test"
}

Run composer update
Modify description in src/Test/A/composer.json
Run composer update
src/Test/A is empty, while the junction vendor/test/a remains

The text was updated successfully, but these errors were encountered:

ghost · 2016-03-13T14:36:34Z

I don't know whats going on.. the problem does not appear when i try it with xdebug..

curry684 · 2016-03-13T19:06:16Z

That's really weird, that should have no influence at all. Is it completely reproducable with xdebug disabled?

ghost · 2016-03-14T00:07:01Z

Yes.
Running it normally (composer update) throws the error mentioned above.
Running it with xdebug (php -dzend_extension=php_xdebug.dll path/to/composer.phar update) works just fine..

I'm running this with PHP 7.0.4 VC14 x64 TS and Xdebug 2.4.0 for the corresponding php version.

alcohol · 2016-03-14T09:10:32Z

Is that the exact command you use to run it with xdebug (does it contain all the flags you pass)?

curry684 · 2016-03-14T09:16:32Z

  Package test/a cannot install to "... src\Test\A" inside its source at "... src\Test\A"

This error is really weird as well in this context. Have you created a path repository with a glob that matches both the install source and destination, ie. "url": "vendor/yourname/*", or multiple path repositories with that result?

If not, please share a bit more about the specific setup. This error indicates you're doing something really special that we may not have foreseen.

alcohol · 2016-03-14T09:21:21Z

Didn't even notice that yet, good catch @curry684.

However, his real issue should be reproducible with the given two composer.json files and directory structure (for anyone running Windows), as that does not seem to have a conflicting source/install path.

ghost · 2016-03-14T09:24:26Z

I removed every single xdebug configuration, so yes, thats the whole command.

There is no special repository - I can literally create these two files in a new directory, run composer update, modify the description and run it again for the error to appear.

curry684 · 2016-03-14T09:26:00Z

What the problem is with the junctioning is that the Symfony filesystem components does not know them, and contains a lot of fallback code for when cleanly removing a folder fails, using RecursiveDirectoryIterator instead, obviously cleaning out the source as well as the target. Thus we needed to have some extra patches in place to prevent that from happening at all costs.

I'll try reproducing the test case when I have time, I'm running PHP7 on Win10 as well so should be possible.

curry684 · 2016-03-14T10:20:12Z

I can't reproduce your issue sadly, but I just noticed the true culprit:

Updating dependencies (including require-dev)
  - Removing test/a (dev-master)

This is where it start going wrong, the rest is just symptoms, as it should log explicitly about the junction:

Updating dependencies (including require-dev)
  - Removing junction for test/a (dev-master)

Can you doublecheck that at that point the folder in Explorer shows a little junction arrow?

If so please share the output of:

php -r "var_dump(is_dir('vendor/test/a'));"
php -r "var_dump(is_link('vendor/test/a'));"
php -r "var_dump(lstat('vendor/test/a'));"

edit: I'm also curious whether composer require test/a && composer remove test/a correctly mentions and removes the junction for you. Could be an easier test case.

I suspect PHP has some internal issues with stat caches here, and xdebug is likely disabling those.

ghost · 2016-03-14T10:31:16Z

Yes it is definitely still a junction. (Restoring the source content also restores it in the junction.)

is_dir('vendor/test/a') => bool(true)
is_link('vendor/test/a') => bool(false)

array(26) {
  [0]=>
  int(4)
  [1]=>
  int(0)
  [2]=>
  int(0)
  [3]=>
  int(1)
  [4]=>
  int(0)
  [5]=>
  int(0)
  [6]=>
  int(4)
  [7]=>
  int(0)
  [8]=>
  int(1457950456)
  [9]=>
  int(1457950456)
  [10]=>
  int(1457950456)
  [11]=>
  int(-1)
  [12]=>
  int(-1)
  ["dev"]=>
  int(4)
  ["ino"]=>
  int(0)
  ["mode"]=>
  int(0)
  ["nlink"]=>
  int(1)
  ["uid"]=>
  int(0)
  ["gid"]=>
  int(0)
  ["rdev"]=>
  int(4)
  ["size"]=>
  int(0)
  ["atime"]=>
  int(1457950456)
  ["mtime"]=>
  int(1457950456)
  ["ctime"]=>
  int(1457950456)
  ["blksize"]=>
  int(-1)
  ["blocks"]=>
  int(-1)
}

Meanwhile I've been checking on my own. If I run composer with xdebug or print anything inside Filesystem::isJunction before the !is_dir | is_link check, lstat mode is 60230 otherweise its 0.

ghost · 2016-03-14T10:45:21Z

I meant the other way around sorry.. 60230 if I dont do anything and just check the mode after the lstat, 0 otherwise.

curry684 · 2016-03-14T11:03:00Z

Hang on, so what you are describing now is that this script:

<?php
$path = 'vendor/test/a';
var_dump(lstat($path)['mode']);
var_dump(is_dir($path) && !is_link($path));
var_dump(lstat($path)['mode']);

Gives different output depending on whether xdebug is enabled
Both show true in the middle
With xdebug disabled outputs 60230 and 0 first and last
With xdebug enabled outputs 0 both first and last

Is this correct?

With my 7.0.2 NTS build I get same output on both:

C:\Projects\ComposerTest>php test.php
int(0)
bool(true)
int(0)

C:\Projects\ComposerTest>php -n test.php
int(0)
8000

bool(true)
int(0)

php -n disables all configs, so no xdebug, if you want to doublecheck that you can add a var_dump(extension_loaded('xdebug')); in there.

ghost · 2016-03-14T11:15:36Z

Nono, I meant composer itself, not that. I still ran the script though, and it's as expected 0, true, 0 both times.

By modifying the source in Filesystem::isJunction i get the following:
var_dump($stat['mode']); before the return statement prints int(60320)

Except if I run it with xdebug or add e.g. echo 'x'; before the !is_dir || is_link condition. Then it prints int(0).

curry684 · 2016-03-14T11:22:58Z

I understand what's happening now and will issue a PR shortly. Would appreciate if you could test it.

Seldaek · 2016-03-14T11:34:25Z

You can run self-update and try with latest composer now.

ghost · 2016-03-14T11:36:45Z

Yep that works. - Thanks a lot!

curry684 · 2016-03-14T11:41:44Z

It's a really fubar situation as a whole hehe. The problem is that without xdebug the stat cache is far more aggressive, and in this specific case you are running into the issue that Composer core itself creates a placeholder folder for all packages it wants to install, with PHP's native functions. The junctions however need to be created via ProcessExecutor as PHP has not wrapped them. So at some point during execution vendor/test/a was a real directory for sure, and PHP never saw it change as we went to the system calls directly for that. So the caches still think it's not a junction while it is during the rollback... chaos ensues, oops.

Must say I've never liked PHP's stat caches, this just proves they can be an unexpected and unseen source of disaster.

Many thanks to you for the detailed reports and thorough testing 👍

ghost · 2016-03-14T12:04:40Z

Yea absolutely confusing.. :P
I still don't understand how printing something cleared the cache though. Especially because it had to be before the condition, which was never true - and didn't do anything before lstat itself.
Do you have any idea why that happened?

curry684 · 2016-03-14T12:10:20Z

To be honest I haven't a clue on that haha. The stat caches are internal magic and the bad part about magic is that it's just as hard to see why it's working as why it's not.

staabm · 2016-03-14T12:57:36Z

sounds like magic of a tick-function or error-handler or something

curry684 · 2016-03-14T13:00:00Z

Could even be output buffering causing file IO that triggers a stat cache overflow causing it to be reloaded correctly. I couldn't reproduce the actual issue once on Win10+PHP7.0.2 NTS, so it could even be that the thread safety is causing it in some freaky interaction with IIS on his computer.

ghost · 2016-03-14T15:59:22Z

I thought I might try that and indeed, using PHP NTS without cache clearing would've worked.

ghost closed this as completed Mar 13, 2016

ghost reopened this Mar 13, 2016

ghost closed this as completed Mar 13, 2016

ghost reopened this Mar 13, 2016

curry684 mentioned this issue Mar 14, 2016

Ensure stat cache is cleared for junctions #5057

Merged

Seldaek closed this as completed in #5057 Mar 14, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Junction content still gets deleted after composer update #5053

Junction content still gets deleted after composer update #5053

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Junction content still gets deleted after composer update #5053

Junction content still gets deleted after composer update #5053

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!