Magento Enterprise Cloud Edition bug: «The file "/composer.json" doesn't exist»

it is the Magento-Cloud Program (magento-cloud CLI)
which is really just a rebranding of Platform.sh

if you want try locally, then install magento-cloud
http://devdocs.magento.com/guides/v2.1/cloud/before/before-workspace-cli.html

If you are able to find the issue is in their code (which I don’t doubt at all) then
the scripts that are responsible for the deploy process i believe are these:

#/var/www/beta.pumpunderwear.com/web/2e/vendor/magento/magento-cloud-configuration/pre-deploy.php


<?php
/**
 * This script contains logic to cleanup outdated caches and restore the contents of mounted directories so that
 * the main deploy hook is able to start.
 */

// Should be deleted at the end of pre-deploy, so presence of flag later indicate if something failed in the pre-deploy.
echo "Setting the pre-deploy flag." . PHP_EOL;
use Magento\MagentoCloud\Environment;
require_once 'src/Magento/MagentoCloud/Environment.php';

touch(Environment::PRE_DEPLOY_FLAG);

$env = new Environment();
$env->log("Starting pre-deploy.");

// Clear redis and file caches
$relationships = $env->getRelationships();
$var = $env->getVariables();
$useGeneratedCodeSymlink = isset($var["GENERATED_CODE_SYMLINK"]) && $var["GENERATED_CODE_SYMLINK"] == 'disabled' ? false : true;
$useStaticContentSymlink = isset($var["STATIC_CONTENT_SYMLINK"]) && $var["STATIC_CONTENT_SYMLINK"] == 'disabled' ? false : true;

if (isset($relationships['redis']) && count($relationships['redis']) > 0) {
    $redisHost = $relationships['redis'][0]['host'];
    $redisPort = $relationships['redis'][0]['port'];
    $redisCacheDb = '1'; // Matches \Magento\MagentoCloud\Console\Command\Deploy::$redisCacheDb
    $env->execute("redis-cli -h $redisHost -p $redisPort -n $redisCacheDb flushdb");
}

$fileCacheDir = Environment::MAGENTO_ROOT . '/var/cache';
if (file_exists($fileCacheDir)) {
    $env->execute("rm -rf $fileCacheDir");
}

$mountedDirectories = ['app/etc', 'pub/media'];

/**
 * optionally symlink DI assets from build resources directory(var/generation to init/var/generation
 * (var/di -> init/var/di, var/generation -> init/var/generation)
 **/

$buildDir = realpath(Environment::MAGENTO_ROOT . 'init') . '/';
if ($useGeneratedCodeSymlink) {
    $varDir = realpath(Environment::MAGENTO_ROOT . 'var') . '/';
    $env->removePathInBackground('var/di');
    $env->removePathInBackground('var/generation');
    if (symlink($buildDir . 'var/generation', $varDir . 'generation')) {
        $env->log('Symlinked var/generation to init/var/generation');
    } else {
        $env->log('Failed to symlink var/generation to init/var/generation');
    }

    if (symlink($buildDir . 'var/di', $varDir . 'di')) {
        $env->log('Symlinked var/di to init/var/di');
    } else {
        $env->log('Failed to symlink var/di to init/var/di');
    }
} else {
    $env->atomicCopyPath('init/var/generation', 'var/generation');
    $env->atomicCopyPath('init/var/di', 'var/di');
}

/**
 * Handle case where static content is deployed during build hook:
 *  1. set a flag to be read by magento-cloud:deploy
 *  2. Either copy or symlink files from init/ directory, depending on strategy
 */
if (file_exists(Environment::MAGENTO_ROOT . 'init/' . Environment::STATIC_CONTENT_DEPLOY_FLAG)) {
    $env->log("Static content deployment was performed during build hook");
    $env->removeStaticContent();
    $env->setStaticDeployInBuild(true);

    if ($useStaticContentSymlink) {
        $env->log("Symlinking static content from pub/static to init/pub/static");

        // Symlink pub/static/* to init/pub/static/*
        $staticContentLocation = realpath(Environment::MAGENTO_ROOT . 'pub/static') . '/';
        if (file_exists($buildDir . 'pub/static')) {
            $dir = new \DirectoryIterator($buildDir . 'pub/static');
            foreach ($dir as $fileInfo) {
                $fileName = $fileInfo->getFilename();
                if (!$fileInfo->isDot() && symlink($buildDir . 'pub/static/' . $fileName, $staticContentLocation . '/' . $fileName)) {
                    $env->log('Symlinked ' . $staticContentLocation . '/' . $fileName . ' to ' . $buildDir . 'pub/static/' . $fileName);
                }
            }
        }
    } else {
        $env->log("Copying static content from init/pub/static to pub/static");
        copyFromBuildDir('pub/static', $env);
    }
}

// Restore mounted directories
$env->log("Copying writable directories back.");

foreach ($mountedDirectories as $dir) {
    copyFromBuildDir($dir, $env);
}

if (file_exists(Environment::REGENERATE_FLAG)) {
    $env->log("Removing var/.regenerate flag");
    unlink(Environment::REGENERATE_FLAG);
}

$env->log("Pre-deploy complete.");
unlink(Environment::PRE_DEPLOY_FLAG);

/**
 * @param string $dir The directory to copy. Pass in its normal location relative to Magento root with no prepending
 *                    or trailing slashes
 * @param Environment $env
 */
function copyFromBuildDir($dir, Environment $env) {
    if (!file_exists($dir)) {
        mkdir($dir);
        $env->log(sprintf('Created directory: %s', $dir));
    }
    $env->execute(sprintf('/bin/bash -c "shopt -s dotglob; cp -R ./init/%s/* %s/ || true"', $dir, $dir));
    $env->log(sprintf('Copied directory: %s', $dir));
}

#/var/www/beta.pumpunderwear.com/web/2e/vendor/magento/magento-cloud-configuration/patch.php


<?php

use Magento\MagentoCloud\Environment;

require_once 'src/Magento/MagentoCloud/Environment.php';
$env = new Environment();

$env->log("Copying static.php to front-static.php");
copy(Environment::MAGENTO_ROOT . 'pub/static.php', Environment::MAGENTO_ROOT . 'pub/front-static.php');

$dirName = __DIR__ . '/patches';

$files = glob($dirName . '/*');
sort($files);
foreach ($files as $file) {
    $cmd = 'git apply '  . $file;
    $env->execute($cmd);
}

copy(Environment::MAGENTO_ROOT . 'app/etc/di.xml', Environment::MAGENTO_ROOT . 'app/di.xml');
$enterpriseFolder = Environment::MAGENTO_ROOT . 'app/enterprise';
if(!file_exists($enterpriseFolder)){
    mkdir($enterpriseFolder, 0777, true);
}
copy(Environment::MAGENTO_ROOT . 'app/etc/enterprise/di.xml', Environment::MAGENTO_ROOT . 'app/enterprise/di.xml');

$sampleDataDir = Environment::MAGENTO_ROOT . 'vendor/magento/sample-data-media';
if (file_exists($sampleDataDir)) {
    $env->log("Sample data media found. Marshalling to pub/media.");
    $destination = Environment::MAGENTO_ROOT . '/pub/media';
    foreach (
        $iterator = new \RecursiveIteratorIterator(
            new \RecursiveDirectoryIterator($sampleDataDir, \RecursiveDirectoryIterator::SKIP_DOTS),
            \RecursiveIteratorIterator::SELF_FIRST) as $item
    ) {
        if ($item->isDir()) {
            if (!file_exists($destination . DIRECTORY_SEPARATOR . $iterator->getSubPathName())) {
                mkdir($destination . DIRECTORY_SEPARATOR . $iterator->getSubPathName());
            }
        } else {
            copy($item, $destination . DIRECTORY_SEPARATOR . $iterator->getSubPathName());
        }
    }
}

#/var/www/beta.pumpunderwear.com/web/2e/vendor/magento/magento-cloud-configuration/src/Magento/MagentoCloud/Environment.php

<?php
/**
 * Copyright © 2016 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */

namespace Magento\MagentoCloud;

/**
 * Contains logic for interacting with the server environment
 */
class Environment
{
    const MAGENTO_ROOT = __DIR__ . '/../../../../../../';
    const STATIC_CONTENT_DEPLOY_FLAG = 'var/.static_content_deploy';
    const PRE_DEPLOY_FLAG = self::MAGENTO_ROOT . 'var/.predeploy_in_progress';
    const REGENERATE_FLAG = self::MAGENTO_ROOT . 'var/.regenerate';

    public $writableDirs = ['var/di', 'var/generation', 'app/etc', 'pub/media'];

    /**
     * Get routes information from MagentoCloud environment variable.
     *
     * @return mixed
     */
    public function getRoutes()
    {
        return json_decode(base64_decode($_ENV["MAGENTO_CLOUD_ROUTES"]), true);
    }

    /**
     * Get relationships information from MagentoCloud environment variable.
     *
     * @return mixed
     */
    public function getRelationships()
    {
        return json_decode(base64_decode($_ENV["MAGENTO_CLOUD_RELATIONSHIPS"]), true);
    }

    /**
     * Get custom variables from MagentoCloud environment variable.
     *
     * @return mixed
     */
    public function getVariables()
    {
        return json_decode(base64_decode($_ENV["MAGENTO_CLOUD_VARIABLES"]), true);
    }


    public function log($message)
    {
        echo sprintf('[%s] %s', date("Y-m-d H:i:s"), $message) . PHP_EOL;
    }

    public function execute($command)
    {
        $this->log('Command:'.$command);

        exec(
            $command,
            $output,
            $status
        );

        $this->log('Status:'.var_export($status, true));
        $this->log('Output:'.var_export($output, true));

        if ($status != 0) {
            throw new \RuntimeException("Command $command returned code $status", $status);
        }

        return $output;
    }

    public function backgroundExecute($command)
    {
        $command = "nohup {$command} 1>/dev/null 2>&1 &";
        $this->log("Execute command in background: $command");
        shell_exec($command);
    }

    public function setStaticDeployInBuild($flag)		
    {
        if ($flag) {
         $this->log('Setting flag file ' . Environment::STATIC_CONTENT_DEPLOY_FLAG);
         touch(Environment::MAGENTO_ROOT . Environment::STATIC_CONTENT_DEPLOY_FLAG);
        } else {
            if ($this->isStaticDeployInBuild()) {
                    $this->log('Removing flag file ' . Environment::STATIC_CONTENT_DEPLOY_FLAG);
                    unlink(Environment::MAGENTO_ROOT . Environment::STATIC_CONTENT_DEPLOY_FLAG);
                }
        }
    }

    public function isStaticDeployInBuild()
    {
        return file_exists(Environment::MAGENTO_ROOT . Environment::STATIC_CONTENT_DEPLOY_FLAG);
    }

    public function removeStaticContent()
    {
        // atomic move within pub/static directory
        $staticContentLocation = realpath(Environment::MAGENTO_ROOT . 'pub/static/') . '/';
        $timestamp = time();
        $oldStaticContentLocation = $staticContentLocation . 'old_static_content_' . $timestamp;

        $this->log("Moving out old static content into $oldStaticContentLocation");

        if (!file_exists($oldStaticContentLocation)) {
            mkdir($oldStaticContentLocation);
        }

        $dir = new \DirectoryIterator($staticContentLocation);

        foreach ($dir as $fileInfo) {
            $fileName = $fileInfo->getFilename();
            if (!$fileInfo->isDot() && strpos($fileName, 'old_static_content_') !== 0) {
                $this->log("Rename " . $staticContentLocation . '/' . $fileName . " to " . $oldStaticContentLocation . '/' . $fileName);
                rename($staticContentLocation . '/' . $fileName, $oldStaticContentLocation . '/' . $fileName);
            }
        }

        $this->log("Removing $oldStaticContentLocation in the background");
        $this->backgroundExecute("rm -rf $oldStaticContentLocation");

        $preprocessedLocation = realpath(Environment::MAGENTO_ROOT . 'var') . '/view_preprocessed';
        if (file_exists($preprocessedLocation)) {
            $oldPreprocessedLocation = $preprocessedLocation . '_old_' . $timestamp;
            $this->log("Rename $preprocessedLocation  to $oldPreprocessedLocation");
            rename($preprocessedLocation, $oldPreprocessedLocation);
            $this->log("Removing $oldPreprocessedLocation in the background");
            $this->backgroundExecute("rm -rf $oldPreprocessedLocation");
        }
    }

    public function removePathInBackground($relativePath)
    {
        $fullPath = rtrim(realpath(Environment::MAGENTO_ROOT) . '/' . $relativePath, '/');
        $timestamp = time();
        if (file_exists($fullPath)) {
            $backgroundLocation = $fullPath . '_old_' . $timestamp;
            $this->log("Rename $fullPath  to $backgroundLocation");
            rename($fullPath, $backgroundLocation);
            $this->log("Removing $backgroundLocation in the background");
            $this->backgroundExecute("rm -rf $backgroundLocation");
        }
    }

    public function atomicCopyPath($relativeSourcePath, $relativeDestinationPath) {
        $fullDestinationPath = rtrim(realpath(Environment::MAGENTO_ROOT) . '/' . $relativeDestinationPath, '/');
        $fullSourcePath = rtrim(realpath(Environment::MAGENTO_ROOT) . '/' . $relativeSourcePath, '/');
        $timestamp = time();
        if (file_exists($fullSourcePath)) {
            $tmpLocation = $fullDestinationPath . '_tmp_' . $timestamp;
            $this->log("Copy $fullSourcePath  to $tmpLocation");
            $this->execute(sprintf('bash -c "shopt -s dotglob; cp -R %s/ %s/ || true"', $fullSourcePath, $tmpLocation));
            $this->removePathInBackground($relativeDestinationPath);
            $this->log("Rename $tmpLocation to $fullDestinationPath");
            rename($tmpLocation, $fullDestinationPath);
        }
    }
}

So which code line exactly generates the «The file “/composer.json” doesn’t exist» message?
I do not see it in the code you have published, and it looks like the message is generated by another code.
If you do not know the answer, then ask it from Magento Enterprise support, because it is their code.

No I don’t know the answer but the only difference between the code in default Magento and cloud is the folder vendor/magento/magento-cloud-configuration
so it must be in there somewhere.

Magento support takes 2 weeks to repsond to even th most urgent issues… their response for this was to un-install Stripe… obviously we can’t run without a payment processor and we also can’t wait two weeks for Magento to respond

the /composer.json line that I wrote was just what I had seen in the output of the deployment and it is only my assumption that it causes the problem. So it could be elsewhere in the magento-cloud-configuration code that Stripe is not compatible with.

You can see directly by trying to install Stripe on the beta server and then adding the code to git and pushing to magento master. You will see the results directly.

As a guess, the message could be generated by the Symfony Config Component.
It is used by Magento 2.

The method Symfony\Component\Config\Resource\FileResource::__construct() has the following implementation:

So it definitely can generate the «The file “/composer.json” doesn’t exist».
Can you log this exception and get it backtrace?
I am unable to do it myself, because you said your production filesystem is read-only, and, as I understand, the message is generated by the production instance.

Oh, I am wrong, it is not the Symfony Config Component, because its message ends on «does not exist», not on «doesn’t exist»

All these messages (and the error «The file “/composer.json” doesn’t exist» message) are generated by an unknown black-box closed source script/program:

Building application ‘mymagento’ (runtime type: php:7.0, tree ID: 4be8741)
Generating runtime configuration.

Moving the application to the output directory
W: Overriding path robots.txt that already exists in destination.
Prewarming composer cache.
Pre-downloaded 2 packages referenced in composer.lock

These strings are absent in your beta and production websites source code.
So the only way to investigate the case and this unknown black-box closed source program requirements is to contact the program author - Magento Enterprise team.
Ask them:

  • Where is exactly the «The file “/composer.json” doesn’t exist» message generated?
  • Why is the message generated? What are the hidden requirements about such «composer.json» files?

I used the following commands to look for the messages:

grep -rnw . -e "Moving the application"
find . -name '*.*' -print | xargs grep "Moving the application"

They all return nothing.

        
        
                                                             
          [Magento\Framework\Exception\FileSystemException]  
          The file "/composer.json" doesn't exist            
                                                             
        
        
        module:enable [-f|--force] [--all] [-c|--clear-static-content] [--magento-init-params="..."] [module1] ... [moduleN]
        
        
        
        
                                                                         
          [RuntimeException]                                             
          Command php ./bin/magento module:enable --all returned code 1  
                                                                         
        
        
        magento-cloud:build

it is this command during the process where it fails:
php ./bin/magento magento-cloud:build

Yet again:

All these messages (and the error «The file “/composer.json” doesn’t exist» message) are generated by an unknown black-box closed source script/program

If you want to prove opposite, then provide THE EXACT source code where these messages are generated.

./Event/Collection.php: * If event doesn’t exist creates new one and returns it
./App/MaintenanceMode.php: * It is going to work much faster in 99% of cases: the isOn() will return false whenever file doesn’t exist.
./App/State/CleanupFiles.php: $messages[] = “The directory ‘{$dirPath}’ doesn’t exist - skipping cleanup”;
./Search/Request/Builder.php: throw new NonExistingRequestNameException(new Phrase(“Request name ‘%1’ doesn’t exist.”, [$requestName]));
./Search/Test/Unit/Request/BuilderTest.php: * @expectedExceptionMessage Request name ‘rn’ doesn’t exist.
./View/TemplateEngineFactory.php: * @throws \InvalidArgumentException If template engine doesn’t exist
./File/Mime.php: throw new \InvalidArgumentException(“File ‘$file’ doesn’t exist”);
./File/Test/Unit/MimeTest.php: * @expectedExceptionMessage File ‘nonexistent.file’ doesn’t exist
./Data/Structure.php: * @throws LocalizedException if doesn’t exist
./System/Ftp.php: throw new \Exception(“Local file doesn’t exist: {$local}”);
./Profiler/Driver/Standard/Stat.php: * @throws \InvalidArgumentException if timer doesn’t exist
./Profiler/Driver/Standard/Stat.php: * @throws \InvalidArgumentException if timer doesn’t exist
./Profiler/Driver/Standard/Output/Factory.php: sprintf(“Cannot create standard driver output, class “%s” doesn’t exist.”, $class)
./Profiler/Driver/Factory.php: sprintf(“Cannot create profiler driver, class “%s” doesn’t exist.”, $class)
./Profiler/Test/Unit/Driver/Standard/StatTest.php: * @expectedExceptionMessage Timer “unknown_timer” doesn’t exist.
./Profiler/Test/Unit/Driver/Standard/StatTest.php: * @expectedExceptionMessage Timer “unknown_timer” doesn’t exist.
./Profiler/Test/Unit/Driver/Standard/StatTest.php: * @expectedMessage Timer “foo” doesn’t exist.

./magento2-base/setup/src/Magento/Setup/Model/Installer.php: $this->log->log("The file '{$absolutePath}' doesn't exist - skipping cleanup");

These messages are unrelated to the error message.

Local file doesn’t exist: {$local}

The word «Local» is absent in the error message.

The file ‘{$absolutePath}’ doesn’t exist - skipping cleanup

The «- skipping cleanup» suffix is absent in the error message.

does it have anything to do with this :

PHP Fatal error:  Uncaught RuntimeException: ObjectManager isn't initialized in /var/www/beta.pumpunderwear.com/web/2e/vendor/magento/framework/App/ObjectManager.php:31
Stack trace:
#0 /var/www/beta.pumpunderwear.com/web/2e/vendor/mage2pro/core/Core/lib/om.php(37): Magento\Framework\App\ObjectManager::getInstance()
#1 /var/www/beta.pumpunderwear.com/web/2e/vendor/mage2pro/core/Core/lib/om.php(29): df_om()
#2 /var/www/beta.pumpunderwear.com/web/2e/vendor/mage2pro/core/Core/lib/cache.php(250): {closure}('Magento\\Framewo...')
#3 /var/www/beta.pumpunderwear.com/web/2e/vendor/mage2pro/core/Core/lib/om.php(29): dfcf(Object(Closure), Array)
#4 /var/www/beta.pumpunderwear.com/web/2e/vendor/mage2pro/core/Core/lib/module.php(37): df_o('Magento\\Framewo...')
#5 /var/www/beta.pumpunderwear.com/web/2e/vendor/mage2pro/core/Core/lib/module.php(174): df_module_dir('Df_Core')
#6 /var/www/beta.pumpunderwear.com/web/2e/vendor/mage2pro/core/Core/lib/composer.php(34): df_module_path('Df_Core')
#7 /var/www/beta.pumpunderwear.com/web/2e/vendor/mage in /var/www/beta.pumpunderwear.com/web/2e/vendor/magento/framework/App/ObjectManager.php on line 31
PHP Stack trace:
PHP   1. {main}() /var/www/beta.pumpunderwear.com/web/2e/bin/magento:0
PHP   2. require() /var/www/beta.pumpunderwear.com/web/2e/bin/magento:14
PHP   3. require_once() /var/www/beta.pumpunderwear.com/web/2e/app/bootstrap.php:30
PHP   4. include() /var/www/beta.pumpunderwear.com/web/2e/app/autoload.php:21
PHP   5. ComposerAutoloaderInitc835b588ff1adeacac61af3874973a0f::getLoader() /var/www/beta.pumpunderwear.com/web/2e/vendor/autoload.php:7
PHP   6. composerRequirec835b588ff1adeacac61af3874973a0f() /var/www/beta.pumpunderwear.com/web/2e/vendor/composer/autoload_real.php:61

No.

So what am I supposed to do with this? Only solution offered so far by magento is remove module … is that where we leave this ? When I do remove stripe the Magento cloud deploy Process does in fact work as they say …

Learn the answer here: Magento Enterprise Cloud Edition bug: «The file "/composer.json" doesn't exist»
If you do not understand it, re-read it again.
Why is it so hard to understand what I write from the first attempt?!?

It’s not hard to understand, its just not a real answer. Magento says this:

Vidal Yesterday at 00:34
Hello Sam,

Unfortunately, the creator of the extension must determine the cause of the issue between Magento and the extension. The integration of any 3rd party application is the responsibility of the SI and/or the creator of the extension.

Best,
Vidal

Meanwhile we can’t even deploy Magento to the Cloud without removing Stripe our pay processor. And neither you or Magento seem to care.

All my extensions are 100% Open Source and published on GitHub: https://github.com/mage2pro
So they can be easily explored and debugged by other developers and customers.
Magento Enterprise Cloud is a closed source and undocumented (!) service with (as you said) 2 weeks delay in customer support.
I do not know what is the cause of your issue, and I do not have any ability to debug it because of lack of the Cloud’s source code and specification.
I recommend you just leave Magento Cloud and use your own server, which you can control and debug.

Luckily, I was able to reproduce a similar issue and investigate it:

With a high probability it is the same issue as yours, and I have created a workaround for it the 2.9.9 version of the mage2pro/core package.

Just upgrade your Stripe extension and try to redeploy your website to production again.