# Ghost product images (2026-09-28)

Why a product's main image (or any image in its gallery) can show as a broken/empty thumbnail even though the product record says it has one, how it happened, what was fixed, and how to find/clean up existing cases.

---

## 1. What a "ghost" image is

A product's `images` column (`products_products.images`, cast to `array`) holds a list of storage paths. The gallery/edit screen renders each path via `Storage::disk($disk)->url($path)`. A "ghost" entry is a path recorded in that array where **no file actually exists at that path on the active disk** — the DB says the image exists, storage disagrees, so the browser shows a broken image icon.

Found investigating product **LR07989** (id 5601) — one of its three recorded images
(`9be76904-84fb-4f19-b30b-08e678b05950.jpg`) didn't exist on R2, while the other two did.

## 2. Root cause (fixed)

Production's file disk is R2 (`config/filesystems.php`, `'r2'`), configured with **`'throw' => false`**. That means a failed Flysystem S3 write doesn't throw — it just returns `false`.

Every image-storing path in the app (`plugins/webkul/imports/src/ImportManager.php`: `downloadAndStoreImages()` and `storeLocalImage()`) used to ignore that return value:

```php
Storage::disk($disk)->put($filename, $response->body(), ['visibility' => 'public']);
$stored[] = $filename;   // ran even if put() just returned false
```

So a transient R2 write failure (timeout, rate limit, blip) on **one image out of a batch** silently recorded that filename as "stored" anyway — exactly the "just the odd image" pattern. Nothing logged it, nothing retried it. For Allmakes/PSP-sourced products it's worse: the scraper marks `allmakes_psp_results.image_fetched_at` as done once `$stored` is non-empty, so it never revisits a product that got a ghost entry.

**Fixed** in `ImportManager.php` — both methods now throw a `RuntimeException` when `put()` returns false. Every call site (PSP scrape `ScrapeAllmakesProductData.php`, `ProcessProductImageImportJob`, `ProcessProductImageZipImportJob`, `AllmakesNewProduct.php`) already wraps the call in try/catch, so nothing crashes: PSP scrape logs a warning and leaves `image_fetched_at` null (retries next scrape run), CSV/ZIP imports mark that row failed with a visible error instead of silently "succeeding" with a broken image.

Test: `tests/Unit/ImportManagerImageStorageTest.php` (3 tests) — covers both the throw-on-failure and still-works-on-success paths using a mocked disk.

**This fix covers:** PSP/Allmakes scrape, CSV/ZIP bulk image import, the "Add new product" PSP image fetch — i.e. every current automated image-storing code path.

**This fix does NOT cover:** manually dragging a file into the Images panel on the product edit form. That goes through Filament's own vendor upload code (`vendor/filament/forms/src/Components/BaseFileUpload.php`, `saveUploadedFileUsing`), which calls Laravel's `storePubliclyAs()` — same disk, same `throw => false` exposure, but it's vendor code, not something this fix touches. In practice much lower risk (one manual upload vs. an automated batch of many requests — more requests, more chances one fails), but it's a known gap. Can be closed with a custom `->saveUploadedFileUsing()` override on the `FileUpload::make('images')` field in `plugins/webkul/products/src/Filament/Resources/ProductResource.php` if it ever turns out to matter.

**LR07989's specific ghost entry predates this fix anyway** — its filename (`9be76904-....jpg`, no directory, no SKU prefix) doesn't match any current code path's naming convention (everything today writes under `product_images/not_watermarked/<sku>_<n>_<ulid>.ext`). It's most likely leftover from the original data migration into R2, not something the currently-fixed bug actively produced. The fix still matters going forward for any *new* ghost entries.

## 3. Diagnosing one product

Server has no web UI for this — use `php artisan tinker --execute`, not the interactive shell (its REPL chokes on multi-line chained method calls pasted from outside — write a script file and `require` it instead).

```bash
cat > /tmp/check-product.php <<'EOF'
<?php
$disk = config('filesystems.disks.r2') ? 'r2' : config('filesystems.default', 'public');
echo "Active disk: $disk\n";

$p = \Webkul\Product\Models\Product::find(PRODUCT_ID_HERE);   // or ->where('sku', '...')->first()

if (! $p) { echo "Not found.\n"; exit; }

echo "Product id: {$p->id}, sku: {$p->sku}\n";
foreach ($p->images ?? [] as $path) {
    $exists = \Storage::disk($disk)->exists($path);
    echo ($exists ? 'OK   ' : 'GHOST') . " - $path\n";
}
EOF
php artisan tinker --execute="require '/tmp/check-product.php';"
```

## 4. Fixing one product (drop just the ghost entry)

```bash
cat > /tmp/fix-product.php <<'EOF'
<?php
$disk = config('filesystems.disks.r2') ? 'r2' : config('filesystems.default', 'public');
$p = \Webkul\Product\Models\Product::find(PRODUCT_ID_HERE);
if (! $p) { echo "Not found.\n"; exit; }

$before = $p->images ?? [];
$good = array_values(array_filter($before, fn ($path) => \Storage::disk($disk)->exists($path)));

echo "Before: " . count($before) . " images. After: " . count($good) . " images.\n";
foreach (array_diff($before, $good) as $removed) { echo "Removing: $removed\n"; }

$p->images = $good;
$p->save();
echo "Saved.\n";
EOF
php artisan tinker --execute="require '/tmp/fix-product.php';"
```

If the product is Allmakes/PSP-sourced, also clear the "already fetched" flag so the next scrape re-downloads a replacement image (dropping a ghost entry alone doesn't trigger a re-fetch):

```bash
php artisan tinker --execute="\DB::table('allmakes_psp_results')->where('product_id', PRODUCT_ID_HERE)->update(['image_fetched_at' => null]);"
```

## 5. Catalog-wide audit (how many products are affected)

Checking every image with one `exists()` call each would be ~40,000 sequential HTTP round-trips to R2 (30–90+ minutes, based on local catalog size of ~20k products / ~39.5k image entries). Instead: **list the whole disk once** (a handful of paginated LIST calls) into an in-memory set, then check membership locally — turns it into roughly a minute.

```bash
cat > /tmp/audit-ghost-images-fast.php <<'EOF'
<?php
use Symfony\Component\Console\Helper\ProgressBar;
use Symfony\Component\Console\Output\ConsoleOutput;

$disk = config('filesystems.disks.r2') ? 'r2' : config('filesystems.default', 'public');
$output = new ConsoleOutput();

$output->writeln("Active disk: $disk");
$output->writeln("Listing all files on disk (one pass, this is the only slow-ish step)...");

$start = microtime(true);
$existing = [];
foreach (\Storage::disk($disk)->allFiles() as $f) {
    $existing[$f] = true;
}
$listSeconds = round(microtime(true) - $start, 1);
$output->writeln('Listed ' . count($existing) . " files in {$listSeconds}s.");
$output->writeln('');

$totalWithImages = \Webkul\Product\Models\Product::whereNotNull('images')->where('images', '!=', '[]')->count();

$bar = new ProgressBar($output, $totalWithImages);
$bar->setFormat(' %current%/%max% [%bar%] %percent:3s%%  elapsed: %elapsed:6s%  eta: %estimated:-6s%');
$bar->start();

$affectedProducts = 0;
$ghostCount = 0;
$checked = 0;
$ghostReport = [];

\Webkul\Product\Models\Product::query()
    ->whereNotNull('images')
    ->where('images', '!=', '[]')
    ->orderBy('id')
    ->chunkById(500, function ($products) use ($existing, &$affectedProducts, &$ghostCount, &$checked, &$ghostReport, $bar) {
        foreach ($products as $p) {
            $images = $p->images ?? [];
            $ghosts = [];
            foreach ($images as $path) {
                $checked++;
                if (! isset($existing[$path])) {
                    $ghosts[] = $path;
                }
            }
            if (! empty($ghosts)) {
                $affectedProducts++;
                $ghostCount += count($ghosts);
                $ghostReport[] = ['id' => $p->id, 'sku' => $p->sku, 'ghosts' => $ghosts];
            }
            $bar->advance();
        }
    });

$bar->finish();
$output->writeln('');
$output->writeln('');
$output->writeln('---');
$output->writeln("Checked $checked image paths across $totalWithImages products.");
$output->writeln("$affectedProducts product(s) affected, $ghostCount ghost entr" . ($ghostCount === 1 ? 'y' : 'ies') . ' total.');
$output->writeln('');

if (! empty($ghostReport)) {
    $lines = [];
    foreach ($ghostReport as $r) {
        $lines[] = "Product {$r['id']} (sku: {$r['sku']}): " . count($r['ghosts']) . ' ghost(s)';
        foreach ($r['ghosts'] as $g) { $lines[] = "    - $g"; }
    }
    file_put_contents('/tmp/ghost-images-report.txt', implode("\n", $lines) . "\n");

    $output->writeln('First 50 lines below — full list saved to /tmp/ghost-images-report.txt:');
    foreach (array_slice($lines, 0, 50) as $l) { $output->writeln($l); }
    if (count($lines) > 50) { $output->writeln('... see /tmp/ghost-images-report.txt for the rest'); }
}
EOF
php artisan tinker --execute="require '/tmp/audit-ghost-images-fast.php';"
```

Run this directly in the terminal (don't redirect to a file — the progress bar needs a real terminal to redraw). The full per-product ghost list is saved to `/tmp/ghost-images-report.txt` on the server regardless of terminal scrollback.

**Result (run 2026-09-28):**

- Checked ~39,500 image paths across ~20,158 products with images.
- **2,693 products affected**, **2,693 ghost entries** total (every affected product had exactly one).
- **2,692 of 2,693 (99.96%)** share the exact same shape: a bare `<uuid>.jpg` with no directory and no SKU prefix — one single systemic cause, not scattered failures. The one outlier (`jlr-assigned-sku-diagrams/12774/771-1788425222.png`) is an unrelated, isolated missing diagram file.
- Full per-product list saved to `/tmp/ghost-images-report.txt` on the server (regenerate with the script above if it's gone).

### Where the two batches came from

Correlating the 2,693 affected products by `created_at` split them into two tight clusters, not a spread — confirming one-off historical events rather than ongoing drip damage:

| Date | Count | Source |
|---|---|---|
| 2026-05-08 | 1,375 | No matching row in `imports_import_jobs` under any type — not done through the in-app Import feature. Likely a pre-launch/early catalog seed script run directly, predating today's logged import path. Not pinned down further. |
| 2026-06-17 | 1,318 | `imports_import_jobs` id=36, type `product`, file `01KVAQW380Q99B63FSXMAG3JYA.csv`, **11,420 rows, status completed, 0 reported failures**. Concrete proof the bug in section 2 was live that day — the import "succeeded" with zero visible errors while silently leaving ~1,318 broken image references behind. |

Vendor spread across the 2,693 is broad (Allmakes 4x4, Land Rover, OEM, Autotec, BWI, Bosch, Delphi, ATE, Nissens, Britpart, Terrafirma, Superseded, Proevo+, ...) — **all 2,693 turned out to be eligible for a PSP re-pull anyway** (linked to the Allmakes partner in `products_product_suppliers` with a `product_code`), regardless of their vendor label. Allmakes/PSP is evidently the supplier behind many of these OEM-branded parts, not just their own-branded stock — confirmed by testing a "BWI"-branded product (LR087084W) which repaired successfully through the same PSP pipeline.

### Live storefront impact is smaller than the raw count suggests

Testing the repair on LR035543 showed the **live Somerset4x4 storefront was already displaying the correct image**, before any Shopify push happened for that product. Shopify keeps its own uploaded copy of a product's images, decoupled from whatever is currently in Cydekick's `images` column — so a ghost entry breaking the *admin* thumbnail doesn't necessarily mean a customer ever saw a broken image, for any product that was already listed on Shopify before it went ghost. It still needs fixing (admin correctness, and any *future* Shopify push or brand-new listing would fail without a real image on file), but the customer-facing blast radius for already-listed products is smaller than 2,693.

## 6. Bulk repair pipeline (PSP re-pull → reorder → watermark → Shopify push)

Since all 2,693 are Allmakes/PSP-eligible, the fix isn't just "drop the ghost entry" — it's a four-phase pipeline per product, reusing existing, already-tested infrastructure rather than writing new image-fetch/watermark/Shopify code:

1. **PSP re-pull.** `ScrapeAllmakesProductData` (`plugins/webkul/allmakes-psp/src/Jobs/ScrapeAllmakesProductData.php`) already treats any image path that isn't under `product_images/` or doesn't exist on disk as "not current" and re-fetches it (line ~133), and already accepts a `targetProductIds` array — no changes needed, just call it with the ghost list. Run its `handle()` method directly (not `::dispatch()`) to execute synchronously in a tinker script rather than depending on a queue worker.
2. **Reorder + drop ghost.** The scrape job *appends* the freshly-fetched image and leaves old entries (including the ghost) in place — it doesn't reorder or clean up. A snapshot-diff (`images` before vs. after the scrape call) identifies the newly added path per product; the final array is `[new image, ...surviving old images that still exist on disk]`, i.e. the fresh PSP image becomes the first/"main" slot and the dead ghost path is dropped.
3. **Watermark.** `WatermarkProductJob` (`plugins/webkul/image-watermark/src/Jobs/WatermarkProductJob.php`) processes every image in a product's `images` array and is naturally idempotent — it skips any image whose watermark is already current (`config_hash` match), so it's safe to call for a product's full image list even when most of them were already watermarked. Resolve the company's `WatermarkConfig` (`WatermarkConfig::where('company_id', ...)->where('is_enabled', true)->first()`) and call the job's `handle()` directly, same synchronous pattern as phase 1.
4. **Shopify push.** *Not custom-built* — `ManageListingsV2::bulkPush('images_only')` already does exactly this from the admin UI (Manage Listings page: select products, pick a channel, bulk action), dispatching `ShopifyBatchSyncJob` with `syncMode = 'images_only'` (a purpose-built mode, logged as "Shopify Watermark Push"). The repair script replicates that same query logic (channel mapping check, £0.00-price guard, `channel_listings` status update) so results land identically to using the UI, then dispatches the job for real — this one **is** queued (`ShouldQueue`), not run inline, same as the normal UI flow; a queue worker was confirmed already active from the JobLogger console. Scoped to **channel 12 (Somerset4x4 Website)** only per instruction — Tool365 (channel 13) was deliberately left out, run separately later if wanted.

Two Shopify channels exist: `id=12` Somerset4x4 Website (`7arei1-cj.myshopify.com`), `id=13` Tool365 Website (`qnpufk-t0.myshopify.com`).

**Tested and confirmed working**, scaling up in stages: 1 product (LR035543 / id 2821) → 20-product batch → full remaining catalog. The full-scale script is idempotent (checks whether each product is still actually broken before doing anything, skips cleanly if not) so it's safe to re-run if interrupted, and it was deliberately built that way so the same script serves both the batch tests and the final full run without needing a separate "already done" id list to track by hand.

**Full-batch script** (safe to re-run; auto-skips anything already fixed):

```bash
cat > /tmp/repair-all.php <<'EOF'
<?php
use Symfony\Component\Console\Output\ConsoleOutput;
use Webkul\AllmakesPsp\Client\PspClient;
use Webkul\AllmakesPsp\Jobs\ScrapeAllmakesProductData;
use Webkul\AllmakesPsp\Settings\PspSettings;
use Webkul\Channel\Jobs\ShopifyBatchSyncJob;
use Webkul\ImageWatermark\Jobs\WatermarkProductJob;
use Webkul\ImageWatermark\Models\WatermarkConfig;
use Webkul\Imports\ImportManager;
use Webkul\Product\Models\Product;

$output = new ConsoleOutput();
$disk = config('filesystems.disks.r2') ? 'r2' : config('filesystems.default', 'public');

$ids = [];
foreach (file('/tmp/ghost-images-report.txt') as $line) {
    if (preg_match('/^Product (\d+) /', $line, $m)) {
        $ids[] = (int) $m[1];
    }
}
$output->writeln('Candidates: ' . count($ids) . ' (already-fixed ones are auto-skipped, safe to re-run)');

$before = Product::whereIn('id', $ids)->pluck('images', 'id')->map(fn ($imgs) => $imgs ?? [])->toArray();

$output->writeln('--- Phase 1: PSP re-pull ---');
(new ScrapeAllmakesProductData(includeImages: true, includeProductData: false, targetProductIds: $ids))
    ->handle(app(PspSettings::class), app(PspClient::class), app(ImportManager::class));

$output->writeln('--- Phase 2: reorder + drop ghosts ---');
$repaired = [];
$noImageFound = [];
foreach ($ids as $id) {
    $product = Product::find($id);
    if (! $product) { continue; }
    $after = $product->images ?? [];
    $new = array_values(array_diff($after, $before[$id] ?? []));
    $survivingOld = array_values(array_filter($before[$id] ?? [], fn ($p) => \Storage::disk($disk)->exists($p)));
    $final = array_values(array_unique(array_merge($new, $survivingOld)));
    if (empty($final)) { $noImageFound[] = $id; continue; }
    if ($final !== ($before[$id] ?? [])) { $product->images = $final; $product->save(); }
    $repaired[] = $id;
}

$output->writeln('--- Phase 3: watermark ---');
$byCompany = Product::whereIn('id', $repaired)->get(['id', 'sku', 'company_id'])->groupBy('company_id');
foreach ($byCompany as $companyId => $products) {
    $config = WatermarkConfig::where('company_id', $companyId)->where('is_enabled', true)->first();
    if (! $config) { continue; }
    foreach ($products as $p) {
        (new WatermarkProductJob($p->id, $config->id))->handle(app(\Webkul\ImageWatermark\Services\WatermarkProcessor::class));
    }
}

$output->writeln('--- Phase 4: Shopify push (Somerset4x4 only) ---');
$channelId = 12;
$mappedIds = \DB::table('channels_sku_mappings')->where('channel_id', $channelId)->whereIn('product_id', $repaired)
    ->where('is_active', true)->whereNotNull('shopify_variant_id')->pluck('product_id')->toArray();
if (! empty($mappedIds)) {
    $pricedIds = \DB::table('products_products as p')
        ->leftJoin('channels_product_prices as cp', fn ($j) => $j->on('cp.product_id', '=', 'p.id')->where('cp.channel_id', $channelId))
        ->whereIn('p.id', $mappedIds)->whereRaw('COALESCE(cp.price, p.price, 0) > 0')->pluck('p.id')->toArray();
    if (! empty($pricedIds)) {
        \DB::table('channel_listings')->where('channel_id', $channelId)->whereIn('product_id', $pricedIds)
            ->update(['status' => 'pending_sync', 'update_reason' => null, 'updated_at' => now()]);
        ShopifyBatchSyncJob::dispatch($pricedIds, $channelId, 'images_only')->onQueue('high');
    }
}

$output->writeln('Repaired: ' . count($repaired) . ' / No PSP image: ' . count($noImageFound));
EOF
php artisan tinker --execute="require '/tmp/repair-all.php';"
```

(Full version with progress output every 200 products lives in this session's history / can be regenerated — this is the condensed logic for reference.)

**Status as of 2026-09-28: single-product and 20-product batch both confirmed working; full remaining-catalog run pending.** Update this section with the final totals once it's been run.

## 7. A gotcha hit while running these on the server

`php artisan tinker` opens an interactive REPL (Psy Shell). Pasting a multi-line block where a line starts with `->` (a chained method call split across lines) breaks its parser (`unexpected T_OBJECT_OPERATOR`) — it doesn't buffer multi-line statements the way a real PHP file does. Likewise, pasting bash commands (`cat > file <<'EOF' ... EOF`) *into* the tinker prompt fails, since that's shell syntax, not PHP — tinker must be exited first (`exit`), then the heredoc run at the actual bash prompt, then `php artisan tinker --execute="require '/path/to/script.php';"` run separately to execute it non-interactively in one shot. All the scripts in this file follow that write-a-file-then-require-it pattern for exactly this reason.

---

## Related

See `purchase-orders.md` for the unrelated PO-name-uniqueness fix from the same day, which follows the same "R2 disk write / silent failure" theme in spirit (a check-then-insert race) but is a completely separate bug in a different part of the app.