# Shopify Listings — Bulk Architecture & Implementation Record

> **Scope**: This document covers the Shopify channel implementation. It is intended as the base reference when adding future channels (BigCommerce, eBay, etc.) — each will have its own equivalent file.

---

## Shopify API Reference

- **Admin GraphQL API**: https://shopify.dev/docs/api/admin-graphql/latest/queries/products
- **API version in use**: `2024-01` (set in `ShopifyClient.php`)
- **Bulk Operations guide**: https://shopify.dev/docs/api/usage/bulk-operations/imports

---

## Part 1 — Why Bulk? (Original Bottleneck Analysis)

### V1 per-product sync (`SyncProductToChannels`)

Each product sync made approximately **8–12 Shopify API calls**:

| Call | Type | Purpose |
|------|------|---------|
| `getProductTags` | GQL | Fetch existing tags for merging |
| `setInventoryLevel` × 2 | REST POST | Set inventory at each warehouse location |
| `updateVariantPrice` | REST PUT | Update variant price |
| `updateInventoryItemCost` | REST PUT | Update cost price |
| `pushListingAttributes` | REST PUT | Title, description, vendor, handle, images |
| `setProductInStockMetafields` | GQL | In-stock metafield |
| `pushFitmentMetafield` (optional) | GQL | Vehicle fitment JSON metafield |

### Scale problem at 5,500 products

```
5,500 products × ~8 calls = ~44,000 Shopify API calls
Rate limit: 2 calls/sec standard
Time: ~6 hours for a full sync run
```

---

## Part 2 — What Was Built

### Key files

| File | Purpose |
|------|---------|
| `plugins/webkul/channels/src/Jobs/ShopifyBatchSyncJob.php` | Bulk update existing listed products |
| `plugins/webkul/channels/src/Jobs/ShopifyBatchCreateJob.php` | Bulk create new products in Shopify |
| `plugins/webkul/channels/src/Jobs/CreateProductInShopify.php` | Single-product create (still used for row-level action) |
| `plugins/webkul/channels/src/Jobs/SyncProductToChannels.php` | Single-product update (still used in V1) |
| `plugins/webkul/channels/src/Services/ShopifyClient.php` | Shopify API wrapper — contains all bulk methods |
| `plugins/webkul/channels/src/Filament/Pages/ManageListingsV2.php` | V2 page — dispatches both bulk jobs |

---

## Part 3 — `ShopifyBatchSyncJob` (Bulk Update)

Updates products that are already listed (have a Shopify product ID in `channels_sku_mappings`).

### Triggered from

`ManageListingsV2::bulkPush()` — when user selects listed/needs-update products and clicks Sync.

### API call comparison (50 products)

| Old | New |
|-----|-----|
| ~400 REST calls (~200s) | ~6 API calls total + 1 REST per product for publish |

### Phases

| Phase | What it does |
|-------|-------------|
| **1 — Pre-load** | Bulk DB queries: SKU mappings, channel prices, location mappings, warehouse quantities, product data, listing attribute overrides. Also loads `$splitSellWarehouseIds` (location mappings where `is_split_sell_source = true`) and `$supplierMinQtyMap` (default supplier `min_qty` per product, if any split-sell warehouses are configured). |
| **2 — Build JSONL** | One `productUpdate` line per product. Includes: title, `handle`, `redirectNewHandle: true` (see handle note below), descriptionHtml, vendor, tags (via `CategoryTagMapper`), variant price (GID), metafields (in_stock boolean + category). Also pre-fetches existing Shopify image GIDs in one batch GraphQL call (250 products/call). **Images are tracked for Phase 3.5 but NOT included in JSONL** (see image limitation below). |
| **3 — Bulk op** | `createStagedUpload()` → `runBulkMutation()` → `pollBulkOperation()` (polls every 10s, max 1200s) |
| **3.5 — Images via REST (dispatched)** | In `all` mode: products with image changes are chunked into 50-product `images_only` jobs dispatched to `default` queue after Phase 6 completes (see below). In `images_only` mode: runs inline — one REST call per product with the full images array. |
| **3.7 — Publish via bulk GraphQL** | `bulkPublishProducts()` — batched `publishablePublish` mutations (10 products/call). Ensures products appear in all Shopify sales channels. |
| **4 — Inventory** | `inventorySetQuantities` GraphQL mutation, batched 250 items/call. Replaces N×2 REST calls. |
| **5 — Fitment metafields** | REST per product (only for products with `vehicle_fitment` data) |
| **6 — Bulk DB writes** | `channel_listings.status = 'listed'` + `SkuSyncStatus::updateOrCreate()` per mapping/location. Per-product console log entry emitted via `JobLogger::ok()`. **Runs before image dispatch** so products are marked `listed` even if image sync fails or times out. |

#### Image dispatch in `all` mode

After Phase 6, `ShopifyBatchSyncJob` dispatches one `ShopifyBatchSyncJob(productIds, channelId, 'images_only')` per 50-product chunk of the products that have image changes. Each `images_only` job re-fetches the current Shopify image state for its 50 products and runs the REST image update inline.

This avoids the timeout that occurred when ~999 REST image calls were made serially in the main job (taking 500+ seconds at the Shopify REST rate limit). The main job now completes in a few minutes; image sync jobs run in the background queue in parallel.

**Sync mode values:**
- `'all'` (default) — data sync + dispatches image jobs
- `'data_only'` — skips all image handling
- `'images_only'` — skips data sync, runs image REST inline

### Image limitation — why images aren't in the JSONL

Shopify's GraphQL `productUpdate` mutation (including bulk ops) deprecated `ImageInput.src` in API version ~2022-10. Sending `{src: url}` in the JSONL is silently ignored. Only `{id: gid}` references work (keeping/reordering existing images). New image uploads must go through REST — hence Phase 3.5.

The pre-fetch in Phase 2 (`getProductImagesByShopifyIds`) batch-fetches all current Shopify image GIDs (one GraphQL call per 250 products) so Phase 3.5 can build the correct replacement array without additional API calls.

### Image differential sync — why the console shows fewer products than the batch size

Phase 3.5 only updates products whose Cydekick image set **differs** from what Shopify already has. The comparison is by bare filename (CDN query-strings stripped):

```php
// ShopifyClient::getProductImagesByShopifyIds() — filename extraction
$filename = strtok(basename((string) ($img['url'] ?? '')), '?');
// "https://cdn.shopify.com/.../abc123.jpg?v=12345" → "abc123.jpg"

// ShopifyBatchSyncJob Phase 2 — skip if sets match
$shopifyFilenames = array_keys($existingByFilename);
if (array_diff($filenames, $shopifyFilenames) || array_diff($shopifyFilenames, $filenames)) {
    $pendingImageUpdates[$shopifyProductId] = [...];
}
```

If Shopify already holds the exact same filenames as Cydekick, the product is skipped — no REST call needed. The count logged to the console (`syncing images (N products)`) reflects only the products with a filename mismatch.

**Example**: Syncing 100 products and seeing "syncing images (25 products)" means 75 products already have correct, up-to-date images on Shopify. Only 25 have differences (new images added, images removed, or reordered since the last sync).

This is intentional — skipping no-op REST calls saves ~75 API requests per batch. To force all images to re-sync regardless, you would remove the `array_diff` guard in `ShopifyBatchSyncJob` Phase 2, but this is not recommended.

### Worker timeout

The job sleeps during bulk operation polling. The Shopify bulk op itself is handled server-side, so the job just polls. With image sync now dispatched as separate jobs, the main job completes much faster (typically under 10 minutes for a 1000-product batch). Start the queue worker with:
```bash
php -d max_execution_time=0 artisan queue:work --timeout=0 --queue=high,default
```

On the production server, `supervisor_scheduler.md` documents `stopwaitsecs=120` — this gives the worker 2 minutes to gracefully finish after a restart signal before being hard-killed.

---

## Part 4 — `ShopifyBatchCreateJob` (Bulk Create)

Creates brand new products in Shopify for products that have no `channels_sku_mappings` entry. Uses Shopify's `productSet` mutation (the only create-capable mutation whitelisted for `bulkOperationRunMutation` — `productCreate` is NOT supported in bulk ops).

### Triggered from

- `ManageListingsV2::bulkCreate()` — selected products in "Not Linked" tab
- `ManageListingsV2::createAllFiltered()` — all filtered products in "Not Linked" tab

`CreateProductInShopify` is still used for the single-row "create" action (row-level button).

### API call comparison (100 products with images)

| Old (per-job REST) | New (bulk) |
|---|---|
| ~300+ REST calls across 100 jobs, 3–5 min | 1 bulk op + ~100 image REST calls + 1 REST per product for publish, ~90s |

### Phases

| Phase | What it does |
|-------|-------------|
| **1 — Pre-load** | Bulk DB: product data (incl. SKU), channel prices, listing attribute overrides, location mappings, warehouse quantities. Builds `$skuToProductId [sku => productId]` for result correlation. |
| **2 — Build JSONL** | One `productSet` line per product — no `id` field (Shopify creates). Includes: title, `handle` (see handle note below), descriptionHtml, vendor, `status: ACTIVE`, tags, `productOptions`, `variants: [{sku, price, inventoryPolicy: DENY, inventoryItem: {tracked: true}, optionValues, inventoryQuantities}]`, metafields. No images in JSONL (same limitation as updates). Skips products with no SKU or zero price. |
| **3 — Bulk op** | Same flow as update: `createStagedUpload()` → `runBulkMutation()` → `pollBulkOperation()`. Uses `productSet` mutation with variant+inventoryItem output fields so the result JSONL contains the new IDs. |
| **4 — Parse results + write mappings** | Downloads result JSONL via `downloadBulkResultLines($result['url'])`. Per line: extracts `shopifyProductId`, `shopifyVariantId`, `inventoryItemId` per SKU, correlates via `$skuToProductId` back to local product IDs, inserts `channels_sku_mappings`. Collects inventory rows. |
| **4.2 — Publish via REST** | `publishProduct()` called for every new product — `PUT /products/{id}.json` with `published: true`. Required because `productSet` creates products Active but NOT published to any sales channel (unlike REST `POST /products.json` which defaults to published). |
| **4.5 — Images via REST** | All images are new (product just created). Builds `[{src: r2Url, position: n}]` per product, calls `updateProductImages()`. No GID pre-fetch needed. |
| **5 — Inventory** | `inventorySetQuantities` with the inventory item IDs from Phase 4, batched 250/call. |
| **6 — Fitment metafields** | REST per product with fitment data. |
| **7 — Final DB writes** | `channel_listings.status = 'listed'` for all successfully created products. Per-product console log entry emitted via `JobLogger::ok()` with inventory/price/images detail. |

### Handle construction

Both jobs explicitly set `handle` on every product to ensure the SKU appears in the Shopify URL:

```php
$titlePart = trim(preg_replace('/^' . preg_quote($sku, '/') . '\s*/i', '', $displayTitle));
$handle    = Str::slug($sku . ' ' . $titlePart);
// SKU "02AJ82738E" + title "ENGINE-STRIPPED-REMAN" → "02aj82738e-engine-stripped-reman"
```

The regex strips the SKU from the start of the title first to avoid doubling it when `$attrs['title']` already contains the full "02AJ82738E ENGINE-STRIPPED-REMAN" string.

**Sync job** sends `handle` + `redirectNewHandle: true` in the JSONL — Shopify creates a 301 redirect from the old URL to the new one, preserving any existing inbound links:
```json
{"input": {"id": "gid://shopify/Product/123", "handle": "02aj82738e-engine-stripped-reman", "redirectNewHandle": true}}
```

**Create job** sends `handle` only in the JSONL — `redirectNewHandle` is not needed since new products have no previous URL to redirect from.

**Why this is needed**: Shopify auto-generates a handle from the initial product title and never updates it automatically, even when the title is later changed via sync. If not set explicitly, products with a short title (e.g. "ENGINE-SHORT") get handle `engine-short`, losing the SKU from the URL entirely.

### Result JSONL structure

Shopify returns a JSONL file at `currentBulkOperation.url` (an S3 pre-signed URL). Each line is a decoded mutation result. The **actual confirmed format** (from logs) is:

```json
{"data": {"productSet": {"product": {"id": "gid://shopify/Product/123456", "variants": {"nodes": [{"id": "gid://shopify/ProductVariant/789", "sku": "SKU123", "inventoryItem": {"id": "gid://shopify/InventoryItem/456"}}]}}, "userErrors": []}}, "__lineNumber": 0}
```

The parsing code handles both the nested `data.productSet` format and a flat fallback:
```php
$psResult = $line['data']['productSet'] ?? $line['productSet'] ?? null;
```

Numeric IDs are extracted with `last(explode('/', $gid))`.

---

## Part 5 — ShopifyClient Methods Added

All methods live in `plugins/webkul/channels/src/Services/ShopifyClient.php`.

| Method | Purpose |
|--------|---------|
| `getProductImagesByShopifyIds(array $ids): array` | Batch-fetch image GIDs for existing products. Returns `[shopify_product_id => [filename => gid]]`. Used by `ShopifyBatchSyncJob` Phase 2. |
| `createStagedUpload(string $jsonl, string $filename): ?string` | Upload JSONL to Shopify's staged S3 target. Returns the `stagedUploadPath` key. |
| `runBulkMutation(string $mutationStr, string $stagedPath): ?string` | Submit `bulkOperationRunMutation`. Returns bulk operation GID. |
| `pollBulkOperation(?callable $onTick, int $maxWait, int $interval): array` | Poll `currentBulkOperation` until terminal state. Returns `['status', 'objectCount', 'url']`. The `url` is the result JSONL download URL. |
| `downloadBulkResultLines(string $url): array` | Download and parse the bulk op result JSONL. Returns array of decoded JSON objects. Used by `ShopifyBatchCreateJob` Phase 4. |
| `inventorySetQuantities(array $quantities): array` | GraphQL `inventorySetQuantities` mutation, batched 250/call. Requires `ignoreCompareQuantity: true` in the mutation input or Shopify rejects the entire call. Replaces per-product REST inventory calls. |
| `updateProductImages(string $shopifyProductId, array $resolvedImages): bool` | REST `PUT /products/{id}.json` with images array. Used by both bulk jobs for image management. Existing images: `{id: numericId, position: n}`. New images: `{src: url, position: n}`. Shopify deletes absent IDs. |
| `publishProduct(string $shopifyProductId): bool` | REST `PUT /products/{id}.json` with `published: true`. Used by both bulk jobs (Phase 4.2 in create, Phase 3.7 in sync) to publish to all sales channels. Safe to call on already-published products. |

---

## Part 6 — Key Constraints & Gotchas

1. **One bulk op at a time per shop** — `bulkOperationRunMutation` allows only one concurrent operation. The poll loop must complete (COMPLETED/FAILED) before starting another. Dispatching two `ShopifyBatchSyncJob` or `ShopifyBatchCreateJob` jobs simultaneously will cause the second to fail.

1a. **Never include `variants` in the `productUpdate` JSONL** — On Shopify's new product model (2024+), including a `variants` array in `productUpdate` causes Shopify to silently reject the **entire** mutation line. This means title, description, vendor, tags, and metafields all fail to apply — not just the variant fields. The result JSONL (which `ShopifyBatchSyncJob` doesn't download) contains a userError per line, but `objectCount` is still > 0 and status is COMPLETED, so the job reports success. All variant fields (price, cost, barcode, weight) must go exclusively through `productVariantsBulkUpdate` (Phase 3.6b).

2. **`productSet` vs `productCreate` vs `productUpdate` in bulk ops**
   - `productCreate` — NOT supported in `bulkOperationRunMutation`
   - `productUpdate` — supported, but only for updating existing products (requires `id`)
   - `productSet` — supported, creates if no `id`, updates/upserts if `id` is provided

3. **Images not supported in GraphQL productUpdate/productSet JSONL** — `ImageInput.src` was deprecated in Shopify API ~2022-10. New images via `src` in the JSONL are silently ignored. Image management always goes through REST (`PUT /products/{id}.json`).

4. **Result JSONL only available for creates** — `ShopifyBatchSyncJob` ignores `result['url']` because it already has Shopify product IDs. `ShopifyBatchCreateJob` must download and parse it to get the new IDs.

5. **`inventorySetQuantities` requires `ignoreCompareQuantity: true`** — omitting this causes the entire mutation to be rejected with `"The compareQuantity argument must be given to each quantity or ignored using ignoreCompareQuantity"`. Without it, inventory is silently not set.

6. **`productSet` does NOT publish to sales channels** — creating a product via `productSet` (GraphQL bulk op) leaves it Active but unpublished on Online Store, Shop, etc. The REST `POST /products.json` used by `CreateProductInShopify` defaults to `published: true`. Both bulk jobs now call `publishProduct()` after the bulk op to fill this gap. Symptom: product appears in Shopify admin as Active but "Online Store" toggle is OFF.

7. **Handle must be set explicitly in the JSONL** — Shopify auto-generates a handle from the initial product title and never updates it automatically when the title changes. The sync job sends `handle` + `redirectNewHandle: true` in the `productUpdate` JSONL so Shopify updates the URL and creates a 301 redirect. The create job sends `handle` in the `productSet` JSONL (no `redirectNewHandle` needed — new products have no previous URL). Without this, products get a handle based on their short title only (e.g. `engine-short` instead of `02aj89994e-engine-short`).

8. **Worker timeout** — polling jobs sleep inside the worker process. Use `--timeout=600` (or `max_execution_time=0`) on the queue worker for the `high` queue.

9. **Queue worker caches compiled classes** — code changes are NOT picked up by a running queue worker. Always restart the worker after editing job or service classes: stop the process and re-run `php -d max_execution_time=0 artisan queue:work --queue=high,default`.

10. **Console logging** — both bulk jobs use `JobLogger` directly and are added to `$skipClasses` in `ConsoleServiceProvider` to prevent double-logging. Each job emits a per-product `JobLogger::ok()` entry with a detail block (inventory per location, price, images, tags, fitment, metafields).

11. **`CreateProductInShopify` still used** — the single-row "create" action in V2 still dispatches `CreateProductInShopify` synchronously (`dispatchSync`). Only the bulk actions use `ShopifyBatchCreateJob`.

12. **Split-sell min/step quantity** — if any `channels_location_mappings` row for the channel has `is_split_sell_source = true`, Phase 3.6b injects `custom.step_quantity` and `custom.min_quantity` variant metafields on every sync. Logic:
    - Any split-sell warehouse has qty ≥ 1 → push `1` (sell individually, we have own stock to split)
    - All split-sell warehouses have qty = 0 → push the default supplier's `min_qty` from `products_product_suppliers` (e.g. 10 for a product that must be bought in packs of 10 from Allmakes)
    - Falls back to `1` if no supplier row exists
    - The skip guard in Phase 3.6b is widened to always run when split-sell is configured, even when price/cost/barcode/weight are all empty
    - Requires Shopify metafield definitions: namespace `custom`, keys `step_quantity` and `min_quantity`, type `Integer` (on Product variants)

---

## Part 7 — Manage Listings V2 Dispatch Map

| UI action | Method | Job dispatched |
|-----------|--------|---------------|
| Row "Create" button | `createInShopify()` | `CreateProductInShopify::dispatchSync()` |
| Bulk "Create" (selected) | `bulkCreate()` | `ShopifyBatchCreateJob::dispatch()->onQueue('high')` |
| "Create All Filtered" | `createAllFiltered()` | `ShopifyBatchCreateJob::dispatch()->onQueue('high')` |
| Bulk "Sync" (selected) | `bulkPush()` | `ShopifyBatchSyncJob::dispatch()->onQueue('high')` |
| "Sync All Filtered" | `pushAllFiltered()` | `ShopifyBatchSyncJob::dispatch()->onQueue('high')` |
| Row "Sync Now" button | `pushNow()` | `SyncProductToChannels::dispatch()` |