# Shopify Listings — Bulk Sync Architecture & V2 Build Plan

## Overview

This document covers:
1. The bottleneck analysis of the current per-product sync approach (V1)
2. The Shopify Bulk Operations API plan for a future `ShopifyBatchSyncJob`
3. The Manage Listings V2 page — what's built, what's improved, what's shared
4. The implementation roadmap

---

## Part 1 — Current V1 Architecture & Bottleneck Analysis

### V1 Page: `ManageListings.php`

| File | Purpose |
|------|---------|
| `plugins/webkul/channels/src/Filament/Pages/ManageListings.php` | Per-channel listing management page (Livewire) |
| `plugins/webkul/channels/resources/views/filament/pages/manage-listings.blade.php` | V1 blade view |
| `plugins/webkul/channels/src/Jobs/SyncProductToChannels.php` | Per-product update job |
| `plugins/webkul/channels/src/Jobs/CreateProductInShopify.php` | Per-product create job |
| `plugins/webkul/channels/src/Services/ShopifyClient.php` | Shopify API wrapper |

### Per-product API call breakdown (V1 — SyncProductToChannels)

Each product sync makes approximately **8–12 Shopify API calls**:

| Call | Type | Purpose |
|------|------|---------|
| `getProductTags` | GQL | Fetch existing tags for merging |
| `setInventoryLevel` × 2 | REST POST | Set inventory at each warehouse location |
| `updateVariantPrice` | REST PUT | Update variant price |
| `updateInventoryItemCost` | REST PUT | Update cost price |
| `pushListingAttributes` | REST PUT | Title, description, vendor, handle |
| `setProductInStockMetafields` | GQL | In-stock metafield (batched 25/call) |
| `pushFitmentMetafield` (optional) | GQL | Vehicle fitment JSON metafield |
| `pushCategoryMetafields` (optional) | GQL | Category hierarchy metafields |
| `getInventoryItems` + `getInventoryLevelsForItems` | REST GET | Verify pushed quantities |

### Scale problem at 5,500 products

```
5,500 products × ~8 calls = ~44,000 Shopify API calls
Rate limit: 2 calls/sec standard
Time: ~6 hours for a full sync run
```

---

## Part 2 — Shopify Bulk Operations API Plan (Future ShopifyBatchSyncJob)

### What the Bulk Operations API does

Shopify's `bulkOperationRunMutation` lets you upload a JSONL file of mutations to S3, submit it in a single API call, and poll for completion. Shopify processes it server-side. One bulk operation can update thousands of products.

**Constraint**: Only one bulk operation can run at a time per shop. Must poll until COMPLETED/FAILED before starting another.

### Existing proof-of-concept

`plugins/webkul/channels/src/Console/Commands/BulkTagBackfillCommand.php` already implements the full bulk workflow:
- Build JSONL → `createStagedUpload()` → `bulkOperationRunMutation` → `pollUntilDone()`
- This is the foundation to extract into reusable `ShopifyClient` methods.

### API call comparison

| Operation | V1 (per-product) | V2 (bulk) |
|-----------|-----------------|-----------|
| Fetch product tags | 5,500 REST calls | 22 GQL calls (250/batch) |
| Update price + title + description + vendor + tags | ~5,500 REST calls | 1 bulk operation (JSONL) |
| Set inventory quantities | 11,000 REST calls | ~44 GQL calls (250 items/call) |
| Set metafields | ~1,650 GQL calls | 1 bulk operation (JSONL) |
| Verify inventory pull-back | ~16,500 REST calls | 0 (removed — trust pushed values) |
| **Total** | **~44,000 calls (~6 hours)** | **~70 calls (~20–30 minutes)** |

### Phase 1 — `inventorySetQuantities` (quick win, no bulk needed)

Replace individual `setInventoryLevel` REST calls with the `inventorySetQuantities` GraphQL mutation that accepts **up to 250 inventory level updates per call**.

- API: `2023-10+` (compatible with current `2024-01` client)
- Impact: 11,000 REST calls → ~44 GQL calls

New method to add to `ShopifyClient`:
```php
public function inventorySetQuantities(array $quantities): array
// $quantities: [['inventoryItemId' => '...', 'locationId' => '...', 'quantity' => 5], ...]
// Batches in groups of 250, returns ['updated' => n, 'errors' => [...]]
```

### Phase 2 — Bulk product update JSONL

For `productUpdate` mutations via `bulkOperationRunMutation`:

**JSONL line format (one per product):**
```json
{"input": {"id": "gid://shopify/Product/123", "title": "...", "bodyHtml": "...", "vendor": "...", "tags": ["tag1","tag2"], "variants": [{"id": "gid://shopify/ProductVariant/456", "price": "29.99"}]}}
```

**ShopifyClient methods to extract from BulkTagBackfillCommand:**
```php
// Upload JSONL to Shopify's staged S3 target
public function stagedUpload(string $jsonl, string $filename): ?string

// Submit bulk mutation operation, returns operation ID
public function runBulkMutation(string $mutation, string $stagedPath): ?string

// Poll currentBulkOperation until COMPLETED/FAILED/CANCELED
public function pollBulkOperation(): array  // returns ['status' => 'COMPLETED', 'url' => '...']
```

### Phase 3 — Bulk product create JSONL

`productCreate` mutations also work inside `bulkOperationRunMutation`. The result JSONL contains the new Shopify product/variant IDs.

**Result JSONL correlation**: Use the variant SKU as the natural key to map result IDs back to Cydekick product IDs.

```json
// Request:
{"input": {"title": "...", "variants": [{"sku": "ABC123", "price": "29.99"}]}}

// Result JSONL:
{"id": "gid://shopify/Product/789", "variants": [{"id": "gid://shopify/ProductVariant/999", "sku": "ABC123"}]}
```

After bulk create completes: parse result JSONL, look up each SKU in `products_products`, write `channels_sku_mappings`, mark as `needs_update`, dispatch `SyncProductToChannels` for inventory (which can't be set in bulk create).

### Phase 4 — ShopifyBatchSyncJob architecture (6 phases)

```
Phase 1: Pre-pass — ensure vehicle collections exist (fast DB + minimal API)
Phase 2: Fetch all product tags in 250-batch GQL calls
Phase 3: Build update JSONL (merge tags, build payload per product)
Phase 4: Staged upload → bulkOperationRunMutation → poll until done
Phase 5: Parse result JSONL for errors → write to channels_bulk_sync_errors
Phase 6: inventorySetQuantities (batched 250/call)
```

**Worker behaviour during polling**: Job sleeps 15s between polls. During this time the worker is parked but still alive. For long bulk operations (20+ minutes), consider releasing the job back to the queue with a state flag to avoid worker timeouts.

**New table: `channels_bulk_sync_errors`**

```sql
CREATE TABLE channels_bulk_sync_errors (
  id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
  channel_id BIGINT UNSIGNED NOT NULL,
  batch_id VARCHAR(255) NOT NULL,
  product_id BIGINT UNSIGNED NULL,
  shopify_product_id VARCHAR(100) NULL,
  error_message TEXT,
  raw_result JSON,
  created_at TIMESTAMP
);
```

### Implementation roadmap

| Sprint | Work | Est. |
|--------|------|------|
| Sprint 1 | Extract `stagedUpload()`, `runBulkMutation()`, `pollBulkOperation()` from BulkTagBackfillCommand into ShopifyClient | 1–2 hrs |
| Sprint 2 | Add `inventorySetQuantities()` to ShopifyClient + quick-win integration into SyncProductToChannels | 2–3 hrs |
| Sprint 3 | Build `ShopifyBatchSyncJob` (update mode only) — phases 1–5 | 1 day |
| Sprint 4 | Add create mode to ShopifyBatchSyncJob + result JSONL parsing + SKU correlation | 1 day |
| Sprint 5 | Wire V2 "Bulk Sync" action to ShopifyBatchSyncJob, remove per-product jobs from V2 | Half day |

---

## Part 3 — Manage Listings V2

### What V2 is

A separate Filament page (`/admin/channel-listings-v2`) that runs **alongside** V1 for beta testing. Goal: once V2 is stable, make V1 redundant.

### Page files

| File | Purpose |
|------|---------|
| `plugins/webkul/channels/src/Filament/Pages/ManageListingsV2.php` | V2 page class |
| `plugins/webkul/channels/resources/views/filament/pages/manage-listings-v2.blade.php` | V2 blade view |

### Improvements over V1

| Feature | V1 | V2 |
|---------|----|----|
| **Update reason** | Not shown | Displayed below "Needs Update" badge when available |
| **Tab labels** | "Not Listed" / "Pending Sync" | "Not Linked" / "Not Synced" (clearer distinction) |
| **Bulk action bar** | Inline within table | Fixed sticky bar at bottom of screen |
| **Needs Update tab** | No reason column | "Reason" column shows why update is needed |
| **Status badges** | Static text | `pending_sync` shows animated spinner indicator |
| **Mark-all reason** | Sets status only | Also sets `update_reason = 'Manual mark-all'` |
| **DB schema** | No update_reason | `channel_listings.update_reason` varchar(200) column |

### What V2 shares with V1

| Shared | Details |
|--------|---------|
| `HasChannelListingModal` trait | Edit modal for listing attributes — unchanged |
| `getAllChannels()` | Channel overview query — identical |
| `getChannelStats()` | Landing page stats — identical |
| `getActiveChannel()` | Channel lookup — identical |
| `linkVariant()` / `unlinkProduct()` | Link/unlink logic — identical |
| `autoMapBySkу()` | Auto-link by SKU — identical |
| `syncShopifyVariants()` | Pull Shopify catalogue — identical |
| `syncVehicleCollections()` | Collections sync — identical |
| `auditOrphanedCollections()` | Orphan audit — identical |
| `bulkPush()` / `pushNow()` | Sync actions use same jobs (V1 jobs) for now |
| `bulkCreate()` / `createInShopify()` | Create actions use same jobs (V1 jobs) for now |
| `cancelPendingSync()` | Cancel queued jobs — identical |
| `channel_listings` table | Same DB table |
| `channels_sku_mappings` table | Same DB table |
| `JobLogger` | Same console logging |

### Future: when ShopifyBatchSyncJob is built

V2's `bulkPush()` and `bulkCreate()` will be swapped to dispatch `ShopifyBatchSyncJob` instead of per-product jobs. V1 stays on per-product jobs — this is the key divergence point.

### update_reason tracking

The `update_reason` column is populated by:

| Source | Reason text |
|--------|-------------|
| V2 `markAllListedAsNeedsUpdate()` | `"Manual mark-all"` |
| V2 `bulkMarkNeedsUpdate()` (new) | `"Manual — selected"` |
| Pricing plugin (future wiring) | `"Price changed"` |
| Product observer (future wiring) | `"Product data changed"` |
| Stock change observer (future wiring) | `"Stock changed"` |

When `update_reason` is NULL (e.g. legacy rows set before V2), the UI shows "—".

### Tab structure

V2 uses the same status filter logic as V1 but with renamed labels:

| Filter key | V1 label | V2 label | What it shows |
|------------|----------|----------|---------------|
| `listed` | Listed | Listed | `cl.status = 'listed'` |
| `needs_update` | Needs Update | Needs Update | `cl.status = 'needs_update'` |
| `not_synced` | Pending Sync | Not Synced | Linked (has SKU mapping) but not pushed |
| `not_mapped` | Not Listed | Not Linked | No active SKU mapping |
| `error` | Errors | Errors | `cl.status = 'error'` |

### V2 "Needs Update" tab columns

Additional "Reason" column shown only when `statusFilter === 'needs_update'`:

```
| ☐ | Img | Product | Shopify SKU | Price | Status | Reason | Last Synced | Actions |
```

The "Reason" column pulls from `cl.update_reason`. Null = "—".

---

## Part 4 — Navigation

V2 is added as a separate nav item in `ChannelPlugin.php`:

```php
NavigationItem::make('channels-listings-v2')
    ->label('Listings V2 (Beta)')
    ->url(fn () => ManageListingsV2::getUrl())
    ->group(__('admin.navigation.channel'))
    ->icon('heroicon-o-list-bullet')
    ->sort(3)
    ->isActiveWhen(fn () => request()->routeIs('filament.admin.pages.channel-listings-v2'))
    ->visible(fn () => ManageListingsV2::canAccess()),
```

Once V2 is stable and replaces V1:
1. Remove V1 nav item
2. Change V2 slug to `channel-listings`
3. Remove V1 page class and blade

---

## Part 5 — Key constraints & gotchas

1. **`ShouldBeUnique` cache locks** — `SyncProductToChannels` uses `uniqueFor=60` uniqueness via Laravel cache. Before re-dispatching, call `Cache::lock(...)->forceRelease()`. V2 inherits V1's `bulkPush()` which already does this.

2. **Shopify bulk operation — one at a time** — only one `bulkOperationRunMutation` can run per shop simultaneously. The poll loop must complete (COMPLETED/FAILED) before the next bulk op starts. Plan job concurrency accordingly.

3. **`inventorySetQuantities` API** — requires API version `2023-10` or later. Current client uses `2024-01` — compatible.

4. **`pending_sync` status reset** — `cancelPendingSync()` must be updated to handle V2 batch jobs once `ShopifyBatchSyncJob` exists.

5. **Result JSONL parsing** — Shopify's bulk operation result is a JSONL file at `currentBulkOperation.url` (an S3 pre-signed URL). Must download and parse line-by-line. Each line may be a success or `{"errors": [...]}` entry.

6. **Worker timeout** — if `ShopifyBatchSyncJob` polls for 20–30 minutes, the PHP queue worker may timeout. Set queue timeout to ≥1800s for the high queue, or use `releaseAfterCommit` + state flag pattern.