# Supervisor & Laravel Scheduler — Issues and Fixes

## Overview
Laravel's task scheduler requires a long-running process to trigger scheduled commands. This project uses **Supervisord** to manage that process alongside the queue worker.

---

## Processes

| Program | Command | Purpose |
|---------|---------|---------|
| `cydekick-worker` | `queue:work --queue=high,default` | Processes queued jobs — checks `high` queue first (admin-triggered jobs), then `default` (bulk syncs) |
| `cydekick-scheduler` | `schedule:work` | Triggers scheduled tasks every minute |

Config file: `/etc/supervisor/conf.d/cydekick-worker.conf`

---

## Queue Priority — `high` vs `default`

The worker listens on `--queue=high,default`. Laravel checks `high` first after every job finishes, so a job on `high` jumps the entire `default` backlog and runs on the next available gap (< 1 second wait).

### Why this matters

During a mass product sync there can be 15,000+ `SyncProductToChannels` jobs in the `default` queue. Without priority, clicking "Sync Collections & Menus" would queue behind all of them and wait hours. With priority it runs within seconds.

### Queue assignments

| Queue | Jobs | Reason |
|-------|------|--------|
| `high` | `SyncShopifyCollectionsJob` | Admin button — Collections & Menus |
| `high` | `SyncShopifyOrders` | Admin button — sync orders |
| `high` | `SyncShopifyVariants` | Admin button — Sync Shopify Catalogue |
| `high` | `SyncSkuInventoryStatus` | Admin button — SKU inventory check |
| `default` | `SyncProductToChannels` | Bulk — dispatched in thousands for mass sync |
| `default` | `CreateProductInShopify` | Bulk — dispatched in thousands for new listings |
| `default` | `FetchShopifyProductsJob` | Import pipeline |
| `default` | `ImportShopifyProductsJob` | Import pipeline |

The queue is set on the job class itself via `public string $queue = 'high';` so the priority applies regardless of where the job is dispatched from.

### Adding a new job to `high`

Add one property to the job class:

```php
public string $queue = 'high';
```

Use `high` for: admin-triggered one-off operations.  
Use `default` (or omit) for: anything dispatched in a loop for bulk processing.

### Checking queue depth per queue

```bash
# Count jobs per queue in the database
php artisan tinker --execute="
    DB::table('jobs')->select('queue', DB::raw('COUNT(*) as total'))->groupBy('queue')->get()->each(fn(\$r) => print(\$r->queue.': '.\$r->total.PHP_EOL));
"
```

---

## Local XAMPP (no Supervisor)

On local there is no Supervisor — just run the worker manually in a terminal:

```bash
php -d max_execution_time=0 artisan queue:work --queue=high,default
```

- `max_execution_time=0` prevents PHP from killing long-running jobs mid-flight
- `Ctrl+C` to stop the worker
- If you change the `--queue` flag, stop and restart — the worker loads it once at startup

---

## Issues Encountered

### 1. Scheduler not set up at all
**Symptom:** Scheduled tasks (e.g. Allmakes PSP auto-sync) never ran.

**Cause:** Only `cydekick-worker` existed. No scheduler process had been added to supervisord.

**Fix:** Added `[program:cydekick-scheduler]` block to the conf file and reloaded supervisord.

---

### 2. "Could not open input file: artisan" in scheduler log
**Symptom:** `tail -f /var/www/html/Cydekick/storage/logs/scheduler.log` showed nothing but:
```
Could not open input file: artisan
Could not open input file: artisan
...
```

**Cause (1):** `directory=` was missing from the supervisord config. Without it, supervisord starts the process from `/` (root), so PHP couldn't find `artisan` as a relative path.

**Cause (2):** The command used `php` instead of the full path `/usr/bin/php`. Supervisord runs with a minimal environment and doesn't inherit the shell PATH, so `php` was not found correctly.

**Fix:** Added both `directory=` and used full PHP path in the config:
```ini
[program:cydekick-scheduler]
process_name=%(program_name)s
command=/usr/bin/php /var/www/html/Cydekick/artisan schedule:work
directory=/var/www/html/Cydekick
autostart=true
autorestart=true
user=www-data
redirect_stderr=true
stdout_logfile=/var/www/html/Cydekick/storage/logs/scheduler.log
stopwaitsecs=3600
```

**Verify it's working:**
```bash
tail -f /var/www/html/Cydekick/storage/logs/scheduler.log
# Should show every minute:
# INFO  No scheduled commands are ready to run.
```

---

### 3. Schedule fired at wrong time (timezone mismatch)
**Symptom:** Schedule set to `dailyAt('18:06')` never fired at 18:06 BST.

**Cause:** Server was running in UTC. `18:06` in the schedule meant 18:06 UTC = 19:06 BST.

**Fix (step 1):** Changed server timezone to Europe/London:
```bash
sudo timedatectl set-timezone Europe/London
```

**Fix (step 2):** Laravel has its own timezone setting independent of the system clock. Added to `/var/www/html/Cydekick/.env`:
```
APP_TIMEZONE=Europe/London
```

After this, `dailyAt('18:06')` reliably means 18:06 BST, and automatically adjusts for GMT in winter.

---

### 4. "Sync Collections & Menus" never ran on server (missing `--queue=high`)
**Symptom:** Clicking "Sync Collections & Menus" on the server showed the notification but nothing appeared in the console bar. The job was never processed. The button had worked fine locally.

**Cause:** The server's supervisord config was deployed without a `--queue=` argument, so the worker defaulted to processing only the `default` queue. `SyncShopifyCollectionsJob` dispatches to `high` via `$this->onQueue('high')`, so the job sat in the `jobs` table indefinitely.

**Diagnosis:** `cat /etc/supervisor/conf.d/cydekick-worker.conf` showed `queue:work --sleep=3` with no `--queue=` flag.

**Fix:** Added `--queue=high,default` to the worker command and reloaded:
```bash
sudo supervisorctl reread && sudo supervisorctl update && sudo supervisorctl restart cydekick-worker:*
```

**Note:** Also clear the cache lock if the button was clicked before the fix (the lock prevents re-dispatch):
```bash
php artisan tinker --execute="Cache::forget('shopify_collections_syncing_8');"
```
Replace `8` with the actual channel ID.

---

### 5. Schedule changes not picked up
**Symptom:** Changed `routes/console.php` but scheduler kept running old schedule.

**Cause:** `schedule:work` is a long-running daemon. It loads the schedule once at startup and never re-reads the file.

**Fix:** After every schedule change, restart the scheduler:
```bash
php artisan optimize:clear && sudo supervisorctl restart cydekick-scheduler
```

---

### 6. `ShopifyBatchSyncJob` killed mid-flight (stopwaitsecs too short)
**Symptom:** After a deployment or `sudo supervisorctl restart cydekick-worker:*`, in-progress jobs are aborted. With `stopwaitsecs=30`, a running bulk sync job is SIGKILL'd 30 seconds after Supervisor sends SIGTERM. This prevents Phase 6 (the DB writes that mark products as `listed`) from completing, leaving the channel listed count at 0 even after a full sync.

**Cause:** `stopwaitsecs=30` is too short for long-running jobs like `ShopifyBatchSyncJob` (timeout up to 3600 seconds).

**Fix:** Increased `stopwaitsecs` to 120 in the worker program block. This gives the worker 2 minutes to finish the current job step gracefully (or at least reach Phase 6 DB writes) before being hard-killed.

```ini
stopwaitsecs=120
```

**Note:** Even 120s won't save a job mid-bulk-op (which can run for 10+ minutes). The real fix is `ShopifyBatchSyncJob` now moves Phase 6 DB writes *before* the image sync, and dispatches image sync as separate `images_only` jobs so the main job completes quickly.

---

### 7. Long-running `default` job blocks `high` queue (e.g. PSP scrape vs eBay sync)

**Symptom:** `SyncEbayOrders` sits in the `high` queue waiting while `ScrapeAllmakesPsp` runs on `default`. High-priority jobs never get processed until the scrape finishes (which has `$timeout = 0` and can run for hours).

**Cause:** `numprocs=1` means a single worker process handles one job at a time. Queue priority (`--queue=high,default`) only determines which job the worker picks up when it becomes free — it cannot interrupt a job already running.

**Fix:** Increased `numprocs=2` so two worker processes run concurrently. One can be occupied by a long-running `default` job while the other remains free to immediately pick up `high`-priority jobs.

**Note:** Both workers process `--queue=high,default`, so both prefer `high` queue work. A long `default` job only ties up one worker, leaving the other available for admin-triggered ops (eBay sync, Shopify collections, etc.).

---

## Final Working Config

```ini
[program:cydekick-worker]
process_name=%(program_name)s_%(process_num)02d
command=/usr/bin/php /var/www/html/Cydekick/artisan queue:work --queue=high,default --sleep=3 --tries=1 --timeout=0
directory=/var/www/html/Cydekick
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/html/Cydekick/storage/logs/worker.log
stopwaitsecs=120

[program:cydekick-scheduler]
process_name=%(program_name)s
command=/usr/bin/php /var/www/html/Cydekick/artisan schedule:work
directory=/var/www/html/Cydekick
autostart=true
autorestart=true
user=www-data
redirect_stderr=true
stdout_logfile=/var/www/html/Cydekick/storage/logs/scheduler.log
stopwaitsecs=3600
```

---

## Useful Commands

```bash
# Check all processes
sudo supervisorctl status

# Restart after config or schedule changes
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl restart cydekick-scheduler

# Watch scheduler live
tail -f /var/www/html/Cydekick/storage/logs/scheduler.log

# Watch queue worker live
tail -f /var/www/html/Cydekick/storage/logs/worker.log

# Check what schedules are registered and when they next fire
php artisan schedule:list

# Manually trigger the scheduler (useful for testing)
php artisan schedule:run --verbose

# Clear a stuck withoutOverlapping mutex
php artisan schedule:clear-cache

# Check what Laravel thinks the current time is
php artisan tinker --execute="echo now();"
```

---

## Deployment Checklist
After deploying code changes:
```bash
git pull
composer install --no-dev --optimize-autoloader
npm ci --production=false && npm run build
php artisan migrate --force
php artisan view:clear
php artisan optimize:clear
php artisan optimize
sudo supervisorctl restart cydekick-worker:*
sudo supervisorctl restart cydekick-scheduler
sudo supervisorctl status
```