Queues & Horizon

AutoCom uses Laravel Horizon to manage Redis-based queue workers. Horizon provides a real-time dashboard, automatic process balancing, and job monitoring — all integrated with Stancl Tenancy for multi-tenant isolation.

Architecture Overview

                         +-----------------------+
                         |    Redis (Queue)       |
                         +-----------------------+
                                   |
              +----------+---------+---------+----------+
              |          |         |         |          |
        +-----+----+ +--+------+ +---+---+ +---+----+ +----+------+
        |supervisor-| |supervisor| |super-| |super-  | |supervisor-|
        |  default  | |-workflows| |visor-| |visor-  | | module-bus|
        +-----+----+ +--+------+ |builds| | long   | +----+------+
        | default  |  |workflows| +--+---+ +---+----+ | module-bus|
        | + module |  +---------+ |builds| |imports | +-----------+
        | queues   |              +------+ |exports |
        +----------+                       |ai      |
                                           +--------+

Supervisors

Horizon runs 5 supervisors, each tuned for a different workload type:

Supervisor	Queues	Timeout	Tries	Memory	Purpose
supervisor-default	default + module queues	120s	3	128MB	General-purpose jobs
supervisor-workflows	workflows	300s	3	256MB	Workflow execution (long-running)
supervisor-builds	builds	180s	3	256MB	Module compilation (CPU-bound)
supervisor-long	imports, exports, ai	600s	2	256MB	Bulk operations, AI generation
supervisor-module-bus	module-bus	120s	3	128MB	Inter-module API calls and event fan-out

In production, supervisors auto-scale up to 10/5/3/5/5 worker processes respectively.

Multi-Tenant Support

Queue jobs are automatically tenant-aware through Stancl Tenancy's QueueTenancyBootstrapper (enabled in config/tenancy.php). When a job is dispatched within a tenant context:

1. The tenant ID is serialized into the job payload
2. When the worker picks up the job, tenancy is re-initialized
3. All database queries, cache calls, and file operations use the correct tenant context

No special configuration is needed — dispatching a job inside a tenant request automatically makes it tenant-aware.

Configuration

Horizon Config

The main configuration file is backend/config/horizon.php. Key settings:

// config/horizon.php

return [
    'path' => env('HORIZON_PATH', 'horizon'),      // Dashboard URL path
    'prefix' => 'autocom_horizon:',                 // Redis key prefix
    'memory_limit' => 128,                          // Master process memory limit (MB)
    'fast_termination' => true,                     // Faster deployments

    'defaults' => [
        'supervisor-default' => [
            'connection' => 'redis',
            'queue' => ['default'],                 // + module-declared queues
            'balance' => 'auto',
            'maxProcesses' => 1,                    // Overridden per environment
            'tries' => 3,
            'timeout' => 120,
        ],
        // ... other supervisors
    ],

    'environments' => [
        'production' => [
            'supervisor-default' => ['maxProcesses' => 10],
            'supervisor-workflows' => ['maxProcesses' => 5],
            'supervisor-builds' => ['maxProcesses' => 3],
            'supervisor-long' => ['maxProcesses' => 5],
            'supervisor-module-bus' => ['maxProcesses' => 5],
        ],
        'local' => [
            'supervisor-default' => ['maxProcesses' => 3],
            'supervisor-workflows' => ['maxProcesses' => 2],
            'supervisor-builds' => ['maxProcesses' => 1],
            'supervisor-long' => ['maxProcesses' => 2],
            'supervisor-module-bus' => ['maxProcesses' => 2],
        ],
    ],
];

Environment Variables

Variable	Default	Description
QUEUE_CONNECTION	redis	Must be redis for Horizon
REDIS_HOST	127.0.0.1	Redis server host
REDIS_PORT	6379	Redis server port
REDIS_PASSWORD	null	Redis password
HORIZON_PATH	horizon	Dashboard URL path
HORIZON_OPEN	false	Set to true to allow unauthenticated dashboard access
HORIZON_EMAILS	(empty)	Comma-separated emails allowed to access dashboard in production

Horizon Dashboard

Access the Horizon dashboard at:

https://your-api-domain.com/horizon

The dashboard provides:

• Real-time metrics — Jobs processed, failed, throughput per minute
• Queue monitoring — Pending, completed, and failed jobs per queue
• Supervisor status — Worker processes, memory usage, balancing
• Job details — Inspect payloads, tags, exceptions, retry history
• Tag filtering — Filter jobs by tenant:acme, workflow:uuid, etc.

Dashboard Access Control

Access is controlled by HorizonServiceProvider:

// app/Providers/HorizonServiceProvider.php

protected function gate(): void
{
    Gate::define('viewHorizon', function ($user = null) {
        // Always open in local/testing
        if (app()->environment('local', 'testing')) {
            return true;
        }

        // Env toggle: HORIZON_OPEN=true opens dashboard
        if (env('HORIZON_OPEN', false)) {
            return true;
        }

        // Production: whitelist by email
        $allowed = explode(',', env('HORIZON_EMAILS', ''));
        return $user && in_array($user->email, $allowed);
    });
}

Docker Setup

Production (docker-compose.prod.yml)

Horizon runs as its own container alongside the app:

horizon:
  build:
    context: .
    dockerfile: backend/Dockerfile.prod
  command: php artisan horizon
  stop_signal: SIGTERM
  stop_grace_period: 30s
  volumes:
    - app_storage:/var/www/html/storage
    - tenant_modules:/var/www/html/storage/app/tenants
  environment:
    - APP_ENV=production
    - QUEUE_CONNECTION=redis
    - REDIS_HOST=redis
    - REDIS_PASSWORD=${REDIS_PASSWORD}
    - CACHE_DRIVER=redis
  depends_on:
    - app
    - redis
  healthcheck:
    test: ["CMD-SHELL", "php artisan horizon:status | grep -q running || exit 1"]
    interval: 30s
    timeout: 10s
    retries: 3

Key points:

• stop_signal: SIGTERM ensures graceful worker shutdown (finish current job before stopping)
• stop_grace_period: 30s gives workers time to finish
• Shares app_storage volume with the app container for logs and OAuth keys
• Health check via horizon:status command

Development (docker-compose.yml)

Dev also uses Horizon (not queue:work) for consistent behavior:

horizon:
  build:
    context: ./backend
    dockerfile: Dockerfile
  command: php artisan horizon
  stop_signal: SIGTERM
  volumes:
    - ./backend:/var/www
    - ./modules:/var/www/modules:ro
  environment:
    - APP_ENV=local
    - QUEUE_CONNECTION=redis
    - REDIS_HOST=redis
  depends_on:
    - app
    - redis

Useful Commands

# Check Horizon status
docker exec autocom-horizon php artisan horizon:status

# List all supervisors
docker exec autocom-horizon php artisan horizon:list

# Pause all workers (stop processing new jobs)
docker exec autocom-horizon php artisan horizon:pause

# Resume workers
docker exec autocom-horizon php artisan horizon:continue

# Gracefully terminate Horizon (finish current jobs, then stop)
docker exec autocom-horizon php artisan horizon:terminate

# View failed jobs
docker exec autocom-api php artisan queue:failed

# Retry a failed job
docker exec autocom-api php artisan queue:retry <job-id>

# Retry all failed jobs
docker exec autocom-api php artisan queue:retry all

# Take a metrics snapshot (runs automatically every 5 minutes)
docker exec autocom-horizon php artisan horizon:snapshot

Creating Jobs

The TenantAwareJob Trait

All queued jobs should use the TenantAwareJob trait for automatic tenant tagging in Horizon:

namespace App\Jobs;

use App\Jobs\Concerns\TenantAwareJob;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class ProcessOrderJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels, TenantAwareJob;

    public int $tries = 3;
    public int $timeout = 120;
    public int $backoff = 30;

    public function __construct(
        protected string $orderId
    ) {
        // Route to a specific queue (optional)
        $this->onQueue('default');
    }

    public function handle(): void
    {
        // Your job logic — tenant context is automatically set
    }

    /**
     * Custom tags for Horizon (merged with auto tenant tag).
     */
    public function jobTags(): array
    {
        return [
            'order:' . $this->orderId,
        ];
    }
}

The TenantAwareJob trait automatically:

• Adds a tenant:{id} tag when the job is dispatched from a tenant context
• Merges with any custom tags you define in jobTags()
• Makes jobs filterable by tenant in the Horizon dashboard

Dispatching Jobs

// From a controller or service (inside tenant context)
ProcessOrderJob::dispatch($order->id);

// Dispatch to a specific queue
ProcessOrderJob::dispatch($order->id)->onQueue('imports');

// Dispatch with delay
ProcessOrderJob::dispatch($order->id)->delay(now()->addMinutes(5));

// Dispatch after response is sent to client
ProcessOrderJob::dispatchAfterResponse($order->id);

Queue Selection Guide

Queue	Use When
default	General jobs, notifications, quick tasks (<2 min)
workflows	Workflow execution jobs (automatic via ExecuteWorkflowJob)
builds	Module compilation (automatic via BuildTenantModuleJob)
imports	Bulk data imports (CSV, JSON, source imports)
exports	Data export generation
ai	AI content generation, embeddings, long AI calls
module-bus	Inter-module API calls and cross-tenant event fan-out (automatic via Module API Bus)

Module Integration

Modules can declare their own queues and create jobs that integrate with Horizon.

Declaring Module Queues

Add a queues key to your module's module.json:

{
  "name": "MyModule",
  "alias": "my-module",
  "backend": {
    "namespace": "Modules\\MyModule",
    "providers": ["Modules\\MyModule\\MyModuleServiceProvider"],
    "migrations": true,
    "queues": ["my-module-sync", "my-module-webhooks"]
  }
}

Horizon automatically discovers these queues at startup and adds them to the supervisor-default worker pool. No configuration changes needed — just declare and use.

Reserved queue names that already have dedicated supervisors (don't declare these):

• workflows, builds, imports, exports, ai, module-bus

Creating Module Jobs

Create jobs inside your module's namespace:

namespace Modules\MyModule\App\Jobs;

use App\Jobs\Concerns\TenantAwareJob;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class SyncProductsJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels, TenantAwareJob;

    public int $tries = 3;
    public int $timeout = 300;
    public int $backoff = 60;

    public function __construct(
        protected string $storeId
    ) {
        // Use the module's declared queue
        $this->onQueue('my-module-sync');
    }

    public function handle(): void
    {
        // Sync logic here
        // Tenant context is automatically available
        $products = \Modules\MyModule\App\Models\Product::all();
        // ...
    }

    public function jobTags(): array
    {
        return [
            'my-module',
            'sync',
            'store:' . $this->storeId,
        ];
    }

    public function failed(\Throwable $exception): void
    {
        \Log::error("SyncProductsJob failed for store {$this->storeId}", [
            'error' => $exception->getMessage(),
        ]);
    }
}

Dispatching from Module Controllers

namespace Modules\MyModule\App\Http\Controllers;

use App\Http\Controllers\Controller;
use Modules\MyModule\App\Jobs\SyncProductsJob;

class SyncController extends Controller
{
    public function sync(string $storeId)
    {
        SyncProductsJob::dispatch($storeId);

        return response()->json([
            'success' => true,
            'message' => 'Sync job queued successfully.',
        ]);
    }
}

Using Built-in Queues from Modules

Modules can also dispatch to the built-in queues without declaring their own:

// Use the AI queue for AI operations
AiGenerateJob::dispatch($input)->onQueue('ai');

// Use the imports queue for bulk data
BulkImportJob::dispatch($file)->onQueue('imports');

// Use the default queue for quick tasks
SendNotificationJob::dispatch($userId);

Workflow Integration

Modules that register workflow nodes can dispatch jobs from within workflow execution. The ExecuteWorkflowJob runs on the workflows queue and maintains tenant context throughout the workflow:

namespace Modules\MyModule\Nodes\Actions;

use Modules\Workflows\App\Contracts\WorkflowNodeContract;

class MyCustomAction implements WorkflowNodeContract
{
    public function execute(array $input, array $context): array
    {
        // This runs inside a workflow job — tenant context is active
        // You can dispatch sub-jobs to other queues
        SyncProductsJob::dispatch($input['store_id']);

        return [
            'success' => true,
            'message' => 'Sync dispatched',
        ];
    }
}

Monitoring & Troubleshooting

Job Lifecycle

Dispatched → Pending → Reserved → Processing → Completed
                                       ↓
                                    Failed (after max retries)

Checking Queue Health

# View all supervisor processes
php artisan horizon:list

# Check if Horizon is running
php artisan horizon:status

# View pending jobs count per queue
php artisan tinker --execute="
  \$redis = app('redis');
  \$queues = ['default', 'workflows', 'builds', 'imports', 'exports', 'ai'];
  foreach (\$queues as \$q) {
      echo \$q . ': ' . \$redis->llen('queues:' . \$q) . ' pending' . PHP_EOL;
  }
"

Common Issues

Jobs not processing:

• Verify QUEUE_CONNECTION=redis in your environment
• Check that Horizon is running: php artisan horizon:status
• Ensure the job's queue name matches a supervisor's queue list
• Check Redis connectivity: redis-cli ping

Jobs failing silently:

• Check storage/logs/laravel.log for exceptions
• View failed jobs: php artisan queue:failed
• Check the Horizon dashboard for job exception details

Tenant context lost in job:

• Ensure QueueTenancyBootstrapper is in config/tenancy.php bootstrappers
• Verify the job uses SerializesModels trait
• Don't dispatch jobs from outside a tenant context expecting tenant data

Module queues not appearing in Horizon:

• Verify "queues": ["queue-name"] in your module's module.json
• Restart Horizon after adding new queues: php artisan horizon:terminate
• Check that the module is enabled

Deployment Workflow

When deploying new code:

# 1. Build new images
docker compose -f docker-compose.prod.yml build app horizon

# 2. Restart app first (serves new code)
docker compose -f docker-compose.prod.yml up -d app

# 3. Terminate Horizon gracefully (finishes current jobs)
docker exec autocom-horizon php artisan horizon:terminate

# 4. Restart Horizon container (picks up new code)
docker compose -f docker-compose.prod.yml up -d horizon

Horizon's fast_termination setting allows the new instance to start while the old one finishes its current jobs, minimizing downtime.

Next Steps

• Docker Production for full Docker deployment setup
• Creating Modules to build modules that use queues
• Autoscaling for scaling workers under load