# Background Job Processing

<div style="text-align: justify;">

VeloxFactory processes background work through a Redis-backed queue managed by <a href="https://laravel.com/docs/horizon" target="_blank">Laravel Horizon</a>: thumbnail generation, mail dispatch, and everything the Job Scheduler fires. Horizon runs as a single supervised process and manages its own worker pool internally, with dynamic scaling, real-time monitoring and a built-in dashboard.

---

<h3 style="color: #203671; margin-top: 2.2em;">Prerequisites</h3>

Horizon has two hard requirements beyond the base VeloxFactory stack: a running Redis instance and the `php-redis` PHP extension. Neither is optional, Horizon will refuse to start without both.

<h4 style="color: #203671; margin-top: 1.4em;">Redis</h4>

Redis can be installed natively or run as a Docker container. Both work equally well; the choice depends on your infrastructure preferences.

**Native installation:**

```bash
apt install redis-server
systemctl enable redis-server
systemctl start redis-server
```

**Docker container:**

```bash
docker run -d \
  --name redis \
  --restart unless-stopped \
  -p 127.0.0.1:6379:6379 \
  redis:alpine
```

The container binds to `127.0.0.1` only, Redis is not exposed to the network, which is the correct default for a single-server deployment.

Whichever method you choose, verify connectivity before proceeding:

```bash
redis-cli ping
# Expected: PONG
```

<h4 style="color: #203671; margin-top: 1.4em;">PHP Redis Extension</h4>

VeloxFactory is configured to use `phpredis` as the Redis client. The extension must be installed and active for PHP:

```bash
apt install php-redis
systemctl restart php8.2-fpm   # adjust version to match your PHP installation
```

Confirm the extension is loaded:

```bash
php -m | grep redis
# Expected: redis
```

<h4 style="color: #203671; margin-top: 1.4em;">.env Configuration</h4>

With Redis running and the extension installed, update your `.env` to activate Redis as the queue driver:

```dotenv
QUEUE_CONNECTION=redis

REDIS_CLIENT=phpredis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
```

---

<h3 style="color: #203671; margin-top: 2.2em;">Web Server</h3>

VeloxFactory requires a web server that routes all requests through Laravel's `public/index.php` entry point. Both Apache2 and nginx are supported. The Horizon dashboard at `/horizon` and the Reverb WebSocket endpoint require no special routing rules, they are handled by Laravel and PHP-FPM like any other request, with one exception: Reverb needs a WebSocket proxy pass.

<h4 style="color: #203671; margin-top: 1.4em;">Apache2</h4>

Enable `mod_rewrite` before configuring the vhost:

```bash
a2enmod rewrite proxy proxy_http proxy_wstunnel
systemctl restart apache2
```

Virtual host configuration:

```apache
<VirtualHost *:80>
    ServerName veloxfactory.example.com
    DocumentRoot /var/www/veloxfactory/public

    <Directory /var/www/veloxfactory/public>
        Options -Indexes +FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    # Reverb WebSocket proxy
    ProxyPreserveHost On
    ProxyPass /app ws://127.0.0.1:8080/app
    ProxyPassReverse /app ws://127.0.0.1:8080/app

    ErrorLog ${APACHE_LOG_DIR}/veloxfactory-error.log
    CustomLog ${APACHE_LOG_DIR}/veloxfactory-access.log combined
</VirtualHost>
```

Laravel's bundled `.htaccess` in `public/` handles the rewrite rules, no additional configuration needed for URL routing.

<h4 style="color: #203671; margin-top: 1.4em;">nginx</h4>

```nginx
server {
    listen 80;
    server_name veloxfactory.example.com;
    root /var/www/veloxfactory/public;

    index index.php;

    # Laravel URL routing
    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    # PHP-FPM
    location ~ \.php$ {
        fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
        fastcgi_read_timeout 120;
    }

    # Reverb WebSocket proxy
    location /app {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
    }

    # Block dotfile access
    location ~ /\.(?!well-known).* {
        deny all;
    }
}
```

<div style="border-left: 4px solid #5fc75d; background: #f6fdf6; padding: 10px 16px; margin: 16px 0;">
ℹ️ <strong>Adjust the PHP-FPM socket path to match your PHP version.</strong> On systems with multiple PHP versions installed, the socket is typically at <code>/var/run/php/php8.2-fpm.sock</code>. Verify with <code>ls /var/run/php/</code>.
</div>

---

<h3 style="color: #203671; margin-top: 2.2em;">Queues</h3>

VeloxFactory uses three queues with distinct priorities:

<table style="width: 100%; border-collapse: collapse;">
  <thead>
    <tr style="border-top: 1px solid #e6e8ef; border-bottom: 1px solid #e6e8ef;">
      <th style="text-align: left; padding: 6px 10px; white-space: nowrap;">Queue</th>
      <th style="text-align: left; padding: 6px 10px;">Jobs</th>
      <th style="text-align: left; padding: 6px 10px;">Notes</th>
    </tr>
  </thead>
  <tbody>
    <tr style="border-bottom: 1px solid #e6e8ef;">
      <td style="padding: 6px 10px; white-space: nowrap;"><code>high</code></td>
      <td style="padding: 6px 10px;">Thumbnail generation</td>
      <td style="padding: 6px 10px;">Dispatched on-demand when a report is rendered. Processed with highest priority, workers on this queue are never blocked by maintenance routines.</td>
    </tr>
    <tr style="border-bottom: 1px solid #e6e8ef;">
      <td style="padding: 6px 10px; white-space: nowrap;"><code>default</code></td>
      <td style="padding: 6px 10px;">Mail dispatch, the scheduler tick</td>
      <td style="padding: 6px 10px;">Mails sent in background mode are queued here, together with the minute tick that looks for due schedules. Both are short jobs.</td>
    </tr>
    <tr>
      <td style="padding: 6px 10px; white-space: nowrap;"><code>scheduler</code></td>
      <td style="padding: 6px 10px;">Runs of the Job Scheduler</td>
      <td style="padding: 6px 10px;">Scheduled renders and the purge run. These are the long jobs of the instance, minutes rather than seconds, so they get a queue of their own.</td>
    </tr>
  </tbody>
</table>

<div style="border-left: 4px solid #5fc75d; background: #f6fdf6; padding: 10px 16px; margin: 16px 0;">
ℹ️ <strong>The three queues run in separate worker pools.</strong> A purge run on the <code>scheduler</code> queue cannot delay thumbnail generation on <code>high</code> or a mail on <code>default</code>, they never compete for the same worker.
</div>

<h4 style="color: #203671; margin-top: 1.4em;">Mail Dispatch</h4>

A mailing sent with `sendAsync` creates its mail task immediately and hands the send to the `default` queue. The job carries only the ID of the task, never the attachments, so nothing large ever travels through Redis: the files are read from the history disk at send time.

A mail that runs into the dispatch rate limit of its mailer is not failed and not lost. The job is released back onto the queue with a delay until the next window opens, and the task stays *pending* in the meantime. Several workers may try to send at the same time, the limiter counts atomically in Redis, so the configured caps hold regardless of how many workers are running.

<h4 style="color: #203671; margin-top: 1.4em;">Scheduled Work</h4>

Every recurring job of the instance is a record in the Job Scheduler, with its own crontab expression. Laravel's own schedule holds exactly one entry: once a minute it queues `DispatchDueScheduledJobsJob`, which reads the due schedules from the database and queues one run per schedule on the `scheduler` queue.

That includes the purge run, which deletes print tasks, mail tasks, history records, orphaned files and the scheduler's own run log in one pass and in a fixed order. The *Schedule your jobs* chapter covers the retentions and the rest of the configuration.

A scheduled run is attempted once and is never retried: a second attempt would print a second label and send a second mail. Two runs of the same schedule never overlap either. What happened is on the run record, with its trace id.

---

<h3 style="color: #203671; margin-top: 2.2em;">Horizon Supervisors</h3>

Horizon manages workers through internal supervisors, process groups, each responsible for one queue. The system-level Supervisor (Supervisord) only ever manages the single Horizon master process; Horizon itself handles everything below that.

<table style="width: 100%; border-collapse: collapse;">
  <thead>
    <tr style="border-top: 1px solid #e6e8ef; border-bottom: 1px solid #e6e8ef;">
      <th style="text-align: left; padding: 6px 10px; white-space: nowrap;">Supervisor</th>
      <th style="text-align: left; padding: 6px 10px;">Queue</th>
      <th style="text-align: left; padding: 6px 10px;">Balancing</th>
      <th style="text-align: left; padding: 6px 10px;">Notes</th>
    </tr>
  </thead>
  <tbody>
    <tr style="border-bottom: 1px solid #e6e8ef;">
      <td style="padding: 6px 10px; white-space: nowrap;"><code>supervisor-rendering</code></td>
      <td style="padding: 6px 10px;"><code>high</code></td>
      <td style="padding: 6px 10px;">Auto</td>
      <td style="padding: 6px 10px;">Scales worker processes dynamically based on queue depth, up to 10 in production. Job timeout: 120 seconds.</td>
    </tr>
    <tr style="border-bottom: 1px solid #e6e8ef;">
      <td style="padding: 6px 10px; white-space: nowrap;"><code>supervisor-default</code></td>
      <td style="padding: 6px 10px;"><code>default</code></td>
      <td style="padding: 6px 10px;">Simple</td>
      <td style="padding: 6px 10px;">Fixed worker count, up to three processes in production. Runs with lower CPU priority (<code>nice 10</code>), mail dispatch should not compete with rendering for system resources. Job timeout: 300 seconds.</td>
    </tr>
    <tr>
      <td style="padding: 6px 10px; white-space: nowrap;"><code>supervisor-scheduler</code></td>
      <td style="padding: 6px 10px;"><code>scheduler</code></td>
      <td style="padding: 6px 10px;">Simple</td>
      <td style="padding: 6px 10px;">Up to two processes, on its own queue connection <code>redis-scheduler</code> and with lower CPU priority (<code>nice 10</code>). Job timeout: 1800 seconds, long enough for a purge over a large history disk or a batch of scheduled renders.</td>
    </tr>
  </tbody>
</table>

<div style="border-left: 4px solid #5fc75d; background: #f6fdf6; padding: 10px 16px; margin: 16px 0;">
ℹ️ <strong>A connection's <code>retry_after</code> has to stay above the job timeout of every supervisor using it.</strong> A job that runs longer than <code>retry_after</code> is considered lost and handed to a second worker while the first is still working on it, which for a scheduled render means a duplicate print and a duplicate mail. The <code>redis</code> connection allows 360 seconds against a timeout of 300, <code>redis-scheduler</code> allows 1860 against 1800. Raise <code>retry_after</code> first whenever you raise a timeout.
</div>

---

<h3 style="color: #203671; margin-top: 2.2em;">System Supervisor Configuration</h3>

Supervisord keeps three VeloxFactory processes alive and restarts them automatically on failure: the Horizon master, the Reverb WebSocket server, and the scheduler.

```ini
; Horizon - manages all VeloxFactory queue workers internally
[program:veloxfactory-horizon]
directory=/var/www/veloxfactory
command=/usr/bin/php artisan horizon
user=www-data
process_name=%(program_name)s
numprocs=1
autostart=true
autorestart=true
startretries=10
stopasgroup=true
killasgroup=true
stopsignal=TERM
startsecs=3
stopwaitsecs=1860
redirect_stderr=true
stdout_logfile=/var/log/supervisor/veloxfactory-horizon.log
environment=HOME="/home/www-data",PATH="/usr/local/bin:/usr/bin:/bin"

; Reverb - WebSocket server for real-time events
[program:veloxfactory-reverb]
directory=/var/www/veloxfactory
command=/usr/bin/php artisan reverb:start
user=www-data
autostart=true
autorestart=true
startretries=10
stopasgroup=true
killasgroup=true
stopsignal=INT
startsecs=3
stopwaitsecs=60
redirect_stderr=true
stdout_logfile=/var/log/supervisor/veloxfactory-reverb.log
environment=HOME="/home/www-data",PATH="/usr/local/bin:/usr/bin:/bin"

; Scheduler - runs Laravel's task schedule every minute
[program:veloxfactory-schedule]
directory=/var/www/veloxfactory
command=/usr/bin/php artisan schedule:work
user=www-data
autostart=true
autorestart=true
startretries=10
stopasgroup=true
killasgroup=true
stopsignal=INT
startsecs=3
stopwaitsecs=60
redirect_stderr=true
stdout_logfile=/var/log/supervisor/veloxfactory-schedule.log
environment=HOME="/home/www-data",PATH="/usr/local/bin:/usr/bin:/bin"
```

<div style="border-left: 4px solid #5fc75d; background: #f6fdf6; padding: 10px 16px; margin: 16px 0;">
ℹ️ <strong><code>stopsignal=TERM</code> is required for Horizon.</strong> SIGTERM triggers a graceful shutdown, Horizon finishes any in-flight jobs before stopping its workers. Using SIGKILL or SIGINT instead will interrupt running jobs mid-execution and may leave Report History Records in an incomplete state.
</div>

<div style="border-left: 4px solid #F9C846; background: #fffbf0; padding: 12px 16px; margin: 16px 0; color: #525E5A;">
<strong>The scheduler process drives every recurring job of the instance.</strong> Without <code>schedule:work</code> running (or an equivalent cron entry calling <code>schedule:run</code> every minute), no schedule fires: no scheduled render, no cleanup, and the disk keeps filling. A cron entry is the alternative on hosts without Supervisord:<br><code>* * * * * cd /var/www/veloxfactory &amp;&amp; php artisan schedule:run &gt;&gt; /dev/null 2&gt;&amp;1</code>
</div>

After updating the configuration:

```bash
supervisorctl reread
supervisorctl update
supervisorctl status
```

---

<h3 style="color: #203671; margin-top: 2.2em;">The Dashboard</h3>

The Horizon dashboard is available at `/horizon`. It is restricted to users with the `global:admin` permission, the same permission required to manage users and system-wide settings.

The dashboard provides a real-time view of the entire queue system: pending and completed jobs, throughput metrics, worker counts per supervisor, and a full log of failed jobs with their stack traces. Failed jobs can be retried directly from the dashboard without any CLI access.

For scheduled work the Job Scheduler's own run log is the better place to look: it holds one record per run with its trace id, its result and the history record it produced. Horizon shows the job, the run log shows what the job did.

---

<h3 style="color: #203671; margin-top: 2.2em;">Graceful Shutdown &amp; Deployments</h3>

When Horizon is stopped, for deployments, configuration changes, or server maintenance, it finishes any jobs currently in flight before exiting. The Supervisord configuration allows up to 1860 seconds for this drain, matching the longest job timeout of any supervisor, so even a purge run over a large history disk finishes cleanly.

```bash
# Stop Horizon gracefully (in-flight jobs will finish first)
supervisorctl stop veloxfactory-horizon

# Restart after a deployment
supervisorctl restart veloxfactory-horizon
```

Never force-kill the Horizon process during a deployment. Always use `supervisorctl stop` or `supervisorctl restart`, both send SIGTERM and wait for the drain period.

A deployment window shorter than the drain period is worth planning around the schedules: a restart during the nightly purge waits for it, and a schedule whose slot falls into the downtime is dropped rather than fired late.

</div>