Skip to main content

Workers - overview

info

tl;dr: Generally, web requests should fetch data from the database and serialize that data in the response. Hard work should be sent to a background job—either by backgrounding it explicitly or setting up a scheduled job. Backgrounding work provides:

  1. faster web responses
  2. automatic retry of work that fails
  3. rate-limiting certain types of work (when using named workstreams)

A web application is like an ecosystem, with many different players interacting and responding to each other's behavior. Much of this is done in the form of API requests being triggered from clients to your backend application servers, but some things are not. For example, you may find yourself needing a piece of code that runs every hour of every day, or something that just runs on the first Monday of each 3rd month.

Additionally, it is important to keep your web servers fluid. You don't want to expose any endpoints that respond slowly, because, while Node concurrency is helpful, you don't want to lean on it to save you from slow endpoint responses, since this makes you vulnerable to DDoS.

To alleviate the pressure on your web application server, you can offload costly or process-intensive services to background processes, so that your web server can continue to be reliable and performant. Additionally, you can use cron-like features within Psychic to schedule your jobs to run at specific times, such as the third Tuesday of the month at noon UTC.

Logging in Background Jobs​

Background methods can optionally receive a Job parameter as their last argument to access logging functionality:

import { Job } from 'bullmq'

class DataProcessingService extends ApplicationBackgroundedService {
  public static async processLargeDataset(datasetId: string) {
    await this.background('_processLargeDataset', datasetId)
  }

  public static async _processLargeDataset(datasetId: string, job: Job) {
    const dataset = await Dataset.find(datasetId)
    if (!dataset) return

    await job.log(`Starting processing of dataset ${datasetId}`)

    let processedRows = 0

    for (const row of dataset.rows) {
      await this.processRow(row)
      processedRows++

      if (processedRows % 100 === 0) {
        await job.log(`processedRows: ${processedRows}`)
      }
    }

    await job.log(`Completed: processedRows: ${processedRows}`)
  }
}

The Job parameter is optional and always comes last in the method signature — Psychic appends it to every dispatch, which is why no parameter of a backgrounded method may carry a default or a ? (see Services). Job logs are accessible through the BullMQ dashboard and can be retrieved programmatically. Let unexpected errors throw so BullMQ can retry the job; catching and continuing marks the job successful.

Where to go next​

Getting set up

  • installation — adding Psychic Workers to an application
  • config — workstreams, queues, and Redis connections
  • testing — testing backgrounded and scheduled work

Where background work is defined

  • services — backgrounded services, the primary home for background work
  • models — backgrounding a method directly on a model

Capabilities — knobs each backgrounded service can set

  • scheduled — running work on a cron-like cadence
  • backgroundWith — running work once, after a delay, or at a one-off priority
  • retry — how failed jobs are retried
  • priority — running some work ahead of other work
  • named workstreams — isolating work into its own queue
  • rate limiting — throttling work against an external dependency

Patterns — techniques composed from the capabilities above