BullMQ
BullMQ is a job queue for Node.js. It runs on KiviDB without changes to your code: point its connection at a KiviDB database that has the Lua build.
BullMQ is a job queue for Node.js. It runs on KiviDB without changes to your code: point its connection at a KiviDB database that has the Lua build.
Requirements
| KiviDB | 1.1.0 or later |
| Build | Lua (see Integrations) |
| BullMQ | Tested with 6.3 on ioredis 6 |
Queue names need no special form. Hashtags ({name}) are not required.
Create the database
On KiviDB Cloud, turn on Job queues (BullMQ, Sidekiq) when you create the database. For an existing database, turn on Lua scripting under Configuration.
Self-hosted:
docker run -p 6379:6380 kividb/kividb:v1.2.2-lua
Connect
Use the endpoint, port and credentials from the database's overview.
import { Queue, Worker } from "bullmq";
import IORedis from "ioredis";
const connection = new IORedis({
host: "your-name.cloud.kividb.io",
port: 6379,
username: "kividbapp",
password: process.env.KIVIDB_PASSWORD,
maxRetriesPerRequest: null, // required by BullMQ for workers
});
const queue = new Queue("email", { connection });
await queue.add("welcome", { to: "ada@example.com" });
new Worker(
"email",
async (job) => {
console.log("sending to", job.data.to);
},
{ connection, concurrency: 20 },
);
With TLS turned on for the database, add the CA certificate from Configuration and use the TLS port:
const connection = new IORedis({
host: "your-name.cloud.kividb.io",
port: 6380,
username: "kividbapp",
password: process.env.KIVIDB_PASSWORD,
tls: { ca: fs.readFileSync("kividb-ca.pem") },
maxRetriesPerRequest: null,
});
What is tested
Our suite runs these against KiviDB and against Redis 8, and expects the same result from both:
- Jobs added in bulk and processed by several workers, each exactly once
- Return values stored on the job
- Delayed jobs
- Retries with fixed and exponential backoff, and failure after the last attempt
- Priorities, and pausing and resuming a queue
- Flows: a parent job that waits for its children and reads their results
- Rate limiting
- Repeating jobs (job schedulers), and removing a schedule
- Recovery of a job whose worker died
- Queue events, and waiting for a job to finish
- Deduplication by job id
obliterate, leaving no keys behind
Keep your jobs
- Nothing is evicted. BullMQ needs a database that never drops keys to
make room. KiviDB Cloud databases run with
noeviction: at the memory limit a write is refused with an error, and no job is removed. - Persistence. A queue is data you cannot recompute. Paid plans keep an append-only log on disk; see Persistence.
- Completed jobs. BullMQ keeps every completed and failed job until told
otherwise. Set
removeOnCompleteandremoveOnFailso they do not fill the database:
await queue.add("welcome", data, {
removeOnComplete: { age: 3600, count: 1000 },
removeOnFail: { age: 24 * 3600 },
});
Troubleshooting
ERR Lua scripting is not supported by KiviDB. The database is on the
default build. Turn on Lua scripting.
NOPERM User default has no permissions to run the 'hello' command. The
database was created before this was allowed. Change a database password
under Access Credentials once, which rewrites the database's users, and
connect again.