background-job-processing
Implement background job processing systems with task queues, workers, scheduling, and retry mechanisms. Use when handling long-running tasks, sending emails, generating reports, and processing large datasets asynchronously.
- Installs
- 483
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/aj-geddes/useful-ai-prompts --skill background-job-processing- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads background-job-processing automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Background Job Processing
Table of Contents
Overview
Build robust background job processing systems with distributed task queues, worker pools, job scheduling, error handling, retry policies, and monitoring for efficient asynchronous task execution.
When to Use
- Handling long-running operations asynchronously
- Sending emails in background
- Generating reports or exports
- Processing large datasets
- Scheduling recurring tasks
- Distributing compute-intensive operations
Quick Start
Minimal working example:
# celery_app.py
from celery import Celery
from kombu import Exchange, Queue
import os
app = Celery('myapp')
# Configuration
app.conf.update(
broker_url=os.getenv('REDIS_URL', 'redis://localhost:6379/0'),
result_backend=os.getenv('REDIS_URL', 'redis://localhost:6379/0'),
task_serializer='json',
accept_content=['json'],
result_serializer='json',
timezone='UTC',
enable_utc=True,
task_track_started=True,
task_time_limit=30 * 60, # 30 minutes
task_soft_time_limit=25 * 60, # 25 minutes
broker_connection_retry_on_startup=True,
)
# Queue configuration
default_exchange = Exchange('tasks', type='direct')
app.conf.task_queues = (
// ... (see reference guides for full implementation)
Reference Guides
Detailed implementations in the references/ directory:
| Guide | Contents |
|---|---|
| Python with Celery and Redis | Python with Celery and Redis |
| Node.js with Bull Queue | Node.js with Bull Queue |
| Ruby with Sidekiq | Ruby with Sidekiq |
| Job Retry and Error Handling | Job Retry and Error Handling |
| Monitoring and Observability | Monitoring and Observability |
Best Practices
✅ DO
- Use task timeouts to prevent hanging jobs
- Implement retry logic with exponential backoff
- Make tasks idempotent
- Use job priorities for critical tasks
- Monitor queue depths and job failures
- Log job execution details
- Clean up completed jobs
- Set appropriate batch sizes for memory efficiency
- Use dead-letter queues for failed jobs
- Test jobs independently
❌ DON'T
- Use synchronous operations in async tasks
- Ignore job failures
- Make tasks dependent on external state
- Use unbounded retries
- Store large objects in job data
- Forget to handle timeouts
- Run jobs without monitoring
- Use blocking operations in queues
- Forget to track job progress
- Mix unrelated operations in one job
FAQ
What does the background-job-processing skill do?
Implement background job processing systems with task queues, workers, scheduling, and retry mechanisms. Use when handling long-running tasks, sending emails, generating reports, and processing large datasets asynchronously.
How do I install background-job-processing?
Run npx skills add https://github.com/aj-geddes/useful-ai-prompts --skill background-job-processing in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/background-job-processing/ (personal) or .claude/skills/background-job-processing/ (project).
Which AI agents can use background-job-processing?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is background-job-processing safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains background-job-processing?
It is published by aj-geddes in aj-geddes/useful-ai-prompts. It has 483 recorded installs.
More from aj-geddes/useful-ai-prompts
nodejs-express-server
aj-geddes/useful-ai-prompts
Build production-ready Express.js servers with middleware, authentication, routing, and database integration. Use when creating REST APIs, managing requests/responses, implementing middleware chains, and handling server logic.
code-review-analysis
aj-geddes/useful-ai-prompts
Perform comprehensive code reviews with best practices, security checks, and constructive feedback. Use when reviewing pull requests, analyzing code quality, checking for security vulnerabilities, or providing code improvement suggestions.
rest-api-design
aj-geddes/useful-ai-prompts
Design RESTful APIs following best practices for resource modeling, HTTP methods, status codes, versioning, and documentation. Use when creating new APIs, designing endpoints, or improving existing API architecture.
jenkins-pipeline
aj-geddes/useful-ai-prompts
Build Jenkins declarative and scripted pipelines with stages, agents, parameters, and plugins. Implement multi-branch pipelines and deployment automation.
architecture-diagrams
aj-geddes/useful-ai-prompts
Create system architecture diagrams using Mermaid, PlantUML, C4 model, flowcharts, and sequence diagrams. Use when documenting architecture, system design, data flows, or technical workflows.
ansible-automation
aj-geddes/useful-ai-prompts
Infrastructure automation and configuration management using Ansible playbooks, roles, and inventory. Use for deploying applications, patching, and managing servers.
Popular ai & agents skills
google-agents-cli-publish
google/agents-cli
This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).
google-agents-cli-eval
google/agents-cli
This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).
skill-creator
anthropics/skills
Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.
codex-pet
prime-skills/runcomfy-agent-skills
Codex Pet generator on RunComfy. Build a Codex-compatible Codex Pet spritesheet.webp + pet.json from a single reference image, drop it into ${CODEXHOME:-$HOME/.codex}/pets/ / and Codex picks it up as a custom Codex Pet next to the 8 built-ins. This skill produces the exact Codex Pet atlas Codex expects (1536x1872 PNG/WebP, 8 cols x 9 rows, 192x208 cells, 9 animation states — idle, running-right, running-left, waving, jumping, failed, waiting, running, review). Calls OpenAI GPT Image 2 edit ONCE via the local RunComfy CLI as runcomfy run openai/gpt-image-2/edit to produce a canonical Codex Pet pose, then assembles all 9 animation rows programmatically with ImageMagick micro-transforms — no Codex Pro, no $imagegen, no OPENAIAPIKEY required, only RUNCOMFYTOKEN. Triggers on "codex pet", "create codex pet", "make codex pet", "hatch codex pet", "/hatch image", "desktop pet codex", "codex pets", "spritesheet.webp", or any explicit ask to build a custom pet for OpenAI Codex.
full-output-enforcement
leonxlnx/taste-skill
Overrides default LLM truncation behavior. Enforces complete code generation, bans placeholder patterns, and handles token-limit splits cleanly. Apply to any task requiring exhaustive, unabridged output.
executing-plans
obra/superpowers
Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available