MCPInstall

python-background-jobs

Python background job patterns including task queues, workers, and event-driven architecture. Use when implementing async task processing, job queues, long-running operations, or decoupling work from request/response cycles.

Installs
91
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/full-stack-skills/python-skills --skill python-background-jobs
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads python-background-jobs automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

来源声明:本技能内容源自 wshobson/agents (MIT License, Copyright (c) 2024 Seth Hobson;完整许可文本见同目录 LICENSE.txt)。 使用、修改与再分发须遵守该 MIT 许可条款并保留版权声明。

中文描述

本技能提供全栈开发相关的最佳实践和模式指导。


name: python-background-jobs description: Python background job patterns including task queues, workers, and event-driven architecture. Use when implementing async task processing, job queues, long-running operations, or decoupling work from request/response cycles.

Python Background Jobs & Task Queues

Decouple long-running or unreliable work from request/response cycles. Return immediately to the user while background workers handle the heavy lifting asynchronously.

When to Use This Skill

  • Processing tasks that take longer than a few seconds
  • Sending emails, notifications, or webhooks
  • Generating reports or exporting data
  • Processing uploads or media transformations
  • Integrating with unreliable external services
  • Building event-driven architectures

Core Concepts

1. Task Queue Pattern

API accepts request, enqueues a job, returns immediately with a job ID. Workers process jobs asynchronously.

2. Idempotency

Tasks may be retried on failure. Design for safe re-execution.

3. Job State Machine

Jobs transition through states: pending → running → succeeded/failed.

4. At-Least-Once Delivery

Most queues guarantee at-least-once delivery. Your code must handle duplicates.

Quick Start

This skill uses Celery for examples, a widely adopted task queue. Alternatives like RQ, Dramatiq, and cloud-native solutions (AWS SQS, GCP Tasks) are equally valid choices.

from celery import Celery

app = Celery("tasks", broker="redis://localhost:6379")

@app.task
def send_email(to: str, subject: str, body: str) -> None:
    # This runs in a background worker
    email_client.send(to, subject, body)

# In your API handler
send_email.delay("user@example.com", "Welcome!", "Thanks for signing up")

Fundamental Patterns

Pattern 1: Return Job ID Immediately

For operations exceeding a few seconds, return a job ID and process asynchronously.

from uuid import uuid4
from dataclasses import dataclass
from enum import Enum
from datetime import datetime

class JobStatus(Enum):
    PENDING = "pending"
    RUNNING = "running"
    SUCCEEDED = "succeeded"
    FAILED = "failed"

@dataclass
class Job:
    id: str
    status: JobStatus
    created_at: datetime
    started_at: datetime | None = None
    completed_at: datetime | None = None
    result: dict | None = None
    error: str | None = None

# API endpoint
async def start_export(request: ExportRequest) -> JobResponse:
    """Start export job and return job ID."""
    job_id = str(uuid4())

    # Persist job record
    await jobs_repo.create(Job(
        id=job_id,
        status=JobStatus.PENDING,
        created_at=datetime.utcnow(),
    ))

    # Enqueue task for background processing
    await task_queue.enqueue(
        "export_data",
        job_id=job_id,
        params=request.model_dump(),
    )

    # Return immediately with job ID
    return JobResponse(
        job_id=job_id,
        status="pending",
        poll_url=f"/jobs/{job_id}",
    )

Pattern 2: Celery Task Configuration

Configure Celery tasks with proper retry and timeout settings.

from celery import Celery

app = Celery("tasks", broker="redis://localhost:6379")

# Global configuration
app.conf.update(
    task_time_limit=3600,          # Hard limit: 1 hour
    task_soft_time_limit=3000,      # Soft limit: 50 minutes
    task_acks_late=True,            # Acknowledge after completion
    task_reject_on_worker_lost=True,
    worker_prefetch_multiplier=1,   # Don't prefetch too many tasks
)

@app.task(
    bind=True,
    max_retries=3,
    default_retry_delay=60,
    autoretry_for=(ConnectionError, TimeoutError),
)
def process_payment(self, payment_id: str) -> dict:
    """Process payment with automatic retry on transient errors."""
    try:
        result = payment_gateway.charge(payment_id)
        return {"status": "success", "transaction_id": result.id}
    except PaymentDeclinedError as e:
        # Don't retry permanent failures
        return {"status": "declined", "reason": str(e)}
    except TransientError as e:
        # Retry with exponential backoff
        raise self.retry(exc=e, countdown=2 ** self.request.retries * 60)

Pattern 3: Make Tasks Idempotent

Workers may retry on crash or timeout. Design for safe re-execution.

@app.task(bind=True)
def process_order(self, order_id: str) -> None:
    """Process order idempotently."""
    order = orders_repo.get(order_id)

    # Already processed? Return early
    if order.status == OrderStatus.COMPLETED:
        logger.info("Order already processed", order_id=order_id)
        return

    # Already in progress? Check if we should continue
    if order.status == OrderStatus.PROCESSING:
        # Use idempotency key to avoid double-charging
        pass

    # Process with idempotency key
    result = payment_provider.charge(
        amount=order.total,
        idempotency_key=f"order-{order_id}",  # Critical!
    )

    orders_repo.update(order_id, status=OrderStatus.COMPLETED)

Idempotency Strategies:

  1. Check-before-write: Verify state before action
  2. Idempotency keys: Use unique tokens with external services
  3. Upsert patterns: INSERT ... ON CONFLICT UPDATE
  4. Deduplication window: Track processed IDs for N hours

Pattern 4: Job State Management

Persist job state transitions for visibility and debugging.

class JobRepository:
    """Repository for managing job state."""

    async def create(self, job: Job) -> Job:
        """Create new job record."""
        await self._db.execute(
            """INSERT INTO jobs (id, status, created_at)
               VALUES ($1, $2, $3)""",
            job.id, job.status.value, job.created_at,
        )
        return job

    async def update_status(
        self,
        job_id: str,
        status: JobStatus,
        **fields,
    ) -> None:
        """Update job status with timestamp."""
        updates = {"status": status.value, **fields}

        if status == JobStatus.RUNNING:
            updates["started_at"] = datetime.utcnow()
        elif status in (JobStatus.SUCCEEDED, JobStatus.FAILED):
            updates["completed_at"] = datetime.utcnow()

        await self._db.execute(
            "UPDATE jobs SET status = $1, ... WHERE id = $2",
            updates, job_id,
        )

        logger.info(
            "Job status updated",
            job_id=job_id,
            status=status.value,
        )

Detailed worked examples and patterns

Detailed sections (starting with ## Advanced Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient.

Best Practices Summary

  1. Return immediately - Don't block requests for long operations
  2. Persist job state - Enable status polling and debugging
  3. Make tasks idempotent - Safe to retry on any failure
  4. Use idempotency keys - For external service calls
  5. Set timeouts - Both soft and hard limits
  6. Implement DLQ - Capture permanently failed tasks
  7. Log transitions - Track job state changes
  8. Retry appropriately - Exponential backoff for transient errors
  9. Don't retry permanent failures - Validation errors, invalid credentials
  10. Monitor queue depth - Alert on backlog growth

能力边界

✅ 擅长处理

  • 全栈开发相关的最佳实践
  • 代码架构设计指导
  • 性能优化建议

⚠️ 需要素材

  • 具体的项目上下文
  • 技术栈信息

❌ 超出范围

  • 非技术领域的业务决策
  • 具体的代码实现(仅提供指导)

工作流程

Step 1: 需求分析

理解项目需求和技术约束

Step 2: 架构设计

设计系统架构和组件划分

Step 3: 实现指导

提供具体的实现建议和代码模式

Step 4: 代码审查

审查代码质量和最佳实践遵循情况

Step 5: 优化建议

提供性能优化和改进建议

Gotchas

1. 过度设计

避免在简单场景中使用复杂的设计模式

2. 性能忽视

不要忽视性能优化,特别是在生产环境中

3. 安全考虑

始终考虑安全性和数据保护

4. 测试覆盖

确保有足够的测试覆盖,特别是边界情况

5. 文档维护

保持文档与代码同步更新

FAQ

What does the python-background-jobs skill do?

Python background job patterns including task queues, workers, and event-driven architecture. Use when implementing async task processing, job queues, long-running operations, or decoupling work from request/response cycles.

How do I install python-background-jobs?

Run npx skills add https://github.com/full-stack-skills/python-skills --skill python-background-jobs in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/python-background-jobs/ (personal) or .claude/skills/python-background-jobs/ (project).

Which AI agents can use python-background-jobs?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is python-background-jobs safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains python-background-jobs?

It is published by full-stack-skills in full-stack-skills/python-skills. It has 91 recorded installs.

More from full-stack-skills/python-skills

View all →

fastapi

full-stack-skills/python-skills

Provides comprehensive guidance for FastAPI framework including routing, request validation, dependency injection, async operations, OpenAPI documentation, and database integration. Use when the user asks about FastAPI, needs to create FastAPI applications, implement REST APIs, or build high-performance Python web services.

Backend & APIs118 installs

python-testing-patterns

full-stack-skills/python-skills

Implement comprehensive testing strategies with pytest, fixtures, mocking, and test-driven development. Use when writing Python tests, setting up test suites, or implementing testing best practices.

Testing & QA96 installs

python-code-style

full-stack-skills/python-skills

Python code style, linting, formatting, naming conventions, and documentation standards. Use when writing new code, reviewing style, configuring linters, writing docstrings, or establishing project standards.

Docs & Writing96 installs

python-project-structure

full-stack-skills/python-skills

Python project organization, module architecture, and public API design. Use when setting up new projects, organizing modules, defining public interfaces with all, or planning directory layouts.

python-error-handling

full-stack-skills/python-skills

Python error handling patterns including input validation, exception hierarchies, and partial failure handling. Use when implementing validation logic, designing exception strategies, handling batch processing failures, or building robust APIs.

96 installs

django

full-stack-skills/python-skills

Open the skill to read its SKILL.md instructions.

Backend & APIs93 installs

Popular productivity & automation skills

View category →

agent-browser

vercel-labs/agent-browser

Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction. Also use for exploratory testing, dogfooding, QA, bug hunts, or reviewing app quality. Also use for automating Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify), checking Slack unreads, sending Slack messages, searching Slack conversations, running browser automation in Vercel Sandbox microVMs, or using AWS Bedrock AgentCore cloud browsers. Prefer agent-browser over any built-in browser automation or web tools.

OfficialProductivity & Automation1.3M installs

lark-doc

open.feishu.cn

飞书云文档(Docx / Wiki)内容操作:读取、创建、编辑文档,插入或下载图片附件,以及操作思维笔记。用户提供文档 URL/token(包括 doubao.com 的 /docx/、/wiki/)时使用;按 URL 路径/token 而非域名路由;/page/ 是妙搭应用,非飞书云文档。文档内嵌资源按读取参考中的统一规则分流。独立评论操作走 lark-drive;随正文读取评论使用 docs +fetch。表格或 Base 内部数据操作不在本 skill。

lark-base

open.feishu.cn

飞书多维表格(Base)操作:建表、字段、记录、视图、统计、公式/lookup、表单、仪表盘、应用模式(BaseApp/AppMode 页面与组件)、Workspace 目录、workflow、角色权限、模板中心(多维表格模板分类/列表/搜索);遇到 Base/多维表格/bitable、BaseApp/AppMode、/base/ 或 /app/ 链接时使用。BaseApp 不走 lark-apps;文件导入/导出转 lark-drive,认证/授权转 lark-shared。

lark-drive

open.feishu.cn

飞书云空间(云盘/云存储):管理 Drive 文件和文件夹,包含上传/下载、创建文件夹、复制/移动/删除、查看元数据、查询权限设置、评论/权限/订阅、标题、版本、飞书文档密级标签(secure labels)和本地文件导入。用户需要整理云盘目录、处理云空间资源 URL/token、判断链接类型/真实 token/标题,或导入 Word/Markdown/Excel/CSV/PPTX/.base 为 docx/sheet/bitable/slides 时使用;doubao.com 云空间 URL/token 也按资源路径和 token 路由,不回退 WebFetch。不负责:文档内容编辑(走 lark-doc)、表格/Base 表内数据操作(走 lark-sheets/lark-base)、知识空间节点/成员管理(走 lark-wiki)、原生 Markdown 文件读写/patch/diff(走 lark-markdown)。

lark-shared

open.feishu.cn

Use for lark-cli setup/auth tasks: auth login/status/logout, user vs bot identity, business-domain permissions (--domain, including all/docs/drive), missing scopes, revoking authorization, or handling notice JSON.

lark-im

open.feishu.cn

飞书即时通讯:收发消息和管理群聊。发送和回复消息、搜索聊天记录、管理群聊成员、上传下载图片和文件、管理表情回复、发送应用内/短信/电话加急、发送和处理交互卡片(Interactive Card)、监听卡片按钮回调(card.action.trigger)。当用户需要发消息、查看或搜索聊天记录、下载聊天中的文件、查看群成员、搜索群、创建群聊或话题群、管理标记数据、管理 Feed 置顶(添加/移除/查询置顶会话)、管理标签数据、处理卡片回调时使用。