chunking-strategy

Q: How do I install chunking-strategy?

Run npx skills add https://github.com/giuseppe-trisciuoglio/developer-kit --skill chunking-strategy in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/chunking-strategy/ (personal) or .claude/skills/chunking-strategy/ (project).

Q: Which AI agents can use chunking-strategy?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Q: Is chunking-strategy safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Q: Who maintains chunking-strategy?

It is published by giuseppe-trisciuoglio in giuseppe-trisciuoglio/developer-kit. It has 3,849 recorded installs.

by giuseppe-trisciuoglioAI & Agents

Provides chunking strategies for RAG systems. Generates chunk size recommendations (256-1024 tokens), overlap percentages (10-20%), and semantic boundary detection methods. Validates semantic coherence and evaluates retrieval precision/recall metrics. Use when building retrieval-augmented generation systems, vector databases, or processing large documents.

Installs: 3.8K
Last 24h: +8
Trending rank: #9523
Security: Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/giuseppe-trisciuoglio/developer-kit --skill chunking-strategy

01
Install
Run the command in your project. The CLI asks which agents to install for.
02
Describe the task
Ask your agent for something this skill covers. It loads chunking-strategy automatically when relevant.
03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

Chunking Strategy for RAG Systems

Overview

Provides chunking strategies for RAG systems, vector databases, and document processing. Recommends chunk sizes, overlap percentages, and boundary detection methods; validates semantic coherence; evaluates retrieval metrics.

When to Use

Use when building or optimizing RAG systems, vector search pipelines, document chunking workflows, or performance-tuning existing systems with poor retrieval quality.

Instructions

Choose Chunking Strategy

Select based on document type and use case:

Fixed-Size Chunking (Level 1)
- Use for simple documents without clear structure
- Start with 512 tokens and 10-20% overlap
- Adjust: 256 for factoid queries, 1024 for analytical
Recursive Character Chunking (Level 2)
- Use for documents with structural boundaries
- Hierarchical separators: paragraphs → sentences → words
- Customize for document types (HTML, Markdown, JSON)
Structure-Aware Chunking (Level 3)
- Use for structured content (Markdown, code, tables, PDFs)
- Preserve semantic units: functions, sections, table blocks
- Validate structure preservation post-split
Semantic Chunking (Level 4)
- Use for complex documents with thematic shifts
- Embedding-based boundary detection with 0.8 similarity threshold
- Buffer size: 3-5 sentences
Advanced Methods (Level 5)
- Late Chunking for long-context models
- Contextual Retrieval for high-precision requirements
- Monitor computational cost vs. retrieval gain

Reference: references/strategies.md.

Implement Chunking Pipeline

Pre-process documents
- Analyze structure, content types, information density
- Identify multi-modal content (tables, images, code)
Select parameters
- Chunk size: embedding model context window / 4
- Overlap: 10-20% for most cases
- Strategy-specific settings
Process and validate
- Apply chunking strategy
- Validate coherence: run evaluate_chunks.py --coherence (see below)
- Test with representative documents
Evaluate and iterate
- Measure precision and recall
- If precision < 0.7: reduce chunk_size by 25% and re-evaluate
- If recall < 0.6: increase overlap by 10% and re-evaluate
- Monitor latency and memory usage

Reference: references/implementation.md.

Validate Chunk Quality

Run validation commands to assess chunk quality:

# Check semantic coherence (requires sentence-transformers)
python -c "
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
chunks = [...]  # your chunks
embeddings = model.encode(chunks)
similarity = (embeddings @ embeddings.T).mean()
print(f'Cohesion: {similarity:.3f}')  # target: 0.3-0.7
"

# Measure retrieval precision
python -c "
relevant = sum(1 for c in retrieved if c in relevant_chunks)
precision = relevant / len(retrieved)
print(f'Precision: {precision:.2f}')  # target: >= 0.7
"

# Check chunk size distribution
python -c "
import numpy as np
sizes = [len(c.split()) for c in chunks]
print(f'Mean: {np.mean(sizes):.0f}, Std: {np.std(sizes):.0f}')
print(f'Min: {min(sizes)}, Max: {max(sizes)}')
"

Reference: references/evaluation.md.

Examples

Fixed-Size Chunking

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=256,
    chunk_overlap=25,
    length_function=len
)
chunks = splitter.split_documents(documents)

Structure-Aware Code Chunking

import ast

def chunk_python_code(code):
    tree = ast.parse(code)
    chunks = []
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
            chunks.append(ast.get_source_segment(code, node))
    return chunks

Semantic Chunking

def semantic_chunk(text, similarity_threshold=0.8):
    sentences = split_into_sentences(text)
    embeddings = generate_embeddings(sentences)
    chunks, current = [], [sentences[0]]
    for i in range(1, len(sentences)):
        sim = cosine_similarity(embeddings[i-1], embeddings[i])
        if sim < similarity_threshold:
            chunks.append(" ".join(current))
            current = [sentences[i]]
        else:
            current.append(sentences[i])
    chunks.append(" ".join(current))
    return chunks

Best Practices

Core Principles

Balance context preservation with retrieval precision
Maintain semantic coherence within chunks
Optimize for embedding model context window constraints

Implementation

Start with fixed-size (512 tokens, 15% overlap)
Iterate based on document characteristics
Test with domain-specific documents before deployment

Pitfalls to Avoid

Over-chunking: context-poor small chunks
Under-chunking: missing information in oversized chunks
Ignoring semantic boundaries and document structure
One-size-fits-all for diverse content types

Constraints and Warnings

Resource Considerations

Semantic methods require significant compute resources
Late chunking needs long-context embedding models
Complex strategies increase processing latency
Monitor memory for large document batches

Quality Requirements

Validate semantic coherence post-processing
Test with representative documents before deployment
Ensure chunks maintain standalone meaning
Implement error handling for malformed content

References

strategies.md - Detailed strategies
implementation.md - Implementation guidelines
evaluation.md - Performance metrics
tools.md - Libraries and frameworks
research.md - Research papers
advanced-strategies.md - 11 advanced methods
semantic-methods.md - Semantic approaches
visualization-tools.md - Visualization tools

FAQ

What does the chunking-strategy skill do?

How do I install chunking-strategy?

Run npx skills add https://github.com/giuseppe-trisciuoglio/developer-kit --skill chunking-strategy in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/chunking-strategy/ (personal) or .claude/skills/chunking-strategy/ (project).

Which AI agents can use chunking-strategy?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is chunking-strategy safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains chunking-strategy?

It is published by giuseppe-trisciuoglio in giuseppe-trisciuoglio/developer-kit. It has 3,849 recorded installs.