
content-similarity-checker
Compare document similarity using TF-IDF, cosine similarity, and Jaccard index. Use for plagiarism detection, duplicate finding, or content matching.
- Installs
- 74
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/dkyazzentwatwa/chatgpt-skills --skill content-similarity-checker- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads content-similarity-checker automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Content Similarity Checker
Compare documents and text for similarity using multiple algorithms.
Features
- Cosine Similarity: TF-IDF based comparison
- Jaccard Similarity: Set-based comparison
- Levenshtein Distance: Edit distance for short texts
- Batch Comparison: Compare multiple documents
- Similarity Matrix: Pairwise comparison of all documents
- Reports: Detailed similarity reports
Quick Start
from similarity_checker import SimilarityChecker
checker = SimilarityChecker()
# Compare two texts
score = checker.compare(
"The quick brown fox jumps over the lazy dog",
"A fast brown fox leaps over a sleepy dog"
)
print(f"Similarity: {score:.2%}")
# Compare documents
score = checker.compare_files("doc1.txt", "doc2.txt")
CLI Usage
# Compare two texts
python similarity_checker.py --text1 "Hello world" --text2 "Hello there world"
# Compare two files
python similarity_checker.py --file1 doc1.txt --file2 doc2.txt
# Compare all files in folder
python similarity_checker.py --folder ./documents/ --output matrix.csv
# Use specific algorithm
python similarity_checker.py --file1 doc1.txt --file2 doc2.txt --method jaccard
# Find similar documents (threshold)
python similarity_checker.py --folder ./documents/ --threshold 0.7
# JSON output
python similarity_checker.py --file1 doc1.txt --file2 doc2.txt --json
API Reference
SimilarityChecker Class
class SimilarityChecker:
def __init__(self, method: str = "cosine")
# Text comparison
def compare(self, text1: str, text2: str) -> float
def compare_files(self, file1: str, file2: str) -> float
# Multiple algorithms
def compare_all_methods(self, text1: str, text2: str) -> dict
# Batch comparison
def compare_to_corpus(self, text: str, corpus: list) -> list
def similarity_matrix(self, documents: list) -> pd.DataFrame
def find_duplicates(self, documents: list, threshold: float = 0.8) -> list
# Folder operations
def compare_folder(self, folder: str, threshold: float = None) -> dict
def find_most_similar(self, text: str, folder: str, top_n: int = 5) -> list
# Report
def generate_report(self, output: str) -> str
Similarity Methods
Cosine Similarity (Default)
Best for comparing documents of different lengths:
checker = SimilarityChecker(method="cosine")
score = checker.compare(text1, text2)
# Returns: 0.0 to 1.0
Jaccard Similarity
Good for comparing sets of words/tokens:
checker = SimilarityChecker(method="jaccard")
score = checker.compare(text1, text2)
# Returns: 0.0 to 1.0
Levenshtein (Edit Distance)
Best for short texts, typo detection:
checker = SimilarityChecker(method="levenshtein")
score = checker.compare(text1, text2)
# Returns: 0.0 to 1.0 (normalized)
TF-IDF + Cosine
Advanced: considers term importance:
checker = SimilarityChecker(method="tfidf")
score = checker.compare(text1, text2)
Batch Comparison
Compare to Corpus
checker = SimilarityChecker()
target = "Machine learning is a subset of artificial intelligence."
corpus = [
"AI includes machine learning and deep learning.",
"Python is a programming language.",
"Neural networks power deep learning systems."
]
results = checker.compare_to_corpus(target, corpus)
# Returns:
[
{"index": 0, "similarity": 0.65, "text": "AI includes..."},
{"index": 2, "similarity": 0.42, "text": "Neural networks..."},
{"index": 1, "similarity": 0.12, "text": "Python is..."}
]
Similarity Matrix
documents = [
"Document one content...",
"Document two content...",
"Document three content..."
]
matrix = checker.similarity_matrix(documents)
# Returns DataFrame:
# doc_0 doc_1 doc_2
# doc_0 1.000 0.750 0.320
# doc_1 0.750 1.000 0.410
# doc_2 0.320 0.410 1.000
Find Duplicates
documents = [...] # List of texts
duplicates = checker.find_duplicates(documents, threshold=0.85)
# Returns:
[
{"doc1_index": 0, "doc2_index": 3, "similarity": 0.92},
{"doc1_index": 2, "doc2_index": 7, "similarity": 0.88}
]
Compare All Methods
Get similarity scores from all algorithms:
checker = SimilarityChecker()
results = checker.compare_all_methods(text1, text2)
# Returns:
{
"cosine": 0.82,
"jaccard": 0.65,
"levenshtein": 0.71,
"tfidf": 0.78,
"average": 0.74
}
Folder Operations
Compare All Files in Folder
checker = SimilarityChecker()
results = checker.compare_folder("./documents/")
# Returns:
{
"files": ["doc1.txt", "doc2.txt", "doc3.txt"],
"comparisons": 3,
"similar_pairs": [
{"file1": "doc1.txt", "file2": "doc3.txt", "similarity": 0.87}
],
"matrix": <DataFrame>
}
Find Most Similar to Query
query = "Your search text here..."
results = checker.find_most_similar(query, "./documents/", top_n=5)
# Returns:
[
{"file": "doc3.txt", "similarity": 0.89},
{"file": "doc1.txt", "similarity": 0.72},
...
]
Output Format
Comparison Result
result = checker.compare_with_details(text1, text2)
# Returns:
{
"similarity": 0.82,
"method": "cosine",
"text1_length": 150,
"text2_length": 180,
"common_words": 25,
"unique_words_text1": 10,
"unique_words_text2": 15,
"interpretation": "High similarity - likely related content"
}
Example Workflows
Plagiarism Check
checker = SimilarityChecker()
submission = open("student_paper.txt").read()
results = checker.compare_folder("./source_materials/")
suspicious = [p for p in results["similar_pairs"] if p["similarity"] > 0.6]
if suspicious:
print(f"Warning: Found {len(suspicious)} potentially similar sources")
for p in suspicious:
print(f" {p['file1']} matches {p['file2']}: {p['similarity']:.0%}")
Document Deduplication
checker = SimilarityChecker()
# Load all documents
docs = {}
for file in Path("./articles/").glob("*.txt"):
docs[file.name] = file.read_text()
# Find near-duplicates
duplicates = checker.find_duplicates(list(docs.values()), threshold=0.9)
print(f"Found {len(duplicates)} duplicate pairs")
Content Matching
checker = SimilarityChecker()
query = "Best practices for Python web development"
results = checker.find_most_similar(query, "./blog_posts/", top_n=10)
print("Most relevant articles:")
for r in results:
print(f" {r['file']}: {r['similarity']:.0%} match")
Dependencies
- scikit-learn>=1.3.0
- nltk>=3.8.0
- numpy>=1.24.0
- pandas>=2.0.0
FAQ
What does the content-similarity-checker skill do?
Compare document similarity using TF-IDF, cosine similarity, and Jaccard index. Use for plagiarism detection, duplicate finding, or content matching.
How do I install content-similarity-checker?
Run npx skills add https://github.com/dkyazzentwatwa/chatgpt-skills --skill content-similarity-checker in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/content-similarity-checker/ (personal) or .claude/skills/content-similarity-checker/ (project).
Which AI agents can use content-similarity-checker?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is content-similarity-checker safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains content-similarity-checker?
It is published by dkyazzentwatwa in dkyazzentwatwa/chatgpt-skills. It has 74 recorded installs.
More from dkyazzentwatwa/chatgpt-skills
ocr-document-processor
dkyazzentwatwa/chatgpt-skills
Extract text and structure from scans, images, and scanned PDFs. Use for OCR, searchable PDFs, table extraction, receipt parsing, and business card parsing.
crypto-ta-analyzer
dkyazzentwatwa/chatgpt-skills
Run multi-indicator technical analysis on crypto or market OHLCV data. Use for deterministic trend, momentum, volume, and divergence analysis.
text-summarizer
dkyazzentwatwa/chatgpt-skills
Generate extractive summaries from long text documents. Control summary length, extract key sentences, and process multiple documents.
financial-calculator
dkyazzentwatwa/chatgpt-skills
Run loan, investment, NPV, retirement, savings, and risk calculations with schedules and charts. Use for deterministic financial modeling tasks.
document-converter-suite
dkyazzentwatwa/chatgpt-skills
Convert PDFs, Office docs, markdown, HTML, and tables between editable formats. Use for best-effort document conversion, PDF utilities, form filling, and table extraction.
image-enhancement-suite
dkyazzentwatwa/chatgpt-skills
Process images for cleanup, conversion, metadata, comparison, icons, palettes, collages, and sprite sheets. Use for single-image or batch image workflows.
Popular other skills
hyperframes-cli
heygen-com/hyperframes
The HyperFrames CLI by job - scaffold, check, look at frames, edit the timeline, preview, render locally or in the cloud, publish, project history - plus reading its JSON and recovering from build or render failures.
triage
mattpocock/skills
Move issues and external PRs through a state machine of triage roles, categorise, verify, grill if needed, and write agent-ready briefs.
ask-matt
mattpocock/skills
Ask which skill or flow fits your situation. A router over the skills in this repo.
to-questionnaire
mattpocock/skills
Turn a decision you can't fully answer into a questionnaire for someone else to fill in.
loop-me
mattpocock/skills
Grill me about specs for the workflows I want to build, within this workspace.