Folder Organization Best Practices

Expert guidance for organizing project directories, establishing file naming conventions, and maintaining clean, navigable project structures for research and development work.

When to Use This Skill

• Setting up new projects
• Reorganizing existing projects
• Establishing team conventions
• Creating reproducible research structures
• Managing data-intensive projects

Core Principles

1. Predictability - Standard locations for common file types
2. Scalability - Structure grows gracefully with project
3. Separation of Concerns - Code, data, documentation, outputs separated
4. Discoverability - Easy for others (and future you) to navigate
5. Version Control Friendly - Large/generated files excluded appropriately

Quick Reference: Standard Project Structure

Use the structure matching your project type. Full templates in project-structures.md.

Research/Analysis: data/{raw,processed,external}, notebooks/, src/, scripts/, tests/, docs/, results/, config/

Development: src/package_name/, tests/, docs/, examples/, .github/workflows/

Bioinformatics: data/{raw,reference,processed}, workflows/, config/, scripts/, results/, logs/

Data Analysis with Notebooks: notebooks/, figures/, data/, tests/, scripts/, docs/, archives/

Add MANIFEST.md files for token-efficient navigation (see manifest-system.md).

File Naming Rules

Detailed conventions in naming-conventions.md. Key rules:

1. Use lowercase with hyphens or underscores (not spaces or CamelCase)
2. Be descriptive but concise - process-telomere-data.py not script.py
3. Use consistent separators - hyphens for files, underscores for Python modules
4. Include version/date for important outputs - report-2026-01-23.pdf
5. Zero-pad sequences - 01-exploration.ipynb, 02-analysis.ipynb
6. Standardize session notes in session-saves/ directory

Version Control

DO commit: Source code, documentation, config files, small test data (<1MB), requirements files, READMEs

DON'T commit: Large data files, generated outputs, environment dirs, logs, temp files, API keys/secrets

.gitignore Template

# Claude Code
.claude/

# Python
__pycache__/
*.py[cod]
.venv/
venv/

# Jupyter
.ipynb_checkpoints/

# Data
data/raw/
data/processed/
*.fastq.gz
*.bam

# Outputs
results/
outputs/
*.png
*.pdf

# Logs & Environment
logs/
*.log
.env
.DS_Store

Data Organization

• Never modify raw data - keep originals in data/raw/ (read-only if possible)
• Document data provenance (source, download date)
• Use hierarchy: raw/ -> interim/ -> processed/ -> external/

Common Anti-Patterns

Avoid these patterns:

• Flat structure: Everything in root directory with no organization
• Ambiguous naming: notebook1.ipynb, test.ipynb, analysis_new.ipynb
• Mixed concerns: Data files and output images inside src/ directory

Deprecation Strategy

Deprecate rather than delete to maintain history and recovery options. Full guide in deprecation-guide.md.

Key pattern:

1. Create deprecated/ subdirectory with descriptive name
2. Move old files there
3. Write README.md explaining what, when, why, and how to recover
4. For scattered deprecated dirs, consolidate to single root deprecated/

Project Reorganization

Full systematic guide in reorganization-guide.md. Critical steps:

1. Update all file paths in scripts and notebooks after moving files
2. Search for references: grep -r "\.json\|\.csv\|\.png" --include="*.ipynb" --include="*.py"
3. Verify file counts match expectations (no files lost)
4. Remove temp files (.bak, .ipynb_checkpoints)
5. Test that notebooks/scripts still run correctly

Documentation Standards

Full guide in documentation-organization.md. Key points:

• Every project needs a README with: description, installation, usage, structure, data, results
• After major changes, create dated summary documents
• Consolidate scattered docs into documentation/ directory
• Keep only README.md, LICENSE, .gitignore in project root

MANIFEST System

Token-efficient project navigation system. Full documentation in manifest-system.md.

• Provides 85-90% token reduction for session startup
• Root MANIFEST.md gives complete project overview in ~1,500 tokens
• Subdirectory MANIFESTs for data/, figures/, scripts/, documentation/
• Update at end of every session with /update-manifest

Project Cleanup: Identifying Essential Files

Detailed process in deprecation-guide.md (section "Project Cleanup"). Summary:

1. Analyze notebooks to find referenced figures
2. Map figures to generating scripts
3. Move unused files to deprecated/ with descriptive subdirectory names
4. Document what was kept in MINIMAL_ESSENTIAL_FILES.md
5. Verify notebooks still work with cleaned structure

Integration with Other Skills

• python-environment - Environment setup and management
• claude-collaboration - Team workflow best practices
• jupyter-notebook-analysis - Notebook organization standards
• data-backup - Backup system should include MANIFESTs
• project-sharing - Include MANIFESTs in shared packages

Quick Setup

# Create standard research project structure
mkdir -p data/{raw,processed,external} notebooks scripts src tests docs results config
touch README.md .gitignore environment.yml

Supporting Files

References and Resources

• Cookiecutter Data Science
• A Quick Guide to Organizing Computational Biology Projects
• Good Enough Practices in Scientific Computing