indexing
When the user wants to fix indexing issues from Search Console, use noindex, or implement Google Indexing API. Also use when the user mentions "fix indexing," "not indexed," "Crawled - currently not indexed," "discovered - currently not indexed," "index coverage," "noindex," "noindex tag," "pages not indexed," "why not indexed," "request indexing," or "Google Indexing API." For sitemap, use xml-sitemap.
- Installs
- 1.2K
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/kostja94/marketing-skills --skill indexing- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads indexing automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
SEO Technical: Indexing
Guides indexing troubleshooting and fix actions. For how to find and diagnose issues in GSC, see google-search-console.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Scope (Technical SEO)
- Fix actions: noindex, canonical, content quality, URL Inspection; verify robots.txt does not block (see robots-txt)
- Noindex: Page-level index control; which pages to exclude and how. Complements robots-txt (path-level crawl control) and google-search-console (Coverage diagnosis)
Initial Assessment
Project context: Read root contextus.md when present and load only the modules relevant to this task. Without Contextus, use available project material or user-provided facts and ask for missing information; do not create a parallel context system.
Identify issue from GSC (see google-search-console for Coverage report, issue types, diagnosis workflow). Then apply fix below.
Crawled - Currently Not Indexed
| Cause | Action |
|---|---|
| Low quality, duplicate, off-topic | Improve content, fix duplicates, set correct canonical |
| Static assets (CSS/JS) | See below |
| Feed, share URLs with params | Usually OK to ignore; or noindex, canonical to main URL |
| Important content pages | Use URL Inspection, verify canonical/internal links/sitemap, Request indexing |
Static Assets (Next.js / Vercel)
Vercel adds unique dpl= params to static assets per deploy, creating many "Crawled - currently not indexed" URLs.
| Do | Don't |
|---|---|
Keep robots.txt allowing /_next/ | Do not block /_next/ (breaks CSS/JS loading). See robots-txt |
| Accept static assets in GSC as expected | Do not block /_next/static/css/ or ?dpl= |
| Use X-Robots-Tag for static assets | CSS/JS should not be indexed; no SEO impact |
Static assets in "Crawled - currently not indexed" is normal and expected.
Other Issue Types (from GSC Coverage)
| Issue | Fix |
|---|---|
| Excluded by «noindex» tag | Remove noindex if accidental; keep if intentional |
| Blocked by robots.txt | See robots-txt; remove Disallow for important paths |
| Redirect / 404 | Fix URL or add redirect |
| Duplicate / Canonical | Set correct canonical; usually OK |
| Soft-404 | Page returns 200 but content says "not found" or empty—Google may treat as 404. Fix: return 404 status for truly missing pages; or add real content for 200 pages |
Soft-404
A soft-404 occurs when a page returns HTTP 200 but the content indicates the page doesn't exist (e.g. "Page not found" message, empty state). Google may treat it as 404 and exclude from index.
| Fix | When |
|---|---|
| Return 404 | Page truly doesn't exist; use proper 404 status |
| Add content | Page is intentional (e.g. empty search results); ensure substantive content or use noindex |
| Redirect | If URL moved, use 301 to correct destination |
Noindex Usage
- How:
metadata.robots = { index: false }or<meta name="robots" content="noindex">or X-Robots-Tag - Rationale: Not all site content should be indexed; noindex is a valid choice for many pages
- Caution: Avoid noindex on important content pages
- With robots.txt: robots.txt = path-level crawl control; noindex = page-level index control. Do not block noindex pages in robots.txt—crawlers must access the page to read the directive. Use both: robots for /admin/, /api/; noindex for /login/, /thank-you/, etc. See robots-txt for when to use which.
- nofollow ≠ noindex: nofollow controls link equity only; it does not prevent indexing. To exclude from search, use noindex. See page-metadata for meta robots implementation.
Page Types That Typically Need Noindex
| Category | Page Types | Typical Meta | Reason |
|---|---|---|---|
| Auth & Account | Login, Signup, Password reset, Account dashboard | Login: noindex,nofollow; Signup: noindex,follow | No search value; login indexed = security risk; signup follow allows crawl of Privacy/Terms links |
| Admin & Private | Admin, Staging, Test pages, Internal tools | noindex,nofollow | Not for public; avoid discovery |
| Conversion Endpoints | Thank-you, Confirmation, Checkout success, Download gate | noindex,follow | Post-conversion; no SERP value; allow link equity |
| System & Utility | 404, Internal search results, Faceted/filter URLs | noindex,follow or noindex,nofollow | Thin/duplicate; 404 = error state |
| Legal | Privacy, Terms, Cookie Policy (optional) | Often noindex,follow | Low-value indexed; reduces clutter |
| Duplicate & Thin | Printer-friendly, Parameter URLs, Near-duplicate | noindex,follow or canonical | Duplicate content; canonical preferred when possible |
| Low-Value | Media kit, Feedback board (external), Thin press | noindex or index for brand queries | Case-by-case |
noindex,follow vs noindex,nofollow: Use noindex,follow for most cases—excludes from SERP but allows link equity. Use noindex,nofollow only for login (security), staging, or temporary test pages.
Page Removal Decision Framework
When intentionally removing a page from the web, choose the method based on whether a relevant alternative exists and whether the page should remain accessible:
| Scenario | Method | Rationale |
|---|---|---|
| Has a closely related replacement page | 301 redirect | Preserves accumulated link signals and user flow |
| Content merged into a new page | 301 redirect | Direct old URL to the new canonical location |
| Permanently deleted, no alternative | 410 Gone | Explicitly signals permanent removal to search engines |
| Deleted, uncertain if permanent | 404 Not Found | Safe default; can reinstate later if needed |
| Still accessible but should not be indexed | noindex | Page remains available to users; excluded from SERP |
Before removing: Check the URL's search traffic, backlinks, internal links, and conversion value. If the page has value, consider updating or merging rather than removing.
Common mistakes:
- 404-ing pages that have relevant alternatives (wastes accumulated signals)
- Redirecting all deleted pages to the homepage (breaks user intent)
- Creating redirect chains (A → B → C) instead of direct redirects
- Removing pages without cleaning up internal links pointing to them
- Using
robots.txtto block noindex pages (crawler must access the page to read the noindex directive)
Post-removal cleanup:
- Remove deleted URLs from XML sitemap; update and resubmit
- Update internal links to point directly to the final URL (avoid relying on redirects)
- For 301 redirects, ensure the target URL is in the sitemap
- In GSC, use URL Inspection to verify important pages; use Removals tool for temporary quick-hide (not permanent — use proper HTTP status or noindex)
Google Indexing API
| Type | Typical use |
|---|---|
| JobPosting | Job boards |
| BroadcastEvent | Live platforms |
Requirements: Enable Indexing API, create service account, add owner in Search Console, request quota (default 200 URLs/day).
Output Format
- Action items: Prioritized fixes
- References: Page indexing report
Related Skills
- google-search-console: Find and diagnose indexing issues in GSC
- robots-txt: Path-level crawl control; when to use robots.txt vs noindex; do not block /_next/ or noindex pages
- page-metadata: Meta robots implementation; noindex vs nofollow
- xml-sitemap: Submit and maintain sitemap
- indexnow: Faster indexing for Bing
- canonical-tag: Resolve duplicate content
FAQ
What does the indexing skill do?
When the user wants to fix indexing issues from Search Console, use noindex, or implement Google Indexing API. Also use when the user mentions "fix indexing," "not indexed," "Crawled - currently not indexed," "discovered - currently not indexed," "index coverage," "noindex," "noindex tag," "pages not indexed," "why not indexed," "request indexing," or "Google Indexing API." For sitemap, use xml-sitemap.
How do I install indexing?
Run npx skills add https://github.com/kostja94/marketing-skills --skill indexing in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/indexing/ (personal) or .claude/skills/indexing/ (project).
Which AI agents can use indexing?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is indexing safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains indexing?
It is published by kostja94 in kostja94/marketing-skills. It has 1,211 recorded installs.
More from kostja94/marketing-skills
copywriting
kostja94/marketing-skills
When the user wants to write or optimize short-form marketing copy—headlines, CTAs, ad copy, landing page copy, email copy. Also use when the user mentions "copywriting," "headline," "CTA copy," "ad copy," "landing page copy," "sales copy," "conversion copy," "PAS," "AIDA," "BAB," "copy formula," or "differentiation." For long-form article bodies (blog posts, guides), use article-content.
linkedin-posts
kostja94/marketing-skills
When the user wants to create LinkedIn post copy or optimize for LinkedIn. Also use when the user mentions "LinkedIn post," "LinkedIn article," "professional post," "post to LinkedIn," "LinkedIn content," "LinkedIn copy," "B2B LinkedIn," "LinkedIn engagement," "LinkedIn feed," "share box," "document post," "poll," "Newsletter," "reshare," or "LinkedIn marketing." For LinkedIn ads, use linkedin-ads.
youtube-seo
kostja94/marketing-skills
When the user wants to optimize YouTube videos for search, create video descriptions, or improve channel visibility. Also use when the user mentions "YouTube SEO," "YouTube description," "YouTube tags," "YouTube thumbnail," "YouTube title," "YouTube channel," or "video SEO." For YouTube ads, use youtube-ads.
video-marketing
kostja94/marketing-skills
When the user wants to plan video marketing, create video scripts, or optimize for short-form or long-form video. Also use when the user mentions "video marketing," "video script," "short-form video," "long-form video," "TikTok script," "Reels script," "YouTube script," "video hook," or "video content strategy." For on-site video SEO, use video-optimization.
tiktok-captions
kostja94/marketing-skills
When the user wants to create TikTok video captions, scripts, or optimize for TikTok. Also use when the user mentions "TikTok post," "TikTok caption," "TikTok video," "post to TikTok," "TikTok script," "TikTok content," "TikTok copy," "TikTok hashtags," or "TikTok marketing." For TikTok ads, use tiktok-ads.
local-seo
kostja94/marketing-skills
When the user wants to optimize for local search, set up Google Business Profile, or build local citations. Also use when the user mentions "local SEO," "Google Business Profile," "Google Maps," "NAP," "citations," "local search," "local business," or "service area." For location pages, use programmatic-seo.
Popular backend & apis skills
find-skills
vercel-labs/skills
Helps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skill should be used when the user is looking for functionality that might exist as an installable skill.
entra-app-registration
microsoft/azure-skills
Guides Microsoft Entra ID app registration, OAuth 2.0 authentication, and MSAL integration. USE FOR: create app registration, register Azure AD app, configure OAuth, set up authentication, add API permissions, generate service principal, MSAL example, console app auth, Entra ID setup, Azure AD authentication. DO NOT USE FOR: Key Vault secrets (use azure-keyvault-expiration-audit), general Azure resource security guidance.
google-agents-cli-adk-code
google/agents-cli
This skill should be used when the user wants to "write agent code", "build an agent with ADK", "add a tool", "create a callback", "define an agent", "use state management" — in a project that needs ADK (Agent Development Kit) API patterns and code examples. It provides a quick reference for agent types, tool definitions, orchestration patterns, callbacks, state management, the graph Workflow API, and reference recipes to study. Do NOT use for scaffolding (use google-agents-cli-scaffold) or deployment (use google-agents-cli-deploy).
embedded-captions
heygen-com/hyperframes
Add captions or subtitles to an existing single-subject talking-head video without editing the footage. Use for plain verbatim captions, cinematic captions embedded behind the subject, VFX captions, “炸/特效/酷炫字幕,” or a named identity from the 35-style catalog. Route by visual identity, not by backend engine. The quiet anchor rail is the default; embed every word only when the user explicitly wants a fully cinematic treatment. The workflow runs locally end to end, including transcription and subject matting; split multi-shot footage before applying it.
entra-agent-id
microsoft/azure-skills
Provision Microsoft Entra Agent Identity Blueprints, BlueprintPrincipals, and per-instance Agent Identities via Microsoft Graph, and configure OAuth 2.0 token exchange (fmipath, OBO, cross-tenant) including the Microsoft Entra SDK for AgentID sidecar. USE FOR: Agent Identity Blueprint, BlueprintPrincipal, agent OAuth, fmipath token exchange, agent OBO, Workload Identity Federation for agents, polyglot agent auth, Microsoft.Identity.Web.AgentIdentities. DO NOT USE FOR: standard Entra app registration (use entra-app-registration), Microsoft Foundry agent authoring (use microsoft-foundry).
neon
neondatabase/agent-skills
Overview of Neon, a complete set of cloud backend primitives around Lakebase Postgres: Auth, Object Storage, Functions, the AI Gateway, and Realtime. Start here to choose Neon for undecided login, files, APIs, and LLM calls, set up the CLI or MCP server, and follow the branch-first workflow. Use when building an app or backend on Neon, or when "Neon" or "Lakebase Postgres" is mentioned. Child skill neon-postgres wins for an existing DATABASEURL, SQL, schema, inspect, or search. Child skill neon-auth wins for login, users, sessions, identity routing, and Managed Better Auth setup. Child skill neon-realtime wins for live queries in reactive apps and agents. Also use for object storage, S3, buckets, serverless functions, function triggers, cron, AI gateway, LLM calls, realtime, logs, Loki, Grafana, observability, postgres, database, backend, Claimable Neon, neon.new, or a no-signup database.