gke-inference
Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers. Use when deploying GKE inference servers, configuring GKE GPU resources for inference, or deploying LLMs on GKE. Don't use for migrating existing AI workloads to GKE (use google-cloud-solution-guided-gke-ai-migration), GKE RAG with Cloud SQL/AlloyDB (use google-cloud-solution-rag-enterprise-search-gke-sqldb), or batch/HPC (use gke-batch-hpc).
- Installs
- 6.9K
- Last 24h
- +73
- Trending rank
- #2560
- Security
- Pass
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/google/skills --skill gke-inference- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads gke-inference automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
Security audits
Independent scans of the skill's instructions and bundled files.
SKILL.md
The instructions your agent reads when the skill is triggered.
GKE AI/ML Inference
Routing Note: For migrating existing AI workloads to GKE, open
google-cloud-solution-guided-gke-ai-migration/SKILL.md. For GKE RAG with Cloud SQL or AlloyDB (pgvector), opengoogle-cloud-solution-rag-enterprise-search-gke-sqldb/SKILL.md.
This reference covers deploying AI/ML inference workloads on GKE using Google's Inference Quickstart (GIQ) and best practices for LLM serving.
MCP Tools:
apply_k8s_manifest,get_k8s_resource,get_k8s_logs,get_k8s_rollout_status,describe_k8s_resource,list_k8s_events. CLI-only:gcloud container ai profiles *
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-inference)"
and append agent-skills/1.0 (skill:gke-inference) to User-Agent headers.
When to Use
- Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
- Generate optimized Kubernetes manifests for inference
- Select GPU/TPU accelerators for model serving
- Configure autoscaling for LLM inference
Prerequisites
- A golden path GKE Autopilot cluster (GPU workloads are supported via ComputeClasses and NAP)
gcloudCLI authenticated- Sufficient GPU/TPU quota in the target region
Workflow
1. Discovery: Find Models and Hardware
# List all supported models
gcloud container ai profiles models list --quiet
# Find valid accelerator/server combinations for a model
gcloud container ai profiles list --model=<MODEL_NAME> --quiet
# Example: what can run Gemma 2 9B?
gcloud container ai profiles list --model=gemma-2-9b-it --quiet
2. Generate Manifest
gcloud container ai profiles manifests create \
--model=<MODEL_NAME> \
--model-server=<SERVER> \
--accelerator-type=<ACCELERATOR> \
--target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml
Parameters:
--model: Model ID (e.g.,gemma-2-9b-it,llama-3-8b)--model-server: Inference server (vllm,tgi,triton,tensorrt-llm)--accelerator-type: GPU/TPU type (nvidia-l4,nvidia-tesla-a100,nvidia-h100-80gb)--target-ntpot-milliseconds: Target Normalized Time Per Output Token (optional, for latency optimization)
Example:
gcloud container ai profiles manifests create \
--model=gemma-2-9b-it \
--model-server=vllm \
--accelerator-type=nvidia-l4 \
--target-ntpot-milliseconds=50 --quiet > inference.yaml
3. Review and Deploy
# Review for placeholders (HF tokens, PVCs)
cat inference.yaml
# Deploy
kubectl apply -f inference.yaml
# Monitor
kubectl get pods -w
kubectl logs -f <POD_NAME>
Some models require Hugging Face tokens. Create a Kubernetes Secret and reference it in the manifest.
GPU ComputeClass for Inference
For Autopilot clusters, create a ComputeClass to target GPU nodes:
apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
name: l4-inference
spec:
priorities:
- machineFamily: g2
gpu:
type: nvidia-l4
count: 1
minCores: 4
minMemoryGb: 16
Accelerator Selection Guide
| Accelerator | Best For | Memory | Relative Cost |
|---|---|---|---|
| NVIDIA T4 | Budget inference, | 16 GB | Lowest |
| : : lightweight legacy : : : | |||
| : : models : : : | |||
| NVIDIA L4 (G2) | Small-medium model | 24 GB | Low |
| : : inference, video, : : : | |||
| : : graphics : : : | |||
| NVIDIA RTX PRO 6000 | Multimodal AI, | 96 GB | Medium |
| : (G4) : high-fidelity 3D, : : : | |||
| : : fine-tuning : : : | |||
| Cloud TPU v5e | Cost-effective | Varies | Medium |
| : : transformer inference : : : | |||
| Cloud TPU v5p | High-performance | Varies | High |
| : : training : : : | |||
| Cloud TPU v6e | High-efficiency next-gen | 32 GB/chip | Medium-High |
| : (Trillium) : training & serving : : : | |||
| Cloud TPU v7x | Ultra-scale inference & | 192 GB/chip | High |
| : (Ironwood) : agentic workflows : : : | |||
| NVIDIA A100 | Large model inference, | 40/80 GB | High |
| : : enterprise ML : : : | |||
| NVIDIA H100 / H200 | Frontier model training, | 80/141 GB | Highest |
| : : high throughput : : : | |||
| NVIDIA B200 (A4) | Blackwell-scale | 192 GB | Highest |
| : : training, FP4 precision : : : | |||
| NVIDIA GB200 (A4X) | Rack-scale AI (Grace | Massive | Highest |
| : : Blackwell Superchip) : : : |
Autoscaling LLM Inference
GPU-based autoscaling
Use custom metrics for GPU utilization:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llm-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llm-server
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: gpu_duty_cycle
target:
type: AverageValue
averageValue: "80"
Best practices for inference autoscaling
- Use DCGM metrics: Golden path enables DCGM monitoring for GPU utilization metrics
- Set appropriate minReplicas: At least 1 for always-on serving; 0 for batch/on-demand
- Tune scale-down delay: LLM model loading is slow; use longer stabilization windows
- Consider queue depth: Scale on pending requests rather than pure GPU utilization for latency-sensitive workloads
Optimization Tips
- Quantization: Use quantized models (GPTQ, AWQ) to reduce GPU memory and increase throughput
- Batching: Configure model server batch size for throughput vs latency trade-off
- Tensor parallelism: Split large models across multiple GPUs within a node
- KV cache optimization: Tune
--gpu-memory-utilizationin vLLM for KV cache allocation
Troubleshooting
| Issue | Cause | Fix |
|---|---|---|
| Invalid | Unsupported tuple | Re-run `gcloud container ai |
| : model/accelerator : : profiles list : | ||
| : combination : : --model=<MODEL>` : | ||
| GPU quota exceeded | Regional quota limit | Request quota increase or |
| : : : try a different region : | ||
| OOM on GPU | Model too large for | Use larger GPU, enable |
| : : accelerator : quantization, or use tensor : | ||
| : : : parallelism : | ||
| Slow cold start | Large model loading from | Use local SSD for model |
| : : registry : caching; pre-pull images : |
FAQ
What does the gke-inference skill do?
Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers. Use when deploying GKE inference servers, configuring GKE GPU resources for inference, or deploying LLMs on GKE. Don't use for migrating existing AI workloads to GKE (use google-cloud-solution-guided-gke-ai-migration), GKE RAG with Cloud SQL/AlloyDB (use google-cloud-solution-rag-enterprise-search-gke-sqldb), or batch/HPC (use gke-batch-hpc).
How do I install gke-inference?
Run npx skills add https://github.com/google/skills --skill gke-inference in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/gke-inference/ (personal) or .claude/skills/gke-inference/ (project).
Which AI agents can use gke-inference?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is gke-inference safe to install?
Security partners have reviewed it: Gen Agent Trust Hub passed; Socket passed; Snyk passed. As with any skill, read the SKILL.md and bundled scripts before installing.
Who maintains gke-inference?
It is published by google in google/skills. It has 6,944 recorded installs.
More from google/skills
bigquery-basics
google/skills
Manages datasets, tables, and jobs in BigQuery. Use when you need to interact with BigQuery, run SQL queries, manage BigQuery resources (datasets, tables, views), or perform basic data ingestion and analysis.
cloud-run-basics
google/skills
Manages Cloud Run services, jobs, and worker pools. Use when you need to deploy applications responding to HTTP requests (services), run event-triggered or scheduled tasks (jobs), or handle always-on pull-based background processing (worker pools).
gemini-api
google/skills
Use when the user asks about using Gemini in an enterprise environment or explicitly mentions Vertex AI, Google Cloud, or Agent Platform. Guides the usage of the Gemini API on Agent Platform with the Google Gen AI SDK. Covers SDK usage (Python, JS/TS, Go, Java, C), capabilities like multimodal inputs, tools, media generation, caching, batch prediction, and Live API.
google-cloud-recipe-auth
google/skills
Provides expert guidance on Identity and Access Management (IAM) and authenticating and authorizing to Google Cloud services and APIs, covering human users, service identities, Application Default Credentials (ADC), and best practices for secure access.
google-cloud-recipe-onboarding
google/skills
Guides a developer's first steps on Google Cloud, covering account creation, billing setup, project management, and deploying a first resource. Use when a new developer wants to initialize their first Google Cloud project, configure billing, and verify deployment. Don't use for enterprise organization setup (use Google Cloud Setup guided flow for that instead). Don't use for complex multi-project architectures.
cloud-sql-basics
google/skills
This file generates or explains Cloud SQL resources. Use this file when the user asks to create a Cloud SQL instance or database for MySQL, PostgreSQL, or SQL Server. Cloud SQL manages third-party MySQL, PostgreSQL, and SQL Server instances as resources in Cloud SQL. For example, when Cloud SQL creates an open-source MySQL instance, the resulting resource is a Cloud SQL for MySQL instance that Google Cloud manages. Cloud SQL handles backups, high availability, and secure connectivity for relational database workloads.
Popular ai & agents skills
google-agents-cli-publish
google/agents-cli
This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).
google-agents-cli-eval
google/agents-cli
This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).
skill-creator
anthropics/skills
Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.
codex-pet
prime-skills/runcomfy-agent-skills
Codex Pet generator on RunComfy. Build a Codex-compatible Codex Pet spritesheet.webp + pet.json from a single reference image, drop it into ${CODEXHOME:-$HOME/.codex}/pets/ / and Codex picks it up as a custom Codex Pet next to the 8 built-ins. This skill produces the exact Codex Pet atlas Codex expects (1536x1872 PNG/WebP, 8 cols x 9 rows, 192x208 cells, 9 animation states — idle, running-right, running-left, waving, jumping, failed, waiting, running, review). Calls OpenAI GPT Image 2 edit ONCE via the local RunComfy CLI as runcomfy run openai/gpt-image-2/edit to produce a canonical Codex Pet pose, then assembles all 9 animation rows programmatically with ImageMagick micro-transforms — no Codex Pro, no $imagegen, no OPENAIAPIKEY required, only RUNCOMFYTOKEN. Triggers on "codex pet", "create codex pet", "make codex pet", "hatch codex pet", "/hatch image", "desktop pet codex", "codex pets", "spritesheet.webp", or any explicit ask to build a custom pet for OpenAI Codex.
full-output-enforcement
leonxlnx/taste-skill
Overrides default LLM truncation behavior. Enforces complete code generation, bans placeholder patterns, and handles token-limit splits cleanly. Apply to any task requiring exhaustive, unabridged output.
executing-plans
obra/superpowers
Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available