MCPInstall

vlm

by answerzhaoMedia & Creative

Implement vision-based AI chat capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to analyze images, describe visual content, or create applications that combine image understanding with conversational AI. Supports image URLs and base64 encoded images for multimodal interactions.

Installs
343
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/answerzhao/agent-skills --skill vlm
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads vlm automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

VLM(Vision Chat) Skill

This skill guides the implementation of vision chat functionality using the z-ai-web-dev-sdk package, enabling AI models to understand and respond to images combined with text prompts.

Skills Path

Skill Location: {project_path}/skills/VLM

this skill is located at above path in your project.

Reference Scripts: Example test scripts are available in the {Skill Location}/scripts/ directory for quick testing and reference. See {Skill Location}/scripts/vlm.ts for a working example.

Overview

Vision Chat allows you to build applications that can analyze images, extract information from visual content, and answer questions about images through natural language conversation.

IMPORTANT: z-ai-web-dev-sdk MUST be used in backend code only. Never use it in client-side code.

Prerequisites

The z-ai-web-dev-sdk package is already installed. Import it as shown in the examples below.

CLI Usage (For Simple Tasks)

For simple image analysis tasks, you can use the z-ai CLI instead of writing code. This is ideal for quick image descriptions, testing vision capabilities, or simple automation.

Basic Image Analysis

# Describe an image from URL
z-ai vision --prompt "What's in this image?" --image "https://example.com/photo.jpg"

# Using short options
z-ai vision -p "Describe this image" -i "https://example.com/image.png"

Analyze Local Images

# Analyze a local image file
z-ai vision -p "What objects are in this photo?" -i "./photo.jpg"

# Save response to file
z-ai vision -p "Describe the scene" -i "./landscape.png" -o description.json

Multiple Images

# Analyze multiple images at once
z-ai vision \
  -p "Compare these two images" \
  -i "./photo1.jpg" \
  -i "./photo2.jpg" \
  -o comparison.json

# Multiple images with detailed analysis
z-ai vision \
  --prompt "What are the differences between these images?" \
  --image "https://example.com/before.jpg" \
  --image "https://example.com/after.jpg"

With Thinking (Chain of Thought)

# Enable thinking for complex visual reasoning
z-ai vision \
  -p "Count the number of people in this image and describe their activities" \
  -i "./crowd.jpg" \
  --thinking \
  -o analysis.json

Streaming Output

# Stream the vision analysis
z-ai vision -p "Describe this image in detail" -i "./photo.jpg" --stream

CLI Parameters

  • --prompt, -p <text>: Required - Question or instruction about the image(s)
  • --image, -i <URL or path>: Optional - Image URL or local file path (can be used multiple times)
  • --thinking, -t: Optional - Enable chain-of-thought reasoning (default: disabled)
  • --output, -o <path>: Optional - Output file path (JSON format)
  • --stream: Optional - Stream the response in real-time

Supported Image Formats

  • PNG (.png)
  • JPEG (.jpg, .jpeg)
  • GIF (.gif)
  • WebP (.webp)
  • BMP (.bmp)

When to Use CLI vs SDK

Use CLI for:

  • Quick image analysis
  • Testing vision model capabilities
  • One-off image descriptions
  • Simple automation scripts

Use SDK for:

  • Multi-turn conversations with images
  • Dynamic image analysis in applications
  • Batch processing with custom logic
  • Production applications with complex workflows

Recommended Approach

For better performance and reliability, use base64 encoding to pass images to the model instead of image URLs.

Supported Content Types

The Vision Chat API supports three types of media content:

1. image_url - For Image Files

Use this type for static images (PNG, JPEG, GIF, WebP, etc.)

{
    role: 'user',
    content: [
        { type: 'text', text: prompt },
        { type: 'image_url', image_url: { url: imageUrl } }
    ]
}

2. video_url - For Video Files

Use this type for video content (MP4, AVI, MOV, etc.)

{
    role: 'user',
    content: [
        { type: 'text', text: prompt },
        { type: 'video_url', video_url: { url: videoUrl } }
    ]
}

3. file_url - For Document Files

Use this type for document files (PDF, DOCX, TXT, etc.)

{
    role: 'user',
    content: [
        { type: 'text', text: prompt },
        { type: 'file_url', file_url: { url: fileUrl } }
    ]
}

Note: You can combine multiple content types in a single message. For example, you can include both text and multiple images, or text with both an image and a document.

Basic Vision Chat Implementation

Single Image Analysis

import ZAI from 'z-ai-web-dev-sdk';

async function analyzeImage(imageUrl, question) {
  const zai = await ZAI.create();

  const response = await zai.chat.completions.createVision({
    messages: [
      {
        role: 'user',
        content: [
          {
            type: 'text',
            text: question
          },
          {
            type: 'image_url',
            image_url: {
              url: imageUrl
            }
          }
        ]
      }
    ],
    thinking: { type: 'disabled' }
  });

  return response.choices[0]?.message?.content;
}

// Usage
const result = await analyzeImage(
  'https://example.com/product.jpg',
  'Describe this product in detail'
);
console.log('Analysis:', result);

Multiple Images Analysis

import ZAI from 'z-ai-web-dev-sdk';

async function compareImages(imageUrls, question) {
  const zai = await ZAI.create();

  const content = [
    {
      type: 'text',
      text: question
    },
    ...imageUrls.map(url => ({
      type: 'image_url',
      image_url: { url }
    }))
  ];

  const response = await zai.chat.completions.createVision({
    messages: [
      {
        role: 'user',
        content: content
      }
    ],
    thinking: { type: 'disabled' }
  });

  return response.choices[0]?.message?.content;
}

// Usage
const comparison = await compareImages(
  [
    'https://example.com/before.jpg',
    'https://example.com/after.jpg'
  ],
  'Compare these two images and describe the differences'
);

Base64 Image Support

import ZAI from 'z-ai-web-dev-sdk';
import fs from 'fs';

async function analyzeLocalImage(imagePath, question) {
  const zai = await ZAI.create();

  // Read image file and convert to base64
  const imageBuffer = fs.readFileSync(imagePath);
  const base64Image = imageBuffer.toString('base64');
  const mimeType = imagePath.endsWith('.png') ? 'image/png' : 'image/jpeg';

  const response = await zai.chat.completions.createVision({
    messages: [
      {
        role: 'user',
        content: [
          {
            type: 'text',
            text: question
          },
          {
            type: 'image_url',
            image_url: {
              url: `data:${mimeType};base64,${base64Image}`
            }
          }
        ]
      }
    ],
    thinking: { type: 'disabled' }
  });

  return response.choices[0]?.message?.content;
}

Advanced Use Cases

Conversational Vision Chat

import ZAI from 'z-ai-web-dev-sdk';

class VisionChatSession {
  constructor() {
    this.messages = [];
  }

  async initialize() {
    this.zai = await ZAI.create();
  }

  async addImage(imageUrl, initialQuestion) {
    this.messages.push({
      role: 'user',
      content: [
        {
          type: 'text',
          text: initialQuestion
        },
        {
          type: 'image_url',
          image_url: { url: imageUrl }
        }
      ]
    });

    return this.getResponse();
  }

  async followUp(question) {
    this.messages.push({
      role: 'user',
      content: [
        {
          type: 'text',
          text: question
        }
      ]
    });

    return this.getResponse();
  }

  async getResponse() {
    const response = await this.zai.chat.completions.createVision({
      messages: this.messages,
      thinking: { type: 'disabled' }
    });

    const assistantMessage = response.choices[0]?.message?.content;
    
    this.messages.push({
      role: 'assistant',
      content: assistantMessage
    });

    return assistantMessage;
  }
}

// Usage
const session = new VisionChatSession();
await session.initialize();

const initial = await session.addImage(
  'https://example.com/chart.jpg',
  'What does this chart show?'
);
console.log('Initial analysis:', initial);

const followup = await session.followUp('What are the key trends?');
console.log('Follow-up:', followup);

Image Classification and Tagging

import ZAI from 'z-ai-web-dev-sdk';

async function classifyImage(imageUrl) {
  const zai = await ZAI.create();

  const prompt = `Analyze this image and provide:
1. Main subject/category
2. Key objects detected
3. Scene description
4. Suggested tags (comma-separated)

Format your response as JSON.`;

  const response = await zai.chat.completions.createVision({
    messages: [
      {
        role: 'user',
        content: [
          {
            type: 'text',
            text: prompt
          },
          {
            type: 'image_url',
            image_url: { url: imageUrl }
          }
        ]
      }
    ],
    thinking: { type: 'disabled' }
  });

  const content = response.choices[0]?.message?.content;
  
  try {
    return JSON.parse(content);
  } catch (e) {
    return { rawResponse: content };
  }
}

OCR and Text Extraction

import ZAI from 'z-ai-web-dev-sdk';

async function extractText(imageUrl) {
  const zai = await ZAI.create();

  const response = await zai.chat.completions.createVision({
    messages: [
      {
        role: 'user',
        content: [
          {
            type: 'text',
            text: 'Extract all text from this image. Preserve the layout and formatting as much as possible.'
          },
          {
            type: 'image_url',
            image_url: { url: imageUrl }
          }
        ]
      }
    ],
    thinking: { type: 'disabled' }
  });

  return response.choices[0]?.message?.content;
}

Best Practices

1. Image Quality and Size

  • Use high-quality images for better analysis results
  • Optimize image size to balance quality and processing speed
  • Supported formats: JPEG, PNG, WebP

2. Prompt Engineering

  • Be specific about what information you need from the image
  • Structure complex requests with numbered lists or bullet points
  • Provide context about the image type (photo, diagram, chart, etc.)

3. Error Handling

async function safeVisionChat(imageUrl, question) {
  try {
    const zai = await ZAI.create();
    
    const response = await zai.chat.completions.createVision({
      messages: [
        {
          role: 'user',
          content: [
            { type: 'text', text: question },
            { type: 'image_url', image_url: { url: imageUrl } }
          ]
        }
      ],
      thinking: { type: 'disabled' }
    });

    return {
      success: true,
      content: response.choices[0]?.message?.content
    };
  } catch (error) {
    console.error('Vision chat error:', error);
    return {
      success: false,
      error: error.message
    };
  }
}

4. Performance Optimization

  • Cache SDK instance creation when processing multiple images
  • Use appropriate image formats (JPEG for photos, PNG for diagrams)
  • Consider image preprocessing for large batches

5. Security Considerations

  • Validate image URLs before processing
  • Sanitize user-provided image data
  • Implement rate limiting for public-facing APIs
  • Never expose SDK credentials in client-side code

Common Use Cases

  1. Product Analysis: Analyze product images for e-commerce applications
  2. Document Understanding: Extract information from receipts, invoices, forms
  3. Medical Imaging: Assist in preliminary analysis (with appropriate disclaimers)
  4. Quality Control: Detect defects or anomalies in manufacturing
  5. Content Moderation: Analyze images for policy compliance
  6. Accessibility: Generate alt text for images automatically
  7. Visual Search: Understand and categorize images for search functionality

Integration Examples

Express.js API Endpoint

import express from 'express';
import ZAI from 'z-ai-web-dev-sdk';

const app = express();
app.use(express.json());

let zaiInstance;

// Initialize SDK once
async function initZAI() {
  zaiInstance = await ZAI.create();
}

app.post('/api/analyze-image', async (req, res) => {
  try {
    const { imageUrl, question } = req.body;

    if (!imageUrl || !question) {
      return res.status(400).json({ 
        error: 'imageUrl and question are required' 
      });
    }

    const response = await zaiInstance.chat.completions.createVision({
      messages: [
        {
          role: 'user',
          content: [
            { type: 'text', text: question },
            { type: 'image_url', image_url: { url: imageUrl } }
          ]
        }
      ],
      thinking: { type: 'disabled' }
    });

    res.json({
      success: true,
      analysis: response.choices[0]?.message?.content
    });
  } catch (error) {
    res.status(500).json({
      success: false,
      error: error.message
    });
  }
});

initZAI().then(() => {
  app.listen(3000, () => {
    console.log('Vision chat API running on port 3000');
  });
});

Troubleshooting

Issue: "SDK must be used in backend"

  • Solution: Ensure z-ai-web-dev-sdk is only imported and used in server-side code

Issue: Image not loading or being analyzed

  • Solution: Verify the image URL is accessible and returns a valid image format

Issue: Poor analysis quality

  • Solution: Provide more specific prompts and ensure image quality is sufficient

Issue: Slow response times

  • Solution: Optimize image size and consider caching frequently analyzed images

Remember

  • Always use z-ai-web-dev-sdk in backend code only
  • The SDK is already installed - import as shown in examples
  • Structure prompts clearly for best results
  • Handle errors gracefully in production applications
  • Consider user privacy when processing images

FAQ

What does the vlm skill do?

Implement vision-based AI chat capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to analyze images, describe visual content, or create applications that combine image understanding with conversational AI. Supports image URLs and base64 encoded images for multimodal interactions.

How do I install vlm?

Run npx skills add https://github.com/answerzhao/agent-skills --skill vlm in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/vlm/ (personal) or .claude/skills/vlm/ (project).

Which AI agents can use vlm?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is vlm safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains vlm?

It is published by answerzhao in answerzhao/agent-skills. It has 343 recorded installs.

More from answerzhao/agent-skills

View all →

web-reader

answerzhao/agent-skills

Implement web page content extraction capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to scrape web pages, extract article content, retrieve page metadata, or build applications that process web content. Supports automatic content extraction with title, HTML, and publication time retrieval.

Frontend & Design1.7K installs

asr

answerzhao/agent-skills

Open the skill to read its SKILL.md instructions.

AI & Agents205 installs

tts

answerzhao/agent-skills

Implement text-to-speech (TTS) capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to convert text into natural-sounding speech, create audio content, build voice-enabled applications, or generate spoken audio files. Supports multiple voices, adjustable speed, and various audio formats.

Media & Creative112 installs

web-search

answerzhao/agent-skills

Implement web search capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to search the web, retrieve current information, find relevant content, or build applications with real-time web search functionality. Returns structured search results with URLs, snippets, and metadata.

Productivity & Automation100 installs

frontend-design

answerzhao/agent-skills

Open the skill to read its SKILL.md instructions.

Frontend & Design46 installs

image-generation

answerzhao/agent-skills

Implement AI image generation capabilities using the z-ai-web-dev-sdk. Use this skill when the user needs to create images from text descriptions, generate visual content, create artwork, design assets, or build applications with AI-powered image creation. Supports multiple image sizes and returns base64 encoded images. Also includes CLI tool for quick image generation.

Media & Creative45 installs

Popular media & creative skills

View category →

hyperframes

heygen-com/hyperframes

Read first to make, edit, animate or render any video with HyperFrames (HTML rendered to video), including launch films, explainers, music videos, charts, maps, vertical clips, title cards, captions and overlays. Holds the render contract, the work loop and what each kind of video needs.

Media & Creative1.2M installs

media-use

heygen-com/hyperframes

Get or make any media for a HyperFrames video - music, sound effects, images, icons, logos, voiceover, avatar and generated video, transcripts, cutouts, colour grades and looks - with HeyGen or free local tools, and mix placed audio. Also for footage that looks dark, flat or dull, or should be stylized or hidden.

Media & Creative1.1M installs

hyperframes-registry

heygen-com/hyperframes

About 450 ready-made HyperFrames blocks and components installed as editable source - transitions, overlays, titles, kinetic type, captions, charts, map overlays, 3D, camera moves. Use it before hand-building any named visual (CRT scanlines, glitch, film grain, a chart); also for hyperframes add and catalog.

Media & Creative1M installs

hyperframes-creative

heygen-com/hyperframes

Non-animation creative direction for HyperFrames videos. Use for design spec (frame.md / design.md) handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns, and brand / style decisions. For atomic motion patterns and scene blueprints, use hyperframes-animation.

Media & Creative916.5K installs

ai-video-generation

101-skills/superpowers

Generate AI videos with Google Veo, Seedance 2.0, HappyHorse, Wan, Grok and 40+ models via inference.sh CLI. Models: Veo 3.1, Veo 3, Seedance 2.0, HappyHorse 1.0, Wan 2.5, Grok Imagine Video, OmniHuman, Fabric, HunyuanVideo. Capabilities: text-to-video, image-to-video, reference-to-video, video editing, lipsync, avatar animation, video upscaling, foley sound. Use for: social media videos, marketing content, explainer videos, product demos, AI avatars. Triggers: video generation, ai video, text to video, image to video, veo, animate image, video from image, ai animation, video generator, generate video, t2v, i2v, ai video maker, create video with ai, runway alternative, pika alternative, sora alternative, kling alternative, seedance, happyhorse

Media & Creative881.5K installs

ai-image-generation

101-skills/superpowers

Generate AI images with GPT-Image-2, FLUX, Gemini, Grok, Seedream, Reve and 50+ models via inference.sh CLI. Models: GPT-Image-2, FLUX Dev LoRA, FLUX.2 Klein LoRA, Gemini 3 Pro Image, Grok Imagine, Seedream 4.5, Reve, ImagineArt. Capabilities: text-to-image, image-to-image, inpainting, LoRA, image editing, upscaling, text rendering. Use for: AI art, product mockups, concept art, social media graphics, marketing visuals, illustrations. Triggers: flux, image generation, ai image, text to image, stable diffusion, generate image, ai art, midjourney alternative, dall-e alternative, text2img, t2i, image generator, ai picture, create image with ai, generative ai, ai illustration, grok image, gemini image, gpt image, openai image, chatgpt image

Media & Creative880.3K installs