diff --git a/packages/ai/README.md b/packages/ai/README.md index 5e5526b6..042c1bb5 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -16,6 +16,10 @@ Unified LLM API with automatic model discovery, provider configuration, token an - [Validating Tool Arguments](#validating-tool-arguments) - [Complete Event Reference](#complete-event-reference) - [Image Input](#image-input) +- [Image Generation](#image-generation) + - [Basic Image Generation](#basic-image-generation) + - [Streaming Image Events](#streaming-image-events) + - [Notes and Limitations](#notes-and-limitations) - [Thinking/Reasoning](#thinkingreasoning) - [Unified Interface](#unified-interface-streamsimplecompletesimple) - [Provider-Specific Options](#provider-specific-options-streamcomplete) @@ -419,6 +423,117 @@ for (const block of response.content) { } ``` +## Image Generation + +Image generation uses a separate API surface from text/chat generation. Use `getImageModel()` / `getImageModels()` / `getImageProviders()` to discover image-generation models, `images()` to stream events, and `generateImages()` to get the final result. + +Do not use `stream()` or `complete()` for image generation. Request options, error handling, and event/result flow follow the same overall pattern as the streaming APIs, but use image-specific types and events. + +### Basic Image Generation + +```typescript +import { getImageModel, generateImages } from '@mariozechner/pi-ai'; + +const model = getImageModel('openrouter', 'google/gemini-2.5-flash-image'); + +const result = await generateImages(model, { + input: [{ type: 'text', text: 'Generate a red circle on a plain white background.' }] +}, { + apiKey: process.env.OPENROUTER_API_KEY +}); + +for (const block of result.output) { + if (block.type === 'text') { + console.log(block.text); + } else if (block.type === 'image') { + console.log(block.mimeType); + console.log(block.data.substring(0, 32)); + } +} +``` + +Some models also support image input: + +```typescript +import { readFileSync } from 'fs'; + +const imageBuffer = readFileSync('input.png'); +const result = await generateImages(model, { + input: [ + { type: 'text', text: 'Create a variation of this image with a blue background.' }, + { type: 'image', data: imageBuffer.toString('base64'), mimeType: 'image/png' } + ] +}, { + apiKey: process.env.OPENROUTER_API_KEY +}); +``` + +Check capabilities on the model metadata: + +```typescript +console.log(model.input); // ['text', 'image'] +console.log(model.output); // ['image'] or ['image', 'text'] +``` + +### Streaming Image Events + +`images()` returns an `AssistantImagesEventStream` with image-specific events: + +```typescript +import { getImageModel, images } from '@mariozechner/pi-ai'; + +const model = getImageModel('openrouter', 'google/gemini-2.5-flash-image'); +const s = images(model, { + input: [{ type: 'text', text: 'Generate a simple green triangle.' }] +}, { + apiKey: process.env.OPENROUTER_API_KEY +}); + +for await (const event of s) { + switch (event.type) { + case 'start': + console.log(`Starting image generation with ${event.partial.model}`); + break; + case 'image_start': + console.log(`Image block started at index ${event.contentIndex}`); + break; + case 'image_end': + console.log(`Image block complete: ${event.image.mimeType}`); + break; + case 'done': + console.log(`Finished: ${event.reason}`); + break; + case 'error': + console.error(`Error: ${event.error.errorMessage}`); + break; + } +} + +const finalResult = await s.result(); +``` + +| Event Type | Description | Key Properties | +|------------|-------------|----------------| +| `start` | Stream begins | `partial`: Initial `AssistantImages` structure | +| `image_start` | Image block starts | `contentIndex`: Position in output array | +| `image_end` | Image block complete | `image`: Final `ImageContent`, `contentIndex`: Position | +| `done` | Stream complete | `reason`: Stop reason (`"stop"`), `images`: Final `AssistantImages` | +| `error` | Error occurred | `reason`: Error type (`"error"` or `"aborted"`), `error`: Final `AssistantImages` | + +Text output, if the model returns any, is available on the final `AssistantImages.output` array. + +### Notes and Limitations + +- Use `getImageModel(...)`, not `getModel(...)`. +- Use `images()` / `generateImages()`, not `stream()` / `complete()`. +- Image-generation models do not participate in tool calling. +- Outputs are returned as base64-encoded `ImageContent` blocks in `AssistantImages.output`. +- Some models return only images, others return images plus text. Check `model.output`. +- Some models accept image input, others are text-to-image only. Check `model.input`. +- Like the streaming APIs, image generation supports options such as `apiKey`, `signal`, `headers`, `onPayload`, and `onResponse`, and results may include `stopReason`, `responseId`, and `usage`. +- If you want a model to analyze images in a conversation or call tools, use the regular `stream()` / `complete()` APIs with a model that supports image input. +- At the moment, image generation is available through only one provider, OpenRouter. + ## Thinking/Reasoning Many models support thinking/reasoning capabilities where they can show their internal thought process. You can check if a model supports reasoning via the `reasoning` property. If you pass reasoning options to a non-reasoning model, they are silently ignored.