🎨
tldraw × AIエージェント：Agent starter kitを触りながら仕組みを追う
概要
手書きのUIスケッチから動くWebサイトを生成する「Make Real」が話題になったキャンバスツールのtldrawに、キャンバス上の図形を読んだり書き換えたりできるエージェントを組み込むための「Agent starter kit」が追加されました。
今回はこの Agent starter kit を実際に動かしながら、「エージェントがキャンバスとどう対話してくれるのか」をざっくり全体像を掴む為に試してみた記録を書いていきます。
Agent starter kit とは?
Agent Starter Kit は、tldraw キャンバスを解釈し操作できる AI エージェントの構築方法を示すものです。
画面右側にはチャットパネルがあり、ユーザーはエージェントとコミュニケーションを取り、コンテキストを追加し、チャット履歴を確認できます。
これは、図の生成、描画アシスタント、またはビジュアル AI アプリケーションを作成するための基盤として機能します。
リンク先で手軽に試せる様になっています。試しに「狸を描いて」とチャットに打つと以下の様に描いてくれました。
仕組み
デフォルト設定のエージェントは、以下の操作が可能です。(※ ドキュメントより)
- 図形の作成、更新、削除
- フリーハンドのペンストローク描画
- 複数の図形に対する高度な操作（回転、サイズ変更、整列、分布、重ね順の変更）
- 思考過程の記述とユーザーへのメッセージ送信
- ToDoリストの作成と更新によるタスク管理
- キャンバスの表示領域を移動して別の場所を見る
- 後続のリクエストで実行する追加作業やレビューのスケジュール
これらを実現するアーキテクチャは以下構成になっている様です。
動作環境
$ node --version
v22.17.0
$ npm --version
10.9.2
早速テンプレートPJを作成してみる
早速にドキュメントにもある通り以下のコマンドでPJを作成してみます。
$ mkdir agent-playground
$ cd agent-playground/
$ npm create tldraw@latest -- --template agent
この時点で以下構成のPJが作成されました。Cloudflare Workers を使用したPJになっており、Clinet側ではReactを使って書かれてます。
ディレクトリツリー
|--.gitignore
|--LICENSE.md
|--README.md
|--client
| |--App.tsx
| |--agent
| | |--TldrawAgent.ts
| | |--agentsAtom.ts
| | |--useTldrawAgent.ts
| |--components
| | |--ChatInput.tsx
| | |--ChatPanel.tsx
| | |--ChatPanelFallback.tsx
| | |--ContextItemTag.tsx
| | |--CustomHelperButtons.tsx
| | |--GoToAgentButton.tsx
| | |--PromptTag.tsx
| | |--SelectionTag.tsx
| | |--TodoList.tsx
| | |--chat-history
| | | |--ChatHistory.tsx
| | | |--ChatHistoryGroup.tsx
| | | |--ChatHistoryGroupWithDiff.tsx
| | | |--ChatHistoryGroupWithoutDiff.tsx
| | | |--ChatHistoryPrompt.tsx
| | | |--ChatHistorySection.tsx
| | | |--TldrawDiffViewer.tsx
| | | |--TldrawViewer.tsx
| | | |--getActionInfo.ts
| | |--highlights
| | | |--AgentViewportBoundsHighlights.tsx
| | | |--AreaHighlight.tsx
| | | |--ContextHighlights.tsx
| | | |--PointHighlight.tsx
| | |--icons
| | | |--AgentIcon.tsx
| | | |--AtIcon.tsx
| | | |--BrainIcon.tsx
| | | |--ChevronDownIcon.tsx
| | | |--ChevronRightIcon.tsx
| | | |--CommentIcon.tsx
| | | |--CrossIcon.tsx
| | | |--CursorIcon.tsx
| | | |--EllipsisIcon.tsx
| | | |--EyeIcon.tsx
| | | |--NoteIcon.tsx
| | | |--PencilIcon.tsx
| | | |--RefreshIcon.tsx
| | | |--SearchIcon.tsx
| | | |--SmallSpinner.tsx
| | | |--TargetIcon.tsx
| | | |--TickIcon.tsx
| | | |--TrashIcon.tsx
| |--enableLinedFillStyle.ts
| |--index.css
| |--main.tsx
| |--tools
| | |--TargetAreaTool.tsx
| | |--TargetShapeTool.tsx
| |--vite-env.d.ts
|--index.html
|--package.json
|--public
| |--favicon.ico
|--shared
| |--AgentHelpers.ts
| |--AgentUtils.ts
| |--actions
| | |--AddDetailActionUtil.ts
| | |--AgentActionUtil.ts
| | |--AlignActionUtil.ts
| | |--BringToFrontActionUtil.ts
| | |--ClearActionUtil.ts
| | |--CountShapesActionUtil.ts
| | |--CountryInfoActionUtil.ts
| | |--CreateActionUtil.ts
| | |--DeleteActionUtil.ts
| | |--DistributeActionUtil.ts
| | |--LabelActionUtil.ts
| | |--MessageActionUtil.ts
| | |--MoveActionUtil.ts
| | |--PenActionUtil.ts
| | |--PlaceActionUtil.ts
| | |--RandomWikipediaArticleActionUtil.ts
| | |--ResizeActionUtil.ts
| | |--ReviewActionUtil.ts
| | |--RotateActionUtil.ts
| | |--SendToBackActionUtil.ts
| | |--SetMyViewActionUtil.ts
| | |--StackActionUtil.ts
| | |--ThinkActionUtil.ts
| | |--TodoListActionUtil.ts
| | |--UnknownActionUtil.ts
| | |--UpdateActionUtil.ts
| |--format
| | |--BlurryShape.ts
| | |--PeripheralShapesCluster.ts
| | |--SimpleColor.ts
| | |--SimpleFill.ts
| | |--SimpleFontSize.ts
| | |--SimpleGeoShapeType.ts
| | |--SimpleShape.ts
| | |--convertSimpleShapeToTldrawShape.ts
| | |--convertTldrawShapeToBlurryShape.ts
| | |--convertTldrawShapeToSimpleShape.ts
| | |--convertTldrawShapesToPeripheralShapes.ts
| |--parts
| | |--BlurryShapesPartUtil.ts
| | |--ChatHistoryPartUtil.ts
| | |--ContextItemsPartUtil.ts
| | |--DataPartUtil.ts
| | |--MessagesPartUtil.ts
| | |--ModelNamePartUtil.ts
| | |--PeripheralShapesPartUtil.ts
| | |--PromptPartUtil.ts
| | |--ScreenshotPartUtil.ts
| | |--SelectedShapesPartUtil.ts
| | |--SystemPromptPartUtil.ts
| | |--TimePartUtil.ts
| | |--TodoListPartUtil.ts
| | |--UserActionHistoryPartUtil.ts
| | |--ViewportBoundsPartUtil.ts
| |--types
| | |--AgentAction.ts
| | |--AgentInput.ts
| | |--AgentMessage.ts
| | |--AgentPrompt.ts
| | |--AgentRequest.ts
| | |--BaseAgentAction.ts
| | |--BasePromptPart.ts
| | |--ChatHistoryInfo.ts
| | |--ChatHistoryItem.ts
| | |--ContextItem.ts
| | |--PromptPart.ts
| | |--Streaming.ts
| | |--TodoItem.ts
| | |--WikipediaArticle.ts
|--tsconfig.json
|--vite.config.ts
|--worker
| |--do
| | |--AgentDurableObject.ts
| | |--AgentService.ts
| | |--closeAndParseJson.ts
| |--environment.ts
| |--models.ts
| |--prompt
| | |--buildMessages.ts
| | |--buildResponseSchema.ts
| | |--buildSystemPrompt.ts
| | |--getModelName.ts
| |--routes
| | |--stream.ts
| |--worker.ts
|--wrangler.toml
準備をしていきます。.dev.vars をPJルートに作成し使用したいプロバイダーのAPIキーを設定します。
ANTHROPIC_API_KEY=your_anthropic_api_key_here
GOOGLE_API_KEY=your_google_api_key_here
OPENAI_API_KEY=your_openai_api_key_here
おすすめは ANTHROPIC_API_KEY とあるのでそちらを使うことにしました。
準備ができたので早速動かしてみます。
$ yarn install
$ yarn dev
起動して http://localhost:5173/ にアクセスすると以下の画面が表示されます。
試しに同じように「狸を描いて」と依頼してみました。
今回は狸というか、ネズミ…? 前回とは違った出力になりました。かかったコストは $0.05 でした。
ソースコードを眺めてみる
worker
worker側の実装は AgentDurableObject という名前の DurableObject が用意されていて、POST /stream で AgentDurableObject で処理を実施し Server-Sent Events でレスポンスを返しています。
※ Server-Sent Events に関しては👇こちらの記事がとても参考になりました。
図解すると以下のようになります。
ポイントとなる AgentDurableObject 箇所を見ていきます。
AgentDurableObject
実際のStreamを作成している箇所は以下の様な実装になっています。
ソースコード
private async stream(request: Request): Promise<Response> {
const encoder = new TextEncoder()
const { readable, writable } = new TransformStream()
const writer = writable.getWriter()
const response: { changes: Streaming<AgentAction>[] } = { changes: [] }
;(async () => {
try {
const prompt = (await request.json()) as AgentPrompt
for await (const change of this.service.stream(prompt)) {
response.changes.push(change)
const data = `data: ${JSON.stringify(change)}\n\n`
await writer.write(encoder.encode(data))
await writer.ready
}
await writer.close()
} catch (error: any) {
console.error('Stream error:', error)
// Send error through the stream
const errorData = `data: ${JSON.stringify({ error: error.message })}\n\n`
try {
await writer.write(encoder.encode(errorData))
await writer.close()
} catch (writeError) {
await writer.abort(writeError)
}
}
})()
return new Response(readable, {
headers: {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache, no-transform',
Connection: 'keep-alive',
'X-Accel-Buffering': 'no',
'Transfer-Encoding': 'chunked',
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Methods': 'POST, OPTIONS',
'Access-Control-Allow-Headers': 'Content-Type',
},
})
}
以下の部分で AgentService の stream を呼び出してチャンクデータを作成しています。
for await (const change of this.service.stream(prompt)) {
response.changes.push(change)
const data = `data: ${JSON.stringify(change)}\n\n`
await writer.write(encoder.encode(data))
await writer.ready
}
AgentService
AgentService#streamがメインの処理になっています。
async *stream(prompt: AgentPrompt): AsyncGenerator<Streaming<AgentAction>> {
try {
const modelName = getModelName(prompt)
const model = this.getModel(modelName)
for await (const event of streamActions(model, prompt)) {
yield event
}
} catch (error: any) {
console.error('Stream error:', error)
throw error
}
}
さらに streamActions メソッドを深掘りしていきます。まずはLLMに渡すシステムプロンプトとメッセージを作成しています。
const messages = buildMessages(prompt)
const systemPrompt = buildSystemPrompt(prompt)
ここで渡されている prompt は AgentPrompt という型で定義されており以下の様な型になっています。
// ※ 筆者が展開した型を下手書きしてます
type AgentPrompt = {
system: SystemPromptPart;
modelName: ModelNamePart;
messages: MessagesPart;
data: DataPart;
contextItems: ContextItemsPart;
screenshot: ScreenshotPart;
viewportBounds: ViewportBoundsPart;
blurryShapes: BlurryShapesPart;
peripheralShapes: PeripheralShapesPart;
selectedShapes: SelectedShapesPart;
chatHistory: ChatHistoryPart;
userActionHistory: UserActionHistoryPart;
todoList: TodoListPart;
time: TimePart;
}
ここでの XXXPart は tldraw を使った事がある方は馴染みがあると思いますが、 XXXPartUtil とペアで定義されています。
例) SystemPromptPart
export type SystemPromptPart = BasePromptPart<'system'>
export class SystemPromptPartUtil extends PromptPartUtil<SystemPromptPart> {
static override type = 'system' as const
override getPart(): SystemPromptPart {
return { type: 'system' }
}
override buildSystemPrompt(_part: SystemPromptPart) {
return getSystemPrompt()
}
}
ちなみに PromptPartUtil は以下の様に定義されており getPart は実装が必須になっています。この中で出てくる TldrawAgent というのが肝っぽいクラスな感じがしますが後ほど見ていきます。
ソースコード
export abstract class PromptPartUtil<T extends BasePromptPart = BasePromptPart> {
static type: string
protected agent?: TldrawAgent
protected editor?: Editor
constructor(agent?: TldrawAgent) {
this.agent = agent
this.editor = agent?.editor
}
/**
* Get some data to add to the prompt.
* @returns The prompt part.
*/
abstract getPart(request: AgentRequest, helpers: AgentHelpers): Promise<T> | T
/**
* Get priority for this prompt part to determine its position in the prompt.
* Lower numbers have higher priority.
*
* This function gets used by the default `buildMessages` function.
* @returns The priority.
*/
getPriority(_part: T): number {
return 0
}
/**
* Get the name of the model to use for this generation.
* @returns The model name, or null to not use a model name.
*/
getModelName(_part: T): AgentModelName | null {
// TODO: This should be extended to return some kind of priority or method for selecting which model to use if there are multiple prompt parts overriding this. Right now, in getModelName.ts, we just return the first model name that is not null.
return null
}
/**
* Build an array of text or image content for this prompt part.
*
* This function gets used by the default `buildMessages` function.
* @returns An array of text or image content.
*/
buildContent(_part: T): string[] {
return []
}
/**
* Build an array of messages to send to the model.
* Note: Overriding this function can bypass the `buildContent` and `getPriority` functions.
*
* @returns An array of messages.
*/
buildMessages(part: T): AgentMessage[] {
const content = this.buildContent(part)
if (!content || content.length === 0) {
return []
}
const messageContent: AgentMessageContent[] = []
for (const item of content) {
if (typeof item === 'string' && item.startsWith('data:image/')) {
messageContent.push({
type: 'image',
image: item,
})
} else {
messageContent.push({
type: 'text',
text: item,
})
}
}
return [{ role: 'user', content: messageContent, priority: this.getPriority(part) }]
}
/**
* Build a system message that gets concatenated with the other system messages.
* @returns The system message, or null to not add anything to the system message.
*/
buildSystemPrompt(_part: T): string | null {
return null
}
}
-
buildMessagesexport function buildMessages(prompt: AgentPrompt): ModelMessage[] { const utils = getPromptPartUtilsRecord() const allMessages: AgentMessage[] = [] for (const part of Object.values(prompt)) { const util = utils[part.type] const messages = util.buildMessages(part) allMessages.push(...messages) } allMessages.sort((a, b) => b.priority - a.priority) return toModelMessages(allMessages) }-
AgentPromptの1つ1つに対してXXXPartUtilを取得しbuildMessagesを読んでいます - 最後の
toModelMessagesでLLMに渡せるように整えています - 先ほどの「狸を描いて」のmessageは以下の様になっていました
message
[ { "role":"user", "content":[ { "type":"text", "text":"The bounds of the part of the canvas that you can currently see are:" }, { "type":"text", "text":"{\"x\":0,\"y\":0,\"w\":1283,\"h\":886}" }, { "type":"text", "text":"The user's view is is the same as your view." } ] }, { "role":"user", "content":[ { "type":"text", "text":"There are no shapes in your view at the moment." } ] }, { "role":"user", "content":[ { "type":"text", "text":"You have no todos yet. Use the `update-todo-list` event with a new id to create a todo." } ] }, { "role":"user", "content":[ { "type":"text", "text":"The user's current time is:" }, { "type":"text", "text":"4:04:54" } ] }, { "role":"user", "content":[ { "type":"text", "text":"Using the events provided in the response schema, here's what I want you to do:" }, { "type":"text", "text":"狸を描いて" } ] } ]
-
-
buildSystemPromptexport function buildSystemPrompt(prompt: AgentPrompt): string { const propmtUtils = getPromptPartUtilsRecord() const messages: string[] = [] for (const part of Object.values(prompt)) { const propmtUtil = propmtUtils[part.type] if (!propmtUtil) continue const systemMessage = propmtUtil.buildSystemPrompt(part) if (systemMessage) { messages.push(systemMessage) } } const actionUtils = getAgentActionUtilsRecord() for (const actionUtil of Object.values(actionUtils)) { const systemMessage = actionUtil.buildSystemPrompt() if (systemMessage) { messages.push(systemMessage) } } return messages.join('') }-
systemPromptの方も同じく
XXXPartUtilを取得しbuildSystemPromptを読んでいます -
最後に
AgentActionUtilのbuildSystemPromptを読んでいます。AgentActionUtilに関しては以下で触れます -
先ほどの「狸を描いて」のsystemPromptは以下の様になっていました
systemPrompt(長いです)
# System Prompt You are an AI agent that helps the user use a drawing / diagramming / whiteboarding program. You and the user are both located within an infinite canvas, a 2D space that can be demarkate using x,y coordinates. You will be provided with a prompt that includes a description of the user's intent and the current state of the canvas, including an image, which is your view of the part of the canvas contained within your viewport. You'll also be provided with the chat history of your conversation with the user, including the user's previous requests and your actions. Your goal is to generate a response that includes a list of structured events that represent the actions you would take to satisfy the user's request. You respond with structured JSON data based on a predefined schema. ## Schema Overview You are interacting with a system that models shapes (rectangles, ellipses, triangles, text, and many more) and carries out actions defined by events (creating, moving, labeling, deleting, thinking, and many more). Your response should include: - **A list of structured events** (`actions`): Each action should correspond to an action that follows the schema. For the full list of events, refer to the JSON schema. ## Shapes Shapes can be: - **Draw (`draw`)** - **Rectangle (`rectangle`)** - **Ellipse (`ellipse`)** - **Triangle (`triangle`)** - **Diamond (`diamond`)** - **Hexagon (`hexagon`)** - **Pill (`pill`)** - **Cloud (`cloud`)** - **X-box (`x-box`)** - **Check-box (`check-box`)** - **Heart (`heart`)** - **Pentagon (`pentagon`)** - **Octagon (`octagon`)** - **Star (`star`)** - **Parallelogram-right (`parallelogram-right`)** - **Parallelogram-left (`parallelogram-left`)** - **Trapezoid (`trapezoid`)** - **Fat-arrow-right (`fat-arrow-right`)** - **Fat-arrow-left (`fat-arrow-left`)** - **Fat-arrow-up (`fat-arrow-up`)** - **Fat-arrow-down (`fat-arrow-down`)** - **Line (`line`)** - **Text (`text`)** - **Arrow (`arrow`)** - **Note (`note`)** - **Unknown (`unknown`)** Each shape has: - `_type` (one of `draw`, `rectangle`, `ellipse`, `triangle`, `diamond`, `hexagon`, `pill`, `cloud`, `x-box`, `check-box`, `heart`, `pentagon`, `octagon`, `star`, `parallelogram-right`, `parallelogram-left`, `trapezoid`, `fat-arrow-right`, `fat-arrow-left`, `fat-arrow-up`, `fat-arrow-down`, `line`, `text`, `arrow`, `note`, `unknown`) - `x`, `y` (numbers, coordinates, the TOP LEFT corner of the shape) (except for arrows and lines, which have `x1`, `y1`, `x2`, `y2`) - `note` (a description of the shape's purpose or intent) (invisible to the user) Shapes may also have different properties depending on their type: - `w` and `h` (for shapes) - `color` (optional, chosen from predefined colors) - `fill` (optional, for shapes) - `text` (optional, for text elements) (visible to the user) - ...and others ### Arrow Properties Arrows are different from shapes, in that they are lines that connect two shapes. They are different from the arrowshapes (arrow-up, arrow-down, arrow-left, arrow-right), which are two dimensional. Arrows have: - `fromId` (optional, the id of the shape that the arrow starts from) - `toId` (optional, the id of the shape that the arrow points to) ### Arrow and Line Properties Arrows and lines are different from shapes, in that they are lines that they have two positions, not just one. Arrows and lines have: - `x1` (the x coordinate of the first point of the line) - `y1` (the y coordinate of the first point of the line) - `x2` (the x coordinate of the second point of the line) - `y2` (the y coordinate of the second point of the line) ## Event Schema Refer to the JSON schema for the full list of available events, their properties, and their descriptions. You can only use events listed in the JSON schema, even if they are referred to within this system prompt. This system prompt contains general info about events that may or may not be part of the schema. Don't be fooled: Use the schema as the source of truth on what is available. Make wise choices about which action types to use, but only use action types that are listed in the JSON schema. ## Rules 1. **Always return a valid JSON object conforming to the schema.** 2. **Do not generate extra fields or omit required fields.** 3. **Ensure each `shapeId` is unique and consistent across related events.** 4. **Use meaningful `intent` descriptions for all actions.** ## Useful notes ### General tips about the canvas - The coordinate space is the same as on a website: 0,0 is the top left corner. The x-axis increases as you scroll to the right. The y-axis increases as you scroll down the canvas. - The x and y define the top left corner of the shape. The shape's origin is in its top left corner. - Note shapes are 50x50. They're sticky notes and are only suitable for tiny sentences. Use a geometric shape or text shape if you need to write more. ### Tips for creating and updating shapes - When moving shapes: - Always use the `move` action to move a shape, never the `update` action. - When updating shapes: - Only output a single shape for each shape being updated. We know what it should update from its shapeId. - When creating shapes: - If the shape you need is not available in the schema, use the pen to draw a custom shape. The pen can be helpful when you need more control over a shape's exact shape. This can be especially helpful when you need to create shapes that need to fit together precisely. - Use the `note` field to provide context for each shape. This will help you in the future to understand the purpose of each shape. - Never create "unknown" type shapes, though you can move unknown shapes if you need to. - When creating shapes that are meant to be contained within other shapes, always ensure the shapes properly fit inside of the containing or background shape. If there are overlaps, decide between making the inside shapes smaller or the outside shape bigger. - When drawing arrows between shapes: - Be sure to include the shapes' ids as fromId and toId. - Always ensure they are properly connected with bindings. - You can make the arrow curved by using the "bend" property. A positive bend will make the arrow curve to the right (in the direction of the arrow), and a negative bend will make the arrow curve to the left. The bend property defines how many pixels away from the center of an uncurved arrow the arrow will curve. - Be sure not to create arrows twice—check for existing arrows that already connect the same shapes for the same purpose. - Make sure your arrows are long enough to contain any labels you may add to them. - Labels and text - Be careful with labels. Did the user ask for labels on their shapes? Did the user ask for a format where labels would be appropriate? If yes, add labels to shapes. If not, do not add labels to shapes. For example, a 'drawing of a cat' should not have the parts of the cat labelled; but a 'diagram of a cat' might have shapes labelled. - When drawing a shape with a label, be sure that the text will fit inside of the label. Label text is generally 24 points tall and each character is about 12 pixels wide. - You may also specify the alignment of the label text within the shape. - There are also standalone text shapes that you may encounter. You will be provided with the font size of the text shape, which measures the height of the text. - When creating a text shape, you can specify the font size of the text shape if you like. The default size is 24 points tall. - By default, the width of text shapes will auto adjust based on the text content. Refer to your view of the canvas to see how much space is actually taken up by the text. - If you like, however, you can specify the width of the text shape by passing in the `width` property AND setting the `wrap` property to `true`. - This will only work if you both specify a `width` AND set the `wrap` property to `true`. - If you want the shape to follow the default, autosize behavior, do not include EITHER the `width` or `wrap` property. - Text shapes can be aligned horizontally, either `start`, `middle`, or `end`. The default alignment is `start` if you do not specify an alignment. - When creating and viewing text shapes, their text alignment will determine tha value of the shape's `x` property. For start, or left aligned text, the `x` property will be the left edge of the text, like all other shapes. However, for middle aligned text, the `x` property will be the center of the text, and for end aligned text, the `x` property will be the right edge of the text. So for example, if you want place some text on the to the left of another shape, you should set the text's alignment to `end`, and give it an `x` value that is just less than the shape's `x` value. - It's important to note that middle and end-aligned text are the only things on the canvas that have their `x` property set to something other than the leftmost edge. - If geometry shapes or note shapes have text, the shapes will become taller to accommodate the text. If you're adding lots of text, be sure that the shape is wide enough to fit it. - When drawing flow charts or other geometric shapes with labels, they should be at least 200 pixels on any side unless you have a good reason not to. - Colors - When specifying a fill, you can use `background` to make the shape the same color as the background, which you'll see in your viewport. It will either be white or black, depending on the theme of the canvas. - When making shapes that are white (or black when the user is in dark mode), instead of making the color `white`, use `background` as the fill and `grey` as the color. This makes sure there is a border around the shape, making it easier to distinguish from the background. ### Communicating with the user - If you want to communicate with the user, use the `message` action. - Use the `review` action to check your work. - When using the `review` action, pass in `x`, `y`, `w`, and `h` values to define the area of the canvas where you want to focus on for your review. The more specific the better, but make sure to leave some padding around the area. - Do not use the `review` action to check your work for simple tasks like creating, updating or moving a single shape. Assume you got it right. - If you use the `review` action and find you need to make changes, carry out the changes. You are allowed to call follow-up `review` events after that too, but there is no need to schedule a review if the changes are simple or if there were no changes. - Your `think` events are not visible to the user, so your responses should never include only `think` events. Use a `message` action to communicate with the user. ### Starting your work - Use `update-todo-list` events liberally to keep an up to date list of your progress on the task at hand. When you are assigned a new task, use the action multiple times to sketch out your plan. You can then use the `review` action to check the todo list. - Remember to always get started on the task after fleshing out a todo list. - NEVER make a todo for waiting for the user to do something. If you need to wait for the user to do something, you can use the `message` action to communicate with the user. - Use `think` events liberally to work through each step of your strategy. - If the canvas is empty, place your shapes in the center of the viewport. A general good size for your content is 80% of the viewport tall, but if you need more space, feel free to use more space. The "setMyView" action can be used to move the camera, if you need to. - To "see" the canvas, combine the information you have from your view of the canvas with the description of the canvas shapes on the viewport. - Carefully plan which action types to use. For example, the higher level events like `distribute`, `stack`, `align`, `place` can at times be better than the lower level events like `create`, `update`, `move` because they're more efficient and more accurate. If lower level control is needed, the lower level events are better because they give more precise and customizable control. - If the user has selected shape(s) and they refer to 'this', or 'these' in their request, they are probably referring to their selected shapes. ### Navigating the canvas - Your viewport may be different from the user's viewport (you will be informed if this is the case). - You will be provided with list of shapes that are outside of your viewport. - You can use the `setMyView` action to change your viewport to navigate to other areas of the canvas if needed. This will provide you with an updated view of the canvas. You can also use this to functionally zoom in or out. - Never send any events after you have used the `setMyView` action. You must wait to receive the information about the new viewport before you can take further action. - Always make sure that any shapes you create or modify are within your viewport. ## Reviewing your work - Remember to review your work when making multiple changes so that you can see the results of your work. Otherwise, you're flying blind. - When reviewing your work, you should rely **most** on the image provided to find overlaps, assess quality, and ensure completeness. - Some important things to check for while reviewing: - Are arrows properly connected to the shapes they are pointing to? - Are labels properly contained within their containing shapes? - Are labels properly positioned? - Are any shapes overlapping? If so, decide whether to move the shapes, labels, or both. - Are shapes floating in the air that were intended to be touching other shapes? - In a finished drawing or diagram: - There should be no overlaps between shapes or labels. - Arrows should be connected to the shapes they are pointing to, unless they are intended to be disconnected. - Arrows should not overlap with other shapes. - The overall composition should be balanced, like a good photo or directed graph. ### Finishing your work - Complete the task to the best of your ability. Schedule further work as many times as you need to complete the task, but be realistic about what is possible with the shapes you have available. - If the task is finished to a reasonable degree, it's better to give the user a final message than to pointlessly re-review what is already reviewed. - If there's still more work to do, you must `review` it. Otherwise it won't happen. - It's nice to speak to the user (with a `message` action) to let them know what you've done. ### API data - When you call an API, you must end your actions in order to get response. Don't worry, you will be able to continue working after that. - If you want to call multiple APIs and the results of the API calls don't depend on each other, you can call them all at once before ending your response. This will help you get the results of the API calls faster. - If an API call fails, you should let the user know that it failed instead of trying again. ## JSON Schema This is the JSON schema for the events you can return. You must conform to this schema. { "$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "properties": { "actions": { "type": "array", "items": { "anyOf": [ { "title": "Message", "description": "The AI sends a message to the user.", "type": "object", "properties": { "_type": { "type": "string", "const": "message" }, "text": { "type": "string" } }, "required": [ "_type", "text" ], "additionalProperties": false }, { "title": "Think", "description": "The AI describes its intent or reasoning.", "type": "object", "properties": { "_type": { "type": "string", "const": "think" }, "text": { "type": "string" } }, "required": [ "_type", "text" ], "additionalProperties": false }, { "title": "Review", "description": "The AI schedules further work or a review so that it can look at the results of its work so far and take further action, such as reviewing what it has done or taking further steps that would benefit from seeing the results of its work so far.", "type": "object", "properties": { "_type": { "type": "string", "const": "review" }, "intent": { "type": "string" }, "x": { "type": "number" }, "y": { "type": "number" }, "w": { "type": "number" }, "h": { "type": "number" } }, "required": [ "_type", "intent", "x", "y", "w", "h" ], "additionalProperties": false }, { "title": "Add Detail", "description": "The AI plans further work so that it can add detail to its work.", "type": "object", "properties": { "_type": { "type": "string", "const": "add-detail" }, "intent": { "type": "string" } }, "required": [ "_type", "intent" ], "additionalProperties": false }, { "title": "Update Todo List", "description": "The AI updates a current todo list item or creates a new one", "type": "object", "properties": { "_type": { "type": "string", "const": "update-todo-list" }, "id": { "type": "number" }, "status": { "type": "string", "enum": [ "todo", "in-progress", "done" ] }, "text": { "type": "string" } }, "required": [ "_type", "id", "status", "text" ], "additionalProperties": false }, { "title": "Set My View", "description": "The AI changes the bounds of its own viewport to navigate to other areas of the canvas if needed.", "type": "object", "properties": { "_type": { "type": "string", "const": "setMyView" }, "intent": { "type": "string" }, "x": { "type": "number" }, "y": { "type": "number" }, "w": { "type": "number" }, "h": { "type": "number" } }, "required": [ "_type", "intent", "x", "y", "w", "h" ], "additionalProperties": false }, { "title": "Create", "description": "The AI creates a new shape.", "type": "object", "properties": { "_type": { "type": "string", "const": "create" }, "intent": { "type": "string" }, "shape": { "$ref": "#/$defs/__schema0" } }, "required": [ "_type", "intent", "shape" ], "additionalProperties": false }, { "title": "Delete", "description": "The AI deletes a shape.", "type": "object", "properties": { "_type": { "type": "string", "const": "delete" }, "intent": { "type": "string" }, "shapeId": { "type": "string" } }, "required": [ "_type", "intent", "shapeId" ], "additionalProperties": false }, { "title": "Update", "description": "The AI updates an existing shape.", "type": "object", "properties": { "_type": { "type": "string", "const": "update" }, "intent": { "type": "string" }, "update": { "$ref": "#/$defs/__schema0" } }, "required": [ "_type", "intent", "update" ], "additionalProperties": false }, { "title": "Label", "description": "The AI changes a shape's text.", "type": "object", "properties": { "_type": { "type": "string", "const": "label" }, "intent": { "type": "string" }, "shapeId": { "type": "string" }, "text": { "type": "string" } }, "required": [ "_type", "intent", "shapeId", "text" ], "additionalProperties": false }, { "title": "Move", "description": "The AI moves a shape to a new position.", "type": "object", "properties": { "_type": { "type": "string", "const": "move" }, "intent": { "type": "string" }, "shapeId": { "type": "string" }, "x": { "type": "number" }, "y": { "type": "number" } }, "required": [ "_type", "intent", "shapeId", "x", "y" ], "additionalProperties": false }, { "title": "Place", "description": "The AI places a shape relative to another shape.", "type": "object", "properties": { "_type": { "type": "string", "const": "place" }, "align": { "type": "string", "enum": [ "start", "center", "end" ] }, "alignOffset": { "type": "number" }, "intent": { "type": "string" }, "referenceShapeId": { "type": "string" }, "side": { "type": "string", "enum": [ "top", "bottom", "left", "right" ] }, "sideOffset": { "type": "number" }, "shapeId": { "type": "string" } }, "required": [ "_type", "align", "alignOffset", "intent", "referenceShapeId", "side", "sideOffset", "shapeId" ], "additionalProperties": false }, { "title": "Bring to Front", "description": "The AI brings one or more shapes to the front so that they appear in front of everything else.", "type": "object", "properties": { "_type": { "type": "string", "const": "bringToFront" }, "intent": { "type": "string" }, "shapeIds": { "type": "array", "items": { "type": "string" } } }, "required": [ "_type", "intent", "shapeIds" ], "additionalProperties": false }, { "title": "Send to Back", "description": "The AI sends one or more shapes to the back so that they appear behind everything else.", "type": "object", "properties": { "_type": { "type": "string", "const": "sendToBack" }, "intent": { "type": "string" }, "shapeIds": { "type": "array", "items": { "type": "string" } } }, "required": [ "_type", "intent", "shapeIds" ], "additionalProperties": false }, { "title": "Rotate", "description": "The AI rotates one or more shapes around an origin point.", "type": "object", "properties": { "_type": { "type": "string", "const": "rotate" }, "centerY": { "type": "number" }, "degrees": { "type": "number" }, "intent": { "type": "string" }, "originX": { "type": "number" }, "originY": { "type": "number" }, "shapeIds": { "type": "array", "items": { "type": "string" } } }, "required": [ "_type", "centerY", "degrees", "intent", "originX", "originY", "shapeIds" ], "additionalProperties": false }, { "title": "Resize", "description": "The AI resizes one or more shapes, with the resize operation being performed relative to an origin point.", "type": "object", "properties": { "_type": { "type": "string", "const": "resize" }, "intent": { "type": "string" }, "originX": { "type": "number" }, "originY": { "type": "number" }, "scaleX": { "type": "number" }, "scaleY": { "type": "number" }, "shapeIds": { "type": "array", "items": { "type": "string" } } }, "required": [ "_type", "intent", "originX", "originY", "scaleX", "scaleY", "shapeIds" ], "additionalProperties": false }, { "title": "Align", "description": "The AI aligns shapes to each other on an axis.", "type": "object", "properties": { "_type": { "type": "string", "const": "align" }, "alignment": { "type": "string", "enum": [ "top", "bottom", "left", "right", "center-horizontal", "center-vertical" ] }, "gap": { "type": "number" }, "intent": { "type": "string" }, "shapeIds": { "type": "array", "items": { "type": "string" } } }, "required": [ "_type", "alignment", "gap", "intent", "shapeIds" ], "additionalProperties": false }, { "title": "Distribute", "description": "The AI distributes shapes horizontally or vertically.", "type": "object", "properties": { "_type": { "type": "string", "const": "distribute" }, "direction": { "type": "string", "enum": [ "horizontal", "vertical" ] }, "intent": { "type": "string" }, "shapeIds": { "type": "array", "items": { "type": "string" } } }, "required": [ "_type", "direction", "intent", "shapeIds" ], "additionalProperties": false }, { "title": "Stack", "description": "The AI stacks shapes horizontally or vertically. Note that this doesn't align shapes, it only stacks them along one axis.", "type": "object", "properties": { "_type": { "type": "string", "const": "stack" }, "direction": { "type": "string", "enum": [ "vertical", "horizontal" ] }, "gap": { "type": "number" }, "intent": { "type": "string" }, "shapeIds": { "type": "array", "items": { "type": "string" } } }, "required": [ "_type", "direction", "gap", "intent", "shapeIds" ], "additionalProperties": false }, { "title": "Clear", "description": "The agent deletes all shapes on the canvas.", "type": "object", "properties": { "_type": { "type": "string", "const": "clear" } }, "required": [ "_type" ], "additionalProperties": false }, { "title": "Pen", "description": "The AI draws a freeform line with a pen. This is useful for drawing custom paths that are not available with the other available shapes. The \"smooth\" style will automatically smooth the line between points. The \"straight\" style will render a straight line between points. The \"closed\" property will determine if the drawn line gets automatically closed to form a complete shape or not. Remember that the pen will be *down* until the action is over. If you want to lift up the pen, start a new pen action.", "type": "object", "properties": { "_type": { "type": "string", "const": "pen" }, "color": { "$ref": "#/$defs/__schema1" }, "closed": { "type": "boolean" }, "fill": { "$ref": "#/$defs/__schema2" }, "intent": { "type": "string" }, "points": { "type": "array", "items": { "type": "object", "properties": { "x": { "type": "number" }, "y": { "type": "number" } }, "required": [ "x", "y" ], "additionalProperties": false } }, "style": { "type": "string", "enum": [ "smooth", "straight" ] } }, "required": [ "_type", "color", "closed", "fill", "intent", "points", "style" ], "additionalProperties": false }, { "title": "Get inspiration", "description": "The AI gets inspiration from a random Wikipedia article.", "type": "object", "properties": { "_type": { "type": "string", "const": "getInspiration" } }, "required": [ "_type" ], "additionalProperties": false }, { "title": "Country info", "description": "The AI gets information about a country by providing its country code, eg: \"de\" for Germany.", "type": "object", "properties": { "_type": { "type": "string", "const": "countryInfo" }, "code": { "type": "string" } }, "required": [ "_type", "code" ], "additionalProperties": false }, { "title": "Count", "description": "The AI requests to count the number of shapes in the canvas. The answer will be provided to the AI in a follow-up request.", "type": "object", "properties": { "_type": { "type": "string", "const": "count" }, "expression": { "type": "string" } }, "required": [ "_type", "expression" ], "additionalProperties": false } ] } } }, "required": [ "actions" ], "additionalProperties": false, "$defs": { "__schema0": { "anyOf": [ { "title": "Draw Shape", "description": "A draw shape is a freeform shape that was drawn by the pen tool. To create new draw shapes, the AI must use the pen event because it gives more control.", "type": "object", "properties": { "_type": { "type": "string", "const": "draw" }, "color": { "$ref": "#/$defs/__schema1" }, "fill": { "$ref": "#/$defs/__schema2" }, "note": { "type": "string" }, "shapeId": { "type": "string" } }, "required": [ "_type", "color", "note", "shapeId" ], "additionalProperties": false }, { "type": "object", "properties": { "_type": { "type": "string", "enum": [ "rectangle", "ellipse", "triangle", "diamond", "hexagon", "pill", "cloud", "x-box", "check-box", "heart", "pentagon", "octagon", "star", "parallelogram-right", "parallelogram-left", "trapezoid", "fat-arrow-right", "fat-arrow-left", "fat-arrow-up", "fat-arrow-down" ] }, "color": { "$ref": "#/$defs/__schema1" }, "fill": { "$ref": "#/$defs/__schema2" }, "h": { "type": "number" }, "note": { "type": "string" }, "shapeId": { "type": "string" }, "text": { "$ref": "#/$defs/__schema3" }, "textAlign": { "type": "string", "enum": [ "start", "middle", "end" ] }, "w": { "type": "number" }, "x": { "type": "number" }, "y": { "type": "number" } }, "required": [ "_type", "color", "fill", "h", "note", "shapeId", "w", "x", "y" ], "additionalProperties": false }, { "type": "object", "properties": { "_type": { "type": "string", "const": "line" }, "color": { "$ref": "#/$defs/__schema1" }, "note": { "type": "string" }, "shapeId": { "type": "string" }, "x1": { "type": "number" }, "x2": { "type": "number" }, "y1": { "type": "number" }, "y2": { "type": "number" } }, "required": [ "_type", "color", "note", "shapeId", "x1", "x2", "y1", "y2" ], "additionalProperties": false }, { "type": "object", "properties": { "_type": { "type": "string", "const": "text" }, "color": { "$ref": "#/$defs/__schema1" }, "fontSize": { "type": "number" }, "note": { "type": "string" }, "shapeId": { "type": "string" }, "text": { "$ref": "#/$defs/__schema3" }, "textAlign": { "type": "string", "enum": [ "start", "middle", "end" ] }, "width": { "type": "number" }, "wrap": { "type": "boolean" }, "x": { "type": "number" }, "y": { "type": "number" } }, "required": [ "_type", "color", "note", "shapeId", "text", "x", "y" ], "additionalProperties": false }, { "type": "object", "properties": { "_type": { "type": "string", "const": "arrow" }, "color": { "$ref": "#/$defs/__schema1" }, "fromId": { "anyOf": [ { "type": "string" }, { "type": "null" } ] }, "note": { "type": "string" }, "shapeId": { "type": "string" }, "text": { "type": "string" }, "toId": { "anyOf": [ { "type": "string" }, { "type": "null" } ] }, "x1": { "type": "number" }, "x2": { "type": "number" }, "y1": { "type": "number" }, "y2": { "type": "number" }, "bend": { "type": "number" } }, "required": [ "_type", "color", "fromId", "note", "shapeId", "toId", "x1", "x2", "y1", "y2" ], "additionalProperties": false }, { "type": "object", "properties": { "_type": { "type": "string", "const": "note" }, "color": { "$ref": "#/$defs/__schema1" }, "note": { "type": "string" }, "shapeId": { "type": "string" }, "text": { "$ref": "#/$defs/__schema3" }, "x": { "type": "number" }, "y": { "type": "number" } }, "required": [ "_type", "color", "note", "shapeId", "x", "y" ], "additionalProperties": false }, { "title": "Unknown Shape", "description": "A special shape that is not represented by one of the canvas's core shape types. The AI cannot create these shapes, but it *can* interact with them. eg: The AI can move these shapes. The `subType` property contains the internal name of the shape's type.", "type": "object", "properties": { "_type": { "type": "string", "const": "unknown" }, "note": { "type": "string" }, "shapeId": { "type": "string" }, "subType": { "type": "string" }, "x": { "type": "number" }, "y": { "type": "number" } }, "required": [ "_type", "note", "shapeId", "subType", "x", "y" ], "additionalProperties": false } ] }, "__schema1": { "type": "string", "enum": [ "red", "light-red", "green", "light-green", "blue", "light-blue", "orange", "yellow", "black", "violet", "light-violet", "grey", "white" ] }, "__schema2": { "type": "string", "enum": [ "none", "tint", "background", "solid", "pattern" ] }, "__schema3": { "type": "string" } } }- systemPrompt内には「あなたは、あらかじめ定義されたスキーマに基づいて、構造化されたJSONデータで応答します」とあり、JSONスキーマ定義も書かれています。
- このJSONデータをclient側ではParseして後述する
AgentActionという形でキャンバスに反映しています
-
これでプロンプトの準備が出来たので、ai パッケージの streamText を使ってプロンプトを投げています。レスポンスを順次 Server-Sent Events でクライアントに返している様です。
client
次にクライアント側を見てみます。コンポーネントの大枠は App.tsx で以下のように定義されています。
<TldrawUiToastsProvider>
<div className="tldraw-agent-container">
<div className="tldraw-canvas">
<Tldraw
persistenceKey="tldraw-agent-demo"
tools={tools}
overrides={overrides}
components={components}
>
<AppInner setAgent={setAgent} />
</Tldraw>
</div>
<ErrorBoundary fallback={ChatPanelFallback}>
{agent && <ChatPanel agent={agent} />}
</ErrorBoundary>
</div>
</TldrawUiToastsProvider>
<AppInner /> 内の処理は以下の様になっています。
function AppInner({ setAgent }: { setAgent: (agent: TldrawAgent) => void }) {
const editor = useEditor()
const agent = useTldrawAgent(editor, AGENT_ID)
useEffect(() => {
if (!editor || !agent) return
setAgent(agent)
;(window as any).editor = editor
;(window as any).agent = agent
}, [agent, editor, setAgent])
return null
}
ここでのポイントとしては tldraw で内部のAPIにアクセスできる Editor インスタンスを前にも出てきた TldrawAgent 初期化時に渡しており、AIエージェントからのレスポンスを受けてEditor インスタンス経由でボード(TLStore)を操作している様です。
全体のざっくりとした図としては以下になるかと思います。
👆の図での「Agentからのレスポンスを変換」する処理で AgentActionUtil を使っています。
AgentActionUtil
実際に何かしらボードを更新するActionを定義したものになっていそうで、現時点(2025/11)以下のActionUtilが存在しています。
| ActionUtil名 | 機能 |
|---|---|
| AddDetailActionUtil | Actionに詳細を追加するための計画を立てる |
| AlignActionUtil | シェイプを軸に沿って整列(上下左右、中央) |
| BringToFrontActionUtil | シェイプを最前面に移動 |
| ClearActionUtil | キャンバス上のすべてのシェイプを削除 |
| CountShapesActionUtil | キャンバス上のシェイプ数をカウント |
| CountryInfoActionUtil | 国コードから国の情報を取得(REST Countries API使用) |
| CreateActionUtil | 新しいシェイプ(図形)を作成 |
| DeleteActionUtil | シェイプを削除 |
| DistributeActionUtil | シェイプを水平または垂直に均等配置 |
| LabelActionUtil | シェイプのテキストラベルを変更 |
| MessageActionUtil | ユーザーにメッセージを送信 |
| MoveActionUtil | シェイプを新しい位置に移動 |
| PenActionUtil | ペンツールでフリーフォームの線を描画(smooth/straight スタイル対応) |
| PlaceActionUtil | 他のシェイプを基準にした相対位置にシェイプを配置 |
| RandomWikipediaArticleActionUtil | ランダムなWikipedia記事を取得してインスピレーションを得る |
| ResizeActionUtil | シェイプのサイズを変更(原点を基準にスケール変更) |
| ReviewActionUtil | 作業結果を確認するためのレビューをスケジュール |
| RotateActionUtil | シェイプを原点を中心に回転 |
| SendToBackActionUtil | シェイプを最背面に移動 |
| SetMyViewActionUtil | AI自身のビューポートを変更してキャンバスの他のエリアに移動 |
| StackActionUtil | シェイプを水平または垂直に積み重ねる |
| ThinkActionUtil | AIの意図や推論を記述 |
| TodoListActionUtil | Todoリストアイテムを更新または作成 |
| UnknownActionUtil | 未知のアクションタイプ |
| UpdateActionUtil | 既存のシェイプを更新 |
RandomWikipediaArticleActionUtil なんかは面白いActionですね 👀
各ActionUtilのベースクラスの実装は以下になっています。
ソースコード
export abstract class AgentActionUtil<T extends BaseAgentAction = BaseAgentAction> {
static type: string
protected agent?: TldrawAgent
protected editor?: Editor
constructor(agent?: TldrawAgent) {
this.agent = agent
this.editor = agent?.editor
}
/**
* Get a schema to use for the model's response.
* @returns The schema, or null to not use a schema
*/
getSchema(): z.ZodType<T> | null {
return null
}
/**
* Get information about the action to display within the chat history UI.
* Return null to not show anything.
* Defaults to the stringified action if not set.
*/
getInfo(_action: Streaming<T>): Partial<ChatHistoryInfo> | null {
return {}
}
/**
* Transforms the action before saving it to chat history.
* Useful for sanitizing or correcting actions.
* @returns The transformed action, or null to reject the action
*/
sanitizeAction(action: Streaming<T>, _helpers: AgentHelpers): Streaming<T> | null {
return action
}
/**
* Apply the action to the editor.
* Any changes that happen during this function will be displayed as a diff.
*/
applyAction(_action: Streaming<T>, _helpers: AgentHelpers): Promise<void> | void {
// Do nothing by default
}
/**
* Whether the action gets saved to history.
*/
savesToHistory(): boolean {
return true
}
/**
* Build a system message that gets concatenated with the other system messages.
* @returns The system message, or null to not add anything to the system message.
*/
buildSystemPrompt(): string | null {
return null
}
}
export interface AgentActionUtilConstructor<T extends BaseAgentAction = BaseAgentAction> {
new (agent: TldrawAgent, editor: Editor): AgentActionUtil<T>
type: T['_type']
}
applyAction メソッドで実際にActionを適用しているようで、TldrawAgent が保持している Editor を通じて適用している様です。
例): BringToFrontActionUtil
override applyAction(action: Streaming<BringToFrontAction>) {
if (!this.agent) return
if (!action.shapeIds) return
this.agent.editor.bringToFront(
action.shapeIds.map((shapeId) => `shape:${shapeId}` as TLShapeId)
)
}
まとめ
全体として、Agent starter kit は「tldraw のキャンバス状態 → プロンプト化 → LLM の JSON レスポンス → Editor 操作」という一連のパイプラインがきれいに分割されていて、自分で独自のアクションやプロンプトパーツを差し込んで拡張しても面白そうという印象でした。
Discussion