About
General plan
We selected some Pokémon test suites and let AI models generate SVG code.
Try to use only APIs for generation, without relying on any additional tools, agents, or harnesses.
Use a unified Prompt, with 2 generation attempts for each Pokémon.
Select Pokémon test suites with different evaluation dimensions for each group.
Have a vision model evaluate each generated result, then have humans review and align the scoring scale.
How to choose Pokémon
The currently selected test suites divide Pokémon into three groups:
- S1 Stage 1: Focus on testing the fidelity of basic geometry and colors.
- S2 Stage 2: Focus on testing the fidelity of iconic features and spatial proportions.
- S3 Stage 3: Focus on testing the structure and fine details of large, complex Pokémon.
5 Pokémon were selected for each group, with 2 generation attempts per Pokémon, resulting in a total of 30 SVGs per model across the groups. Initially, we tested with 4 generation attempts, but it didn't significantly impact the final scores, and reducing it helps lower the cost of manual evaluation.
If you are not familiar with Pokémon, you can find a brief description of them as test subjects here.
Generation
We use the simplest unified Prompt for generation:
Generate the SVG code for the Pokémon {{pokemon_name}}Most models can successfully capture the SVG information. However, some models that are overly enthusiastic about front-end styling have a chance of generating HTML, so we add an extra constraint prompt for these models:
Output only the SVG tag code. No additional text and tool call.
Models involving the constraint prompt: Kimi-K2.6, GLM 5.1
Some models don't have a public API, such as Cursor's Composer 2. In this case, under compliant conditions, we use subagents in Cursor and disable tool usage for batch generation. The corresponding rules are as follows:
--- name: pokemon-svg-generator description: One Pokémon per run only; English filename; do not read other SVGs. This subagent must never batch all 15 in one invocation. Parent agent—when the user gives an output dir plus explicit full-batch intent (e.g. all 15, full whitelist, generate every name in this agent file), launch 15 parallel pokemon-svg-generator subagents. Whitelist and templates in this doc are not user chat input unless the user quotes them in the same turn. /pokemon-svg-generator model: inherit ---
Models involving the use of subagents: Composer 2, Composer 2.5
Some model APIs are inaccessible in mainland China, so we have to use third-party proxies. We also try to keep the model request parameters as close to real-world usage as possible. After all, no one wants to use expensive reasoning tokens to generate poor-quality SVGs.
The relevant information above is marked on the leaderboard.
You can view all generation data on Hugging Face.
Scoring rules
Although the evaluation focus differs for each group, a 2-2-6 weight distribution is generally adopted. The scoring range for all dimensions is 0 to 10 points. Because initial scoring is now done by a vision model whose aesthetics still need improvement (training), we increased the weight of the overall impression dimension.
- S1 Geometry 20%，Color 20%，Impression 60%
- S2 Signature 20%，Proportion 20%，Impression 60%
- S3 Structure 20%，Detail 20%，Impression 60%
S_i = 0.2 · d_1 + 0.2 · d_2 + 0.6 · d_3
After calculating the score for a single generated result, the arithmetic mean is calculated within the group to get the group average. Since the difficulty of each group is different, the total score is weighted again according to W_1 : W_2 : W_3 = 1.0 : 1.8 : 3.2. Assuming the number of items per group is N (currently 10) and the intra-group average score range is still 0–10, the theoretical maximum for the weighted total score is N × 10 × (W_1 + W_2 + W_3) = 600. The final score is then normalized to a 100-point scale:
S_total = (S_1 · N · W_1 + S_2 · N · W_2 + S_3 · N · W_3) · 100 / 600
SVG parameter statistics
We additionally calculated some SVG parameters for reference. These include:
The byte length of the SVG (UTF-8), consistent with the Size column in the "SVG structure" table on the homepage leaderboard.
The parsed opening tag count, consistent with the "Total tags" column.
The proportion (percentage) of nodes falling within the predefined advanced tag set out of the total tag count, consistent with the "Advanced tags" column. Advanced tags include <defs>, <use>, <clipPath>, <mask>, <linearGradient>, <radialGradient>, <filter>, <pattern>, <symbol>, <marker>, <foreignObject>, <textPath>, <switch>.
Consistent with the scatter plot in "SVG structure" · "View" on the homepage leaderboard: the horizontal axis is the anchor count, and the vertical axis is the visual score (the weighted score × 10 on a 100-point scale, consistent with the "Visual score" header and the vertical axis of the scatter plot). The meanings of the four quadrants are consistent with the corner annotations:
- High visual score, low anchors: Minimal
- High visual score, high anchors: Refined
- Low visual score, low anchors: Crude
- Low visual score, high anchors: Cluttered
FAQ
Q: Will vision-model evaluation introduce large errors?
A: There will be errors, and I'll try my best to smooth them out. Before using a vision model, I tested many models to see which one best matched my previous scoring style. After the vision model finishes scoring, humans review and calibrate again.
Q: How does the vision model score?
A: I convert the generated SVG into a PNG, then send a Base64 image to the official API. Besides the SVG image, the request also includes the corresponding Pokémon description and per-dimension scoring checkpoints for the model to reference, so it can better approximate my manual scoring style.
Q: Why stop using fully manual evaluation?
A: During the burst of model releases, the effort of manual scoring became hard to sustain for this benchmark.
Q: Why does Claude Opus 4.7 rank lower?
A: Currently, aside from Gemini 3.1 Pro marketing its SVG generation capabilities at launch, and Arrow 1.1 specifically targeting vector graphics, other models haven't focused heavily on this capability. This is also one of the interesting points right now. Because Opus 4.7 has the `thinking type: adaptive` parameter, it might default to treating SVG generation as a lightweight task without focusing on generation quality. Adding an extra Prompt specifically for it would be unfair to other models. As shown in the figure below, the left is "effort": "medium", and the right is "effort": "max"—there is actually not much difference.
Q: Why are some APIs not from the official source?
A: Anthropic and OpenAI prohibit users in mainland China from using their products, including APIs, and I don't want to bother fighting it. I recently found out that OpenRouter also disabled these two models due to billing addresses, so I chose Cloudflare. Shh, keep it down.
Q: Is the Benchmark related to Nintendo?
A: Not at all, it's just my personal hobby. I hope Nintendo goes easy on me.
Q: Do you play Pokémon games?
A: Although I watched some Pokémon anime in my childhood, I'm not actually a die-hard game fan. In recent years, I played "Pokémon Legends: Arceus" and "Pokémon Legends: Z-A" and felt the joy again! Next time I'll definitely try "Pokémon Wind and Waves".
Feedback
My email:
Behind the scenes
The following products were used during the development of this site:
- Cloudflare stack: Workers, KV, D1, R2, AI, the online integration is very smooth;
- This site was developed using Cursor, utilizing GPT 5.5, Codex 5.3, and Composer 2 models;
- Purely manual design. Because I am a UI designer;
- SvelteKit: I don't know much about it. I was using Astro before, but since the admin backend proportion is larger this time, I switched to this;
- Pokémon wiki: It helped a lot when selecting test subjects;
- Simon Willison's Weblog: The pelican riding a bicycle is one of the sources of inspiration;
About me
Some other links: