Muse 1.2 wrote a terrible "smart summaries" extension for my pi setup. It was sending every single steamed chunk for summarization instead of waiting for the full CMD.

This is an error I would expect from sonnet 4, not a model that was supposedly just a few points behind sol.

I honestly can't believe serious people are making this argument on a straight face.

Gemini 3.7 flash outputs so many tokens per answer it doesn't matter how fast its TPS is, sol will end up being both cheaper and faster than Gemini. So ppl are paying more for a given task, waiting longer and using a dumber intelligence because "TPS number shiny".

Gemini 3.8 outputs 11k more tokens PER TASK on average in AAII than 3.7 putting it dead last in output tokens per task in the leaderboard.

There are numerous benchmarks that measure cost per task, which factors out tokens entirely. Gemini 3.8 flash is significantly lower than Sol on basically all of them

https://artificialanalysis.ai/#cost-tabs

That said, Luna is the undisputed king here at the moment and is what I use as my workhorse model.

>There are numerous benchmarks that measure cost per task, which factors out tokens entirely. Gemini 3.8 flash is significantly lower than Sol on basically all of them https://artificialanalysis.ai/#cost-tabs

Not sure if you read your own link but Sol 56 high ranks smack between Gemini 3.8 flash medium and high. Gemini 3.8 flash comes in as more expensive per task than Sol 56 high according to artificial analysis.

Luna high is literally 30X cheaper than Gemini 3.8 flash high.

You can limit the model viewer and they're getting better at testing multiple effort levels now: https://artificialanalysis.ai/?models=gpt-5-6-sol-medium%2Cg...

One reason is clear: Sol uses dramatically fewer output tokens than Gemini 38 flash https://artificialanalysis.ai/?models=gemini-3-8-flash%2Cgem...

I open the link and I see Flash 3.8 high at 0.58 and Sol at 0.95. I don't understand why you say that "Sol 56 high ranks smack between Gemini 3.8 flash medium and high" but that is clearly wrong.

On cost per intelligence task, Gemini38flash and Sol56 trade back and forth on cost depending on effort level. https://i.imgur.com/zPaWPXx.png As seen in this image, literally: Sol56 high ranks in between Gemini 38 medium and high. The image proves it.

I also included Sol56 xhigh, which ranks above even Gemini38 high.

I don't know if my code is just "complex", but I find that Luna on max ignores the surrounding style and completely ignores logical consequences of a change, like just writing `del arg1, del arg2, ...` instead of dropping it from the surrounding code. All LLMs make questionable decisions at times, but Luna requires so much guidance that it's faster to just type it out yourself. What kind of routine tasks can one accomplish with such a model?

Do you have code formatters, linters and static analysis?

I can get extremely dumb models to get our code style correct because of those guard rails and a specific style document.

It's so funny how many people diverge on the same model.

Ps. For the last week I diverged to Luna too, still need to check 3.8 flash.

But 3.6 flash was my go-to model 3 weeks ago and before it was deepseek flash/pro for a while.

None of the claude models seemed cost effective though.

AA isn't the only benchmark

https://deepswe.datacurve.ai/

This only makes me understand how flawed AAII is. This Qwen model is nowhere close to the other models in that score range.

If you're working on smol? How complex is your work? The agent has to do everything using sed? Did you write your own tools? I guess my question is, why aren't you using pi?

The difference in tokens between the two also makes super curious. The system prompt can't be that different (I'd even bet Pi's shorter) and the 4 tools shouldn't make as much of a difference. I'm gonna have to try it.

smol only has 1 tool: sh

the system prompt of smol is shorter than the system prompt of Pi

smol has no system prompt

system prompt of Pi 0.83.0

""" You are an expert coding assistant operating inside pi, a coding agent harness. You help users by reading files, executing commands, editing code, and writing new files.

Available tools: - read: Read file contents - bash: Execute bash commands (ls, grep, find, etc.) - edit: Make precise file edits with exact text replacement, including multiple disjoint edits in one call - write: Create or overwrite files

In addition to the tools above, you may have access to other custom tools depending on the project.

Guidelines: - Use bash for file operations like ls, rg, find - Use read to examine files instead of cat or sed. - Inspect PI_* environment variables for current model and session details. - Use edit for precise changes (edits[].oldText must match exactly) - When changing multiple separate locations in one file, use one edit call with multiple entries in edits[] instead of multiple edit calls - Each edits[].oldText is matched against the original file, not after earlier edits are applied. Do not emit overlapping or nested edits. Merge nearby changes into one edit. - Keep edits[].oldText as small as possible while still being unique in the file. Do not pad with large unchanged regions. - Use write only for new files or complete rewrites. - Be concise in your responses - Show file paths clearly when working with files

Pi documentation (read only when the user asks about pi itself, its SDK, extensions, themes, skills, or TUI): - Main documentation: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/README.md - Additional docs: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/docs - Examples: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/examples (extensions, custom tools, SDK) - When reading pi docs or examples, resolve docs/... under Additional docs and examples/... under Examples, not the current working directory - When asked about: extensions (docs/extensions.md, examples/extensions/), themes (docs/themes.md), skills (docs/skills.md), prompt templates (docs/prompt-templates.md), TUI components (docs/tui.md), keybindings (docs/keybindings.md), SDK integrations (docs/sdk.md), custom providers (docs/custom-provider.md), adding models (docs/models.md), pi packages (docs/packages.md), environment variables (docs/environment-variables.md) - When working on pi topics, read the docs and examples, and follow .md cross-references before implementing - Always read pi .md files completely and follow links to related docs (e.g., tui.md for TUI API details) Current working directory: /workspace """

DeepSeek being DeepSeek. v3 and R1 went over the same and had multiple versions

Source? The most trusted benchmark right now (deepSWE) scores better or just as well on their minimal harness than when using CC or codex

deepSWE clearly doesn't need complex tool calling?

I wonder if you're as cynical and untrustworthy of American companies as well or is it more of a racism kinda thing

Everyone should distrust them equally. Only local agents in a detached network namespace are safe from data leaks. It is perfectly reasonable to assume they are using our sessions to train on, since everything else short of nuclear launch codes is already there, and they need to keep feeding it.

This is an extremely weird comment that doesn't add anything to the conversation.

Here on HN we discuss facts, jumping straight into racism has no place here.

Everything is more expensive than deepseek. They aren't frontier in intelligence but they are the frontier in cost per intelligence

What surprised me the most was the fact that people have enough disposable income to pay for search to make this viable

Search is one of the most critical pieces of technology for most people on the web. That, combined with the point another comment made that indicates the demographic of Kagi largely revolves around IT/SWE (who generally have more disposable income than other types of jobs), shows good reason why Kagi could survive long term