Vegas was such a great solution to gambling for Americans. We had it contained to a specific place, making it easy to access but also easy to avoid.
Cryptocurrency, which turned into just another form of gambling, seems to have been the corrupting influence that allowed it to spread into people's homes. I hope we can reverse it somehow.
First, congrats to the team on launching something genuinely interesting and new.
Seems like a more accurate title would be "Jev: Trading general purpose generation for fast typed inference" or something like that.
This is interesting, but the speed comparison seems misleading? A generative model that can output code in a Turing-complete language can do anything a computer can do.
Jev can only generate structured output, right? This is probably super useful for classification/routing/scoring, but it's nothing like the code generating models we're all using today for code and automation.
Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value. You can enforce structured output from an LLM too, with an appropriate harness, etc.
Assuming there's no funny business, the Doom demo is cool.
When they say "can't hallucinate" they mean they produce a confidence value for every result, so you could see for example it has 0.1 confidence, and you can disregard the result - that'd be different from hallucinating where it believes it's correct
I would be happy enough with: only produces what it can verify with sources.
If you eg try to remember a court case (ie produce the reference via LLM token generation only), it's easy enough to check with your data whether it really exists. Similar for following links and other references.
If your data or sources are wrong, obviously your report about them will be wrong. But I wouldn't call that a hallucination.
It's not a binary thing. You can get closer or further away from that standard.
And humans also behave differently in different contexts. A conversation at the pub has more such hallucinations than a formal deposit in court. For the latter, a good lawyer will look at her shoes, when you ask him what colour her laces are.
No, I don't believe so. Hallucinations are not "high probability" in a real sense. They are an artifact of the random walk the inference algorithm takes, which causes it to latch on to and chase attractors in the noise. This random walk behavior is necessary for chat interfaces to be useful, but are less critical to typed output predictors. I'm guessing they found some optimization that is possible if you give up caring about chat.
What we would want to see if a confidence value that is in line with the actual correctness. If the value is 0.9 for 1000 different answers, then approximately 900 of those answers should be correct.
The probability values don’t really represent confidence in modern LLMs though, especially after RLHF and RLVR.
System One says they use RLCD, Reinforcement Learning for Calibrated Decisions, which presumably has accurate probabilities as an explicit optimisation goal.
RLVR generally upweights tokens along the whole thinking trace that led to a correct answer, whether each token was "correct" or not. RLVR doesn't train a model to output an 80% likelihood, it just trains it to produce correct answers, and not to produce incorrect ones.
System One hasn't said how RLCD works, but they do say it is explicitly training models to output "calibrated" probabilities, which makes it distinct from RLVR. This is how they describe it:
> System One models are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty.
In RLCD (which is now an RL acronym that has 3 different unrelated expansions!), you basically massively negatively reward a distribution that is {yes: 0.9, no: 0.1} if the answer was no, and less negatively reward a {yes: 0.6, no: 0.4}. Many nuances when designing the details, but that is the rough idea.
It is a known existing thing variously called "calibrated RL" or such.
Implementing it on top of LLMs was difficult to get it to work, they seem to have done it up so its good enough for a polished product that works in a wide variety of usecases at the same time. I got accepted from the waitlist and it's really neat. Edit: it is now on vercel gateway.
One thing to note, the out of distribution behaviour will be different from what we are used to with regular LLMs. Theoretically, it should be worse, but practically, it depends on their method.
that's right, but because these models are probabilistic, it's also possible to be confidently wrong (and all future models will be smarter still and still have that possibility)
Correct. Not to say we're getting into the weeds of probability here as well.
"What are the odds a thunder will strike in Paris at 1pm UTC of 2026-09-16" - that could be a 0.001 chance going from blind historical measurements; 0.01 if it's raining; or 1 or 1 after the date has passed.
Nothing, but imagine using LLMs for a classification task
People out there are so resigned to the models being unreliable that they are really doing things like hallucinating deliberately, and then matching the hallucinations to embeddings -
I'm certainly not resigned to that, at least for classification.
Even non-frontier models are absurdly good at this in a broad sense.
Which would make it hard to judge "a model that will never produce unreliable outputs in the first place" against something that is already really, really good and exceptional in domain-specific areas with the tiniest amount of elbow grease.
What about the LLM calls though that are done midchain? In the Home Assistant video the multi-intent prompt gets split using what looks like a traditional llm model, which I'm assuming is vulnerable to classical hallucinations.
Has LLM become so synonymous with Generative Transformer that other high-parameter count models that interpret language need a different name?
For all we know this might be a non-language-generative transformer e.g. a transformer where the decoder produces confidence scores rather than language. Please provide more likely architectures if you know them, I'm genuinely curious.
I think the meaning of can't hallucinate in this model is that the type won't be hallucinated.
So if the generated schema is for a tool call for calculator, then the numbers will be valid numbers for sure (and not random words).
To me, it looks similar to BNF schema already introduced and implemented few years ago: generally speaking - it limits the next token that is allowed to be generated, probs are drawn from a subset tokens.
(tbh, I'm not sure why it didn't pick up as a more standard interface to LLMs, as it made a lot of sense back then, and now.)
It did, no? If you send a JSON schema to an LLM endpoint, at least for OpenAI and Anthropic, they turn it into a grammar behind the scenes and use it for constrained decoding.
Yeah, I thought about constrained generation as well. I've actually done something similar with local models before. And you can even get a "confidence" score by looking at the logits (something along the lines of logprob("YES") + logprob("Yes") + logprob("yes") - logprob("NO")...
There's also a cheeky "one of the models hallucinated a link" in the wiki jump example that most likely could have been avoided by properly using grammars. You can setup constrained gen so that only valid options (say from a list) can be outputted. Their own inference lib likely does that. So comparing to one that doesn't is a bit cheeky.
That being said, after a brief look at the site I could see this working. Especially if this can be ran locally, the speed and cost can enable some workflows where you have this as an "overseer" layer over say a cli agent. After each step you run through a list of "questions" ("is the task completed?" -> yes -> "does the edit touch files it shouldn't" / "does the edit follow our code writing policies") etc.
edit: extra points if the "question" rubric is also generated by a higher abstraction model. Say "/goal Build out auth" -> generate_rubrics(goal) -> "Is auth implemented on all endpoints" / "Has code touched anything else than auth" / "is this following the best practices" / ...
AFAICT it is the same interface as you describe, but the underlying inference algorithm is fundamentally different, hence the speed gains. There is an application I am currently working on right now where this typed output predictor is the performance bottleneck. I'd be very interested to see how this performs.
I don’t think it’s misleading if you compare on the use cases they suggested. It’s faster and cheaper (no idea if higher quality), so it’s immediately interesting for certain things.
And if you buy their RLCD claims, this might be even better than huge models that know a bunch of irrelevant things.
This is likely still an LLM (in the purest definition of a language model with relatively many parameters) since the inputs are natural language, just not a generative LLM as the output is something other than more language.
It is frontier in the sense it is exploring an unexplored domain. I do agree on questioning the comparatives though. Speed/cost is indeed relevant for problems that can be framed as structured decisions only. The question is, would defining a structured decision model be a structured decision model itself? This would significantly increase the application domain.
How is this not a frontier model? It's bleeding edge in its own niche. It's not a frontier LLM; however, applicable to many of the things people use LLMs for.
It's nothing like a traditional LLM and so should not be compared to one. It's a heavily constrained, tiny model that can only produce a probability score or a yes/no answer over pre-defined selections. It has no long-context capacity.
I mean, imagine comparing this thing to Astra, it's hilarious. They don't even tell you what the max input size is, and they only allow 10 possible answers to choose from for the Choice mode. It's probably like a 1billion param model. They say it's "not small", but there's zero reason to believe that.
I suspect someone will be able to recreate this within a week by piecing together open-weight models.
> It's nothing like a traditional LLM and so should not be compared to one.
Frontier LLMs are expensive jack of all trades. You can absolutely compare them to purpose-built tools on any domain they touch. Engineering is all about assessing tradeoffs.
I'm biased but I wouldn't call it misleading - generating text is super awesome and flexible, (we describe that in the blog post - and I personally use string models all the time) but it's true you pay a high tax for autoregressive generation
> Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value.
that is likely true of all ML! perhaps we could debate semantics, but I don't think it's fair to say a random forest "hallucinates" in the way LLMs do
His claim was that the title is misleading, not sure how it's relevant to that claim that you use "string models" (full LLMs).
The original title before it changed less than an hour ago was:
"Jev: New frontier model 40-400x cheaper and 20-200x faster"
I'm going to agree that was misleading.
And on the second point:
>>Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value.
>that is likely true of all ML! perhaps we could debate semantics, but I don't think it's fair to say a random forest "hallucinates" in the way LLMs do"
Also going to disagree here, and I don't think it's semantics.
No. Your launch post puts “0%” on a hallucination chart, then explains that the number comes from guaranteed schema matching.
You’ve already agreed that this doesn’t establish correctness. An approve for an unauthorized action still meets the schema guarantee.
That’s why I find the messaging misleading. You’re acknowledging the limitations in these replies while defending the broader reliability pitch.
Even granting that each answer is calibrated individually, that doesn’t establish calibration of the decision that combines them.
Sure, I can threshold a composite score, but there may be many wrong answers with the same score. An unauthorized action doesn’t become acceptable because it scores highly on the other dimensions.
I still have to define the constraints and test which wrong actions get through the complete workflow on my own data. That’s a substantial part of the work being pushed back onto the developer.
And furthermore, because the model is forced to answer in a boolean (if in boolean mode), if the user input is outside of the range of a boolean, it's forced to hallucinate. It can't abstain.
User input: "Hey, have your human support agent call me, tomorrow at 5pm."
Model input: "Does the user want to speak to a human support agent?"
Output: Yes.
I imagine that your model would produce this, and I think it's fair to say this is a hallucination. A human would caveat it with: "Yes, but not right now.", your model is incapable of that. Yes is technically correct, but within the context of being in a live chat, a human would understand that the caveat is required.
No, the model has answered correctly. Your question is poorly phrased (possibly deliberately).
Your question would correctly classify the user's input as requesting a human support agent, but at an indeterminate time.
If you wanted to determine whether the user wants to speak to a human support agent immediately, you would have to correctly qualify your question, e.g. "Does the user want to speak to a human support agent now?". You could have another question which is "Is the user requesting a call-back from a human support agent?". Or you could have a multiple choice query which would filter the conversation into one of a number of pre-written possibilities.
This is nothing to do with accuracy or hallucination. It's a different method of interacting with the model where you are relied upon to be precise.
"Hallucination and type-safety are intrinsically related"
I'm not entirely sure why we're conflating type safety with, I guess, value or output safety.
"Would you say a linear classifier hallucinates?"
No, but it can be (and often is) mathematically correct and functionally incorrect. It doesn't help to say "a linear classifier can't hallucinate" when you get even 99% accuracy. That's 100% a semantic play, and it doesn't help when the picture of a dog is labeled cat and the response is "yeah but that's not a hallucination, only stupid LLMs do that"
Let's say classifiers don't hallucinate. To make a fair comparison we should constrain LLMs to the same classification task. In that case, no, LLMs also don't hallucinate.
- Give Jev and LLM the same input
- Lock down both to approved/rejected/unknown (LLM restricts on decoding)
- Both can be wrong, but neither can hallucinate (invent an another option).
A hallucination in the context of LLMs is generally understood as an incorrect answer presented as factual. If you claim that "x can't hallucinate" in the context of LLMs, you're saying that x always gives accurate answers. It does not matter whether the answer is type safe. If its value is incorrect, it's a hallucination.
Just to be sure that I understand, you're saying that your model "can't hallucinate" because it only outputs a single thing, right? In this way, an LLM can't hallucinate either if I prompt it to do a classification task with a discrete set of possible outputs, right? (Assuming I reject non-conforming output. Actually, maybe what you're saying is that your system can't output non-conforming output?)
For e.g. classification tasks, even in 2026 people are doing things like hallucinating deliberately, and then matching the hallucinations to embeddings -
From a quick look at this it looks like it could easily generate natural language text by following a structured representation like UMR (Uniform Meaning Representation) or the similar representation the Abstract-Wikipedia folks will be working on for generic encyclopedic text (which will be heavily informed by Universal Dependencies). These are basically linguistically principled and frame-based counterparts to a programming language AST, that can be then converted to natural language (in a broadly language-independent way, to the extent that semantics and pragmatics make that feasible) via some sort of NLG rendering.
(To be clear, this one raw model does not support outputing a full AST directly - it wants to output "choice" among fixed options, "score" on a sliding scale, or a true/false answer (all of these with confidence scores attached), so building the AST/structure would be a code-driven (or even perhaps outside LLM-driven in some more challenging cases) multi-step affair where the model would essentially be playing a "game" of building the structured output step by step and getting a revised partial state back. But one could expect this to lead to interesting results.)
Only if you think that everyone cares about self-driving. Lots of niches require structured domains; self-driving is just one that has a lot of capital thrown at it.
> Assuming there's no funny business, the Doom demo is cool.
The Doom demo seems very funny business. They're not feeding it video, they're feeding it a text description of what's going on in the game. It's not reading pixel data.
I think LLMs would play a lot better with that input too but Jev does seem to have a huge speed advantage; I don't know if the other models could do that in real-time.
> but it's nothing like the code generating models we're all using today for code and automation.
Is this true? Code is structured output. At the very least it seems like a question of degree rather than kind.
While the LLMs we're using today are limited to sequenced text, it seems that a model like Jev could excel at coding on a more structural level (factoring, controls) by working within the constraints of an actual language specification and supplemental domain model. I don't know, though -- maybe that's too deep and complex.
I feel like the power of the approach presented here is that it gives a model a proper "language" to describe computations directly vs moving tape silliness.
I foresee this to be the path moving forward - giving AI models understanding of the computation directly(as well as compositional rules) This feels like a short path towards total software in many areas.
Agreed. It's a wildly dishonest presentation of their product from many perspectives, which is a shame because it might actually have some good use cases.
The comparison between LLM speed and Jev speed is misleading, because they're using autoregression to generate all of the type names, all of the schema, etc. A closer comparison would be if the LLM was purely outputting the raw numbers. Even then, comparisons to LLMs are pointless because you could train a transformer on the same sort of task that Jev is doing and get even better performance yet again, and a smaller model. I suspect this is some form of stripped down diffusion language model.
You really have to do a lot of hand holding here, and map out your problem space manually, and very carefully, to get any sort of accuracy. For example:
> Keep each Score to one dimension. If a description says “punctual and smart and experienced”, the question is measuring three things, and an input that is high on one and low on another can’t be placed. Confidence drops and the score means less. Split it into one Score per thing and combine them in code
If you don't perfectly represent the distributions of possible answers then you'll likely get garbage results. As far as probabilistic state machines are concerned, I'd say creating the distributions of possible answers, and their hierarchy, is the actual hard part.
One of their examples is:
- "state": "I have asked three times now. Can I please just talk to a real person?"
- "Is the customer asking for a human agent?"
Imagine the users request is: "I want your human agent to call me tomorrow at 5pm."
Human conversation is fuzzy, getting useful reliable results out of this is going to be a challenge. Of course, you could add follow up checks like: "Do they want that now, or later?" -> if later -> "Do they want that tomorrow, or the day after?" and so on... But now you're building an LLM out of if statements. I am skeptical of whether this model has much utility for fluid language interpretation - I suspect it'll only be useful for scenarios where you've tightly constrained the answer space but want to use fuzzy language to describe it. Like:
- Question to human: "Would you like a support agent RIGHT NOW?"
- Their response: Yes | Yeah | Mhmm | ye sure (any possible yes signal)
Model input: "Did they ask for a support agent?"
Still... a tiny LLM could accomplish this sort of thing without problem. And that doesn't stop someone from saying: "No, not right now. But tomorrow." - and the tomorrow would get missed. I think this is why people haven't really tried this approach much already.
Also their Doom demo is on structured state, not on images. Meaning, the enemies must be being served to the model as coordinates (or the exact angle of projectiles that hit the player), otherwise it'd have to scan every pixel of the 360 degrees to know whether an enemy is in front of the crosshair or not. You can see from the map below that it's also choosing travel checkpoints/destinations through walls. So they've severely cooked this to make it look far more capable than it is in practice, and any speed advantage that is offered here is not factoring in the shortcuts it is taking, the training on the map, and the fact that it can cheat because the structured state it is using is not bound by obstructions.
For side projects I pretty much exclusively use Luna xhigh. The $20/mo plan with the recent generous resets is more than enough for me. Sometimes I reach the 5hr limit, but haven't reached the weekly limit yet.
The most recent project it finished was a SIP client for an ESP32 in-wall touch panel that I got from AliExpress for $50. It rings when someone is at my doorbell and let's me answer calls and see video. Yes an ESP32 can stream H.264 video :D
My only complaint with Luna is it seems to give up when the work is half finished, and I often need to tell it to continue. But I feel this is mainly a harness problem. I just use it in ChatGPT/Codex as it gives me easy remote access to check in on what it's doing.
(At my dayjob I usually spend $200+/day with Opus/Fable)
Use of closed models is unprofessional, and depending on your field negligent. The fact that it has been widely normalized does not make it less so.
You're handing over your (presumably your customer/employers) data to an unaccountable third party which has demonstrated itself willing to commit criminal acts, and to take other people's data without permission. Your ability to continue to perform this work can be withdrawn at any time for any (or no) reason. You have little ability to validate that the work is being performed as expected and isn't being silently nerfed or outright subverted based on competitive considerations, bribes, overactive 'safety', or cost management.
Outsourcing to a black box would be a reasonable expectation if you asked a non-professional to perform the work. A professional should be able to account for the tools they use.
> Your ability to continue to perform this work can be withdrawn at any time for any (or no) reason.
Thanks to the fact that there are no widespread stories about this actually occurring in practice, at least not yet, people do not take it as a relevant risk at the moment.
> You have little ability to validate that the work is being performed as expected and isn't being silently nerfed or outright subverted based on competitive considerations, bribes, overactive 'safety', or cost management.
Yes, I agree that this is a real concern that many people might rightfully have. And I am unaware of any way to mitigate this concern while using black box AI models. Because the only thing that their creators can do is to tell their customers: "trust us". But there is no way to objectively verify whether they serve tainted AI model responses or not.
I think that's a bit strong of an assertion. How many people use copilot daily under an enterprise agreement? I don't necessarily disagree in spirit, especially given the questionable data sanitization around the recent Navier-Stokes announcement, but most companies disclose huge amounts of data regularly to hopefully-trusted third parties. I think the internet -- and a good share of the world's commerce -- would grind to a halt if we suddenly stopped. Setting up, securing, and maintaining local models for even a small user base is non-trivial and there is way more demand than supply for that skillset right now.
I disagree; I just spent 15x dogfooding some Claude setup I rolled out to the org making changes that would have cost me less then a dollar had I used Luna and I would have got the same, if not better results; better because it would have been faster so I could have iterated more.
That way I have a chance to diff with the previous iteration and clean up comments, modify skills, etc. also if it bonks on a step I'm one snapshot away from trying again...
Is there a place people share their workflows other than HN comments?
I wish I knew, I haven't had any place to point people to. I'm going to start sharing this on YouTube since I already spend a few hours each week talking some friend or user through the latest best practices.
I just tell everyone to use Fable 5.1 for everything at this point. Astra is unfortunately a dud, I'm sure they will try to fix a bunch of it with GPT-6.1 but OAI has had this issue for awhile now where every other generation has some sort of strange tic, or reward hacking issue, or something. It's almost like they are balancing the RL on the tip of a needle.
Opus 5 has issues too, comment-slop, claude-ish, etc.
5.1 on the other hand can seemingly do no wrong. Easy to work with, writes human-level code. Expensive, yes, but even at Low effort it's well worth it.
Quite good as work horses? To me Luna is the work horse and Sol and Astra are prancing thoroughbreds. If I use Sol or Astra for anything other than curated reasoning and planning I will burn through my usage limits in an hour.
Same. I use Luna for research/scout/test subagents, Sol for coordinator, Terra for delegate, and Astra high for review and simplify. Even with Astra in there, it’s fresh context, and I’m consistently amazed by how far I can stretch my $20 subscription with really good results.
I used to do this but recently I switched to having Fable 5.1 spawn forks of itself rather than Opus subagents. Yes it's more expensive but you don't pay for reads that already happened pre-fork, and you end up doing less rework since Fable agents are just much smarter.
I agree with you on this. Opus feels tedious and it cannot be stopped from doing change-narration comments, but Fable feels like a real collaborator. I am usually pretty happy with the code it writes.
- Occasionally has strange tics around asking for permission for obvious next-steps, implied actions, etc.
- It's very expensive, both in terms of tokens and % usage on subscription plans.
- Relatedly, effort level is unintuitive. Sometimes it seems like higher effort levels are actually cheaper due to not under-thinking and needing to correct work. But other times they are overkill and send the model into rabbitholes.
That said, it's fantastic as a code-reviewer or "hunter seeker". It's better at finding bugs than Fable and "Get this well articulated task done single-mindedly" is an Astra-shaped task.
I gave up on fable 5 after I asked it to critique my PR and it spit out a page of complete nonsense technical jargon. Like, to the point that I had to review the feedback with other models and try to parse what it was saying and ultimately it wasn't even right. Compare to Astra and Sol where I can almost forget there's a model and just speak/read naturally.
I think I should give 5.1 another chance but I am just so triggered by the way it talks after spending so long battling fable 5.
Also I'm starting to wonder if the latest round of models have finally saturated for my personal coding needs. I mean obviously not for taste and judgement, but those barely seem to improve with model generations. For just spitting out a 1000-line feature I've vaguely scoped out, Astra feels basically as good as I need.
Interesting, in my experience Astra is a marked improvement over both Sol 5.6 and Fable 5.1. Its output feels a lot more natural, and it is just less "dumb." But individual experiences may vary.
It's a great model and you're right it does feel quite natural at times while Fable 5.1 still has a claude-ish shape to it. Unfortunately I just find that it's not reliable enough as a daily driver and ends up performing specialist tasks rather than being the primary pane of glass.
Astra is quite crap (enters reasoning loops like Gemini used to and fails to actually work on a task - would say yes this needs fixing, so I say go ahead and then it will spend half an hour coming back with yes this needs fixing and not doing any fix) and Fable/Opus unusable in many instances (they struggle to generate coherent English let alone code).
Out of these only Sol is quite useful - actually finishes a task, though you need to interrupt often as it likes to wander into its comfort zone.
Since you asked, the answer is that I built and use an agent multiplexer called Clor https://clor.com
I have a $200/mo Claude subscription and a $200/mo Codex subscription, and I'm signed in to both. The Docker containers keep each session isolated, so dev servers, browser testing, etc. can work without conflicts.
It includes `/ask-claude` and `/ask-codex` skills that I use very frequently to have the Claude or Codex harness call out to the other one for advice on plans, bug repro, code review, etc.
The agents run in total "yolo" mode, so there are no permission prompts to approve. The risk is mitigated by the Docker containers (which don't necessarily provide a security barrier but do limit accidents).
I was doing this manually in Ghostty tabs for a long time, and it got painful, so I built a much more sophisticated version that I (and my friends/colleagues) could use.
Generally using Claude Code with Fable 5.1 (high) to plan and implement (Opus 5 (medium) as the implementer subagents), and using Codex with Astra high to review the plan and review the implementers' output.
I use a workflow that has different named subagents. [1] Agent profiles can be pinned to models. So you set the model you want on your main thread as the orchestrator. Create an agent for the "planner", "implementer", and "reviewer" and set the model you want for each. Right now I am orchestrating and implementing with Deepseek, planning with Astra, and reviewing with Opus.
I am doing this with the Pi harness right now. To use a Claude monthly plan you need to use the pi-claude-bridge plugin.
If you are using just Claude for example you can use Sonnet as the implementer and Fable/Opus as the planner.
Midwit: "No, you see, you need a deterministic 12-stage multi-agent orchestration framework with vector embedding semantic routing, and five open weight models with custom harnesses!"
Working on Clor, an agent multiplexer for Claude and Codex with shared memory. It's free on your own machines.
The idea is to help developers multiplex agent sessions and run Claude and Codex together, which is much more powerful than running one or the other alone.
My secret goal is to figure out how to make software development the kind of focused, meditative work it used to be.
These are the sober and informed takes that we need more of.
> the claims from Coxon and his ilk are the most extraordinary a technologist can make, and we must demand evidence commensurate with the claims.
Yes, exactly. These claims do not have sufficient evidence.
> ...you had nothing to fear then — and (at least with respect to extinction risk!) you have nothing to fear now.
Wait, this is another extraordinary claim without evidence, right?
Unless you're going to dispute the power of AI you do have to acknowledge the danger of AI, and that does include the very real possibility (however small) of existential risk.
bruh what? Are we reading the same blog? Bryan says the responsibility is the person making the extraordinary claim. If tomorrow Jacob Coxon comes on CNN and says "Jacob Gold is an extinction risk to humanity". What are we supposed to do? Put you in a bunker and never let you see the light of day until someone proves otherwise?
If someone doesn't accept an extraordinary claim without evidence, that doesn't mean they are making an extraordinary claim.
What's the evidence for there being an existential risk? We are provided scenarios which read like science fiction about RSI and ASI right around the corner, resulting in magic sounding technology that can do anything the person making the claim wants it to do, because it can just make itself smart enough in a short amount of time. But the person making the claim has to actually show how such a thing is possible in the real world, not just a story.
The clear and predictable power of AI is the evidence for existential risk. Even if we don't get RSI or ASI, there's a risk we'll automate our world, then it'll break and we'll starve, etc.
> The clear and predictable power of AI is the evidence for existential risk
* The "predictable power of AI" is very advanced predictive text. What a lot you can do with that, and there are clear limits.
* AI has no intent. The greedheads who find themselves in these positions of power have clear intent (often but not always bordering on and actively becoming misanthropic) put their intentions on AI - hence to doom mongering
* Who will starve with the failure of agriculture? A few, a lot, but not everybody. We are good at this - have been doing it a lot longer than computing or science
Good god. You are burden shifting.
a) big claim no evidence. b). says a has no evidence; therefore, I will continue to not let fear manipulate me into believing a claim without evidence.
I have a bridge to sell to anyone who believes a big claim without evidence.
There are no clear and plausible pathways to human extinction level events that AI makes worse or more likely in any way. It is obviously not an extraordinary claim at all.
The students in the computer lab had no guarantee that they wouldn't, say, download a copy of Napster infected with the CIH virus later. The fact that they were not under imminent threat from some kind of Hollywood-style network worm did not mean they were immune from more realistic attack vectors.
Likewise, one can quite reasonably say there is no credible existential, Hollywood-style threat from AI in the foreseeable future while recognizing far lower-stakes, yet important risks that need to be addressed.
the earth has been 1 decision away from explosion for nearly a hundred years now. do you anticipate a change to this agenda very soon? personally I imagine the same trajectory continuing.
certainly I behave like the article's author and yourself all the time, this isn't meant to be some sort of moral point. watching all these smart people talk so confidently regarding things nobody knows about, it reminds of the confidence ai shows in hallucination. would you say that the singular choice of vasili arkhipov did not prevent annihilation?
> would you say that the singular choice of vasili arkhipov did not prevent annihilation?
He prevented a catastrophe, but not annihilation. We were not at risk of that in 1962 and even if he had decided to go with the others, more decisions would have been needed (not just his) to fully escalate to full scale nuclear war. I will reiterate: We have never been one decision away from full scale nuclear war.
But in case you don't understand why, it's because no one person can actually launch all the missiles. And considering the two major arsenals (US and USSR), there has never been a time when two people could make the same decision (launch) and actually launch all the missiles. The orders still have to go out and acted on, many decisions have to be made in order to have full scale nuclear war and come close to annihilation.
I like the way you frame your opinion as a truth with an obligation to be understood. still I disagree that a decision can only be said to affect its direct successors.
If you think AI will be powerful enough to change the world in huge ways, then it does seem reasonable to assume some level of risk of destroying the world too.
Stating that the probability is zero when we simply don't know what the probabilities are does seem like an extraordinary claim.
Imagine how reassuring it would be to people if we had evidence that there's no existential risk?
> If you think AI will be powerful enough to change the world in huge ways, then it does seem reasonable to assume some level of risk of destroying the world too.
There's an opportunity cost in the doomsday prophesying, though.
The media coverage of "these extremely capable robots might kill us (according to the guys who sell the robots)" comes at the cost of coverage of the real, present issues surrounding the tech oligarchs that we're already facing.
Sure, there are some failure modes that lead to some really bad stuff, but human extinction seems exceedingly unlikely to be one of them, and "10%" is pulled straight out of Dario's derriere.
Far more likely are economic disruptions that impact the tenuous balance between labor and capital and lead to unpredictable societal upheaval.
No it does not seem like an extraordinary claim, unless this is your first time hearing such claims. For the rest of us, we've heard this every decade and it turns out to be entirely untrue.
Nuclear, Overpopulation, Peak Oil, Y2K Bug, Global Warming.
Someone who has spent $100M+ on Google Ads explained to me that Google is doing everything possible to juice their revenue right now, in ways they've never seen before.
Seems like two things 1) Google is losing at AI and Sundar wants to mask this 2) AI is going to destroy their ad business and they want to get while the getting is good.
I think the 3.8 Flash release shows they are doing just fine at AI. Unlike OpenAI and Anthropomorphic, Google doesn’t have to release new models to capture VC sentiment every three months. They have the hardware, data and money. And when they release a model it’s very good.
They can be doing fine at AI models but not so well with AI revenue.
Although I am curious if they are not worried and playing the long game confident that their distribution will eventually help them catch up (and worst case they own some of Anthropic). But obviously Anthropic and OpenAI have their own distribution now too.
Nobody is doing well enough with AI revenue ATM (https://isaiprofitable.com/). At the moment it is probably more about market share of users, that is where the investment capital will need to be clawed back from when it starts to be called back (or replacement money will need to come from if the investment cycle simply stops giving generously).
While true, Google is particularly vulnerable to AI disrupting the traditional internet ad model. They have to figure out how to make AI profitable or Google will suffer catastrophic revenue, profit, and stock price declines in short order.
Can they not just do what they expected sites to do when their search results started including key extracts from sites, significantly reducing click-throughs to news pages and such?
[couldn't resist the snark, I don't exactly feel sorry for them here!]
I think this is their obvious strategy. What's not so obvious is that they'll succeed. Their share of consumer LLM users is around 27%. Their share of search engine users is above 91%. If current growth trajectory holds (which is admittedly unpredictable), Google will command far less of the global internet ad market.
I expect aggregate ad revenue to climb, so this might sting less, but much of Google's stock price imputes their dominant market position (and implied moat). A large proportion of users don't first navigate to Google anymore. They ask ChatGPT. Commoditised LLMs (or worse, a dominant provider which isn't Google) is a very bad scenario for Google.
I think they're going to have to diversify. Cloud compute revenue keeps climbing. Their deal with Apple for "small" LLMs suggests they might have opportunities to specialise models for specific tasks. They still command a massive lead in things like Maps and Gmail, and they can lean on those.
I do wonder if the more they goose their ad business the faster it eventually falls. I know I am more and more reluctant to view many normal Web page ads at all and usually immediately click away the second a site gets annoying
I block ads because I don’t want my family exposed to scams, clickbait, and offensive content. If ads weren’t so annoying, I wouldn’t put too much effort into blocking them.
Also: Creepy surveillance, and a delivery mechanism for malware/viruses.
Online advertising has completely broken from anything resembling the traditional social-contract of magazine ads, billboard, signs, etc. They're preying on your safety and privacy, not just your attention.
The other companies are stealing people's work for headlines and sabotaging each other. Google is offering a whole bunch of Gemini APIs that do real shit, with the same terms and contracts and the existing products for most of them.
A large chunk of the population don't use Google to search anymore. They go straight for the GenAIs. That's got to be scaring a lot of managerial folks at Google.
Slide 65 of this (very nice) set of slides [1] broke down search vs genAI use to get information. "Mostly genAI was not breaking 25% in all age brackets... Curious to see how that will evolve
The Gen AIs are doing the search for people. So hopefully this forces google to switch their business model from manipulative ads against human users to just charging for API access for Gen AI models.
No, Google will just do "product placement" in the AI output.
P.S. Google's real problem is that their web search is absolutely atrocious and completely broken. Even Qwen is better at searching the web than Google's AI search box.
I think Google's big bet (and Microsoft's before them) is in centralization of AI services, hosted using their model, and that AI at the edge will always be a bit player.
But what I see and hear from technical consumers, businesses, and govts is that they would prefer their data stay local, and like the idea of edge compute being used for AI.
Claude/OpenAI's direction with their Apps enabling deeper integration is a good direction, but the trust with data is still a growing concern. OpenAI's math actions on the NS problem degrades their reputation a lot.
Google’s cash cow is ads. I think the writing on the wall is ChatGPT and other AI search engines will dominate. It is to be seen how Google will respond. But in the end, their monopoly on web search is coming to an end.
Generated via ChatGPT, as google-ing it would have taken too much time:
Rank Cloud Approx. annual revenue Growth My assessment
1 AWS ~$129B ~20% Clear leader
2 Microsoft Azure ~$90B ~40%+ Strong #2, rapidly closing
3 Google Cloud ~$59B 36% Fastest-growing historically, but smaller
which is to say that Google is still way behind AWS and Azure. Also, lots and lots of people have switched to using AI-interfaces now, I'm not hearing "I've googled that and that's what I found" anymore, it's all "I've chatgpt-ed and this is what it told me".
It's not about releasing models. Consumer market share of chatgpt is still >60%, if not more. Their search engine was hit and as soon as product search starts declining on Google this will be a code red. They're in an extreme nicle and dime mode.
How are they going to generate revenue if their old model is Ads on search and the new model is yet to show how revenue can be generated. ChatGPT (on free) is showing ads now but they are not targeted and I highly doubt they are worth anything.
very HN take. Reality is 3.8 flash is good at what it does (being fast + cheap)... but that market is becoming very saturated and google is not competing at all at the frontier where major spend lies.
It's ridiculously fast, and generally very good with real-world knowledge. It's my go-to for anything non-coding, and for quick fixes/questions/prototypes.
If metrics measured what most people need from an LLM, it's competitive, IMO, and with their TPU chips, it's probably economically competitive.
Today search results are a page full of ads. They can't get away with that unless they completely corner down the LLM market.
I forget the name but I see one LLM chatbox that looked like one of those old world news media websites plastered from top to bottom with jumpy obnoxious ads. It did offer some good models for free. I honestly might even consider using it if I could remember what it was called.
If they turned Gemini into that today I think people would just laugh, shake their head and close the app.
Openrouter isn't even 00.01% of tokens used on a daily basis. China alone uses 500 Trillion tokens a day and they're only 1/8th of the world. Much of the world has a google account, use your brain.
>Google is doing everything possible to juice their revenue right now, in ways they've never seen before
My experience confirms this: many years ago I had a website banned from serving Google Ads. They said my pages didn't have enough content.
Then a competitor started showing ads on the exact same type of pages where I had previously been able to serve them. So I tried getting my website approved again and -- surprise -- now I can serve ads again.
I think the reality is more a twist on 2): "AI is going to destroy their ad business and they want to goose their revenues while they can, hoping their AI business can compensate, but they know the AI business will never be as lucrative as the ad business."
Firstly, try to imagine stuffing as many ads as there are on a SERP into a chatbot conversation. Good way to lose users.
Secondly, Google spent decades hyper-optimizing their ad business monopoly for maximum profit, including going to extents that were recently found unlawful. People should really look into the findings of the last two Antitrust cases against Google, both of which it lost but for which it suffered only slaps on wrists. The details are eye-opening, including the bits about how Google, leveraging its prime position as the middle-man playing all sides against each other, manipulated ad auctions to make itself more money at the expense of its customers.
Even Microsoft was dinged for screwing over just their competitors, not their customers.
But now because of the paradigm shift in how people discover information, very little of the Google ad monoploy advantage transfers over. I think their AI business will keep growing, even if they don't have the best models, and it will be an exceptional source of revenue.
But it will still be no match to their ad business -- a cash cow of incomprehensible proportions -- and that is the fundamental problem they face today.
I don't think it's related to AI, at least not originally. To me this is about the slow death of display advertising. The arms race of ad blockers and ad blocker blockers and such has created a world where people hate display ads, especially because they require so much creepy data to even work in the first place.
AI probably would help their display ad business overall -- they finally just launched a "have Gemini create your ad!" feature, and I imagine having to hire a designer was a common blocker for SMBs.
My understanding was that the Ad business was starting to suffer because of how much bot activity there's been both in social media and indexed search (Google).
But I wouldn't write off ads yet. It seems plausible to me that LLMs will be able to slightly alter their output to casually mention paying brands and to profile users better than we could before.
everyone (on all sides!) is losing at AI and we'll see increasingly frantic attempts from all players to paper over the cracks as this extreme resource misallocation bites.
While this says nothing about Google as a company, I am sure AdSense is feeling the same pain as everybody else which livelihood depends on page views - just n orders of magnitudes bigger. So it would not surprise me if they were scraping the bottom of the barrel, so to speak, for any revenue opportunity.
It's still "only" 10-15% of Googles total revenue and less than 10% of it's profits. It's not nothing, but it's also not nearly as important as the dodgy ads business.
Why would Google be losing at AI, they're SOTA. They have all the users.
There's zero chance that AI destroys their ad business. I get served ads from Amazon in podcast RSS feeds that were directly derived from things I said in Gemini chats. It's definitely not destroying ads lol
I'm surprised by the weird levels of incompetence on this website, use your brain or have you already hawked that off to your favorite AI lab.
Make it easy for these companies by passing a law that any AI model a company offers to the public (non-government) has to be released as open weights.
This is how we deflate the bubble safely and completely, without having to introduce a government regulator that is likely to go either too far or not far enough.
This resolves the coordination problem among these potentially good actors.
What? This is a terrible policy. In addition to the obvious AI safety risks, this just amounts to a free R&D subsidy to Amazon for hosting models. It would remove any incentive to do any R&D without AWS level physical infrastructure.
Just like they're already doing with frontier-level open weight models? Do you imagine that Russian and Chinese intelligence don't have copies of the Mythos/Fable weights?
The point is to slow progress of future models. Nothing can stop what has already been created.
* International body which must approve new training runs
* Ban training on copyrighted material
* Hold AI model creators liable
The essential benefit of your idea is making new training runs unprofitable. But I imagine there are other ways to accomplish that, e.g. by taxing AI companies to the point where they are just barely squeaking by financially.
We're going to live in that world forever now, capable adversarial models have already been trained and will be used in a large conflict.
The pragmatic issue is hacking, and current open-weight LLMs are already capable of it. Bioweapons and RSI are pie-in-the-sky theories that distract from the real-world situation that each government wants to exploit.
But hacking has always been a vulnerability, even pre-AI. The scale multiplier here is the number of vulnerable machines, not the intelligence of AI.
> An ounce of prevention can be worth a pound of cure.
It can be, but if you get chemotherapy on a hunch instead of a diagnosis then it could kill you instead. There is no evidence that RSI will ever exist.
> The approach proposed by Amodei looks sensible...
It's hard to imagine any government regulation that didn't look sensible on paper and at first. But we know that government regulation has an extremely mixed record, with lots of good and bad outcomes.
This is a really important topic (and a lot of vested interests involved), so I sent an email to hn@ycombinator.com but if it takes too long for any action to be taken there's no point, right?
(I'll delete this reply if I can later, couldn't think of any other options)
Cryptocurrency, which turned into just another form of gambling, seems to have been the corrupting influence that allowed it to spread into people's homes. I hope we can reverse it somehow.
reply