I don't know how much this API churn is going to help developers who are trying to integrate OAI into real, actual, non-wrapper products. Every vendor-managed state machine that handles conversation, messages, prompt hand-off, etc., has ultimately proven inadequate, presumptive or distracting for my use cases.
At the end of the day, all I ever seem to use is the chat completion API with structured outputs turned on. Despite my "basic" usage, I am employing tool use, recursive conversations, RAG, etc. I don't see the value in outsourcing state management of my "agent" to a 3rd party. I have way more autonomy if I keep things like this local.
The entire premise of these products is that you are feeding a string literal into some black box and it gives you a new string. Hopefully, as JSON or whatever you requested. If you focus just on the idea of composing the appropriate string each time, everything else melts away. This is the only grain that really matters. Think about other ways in which we compose highly-structured strings based upon business state stored in a database. It's literally the exact same thing you do when you SSR a webpage with PHP. The only real difference is how it is served.
I haven't really found any agent framework that gives me anything I need above a simple structured gen call.
As you say, most requests to LLMs are (should be?) prompt-in structure-out, in line with the Unix philosophy of doing precisely one thing well.
Agent frameworks are simply too early. They are layers built to abstract a set of design patterns that are not common. We should only build abstractions when it is obvious that everyone is reinventing the wheel.
In the case of agents, there is no wheel to invent. It's all simple language model calls.
I commonly use the phrase "the language model should be the most boring part of your code". You should be spending most of your time building the actual software and tooling -- LLMs are a small component of your software. Agent frameworks often make the language model too large a character in your codebase, at least for my tastes.
I mirror this sentiment. Even their "function calling" abstraction still hallucinates parameters and schema, and the JSON schema itself is clearly way too verbose and breaks down completely if you feed it anything more complex than 5 very simple function calls. This just seems to build upon their already broken black box abstractions and isn't useful for any real world applications, but it's helpful for getting small proof-of-concept apps going, I guess...
Exactly. You would have to be naive to build a company on top of this kind of API. LLMs are going to be become commodities, and this is OpenAI fighting against that fate as their valuation and continued investment requirements doesn't make any sense otherwise.
If you built on the Assistant API, maybe take the hint and don't just rewrite to the Responses API? Own your product, black box the LLM-of-the-day.
> OpenAI fighting against that fate as their valuation and continued investment requirements doesn't make any sense otherwise.
Is it actually the case that OpenAI couldn't be viable if all they offered was a simple chat completion API on top of the web experience?
It seems to me the devil is all in how the margin plays out. I'd focus on driving down costs and pushing boundaries on foundation models. If you are always a half step ahead, highly reliable and reasonably cheap, your competitors will have a tough time. Valuations can be justified if businesses begin to trust the roadmap and stability of the services.
I'll tell you what's not working right now is the insane model naming scheme and rapid fire vision changes. This kind of stuff is spooking the technology leaders of large prospective customers. Only the most permanently online people can keep things straight. Everyone was super excited and on board with AI in 2024 because who wants to be left out. I think that energy is still justified in many ways, but we've also got to find a way to meet more of the customer base where they are currently at. Wrappers and agentic SDKs are not what these people are looking for. Many F500s already have a gigantic development team who can deal with deep, nasty API integrations and related state contraptions. They're looking for assurances/evidence that OAI's business & product line will remain stable for the next 5+ years before going all-in.
The point of the bear thesis on OpenAI is that training frontier models is extraordinarily expensive. They can’t produce cutting edge models, charge a cheap price, and make a profit all at once
It feels like I'm becoming way too old for all the new computer stuff. I spent 2 decades trying to use every language available to write reliable programs for everyone, and now the whole world is jumping in this black hole / black box controlled by a few big companies where the output is random and definitely not up to my own standards.
It's very sad because we were supposed to do better than those who came before us, but instead we're throwing everything in the trash for a so-called productivity that I don't think even exists out of the influencers' brains.
> You would have to be naive to build a company on top of this kind of API.
You have to be purposefully naive to be a cutting-edge tech entrepreneur in the first place. If you fully acknowledged every risk and roadblock ahead, you’d probably never start.
But that deliberate naiveté is exactly what’s required to launch a startup in VUCA-space. Outsized success comes from exploiting emerging complexities: betting despite ambiguity, adapting quickly on top of uncertainty, and turning volatility into advantage.
This bit feels like we are being pushed away from the existing API for non-technical reasons?
> When using Chat Completions, the model always retrieves information from the web before responding to your query. To use web_search_preview as a tool that models like gpt-4o and gpt-4o-mini invoke only when necessary, switch to using the Responses API.
Porting over to the new Responses API is non-trivial, and we already have history, RAG and other things an assistant needs already.
I can’t find that text in the announcement. In fact it sounds like you have to use a specific model with the chat completions endpoint to get web searches.
Couldn’t have said it better. I’ve developed multiple agents with just function calling, structured outputs and they have been in production for more than a year (back in the day we did not call it agent lol)
I think these is targeted towards people who are already using agent frameworks + OpenAI API.
“ These new tools streamline core agent logic, orchestration, and interactions, making it significantly easier for developers to get started with building agents”
Sounds exactly like “the cloud”, especially AWS. Basically “get married to our platform, build on top of it, and make it hard to leave.” The benefits are that it’s easy to get started. And also that they invested in the infrastructure, but now they are trying to lock you in by storing as much state and data as possible with them withoit an easy way to migrate. So, increase your switching costs. For social networks the benefit was that they had the network effect but that doesn’t apply here.
All of AWS' big money makers are the meat-and-potatoes services around compute, storage, databases etc. where you could drop their offering and replace it with another in a straightforward way. It will cost you to migrate in terms of time and direct spend (those egress fees...), but it's possible. Companies ultimately stay put because the products work and the price is reasonable, but if they tried to 10X the price overnight everyone would eventually bolt.
Yeah they keep pushing higher-level services, but the uptake of these is extremely limited. If you used something like SageMaker, which has an extremely high lock-in factor, it's probably because you're an old school company that don't know what you're doing and AWS held your developer's hand to get the Hello World-level app working, but at least you got your name printed in their case study materials of the project at the end.
I think OpenAI looks at AWS and thinks they can do better. And for their investors, they must do better. But in the end I think the commoditization of LLMs is already almost complete, and this is just a futile attempt to fight it.
outsourcing state to openai & co is great for them as vendor lock-in. the real money in AI will be business- and user-interfacing tools built on top of the vendors and it would be a terrible business decision to not abstract away from the model provider in the background and keep all private data under your domain, also from a data protection / legal point of view
i can understand them trying to prevent their business from becoming a commodity but i don't see that working out for them except for some short term buzz, but others will run with their ideas in domain specific applications
Speaking of string literals, I hate that the state of the art nowadays is to force you to format model inputs as a conversation with separate messages. Ever since OpenAI did that and discontinued the regular completions API it became nearly useless for me since I don't use LLMs for conversation. And because OpenAI is the Apple of LLMs, everyone else is copying that worthless chat messages abstraction and not providing normal completions as they should.
Don't get me wrong, chat completions are nice to have for certain use cases, but that being the only option makes me practically unable to use the model.
At the end of the day, all I ever seem to use is the chat completion API with structured outputs turned on. Despite my "basic" usage, I am employing tool use, recursive conversations, RAG, etc. I don't see the value in outsourcing state management of my "agent" to a 3rd party. I have way more autonomy if I keep things like this local.
The entire premise of these products is that you are feeding a string literal into some black box and it gives you a new string. Hopefully, as JSON or whatever you requested. If you focus just on the idea of composing the appropriate string each time, everything else melts away. This is the only grain that really matters. Think about other ways in which we compose highly-structured strings based upon business state stored in a database. It's literally the exact same thing you do when you SSR a webpage with PHP. The only real difference is how it is served.