It doesn't reflect itself, we only see the UI of a complex process, not the real thing. We don't understand what happened in our brains any better for being able to feel conscious. We can only be conscious of what is cost effective and cost necessary to feel, in order to persist and survive. Animals for example and primitive humans could reproduce without understanding reproduction mechanisms, just the operational side.
It's even worse when everyone around you is using it. How can you keep up? Companies face the same dilemma: investors, competitors, and users already use AI and have factored it into their expectations.
AI is supposed to make people 100x more productive. We know it doesn't because nobody remade Windows in 6 months or Photoshop in 1 month. It's just memorized more common cases, that's all. You used to not be able to oneshot a three.js game, now you can, but that's only because it's memorized more three.js games, not because it's more intelligent.
Even if there's a lot of noise there's clearly something real there. People are shipping more working products than was previously possible, they're debugging faster than was previously possible, and various other things. I mean you can go fishing for things to confirm your skepticism if you want but it's pretty clear to me.
Sure, but that doesn't mean that you can't filter signal from noise.
So the actual problem statement is not "how do I keep up" but "how do I correctly tune my filter", which is solvable.
The biggest challenge there I think is that many people are not prepared for just how sharp and uncompromising that filter needs to be, but that too is solvable.
If you're not going to experiment at all you're not going to be able to do that. Agentic coding was basically a joke the first time I tried it. Now it isn't.
I think we have a pretty good explanation today - it's like embeddings from AI models. Experience is both content and reference, we represent new experience in relation to old experience. That makes representation personal, being made of one's own past experience. This does not explain away pure feeling, but explains how we make discriminations of similarity and difference between our experiences, the contents of qualia, the qualitative aspects.
We also know brains are locked inside a bone box only connected to the outside world by a bundle of unlabeled nerves, there is no direct access. So the brain can only compare patterns of signals it receives from outside. But since this representation-action-learning loop is recursive it cannot be inhabited or known from outside, 3p needs to pay the price of recursion to execute in order to get to 1p.
The gap is that between description and execution, which cannot be crossed for free with cheap description. Execution costs, and that cost is part of what is like being a bat. We can't inhabit their cost pressures since we don't have their context and body. You can't remove the costs of being a bat from "what it is like being a bat" and still get your answer from the comfort of the philosophical armchair.
You nailed it. Asking the question is asking to define from the outside what is an inner recursive process. The question is a simple confusion of domains. This is Humbert Maturana’s main point in Autopoiesis and Cognition (1980, now reissued). Recommend the whole book, as does Terry Winograd. The most intense part is the appendix specifically about the nervous system. Nagel and others knew no neuroscience and are clueless about recursion.
Isn't Maturana's theory that consciousness has to do with language, and the use of language to make distinctions about ourselves and others? To me, this seems clearly insufficient to explain consciousness - qualia totally precede language; one could experience qualia without language, etc.
> qualia totally precede language; one could experience qualia without language, etc.
While I do believe this as well, I don't think there is any way to prove this with current knowledge. You can introspect and separate your experience of a color from your language, but this type of introspection can also be misleading. And that's about all you can do - we don't know of any way to objectively test if another organism experiences qualia, and any historical/evolutionary evidence is also lost.
The relation is not qualia at the base and language on top, even if qualia is more primitive, because language directs action and action leads back to qualia, so they form a recursive loop which cannot be analyzed component by component anymore.
Qualia represents the compressed past experiences acting as a screen on which we represent new experiences, language is compressed past experiences from others and from past generations. Both work to reduce costs of cognition and action. (imho)
Sure. We can't prove that other organisms experience qualia; we can only look at the effects of qualia (e.g. behaviors that are likely to be the product of emotions) and assume that an organism is therefore conscious. The real point, though, is that suggesting language gives rise to consciousness lacks any explanatory power as to why language should be accompanied by consciousness.
Most proponents of the existence of qualia regard them as fundamental to what they call the hard problem of consciousness. People like Chalmers and Searle assume that all of the externally observable behaviors of people could be (at least in principle) explained without the need for qualia; that is, they believe it's possible for p-zombies to exist; this then gives rise to this question of consciousness - if all physical behaviors can be explained without qualia, and yet everyone of us has qualia, how and why does this happen?
I thought the point above was in a similar vein - that qualia and language are theoretically separable phenomena, so that we can imagine a being might have qualia without language, or language without qualia, and so these need to be explained separately. I was trying to point out that we have no proof for the existence of beings that possess these two qualities separately, so that I don't think the theoretical distinction is necessarily true. Just like any volume of gas has a temperature and a pressure, the existence of separate concepts doesn't mean they are physically separable.
You've completely left out the Hard Problem, though, and missed the essay's point.
A large part of the essay is that we have plenty of objective knowledge about how bat sonar works, but we don't know what the subjective experience of sonar is like, and more importantly, knowing about the physical representation, whether in neuronal patterns or embeddings, doesn't get you closer to the subjective experience.
tl;dr RGB(1.0, 0.0, 0.0) !== the subjective experience of red.
No - he's right - it's all relative. Our experience of a color is based on recall of things of that color.
Experimentally it's been shown that if a subject wears color goggle then initially everything will appear color tinted, but after a while normal color perception returns. The quale of "red" is not some absolute thing related to the wavelength (hence neural inputs) of red light.
> Our experience of a color is based on recall of things of that color.
You're getting memory mixed up with current experience. I think you mean to say that experience is based on the neural substrate associated with a color (mostly area V4 in human brains, which causes achromatopsia when damaged bilaterally).
(If memory is mandatory, then infants wouldn't see color when they first open their eyes, which seems unlikely. It also implies that cerebral achromatopsia would be impossible; but the damage that causes achromatopsia is in primary visual cortex, not memory areas.)
---
But again, this misses the point. The RGB triplet is a fact known about the color, but knowing 100% of the facts of the representation does NOT tell you what it is like to experience red.
Consider a perfect future neuroscience lab that uses nanomachines to safely record every neuronal firing, every dendritic voltage shift, every synaptic cleft's neurotransmitter levels, all at microsecond precision, while showing you red circles. This lab knows everything about what your brain does in response to red stimuli. Everything EXCEPT the subjective experience of redness in the participants.
An infant's perception of the world, in the days/weeks/months following birth is not going to be like our own until they have indeed started to form memories/associations of things outside of the womb (where the main color they will be exposed to is red - light filtered through the mother's stomach). Even after birth it's going to be a while (depending on parents/environment) until they have much exposure to certain classes of things, maybe even certain colors (e.g. an indoor baby may not be seeing a lot of green).
RGB triplet has little to do with color vision since we don't have sensors for individual wavelengths - our color cones (most people have 3 types, but some have 4, allowing them to distinguish a lot of spectra that a normal person can't) all have broad gaussian responses and so all respond to all wavelengths, just with different spectral sensitivities.
An infant who first opens their eyes, or is exposed to new colors for the first time, is not going to have the same experience of color as later on, but necessarily there will still be some experience of "color" (e.g. a varying surface attribute, differing by the different neural inputs coming from the retina), but the subconscious associations will of course be different - something is not going to be perceived as "grass green" or "sky blue" until you have experienced those.
Of course you can never know the subjective experience of another person, let alone another animal, since while there will be a lot in common, dictated by brain architecture, it's also going to depend on individual experience. Our senses work by prediction, which is based on personal experience. If you look at a mid-game chess board you are not going to see the same thing as a grandmaster since they will be seeing positions and you will just be seeing pieces.
The real point is that the subjective experience of a color like red is not some absolute thing tied to the neural inputs for "red" (i.e. varying strengths of signal firing from your 3/4 wavelength sensors), since the experience is the same even when those inputs change - color constancy, goggle experiment, etc etc.
You are still missing the point, and talking about everything BUT the disconnect between objectivity and subjectivity. I don't know how many times I have to mention the Hard Problem in the comments before people address it in their responses.
Yes, subjective experience is unique, based in neural architecture, color-blindness, experience, etc, etc etc. All of that is irrelevant to the essay.
> The real point is that the subjective experience of a color like red is not some absolute thing tied to the neural inputs for "red", since the experience is the same even when those inputs change
Not the real point of the essay at all. Please, just go read up on the Hard Problem.
That's not really what Nagel is talking about - the paper is about the difficulty, if not impossibility, of using reductionism to explain some things such as subjective (conscious) experience.
Note that at the beginning of the paper Nagel says "Conscious experience is a widespread phenomenon. It occurs at many levels of animal life ...". His starting point is a willingness to accept that higher order animals are indeed conscious, and that by extension it is indeed like something to be them.
If you want to discuss the hard problem, then you are talking about the wrong paper, and should be reading Chalmers (or Kirk's earlier "Zombies v. Materialists") not Nagel. However, Nagel is of course right, and the p-zombie is a non-sensical construct. If you have a sufficiently advanced cognitive apparatus then of course you can reflect on your own mental life - of course it "feels like something".
The ad hoc heuristics are the domain knowledge baked into the model by human experts, like features, architecture and loss function.
"Evaluation" means environments or datasets, the model is supposed to discover its representations from scaled up experience. That was the bitter lesson - more data and compute beat heuristics.
> If anyone knows of any work towards more open-ended fitness functions, I'd love to read it.
There is research in open-ended learning, see "Why Greatness Cannot Be Planned" by Kenneth O. Stanley. The core idea is that in open-ended scenarios you don't know what action was good except in hindsight because your path is deceptive. So the idea is to replace fitness with novelty search which provides more stepping stones towards the goal.
> There are more elements to discovery though. It is still not clear where the initial working model/hypothesis comes from or how the updates are selected
That is a problem in RL, so we usually do supervised training first, teach it to imitate some trajectories, then do RL to refine the model. RL alone has a huge problem because it might be hard to reach a reward, hence hard to learn the task by pure reinforcement. Humans also combine supervision (learn from books) with search (solving problems) to break the discovery problem. For example, a human with no initial instruction in math would not produce great results no matter how smart they are. The bootstrap was exploration paid for in the past.
SFT + RL connection to model/hypothesis search is insightful. Brute force / scalable search is where Sutton's Bitter Lesson also points to. Once your search domain is small compared to your search budget, that makes a lot of sense.
If I get your meaning right, SFT creates the right inductive bias so that the RL search + reward guidance does the trick.
For novel discovery, the question might then be whether the inductive bias builds a strong enough prison so no new discovery is possible by RL or if the search can escape the boundaries set by SFT given enough randomization and the right reward function.
I know that RL is usually not performed at inference time, but in-context learning mechanisms might be developed by RL to discover at test time. Edit: I would love to hear if that actually happens or not, like new induction heads (https://transformer-circuits.pub/2022/in-context-learning-an...) forming during RL. I really have no idea.
the role of evolution is always a confounding factor as well and all the various analogies to how it maps onto AI research are always not quite satisfactory.
Before they add AI they better fix the frigging search function in settings, it is horrible, you need to know their exact words, and Apple has a funny naming sense. Hierarchies nested so deep you never find anything. I come to use Claude or ChatGPT to tell me the right incantations to find a setting.
I don't even know that it's knowing the exact words, it just seems like sometimes the search decides it's just gonna give you no results because the vibes were off.
I'm almost sure that sometimes searching the same thing will give you the result and sometimes it won't.
> We are going to get near instant software from prompt, multiple ones and then choose the best one.
If you extract the spec from first implementation and reimplement from scratch you get a free testing oracle. Where they diverge you send the agent to decide which one had a bug.
> All my finance and payment domain expertise, all the debugging intuition and distributed system knowledge earned through hours of sweat and tears, is now promptable.
In ML it was even worse, we had to throw away a decade of experience, made irrelevant by the new approaches. Even the most revered activity - designing new architectures - became too expensive to do in real life. Fine-tuning models is what we do now, prompting and evals. Like 90% of what we used to learn is no longer needed. And yes, LLMs can do most new ML activities too, they just need light supervision. I am sometimes ashamed to admit I have stopped coding 12 months ago and never wrote one more line, that after 35 years of coding manually. But I also think we will never be without LLMs again, so no point in preparing for 2016 in 2026
reply