I'm kinda surprised that the mixture of depths paper didn't come up here. It approaches the other direction of sometimes dropping layers:

https://arxiv.org/abs/2404.02258

If the author would like, I self computed a j lens for the 27b version of the qwen model. I used it for my own exploration in this area, and can share it if you want.

I'm not the author, but I would like! Is it feasible to run on the same hardware as the 27b model itself?

The using a J lens is super cheap compared to inference. You basically add a single matrix multiply per layer. You probably wouldn't even notice the overhead in a good implementation.

You should even be able to create a j lens from scratch, but it might take a while. I was able to do it in a few hours on an H100. Creating the J lens is basically the equivalent of calculating a few thousand training steps for a model (256,000 backprops in my case). I've got more details in a blog post:

https://blog.lwarfield.dev/layer-scope/

I'm currently at work and can't those matrixes up until I get home. I'll update this comment with a link later.

I've been experimenting with using J-Space (and final hidden layers) to extract the semantic meaning of words to improve TTS output accuracy using Qwen and Gemma models. My goal is to either map to an alternative token set where heteronyms are preserved or to output parenthesized IPA annotations for ambiguous words (with standard tokens). It's interesting to me that LLMs preserve this data throughout their processing but discard it in the final output.

I've also looked into extracting actions from J-Space to short-circuit a local assistant on low-end hardware. Are there any resources on how to do this training with inexpensive H100 instances (~$8/hr)? I would be opening up the final weights for Qwen/Gemma layers.

Awesome, thanks!

Fwiw you can just Google this for a model and often someone has done it

https://huggingface.co/eyes-ml/Qwen3.8-27B_jacobian-lens

Same for me. Every single time I tried it got flagged. I think this will be my litnus test for if the safeguards are good enough for benign requests.

Best museum in Seattle! It also feels a lot more "hands on", than most museums.

\s Take my angry upvote!

Personally I'm curious to the point of doing borderline LLM archecture research. This come from genuine curiosity, and not a want to use LLMs better.

I haven't gotten much out of for using LLMs though. It makes me understand the short comings of LLMs, and I feel like I got an early insight into how important context management is.

I currently have fable organize a bunch of 5.6 sol agents when working on my personal projects. This makes me wonder if I should add something along the lines of "For tasks that involve visual analysis, have gemini 3.7 look at images generated."

Overall I've been hooked on using agents from different companies for what they are best at (Thanks to Theo). Fable is expensive, but unmatched for planning and top level organization of other agents. Sol is fast, will persistantly go after goals (sometimes to its detriment), and does well with computer use.

I've always wondered why the industry relies on the giant monolithic system prompt. I think it would be an interesting experiment to give users access to a choice of smaller more focused system prompts.

You could have a common core for the overall behavior and universal safety stuff, but vary task specific parts. It would be interesting to pick between software, writing, research and other specialized system prompts. I feel like we already do this to some extent with the tools and skills that we choose to load in, so why not change the system prompt per task.

The reason is because they need one product to cover 100% of their customer base. Combinatorics get nasty real fast at these scales.

Also, why don't they bake in these limitations via reinforcement learning so they can keep the prompt context clear.

Because it is cheaper to have different behavior for internal use, special (e.g., government) customers, etc. if they are differentiated by things like system prompts and external guardrails than if they are different model versions with their own final training regime, and also its faster and cheaper to respond to problems even with a single public version by updating a system prompt than rertraining.

Presumably because these system prompt “limitations” are not given to government contracts, and training it into the model gets increasingly expensive the more versions you have to maintain.

Pi is excellent for this, its system prompt is tiny

Yes it is:

> Same model weights as Mythos 5, deployed with higher-coverage safeguards (see Section 4.5.2.2)

> 6.2 [Appendix redacted] > This appendix describes the criteria for our blocking bioclassifier exemption policy, and has been redacted from the public version of this report for security reasons.

>6.3 [Appendix redacted] > This appendix, redacted from the public version of this report, details the changes made to our constitution to expand classifier coverage to harmful uses in scope for the CB-2 threat model but not the CB-1 threat model, as described in Section 4.5.2.1.

interesting...

EDIT: After reading more I'd recommend looking at Transcript 2.20.A. Its a transcript of claude going over the redactions in the report. The section says its specifically for section 2, but the transcript also mentions other sections.