The result of experiments conducted by Rikiya Takehi et al. from MIT CSAIL 11 2 in their blog post "Sparks of Weight-Space Programming" 8: enabling a frontier large model (referred to as Astra in the text) to directly read and write the weights of Qwen3.5-0.8B within a Python environment, without running gradient descent or training. Using only 50 forward inference passes to examine GSM8K / MATH practice problems, they identified a direction that caused the model to repeatedly loop during reasoning. They subtracted this direction from 64 columns each in the down_proj of layers 15 and 19, modifying a total of 131,072 weights (accounting for 0.016%). This repository contains the modified model, created on 11 2, and has not been indexed in the last 7 periods.
Upvotes 64 · Likes 64 · Downloads 310
Rationale
- The authors state that the original version would think until the token limit (hitting 93.0%) for almost every math problem, whereas the modified version reduces this to 33.1%. Under the same sampling settings, MATH-500 scores rose from 7.9% to 48.1%, GSM8K scores rose from 13.8% to 67.1%, and IFEval, which was not specifically targeted, also rose from 21.0% to 62.3%.
- Verifiable: The modifications are publicly available as a bit-for-bit identical diff on GitHub (exp8_math in RikiyaT/lm_understand-logs). Compared to the original, only two tensors differ, and the model card lists the sha256 hashes for both. akhaliq used ML Intern on H200 to rerun the full MATH-500 benchmark (500 problems × 4 samples) following the original protocol, achieving 51.0% with 31.6% hitting the limit; per-problem results are located in the repository's eval/ directory.
- The significance lies in the method: The same blog post also demonstrated the large model identifying which game a network was trained to play by looking only at its weights, and writing a network from scratch capable of playing small games. This repository is the public checkpoint of the real language model experiment described in the text, allowing for direct download and comparison.
Usage
- vLLM: LLM(model="RikiyaT/Qwen3.5-0.8B-weight-space-programming", dtype="bfloat16", max_model_len=34816). When applying apply_chat_template, set enable_thinking=True. Follow the sampling parameters listed in the model card: temperature 0.6, top_p 0.95, top_k 20, presence_penalty 1.5, max_tokens 32768.
- For math problems, append the phrase "Please reason step by step, and put your final answer within \boxed{}" after the question, and take the last \boxed{} as the answer.
- If you do not wish to deploy, you can first compare the original and modified versions on the same problems in the Space ML-Intern-lab/weight-space-programming-demo (the akhaliq/weight-space-programming-demo listed in the model card redirects here).
Risks
- It is a 0.8B small model; even after fixing MATH-500, it only achieves around 50%, with about one-third of responses still hitting the thinking limit. It is not suitable as a practical math assistant; it is primarily a research demonstration.
- The scores were run by the authors and akhaliq themselves, using long thinking (up to 32,768 tokens) and specific sampling. The low scores of the original version were mainly due to infinite loops failing to produce answers; results will vary with shorter limits or different sampling parameters. During the rerun, the original version only underwent a smoke test on 25 problems.
- The frontier model used to make the modifications is not public. What others can reproduce is this modified checkpoint and the evaluation, not the process of "having a large model modify weights" itself. The repository is the full multimodal version, while the original evaluation used a text-only exported version (the language part tensor hashes are consistent).