/AmagibabaI write about Mechanistic Interpretability and Statistics. 2026-02-07T21:11:32-08:00 Amagibaba / Jekyll © 2026 Amagibaba /assets/img/favicons/favicon.ico /assets/img/favicons/favicon-96x96.png Complexity Series (2 / 3) - Carry-Over Circuits2026-02-05T21:00:00-08:00 2026-02-05T21:00:00-08:00 /posts/carry-over-circuits/ Amagibaba This is the second installment of the “Complexity Series,” where I endeavor to argue that there are certain classes of problems (simple arithmetic being one of them) that Transformer architectures will never be able to solve. In my previous post, I outlined that we have thus far discovered a few algorithms that are being implicitly implemented by LLMs to perform arithmetic (regular or modulo):... Complexity Series (1 / 3) - LLM Arithmetic ⭐2025-11-29T21:00:00-08:00 2025-12-19T09:58:20-08:00 /posts/llm-arithmetic/ Amagibaba This is the first installment of the “Complexity Series”, where I endeavor to argue that there are certain classes of problems (simple arithmetic being one of them) that Transformer architectures will never be able to solve. To get started, I want to do a deep dive on known algorithms that Transformers have used to perform simple arithmetic. There are thus far a few algorithms that researchers... Multi-Layer Latent Space Visualization2025-10-01T22:00:00-07:00 2026-01-07T22:44:31-08:00 /posts/multi-layer-latent-spaces/ Amagibaba In MLPs, What Input Gives What Output? If you had a series of MLP (nn.Linear) layers chained together, you may like to answer the question: if I wanted the final layer outputs’ first element (call $h_1$) to be activated (i.e. $> 0$), what would my input have to be? To set the stage, one may ask: who cares? Well, wouldn’t it be great if you could answer: I want a model’s response to my one-... ICA vs SAEs ⭐2025-05-12T22:00:00-07:00 2025-08-24T23:33:11-07:00 /posts/ica/ Amagibaba Thinking about how Sparse Auto-encoders (SAEs) aim to learn a sparse over-complete basis (where you are trying to triangulate a larger number of sources than you have signals; e.g. you only have 8 microphones in the room, but there are 20 speakers) got me thinking about Independent Component Analysis again. In particular, I wanted to see if I could articulate a mapping between ICA and SAEs. Thi... Thoughts on Hidden Structure in MLP Space2025-04-27T22:00:00-07:00 2025-06-15T23:09:17-07:00 /posts/tegum-factors/ Amagibaba After deep-diving into why SAEs succeed at retrieving superposed features, what their limitations are, and closely inspecting the hidden technical implementations of the sae_lens library, I just wanted to write some quick notes of what I think the MLP space looks like, with respect to ‘true’ features. Anticorrelated (“Mutually Sparse”) Features Prefer to be in the Same Tegum Factor This hypot...