> Changing coordinates. Given a vector x expressed in the standard basis, we change coordinates by applying
x_eig = V^−1 x.
> This operation does not change the vector itself; it only changes how the vector is represented, converting it into eigenvector coordinates.
I'm sure the author knows about the difference between the vector and a coordinate vector given a basis. I just don't like this wording. It's weird to say applying a function doesn't affect a change.
> It's weird to say applying a function doesn't affect a change.
It does affect a change--from one representation into another representation. The underlying object it is representing stays the same, however, which I think is reasonable to emphasize.
That’s active vs passive transformation and AFAIK mathematicians often think in passive ones but physicists would use both.
And the one you quoted is passive.
Another book which I would highly recommend for learning the math (or at least getting a very good idea of how it works) behind machine learning/transformers is "Why Machines Learn: The Elegant Math Behind Modern AI" by Anil Ananthaswamy. I read it this past year and I found it was extremely good at pacing and building up intuition to the "how and why" of most important math concepts and discoveries in ML that have lead to where we are today. For those who don't already have a fairly deep understanding of linear algebra, statistics, and multivariate calculus, this book will be a far more gentle introduction yet still has enough rigor for you to come away with something.
Is it working out well for tech pdf's and books? Once I saw the form factor I realized it would be better than an iPad (less heavy, better battery) and that is the use case I am looking at. Might wait till v2.
From what I understand the iPad mini weighs the same with a slightly larger screen. Has their matte display option.
I have big pockets so for me it’s probably pocketable. But I still would have a phone. So I’ve got a duo on order with a year lease so I’m not stuck if I don’t think it’s great.
I have to confess to considering a mini with cellular instead of a phone altogether though.
My full stack JS jockey buddy studied this in the nascent days of LLMs to prep for an interview and is now generationally wealthy. Sliding doors moment for sure...
I'd note that the Fleuret books gives the foundation for basic deep learning whereas the OP seems to give the necessary foundations for Transformers and related machinery. These later addition what is now "AI" are in total at least as complicated as the original deep learning concepts. Not that just learning basic neural networks but the two book don't appear to cover the same ground.
Only had a quick skim, but looks like the content of this book has very little to do with transformers. Might be more relevant for diffusion models (image/video generation) though.
It's a nice treatment of the subject, but I wish he would post a conventionally-formatted .PDF that puts every page on a full sheet of standard printer paper.
Oh, that's unfortunate! I made it from the A5 version on the site which I downloaded right before making my post. Looks like the author hasn't updated that one.
- Understanding deep learning by Prince[1]. Has both theoretical and programming exercises.
- Deep learning - Foundations and concepts by Bishop[2].
- Deep learning architectures - mathematical approach - Calin[3].
- Mathematical theory of deep learning by Petersen et al[4]. Only handful of exercises each chapter.
- Dive into deep learning[5]. This one is more practical programming hands-on. The exercises is a mix between programming exercises and theoretical.
- EECS 182/282A | Deep Neural Networks 2025 (not a book but a course by Berkley)[6]. I only skimmed the assignments, but they seem very interesting. A mix of theoretical and practical.
I think Bishop is probably the best one for theoretical exercises, balancing between not too difficult but not too easy either. However, if you also want programming exercises, Prince has lots of those. Berkley also seems very interesting, modern and encompassing.
Is there any one who have recommended books on the attention architecture specifically? I have been interested in that for a long time but never gotten around to actually studying it.
I'm v curious how you're incorporating this into your workflow. Is it that when you want to implement something you can call out to these books as rigorous reference material for your agents? I'd never thought of doing such a thing and am curious how you're doing it and how it's worked.
VL-JEPA: Joint Embedding Predictive Architecture for Vision-language [2512.10942v1]
A Multi-Memory Segment System for Generating [2508.15294v3]
Adversarial Cues in Decision Models Used [2610.11436v1]
AGENTIC BACKDOORS VIA PRETRAINING POISONING [2610.37741v1]
Agentic Memory Learning Unified Long-Term [2601.01885v2]
Agentic Semi-Structured Memory [2604.14362v1]
Agents That Know Too Much [2606.26627v1]
AI Agents Enable Adaptive [2606.03811v1]
AI Agents That Matter [2407.01502v1]
AI Assistants Cloud Versus Local LLM [2604.18566v2]
AMA-Bench Evaluating Long-Horizon Memory [2602.22769v2]
A-Mem Agentic Memory for LLM Agents [2502.12110v11]
APEX-MEM Agentic Semi-Structured Memory [2604.14362v1]
ASK ONLY WHEN NEEDED [2604.20572v2]
Attention Is All You Need [1706.03762v7]
Attention Is Not What You Need [2512.19428v1]
Benchmarking Multi-Agent LLM [2603.22651v1]
Beyond Nodes vs. Edges [2604.14685v1]
Beyond Static Summarization [2601.04463v1]
Calibrated Decision Models for Autonomous [2609.28940v1]
Character-and-Scene Episodic Memory for Agents [2602.06051v3]
CLASP Training-Free LLM-Assisted Source Code Watermarking [2510.11251v2]
Cognitive Architectures for Language Agents [2309.02427v3]
COMPUTATIONAL COPYRIGHT [2312.06646v5]
Constitutional AI [2212.08073v1]
Creative Problem Solving in Artificially Intelligent Agents [2204.10358v1]
CyberGym [2506.02548v3]
Deep image search [2602.10809v1]
Developing Surrogates [2602.21588v1]
Diffusion-Based Neural Network Weights Generation [2402.18153v2]
Dynamic LLM-Powered Agent Network [2310.02170v2]
Dynamic Logit-Level Gating [2606.04378v1]
EternalMath [2601.01400v1]
EverMemOS [2601.02163v2]
EvoMemBench Benchmarking Agent Memory [2605.18421v1]
ExploitBench [2605.14153v1]
FADEMEM BIOLOGICALLY-INSPIRED [2601.18642v2]
Finance-Grounded Optimization [2509.04541v2]
Formal Algorithms for Transformers [2207.09238v1]
Geogalactica [2401.00434v2]
HAGE - Harnessing Agentic Memory [2605.09942v1]
HaluMem [2511.03506v1]
Harnessing the Universal Geometry of Embeddings [2505.12540v4]
HIPPOCAMPUS [2602.13594v1]
hipporag [2405.14831v3]
Hybrid Reasoning Agents [2610.18861v1]
Hyper agents [2603.19461v1]
HyperMem [2604.08256v2]
Interactive Simulacra of Human Behavior [2304.03442v2]
Is Complex Agentic Memory Always Needed [2610.30755v1]
Language Modeling Is Compression [2309.10668v2]
Learn from your own latents [2605.27734v1]
Learning on the Job [2607.22157v1]
Learning to Memorize at Test Time [2501.00663v1]
Learning to Rewrite Tool Descriptions [2602.20426v2]
LLM Agents Already Know When to Call Tools [2605.09252v2]
LLM-Enabled Frequency-Aware [2602.19522v1]
LONGMEMEVAL [2410.10813v2]
MAGMA A Multi-Graph based [2601.03236v2]
MemGround [2604.14158v1]
Memory in the Age of AI Agents [2512.13564v2]
Memory Tax [2610.34073v1]
Memp - Exploring Agent Procedural Memory [2508.06433v4]
Mitigating Provenance-Role [2605.25869v1]
MOSS Self-Evolution through Source-Level Rewriting [2605.22794v2]
Multi-task DAG Learning [2306.02831v1]
Natural Language conductor [2512.04388v5]
NeuroGen - Neural Network Parameter Generation [2505.12470v2]
Next-Latent Prediction Transformers Learn Compact World Models [2511.05963v4]
Novel Memory Forgetting Techniques [2604.02280v1]
OMNI-LEAK Orchestrator Multi-Agen [2602.13477v2]
On Understanding Extremely Long-Term Video with Adaptive [2411.12593v3]
Opal Private Memory [2604.02522v1]
Parameter Prediction for Unseen Deep Architectures [2110.13100v1]
Prompt Optimization [2602.18449v1]
PromptMark [2606.20835v1]
RAG-MCP Mitigating Prompt Bloat [2505.03275v1]
RAIN-MERGING [2602.22538v1]
Recursive Self-Improvement through Multi-Agent [2610.12176v1]
Reflection-Based Memory [2506.02158v1]
Refusal in Language Models Is Mediated by a Single Direction [2406.11717v3]
REMEM REASONING WITH EPISODIC MEMORY [2602.13530v3]
Remember When It Matters [2607.08716v1]
Remembering More, Risking More [2605.17830v1]
Sakana Fugu Technical Report [2606.21228v1]
SalesRLAgent [2503.23303v1]
Scaling Long-Form Story Generation [2609.35759v1]
Search-R1- Training LLM [2503.09516v5]
Secure Semantic Communications [2602.22134v2]
Selective Memory for Artificial Intelligence [2603.15994v1]
Skill.state [2608.26263v2]
Smarter Together [2511.08301v1]
SmartSearch [2603.15599v1]
SOLVING A MILLION-STEP LLM TASK [2511.09030v1]
Storage Is Not Memory [2605.04897v1]
StructMem Structured Memory [2604.21748v1]
Structured Belief State and the First Precision [2605.11325v2]
SuperLocalMemory V3.3 [2604.04514v1]
SWE-MeM Learning Adaptive Memory [2606.28434v1]
Synthius-Mem Brain-Inspired Hallucination [2604.11563v1]
The AI Scientist-v2 [2504.08066v1]
The Bitter Lesson of Tool Calling [2608.06370v1]
The Customer Is the Builder [2610.30307v1]
The Gradient of Generative AI Release [2302.04844v1]
The Little Book of Generative AI Foundations [2605.29713v2]
The Malicious Use [1802.07228v2]
The Memory Wars [2508.05867v1]
Thinking Fast and Slow in AI [2110.01834v1]
Time algo [2610.00577v1]
Towards trustworthy agentic AI [2605.23989v1]
Transformers learn in-context by gradient descent [2212.07677v2]
TReMu Towards Neuro-Symbolic Temporal Reasoning [2502.01630v2]
TRINITY AN EVOLVED LLM COORDINATOR [2512.04695v3]
VERBALIZED SAMPLING [2510.01171v3]
Verifiable Agent-to-Agent [2602.02395v1]
Verified Multi-Agent Orchestration [2603.11445v2]
WEBCOACH - A MODEL [2511.12997v1]
XGrammar-2 Dynamic and Efficient Structured Generation [2601.04426v4]
Your Agent’s Memories Are Not Its Own [2607.05029v1]
Zep A Temporal Knowledge Graph Architecture [2501.13956v1]
The very first formula in the text, explicitly described as "as simple as possible," names its variable ê1 with no explanation given. I think this paper needs another pass.
What do you mean "no explanation given"? It says: "In 2 dimensions, the standard (or identity) basis vectors are ê1 = [ 1 0 ] and ê2 = [0 1]." Seems pretty clear to me he's defining ê1 and ê2, or rather telling us that he's using this very common way of naming the basis vectors.
It's just giving a vector a name. It also names ê₂. These particular names are very standard and familiar to most people who've met vectors before.
You can look at those vectors and see that the position of the 1 is the subscript, without explanation. The ^ could do with an explanation that it means unit vector, but if you've met vectors before you likely know that (maybe that's elsewhere in the book, but a quick search didn't find it).
That's absolutely standard notation for a unit basis vector. I haven't read the prerequisites carefully, so if it addressed to the general public maybe?
It's fine to be forgetful of maths that you're not actively using. (I am speaking as a mathematician.) It's also easier than ever to get a quick reminder or introduction, e.g. for this Google something like [what does e1 with ^ on the e mean], or [standard basis vector] from earlier in the sentence, or ask another AI.
I was never actually taught ê as unit vector notation. I did game development for years and am quite familiar with 3D math but a lot of formal math notation is very foreign to me.
I got annoyed at yet another primer that does not put information theory as foundational, so I decided to write my own.
The Little Pamphlet of AI Foundations: An Unintuitive Philosophical Primer
Losses
Intelligence is the ability to explain phenomena. One is more intelligent where they can better explain phenomena.
A phenomenon is any correspondence between data, and an explanation is an encoding of that correspondence. There is an infinite amount of data so an intelligence must be a program that generates new explanations when it sees new data.
A program is a string of bits run by an interpreter. Since strings of bits are data, programs may be generated by other programs, and a program's interpreter may itself be another program or intelligence. This recursion eventually bottoms out in physical ware. Human and artificial intelligences differ mainly in their ware, though human intelligences also have an impredicative recursion property that artificial intelligences are only expected to gain in 2027.
At a low level of intelligence, a calculator takes in numbers and operations through electrical impulses, moves electrons through a series of transistors, which creates new outputs to be displayed on the screen.
At a higher level, a neural network training algorithm takes in examples and outputs weights, usually through preconditioned gradient descent.
At an even higher level, a language model takes in words and outputs possible continuations. To convert this to the correct continuation, an additional error correction code must be provided for every word it outputs. This unexplained gap in the language model, or 'loss', dominates in the limit of a large number of words. To consistently provide short explanations, a model must have a small loss.
At the highest level, an explanation of an environment must encode all data in that environment, including any intelligences. An intelligence encompasses a larger share of its environment if it is harder to encode. Under a fixed loss budget, the hardest intelligence to encode provides explanations weighted exponentially by their length. This is the origin of the common mantras that a short explanation is exponentially more likely to be the correct one, colloquially known as Occam's razor, Solomonoff's lightsaber, and Kolmogorov's complexity.
The important takeaway is that a loss represents the average size of an error correction code, and a smaller loss is better.
Architecture
An intelligence is a program that generates encodings of data correspondences. Choosing an architecture with better data topologies enables it to do so more efficiently.
A data topology consists of data and possible connections between the data, with computational costs determined by the architecture. As computational resources grow superexponentially, the goal of architecting is generally to lower the computational complexity of creating connections, encapsulated colloquially as 'The Bitter Lesson'. To give a few examples:
1. The limited liability corporation lowered the personal risk of starting a business from the size of the business to a fixed cost, making economic connections cheaper to form.
2. Residual networks directly form connections between neuron data and output gradients, bypassing the exponentially weakening connections that previously made deep learning impossible.
3. Flow transport models improve upon continuous normalizing flows by giving target distribution data along the entire generation process.
4. Quadratic attention connects every input datum to every input datum, instead of relying on fringe connections from the nonlinearity in activation functions. Linear attention further lowers this cost by adaptively selecting data to go into a memory bank, and only forming connections with the memory bank.
5. A gradient descent preconditioner modifies the weight flow density, so distributions of weights are squeezed closer together or spread further apart. In the extreme, this no longer looks like gradient descent and looks closer to an intelligence directly writing weight updates based on new examples.
Learning Theory
A perfect intelligence provides explanations weighted exponentially by their length. A Bayesian neural network is a network generated by a perfect intelligence as an explanation for a dataset, and its distribution of weights—its 'partition function'—will be concentrated in regions with exponentially low loss. Analogous to physics, much can be gleaned from the partition function.
In adversarial contexts such as generative adversarial networks or reinforcement learning, examples rotate through a cyclic group of six elements, being positive, neutral, or negative examples depending on what the other intelligences have learned. This creates an anyonic partition function for each learnable feature and the familiar sextessential steep upwards loss curve before a plateau. Similarly, an alternating cycle of learning and forgetting gives the fermionic partition function and a loss curve colloquially referred to as grokking. Most learning is just learning and looks bosonic in nature: losses quickly go down, then settle into an exponential curve.
In the limit of a large number of training examples, the network's loss dominates the size of the explanation, but it still takes a logarithmically increasing number of bits to describe the path the weights take along the weight flow. To be precise, the explanation length grows as
nL(w) + λ log n − (m − 1) log log n
where n is the number of training examples, L is the network's loss, λ the real log canonical threshold, and m its multiplicity. It is possible to subsume the logarithmic terms into the loss by adding normal noise to the weights after each training iteration with variance dηT/λn, where η is the learning rate, T the temperature, and d the model dimension. In the nonsingular case λ = d/2.
From the Dust jacket:
The apocalypse will be televised!
You know what’s worse than breaking up with your girlfriend? Being stuck with her prize-winning show cat. And you know what’s worse than that? An alien invasion, the destruction of all man-made structures on Earth, and the systematic exploitation of all the survivors for a sadistic intergalactic game show. That’s what.
READ MORE
Join Coast Guard vet Carl and his ex-girlfriend’s cat, Princess Donut, as they try to survive the end of the world—or just get to the next level—in a video game–like, trap-filled fantasy dungeon. A dungeon that’s actually the set of a reality television show with countless viewers across the galaxy. Exploding goblins. Magical potions. Deadly, drug-dealing llamas. This ain’t your ordinary game show.
Welcome, Crawler. Welcome to the Dungeon. Survival is optional. Keeping the viewers entertained is not.
I don't like:
> Changing coordinates. Given a vector x expressed in the standard basis, we change coordinates by applying
> This operation does not change the vector itself; it only changes how the vector is represented, converting it into eigenvector coordinates.I'm sure the author knows about the difference between the vector and a coordinate vector given a basis. I just don't like this wording. It's weird to say applying a function doesn't affect a change.
> It's weird to say applying a function doesn't affect a change.
It does affect a change--from one representation into another representation. The underlying object it is representing stays the same, however, which I think is reasonable to emphasize.
> effect*
effect*
That’s active vs passive transformation and AFAIK mathematicians often think in passive ones but physicists would use both. And the one you quoted is passive.
Another book which I would highly recommend for learning the math (or at least getting a very good idea of how it works) behind machine learning/transformers is "Why Machines Learn: The Elegant Math Behind Modern AI" by Anil Ananthaswamy. I read it this past year and I found it was extremely good at pacing and building up intuition to the "how and why" of most important math concepts and discoveries in ML that have lead to where we are today. For those who don't already have a fairly deep understanding of linear algebra, statistics, and multivariate calculus, this book will be a far more gentle introduction yet still has enough rigor for you to come away with something.
Thanks for your advice
Wow, this book is surprisingly readable! I like how he starts from basics of linear algebra.
My favorite little book is François Fleuret's: The Little Book of Deep Learning https://fleuret.org/francois/lbdl.html
I love how that site offers a phone-formatted PDF as an option. How thoughtful!
You may enjoy the reflow feature on Koreader. It fits a pdfs content into a screenwidth single column layout. That text is also zoomable.
Does it work on scanned (non-“digital”) PDFs as well? That has been my biggest pain point so far. Haven’t tried Koreader yet.
i got duo to increase my reading.
Is it working out well for tech pdf's and books? Once I saw the form factor I realized it would be better than an iPad (less heavy, better battery) and that is the use case I am looking at. Might wait till v2.
From what I understand the iPad mini weighs the same with a slightly larger screen. Has their matte display option.
I have big pockets so for me it’s probably pocketable. But I still would have a phone. So I’ve got a duo on order with a year lease so I’m not stuck if I don’t think it’s great.
I have to confess to considering a mini with cellular instead of a phone altogether though.
My old crappy eyes need big screens for this. I have one on order. I’m ambivilating over the screen crack issue.
My full stack JS jockey buddy studied this in the nascent days of LLMs to prep for an interview and is now generationally wealthy. Sliding doors moment for sure...
I'd note that the Fleuret books gives the foundation for basic deep learning whereas the OP seems to give the necessary foundations for Transformers and related machinery. These later addition what is now "AI" are in total at least as complicated as the original deep learning concepts. Not that just learning basic neural networks but the two book don't appear to cover the same ground.
Only had a quick skim, but looks like the content of this book has very little to do with transformers. Might be more relevant for diffusion models (image/video generation) though.
Gee, I think you're right. But it's definitely a leap beyond what was deep learning a while back.
It's a nice treatment of the subject, but I wish he would post a conventionally-formatted .PDF that puts every page on a full sheet of standard printer paper.
I immediately wanted the same thing, so I made it. The book is licensed Creative Commons BY-NC-SA 4.0 International License, so here you go.
https://cdn.filepost.dev/file/filepost/uploads/15/1578f21219...
Thanks.
But fyi for others this is one version behind and missing these updates:
Oh, that's unfortunate! I made it from the A5 version on the site which I downloaded right before making my post. Looks like the author hasn't updated that one.
I just emailed him. Fingers crossed.
Thanks!
Are there any deep learning books with some problems to solve ?
Some that comes to mind:
- Understanding deep learning by Prince[1]. Has both theoretical and programming exercises.
- Deep learning - Foundations and concepts by Bishop[2].
- Deep learning architectures - mathematical approach - Calin[3].
- Mathematical theory of deep learning by Petersen et al[4]. Only handful of exercises each chapter.
- Dive into deep learning[5]. This one is more practical programming hands-on. The exercises is a mix between programming exercises and theoretical.
- EECS 182/282A | Deep Neural Networks 2025 (not a book but a course by Berkley)[6]. I only skimmed the assignments, but they seem very interesting. A mix of theoretical and practical.
I think Bishop is probably the best one for theoretical exercises, balancing between not too difficult but not too easy either. However, if you also want programming exercises, Prince has lots of those. Berkley also seems very interesting, modern and encompassing.
[1] https://udlbook.github.io/udlbook/
[2] https://www.bishopbook.com/
[3] https://link.springer.com/book/10.1007/978-3-030-36721-3
[4] https://arxiv.org/abs/2407.18384
[5] https://d2l.ai/index.html
[6] https://berkeley-cs182.github.io/fa25/
https://www.deep-ml.com
+1. I only learn from doing exercises.
Interesting!
Is there any one who have recommended books on the attention architecture specifically? I have been interested in that for a long time but never gotten around to actually studying it.
Not a book, but 3blue1brown's video on attention is pretty great: https://www.youtube.com/watch?v=eMlx5fFNoYc
formal algorithms for transformers provides perhaps the most succinct introduction to attention I can imagine https://arxiv.org/pdf/2207.09238
Appreciated, I drop all of these into a RAG folder for when I need them later on in projects.
8Gb of whitepapers/books...
I'm v curious how you're incorporating this into your workflow. Is it that when you want to implement something you can call out to these books as rigorous reference material for your agents? I'd never thought of doing such a thing and am curious how you're doing it and how it's worked.
You can source it here in the library tool, it connects to a cloud/local LLM for RAG:
https://vektormemory.com/
Extracted whitepapers, should keep you busy:
VL-JEPA: Joint Embedding Predictive Architecture for Vision-language [2512.10942v1] A Multi-Memory Segment System for Generating [2508.15294v3] Adversarial Cues in Decision Models Used [2610.11436v1] AGENTIC BACKDOORS VIA PRETRAINING POISONING [2610.37741v1] Agentic Memory Learning Unified Long-Term [2601.01885v2] Agentic Semi-Structured Memory [2604.14362v1] Agents That Know Too Much [2606.26627v1] AI Agents Enable Adaptive [2606.03811v1] AI Agents That Matter [2407.01502v1] AI Assistants Cloud Versus Local LLM [2604.18566v2] AMA-Bench Evaluating Long-Horizon Memory [2602.22769v2] A-Mem Agentic Memory for LLM Agents [2502.12110v11] APEX-MEM Agentic Semi-Structured Memory [2604.14362v1] ASK ONLY WHEN NEEDED [2604.20572v2] Attention Is All You Need [1706.03762v7] Attention Is Not What You Need [2512.19428v1] Benchmarking Multi-Agent LLM [2603.22651v1] Beyond Nodes vs. Edges [2604.14685v1] Beyond Static Summarization [2601.04463v1] Calibrated Decision Models for Autonomous [2609.28940v1] Character-and-Scene Episodic Memory for Agents [2602.06051v3] CLASP Training-Free LLM-Assisted Source Code Watermarking [2510.11251v2] Cognitive Architectures for Language Agents [2309.02427v3] COMPUTATIONAL COPYRIGHT [2312.06646v5] Constitutional AI [2212.08073v1] Creative Problem Solving in Artificially Intelligent Agents [2204.10358v1] CyberGym [2506.02548v3] Deep image search [2602.10809v1] Developing Surrogates [2602.21588v1] Diffusion-Based Neural Network Weights Generation [2402.18153v2] Dynamic LLM-Powered Agent Network [2310.02170v2] Dynamic Logit-Level Gating [2606.04378v1] EternalMath [2601.01400v1] EverMemOS [2601.02163v2] EvoMemBench Benchmarking Agent Memory [2605.18421v1] ExploitBench [2605.14153v1] FADEMEM BIOLOGICALLY-INSPIRED [2601.18642v2] Finance-Grounded Optimization [2509.04541v2] Formal Algorithms for Transformers [2207.09238v1] Geogalactica [2401.00434v2] HAGE - Harnessing Agentic Memory [2605.09942v1] HaluMem [2511.03506v1] Harnessing the Universal Geometry of Embeddings [2505.12540v4] HIPPOCAMPUS [2602.13594v1] hipporag [2405.14831v3] Hybrid Reasoning Agents [2610.18861v1] Hyper agents [2603.19461v1] HyperMem [2604.08256v2] Interactive Simulacra of Human Behavior [2304.03442v2] Is Complex Agentic Memory Always Needed [2610.30755v1] Language Modeling Is Compression [2309.10668v2] Learn from your own latents [2605.27734v1] Learning on the Job [2607.22157v1] Learning to Memorize at Test Time [2501.00663v1] Learning to Rewrite Tool Descriptions [2602.20426v2] LLM Agents Already Know When to Call Tools [2605.09252v2] LLM-Enabled Frequency-Aware [2602.19522v1] LONGMEMEVAL [2410.10813v2] MAGMA A Multi-Graph based [2601.03236v2] MemGround [2604.14158v1] Memory in the Age of AI Agents [2512.13564v2] Memory Tax [2610.34073v1] Memp - Exploring Agent Procedural Memory [2508.06433v4] Mitigating Provenance-Role [2605.25869v1] MOSS Self-Evolution through Source-Level Rewriting [2605.22794v2] Multi-task DAG Learning [2306.02831v1] Natural Language conductor [2512.04388v5] NeuroGen - Neural Network Parameter Generation [2505.12470v2] Next-Latent Prediction Transformers Learn Compact World Models [2511.05963v4] Novel Memory Forgetting Techniques [2604.02280v1] OMNI-LEAK Orchestrator Multi-Agen [2602.13477v2] On Understanding Extremely Long-Term Video with Adaptive [2411.12593v3] Opal Private Memory [2604.02522v1] Parameter Prediction for Unseen Deep Architectures [2110.13100v1] Prompt Optimization [2602.18449v1] PromptMark [2606.20835v1] RAG-MCP Mitigating Prompt Bloat [2505.03275v1] RAIN-MERGING [2602.22538v1] Recursive Self-Improvement through Multi-Agent [2610.12176v1] Reflection-Based Memory [2506.02158v1] Refusal in Language Models Is Mediated by a Single Direction [2406.11717v3] REMEM REASONING WITH EPISODIC MEMORY [2602.13530v3] Remember When It Matters [2607.08716v1] Remembering More, Risking More [2605.17830v1] Sakana Fugu Technical Report [2606.21228v1] SalesRLAgent [2503.23303v1] Scaling Long-Form Story Generation [2609.35759v1] Search-R1- Training LLM [2503.09516v5] Secure Semantic Communications [2602.22134v2] Selective Memory for Artificial Intelligence [2603.15994v1] Skill.state [2608.26263v2] Smarter Together [2511.08301v1] SmartSearch [2603.15599v1] SOLVING A MILLION-STEP LLM TASK [2511.09030v1] Storage Is Not Memory [2605.04897v1] StructMem Structured Memory [2604.21748v1] Structured Belief State and the First Precision [2605.11325v2] SuperLocalMemory V3.3 [2604.04514v1] SWE-MeM Learning Adaptive Memory [2606.28434v1] Synthius-Mem Brain-Inspired Hallucination [2604.11563v1] The AI Scientist-v2 [2504.08066v1] The Bitter Lesson of Tool Calling [2608.06370v1] The Customer Is the Builder [2610.30307v1] The Gradient of Generative AI Release [2302.04844v1] The Little Book of Generative AI Foundations [2605.29713v2] The Malicious Use [1802.07228v2] The Memory Wars [2508.05867v1] Thinking Fast and Slow in AI [2110.01834v1] Time algo [2610.00577v1] Towards trustworthy agentic AI [2605.23989v1] Transformers learn in-context by gradient descent [2212.07677v2] TReMu Towards Neuro-Symbolic Temporal Reasoning [2502.01630v2] TRINITY AN EVOLVED LLM COORDINATOR [2512.04695v3] VERBALIZED SAMPLING [2510.01171v3] Verifiable Agent-to-Agent [2602.02395v1] Verified Multi-Agent Orchestration [2603.11445v2] WEBCOACH - A MODEL [2511.12997v1] XGrammar-2 Dynamic and Efficient Structured Generation [2601.04426v4] Your Agent’s Memories Are Not Its Own [2607.05029v1] Zep A Temporal Knowledge Graph Architecture [2501.13956v1]
Can you share what you have so far in that folder?
They spell it out quite clearly: "8Gb of whitepapers/books..."
We do a lil copyright infringement
Holy..
Care to share the list of titles in a gist?
Someone should add it to the hacker news curated paper list! Along side the video and book list.
The very first formula in the text, explicitly described as "as simple as possible," names its variable ê1 with no explanation given. I think this paper needs another pass.
What do you mean "no explanation given"? It says: "In 2 dimensions, the standard (or identity) basis vectors are ê1 = [ 1 0 ] and ê2 = [0 1]." Seems pretty clear to me he's defining ê1 and ê2, or rather telling us that he's using this very common way of naming the basis vectors.
It's just giving a vector a name. It also names ê₂. These particular names are very standard and familiar to most people who've met vectors before.
You can look at those vectors and see that the position of the 1 is the subscript, without explanation. The ^ could do with an explanation that it means unit vector, but if you've met vectors before you likely know that (maybe that's elsewhere in the book, but a quick search didn't find it).
That's absolutely standard notation for a unit basis vector. I haven't read the prerequisites carefully, so if it addressed to the general public maybe?
It is perhaps addressed to a different crowd?
It’s not okay to be forgetful of maths anymore. Whats a dot product, whats a unit vector, how does matrix multiplication work..
It's fine to be forgetful of maths that you're not actively using. (I am speaking as a mathematician.) It's also easier than ever to get a quick reminder or introduction, e.g. for this Google something like [what does e1 with ^ on the e mean], or [standard basis vector] from earlier in the sentence, or ask another AI.
I was never actually taught ê as unit vector notation. I did game development for years and am quite familiar with 3D math but a lot of formal math notation is very foreign to me.
Ah, "intuition": one man's cheese is another man's rotten milk.
I got annoyed at yet another primer that does not put information theory as foundational, so I decided to write my own.
The Little Pamphlet of AI Foundations: An Unintuitive Philosophical Primer
Losses
Intelligence is the ability to explain phenomena. One is more intelligent where they can better explain phenomena.
A phenomenon is any correspondence between data, and an explanation is an encoding of that correspondence. There is an infinite amount of data so an intelligence must be a program that generates new explanations when it sees new data.
A program is a string of bits run by an interpreter. Since strings of bits are data, programs may be generated by other programs, and a program's interpreter may itself be another program or intelligence. This recursion eventually bottoms out in physical ware. Human and artificial intelligences differ mainly in their ware, though human intelligences also have an impredicative recursion property that artificial intelligences are only expected to gain in 2027.
At a low level of intelligence, a calculator takes in numbers and operations through electrical impulses, moves electrons through a series of transistors, which creates new outputs to be displayed on the screen.
At a higher level, a neural network training algorithm takes in examples and outputs weights, usually through preconditioned gradient descent.
At an even higher level, a language model takes in words and outputs possible continuations. To convert this to the correct continuation, an additional error correction code must be provided for every word it outputs. This unexplained gap in the language model, or 'loss', dominates in the limit of a large number of words. To consistently provide short explanations, a model must have a small loss.
At the highest level, an explanation of an environment must encode all data in that environment, including any intelligences. An intelligence encompasses a larger share of its environment if it is harder to encode. Under a fixed loss budget, the hardest intelligence to encode provides explanations weighted exponentially by their length. This is the origin of the common mantras that a short explanation is exponentially more likely to be the correct one, colloquially known as Occam's razor, Solomonoff's lightsaber, and Kolmogorov's complexity.
The important takeaway is that a loss represents the average size of an error correction code, and a smaller loss is better.
Architecture
An intelligence is a program that generates encodings of data correspondences. Choosing an architecture with better data topologies enables it to do so more efficiently.
A data topology consists of data and possible connections between the data, with computational costs determined by the architecture. As computational resources grow superexponentially, the goal of architecting is generally to lower the computational complexity of creating connections, encapsulated colloquially as 'The Bitter Lesson'. To give a few examples:
1. The limited liability corporation lowered the personal risk of starting a business from the size of the business to a fixed cost, making economic connections cheaper to form.
2. Residual networks directly form connections between neuron data and output gradients, bypassing the exponentially weakening connections that previously made deep learning impossible.
3. Flow transport models improve upon continuous normalizing flows by giving target distribution data along the entire generation process.
4. Quadratic attention connects every input datum to every input datum, instead of relying on fringe connections from the nonlinearity in activation functions. Linear attention further lowers this cost by adaptively selecting data to go into a memory bank, and only forming connections with the memory bank.
5. A gradient descent preconditioner modifies the weight flow density, so distributions of weights are squeezed closer together or spread further apart. In the extreme, this no longer looks like gradient descent and looks closer to an intelligence directly writing weight updates based on new examples.
Learning Theory
A perfect intelligence provides explanations weighted exponentially by their length. A Bayesian neural network is a network generated by a perfect intelligence as an explanation for a dataset, and its distribution of weights—its 'partition function'—will be concentrated in regions with exponentially low loss. Analogous to physics, much can be gleaned from the partition function.
In adversarial contexts such as generative adversarial networks or reinforcement learning, examples rotate through a cyclic group of six elements, being positive, neutral, or negative examples depending on what the other intelligences have learned. This creates an anyonic partition function for each learnable feature and the familiar sextessential steep upwards loss curve before a plateau. Similarly, an alternating cycle of learning and forgetting gives the fermionic partition function and a loss curve colloquially referred to as grokking. Most learning is just learning and looks bosonic in nature: losses quickly go down, then settle into an exponential curve.
In the limit of a large number of training examples, the network's loss dominates the size of the explanation, but it still takes a logarithmically increasing number of bits to describe the path the weights take along the weight flow. To be precise, the explanation length grows as
where n is the number of training examples, L is the network's loss, λ the real log canonical threshold, and m its multiplicity. It is possible to subsume the logarithmic terms into the loss by adding normal noise to the weights after each training iteration with variance dηT/λn, where η is the learning rate, T the temperature, and d the model dimension. In the nonsingular case λ = d/2.We are recommending books for HackerBooks.org I see. Good, this is why I have come here also.
Let me tell you about one of the greatest book series of our generation: Dungeon Crawler Carl, By Matt Dinniman
https://mattdinniman.com/books/
From the Dust jacket: The apocalypse will be televised!
You know what’s worse than breaking up with your girlfriend? Being stuck with her prize-winning show cat. And you know what’s worse than that? An alien invasion, the destruction of all man-made structures on Earth, and the systematic exploitation of all the survivors for a sadistic intergalactic game show. That’s what. READ MORE
Join Coast Guard vet Carl and his ex-girlfriend’s cat, Princess Donut, as they try to survive the end of the world—or just get to the next level—in a video game–like, trap-filled fantasy dungeon. A dungeon that’s actually the set of a reality television show with countless viewers across the galaxy. Exploding goblins. Magical potions. Deadly, drug-dealing llamas. This ain’t your ordinary game show.
Welcome, Crawler. Welcome to the Dungeon. Survival is optional. Keeping the viewers entertained is not.
Published sept 21, 2020 RePublished Aug 27, 2024
The street marketing around this book is frankly baffling, do you suppose this is relevant?
I guess not. My bad.