Weirdly complex to read yet light on key technical details. My TLDR (as an old clueless electronics engineer) was the compute part is photonic/analog, lasers and waveguides, yet we still require 50 billion transistors performing the (I guess non-compute) parts such as ADC, I/O, memory etc. The bottom line is 65 TOPS for <80W - The processing (optical) part consuming 1.65W and the 'helper electronics' consuming the rest so scaling the (optical) processing should not have the thermal bottlenecks of a solely transistor based processor. Also parallelism of the optical part though using different wavelengths of light as threads may be possible.
Nothing about problems, costs, or can the helper electronics eventually use photonics.
I remember a TV Program in the UK from the 70's (tomorrows world I think) that talked about this so I am guessing silicon was just more cost effective until now. Still taking it at face value I would say it is quite an exciting technology.
You are also forgetting the ridiculous amount of software bloat literally introduced at the OS level. https://www.youtube.com/watch?v=tCNmhpcjJCQ Look at how fast windows 7 works compared to windows 11. Think of how much telemetry, bloatware windows 10 and 11 ship with. A new methodology that teases a different direction from silicon chips is definitely a must step but software vendors need to come together as well and highlight the bloat problem. Perhaps for the next windows release, in addition to Home, Professional and Pro, how about Windows Maximal, a highly streamlined version of windows with only the absolute minimal set of things installed to get your system running at 2x speeds compared to Windows Home. Charge it double maybe?
A few years ago I ran Windows XP in a VM inside Windows 10.
When you pressed Win+E, Windows opens an explorer window (in both versions).
In XP this happens in the span of a single video frame.
In Windows 10, first nothing happens, then a big white rectangle, then you get to watch all the UI elements get painted in one by one.
The really impressive part is that this was before they rewrote explorer as an electron app! I think it might actually be faster now that it's an electron app.
It's always cited as an example, and it's highly impressive yes, but it's also highly focused, with a very partial feature set, I'm not sure that type of coding is applicable to mass scale software.
Interesting. Questions, the Nature paper being expensively paywalled:
- Is the analog computation actually done with light? What's the actual compute element like? Do they have an analog photonic multiplier? Those exist, and have been scaling up for a while.[1] The announcement isn't clear on how much compute is photonic. There are still a lot of digital components involved. Is it worth it to go D/A, generate light, do some photonic operations, go A/D, and put the bits back into memory? That's been the classic problem with photonic computing. Memory is really hard, and without memory, pretty soon you have to go back to a domain where you can store results. Pure photonic systems do exist, such as fiber optic cable amplifiers, but they are memoryless.
- If all this works, is loss of repeatability going to be a problem?
Maybe not the specific photonic system that they are describing. Which I'm sure has some significant improvements over what existed then. But the idea of using analog approximations of existing neural net AI models, to allow us to run AI models far more cheaply, with far less energy.
Whether or not this system is the one that wins out, I'm very sure that AI run on an analog system will have a very important role to play in the future. It will allow technologies like guiding autonomous robots with AI models running on hardware inside of the robot.
In 25 years we'll have #GlassModels. A "chip", which is a passive device (just a complex lens) made only of glass or graphene, which can do an "AI Inference" simply by shining the "input tokens" thru it. (i.e. arrays of photons). In other words, the "numeric value" at one MLP "neuron input" will be the amplitude of the light (number of simultaneous photons).
All addition, multiplication, and tanh functions will be done by photon superposition/interference effects, and it will consume zero power (since it's only a complex "lens").
It will probably do parallel computations where each photon frequency range will not interfere with other ranges, allowing multiple "inferences" to be "Shining Thru" simultaneously.
This design will completely solve the energy crisis and each inference will take the same time as it takes light to travel a centimeter. i.e. essentially instantaneous.
For years I've been fascinated by those little solar-powered calculators. In a weird way, they're devices that enable us to cast hand shadows to do arithmetic.
Lookup "Analog Optical Computing". There was recently a breakthrough just last week where optical computing researchers were able to use photon interference effects to do mathematical operations purely in analog! That means no 0s and 1s, just pure optics. Paste all that into Gemini to learn more.
This is always a hilarious take. If you inflation adjust a 386 PC from the early 90s when 486's were on the market you'd find they range in excess of $3000 and the 486s are in the $5000 zone. Computers are incredibly cheap now. What isn't cheap is the bleeding edge. A place fewer and fewer people have to be at, which leads to lower demand and higher prices to compensate.
It is crazy you can buy a used laptop for $15 and do something meaningful with like writing code (meaningful as in make money)
I used to have this weird obsession of doing this, buying old chromebooks putting linux on them, with 4GB of RAM it was still useful but I realize nowadays for "ideal" computing it seems 16GB is a min for RAM
It was kind or that way in early days of high end personal computing. I remember seeing an ad in the early 90s for a 486 laptop that was $6,000. Historically prices have always gone down. You just have to wait. SoTA is always going to go for a premium.
> That irked me too. "_Bleeding edge" consumer GPUs are ...", sure, but you wait 6 months and you have it at a fraction of the cost.
That's demonstrably false. The RTX 4090 released in 2022 with an MSRP of $1,600. Today you'd be hard pressed to find one below $3K that isn't a scam.
The reality is that NVIDIA is taking advantage of their market dominance to increase their markup with every generation of products[1], even when accounting for inflation and price-to-performance. The 50 series is even more egregious, since it delivers a marginal performance increase, yet the marketing relies heavily on frame generation. The trickling supply and scalpers are doing the rest.
AMD and Intel have a more reasonable pricing strategy, but they don't compete at the higher end.
> Critically, this processor achieves accuracies approaching those of conventional 32-bit floating-point digital systems “out-of-the-box,” without relying on advanced methods such as fine-tuning or quantization-aware training.
It's an analog system. Which means that accuracy is naturally limited.
However a single analog math operation requires the same energy as a single bit flip in a digital computer. And it takes a lot of bit flips to do a single floating point operation. So a digital calculation can be approximated with far less energy and hardware. And neural nets don't need digital precision to produce useful results.
It seems weirdly backwards. They don’t do techniques like quantization aware tuning to increase the accuracy of the coprocessor, right? (I mean that’s nonsense). They use those techniques, to allow them to use less accurate coprocessors, I thought.
I think they are just saying the coprocessor is pretty accurate, so they don’t need to use these advanced techniques.
Weirdly complex to read yet light on key technical details. My TLDR (as an old clueless electronics engineer) was the compute part is photonic/analog, lasers and waveguides, yet we still require 50 billion transistors performing the (I guess non-compute) parts such as ADC, I/O, memory etc. The bottom line is 65 TOPS for <80W - The processing (optical) part consuming 1.65W and the 'helper electronics' consuming the rest so scaling the (optical) processing should not have the thermal bottlenecks of a solely transistor based processor. Also parallelism of the optical part though using different wavelengths of light as threads may be possible. Nothing about problems, costs, or can the helper electronics eventually use photonics.
I remember a TV Program in the UK from the 70's (tomorrows world I think) that talked about this so I am guessing silicon was just more cost effective until now. Still taking it at face value I would say it is quite an exciting technology.
Insightful Note. thank you.
You are also forgetting the ridiculous amount of software bloat literally introduced at the OS level. https://www.youtube.com/watch?v=tCNmhpcjJCQ Look at how fast windows 7 works compared to windows 11. Think of how much telemetry, bloatware windows 10 and 11 ship with. A new methodology that teases a different direction from silicon chips is definitely a must step but software vendors need to come together as well and highlight the bloat problem. Perhaps for the next windows release, in addition to Home, Professional and Pro, how about Windows Maximal, a highly streamlined version of windows with only the absolute minimal set of things installed to get your system running at 2x speeds compared to Windows Home. Charge it double maybe?
A few years ago I ran Windows XP in a VM inside Windows 10.
When you pressed Win+E, Windows opens an explorer window (in both versions).
In XP this happens in the span of a single video frame.
In Windows 10, first nothing happens, then a big white rectangle, then you get to watch all the UI elements get painted in one by one.
The really impressive part is that this was before they rewrote explorer as an electron app! I think it might actually be faster now that it's an electron app.
Just look at the demoscene for some great example of what it means to fully exploit what the hardware is actually capable of.
It's always cited as an example, and it's highly impressive yes, but it's also highly focused, with a very partial feature set, I'm not sure that type of coding is applicable to mass scale software.
Interesting. Questions, the Nature paper being expensively paywalled:
- Is the analog computation actually done with light? What's the actual compute element like? Do they have an analog photonic multiplier? Those exist, and have been scaling up for a while.[1] The announcement isn't clear on how much compute is photonic. There are still a lot of digital components involved. Is it worth it to go D/A, generate light, do some photonic operations, go A/D, and put the bits back into memory? That's been the classic problem with photonic computing. Memory is really hard, and without memory, pretty soon you have to go back to a domain where you can store results. Pure photonic systems do exist, such as fiber optic cable amplifiers, but they are memoryless.
- If all this works, is loss of repeatability going to be a problem?
[1] https://ieeexplore.ieee.org/document/10484797
This paradigm for computing was already covered three years ago by Veratasium in https://www.youtube.com/watch?v=GVsUOuSjvcg.
Maybe not the specific photonic system that they are describing. Which I'm sure has some significant improvements over what existed then. But the idea of using analog approximations of existing neural net AI models, to allow us to run AI models far more cheaply, with far less energy.
Whether or not this system is the one that wins out, I'm very sure that AI run on an analog system will have a very important role to play in the future. It will allow technologies like guiding autonomous robots with AI models running on hardware inside of the robot.
In 25 years we'll have #GlassModels. A "chip", which is a passive device (just a complex lens) made only of glass or graphene, which can do an "AI Inference" simply by shining the "input tokens" thru it. (i.e. arrays of photons). In other words, the "numeric value" at one MLP "neuron input" will be the amplitude of the light (number of simultaneous photons).
All addition, multiplication, and tanh functions will be done by photon superposition/interference effects, and it will consume zero power (since it's only a complex "lens").
It will probably do parallel computations where each photon frequency range will not interfere with other ranges, allowing multiple "inferences" to be "Shining Thru" simultaneously.
This design will completely solve the energy crisis and each inference will take the same time as it takes light to travel a centimeter. i.e. essentially instantaneous.
If you contextualize Star Trek's "isolinear chips" to be something like this, they start to seem considerably more sensible.
Has anyone built a physical ASIC that embeds a full model yet?
For years I've been fascinated by those little solar-powered calculators. In a weird way, they're devices that enable us to cast hand shadows to do arithmetic.
Lookup "Analog Optical Computing". There was recently a breakthrough just last week where optical computing researchers were able to use photon interference effects to do mathematical operations purely in analog! That means no 0s and 1s, just pure optics. Paste all that into Gemini to learn more.
I stopped reading after "Soon, you will not be able to afford your computer. Consumer GPUs are already prohibitively expensive."
This is always a hilarious take. If you inflation adjust a 386 PC from the early 90s when 486's were on the market you'd find they range in excess of $3000 and the 486s are in the $5000 zone. Computers are incredibly cheap now. What isn't cheap is the bleeding edge. A place fewer and fewer people have to be at, which leads to lower demand and higher prices to compensate.
It is crazy you can buy a used laptop for $15 and do something meaningful with like writing code (meaningful as in make money)
I used to have this weird obsession of doing this, buying old chromebooks putting linux on them, with 4GB of RAM it was still useful but I realize nowadays for "ideal" computing it seems 16GB is a min for RAM
It's like the black Mac from 2007, I know its tech is outdated but I want it
Definitely one of my favourite machines of all time (until the plastic started falling off).
That's related more to NVidia's discovery that they could get away with huge margins, and the China GPU projects for graphics being years behind.[1]
[1] https://www.msn.com/en-in/money/news/china-s-first-gaming-gp...
It was kind or that way in early days of high end personal computing. I remember seeing an ad in the early 90s for a 486 laptop that was $6,000. Historically prices have always gone down. You just have to wait. SoTA is always going to go for a premium.
That irked me too. "_Bleeding edge" consumer GPUs are ...", sure, but you wait 6 months and you have it at a fraction of the cost.
It's like saying "cars are already prohibitively expensive" whilst looking a Ferraris.
> That irked me too. "_Bleeding edge" consumer GPUs are ...", sure, but you wait 6 months and you have it at a fraction of the cost.
That's demonstrably false. The RTX 4090 released in 2022 with an MSRP of $1,600. Today you'd be hard pressed to find one below $3K that isn't a scam.
The reality is that NVIDIA is taking advantage of their market dominance to increase their markup with every generation of products[1], even when accounting for inflation and price-to-performance. The 50 series is even more egregious, since it delivers a marginal performance increase, yet the marketing relies heavily on frame generation. The trickling supply and scalpers are doing the rest.
AMD and Intel have a more reasonable pricing strategy, but they don't compete at the higher end.
[1]: https://www.digitaltrends.com/computing/nvidias-pricing-stra...
> Critically, this processor achieves accuracies approaching those of conventional 32-bit floating-point digital systems “out-of-the-box,” without relying on advanced methods such as fine-tuning or quantization-aware training.
Hmm... what? So it is not accurate?
It's an analog system. Which means that accuracy is naturally limited.
However a single analog math operation requires the same energy as a single bit flip in a digital computer. And it takes a lot of bit flips to do a single floating point operation. So a digital calculation can be approximated with far less energy and hardware. And neural nets don't need digital precision to produce useful results.
> neural nets don't need digital precision to produce useful results.
The point - as shown by the original implementation...
It seems weirdly backwards. They don’t do techniques like quantization aware tuning to increase the accuracy of the coprocessor, right? (I mean that’s nonsense). They use those techniques, to allow them to use less accurate coprocessors, I thought.
I think they are just saying the coprocessor is pretty accurate, so they don’t need to use these advanced techniques.