Q Labs' Dust replaces the backward pass with noisy forward experiments. Then an open-source speedrun made the whole idea considerably more interesting.
For roughly forty years, neural networks have operated under a rather strict arrangement.
The network makes a prediction.
The prediction is wrong.
Backpropagation calculates how much each part of the network contributed to the mistake.
An optimiser adjusts the weights.
Repeat this process a few trillion times, spend an extraordinary amount of money on GPUs, and eventually you get something capable of writing Python, analysing medical research and confidently inventing a nonexistent airport.
An impressive system.
But what if we simply removed the backward pass?
Not improved it.
Not made it cheaper.
Removed it.
That sounds like a particularly effective way to stop a neural network from learning anything.
Yet Q Labs has published a method called Dust that trains transformer language models without running backpropagation.
And then launched an open-source project called Noise to see how much further people could push the idea.
I've been looking through the research, and the interesting part isn't that someone has found another way to train a small transformer.
It's how they did it.
Because the method challenges a fairly fundamental assumption about what the training process needs to look like.
First, a distinction: Dust is the method. Noise is the experiment.
There are two projects worth separating.
Dust: Pretraining Transformers Without Backpropagation is the research work by Samip Dahal, Bishwas Mandal, Serdar Gülbahar and Akshay Vegesna.
It describes a zeroth-order approach to training transformers: instead of obtaining exact gradients through backpropagation, the system estimates useful update directions by perturbing the network and observing what happens.
Noise is the open-source speedrun built around that idea.
The challenge is straightforward.
Train a transformer without using a backward pass.
The learning signal must come from forward evaluations.
Then compete on either the quality of the resulting model or how quickly a specified loss can be reached.
It's a rather appealing research format.
Instead of arguing endlessly about which training method ought to be better, give everybody a model, a dataset, a scoring procedure and some GPU time.
Then see what actually works.
But before we get to the results, we need to understand what Dust does differently.
Rather than calculating the gradient, annoy the network and observe its reaction
Imagine you're trying to improve a machine but you're not allowed to inspect the internal mechanism.
You can still experiment.
Turn a control slightly to the left.
Observe the output.
Turn it slightly to the right.
Observe again.
If one change improves the result, you've learned something useful about the direction to move.
This is broadly the idea behind zeroth-order optimisation.
You estimate how to improve a system using measurements of its outputs rather than analytical derivatives.
The obvious problem is scale.
A neural network contains millions, billions or even hundreds of billions of parameters.
If you start randomly changing individual weights and measuring what happens, you can spend a remarkable amount of computation discovering approximately nothing.
This isn't a new problem.
Evolution strategies and other perturbation-based approaches have explored it for years.
Dust's interesting trick is to change where the noise is introduced.
Not directly into the model's weights.
Into its activations.
More specifically, the method adds Gaussian noise to the outputs of linear layers, independently for different tokens.
Then it evaluates what those perturbations do to the model's predictions.
Some changes improve the result.
Others make it worse.
By correlating the perturbations with changes in loss, Dust estimates which direction the layer's output should move.
And because the input to that layer is already known from the forward pass, the estimated output error can be converted into an approximate weight gradient.
That gradient can then be passed to an ordinary optimiser such as SGD or Adam.
So the optimiser doesn't disappear.
The gradients don't disappear either.
What disappears is the need to calculate them by propagating derivatives backwards through the entire network.
It's a very different proposition from the idea that neural networks can simply train themselves without gradients.
They still need a useful direction for changing their weights.
Dust obtains that direction experimentally.
Every token becomes a little laboratory
The cleverest part is the treatment of tokens.
Traditional weight-space perturbation methods often need to evaluate different perturbed models separately.
Each candidate is expensive.
Dust instead perturbs internal activations independently at each token position.
This means one forward evaluation can simultaneously collect information from many different perturbations.
The researchers describe this as a virtual population.
A sequence containing thousands of tokens can provide thousands of token-specific experiments without creating thousands of separately materialised weight configurations.
That doesn't mean the entire training process happens in one forward pass.
Dust still uses many perturbation draws and considerable computation.
But it extracts far more useful experimental information from each pass than a straightforward weight-space search.
There's another complication, of course.
Transformers have attention.
Changing an activation at one token may affect not only that token's prediction but also predictions at later positions.
So the question becomes slightly more involved than whether an individual token's own loss improved.
Dust accounts for this with token-level reward assignment, including information from later tokens where appropriate.
The attention components require special treatment, because their contribution isn't always clearly visible in the immediate token loss.
The paper describes separate perturbation procedures and credit assignment based on estimated changes at the attention output.
In other words, this isn't just random noise sprayed over a transformer while someone hopes for the best.
It's a carefully organised system for measuring which disturbances deserve credit.
The randomness provides the experiments.
The credit-assignment mechanism turns the experiments into learning.
The results are interesting. The computational bill is less romantic.
The Dust paper reports experiments with transformer models ranging from approximately 2 million to 243 million parameters.
Its main model has 37.7 million parameters.
The researchers evaluate training on FineWeb text and compare the method with conventional backpropagation and a weight-space evolution-strategy baseline called EGGROLL.
Their findings challenge the assumption that zeroth-order training becomes hopelessly ineffective as networks grow.
At sufficiently large perturbation populations, Dust approaches the performance of backpropagation under SGD and surpasses it in several tested settings.
It also performs dramatically better than the weight-space evolution-strategy baseline.
The authors estimate an advantage of roughly 1,000 to 10,000 times over EGGROLL in the relevant comparisons.
That number needs careful handling.
It's not a claim that Dust trains language models 10,000 times faster than conventional backpropagation.
It is a comparison against another perturbation-based training approach, partly based on extrapolating how large a population EGGROLL would need to match Dust.
And the paper's authors are explicit about the main limitation.
Dust is not yet computationally efficient enough to replace backpropagation in ordinary large-scale training.
In the original Noise benchmark, the Dust baseline reached validation loss 4.167 using Adam, while the backpropagation baseline reached 3.769.
With loss, lower is better.
So the original result was impressive for a forward-only training method, but it wasn't a general victory over conventional training.
Which makes what happened next rather entertaining.
Then the open-source crowd got involved
Q Labs turned the research into Noise, a constrained training challenge.
The initial benchmark used the 37.7-million-parameter transformer, approximately 10 million training tokens and a fixed validation procedure.
The challenge was to find a better learning algorithm without running a backward pass.
One track rewarded lower validation loss.
Another rewarded reaching a target loss as quickly as possible.
Then people started experimenting.
Different credit-assignment strategies.
Different optimisers.
Orthogonal perturbations.
Structured probes.
Better use of attention.
More efficient GPU kernels.
Less redundant computation.
At one point, contributors replaced some of the random sampling with structured Hadamard-based probing.
Others found ways to recover information that faster approximations had thrown away.
And the results moved quickly.
According to Q Labs' October 2026 update, the original speedrun time fell from approximately 1.76 hours to 1.6 seconds in ten days.
More than a 4,000-fold improvement.
In the same original benchmark, the best reported validation loss improved from 4.167 to 3.676, passing the published Adam backprop baseline of 3.769.
There's an important qualification here too: that doesn't establish superiority over every backpropagation configuration. Stronger conventional optimisers and different hyperparameters can change the comparison.
Still, that's an extraordinary rate of improvement for a research challenge.
It also tells us something useful.
The original Dust implementation was only one point in a large design space.
Once other researchers began changing the estimation method, update strategy and execution kernels, the possibilities expanded considerably.
By late October, the competition had moved on to a larger 30-million-token setting, with an open-architecture track added as well.
The initial 10-million-token records are therefore best understood as the first phase of the experiment, not a permanent benchmark of what the approach can achieve.
And this is precisely why I find open research challenges like this interesting.
A paper introduces a mechanism.
The community discovers that the first implementation is far from the final word.
Then somebody finds a completely different way to use the same principle.
Suddenly, a respectable academic result has become a rather lively engineering competition.
But the most interesting possibility isn't faster training
Here's where the research starts to become genuinely provocative.
Backpropagation depends on differentiability.
If you want to propagate gradients through a system, the operations along that path must support the necessary derivatives, or you need some workaround that supplies them.
That's perfectly manageable for conventional neural-network layers.
Less convenient when your model begins incorporating things such as external programs, discrete decisions, search procedures, simulators or memory systems.
Consider a model that calls a search engine halfway through its computation.
Or one that interacts with a simulator.
Or a recurrent architecture that performs a large number of internal reasoning steps.
The overall system may have a perfectly meaningful objective.
Its behaviour can be evaluated.
But obtaining an exact gradient through every component may be difficult, expensive or impossible.
A training method based on forward experiments offers a different possibility.
Instead of insisting on differentiating through the entire process, you measure whether perturbations made the final behaviour better or worse.
That's a much more general idea.
It doesn't automatically solve the credit-assignment problem, of course.
In fact, introducing a complicated nondifferentiable component may make the signal noisier and harder to attribute to individual decisions.
But it changes the fundamental constraint.
The model doesn't necessarily need to be one enormous, neatly differentiable mathematical object.
It may eventually become a trainable system containing neural networks, tools, programs, memory and iterative computation.
That's the part that interests me most.
Not eliminating a backward pass to save a few seconds.
But expanding what we're allowed to put inside the learning process.
And then there are looped transformers
Another potential application is recurrent or looped transformer architectures.
The intuition is attractive.
Instead of constantly adding more layers, why not let the network reuse computation over multiple steps?
Allow it to iterate.
Refine intermediate representations.
Perform more internal work before producing an answer.
The problem is that training very long computational chains with conventional backpropagation through time can become expensive and difficult.
Memory usage, numerical behaviour and the depth of the dependency chain all matter.
A forward-based credit-assignment mechanism could offer a different way to train such systems without propagating gradients through every iteration.
But there's a critical distinction between a promising direction and a demonstrated result.
The Dust paper did not demonstrate a superior large-scale training system for looped transformers, external-search architectures or simulator-based agents.
The authors identify those as possibilities for future research.
That's a perfectly respectable outcome.
Not every interesting paper needs to have solved its own long-term implications.
Sometimes showing that a supposedly impractical approach can work at all is the important step.
Three reasons not to declare backpropagation dead
First, compute still matters.
A large population of perturbation experiments can be expensive, even when those experiments are organised efficiently.
Backpropagation has decades of optimisation behind it.
Modern GPU hardware, software frameworks and training pipelines are extremely good at executing it.
Matching a training loss doesn't necessarily mean matching the total cost of reaching that loss.
Second, the evidence is still relatively small-scale.
Training a 37.7-million-parameter transformer is meaningful research.
But it's not the same as demonstrating a practical alternative for models with tens or hundreds of billions of parameters.
Even the larger model experiments in the paper remain far below frontier-scale language-model training.
Third, the experiments are not yet demonstrations of better real-world reasoning.
Lower cross-entropy on a held-out text dataset is useful.
But it doesn't prove better tool use, better generalisation, more reliable planning or superior reasoning.
Those claims would require separate experiments.
The researchers themselves leave many of these questions open.
So no, I wouldn't start redesigning every training system around Dust tomorrow.
But I would absolutely keep watching it.
What I think Noise gets particularly right
The most useful part of the project may be its research structure.
Fix the task.
Publish the baseline.
Define the evaluation.
Let people compete.
Make their implementations inspectable.
And, when the benchmark starts becoming too easy to optimise specifically, raise the difficulty.
The move from 10 million to 30 million training tokens matters for exactly this reason.
A clever optimisation can dominate a tiny benchmark without generalising to larger problems.
Changing the scale forces contributors to confront that possibility.
It's the distinction between winning a benchmark and discovering a learning algorithm that actually scales.
Noise doesn't eliminate this risk.
But at least the project openly acknowledges it.
And the experiments are reproducible enough for the community to inspect and challenge them.
That's a much healthier approach than announcing that an entire field of machine learning has become obsolete because someone found an attractive number in a benchmark table.
The really interesting question
For me, Dust and Noise raise a broader question about how we design AI systems.
For decades, we've built increasingly sophisticated architectures around the training mechanism that worked best.
Backpropagation.
It has been astonishingly successful.
But its success also shaped the kinds of systems we chose to build.
What happens when we get better at assigning credit without needing to calculate derivatives through everything?
Could we train systems that incorporate discrete tools and external state more naturally?
Could we allow models to perform long sequences of internal computation without making training prohibitively expensive?
Could optimisation explore useful directions that ordinary first-order gradients don't reveal?
Possibly.
The existing experiments don't prove those outcomes.
But they make the questions worth investigating.
And that's enough to get my attention.
Because the real promise isn't that backpropagation will disappear next Tuesday.
It's that, eventually, the architecture of an intelligent system might no longer need to be designed around the convenience of its backward pass.
That would be a much more interesting change than another percentage point on a benchmark.
For now, Q Labs has shown that you can remove a fundamental part of the conventional training procedure, replace it with carefully organised experiments and still make a transformer learn.
Then an open-source community demonstrated how much optimisation was still available.
Backpropagation is very much alive.
But for the first time in a while, it has some unusually noisy competition.
And I rather like that.
Sources
Dust: Pretraining Transformers Without Backpropagation — Q Labs
Noise — GitHub repository, rules, implementations and records
Noise: Speedrun for Pretraining Without Backpropagation — Q Labs