NeurIPS 2026 spotlight

Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks

TelepathNew York University

Deep stacks of slow neurons fall behind

\[\tau\,\dot y_\ell = -y_\ell + y_{\ell-1}, \qquad y_0 = \class{c-z}{\text{input}}\]

Each layer is a leaky integrator driven by the layer above it.

Neurons look ahead

\[\chi_\tau = 1 + \tau\,\frac{d}{dt}\]

The look-ahead undoes the leaky integration.

Each layer, as before

\[\tau\,\dot y_\ell = -y_\ell + y_{\ell-1}\]

With look-ahead on each layer’s input

\[\tau\,\dot y_\ell = -y_\ell + \class{c-yh}{\chi_\tau\big(y_{\ell-1}\big)}\]
\[\phantom{\tau\,\dot y_\ell} = -y_\ell + y_{\ell-1} + \class{c-yh}{\tau\,\dot y_{\ell-1}}\]

Prospective firing: cortex, retina, cerebellum, muscle spindles

Start from an energy principle

Energy of one unit

\[E = \gamma\,\lvert \class{c-y}{y} - \class{c-z}{z}\rvert^2 + (1-\gamma)\,\lvert \class{c-y}{y} - \class{c-yh}{\hat y}\rvert^2\]

A unit balances the bottom-up input against an internal prediction. γ sets the trade-off.

Descend the energy

\[\tau\,\dot{\class{c-y}{y}} = -\frac{\partial E}{\partial y^{\dagger}}\]
\[\tau\,\dot{\class{c-y}{y}} = -\class{c-y}{y} + \gamma\,\class{c-z}{z} + (1-\gamma)\,\class{c-yh}{\hat y}\]

A leaky integrator relaxing toward a blend of input and prediction.

Heeger (2017), Theory of cortical function, PNAS

The prediction is a look-ahead of an oscillation

Simplest model of an oscillation

\[\dot y = i\omega\, y\]

Exact rotation over one time constant

\[y(t+\tau) = e^{i\omega\tau}\, y(t)\]

Look-ahead: the first-order version of that rotation

\[\class{c-yh}{\hat y} = \chi_\tau(y)\big|_{\dot y = i\omega y} = \class{c-yh}{(1 + i\omega\tau)}\, y\]

Internal-model principle (Francis and Wonham, 1976)

The Recursive Quadrature Filter

\[\tau\dot y = -y + \gamma z + (1-\gamma)(1+i\omega\tau)\,y\]
stable for any γ in (0, 1)two learnable numbers per unit
State in the complex plane
Impulse response: a quadrature pair
Frequency response
\[\Longrightarrow\;\; \boxed{\dot y = \class{c-lam}{(-\alpha + i\,\omega_0)}\, y + \alpha z}\]
\[\alpha = \gamma/\tau \qquad \omega_0 = (1-\gamma)\,\omega\]

Every RQF layer is a diagonal state-space model

RQF layer
\[\dot{\mathbf y} = \class{c-lam}{\mathrm{diag}\big({-\boldsymbol\alpha} + i\,\boldsymbol\omega_0\big)}\,\mathbf y + \class{c-B}{\mathrm{diag}\big(\boldsymbol\alpha\big)\,W}\;\class{c-x}{\mathbf x}\]
Diagonal SSM
\[\dot{\tilde{\mathbf y}} = \class{c-lam}{\Lambda}\,\tilde{\mathbf y} + \class{c-B}{W}\,\class{c-x}{\mathbf x}\]

One constraint: each channel’s input gain is tied to its own decay rate, α = γ/τ.

Every RQF layer is a diagonal SSM; not every diagonal SSM is an RQF.
RQFs train with any method built for diagonal SSMsParallel scans, FFT convolutions, and the rest of the SSM toolkit, which is how they reach sequences of 16,384 steps.

Look-ahead on the input: two taps, zero parameters

The RQF

\[\dot y = \class{c-lam}{(-\alpha + i\,\omega_0)}\, y + \alpha\, z\]

Make the input drive prospective

\[\dot y = \class{c-lam}{(-\alpha + i\,\omega_0)}\, y + \alpha\, \class{c-pro}{\chi_\tau(z)} = \class{c-lam}{(-\alpha + i\,\omega_0)}\, y + \alpha\,\big(z + \class{c-pro}{\tau\,\dot z}\big)\]

One discretized step, instantaneous input

\[y_{n+1} = \bar A\, y_n + \bar B\, z_{n+1}\]

With prospective input

\[y_{n+1} = \bar A\, y_n + \big[\bar B + \class{c-pro}{\gamma \bar A}\big]\, z_{n+1} \;\class{c-pro}{-\; \gamma \bar A\, z_n}\]
yn A zn+1 B B + γA + yn+1 zn −γA
  # one step of an RQF or any diagonal SSM  y = A_bar * y + B_bar * z[t]  y = A_bar * y + B_plus * z[t] + B_minus * z_prev  z_prev = z[t]  # B_plus = B_bar + γ·A_bar,  B_minus = −γ·A_bar
0 new parameterssame transition Āsame parallel scanany diagonal SSM

Small, interpretable, and it scales

0%

Raw-audio Speech Commands with 31.9k parameters. No residuals, gating, or normalization.

0%

Path-X, 16,384 steps with a minimal six-layer stack. 86.3% with a selective RQF.

0%

Selective Copying with a selective RQF. Look-ahead: 5 of 5 seeds; without it, 2 of 5.

Prospective input beat instantaneous input across RQF, S5, and ORGaNICs
Derived from cortex.Trained like an SSM.
Poster in Sydney
Hall 1–4
Thu, Dec 10
10:00 AM–1:00 PM AEDT
Scan for the paper
1 / 9
→ next   ← back   N notes   F full screen   D theme   H hide controls