Neural differential equation

Neural differential equations are a class of models in machine learning that combine neural networks with the mathematical framework of differential equations. These models provide an alternative approach to neural network design, particularly for systems that evolve over time or through continuous transformations.

The most common type, a neural ordinary differential equation (neural ODE), defines the evolution of a system's state using an ordinary differential equation whose dynamics are governed by a neural network: $${\displaystyle {\frac {\mathrm {d} \mathbf {h} (t)}{\mathrm {d} t}}=f_{\theta }(\mathbf {h} (t),t).}$$In this formulation, the neural network parameters θ determine how the state changes at each point in time. This approach contrasts with conventional neural networks, where information flows through discrete layers indexed by natural numbers. Neural ODEs instead use continuous layers indexed by positive real numbers, where the function ${\displaystyle h:\mathbb {R} _{\geq 0}\to \mathbb {R} }$ represents the network's state at any given layer depth t.

Neural ODEs can be understood as continuous-time control systems, where their ability to interpolate data can be interpreted in terms of controllability. They have found applications in time series analysis, generative modeling, and the study of complex dynamical systems.