Introduction to Automatic Differentiation

Automatic differentiation (AD) is the backbone of modern deep learning. Unlike symbolic differentiation (which manipulates expressions) or numerical differentiation (which approximates via finite differences), AD computes exact derivatives by decomposing programs into elementary operations and applying the chain rule systematically.

The Chain Rule

Given a composition of functions $f=f_n\circ f_{n-1}\circ \cdots \circ f_1$, the chain rule tells us:

$$\frac{\partial f}{\partial x}=\frac{\partial f_n}{\partial f_{n-1}}\cdot \frac{\partial f_{n-1}}{\partial f_{n-2}}\cdots \frac{\partial f_2}{\partial f_1}\cdot \frac{\partial f_1}{\partial x}$$

The order in which we multiply these Jacobians gives rise to two modes of AD.

Forward Mode

Forward mode propagates derivatives alongside the computation, from inputs to outputs. For each input variable $x_i$, we track a dual number $(v_i,{\dot{v}}_i)$ where ${\dot{v}}_i=\frac{\partial v_i}{\partial x}$.

For a function $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$, forward mode computes one column of the Jacobian per pass — efficient when $n\ll m$.

Reverse Mode

Reverse mode — what frameworks like PyTorch call “backpropagation” — propagates derivatives from outputs to inputs. It first performs a forward pass to build a computation graph, then walks it backward:

$${\bar{v}}_i=\sum _{j\in \text{children}(i)}{\bar{v}}_j\cdot \frac{\partial v_j}{\partial v_i}$$

where ${\bar{v}}_i=\frac{\partial L}{\partial v_i}$ is the adjoint. This computes one row of the Jacobian per pass — efficient when $m\ll n$, which is exactly the case in ML (scalar loss, many parameters).

A Minimal Value Class

Here’s the core of a reverse-mode AD engine in TypeScript:

class Value {
  data: number;
  grad: number = 0;
  private backward: () => void = () => {};
  private children: Value[];

  constructor(data: number, children: Value[] = []) {
    this.data = data;
    this.children = children;
  }

  add(other: Value): Value {
    const out = new Value(this.data + other.data, [this, other]);
    out.backward = () => {
      this.grad += out.grad;
      other.grad += out.grad;
    };
    return out;
  }

  mul(other: Value): Value {
    const out = new Value(this.data * other.data, [this, other]);
    out.backward = () => {
      this.grad += other.data * out.grad;
      other.grad += this.data * out.grad;
    };
    return out;
  }

  backprop(): void {
    const topo: Value[] = [];
    const visited = new Set<Value>();
    const build = (v: Value) => {
      if (!visited.has(v)) {
        visited.add(v);
        v.children.forEach(build);
        topo.push(v);
      }
    };
    build(this);
    this.grad = 1;
    for (const v of topo.reverse()) {
      v.backward();
    }
  }
}

Interactive Computation Graph

Try adjusting the inputs below to see how gradients flow through a simple computation $y=w\cdot x+b$:

x = w = b = Show Gradients

Computation graph: y = w·x + b. Toggle gradients to see ∂y/∂(each variable).

Key Takeaways

• Forward mode is $O(n)$ passes for $n$ inputs — good for few inputs, many outputs
• Reverse mode is $O(m)$ passes for $m$ outputs — ideal for ML’s scalar-loss setup
• The computation graph records operations during the forward pass, then gradients propagate backward through the graph
• TSTorch implements reverse-mode AD, just like PyTorch, but targeting TypeScript and WebGPU