matthewgrad

Background

A while back, I stumbled upon Andrej Karpathy's "Neural Networks: Zero to Hero" series. As someone who was trying to learn about machine learning and how I can build my own models, it interested me.

Fast forward a couple of weeks, and I decided to start the course with Mr. Karpathy's video lecture explaining how he built micrograd. Along the way, he provides a very accessible breakdown of how neural nets work, including the math behind them and the actual implementation of it.

The entire lecture is around two and a half hours, so I split it across three days. By the end of it, I had created matthewgrad, a small automatic gradient engine and neural network library.

Breakdown

Project structure

matthewgrad/
├── matthewgrad/
│   ├── init.py
│   └── engine.py
│   └── nn.py
│   └── viz_utils.py
├── test.py
└── README.md

• engine.py: Core autograd engine.
  • Defines Value.

class Value:
    def __init__(self, data, _children=(), _op="", label=""):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op
        self.label = label
    # ...
    def backward(self):
        # ...

• Tracks operations and gradients.
• Supports .backward() for reverse-mode autodiff.
• nn.py: Neural network components.
  • Module base class.

class Module:
    def zero_grad(self):
        for p in self.parameters():
            p.grad = 0
    def parameters(self):
        return []

• Neuron, Layer, and MLP.

class Neuron(Module):
    def __init__(self, nin):
        self.w  = [Value(random.uniform(-1, 1)) for _ in range(nin)]
        self.b = Value(random.uniform(-1, 1))
    # ...
class Layer(Module):
    def __init__(self, nin, nout): # `nin` = number of inputs; `nout` = number of ouputs
        self.neurons = [Neuron(nin) for _ in range(nout)]
    # ...
class MLP(Module):
    def __init__(self, nin, nouts): # `nin` = number of inputs; `nouts` = number of outputs
        sz = [nin] + nouts
        self.layers = [Layer(sz[i], sz[i+1]) for i in range(len(nouts))]
        self.loss = 0.0
    # ...
    def train(self, max_iter, X_train, y_train):
        # ...
    def predict(self, X):
        # ... 

• Simple training loop with gradient descent.
• viz_utils.py: Graph visualization helpers.
  • trace(root) collects graph nodes and edges.

def trace(root):
    # builds a set of all nodes and edges in a graph 
    nodes, edges = set(), set()
    def build(v):
        if v not in nodes:
            nodes.add(v)
            for child in v._prev:
                edges.add((child, v))
                build(child)
    build(root)
    return(nodes, edges)

• draw_dot(root) renders the computation graph.

def draw_dot(root):
    dot = Digraph(format="svg", graph_attr={"rankdir": "LR"}) # left to right
    nodes, edges = trace(root)
    for n in nodes:
        uid = str(id(n))
        dot.node(name=uid, label="{ %s | data %.4f | grad %.4f }" % (n.label, n.data, n.grad), shape="record")
        if n._op:
            dot.node(name=uid+n._op, label =n._op)
            dot.edge(uid+n._op, uid)
    for n1, n2 in edges:
        dot.edge(str(id(n1)), str(id(n2)) + n2._op)
    return dot

• test.py: Example usage.
  • Builds a 3 → 8 → 8 → 1 MLP.
  • Trains on a small toy dataset.
  • Prints prediction and loss.
  • Can render the computation graph.

How it works

matthewgrad uses a tiny reverse-mode autodiff engine:

1. Each Value stores:

   • A scalar data value.
   • A scalar grad.
   • References to previous values in the computation graph.

2. Operations like +, *, tanh, and exp create new Value nodes and define local backward functions.

3. Calling .backward() on the final loss:

   • Builds a topological order of the graph using a topological sort.
   • Propagates gradients from the output back to all parameters.

4. nn.py uses those gradients to update weights with stochastic gradient descent.

Experimentation

Out of curiosity, I created a computation graph of an MLP with 1000 inputs, two layers of 8 nodes, and 1 output. The result:

Links

Check out the project on GitHub.

Watch Mr. Karpathy's video lecture.