How Neural Networks Work: Step-by-Step Guide with Code

What does a neural network do when it “learns”?

Between the first example a network sees and the last, what has changed inside it? The answer fits in one phrase: the weights. A neural network is a function with many parameters, and training it means adjusting those parameters bit by bit so that the error goes down. There is nothing more underneath.

Here is the framework: the multilayer perceptron (MLP) is a supervised learning algorithm. According to the scikit-learn documentation, it learns a function f: R^m → R^o from a set of examples, where m is the number of input dimensions and o the number of output dimensions. “Supervised” means that every example comes with the correct answer.

We will do this in two passes: first a minimal version written by hand with NumPy, then the same loop with PyTorch, so you can see what the library takes off your hands. If you want to place this within the bigger picture first, you can start with what machine learning is.

Let’s open the box: neurons and layers

An artificial neuron does two things. First, it takes a weighted linear sum of its inputs: w₁x₁ + w₂x₂ + … + w_mx_m. Then it passes the result through a non-linear activation function, such as the hyperbolic tangent (this is how the scikit-learn documentation describes it). The w are the weights; the x are the inputs.

These neurons are organised into layers: an input layer, one or more hidden layers and an output layer. The output layer receives the values from the last hidden layer and transforms them into the final values.

Diagram with three columns: two input nodes on the left, three nodes of a hidden layer in the middle and one output node on the right. Every node in one column is connected to all the nodes in the next by lines that represent the weights.InputHidden layerOutputweights W1weights W2

Why hidden layers and non-linearity are needed

What distinguishes an MLP from logistic regression is that, between the input and the output, it has non-linear hidden layers (according to the same documentation). My intuition, without proof: if every layer only added and multiplied, the whole thing would still be a single linear operation, however many layers you stacked. The activation function is what prevents that collapse.

And what do the hidden units do? In the work of Rumelhart, Hinton and Williams, published in Nature in 1986, they describe how, as a result of the weight adjustments, these internal units come to represent important features of the task. Nobody tells the network which features to look for: they appear because they help reduce the error.

Measuring the error: the loss function

To correct something you need to know how far off it is. The loss function L(y) = L(M(x)) returns a single number, a scalar, that expresses how far the prediction landed from the ideal output for a given input, as the PyTorch autograd tutorial explains. The idea was already there in the 1986 paper: minimise a measure of the difference between the actual output vector and the desired one.

For classification, scikit-learn uses cross-entropy and applies softmax at the output when there are several classes. Intuitively, softmax turns the outputs into values that can be read as a split of confidence between classes. Cross-entropy penalises heavily a network that gives little confidence to the correct class.

How the weights get corrected: backpropagation and gradient descent

The intuition: for each weight, work out how much it contributed to the error and nudge it in the direction that reduces it. In scikit-learn, once the loss has been computed, a backward pass propagates it from the output layer to the earlier ones and gives each weight its update value.

It is worth keeping the terms apart. Backpropagation computes the gradients. Gradient descent (or some variant of it) uses those gradients to update the weights. The scikit-learn documentation puts it precisely: the MLP is trained with some form of gradient descent whose gradients are computed with backpropagation.

On the origins: the Nature paper, 323, 533–536 (1986), was received on 1 May and accepted on 31 July of that year. Its authors point out that the ability to create new, useful features distinguishes backpropagation from earlier, simpler methods such as the perceptron-convergence procedure.

Two settings you will see in any optimiser, according to the PyTorch training tutorial:

The learning rate sets the size of each step.
Momentum accumulates the gradients from previous steps and gives the optimiser inertia: it smooths out oscillations and speeds up in directions where the gradient stays consistent.

And the starting point: random initial weights.

The training loop in code

It is easier to see in code. A network with one hidden layer that learns XOR, a function that no single neuron can solve. Each cycle does four things: forward pass, loss, gradients by hand and update.

import numpy as np

rng = np.random.default_rng(1)

# XOR: four examples with their answer
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

# Random initial weights, 4 hidden neurons
W1 = rng.normal(0, 1, (2, 4)); b1 = np.zeros((1, 4))
W2 = rng.normal(0, 1, (4, 1)); b2 = np.zeros((1, 1))
lr = 0.5

for paso in range(5000):
    # Forward pass
    h = np.tanh(X @ W1 + b1)
    p = 1 / (1 + np.exp(-(h @ W2 + b2)))

    # Loss: binary cross-entropy
    perdida = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))

    # Backward pass: gradients by hand
    dz2 = (p - y) / len(X)
    dW2 = h.T @ dz2
    db2 = dz2.sum(axis=0, keepdims=True)
    dz1 = (dz2 @ W2.T) * (1 - h ** 2)   # derivative of tanh
    dW1 = X.T @ dz1
    db1 = dz1.sum(axis=0, keepdims=True)

    # Gradient descent
    W1 -= lr * dW1; b1 -= lr * db1
    W2 -= lr * dW2; b2 -= lr * db2

    if paso % 1000 == 0:
        print(paso, perdida)

# Check: the four predictions should end up close to 0, 1, 1 and 0
print("predictions:", p.ravel().round(2))
assert np.all(np.abs(p - y) < 0.1), "the network has not learned XOR"

What you should see is the printed loss going down and, at the end, the four predictions landing close to 0, 1, 1 and 0. The backward-pass section is the important part: each gradient is obtained by chaining the one from the next layer, which is why it is called backpropagation. It also shows why the derivative of the activation (1 - h ** 2) appears in the calculation.

Writing this by hand is manageable with two layers. The Learning PyTorch with Examples tutorial says it plainly: it is not a problem for a small two-layer network, but it quickly gets complicated for large, complex ones.

The same loop with PyTorch

This is where automatic differentiation comes in. With autograd, the forward pass defines a computational graph: the nodes are tensors and the edges are functions that produce tensors from other tensors. Backpropagating through that graph gives the gradients. Each tensor stores the history of the tensors and the function that created it.

That is why, in an nn.Module, you only define forward; the backward is generated for you (neural networks tutorial). The learnable parameters come from parameters().

import torch
import torch.nn as nn
torch.manual_seed(0)

X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = torch.tensor([[0.], [1.], [1.], [0.]])

class Red(nn.Module):
    def __init__(self):
        super().__init__()
        self.oculta = nn.Linear(2, 4)
        self.salida = nn.Linear(4, 1)

    def forward(self, x):
        return torch.sigmoid(self.salida(torch.tanh(self.oculta(x))))

model = Red()

def loss_fn(p, y):  # binary cross-entropy using tensor operations
    return -torch.mean(y * torch.log(p) + (1 - y) * torch.log(1 - p))

optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)

for paso in range(2000):
    optimizer.zero_grad()          # set the gradients to zero
    outputs = model(X)             # forward pass
    loss = loss_fn(outputs, y)     # loss
    loss.backward()                # backpropagation (autograd)
    optimizer.step()               # update the weights
    if paso % 500 == 0:
        print(paso, loss.item())

print("predictions:", outputs.detach().ravel().round(decimals=2))
assert torch.all((outputs - y).abs() < 0.1), "the network has not learned XOR"

The five lines of the loop are the ones shown in the training tutorial: zero_grad(), model(inputs), loss_fn(outputs, labels), loss.backward() and optimizer.step(). The blocks we wrote by hand in NumPy (dz2, dz1, dW1…) shrink to loss.backward(). Without that hand-written gradient calculation, the rest of the loop is the same.

If “deep learning” sounds to you like more of the same with more layers, this deep learning guide (in Spanish) follows that thread.

Common mistakes and where it breaks

Forgetting to zero the gradients. Gradients accumulate, so you need to call optimizer.zero_grad() on every iteration (PyTorch neural networks tutorial). Without that line, each step carries over the calculation from the previous one.
Overfitting. A model that gets the training data right and fails on the test data has not learned: it has memorised. That is why the data is split; the scikit-learn example on regularisation uses train_test_split(X, y, test_size=0.4, random_state=42).
Regularisation. The alpha parameter is a penalty that limits the size of the weights. Raising it can fix high variance, a sign of overfitting, and it smooths the decision boundary.
Local minima. With hidden layers, the loss is non-convex and has more than one local minimum. Different random initialisations can give different validation accuracy.
Scaling and hyperparameters. The MLP is sensitive to feature scaling and requires tuning the number of hidden neurons, the number of layers and the number of iterations.
Input size. The LeNet example in the PyTorch tutorial expects 32×32 images; to use it with MNIST you have to resize them.

A starting tip from scikit-learn: begin with few hidden neurons and few layers, because backpropagation has a high time complexity. If you want to see where this model fits among other methods, go through the most common types of machine learning algorithms.

To practise

Run the NumPy code and change the learning rate (try very small and very large values) and the number of hidden neurons. Notice, without looking for exact figures, how the loss curve changes.
With a somewhat larger dataset, add a penalty on the size of the weights, in the style of alpha, and compare behaviour on training versus test data.
Rewrite the loop in PyTorch, remove optimizer.zero_grad() and see what happens to the training.

We have stopped here, at how it works on the inside.

Sources

Image: geralt on Pixabay

Translated and adapted by Ollie.

How Neural Networks Work: A Step-by-Step Guide with Code