Machine Learning Algorithms and When to Use Each One

You have a table of data, a column you want to predict and a long list of algorithms with impressive names. Where do you start? A catalogue of ten names won’t answer that. A decision guide will.

The scikit-learn documentation frames it the same way in its guide “Choosing the right estimator”: each estimator suits a particular kind of data and problem. Its diagram is a rough guide, and the “Try next” arrows tell you what to try if the first estimator doesn’t give the result you want. We’ll follow that idea: from the simplest to the most flexible, with the reason for each step.

What this guide covers (and what it doesn’t)

I focus on the algorithms I can back with documentation and published studies: linear models, logistic regression, decision trees, random forests, gradient boosting and neural networks. Other classics, such as k-nearest neighbours, support vector machines, Naive Bayes, k-means or principal component analysis, are left out here because the sources I consulted don’t cover them in detail. That is why the title doesn’t promise a round number.

What supervised models have in common

Let’s open the box. A machine learning model is a function with parameters. You give it examples with their correct answers, it measures how wrong it is and adjusts the parameters to be a little less wrong. This is repeated many times. If you want the big picture, you’ll find it in What is Machine Learning?.

It helps to separate two kinds of problem:

Classification: predicting a category (spam or not spam).
Regression: predicting a numerical value (the price of a house).

Several algorithms work for both. Decision trees, for example, are used for classification and for regression, according to the scikit-learn documentation.

Linear regression: the starting point

Linear regression assumes that the target value is a linear combination of the features. According to the documentation on linear models, ŷ(w, x) = w0 + w1x1 + … + wp xp. Each weight w tells you how much the prediction goes up or down when its feature increases by one unit, and w0 is the base value. That is why you can read the model weight by weight.

When to use it: when you expect a more or less linear relationship and want a model you can explain. My opinion: it is worth trying it every time as a baseline. If a complex model doesn’t clearly beat it, the extra complexity isn’t paying for itself.

It’s easier to see in code. This example fits a straight line by gradient descent, that is, by moving the weights a little in the direction that reduces the error:

import numpy as np

rng = np.random.default_rng(0)
x = rng.uniform(0, 10, 100)
y = 3 + 2 * x + rng.normal(0, 1, 100)  # synthetic data: intercept 3, slope 2, noise

w0, w1 = 0.0, 0.0
tasa = 0.01  # how far the weights move at each step

for paso in range(2000):
    pred = w0 + w1 * x
    error = pred - y
    perdida = np.mean(error ** 2)       # mean squared error
    w0 -= tasa * 2 * np.mean(error)     # gradient with respect to w0
    w1 -= tasa * 2 * np.mean(error * x) # gradient with respect to w1
    if paso % 400 == 0:
        print(paso, perdida)

print("w0:", w0, "w1:", w1)
assert abs(w0 - 3) < 0.2 and abs(w1 - 2) < 0.1, "the weights have not moved close to 3 and 2"

This is the important part: the loss should go down, and w0 and w1 should approach the values we used to generate the data (3 and 2), with some deviation because of the noise. If you raise the learning rate too far, the weights jump from side to side and the loss stops falling.

Logistic regression: a linear model that classifies

The name is misleading. Despite the word “regression”, scikit-learn implements it as a linear model for classification. It is a special case of the generalised linear models, with a Binomial/Bernoulli distribution and a logit link. The intuition behind the link: the weighted sum can be any number, and it has to be turned into something between 0 and 1 that can be read as a probability.

That probability is turned into a class by applying a threshold, which in scikit-learn is 0.5 by default. When to use it: in binary classification, when you care about the probability and want a model that is simple to explain. Here is what the last step looks like, with weights set by hand (there is no training here, only the mechanics of prediction):

import numpy as np

def sigmoide(z):
    return 1 / (1 + np.exp(-z))  # turns any number into a value between 0 and 1

# Two features per example; weights chosen by hand
X = np.array([[1.0, 2.0], [4.0, 0.5], [0.2, 0.1]])
pesos = np.array([0.8, -0.5])
sesgo = 0.1

z = X @ pesos + sesgo          # weighted sum
prob = sigmoide(z)             # probability of the positive class
clase = (prob >= 0.5).astype(int)  # threshold of 0.5

print(prob)
print(clase)
assert (clase == [0, 1, 1]).all(), "the classes are not 0, 1 and 1"

The threshold is your decision, not a law. It lets you adjust which kind of error costs you more, even though the library’s default value is 0.5.

Decision trees: rules you can read

According to the documentation, a decision tree is a non-parametric supervised learning method for classification and regression. The advantages it lists:

It is simple to understand and interpret, and it can be visualised.
It requires little data preparation.
It is a “white box” model: you can follow why it makes each prediction.

Where it breaks:

Overfitting: it can create over-complex structures that don’t generalise. This is mitigated with pruning, a minimum number of samples per leaf or a maximum depth.
Instability: small variations in the data can produce a very different tree.
No extrapolation: its predictions are piecewise constant, so they don’t extrapolate well outside the range seen in training.
Categorical variables: the scikit-learn implementation does not support them for now, according to the documentation for version 1.9.1.

A tree that scores well on training data and fails on test data hasn’t learned: it has memorised. I come back to this in “Common mistakes”.

Random forests and gradient boosting: many trees together

A single tree is unstable, so many are combined. The documentation on ensembles describes two opposing strategies.

Bagging and random forests. Several instances of an estimator are built on random subsets of the training data and their predictions are aggregated (majority vote in classification). This reduces variance, because the trees have partially decoupled errors. It works best with strong, complex models, such as fully grown trees.

Gradient boosting. The trees are built in sequence and each one is trained to correct the errors of the previous ones. Boosting works best with weak models. In the histogram-based variant (HGBT), random forests may need more trees to reach the same accuracy.

On the left, three trees trained in parallel on random subsets send their predictions to a final vote. On the right, three chained trees: each one corrects the errors of the previous one and the final result is the sum of all of them.Bagging (parallel)Tree 1Tree 2Tree 3Final voteBoosting (chained)Tree 1Tree 2Tree 3Each tree corrects the errorsof the previous ones

The price of boosting is interpretability. Gradient boosting models comprise hundreds of regression trees, and they cannot be understood by inspecting the trees one by one.

The best-known implementation is XGBoost, a scalable, open-source tree boosting system described by Chen and Guestrin in “XGBoost: A Scalable Tree Boosting System” (KDD 2016). The paper includes a sparsity-aware algorithm for sparse data and a weighted quantile sketch for approximate learning. Its most cited figure is a historical one: of the 29 winning solutions to Kaggle challenges published on the Kaggle blog during 2015, 17 used XGBoost. That is a snapshot from 2015, not a current ranking.

Neural networks and tabular data: when they are not the first choice

Neural networks dominate in images and language, but that doesn’t make them the best starting point for every table. If you want to see how they work inside, there is how neural networks work and, for the many-layered variety, what deep learning is.

What published studies say about tabular data:

The survey “Deep Neural Networks and Tabular Data: A Survey” (v3, 29 June 2022) concludes that gradient-boosted tree ensembles still mostly outperform deep learning models on these tasks. XGBoost, LightGBM and CatBoost win on most of the datasets considered, with notably shorter training times.
Recent empirical studies (Grinsztajn et al., 2022; Borisov et al., 2022; McElfresh et al., 2023) suggest the same, as reported in “Tabular Data: Is Deep Learning all you need?”.

There are counterpoints. According to the 2022 survey, one line of work shows that multilayer perceptrons with “cocktails” regularisation can outperform tree-based architectures. But the per-dataset hyperparameter optimisation this requires costs far more compute than gradient boosting. And in the October 2025 version of “Tabular Data: Is Deep Learning all you need?”, the meta-learned models TabPFNv2 and TabICL outperform tree-based models on most of the datasets in that comparison.

My reading, as an opinion: as of November 2026 the debate is still open, and the answer depends on your compute and tuning budget. For the research context, head to Kia’s articles.

How to choose

Do you want a readable baseline? Linear regression (numerical value) or logistic regression (category).
Do you want visible rules? A tree with limited depth.
Are you after the highest accuracy on tabular data? Tree ensembles: random forest or gradient boosting.
Isn’t the first one enough? Follow the “Try next” arrows in the scikit-learn guide.

Another opinion of mine: in most real problems, the data matters more than the algorithm. Cleaning labels and variables usually pays off more than switching model.

Common mistakes

A tree with no depth limit. It memorises the training data.
Trusting a single tree. It is unstable when the data changes slightly.
Expecting a tree to extrapolate. Its predictions are piecewise constant.
Evaluating on the training data. Always measure on data the model has not seen.
Using deep learning without a budget for tuning hyperparameters. That is exactly the cost the cited survey points to.

To practise

Take a tabular dataset of your own and train a logistic regression and a decision tree. Vary the tree’s maximum depth and compare the accuracy on training data with the accuracy on test data. The point where the first keeps rising and the second levels off or drops is where memorisation begins. Then try a random forest and see how much that gap shrinks.

To see the debate between trees and deep learning from the research side, the next step is Kia’s articles.

Sources

Image: Lalmch on Pixabay

Translated and adapted by Ollie.

The Most Common Machine Learning Algorithms, and When to Use Each One