Functions on davethehuman.com

softmax

Tue, 13 Oct 2026 00:00:00 +0000

The softmax function (also known as normalized exponential) is a generalization and extension of the sigmoid to move from binary classification to multi-class classification.

It takes a vector of $K$ real numbers (logits) and normalizes it into a probability distribution consisting of $K$ probabilities proportional to the exponentials of the input numbers.

For an input vector $\mathbf{z} = (z_1, z_2, \dots, z_K) \in \mathbb{R}^K$, the Softmax function $\sigma(\mathbf{z})_i$ for a specific component $i$ is defined as:

tanh

Sun, 27 Sep 2026 00:00:00 +0000

The tanh (hyperbolic tangent) function maps any real valued number in a range between -1 and 1. Similar to the sigmoid, its shape is a continuous “S” but it is centered over 0. It is commonly used in hidden layers of shallow neural networks and RNNs, since it helps them converge faster during training allowing both positive and negative values, and preventing the vanishing gradient problem.

$$tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$$

sigmoid

Sat, 26 Sep 2026 00:00:00 +0000

The sigmoid function is used in Logistic Regression to transform the linear output in a probability between 0 and 1.

For a linear score $z = X\beta$, the sigmoid function is:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

When negative values are also needed while keeping the sigmoid’s smooth, bounded output, tanh is used.

Implementation
#

A correct implementation of sigmoid has two branches: one for negative values and one for positive values.
The original formula at the top of the page is correct for every real number, but a computer has a largest representable floating-point value that can lead to overflow with exponential operation on large positive numbers.

Functions on davethehuman.com

softmax

tanh

sigmoid

Implementation #

Implementation
#