sigmoid

The sigmoid function is used in Logistic Regression to transform the linear output in a probability between 0 and 1.

For a linear score $z = X\beta$, the sigmoid function is:

When negative values are also needed while keeping the sigmoid’s smooth, bounded output, tanh is used.

Implementation

#

A correct implementation of sigmoid has two branches: one for negative values and one for positive values.
The original formula at the top of the page is correct for every real number, but a computer has a largest representable floating-point value that can lead to overflow with exponential operation on large positive numbers.

For example, in case of z=-1000 the formula asks to compute exp(1000), which is too large and overflows, while the final sigmoid should simply be very close to zero.

We choose the formula according to the sign of z so the value passed to exp is always zero or negative:

• Nonnegative score (z >= 0): keep the standard formula. Here -z <= 0, so exp(-z) is safe. For example, if z = 1000, we compute exp(-1000), which safely becomes approximately zero, and the probability becomes approximately one.
• Negative score (z < 0): rearrange the same formula by multiplying its numerator and denominator by exp(z): $$\frac{1}{1+e^{-z}}\times\frac{e^z}{e^z}=\frac{e^z}{1+e^z}.$$Now, if z = -1000, we compute exp(-1000), which safely becomes approximately zero. The probability also becomes approximately zero. We never compute the dangerous exp(1000).

The two branches are not two different sigmoid functions. They are two algebraically equivalent ways to calculate the same function, selected only to keep the intermediate exponential safe.

def _sigmoid(scores):
	scores = np.asarray(scores, dtype=float)
	probabilities = np.zeros_like(scores)

	nonnegative = scores >= 0
	probabilities[nonnegative] = 1.0 / (1.0 + np.exp(-scores[nonnegative]))

	negative = scores < 0
	exp_scores = np.exp(scores[negative])
	probabilities[negative] = exp_scores / (1.0 + exp_scores)
	return probabilities