Statistics on davethehuman.com

gini index

Tue, 13 Oct 2026 00:00:00 +0000

Gini Index or Gini Impurity is a metric used to measure the frequency with which a randomly chosen element from a dataset would be incorrectly labeled if it were randomly labeled according to the distribution of labels in the subset.

In Machine Learning, it is primarily used as a splitting criterion by the algorithm to build decision trees.

Formula
#

The Formula For a dataset with $C$ total classes, where $p_i$ is the probability of an item belonging to class $i$:

variance (statistics)

Tue, 13 Oct 2026 00:00:00 +0000

In statistics, Variance measures data dispersion around a mean.
It is a descriptive statistical measure that quantifies how far individual data points deviate from the mean of a dataset.

Depending on the fact that the dataset represent a full population or a sample, the formula is:

$\sigma^2 = \frac{\sum_{i=1}^{N} (x_i - \mu)^2}{N}$ for the entire population
$s^2 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n - 1}$ for a sample

In numpy, it can be implemented as:

entropy

Sat, 10 Oct 2026 00:00:00 +0000

Entropy is a statistics concept used to measure disorder in a sample.
It is applied to calculate the information gain of a decision tree’s split through node impurity, and can be described with the formula:

$$ E = -\sum_{i=1}^{n} P(X) \log_2 P(X) $$

In a decision tree’s node, $P(X)$ is the number of times a given class has occurred in the node divided by the number of total examples $n$ in the node.
It is basically the probability of encountering that class in the node’s sample.

bias

Tue, 29 Sep 2026 00:00:00 +0000

In statistics and machine learning, Bias refers to the error coming from model oversimplification and leading to underfitting.
A model with high bias is not able to capture the true complexity of the data, leading to poor performance on both training set and test set.
The bias is represented by the difference between the expected or true value of the target variable and the predicted value of the model.

bias-variance trade-off