Decisiontrees on davethehuman.com

gini index

Tue, 13 Oct 2026 00:00:00 +0000

Gini Index or Gini Impurity is a metric used to measure the frequency with which a randomly chosen element from a dataset would be incorrectly labeled if it were randomly labeled according to the distribution of labels in the subset.

In Machine Learning, it is primarily used as a splitting criterion by the algorithm to build decision trees.

Formula
#

The Formula For a dataset with $C$ total classes, where $p_i$ is the probability of an item belonging to class $i$:

entropy

Sat, 10 Oct 2026 00:00:00 +0000

Entropy is a statistics concept used to measure disorder in a sample.
It is applied to calculate the information gain of a decision tree’s split through node impurity, and can be described with the formula:

$$ E = -\sum_{i=1}^{n} P(X) \log_2 P(X) $$

In a decision tree’s node, $P(X)$ is the number of times a given class has occurred in the node divided by the number of total examples $n$ in the node.
It is basically the probability of encountering that class in the node’s sample.

Random Forest

Sat, 26 Sep 2026 00:00:00 +0000

The Random Forest is an ensemble of decision trees that reduces overfitting by using:

bagging to reduce variance
random selection of features from the full set of features at each split to prevent the important features from always being present at the top of each tree

It is important to note that we can’t overfit by putting in too many trees as they just reduce variance; at some point, variance reduction will just stop and we need to take into account computation time.

Decisiontrees on davethehuman.com

gini index

Formula #

entropy

Random Forest

Formula
#