Logitsticregression on davethehuman.com

Logitsticregression on davethehuman.comhttps://davethehuman.com/tags/logitsticregression/Recent content in Logitsticregression on davethehuman.comHugo -- gohugo.ioen© 2026 Dave the humanTue, 13 Oct 2026 00:00:00 +0000softmaxhttps://davethehuman.com/notes/softmax/Tue, 13 Oct 2026 00:00:00 +0000https://davethehuman.com/notes/softmax/<p> The <em>softmax</em> function (also known as <em><strong>normalized exponential</strong></em>) is a generalization and extension of the <a href="https://davethehuman.com/notes/sigmoid/">sigmoid</a> to move from <a href="https://davethehuman.com/coming-soon/" rel="nofollow">binary classification</a> to <a href="https://davethehuman.com/coming-soon/" rel="nofollow">multi-class classification</a>.</p> <p>It takes a vector of $K$ real numbers (<a href="https://davethehuman.com/coming-soon/" rel="nofollow">logits</a>) and <a href="https://davethehuman.com/coming-soon/" rel="nofollow">normalizes</a> it into a <a href="https://davethehuman.com/coming-soon/" rel="nofollow">probability distribution</a> consisting of $K$ probabilities proportional to the exponentials of the input numbers.</p> <p>For an input vector $\mathbf{z} = (z_1, z_2, \dots, z_K) \in \mathbb{R}^K$, the Softmax function $\sigma(\mathbf{z})_i$ for a specific component $i$ is defined as: </p>early stoppinghttps://davethehuman.com/notes/early-stopping/Thu, 08 Oct 2026 00:00:00 +0000https://davethehuman.com/notes/early-stopping/<p><em>Early stopping</em> is a <a href="https://davethehuman.com/coming-soon/" rel="nofollow">regularization</a> technique that stops training when performance on a <a href="https://davethehuman.com/coming-soon/" rel="nofollow">held-out</a> validation set stops improving, helping prevent <a href="https://davethehuman.com/coming-soon/" rel="nofollow">overfitting</a>.</p> <p><a href="https://davethehuman.com/coming-soon/" rel="nofollow">Training loss</a> may keep decreasing while <a href="https://davethehuman.com/coming-soon/" rel="nofollow">validation loss</a> begins to rise: the model is increasingly fitting details that do not <a href="https://davethehuman.com/coming-soon/" rel="nofollow">generalize</a>.</p> <p>A typical procedure monitors validation loss, saves the best <a href="https://davethehuman.com/coming-soon/" rel="nofollow">model checkpoint</a>, and stops after a chosen number of evaluations without meaningful improvement. This waiting period is called <a href="https://davethehuman.com/coming-soon/" rel="nofollow">patience</a>. The best checkpoint is then restored.</p> <p>Another way to apply early stopping is to monitor the algorithm <a href="https://davethehuman.com/coming-soon/" rel="nofollow">convergence</a>: if further iterations barely change the parameters, we can stop when the largest absolute update of <code>w</code> and <code>b</code> is below <code>tolerance</code>. This is implemented in <a href="https://davethehuman.com/notes/linear-regression-from-scratch/">Linear Regression from scratch</a> and <a href="https://davethehuman.com/notes/logistic-regression-from-scratch/">Logistic Regression from scratch</a>.</p>Logistic Regression from scratchhttps://davethehuman.com/notes/logistic-regression-from-scratch/Thu, 08 Oct 2026 00:00:00 +0000https://davethehuman.com/notes/logistic-regression-from-scratch/<h2 class="relative group">Algorithm and implementation <div id="algorithm-and-implementation" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#algorithm-and-implementation" aria-label="Anchor">#</a> </span> </h2> <p><a href="https://davethehuman.com/notes/logistic-regression/">Logistic Regression</a> formula is the same as the <a href="https://davethehuman.com/notes/linear-regression/">Linear Regression</a>’s one passed to a <a href="https://davethehuman.com/notes/sigmoid/">sigmoid</a> function to obtain a <a href="https://davethehuman.com/coming-soon/" rel="nofollow">probability score</a> from <a href="https://davethehuman.com/coming-soon/" rel="nofollow">logits</a> and can be replicated on numpy with:<br> <code>sigmoid(X @ w + b)</code> (or <code>sigmoid(np.dot(X,w))</code>.<br> The implementation of <a href="https://davethehuman.com/coming-soon/" rel="nofollow">gradient descent</a> as an <a href="https://davethehuman.com/coming-soon/" rel="nofollow">optimizer</a> is the same of <a href="https://davethehuman.com/notes/linear-regression-from-scratch/">Linear Regression from scratch</a>, what changes is the <a href="https://davethehuman.com/coming-soon/" rel="nofollow">loss</a> and the error since we deal with probabilities:</p>accuracyhttps://davethehuman.com/notes/accuracy/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/accuracy/<p><em>Accuracy</em> is the <strong>ratio of the correctly classified observations</strong>. In <a href="https://davethehuman.com/coming-soon/" rel="nofollow">binary classification</a> settings, it checks the sum of <a href="https://davethehuman.com/notes/tp/">TP</a> and <a href="https://davethehuman.com/notes/tn/">TN</a> over all the predictions: </p> $$ \mathrm{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} $$<p>It is very important to use this metric on datasets that <strong>do not suffer from <a href="https://davethehuman.com/coming-soon/" rel="nofollow">class imbalance</a></strong>; for example, a model that always predicts “False” on a dataset having 99.9% of False observations will have a very high 0.999 accuracy score but it won’t be meaningful for detecting the True observations.</p>classification metricshttps://davethehuman.com/notes/classification-metrics/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/classification-metrics/<p>When building a <a href="https://davethehuman.com/coming-soon/" rel="nofollow">classifier</a> we want to minimize the number of misclassified observations. In <a href="https://davethehuman.com/coming-soon/" rel="nofollow">binary</a> cases, we talk about <a href="https://davethehuman.com/notes/tp/">true positives</a>, <a href="https://davethehuman.com/notes/fp/">false positives</a>, <a href="https://davethehuman.com/notes/tn/">true negatives</a> and <a href="https://davethehuman.com/notes/fn/">false negatives</a> that can be visualized in a <strong>confusion matrix</strong>:</p> <p><figure><img class="my-0 rounded-md" loading="lazy" decoding="async" fetchpriority="auto" alt="confusion-matrix" width="2400" height="1640" src="https://davethehuman.com/attachments/confusion-matrix_hu_51d2cad18105cc81.png" srcset="https://davethehuman.com/attachments/confusion-matrix_hu_51d2cad18105cc81.png 800w, https://davethehuman.com/attachments/confusion-matrix_hu_bd6c4f8161120a6d.png 1280w" sizes="(min-width: 768px) 50vw, 65vw" data-zoom-src="https://davethehuman.com/attachments/confusion-matrix.png"></figure> </p> <p>The main classification metrics are:</p> <ul> <li><a href="https://davethehuman.com/notes/accuracy/">accuracy</a></li> <li><a href="https://davethehuman.com/notes/precision/">precision</a></li> <li><a href="https://davethehuman.com/notes/recall/">recall</a></li> <li><a href="https://davethehuman.com/notes/fpr/">false positive rate</a></li> </ul> <p>Their usage depends on the classification task.</p>F1-scorehttps://davethehuman.com/notes/f1-score/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/f1-score/<p><em>F1-score</em> (also called <a href="https://davethehuman.com/coming-soon/" rel="nofollow">harmonic mean</a>) is a <a href="https://davethehuman.com/coming-soon/" rel="nofollow">metric</a> that takes into account both <a href="https://davethehuman.com/notes/precision/">precision</a> and <a href="https://davethehuman.com/notes/recall/">recall</a>: </p> $$ \mathrm{F1} = \frac{2*Precision*Recall}{Precision + Recall} $$<p>It is widely used in <a href="https://davethehuman.com/coming-soon/" rel="nofollow">NLP</a> and it balances precision and recall, handling <a href="https://davethehuman.com/coming-soon/" rel="nofollow">class imbalance</a> and highlighting poor performance on minority classes.</p>FNhttps://davethehuman.com/notes/fn/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/fn/<p>In <a href="https://davethehuman.com/coming-soon/" rel="nofollow">binary classification</a>, a <em>False Negative (FN)</em> is a misclassification of the model that assigns a “False” label to an observation whose label is actually “True”.</p> <p>Used in the <a href="https://davethehuman.com/notes/classification-metrics/">classification metrics</a>.</p>FPhttps://davethehuman.com/notes/fp/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/fp/<p>In <a href="https://davethehuman.com/coming-soon/" rel="nofollow">binary classification</a>, a <em>False Positive (FP)</em> is a misclassification of the model that assigns a “True” label to an observation whose actual label is “False”.</p> <p>Used in the <a href="https://davethehuman.com/notes/classification-metrics/">classification metrics</a>.</p>FPRhttps://davethehuman.com/notes/fpr/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/fpr/<p><em>False Positive Rate (FPR)</em> is the ratio of <strong>the negative observations that are classified as positives</strong> (how many <a href="https://davethehuman.com/notes/fp/">FP</a> over the sum of <a href="https://davethehuman.com/notes/tp/">FP</a> and <a href="https://davethehuman.com/notes/tn/">FN</a>): </p> $$ \mathrm{FPR} = \frac{FP}{FP + TN} $$<p>It is used in <a href="https://davethehuman.com/coming-soon/" rel="nofollow">ROC-AUC</a>.</p>precisionhttps://davethehuman.com/notes/precision/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/precision/<p><em>Precision</em> is the ratio of <strong>the observation classified as positives that are actually positives</strong>. In <a href="https://davethehuman.com/coming-soon/" rel="nofollow">binary classification</a> settings, it takes the number of <a href="https://davethehuman.com/notes/tp/">TP</a> and divides it by the sum of <a href="https://davethehuman.com/notes/tp/">TP</a> and <a href="https://davethehuman.com/notes/fp/">FP</a>: </p> $$ \mathrm{Precision} = \frac{TP}{TP + FP} $$<p>While <a href="https://davethehuman.com/notes/recall/">recall</a> uses in the denominator the total number of observations that are actually positives, precision uses the total number of observations that are <strong>classified as positives</strong>.</p> <p>This metric is useful in cases of <a href="https://davethehuman.com/coming-soon/" rel="nofollow">class imbalance</a> and when the false positive creates severe inconvenience, financial loss or user distrust; a typical task where precision is used as a <a href="https://davethehuman.com/coming-soon/" rel="nofollow">metric</a> is spam detection.</p>recallhttps://davethehuman.com/notes/recall/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/recall/<p><em>Recall</em> is the ratio of <strong>the positive observations that are actually classified as positives</strong>. In <a href="https://davethehuman.com/coming-soon/" rel="nofollow">binary classification</a> setting, it takes the number of <a href="https://davethehuman.com/notes/tp/">TP</a> and divides it by the sum of <a href="https://davethehuman.com/notes/tp/">TP</a> and <a href="https://davethehuman.com/notes/fn/">FN</a>: </p> $$ \mathrm{Recall} = \frac{TP}{TP + FN} $$<p>It is also called <a href="https://davethehuman.com/notes/sensitivity/">sensitivity</a> or <a href="https://davethehuman.com/notes/tpr/">true positive rate</a>. the correctly classified observations.</p> <p>While <a href="https://davethehuman.com/notes/precision/">precision</a> uses in the denominator the total number of observations classified as positives, recall uses the total number of observations that are <strong>actual positives</strong>.</p>sensitivityhttps://davethehuman.com/notes/sensitivity/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/sensitivity/<p>See <a href="https://davethehuman.com/notes/recall/">recall</a>.</p>TNhttps://davethehuman.com/notes/tn/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/tn/<p>In <a href="https://davethehuman.com/coming-soon/" rel="nofollow">binary classification</a>, a <em>True Negative (TN)</em> is a correct classification of the model that assigns a “False” label to an observation whose label is “False”.</p> <p>Used in the <a href="https://davethehuman.com/notes/classification-metrics/">classification metrics</a>.</p>TPhttps://davethehuman.com/notes/tp/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/tp/<p>In <a href="https://davethehuman.com/coming-soon/" rel="nofollow">binary classification</a>, a <em>True Positive (TP)</em> is a correct classification of the model that assigns a “True” label to an observation whose label is “True”.</p> <p>Used in the <a href="https://davethehuman.com/notes/classification-metrics/">classification metrics</a>.</p>TPRhttps://davethehuman.com/notes/tpr/Tue, 29 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/tpr/<p>See <a href="https://davethehuman.com/notes/recall/">recall</a>.</p>Logistic Regressionhttps://davethehuman.com/notes/logistic-regression/Sat, 26 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/logistic-regression/<h2 class="relative group">Formula <div id="formula" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#formula" aria-label="Anchor">#</a> </span> </h2> <p>Taking as a reference the <a href="https://davethehuman.com/notes/linear-regression/">Linear Regression</a>, in <em>Logistic Regression</em> the linear output is converted into a probability between 0 and 1 using the <a href="https://davethehuman.com/notes/sigmoid/">sigmoid</a> function:</p> $$\sigma(z) = \frac{1}{1 + e^{-z}}$$<p>With \(z = \beta_0 + X\beta\) as the linear score, the model’s predicted probability is \(\hat{p} = \sigma(z) = \sigma(\beta_0 + X\beta) = \frac{1}{1 + e^{-(\beta_0 + X\beta)}}\).</p>sigmoidhttps://davethehuman.com/notes/sigmoid/Sat, 26 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/sigmoid/<p> The <em>sigmoid</em> function is used in <a href="https://davethehuman.com/notes/logistic-regression/">Logistic Regression</a> to transform the linear output in a probability between 0 and 1.</p> <p>For a linear score $z = X\beta$, the sigmoid function is: </p> $$\sigma(z) = \frac{1}{1 + e^{-z}}$$<p> <figure><img class="my-0 rounded-md" loading="lazy" decoding="async" fetchpriority="auto" alt="sigmoid" width="1600" height="1100" src="https://davethehuman.com/attachments/sigmoid_hu_37d5cdef045797d8.png" srcset="https://davethehuman.com/attachments/sigmoid_hu_37d5cdef045797d8.png 800w, https://davethehuman.com/attachments/sigmoid_hu_60910846251427ee.png 1280w" sizes="(min-width: 768px) 50vw, 65vw" data-zoom-src="https://davethehuman.com/attachments/sigmoid.png"></figure> </p> <p>When negative values are also needed while keeping the sigmoid’s smooth, bounded output, <a href="https://davethehuman.com/notes/tanh/">tanh</a> is used.</p> <h2 class="relative group">Implementation <div id="implementation" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#implementation" aria-label="Anchor">#</a> </span> </h2> <p>A correct implementation of sigmoid has two branches: one for negative values and one for positive values.<br> The original formula at the top of the page is correct for every <a href="https://davethehuman.com/coming-soon/" rel="nofollow">real number</a>, but a computer has a largest representable floating-point value that can lead to overflow with exponential operation on large positive numbers.</p>intercepthttps://davethehuman.com/notes/intercept/Sun, 20 Sep 2026 00:00:00 +0000https://davethehuman.com/notes/intercept/<p>In <a href="https://davethehuman.com/notes/linear-regression/">Linear Regression</a>, the <em>intercept</em> is the expected value of the <a href="https://davethehuman.com/coming-soon/" rel="nofollow">dependent variable</a> $Y$ when all <a href="https://davethehuman.com/coming-soon/" rel="nofollow">independent variable</a>s are equal to zero. Graphically, it is represented by the intersection point between the regression line and the Y axis on a chart; its value shift the regression line up and down without affecting its tilt or slope.</p> <p><figure><img class="my-0 rounded-md" loading="lazy" decoding="async" fetchpriority="auto" alt="intercept" width="2000" height="1300" src="https://davethehuman.com/attachments/intercept_hu_426627f0b6687085.png" srcset="https://davethehuman.com/attachments/intercept_hu_426627f0b6687085.png 800w, https://davethehuman.com/attachments/intercept_hu_bf603842d8e23f87.png 1280w" sizes="(min-width: 768px) 50vw, 65vw" data-zoom-src="https://davethehuman.com/attachments/intercept.png"></figure> </p> <p>It can be interpreted as a <strong>baseline value</strong>, since it acts as the starting point or baseline prediction for the linear regression model before any independent variables effect is added.</p>