What is Logistic Regression (Logistic Regression), an article to read and understand

Definition of logistic regression

Logistic Regression is a statistical learning method used to solve binary classification problems, where the core objective is to predict the probability of a sample belonging to a specific category based on the input features. The model does this by linearly combining the eigenvalues and mapping the linear output to a probability value between 0 and 1 using an S-shaped function. Logistic regression specializes in dealing with discrete response variables to avoid oversensitivity to outliers. Model training uses maximum likelihood estimation to find the optimal parameters to maximize the probability of the observations. The probabilistic output can be interpreted as the chance of an event occurring, expressing the degree of influence of the feature on the outcome through the odds ratio. Logistic regression can be extended to multi-categorization problems to form multinomial logistic regression. The model assumes linear decision boundaries, but nonlinear relationships can be handled through feature engineering. Key advantages include simplicity of the model, computational efficiency, and ease of interpretation of the results, making it suitable for application scenarios where the importance of features needs to be understood.

Origins of logistic regression

• The Roots of StatisticsThe concept of logistic regression originated in the 19th century in demographic studies, when the Belgian mathematician Werster proposed a logistic function to describe the pattern of population growth, and in the mid-20th century, the statistician Berkson introduced it into biometric experiments to analyze the dose-response relationship and to establish a "logistic model".
• Psychometric Advancement: In the 1950s, psychologist Loos developed the choice model, extending logistic regression to multi-category choice problems. Scholars such as Cox refined the theoretical framework, making logistic regression a standard tool for categorical data analysis.
• Machine Learning Adoption: In the 1980s, with the development of pattern recognition, logistic regression was redefined as a classification algorithm. The generalized linear model in statistical learning theory provides a rigorous mathematical foundation to clarify its relevance to linear regression.
• Increased computing power: In the 1990s, advances in computer technology made maximum likelihood estimation more feasible, and logistic regression began to be applied to large-scale data sets. The integration of logistic regression into statistical software packages promoted its popularization.
• The status of modern data science: In the era of big data in the 21st century, logistic regression maintains an important position as a benchmark reference for complex models. Its interpretability advantage is highly favored in finance, healthcare and other regulatory stringent fields.

The core principle of logistic regression

• probabilistic mapping mechanism: Logistic regression centers on transforming linear predictive values into probabilities, using an S-shaped function as the connecting function. The mathematical form of this function is 1/(1+e^(-z)), with z being a linear combination of features. This function has the property of being smooth and monotonic, ensuring that the probability values are reasonable and derivable.
• Decision boundary formation: The model divides the categories by setting a probability threshold (usually 0.5), which corresponds to a linear decision boundary. In the feature space, the decision boundary is represented as a hyperplane for separating samples of different categories. The location of the boundary is determined by the model parameters, which are learned from the training data.
• odds ratio interpretation: Logistic regression parameters correspond to changes in odds ratios, i.e., multiplicative changes in odds ratios due to changes in feature units. An odds ratio greater than 1 indicates a positive correlation, while less than 1 indicates a negative correlation, providing an intuitive measure of feature influence.
• Maximum likelihood estimation optimization: The training objective is to maximize the likelihood function of the observed data, which is equivalent to minimizing the cross-entropy loss. Optimization algorithms such as gradient descent iteratively update the parameters and eventually converge to the optimal solution. The concavity of the likelihood function guarantees the uniqueness of the solution.
• Linear assumptions and extensions: The underlying logistic regression assumes that features are linearly correlated with the fractional logarithm, but simple nonlinear relationships can be handled by adding interaction terms, polynomial features. Kernel methods or neural networks can further extend their processing power.

Mathematical modeling of logistic regression

• S-shaped function action: The core of the mathematical model is the S-shaped function that maps the linear score z = β₀ + β₁x₁ + ... + βₙxₙ maps to P(y=1|x)=1/(1+e^(-z)). This functional derivative has the elegant mathematical form P(1-P), which facilitates gradient computation.
• Loss function design: A logarithmic loss function is used, formulated as -Σ[yᵢlog(pᵢ)+(1-yᵢ)log(1-pᵢ)]. The convexity of the loss function ensures the stability of the optimization process, with a reasonable penalty for misclassified probability estimates.
• parameter estimation equation: Maximum likelihood estimation derives a set of nonlinear equations for solving the parameter β. These equations do not have analytical solutions and need to be solved iteratively using numerical methods such as the Newton-Raphson method or gradient descent.
• regularization: To prevent overfitting, the loss function often incorporates regularization terms, such as L1 or L2 penalty terms.L1 regularization can generate sparse solutions for automatic feature selection; L2 regularization improves the model generalization ability by shrinking the parameters.
• Multi-category extensions: Multinomial logistic regression uses a flexible maximum function to transform multiple linear outputs into probability distributions. The function is normalized by an exponential score to ensure that all category probabilities sum to 1.

Application Scenarios of Logistic Regression

• Medical Diagnostic Forecasting: Logistic regression is widely used for disease risk prediction, such as estimating the probability of heart disease based on characteristics such as age and blood pressure. The model output aids physicians in clinical decision making and helps balance the sensitivity and specificity of diagnosis.
• Financial Credit Scoring: Banks utilize logistic regression to construct credit scorecards to assess the probability of customer default. The model takes into account income, historical credit and other characteristics, and the results are used in the loan approval process to effectively reduce the risk of bad debt.
• Marketing Response: Companies use logistic regression to predict the probability of customer response to promotions and optimize the allocation of marketing resources. Model inputs include demographic data, purchase history, and other information, which helps to improve marketing conversion rates.
• natural language processing (NLP): In text classification tasks such as sentiment analysis, logistic regression processes bag-of-words features to determine text sentiment polarity. This method is simple and efficient, suitable for real-time application scenarios that require fast response.
• Image Recognition Aid: In computer vision, logistic regression is used as a classification layer in combination with a feature extractor to handle simple image classification tasks. For example, it performs well in handwritten digit recognition benchmarks.

Advantages of logistic regression

• High computational efficiency: The training and prediction process of logistic regression has low computational complexity and is suitable for handling large-scale data or real-time system requirements. The optimization process converges faster and requires relatively less computational resources.
• Probabilistic output is useful: The model provides probabilistic estimates rather than simple classification results, allowing flexibility in adjusting decision thresholds according to actual needs. The probabilistic output supports uncertainty quantification for risk ranking scenarios.
• Highly interpretable: The model parameters directly correspond to feature importance and the concept of odds ratio is easy to understand at the business level. This transparency satisfies regulatory compliance requirements in finance, healthcare and other fields.
• Good robustness: The model is tolerant to noise and irrelevant features, and especially performs more consistently with the addition of regularization. The probabilistic output smoothing property avoids producing extreme predictions.
• Easy to implement and commission: The algorithm structure is simple, and the implementation code is readily available in various programming languages. The debugging process is intuitive, and feature effects can be visualized.

Limitations of logistic regression

• linear boundary constraint: Basic logistic regression can only learn linear decision boundaries and cannot handle complex nonlinear patterns. Need to resort to feature engineering or kernel tricks to increase model complexity.
• feature-related sensitivity: Highly correlated features can lead to unstable parameter estimates and increase variance. Although this can be mitigated by preprocessing methods such as principal component analysis, some interpretability is lost.
• Sample imbalance effects: When the distribution of categories in the data is uneven, the model is biased towards the majority category. A resampling strategy or loss function weighting is needed to rebalance the category impact.
• Outlier vulnerability: Although more robust than linear regression, extreme outliers can still distort probability estimates. This needs to be coupled with outlier detection or the use of a robust loss function.
• Independence assumption requirements: Logistic regression assumes that features are independent of each other, an assumption that is often violated by realistic data. Ignoring the dependency structure between features may degrade model performance.

Training process for logistic regression

• Data preprocessing steps: Preparatory work such as data cleaning, feature normalization, and missing value processing is required before training begins. Categorical variables need to be encoded in numerical form, e.g., by using solo thermal encoding.
• Parameter initialization: The model weights are usually initialized either randomly or with zero values, and different initialization methods can affect the convergence speed. Careful choices are needed to avoid the problem of vanishing or exploding gradients.
• Gradient descent iteration: Minimize the loss function using an optimization algorithm and update the model parameters by calculating the gradient. The setting of the learning rate is crucial, too large leads to oscillations, too small leads to slow convergence.
• Convergence judgment criteria: The training process continues until the loss change is less than a set threshold or the maximum number of iterations is reached. Overfitting can be prevented by using the early stopping method, which is realized by validation set performance monitoring.
• hyperparameter tuning: Key hyperparameters including learning rate, regularization strength, etc. are selected by cross-validation methods. Grid search or random search helps to find the optimal parameter combination.

Explanation of the output of logistic regression

• Probability threshold selection: The default 0.5 threshold can be adjusted according to business needs; increasing the threshold improves the precision rate, while decreasing the threshold is conducive to increasing the recall rate. Subject's work characteristic curve assists the threshold selection process.
• Feature Importance Assessment: The size of the absolute value of the parameter reflects the influence of the feature, and the positive and negative signs indicate the direction of influence. After normalization of the features, the parameters allow cross-feature comparisons.
• Confidence interval construction: Parameter estimates are accompanied by confidence intervals that reflect the uncertainty of the estimates. When the confidence interval does not contain zero, it indicates that the feature is statistically significant.
• Model calibration check: Probability outputs need to be calibrated to ensure that predicted probabilities match actual frequencies. The degree of calibration is assessed through calibration curves or Breyer scores.
• Business Insight Transformation: Translate odds ratios into business terms, e.g., "Each year of age increases the odds of default by 10%." Enhance decision support through storytelling explanations.

Comparison of logistic regression with other models

• Comparison with linear regression: Logistic regression deals with classification problems and linear regression deals with regression problems; logistic regression outputs probabilities and linear regression outputs continuous values; logistic regression uses maximum likelihood estimation and linear regression uses least squares.
• Comparison with Decision Trees: logistic regression provides smooth probability output, decision trees produce hard segmentation results; logistic regression is a global model, decision trees are local models; logistic regression requires feature scaling, decision trees are not sensitive to this.
• Comparison with Support Vector Machines: logistic regression outputs probability values, support vector machines output boundary distances; logistic regression loss functions are everywhere derivable, support vector machines use hinge loss; logistic regression is easier to extend to multiclassification problems.
• Comparison with Neural NetworksLogistic regression is a single-layer structure, while neural networks have a multi-layer structure; logistic regression is highly interpretable, while neural networks are more difficult to interpret; logistic regression is fast to train, while neural networks require a large amount of data support.
• Comparison with plain Bayes: logistic regression is a discriminative model and plain Bayes is a generative model; logistic regression estimates conditional probabilities and plain Bayes estimates joint probabilities; logistic regression has no feature independence assumption requirement.