Author(s: Ibrahim Kovan Machine Learning. This article describes the subject of logistic regression. It is a linear model for classification.
This article explains the mathematical basis of the Logistic Regression algorithm, details how it works with the sklearn libraries, and how to implement it in Python with mathematical equations.
Multiclass classification of linear models with multiple classes is also explained. The Table of Contents is )—- Introduction Linear Classification Models without Sklearn Multiclass Classification. Photo by Roberto Sorin, Unsplash Introduction Linear model are both used to classify and regression.
Logistic Regression is the one used to perform binary classification. Binary classification refers to the fact that the dataset contains 2 outputs (classes). Logistic Regression, which is also the core part of Neural Networks, is another important component. This reduces the cost of error by updating initial values. The flowchart in Figure 1 illustrates how logistic regression is used to classify the data with its 4 classes and 2 features. Figure 1. Figure 1. The user selects the values to initialize these weights and biases. The weights are multiplied with the feature values, and then added to the total by adding the bias. The sum value is then applied to the activation function (figure 2). If it were a sigmoid, activation function(sum) becomes between 0 and 1, and error is calculated using this value. The calculated error value is used to calculate the bias and weight values. Click here for more information about gradient descent. The process can be repeated at the same rate as iterations. Let’s now implement these processes using codes on the breast-cancer dataset. Figure 2. Figure 2. IN[1]import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom sklearn.datasets import load_breast_cancerdata = load_breast_cancerx=data.datay=data.targetprint(“shape of data:”,x.shape)from sklearn.preprocessing import MinMaxScalerscaler=MinMaxScalerx_new=scaler.fit_transform(x)OUT[1]shape of data: (569, 30) IN[2]from sklearn.model_selection import train_test_splitx_train, x_test, y_train, y_test = train_test_split(x_new,y,test_size = 0. 15,random_state=2021)x_train = x_train.Tx_test = x_test.Ty_train = y_train.Ty_test = y_test.T The initial values are set as the weight is 0.1 and the initial bias is 1. The equation of the sigmoid function will be applied. IN[3]def weights_bias(shape): weights = np.full((shape,1),0.1) bias = 1 return weights,biasIN[4]def sigmoid(z): y_predict = 1/(1+ np.exp(-z)) return y_predict Forward propagation and backward propagation are designed as follows. IN[5]def forward_backward(w,b,x_train,y_train): z = np.dot(w.T,x_train) + b y_predict = sigmoid(z) derivative_weight = (np.dot(x_train,((y_predict-y_train).T)))/x_train.shape[1] derivative_bias = np.sum(y_predict-y_train)/x_train.shape[1] gradients = “derivative_weight”: derivative_weight, “derivative_bias”: derivative_bias return gradients The initial values have updated the rate of learning rate and the number of iteration times. IN[6]def update_parameters(w, b, x_train, y_train, learning_rate,iterations): index = [] for i in range(iterations): gradients = forward_backward(w,b,x_train,y_train) w = w – learning_rate gradients[“derivative_weight”] b = b – learning_rate gradients[“derivative_bias”] parameters = “weight”: w,”bias”: b return parameters, gradients The final step of the architecture is to predict current input and if sigmoid function(sum)<=0.5, it belongs to class 0, and if sigmoid function(sum)>0.5, it belongs to class 1. IN[7]def predict(w,b,x_test): z = sigmoid(np.dot(w.T,x_test)+b) y_prediction = np.zeros((1,x_test.shape[1])) for i in range(z.shape[1]): if z[0,i]<= 0.5: y_prediction[0,i] = 0 else: y_prediction[0,i] = 1 return y_prediction Now let’s combine all of them and test it with Learning_rate=0.1 and number of iterations=100: IN[8]def logistic_regression(x_train, y_train, x_test, y_test, learning_rate , iterations): shape = x_train.shape[0] w,b = weights_bias(shape) parameters, gradients = update_parameters(w, b, x_train, y_train, learning_rate,iterations) y_prediction_test = predict(parameters[“weight”],parameters[“bias”],x_test) print(“test accuracy: {}% “.format(100 – np.mean(np.abs(y_prediction_test – y_test)) 100)) logistic_regression(x_train, y_train, x_test, y_test,learning_rate = 0.1, iterations = 100)OUT[8]test accuracy: 91. 86046511627907% Initial values, learning rate, number of iterations are the hyperparameter of the project. You can try other configurations and achieve higher accuracy. Sklearn Library: Linear models for classification Let’s now work with the same dataset, using the sklearnlibrary. We can see the effect of changing C as a hyperparameter on model accuracy. IN[9]from sklearn.linear_model import LogisticRegressionc_list=[0.001,0.01,0.1,1,10]for i in c_list: lrc = LogisticRegression(C=i).fit(x_train.T,y_train.T) lrc_test=lrc.score(x_test.T,y_test.T) lrc_test=round(lrc_test*100,2) print(“C=”,i,”test acc: “, lrc_test,”%”)OUT[9]C= 0.001 test acc: 63. 95 %C= 0. 01 test acc: 77. 91 %C= 0.1 test acc: 93. 02 %C= 1 test acc: 95. 35 %C= 10 test acc: 98. 84 % As seen that, when the Control C value is increased, test accuracy is increased as well. Sklearn has many hyperparameters that can be used to perform logistic regression. All of them are available by clicking the link. Multiclass classification Logistic regression is used to classify binary data. Logistic Regression is used for binary classification. Multiclass classification is based on one vs. rest. By taking into account all classes that are against each other, you can create bias and coefficients for each class. It is then placed in the appropriate class during the predict phase. It is possible to implement the Sklearn library as follows: IN[10] from sklearn.datasets import load_digitsdigits = load_digitsx_digit=digits.datay_digit=digits.targetprint(“shape of data:”), x_digit.shape) from skalern.preprocessing import MinMaxScalerscaler=MinMaxScalerx_digit_new=scaler.fit_transform(x_digit_new,y_digit,test_size_digit_digit_new,y_digit). IN[10]from sklearn.datasets import load_digitsdigits = load_digitsx_digit=digits.datay_digit=digits.targetprint(“shape of data:”,x_digit.shape) from sklearn.preprocessing import MinMaxScalerscaler=MinMaxScalerx_digit_new=scaler.fit_transform(x_digit) from sklearn.model_selection import train_test_splitx_digit_train, x_digit_test, y_digit_train, y_digit_test = train_test_split(x_digit_new,y_digit,test_size = 0. 20,random_state=2021)OUT[10]shape of data: (1797, 64) IN[11]from sklearn.linear_model import LogisticRegressionmulticlass = LogisticRegression(multi_class=’multinomial’)multiclass.fit(x_digit_train,y_digit_train)multiclass_test=multiclass.score(x_digit_test,y_digit_test)multiclass_test=round(multiclass_test*100,2)print(“test acc: “, multiclass_test,”%”)OUT[11]test acc: 95. 28 % Back to the guideline click here. Machine Learning Guideline Linear Models of Classification and Logistic Regression with &without sklearn Library was first published on Medium. People are responding to the story by highlighting it. Published via
