## Publication Latest

# Linear Models for Classification, Logistic Regression, with&without sklearn library

#### Machine Learning

This article deductively breaks down the topic of logistic regression, which is linear models for classification. It explains how the Logistic Regression algorithm works mathematically, how it is implemented with the sklearn library, and finally how it is implemented in python with mathematical equations without the sklearn library. Furthermore, multiclass classification for linear models is explained.

`Table of Contents (TOC)---- Introduction---- Linear Models for Classification without Sklearn---- Linear Models for Classification with Sklearn-----Linear Models for Multiclass Classification`

### Introduction

Linear models are used for classification as well as regression. One of them, Logistic Regression, is used for binary classification as opposed to its name. Binary classification means that the dataset includes 2 outputs(classes). In addition, Logistic Regression is the fundamental part of Neural Networks. It works on minimizing the error(cost) in each iteration by updating the initial values set by the user. Figure 1 shows the flowchart of how the dataset with 4 features and 2 classes is classified with logistic regression.

1. In the training part of logistic regression, 1 weight is assigned for each feature and 1 bias value is added to the system.
2. These weights and bias values are initialized with values selected by the user.
3. Each weight is multiplied by the own feature values and added by adding the bias value.
4. The activation function (figure 2) which is chosen by the user is applied to the sum value.
5. When we assume that it was a sigmoid function, Activation Function(sum) will become between 0–1 and the error is calculated by this value.
6. According to the calculated error value, the weights and bias values are updated with gradient descent (click here) at the rate of learning rate.
7. This process is repeated as many as the number of iterations. Now, let’s implement the above-mentioned processes on the breast cancer dataset with codes.

### Linear Models for Classification without Sklearn library

Dataset is imported and feature values are normalized with MinMaxScaler and the data is separated with train_test_split.

`INimport numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom sklearn.datasets import load_breast_cancerdata = load_breast_cancer()x=data.datay=data.targetprint("shape of data:",x.shape)from sklearn.preprocessing import MinMaxScalerscaler=MinMaxScaler()x_new=scaler.fit_transform(x)OUTshape of data: (569, 30)`
`INfrom sklearn.model_selection import train_test_splitx_train, x_test, y_train, y_test = train_test_split(x_new,y,test_size = 0.15,random_state=2021)x_train = x_train.Tx_test = x_test.Ty_train = y_train.Ty_test = y_test.T`

The initial values are set as the weight is 0.1 and the initial bias is 1. Equation of sigmoid function is applied.

`INdef weights_bias(shape):    weights = np.full((shape,1),0.1)    bias = 1    return weights,biasINdef sigmoid(z):      y_predict = 1/(1+ np.exp(-z))    return y_predict`

Forward propagation and backward propagation are designed as follows.

`INdef forward_backward(w,b,x_train,y_train):    z = np.dot(w.T,x_train) + b    y_predict = sigmoid(z)    derivative_weight = (np.dot(x_train,((y_predict-y_train).T)))/x_train.shape    derivative_bias = np.sum(y_predict-y_train)/x_train.shape                     gradients = {"derivative_weight": derivative_weight, "derivative_bias": derivative_bias}    return gradients`

The initial values have updated the rate of learning rate and the number of iteration times.

`INdef update_parameters(w, b, x_train, y_train, learning_rate,iterations):    index = []    for i in range(iterations):        gradients = forward_backward(w,b,x_train,y_train)        w = w - learning_rate * gradients["derivative_weight"]        b = b - learning_rate * gradients["derivative_bias"]    parameters = {"weight": w,"bias": b}    return parameters, gradients`

The final step of the architecture is to predict current input and if sigmoid function(sum)≤0.5, it belongs to class 0, and if sigmoid function(sum)>0.5, it belongs to class 1.

`INdef predict(w,b,x_test):    z = sigmoid(np.dot(w.T,x_test)+b)    y_prediction = np.zeros((1,x_test.shape))    for i in range(z.shape):        if z[0,i]<= 0.5:            y_prediction[0,i] = 0        else:            y_prediction[0,i] = 1    return y_prediction`

Now let’s combine all of them and test it with Learning_rate=0.1 and number of iterations=100:

`INdef logistic_regression(x_train, y_train, x_test, y_test, learning_rate ,  iterations):    shape =  x_train.shape    w,b = weights_bias(shape)    parameters, gradients = update_parameters(w, b, x_train, y_train, learning_rate,iterations)    y_prediction_test = predict(parameters["weight"],parameters["bias"],x_test)    print("test accuracy: {}% ".format(100 - np.mean(np.abs(y_prediction_test - y_test)) * 100))    logistic_regression(x_train, y_train, x_test, y_test,learning_rate = 0.1, iterations = 100)OUTtest accuracy: 91.86046511627907%`

Initial values, learning rate, number of iterations are the hyperparameter of the project. Other configurations can be tried and higher accuracy may be obtained.

### Linear Models for Classification with Sklearn library

Now let’s work on the same dataset using the sklearn library. When we change the hyperparameter C value, its effect on model accuracy is seen.

`INfrom sklearn.linear_model import LogisticRegressionc_list=[0.001,0.01,0.1,1,10]for i in c_list:    lrc = LogisticRegression(C=i).fit(x_train.T,y_train.T)    lrc_test=lrc.score(x_test.T,y_test.T)    lrc_test=round(lrc_test*100,2)    print("C=",i,"test acc: ", lrc_test,"%")OUTC= 0.001 test acc:  63.95 %C= 0.01 test acc:  77.91 %C= 0.1 test acc:  93.02 %C= 1 test acc:  95.35 %C= 10 test acc:  98.84 %`

As seen that, when the Control C value is increased, test accuracy is increased as well. There are lots of hyperparameters for the logistic regression in Sklearn and all can be accessed by clicking this link.

### Multiclass Classification

It was mentioned that Logistic Regression is used for binary classification. So how can we use logistic regression if there are more than 2 classes? Here, we come across multiclass classification based on the one vs rest principle. For each class, coefficients and bias are produced by taking all other classes against it. In the predict phase, it is placed in the most appropriate class. With the Sklearn library, we can easily implement it as seen below.

`INfrom sklearn.datasets import load_digitsdigits = load_digits()x_digit=digits.datay_digit=digits.targetprint("shape of data:",x_digit.shape)`
`from sklearn.preprocessing import MinMaxScalerscaler=MinMaxScaler()x_digit_new=scaler.fit_transform(x_digit)`
`from sklearn.model_selection import train_test_splitx_digit_train, x_digit_test, y_digit_train, y_digit_test = train_test_split(x_digit_new,y_digit,test_size = 0.20,random_state=2021)OUTshape of data: (1797, 64)`
`INfrom sklearn.linear_model import LogisticRegressionmulticlass = LogisticRegression(multi_class='multinomial')multiclass.fit(x_digit_train,y_digit_train)multiclass_test=multiclass.score(x_digit_test,y_digit_test)multiclass_test=round(multiclass_test*100,2)print("test acc: ", multiclass_test,"%")OUTtest acc:  95.28 %`

#### Back to the guideline click here.

Machine Learning Guideline Linear Models for Classification, Logistic Regression, with&without sklearn library was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Published via Towards AI