Author(s: Ibrahim Kovan) Data Analysis. An overview of data processing: Converting variables — Column Transformers. OneHotEncoding. Label Binarizer. Standardization. Normalization. Robust scaling. Every column is unique and should have its own methods. A conversion of categorical data to numerical data can yield better results. This distinction is important because it allows you to select the best data processing method for your model. To make it easier to allow algorithms to understand the data, this article offers different perspectives. Python applications make all studies more understandable. Table of Contents (TOC). 1. Categorical Variables Column Transformers. OneHotEncoding. Dummy Coding1.2. 1. Label Encoder Ordinal Encoder1.4. Label Binarizer2. Numeric Variables2.1. Standardization2.2. Normalization2.3. L2 Regularization2.4. Robust Scaler Image by Gaelle Marcel at Unsplash 1. Categorical Variables These columns are available for processing by the algorithm. They can either be displayed continuously (continuous features) or presented with no variation at all, such as when looking at the iris dataset. A flower can be one of three types: Iris Setosa or Iris Vericolor. The value of the data cannot lie in any of the three types. These datasets are known as categorical features. It is important to convert categorical data to numeric data these columns to be compatible with the algorithm. This is how you can implement these methods in Python. 1.1. Column Transformers and One Hot Encoding and Dummy Coding are different processes. Multiple preprocessing operations on columns can be done simultaneously with Column Transformers. OneHotEncoding converts each categorical type into a column, and turns the dataset numerical. It adds two new columns to the dataset. For female data, the male option assigns 0, while the female option assigns 1. import numpy as npimport pandas as pdimport matplotlib.pyplot as plt IN[1]data=pd.read_csv(‘toy_dataset.csv’)data Figure 1. OUT[1], Image by author from sklearn.compose import make_column_transformerfrom sklearn.preprocessing import OneHotEncoderfrom sklearn.preprocessing import MinMaxScaler IN[2]ohe=OneHotEncoder(sparse=False)scaler = MinMaxScalerct=make_column_transformer((ohe,[‘City’,’Gender’,’Illness’]), (scaler [‘Income’]),remainder=’passthrough’)data_ct=ct.fit_transform(data)data_ct Figure 2. OUT[2], Image by author IN[3]ohe_True=OneHotEncoder(sparse=True)ohe_with_sparse=ohe_True.fit_transform(data[‘Gender’].values.reshape(-1,1)).toarrayohe_with_sparseOUT[3]array([[0., 1.], [0., 1.], [0., 1.], …, [0., 1.], [0., 1.], [1., 0.]]) The same procedure is done with pandas.get_dummies in Pandas. OneHotEncoder, however, is more suitable for machine learning research. OneHotEncoder comes from the transformer class, and can be applied using the fit command. It can then transform data according to its categories. The pipeline makes it easy to use. IN[4]dum=pd.get_dummies(data,columns=[‘Gender’,’City’,’Illness’],prefix=”,prefix_sep=”) Figure 3. OUT [4], Photo by author. Now, let’s take a look at the OneHotEncoder hyperparameters. Sparse: Because the dataset contains a lot “0” values, sparse can be defined as 0. The dataset is readable if Sparse=False has been set, and.toarray added. Handle_unknown : If this parameter is set as ‘ignore’, and an unidentified category is encountered during transformation, then the one-hot encoded columns resulting for this feature will all be zeros. 1.2. 1. A column with red, white, and blue data might be IN[5]from sklearn.preprocessing import LabelEncoderle=LabelEncoderencoded_data_city=le.fit_transform(data [‘City’]print(“Cities”,np.unique(encoded_data_city)encoded_data_gender=le.fit_transformdata.plain_transformdata.plain_gender=le.la_data_data_data_data_gender_data_data_data_gender_data_gender_data_gender_gender_data_gender_data_data_gender_data_gender_gender_gender_data_gender_data_data_gender_data_data_data_data_data_data_data_data_data_data_data_data_encoded_data_gender_data_gender_gender).OUT IN[5]from sklearn.preprocessing import LabelEncoderle=LabelEncoderencoded_data_city=le.fit_transform(data[‘City’])print(“Cities”,np.unique(encoded_data_city))encoded_data_gender=le.fit_transform(data[‘Gender’])print(“Gender”,np.unique(encoded_data_gender))OUT[5]Cities [0 1 2 3 4 5 6 7]Gender [0 1] 1.3. The Ordinal Encoder Ordinal Encoder is the same as Label Encoder. The difference between Ordinal Encoder is used for features although Label Encoder is used Labels(targets) so that Ordinal Encoder fit the data with (number_of_samples, number_of_features) while Label Encoder (number_of_samples) IN[6]from sklearn.preprocessing import OrdinalEncoderoe=OrdinalEncoder(categories=[dictionary])encoded_data_city=oe.fit_transform(data[[‘City’]])print(“Cities”, np.unique(encoded_data_city))OUT[6]Cities [0. 1. 2. 3. 4. 5. 6. 7.] 1.4. OneHotEncoding Label Binarizer is the same as OneHotEncoding. OHE can be used to store multicolumn data, while LabelBinarizer only works for one column. This is only used in LabelEncoder for one column. IN[7]from sklearn import preprocessinglb = preprocessing.LabelBinarizerlabel_city=lb.fit_transform(data[‘City’])label_cityOUT[7]array([[0, 0, 1, …, 0, 0, 0], [0, 0, 1, …, 0, 0, 0], [0, 0, 1, …, 0, 0, 0], …, [1, 0, 0, …, 0, 0, 0], [1, 0, 0, …, 0, 0, 0], [1, 0, 0, …, 0, 0, 0]]) 2. Numeric Variables Rescale Numeric Variables are very important to the algorithm’s accuracy. All numeric data can be sorted within a specified range using Rescale. These methods are now available. 2.1. Standardization. The column’s average value is given 0; all numeric values surrounding it are given variance 1. Figure 4. Standardization, source IN[8]from sklearn.preprocessing import StandardScalerscaler = StandardScalerdata_standardscaler=scaler.fit_transform(data[[‘Income’]])data_standardscalerOUT[8]array([[-2.03629388], [-1.84753398], [-1.55144865], …, [ 0.82563756], [ 0.82535744], [-0.16013972]]) 2.2. Normalization Numeric data are divided between 0 to 1, with the minimum being 0. and the maximum being 1. Figure 5. Normalization, source IN[9]from sklearn.preprocessing import MinMaxScalerscaler = MinMaxScalerdata_minmaxscaler=scaler.fit_transform(data[[‘Income’]])data_minmaxscalerOUT[9}array([[0.23070001], [0.25722818], [0.29883978], …, [0.6329136 ], [0.63287423], [0.49437324]]) 2.3. Normalization of L2 L2 is also called the Euclidian norm. The data are placed between 1 and 1. Figure 6. Figure 6. The Robust Scaler Median Value and Inter Quantile Range (IQR) are calculated. new_x(i) = [x(i) — median_value]/IQR IN[10]from sklearn.preprocessing import RobustScalerscaler = RobustScalerdata_robust=scaler.fit_transform(data[[‘Income’]])data_robust OUT[10]array([[-2.2530733 ], [-2.05363353], [-1.74079594], …, [ 0.77078379], [ 0.77048782], [-0.27076793]]) Back to the guideline click here. Machine Learning Guideline: An Overview of Data Processing — Converting Variables — Column Transformers and OneHotEncoding… originally appeared in . Medium has people continuing to discuss the story by responding. Published via
