Pipelines are a simple way to keep your data preprocessing and modeling code organized. Specifically, a pipeline bundles preprocessing and modeling steps so you can use the whole bundle as if it were a single step.
Many Data Scientists hack together models without Pipelines… (Kaggle) Kaggle also says Pipelines have some important benefits, such as: Cleaner Code: Accounting for data at each step of preprocessing can get messy. With a Pipeline, you won’t need to manually keep track of your training and validation data at each step. Fewer Bugs: There are fewer opportunities to misapply a step or forget a preprocessing step.
Easier to Productionize: It can be surprisingly hard to transition a model from a prototype to something deployable at scale, but Pipelines can help.
More Options for Model Validation:
We can easily apply cross-validation and other techniques to our Pipelines. So, a Pipeline is a convenient process of designing our data preprocessing and Machine Learning flow. There are certain steps that we must do before the actual ML begins.
These steps are called data-preprocessing and/or feature engineering. Some Pipeline steps include: Converting categorical values to nominal & numerical values Normalizing the range of values per dimension One-Hot encoding categorical values and… Modeling… where we train our ML algorithm.
The overall idea of Pipelines is that we can fuse our complete data processing flow into one single pipeline, and that single pipeline we can further use downstream. Some Pipeline Methods: Image from IBM scalable-machine-learning-for-big-data-with-apache-spark Pipeline as a Machine Learning Algorithm has the following methods…
Fit: Fit basically starts the training Score: Score gives back the predicted value.
Evaluate: Evaluates the model performance on the validation data. One extra advantage of Pipelines is that we can cross-validate, meaning, try out many, many parameters using the very same Pipeline. And this really accelerates the optimization of the algorithm.
So, in summary, pipelines are facilitating our day to day work in machine learning as we can draw from pre-defined data processing steps, we make sure everything is aligned, and we can switch and swap our algorithms as needed. While there are plenty of materials covering Pipelines for machine learning, today we shall focus on Pipelines for machine learning on Big-data, using Apache SparkML.
1. Intro
For this exercise, we shall use the HMP dataset. It’s basically accelerometer recordings from an accelerometer sensor attached to the human body. The data records sensors from humans when performing activities like brush-teeth, comb-hair, eat-soup, and so on. So, we shall preprocess this dataset for Machine Learning tasks. First, manually, then we shall build a SparkML Pipeline to preprocess the dataset automatically for us. This Pipeline can be applied to future datasets. I use Colab for my data exploration. If you need help starting Pyspark in Colab, see this link. So first, we set up Pyspark in Colab…
2. Data Extraction
Note that the data is in a parquet file format. Parquet uses compression and column store, which maps the data layout to the Apache Spark Tungsten memory layout.
So we can see there are different folders in this HMP_dataset. Folders representing different activities such as Brush_teeth, Drink_glass, Getup_bed, Pour_water, Use_telephone. Looking at the text files in the Brush_teeth folder, for example, we can see the accelerometer data represented in three numeric columns we may call X, Y, Z. !head HMP_Dataset/Brush_teeth/Accelerometer-2011-04-11-13-28-18-brush_teeth-f1.txt >>22 49 35 22 49 35 22 52 35 22 52 35 21 52 34
Let’s recursively traverse through these folders in HMP_Dataset and create an Apache spark DataFrame from these text files. Then we just union all DataFrames <df.union(df2)> into one overall DataFrame containing all the data. First, Let’s define the schema of the data frame.
Now let’s traverse through the data using the OS library.
Let’s remove non-action folders from the file list. These are typically folders without an underscore in their names.
Okay, we have all the folders containing the data in one array. Now we can iterate over this array.
So from the Git gist above, first, we define an empty DataFrame and import tqdm for progress-bars. Next, we import the lit library that helps us write String literal columns to an Apache Spark DataFrame.
At this point, all we do is go through each file using the OS library, add the three numeric columns to the X, Y, Z schema we defined earlier, add these to the DataFrame and add two String Literal columns, one for the class of the accelerometer reading and the other for the source file for the reading. Let’s see the schema of the DataFrame… df.printSchema >>root |– x: integer (nullable = true) |– y: integer (nullable = true) |– z: integer (nullable = true) |– class: string (nullable = false) |– source: string (nullable = false)
Let’s see the first 10 rows of the DataFrame. This takes a little while to run, cos as we know, DataFrames in Apache Spark are always lazy…
3. Data Transformation
Now we need to transform the data and create an integer representation of the class column as ML algorithms cannot cope with a String. So we will transform the class into a number of integers using the StringIndexer module.
https://medium.com/media/6dffdbb27a4d9f318b33083af397c6ee/href The StringIndexer is an estimator having both fit and transform methods. So we create a StringIndexer object (indexer), pass the ‘class’ column as inputCol, and ‘classIndex’ as outputCol. Then we fit the DataFrame to the indexer, and transform the DataFrame.
This creates a brand new DataFrame (indexed), which we can see above, containing the classIndex additional column.
4. One-Hot Encoding
With the class index column, we can now do one-hot-encoding in Pyspark… https://medium.com/media/b6c4e390d4c6b66b88bc8026c27a26e7/href Unlike the StringIndexer, OneHotEncoder is a pure transformer, having only the transform method. It uses the same syntax of ‘inputCol’ and ‘outputCol’ we saw in StringIndexer. We pass ‘classIndex’ and ‘categoryVec’ values, respectively.
OneHotEncoder also creates a brand new DataFrame (encoded), with a ‘category_vec’ column added to the previous DataFrame(indexed). One more thing to note is that calling encoded.show(10, False) in the Git gist above, ensures that 10 rows are displayed and False ensures that each column element is fully expanded, cos normally SparkML compresses column cells.
Finally, OneHotEncoder in SparkML doesn’t return several columns containing only zeros and one at a point where a value existed, as we all know… Rather it returns a sparse-vector as seen in the categoryVec column. Thus, for the ‘Drink_glass’ class above, SparkML returns a sparse-vector that basically says there are 13 elements, and at position 2, the class value exists(1.0). 5. VectorAssembler.
The next thing we need to do is to transform our numerical columns X, Y, Z into vectors because sparkML can only work on vector objects.
So let’s import Vectors and VectorAssembler libraries.
The VectorAssembler object gets initialized with the same syntax we used in StringIndexer and OneHotEncoder. We pass list [‘x’, ‘y’, ‘z’] to inputCols, and we specify outputCol = ‘features’. This is also a pure transformer like OneHotEncoder. So we transform the DataFrame from the last step (encoded) into a new DataFrame (features_vectorized) with the features column added. The consistency of SparkML syntax is quite impressive, it reduces the learning curve for Big-Data Enthusiasts…
