Author(s: Abid Ali Awan A new way to build machine learning pipes Designing your very first machine-learning pipeline using just a few lines of code with Orchest. This course will teach you how to prepare the data and train the machine-learning model. Elements by Vecteezy In this article, we will go through all steps required to build a ML (Machine Learning) pipeline. We will be using Kaggle binary classification data COVID-19/SARS B-cell Epitope to analyze, preprocess, train and evaluate our model. As I have previously published a Deepnote article explaining how these models function, we won’t go into detail about code. To classify peptides, we will use SARS-CoV datasets and B-cell datasets. Antibodies with inducing characteristics are classified as positive (1) while those without are considered negative (0). The research paper provides more information about the data. Orchest Orchest can be used to build data pipelines without the need for DAGs or frameworks. It is easy to use and allows you to code Python, R and Julia with the favorite data scientist tool Jupyter notebook. These are steps in Orchest’s pipeline. These executable files are executed within an isolated environment. The connections determine the flow of data. You can monitor your progress and schedule to run the pipeline. This will give you a complete report. Author ML Pipeline Image by Author Scheduling Pipeline Similar to Airflow, scheduling your pipeline can be set up for specific minutes and hours. You don’t have to program or monitor your pipeline. Overview (orchest.readthedocs.io) Installation Installation for the local server is easy for Linux users, but for windows, you can get a similar experience by installing additional applications. Windows Verify that you are using the latest Docker Engine version. Run docker version to verify. WSL 2 must be enabled in Docker. Ubuntu 20. 04 LTS for Windows. The script below can be run within Ubuntu. Linux For Linux you only need to install the docker engine. Then, run the below script to download all necessary dependencies. git clone https://github.com/orchest/orchest.git && cd orchest./orchest install# Verify the installation../orchest version –ext# Start Orchest../orchest start First Project It’s time for us to start our local server. The script below will allow us to run Linux on Windows. Check that Docker Engine works properly. Once the script is executed successfully, you’ll receive a URL that you can paste into your browser. Image by Auhor On the landing page will see this amazing user interface. Click on the Create Project button to create your new project. Image by Author A project contains many pipelines, so now it’s time to create our Vaccine ML pipeline. A pipeline is a file that adds vaccine.orchest files to your directory. This contains metadata for each step. Image by Author For code, we will be using our previous project and focus on building an effective pipeline. Pipeline Machine Learning Pipelines are executable codes that can run many tasks, including preparing models and training them on data Azure Machine Learning. Below is a diagram of the machine learning model used in all machine learning projects. The data flow between one task and another is represented by the arrows. Microsoft Creating Steps Image Credit Click on the New Step button to first create the step. You will need to create the step. If you don’t already have a file in python or.ipynb, click on the New Step button. Image by Author Voila we have successfully created our first step and now we need to create a few more and try to connect the nodes. Image by Author We have added EDA (Exploratory Data Analysis) and Preprocessing step. These were then joined with the Load Data Step so each step can access extracted data. These steps will be coded in detail later. Image by Author To code steps, click on the button Edit in JupyterLab, which will take us to the Jupyter notebook where you can start coding. Image by Author To run all the steps, select all by using a mouse and then click on the blue button on the bottom left called Run Selected Steps. You will see all the steps running one by one. You can either click on the steps to view the logs or go straight to the notebook and see the progress. Orchest/orchest Output Let’s connect nodes and make some code changes. We need to import the orchest data via pandas read_csv Concat.bcell and sars.dataframe. Orchest.output is used to produce the data needed for the next step. Orchest output can take single or multiple variables, and creates dataflow that we can then use for the next steps. The dataflow variable called “data” stores bcell and covid as well as sars and bcell_sars. Import orchest import Pandas as pd # Convert data to a DataFrame. INPUT_DIR = “Data” bcell = pd.read_csv(f”INPUT_DIR/input_bcell.csv”) covid = pd.read_csv(f”INPUT_DIR/input_covid.csv”) sars = pd.read_csv(f”INPUT_DIR/input_sars.csv”) bcell_sars = pd.concat([bcell, sars], axis=0, ignore_index=True) # Output the Vaccine data. print(“Outputting converted Vaccine data…”) orchest.output((bcell, covid, sars, bcell_sars), name=”data”) print(bcell_sars.shape) print(“Success!”) Outputting converted Vaccine data…(14907, 14)Success! We now move on to the input. This takes the four variables, and uses them to analyze and explore data. To demonstrate the data flow between nodes, I added a few Jupyter notebook cell cells. Deepnote. Let’s import the required libraries, including Orchest. https://medium.com/media/5c4cedb89111a092bcbfb023f9f2ce7b/href Use orchest.get_inputs function to create an object and then add the name (‘data’) of the data pipeline variable to extract variables from the previous step. https://medium.com/media/861539f16ce5bc476527e22db25d564c/href As we can see we have successfully loaded the data from the previous task. https://medium.com/media/4ab2e1a492e711b36e0fecbc60cd528e/href Using PCA from sklearn to reduce dimensionality to 2 and use scatter plots to visualize the target distribution. https://medium.com/media/9c01377e27a39548d3814eee732521bb/href Input and Output Let’s use both input and output functions to extract training data and then use it to train our Random Forest Classifier. We will then export the data to be evaluated after training. https://medium.com/media/cfa124e48061ae1c0f75e220a5de27ef/href Input train test split from Preprocessed step using ‘training_data’ as name of data flow. https://medium.com/media/0e57044020113f1b87e6d6281825d821/href Using 400 estimators and fitting our train data set. Our AUC score for a model that does not require hyperparameter tuning is very high, as you can see. https://medium.com/media/af2734c69ef32a4bdfe7e760465c6f3a/href Let’s output both our model and prediction for evaluation. https://medium.com/media/e98ec938e18be9c0f2a3262d76ca402e/href Final Pipeline Loading the data Exploratory Data analysis Using Alteryx EvalML and Microsoft FLAML to preprocess, train multiple modes and then evaluate results Processing data for training Train on Naive Bayes, Random Forest, CatBoost, and LightGBM.
