Scientists acknowledge the reasons behind data unavailability, establishes the scope of missing data, handle the gaps with the right strategy, and avoid common mistakes.
This is a story for anyone who wishes to learn more about missing data and how to deal with it.
Let’s start with “Go straight for _____ meters, and then turn at the second cross,” imagine what would happen to a driver of a car if they were given missing data by a navigator.
No one likes missing data in their databases. Missing Data can have a negative impact on consumer apps and leave users unsure about their decisions. The same applies to data scientists. Data missing from data sets is something we also dislike. It makes our machine learning algorithms inefficient and the raw data unsuitable.
Why is there missing data?
Why are some data parts missing while others are readily available? Before we start strategizing, let’s have a look at how this happens. Data loss can be caused by several factors, including malfunctioning devices that collect data. Eg. Thermometers located at weather stations.
Dust can cause the thermometer to stop recording temperature. This happens for a brief time until it is noticed and cleared. There may be gaps in the survey due to a lack of field staff knowledge. For example, new recruiters might not ask certain details such as — the bonus amount the applicant received in their previous appraisal. Respondents may choose to leave out certain details when filling in feedback forms.
For example, some customers might not want to give income or gender information.
Regardless of their reason, the algorithm’s ability to learn from missing data is affected. Before defining any training program, the Data Scientist needs to identify and fix them. This is the need. What can you do to find missing data from your database? It is crucial to identify the missing data and how you can handle it. You can detect missing data in five ways: Image by Author. Check for empty or null values within a column.
Verify that the column count matches. The count of columns in a dataset should match the row count. You should look for uniformity in the column’s values. Even though the columns match the count, there are some possible errors. Check for invalid columns. Is the range of numeric numbers within which the column is located? Are there any categorical value that corresponds to the list defined? The outliers can be found at each column level.
Yes. I also recommend that outliers be considered missing data. The next step is to determine if there are missing data. What strategy should you use to deal with missing data? All strategies can be divided into one of two types: eliminate, blank or invalid.
Strategy 1: Remove all rows that have at least one missing value
A pseudo algorithm for this would be: For col in columns, delete column end. If row forend is empty, invalid, or blank, then you can lose significant data. This works well when there is a large data volume and missing values are distributed randomly across the database. Random distribution ensures that data is not affected by the removal of rows. One 10 thermometer, located at an unusual location, did not record temperature for 30 minutes in one day. It is possible to introduce bias if all readings from one thermometer are removed.
Strategy 2. Replace the constant default value with one that is consistent
This strategy works for both continuous and categorical variables. Exemple:Clinical Dataset: Boolean Column ‘Available’ indicates the physician’s availability on a specific date. Replace the blank/null/invalid value with the value of ‘False’. Project Management Dataset, if the “Planned Start Date” of an activity is not present, substitute it with the “Project Start Date”, which is a constant.
Strategy 3: Use a UnivariateStatistics to estimate a new value based on the available features
It could also be the mean, median, or mode for a numerical feature. If it is a categorical function, the function could return the most frequently returned value. Exemple:Recruitment Drive Database: Replace the missing value for ‘expected salaries’ with the average of the expected salary’ feature.
Strategy 4: Replace the Multivariate Statistics
This horizontal calculation is performed across many features. Exemple:Recruitment Drive Database: The candidate’s resignation date can be used to determine the value for ‘notice’.
Strategy 5: Iterative replacement
If replacing every value with one constant creates bias in the data, iterative replacement is an effective strategy. Instead of creating a default constant for each feature, create a list that allows you to replace it. You would then go through each feature, replacing any missing values with constants from the list. Ex. In a round-robin manner, go through the customer list and add one from each of red green or blue’ to it.
Strategy 6: Linear forward or backward
If your immediate neighbor has more reliable information that the other rows, this strategy might be a good fit. You can iterate by using a feature. If you find a missing value in a row, then copy that value to the next row. The Iteration can either be forward (top-down), or reverse (bottom-up).
Strategy 7: Retail Store Sales Transactions
Take the previous item’s Batch Id if the Batch Id for an item is not available. The Workplace Access Log: Cascade the Exit Time for the employee who is absent from the log.
Strategy 8: Calculate the value of each neighbor
This is the best strategy for situations where the combined k-Nearest neighbors is better than one immediate neighbor.
This avoids the long-term cascading error of the closest neighbor. You can set a value for K and then calculate the average of all the previous k numbers and add the missing value. Calculate the weighted average if the correlation is not consistent with proximity to the neighbor.
