Author(s: Eija Leena Koponen Original publication on , the World’s Most Reputable AI and Technology News and Media Company. We invite you to become an AI sponsor if you’re working on an AI product or service. helps technology and AI startups scale. We can help you bring your technology to millions. DevOps: How can you bridge the gap between data science and engineering? Photo by the author Data Scientists were hailed as the 21sexiest jobs of today. Most companies in 2021, have included Data Science into their core processes. This includes analyzing customer data and optimizing manual tasks. Data scientists, on the other hand are a diverse group of individuals. Some have a background either in Statistics or in mechanical engineering and others in Physics. Although there has been an increase in output due to actual Data Science Master’s programmes, most people are still self-taught, including me. It seems that the problem is not the data, but the process of bringing the exploratory projects to production. To put it another way, Software Engineers should be able to communicate with Data Scientists in order to make the whole process smoother. Data scientists are often very skilled at fetching data and wrangling it. They can also make visualizations, initial models, notebooks, and other preliminary modeling. Data Scientists need to hand the responsibility for the creation phase of the model, providing it to users and then integrating it into the existing tools. These messy scripts and notebooks can often look like cardboard models to software engineers, which makes it difficult for them to translate into a tangible thing. What are the best steps for everyone to follow to close the gap? These are some steps that we suggest. Image by the author. Git Version Control is a tool that tracks changes made to files or sets of files over time. This allows you to see particular versions in the future. Github and Bitbucket allow you to easily see the changes made by multiple developers working together on one code base. To avoid accidental overwrites or other errors, this is necessary. Google Docs already provides version control, so you can do the same for your code. Your code will not only be protected from being lost or misplaced, but it’ll also be available on Google Docs. Your teammates will be able to see what you’re building so they can pick up from where you left off. The side effect is that code becomes less spaghetti-like if someone else will be reading it. Data Scientists consider reproducible notebooks the ultimate tool for studying your machine. There are many online services available, such as e.g. Kaggle kernels and Sagemaker are just a few of the many online providers available. Data Scientists have many options with notebooks. You can, for example, run your code in whatever order you like — which is what I know people do. They also include lots of checks and “unnecessary”, things that are available to anyone. There’s also the hassle of setting up: which version and pandas libraries, python and other libraries have you installed? What are your dependencies when reproducing what the other DS did? They are still far from being ready for production. Jupyhai is a solution by Valohai. A Valohai execution can be run from your Jupyter notebook. This requires that the entire notebook be executed simultaneously, from top to bottom. Data scientists must keep the code clean and orderly. This principle is not only for Valohai; it will make every notebook easier to read if you write them this way. Voila! Now you are just one step away from the production code. Modular work: Instead of scrolling through the notebook endlessly, you can now think modularly. It takes some extra effort to break down your work and form a pipeline, but this will be worth it in the long-term. It is possible to only run one part at a given time. This allows you to identify the potential errors or bugs and any bottlenecks in your project. You might even want to write tests so that future changes to one part of the project don’t cause problems. This helps you stay organized, and makes it easier to maintain your code. It might be simpler to show a person how your work flows at a higher level. This allows them to focus their attention on one task at a given time. The definition of finished is a vague one. Hopefully many of your projects will turn into products. This is a continuous loop in which there are always new things to improve, iterate, optimize and make even better. Frameworks like agile development can help us again. Although not every aspect of Data Science works, planning sprints, prioritizing each one, clearly defining tasks and deadlines, using retrospectives and demos may make it feel like you have accomplished something. It is also easier to communicate with developers and business stakeholders. Empathy is the last thing. We all come from different backgrounds and each other. Diversity adds to our richness. However, it takes effort to have different thoughts and methods of doing things. Communicating with someone, for example. It is easier to communicate with someone, e.g., from the same educational background. However, it will still be a project that requires only one set or skills. Data Scientists, developers don’t want to ruin your day by creating tools or frameworks that work for you. Data Scientists don’t produce incomprehensible spaghetti codes on purpose. It’s often a different way to think and lack of knowledge better. This industry was 10-15 many years ago. Understanding and learning from other people, trying to view the world through their eyes, e.g. Empathy allows you to do amazing things. You promote the understanding and transferability of the work by giving the Data Scientists and their teams frameworks. Only tackle one workflow problem at a given time, and then move on to MLOps or tooling. DS Dev originally published on https://valohai.com. The original publication of From Notebook to Production appeared in on Medium. People are responding and highlighting this story on Medium. Get the AI newsletter and join thousands of data professionals. We don’t spam and it’s completely free. Stay up-to-date with AI news. Research, projects, and new ideas. We invite you to become a sponsor if you’re building an AI startup or an AI-related service. Published via
