Datacentric AI is a focus on data. It considers the data to be the determinant of success or failure in AI deployments within organizations. A model-centric approach, on the other hand, is more focused on details of the model and less important to the data. It was believed that a model-centric approach would be necessary in order to create accurate models for ML pipelines. Recent AI developments have proven this to be incorrect. Instead, practitioners can build a “good enough model” base that allows them to train, validate, improve accuracy, and gain deeper insight into the nature of the data. AI companies and organizations are focusing their efforts towards a more data-centric AI approach. This is in direct contrast to the model-centric approach. There are many ways that the AI landscape can shift from model-centric to data-centric AI. Snorkel AI is tackling the huge challenge of shifting current AI practices towards more data-centric methods and ending time-wasting modelitis with Snorkel Flow. We will be briefly discussing Snorkel AI. Snorkel AI was created by the inventors of data-centric AI. Source: Snorkel AI. Snorkel AI began as an experiment in the Stanford AI Lab in 2015.. In the beginning, they wanted to find a better interface for machine learning using training data. Snorkel AI has over 50 peer-reviewed publications, published at ICML, Nature, ICLR, IEEE, NeurIPS, and many more, powering the core technology behind Snorkel Flow. Snorkel technology was also developed by Intel, Apple and two of America’s top banks. It has been used in the US Department of Justice and many other major organizations. Snorkel AI Snorkel Flow is the First Data-Centric AI Platform. Snorkel Flow uses weak supervision and programmatic labeling [3] to power its AI development platform. Data science teams and subject-matter experts can quickly collaborate using Snorkel Flow to create highly precise AI applications. It also allows you to manage large amounts of data, create models and analyze them, then deploy the models. Snorkel AI At What Does Snorkel AI excel? Instead of spending months, or even years labeling training data manually, programmatic labeling can be done in hours. All data sources can be used to integrate and manage programmatic data, data cleaning and data slicing. In-platform and via the Python SDK, train and deploy machine learning models of high quality. To quickly identify and fix errors in data, analyze and monitor the performance of model performances. Find out more about Snorkel Flow. Snorkel AI SuperGLUE case study. Using pre-trained, standard models and little tuning, the Snorkel AI Team was able leverage key abstractions to programmatically create and manage training data. This result enabled them to reach a new benchmark, SuperGLUE, which has six tasks to evaluate “general-purpose language comprehension technologies.” The SuperGLUE Benchmark was used to create a new SOTA. Four of the four components were also utilized. SuperGLUE, which is similar to GLUE, has “more challenging tasks” that are selected to maximise difficulty and variety. These are tasks that show significant headroom between strong BERT++ baselines and human performance. Source: SnorkelAI. We then minimally tune the models (baseline models and default learning rate). and find that with applications of the above programming abstractions, we notice improvements of +4.0 points on the SuperGLUE benchmark (indicating a 21% reduction of the gap to human performance). The paper [5] provides updates about Snorkel’s industrial use cases. These include Google using Snorkel Drybell for scientific work in MRI classification, and automated Genomewide Association Study (GWAS), both of which were accepted by Nature Comms. Industrial Case Studies Google has used Snorkel to replace 100k+ hand-annotated labels in critical machine learning pipelines. Snorkel Flow is used by a top US bank to rapidly build AI apps that extract and classify information from documents. Apple developed applications using an internal Snorkel-based Snorkel system. It processed billions of records and answered millions of questions in many languages. There were 2.9x less errors. A Fortune 500 Biotech pioneer leveraged Snorkel Flow to extract critical chronic disease data from clinical trials, accurately processing 300K documents in minutes. References [1] “Snorkel: Rapid Training Data Creation with Weak Supervision.” Alex Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, Chris Re, Stanford University, https://arxiv.org/pdf/1711. 10160.pdf [2] “Weak Supervision: A New Programming Paradigm For Machine Learning.” Alex Ratner, Paroma Varma, Braden Hancock, Chris Re, et al., SAIL Blog, 2019, https://ai.stanford.edu/blog/weak-supervision/ [3] “Interactive Programmatic Labeling for Weak Supervision.” Benjamin Cohen-Wang, Stephen Mussmann, Alex Ratner, Chris Re, KDD, 2019, https://bencw99.github.io/files/kdd2019_dcclworkshop.pdf [4] “Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale.” Stephen H. Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alexander Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Christopher Re, Rob Malkin, SIGMOD, 2019, https://arxiv.org/abs/1812. 00417 [5] Wang, Alex, et al. “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems.” 2019. The SuperGLUE database consists 6 data sets: RTE2, Bar Haim, De Marneffe and 2019), Choice of Plausible Alternatives (COPA) Roemmele, et. al. 2011), Multi-Sentence reading comprehension dataset (MultiRC), Khashabi,, et. al. 2018), Recognizing textual Entailment (merged with RTE1, Dagan, et. al. 2006, RTE2, Bar Haim, et al., 2006, RTE3, Giampiccolo, et al., 2007, and RTE5, Bentivogli, et al., 2009), Word in Context (WiC, Pilehvar, and Camacho-Collados, 2019), and the Winograd Schema Challenge (WSC, Levesque, et al., 2012).
