Awesome Mlops

An awesome list of references for MLOps - Machine Learning Operations :point_right: ml-ops.org

Linkedin Dr. Larysa Visengeriyeva

MLOps Core	MLOps Communities
MLOps Books	MLOps Articles
MLOps Workflow Management	MLOps: Feature Stores
MLOps: Data Engineering (DataOps)	MLOps: Model Deployment and Serving
MLOps: Testing, Monitoring and Maintenance	MLOps: Infrastructure
MLOps Papers	Talks About MLOps
Existing ML Systems	Machine Learning
Software Engineering	Product Management for ML/AI
The Economics of ML/AI	Model Governance, Ethics, Responsible AI
MLOps: People & Processes	Newsletters About MLOps, Machine Learning, Data Science and Co.

Click to expand!

1. Machine Learning Operations: You Design It, You Train It, You Run It!
2. MLOps SIG Specification
3. ML in Production
4. Awesome production machine learning: State of MLOps Tools and Frameworks
5. Udemy “Deployment of ML Models”
6. Full Stack Deep Learning
7. Engineering best practices for Machine Learning
8. :rocket: Putting ML in Production
9. Stanford MLSys Seminar Series
10. IBM ML Operationalization Starter Kit
11. Productize ML. A self-study guide for Developers and Product Managers building Machine Learning products.
12. MLOps (Machine Learning Operations) Fundamentals on GCP
13. ML full Stack preparation
14. MLOps Guide: Theory and Implementation
15. Practitioners guide to MLOps: A framework for continuous delivery and automation of machine learning.
16. MLOps maturity assessment

Click to expand!

1. MLOps.community
2. CDF Special Interest Group - MLOps
3. RsqrdAI - Robust and Responsible AI
4. DataTalks.Club
5. Synthetic Data Community
6. MLOps World Community
7. Marvelous MLOps

1. MLOps Zoomcamp (free)
2. Coursera's Machine Learning Engineering for Production (MLOps) Specialization
3. Udacity Machine Learning DevOps Engineer
4. Made with ML
5. Udacity LLMOps: Building Real-World Applications With Large Language Models

Click to expand!

1. “Machine Learning Engineering” by Andriy Burkov, 2020
2. "ML Ops: Operationalizing Data Science" by David Sweenor, Steven Hillion, Dan Rope, Dev Kannabiran, Thomas Hill, Michael O'Connell
3. "Building Machine Learning Powered Applications" by Emmanuel Ameisen
4. "Building Machine Learning Pipelines" by Hannes Hapke, Catherine Nelson, 2020, O’Reilly
5. "Managing Data Science" by Kirill Dubovikov
6. "Accelerated DevOps with AI, ML & RPA: Non-Programmer's Guide to AIOPS & MLOPS" by Stephen Fleming
7. "Evaluating Machine Learning Models" by Alice Zheng
8. Agile AI. 2020. By Carlo Appugliese, Paco Nathan, William S. Roberts. O'Reilly Media, Inc.
9. "Machine Learning Logistics". 2017. By T. Dunning et al. O'Reilly Media Inc.
10. "Machine Learning Design Patterns" by Valliappa Lakshmanan, Sara Robinson, Michael Munn. O'Reilly 2020
11. "Serving Machine Learning Models: A Guide to Architecture, Stream Processing Engines, and Frameworks" by Boris Lublinsky, O'Reilly Media, Inc. 2017
12. "Kubeflow for Machine Learning" by Holden Karau, Trevor Grant, Ilan Filonenko, Richard Liu, Boris Lublinsky
13. "Clean Machine Learning Code" by Moussa Taifi. Leanpub. 2020
14. E-Book "Practical MLOps. How to Get Ready for Production Models"
15. "Introducing MLOps" by Mark Treveil, et al. O'Reilly Media, Inc. 2020
16. "Machine Learning for Data Streams with Practical Examples in MOA", Bifet, Albert and Gavald`a, Ricard and Holmes, Geoff and Pfahringer, Bernhard, MIT Press, 2018
17. "Machine Learning Product Manual" by Laszlo Sragner, Chris Kelly
18. "Data Science Bootstrap Notes" by Eric J. Ma
19. "Data Teams" by Jesse Anderson, 2020
20. "Data Science on AWS" by Chris Fregly, Antje Barth, 2021
21. “Engineering MLOps” by Emmanuel Raj, 2021
22. Machine Learning Engineering in Action
23. Practical MLOps
24. "Effective Data Science Infrastructure" by Ville Tuulos, 2021
25. AI and Machine Learning for On-Device Development, 2021, By Laurence Moroney. O'Reilly
26. Designing Machine Learning Systems ,2022 by Chip Huyen , O'Reilly
27. Reliable Machine Learning. 2022. By Cathy Chen, Niall Richard Murphy, Kranti Parisa, D. Sculley, Todd Underwood. O'Reilly
28. MLOps Lifecycle Toolkit. 2023. By Dayne Sorvisto. Apress
29. Implementing MLOps in the Enterprise. 2023. By Yaron Haviv, Noah Gift. O'Reilly

Click to expand!

1. Continuous Delivery for Machine Learning (by Thoughtworks)
2. What is MLOps? NVIDIA Blog
3. MLSpec: A project to standardize the intercomponent schemas for a multi-stage ML Pipeline.
4. The 2021 State of Enterprise Machine Learning | State of Enterprise ML 2020: PDF and Interactive
5. Organizing machine learning projects: project management guidelines.
6. Rules for ML Project (Best practices)
7. ML Pipeline Template
8. Data Science Project Structure
9. Reproducible ML
10. ML project template facilitating both research and production phases.
11. Machine learning requires a fundamentally different deployment approach. As organizations embrace machine learning, the need for new deployment tools and strategies grows.
12. Introducting Flyte: A Cloud Native Machine Learning and Data Processing Platform
13. Why is DevOps for Machine Learning so Different?
14. Lessons learned turning machine learning models into real products and services – O’Reilly
15. MLOps: Model management, deployment and monitoring with Azure Machine Learning
16. Guide to File Formats for Machine Learning: Columnar, Training, Inferencing, and the Feature Store
17. Architecting a Machine Learning Pipeline How to build scalable Machine Learning systems
18. Why Machine Learning Models Degrade In Production
19. Concept Drift and Model Decay in Machine Learning
20. Machine Learning in Production: Why You Should Care About Data and Concept Drift
21. Bringing ML to Production
22. A Tour of End-to-End Machine Learning Platforms
23. MLOps: Continuous delivery and automation pipelines in machine learning
24. AI meets operations
25. What would machine learning look like if you mixed in DevOps? Wonder no more, we lift the lid on MLOps
26. Forbes: The Emergence Of ML Ops
27. Cognilytica Report "ML Model Management and Operations 2020 (MLOps)"
28. Introducing Cloud AI Platform Pipelines
29. A Guide to Production Level Deep Learning
30. The 5 Components Towards Building Production-Ready Machine Learning Systems
31. Deep Learning in Production (references about deploying deep learning-based models in production)
32. Machine Learning Experiment Tracking
33. The Team Data Science Process (TDSP)
34. MLOps Solutions (Azure based)
35. Monitoring ML pipelines
36. Deployment & Explainability of Machine Learning COVID-19 Solutions at Scale with Seldon Core and Alibi
37. Demystifying AI Infrastructure
38. Organizing machine learning projects: project management guidelines.
39. The Checklist for Machine Learning Projects (from Aurélien Géron,"Hands-On Machine Learning with Scikit-Learn and TensorFlow")
40. Data Project Checklist by Jeremy Howard
41. MLOps: not as Boring as it Sounds
42. 10 Steps to Making Machine Learning Operational. Cloudera White Paper
43. MLOps is Not Enough. The Need for an End-to-End Data Science Lifecycle Process.
44. Data Science Lifecycle Repository Template
45. Template: code and pipeline definition for a machine learning project demonstrating how to automate an end to end ML/AI workflow.
46. Nitpicking Machine Learning Technical Debt
47. The Best Tools, Libraries, Frameworks and Methodologies that Machine Learning Teams Actually Use – Things We Learned from 41 ML Startups
48. Software Engineering for AI/ML - An Annotated Bibliography
49. Intelligent System. Machine Learning in Practice
50. CMU 17-445/645: Software Engineering for AI-Enabled Systems (SE4AI)
51. Machine Learning is Requirements Engineering
52. Machine Learning Reproducibility Checklist
53. Machine Learning Ops. A collection of resources on how to facilitate Machine Learning Ops with GitHub.
54. Task Cheatsheet for Almost Every Machine Learning Project A checklist of tasks for building End-to-End ML projects
55. Web services vs. streaming for real-time machine learning endpoints
56. How PyTorch Lightning became the first ML framework to run continuous integration on TPUs
57. The ultimate guide to building maintainable Machine Learning pipelines using DVC
58. Continuous Machine Learning (CML) is CI/CD for Machine Learning Projects (DVC)
59. What I learned from looking at 200 machine learning tools | Update: MLOps Tooling Landscape v2 (+84 new tools) - Dec '20
60. Big Data & AI Landscape
61. Deploying Machine Learning Models as Data, not Code — A better match?
62. “Thou shalt always scale” — 10 commandments of MLOps
63. Three Risks in Building Machine Learning Systems
64. Blog about ML in production (by maiot.io)
65. Back to the Machine Learning fundamentals: How to write code for Model deployment. Part 1, Part 2, Part 3
66. MLOps: Machine Learning as an Engineering Discipline
67. ML Engineering on Google Cloud Platform (hands-on labs and code samples)
68. Deep Reinforcement Learning in Production. The use of Reinforcement Learning to Personalize User Experience at Zynga
69. What is Data Observability?
70. A Practical Guide to Maintaining Machine Learning in Production
71. Continuous Machine Learning. Part 1, Part 2. Part 3 is coming soon.
72. The Agile approach in data science explained by an ML expert
73. Here is what you need to look for in a model server to build ML-powered services
74. The problem with AI developer tools for enterprises (and what IKEA has to do with it)
75. Streaming Machine Learning with Tiered Storage
76. Best practices for performance and cost optimization for machine learning (Google Cloud)
77. Lean Data and Machine Learning Operations
78. A Brief Guide to Running ML Systems in Production Best Practices for Site Reliability Engineers
79. AI engineering practices in the wild - SIG | Getting software right for a healthier digital world
80. SE-ML | The 2020 State of Engineering Practices for Machine Learning
81. Awesome Software Engineering for Machine Learning (GitHub repository)
82. Sampling isn’t enough, profile your ML data instead
83. Reproducibility in ML: why it matters and how to achieve it
84. 12 Factors of reproducible Machine Learning in production
85. MLOps: More Than Automation
86. Lean Data Science
87. Engineering Skills for Data Scientists
88. DAGsHub Blog. Read about data science and machine learning workflows, MLOps, and open source data science
89. Data Science Project Flow for Startups
90. Data Science Engineering at Shopify
91. Building state-of-the-art machine learning technology with efficient execution for the crypto economy
92. Completing the Machine Learning Loop
93. Deploying Machine Learning Models: A Checklist
94. Global MLOps and ML tools landscape (by MLReef)
95. Why all Data Science teams need to get serious about MLOps
96. MLOps Values (by Bart Grasza)
97. Machine Learning Systems Design (by Chip Huyen)
98. Designing an ML system (Stanford | CS 329 | Chip Huyen)
99. How COVID-19 Has Infected AI Models (about the data drift or model drift concept)
100. Microkernel Architecture for Machine Learning Library. An Example of Microkernel Architecture with Python Metaclass
101. Machine Learning in production: the Booking.com approach
102. What I Learned From Attending TWIMLcon 2021 (by James Le)
103. Designing ML Orchestration Systems for Startups. A case study in building a lightweight production-grade ML orchestration system
104. Towards MLOps: Technical capabilities of a Machine Learning platform | Prosus AI Tech Blog
105. Get started with MLOps A comprehensive MLOps tutorial with open source tools
106. From DevOps to MLOPS: Integrate Machine Learning Models using Jenkins and Docker
107. Example code for a basic ML Platform based on Pulumi, FastAPI, DVC, MLFlow and more
108. Software Engineering for Machine Learning: Characterizing and Detecting Mismatch in Machine-Learning Systems
109. TWIML Solutions Guide
110. How Well Do You Leverage Machine Learning at Scale? Six Questions to Ask
111. Getting started with MLOps: Selecting the right capabilities for your use case
112. The Latest Work from the SEI: Artificial Intelligence, DevSecOps, and Security Incident Response
113. MLOps: The Ultimate Guide. A handbook on MLOps and how to think about it
114. Enterprise Readiness of Cloud MLOps
115. Should I Train a Model for Each Customer or Use One Model for All of My Customers?
116. MLOps-Basics (GitHub repo) by raviraja
117. Another tool won’t fix your MLOps problems
118. Best MLOps Tools: What to Look for and How to Evaluate Them (by NimbleBox.ai)
119. MLOps vs. DevOps: A Detailed Comparison (by NimbleBox.ai)
120. A Guide To Setting Up Your MLOps Team (by NimbleBox.ai)

1. Open-source Workflow Management Tools: A Survey by Ploomber
2. How to Compare ML Experiment Tracking Tools to Fit Your Data Science Workflow (by dagshub)
3. 15 Best Tools for Tracking Machine Learning Experiments

Click to expand!

1. Feature Stores for Machine Learning Medium Blog
2. MLOps with a Feature Store
3. Feature Stores for ML
4. Hopsworks: Data-Intensive AI with a Feature Store
5. Feast: An open-source Feature Store for Machine Learning
6. What is a Feature Store?
7. ML Feature Stores: A Casual Tour
8. Comprehensive List of Feature Store Architectures for Data Scientists and Big Data Professionals
9. ML Engineer Guide: Feature Store vs Data Warehouse (vendor blog)
10. Building a Gigascale ML Feature Store with Redis, Binary Serialization, String Hashing, and Compression (DoorDash blog)
11. Feature Stores: Variety of benefits for Enterprise AI.
12. Feature Store as a Foundation for Machine Learning
13. ML Feature Serving Infrastructure at Lyft
14. Feature Stores for Self-Service Machine Learning
15. The Architecture Used at LinkedIn to Improve Feature Management in Machine Learning Models.
16. Is There a Feature Store Over the Rainbow? How to select the right feature store for your use case