MONITORING MACHINE LEARNING MODEL DRIFT IN PRODUCTION PIPELINES: METHODS, METRICS, AND DEPLOYMENT CONSIDERATIONS
DOI:
https://doi.org/10.32689/maup.it.2025.1.1Keywords:
моніторинг дрейфу, CI/CD пайплайни, MLOps, стабільність моделей, автоматизація, TensorFlow Extended, Google Vertex AI, AWS SageMaker, data drift, concept drift, label driftAbstract
The relevance of the study is determined by the need to ensure the stability and effectiveness of machine learning models in the context of dynamic changes in data. The problem of model drift – changes in the statistical characteristics of input data or relationships between features and the target variable – leads to a decrease in prediction accuracy. Detecting and monitoring drift in real-time is crucial for maintaining the stability of models, particularly in fields such as finance, healthcare, and cybersecurity, where changes in input data or conditions can significantly affect model performance. The aim of the paper is to investigate methods for monitoring model drift, particularly within the integration of CI/CD pipelines, to ensure their stability in real-world conditions. Special attention is paid to types of drift (data drift, concept drift, label drift) and the metrics used for their detection. The research methods include analyzing existing tools for monitoring and detecting changes in model behavior through the example of financial risk forecasting, as well as evaluating the effectiveness of integrating monitoring into CI/CD. The scientific novelty lies in the proposed comprehensive approach to detecting drift and integrating monitoring into production pipelines using advanced tools such as Google Vertex AI, AWS SageMaker, and TensorFlow Extended, which allow automatic response to changes in data. The use of such technologies improves prediction accuracy and reduces errors in realworld environments. The study confirms the importance of integrating drift monitoring into the continuous process of updating and adapting models to maintain their effectiveness in the context of constantly changing data. The conclusions show that integrating drift monitoring systems into CI/CD pipelines significantly improves the stability and effectiveness of models. Timely detection of drift allows for prompt model adjustments, reducing the likelihood of model degradation. It has been found that for achieving model stability, the automation of monitoring is crucial, as it allows for a prompt response to changes without manual intervention. This enhances the system’s efficiency and reduces risks related to the deterioration of prediction quality.
References
Ackerman S., Farchi E., Raz O., Zalmanovici M., Dube P. Detection of data drift and outliers affecting machine learning model performance over time. arXiv preprint. 2020. arXiv:2012.09258. DOI: https://doi.org/10.48550/arXiv.2012.09258 (date of access: 04.04.2025).
Alibi Detect. Explainable AI for Drift Detection and Model Monitoring. 2025. URL: https://github.com/SeldonIO/alibidetect (date of access: 03.04.2025).
Amazon Web Services. Amazon SageMaker Model Monitor. 2025. URL: https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html (date of access: 03.04.2025).
Amazon Web Services. Machine Learning Lens – AWS Well-Architected Framework. 2025. URL: https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/wellarchitected-machine-learning-lens.pdf (date of access: 03.04.2025).
Celik B., Vanschoren J. Adaptation Strategies for Automated Machine Learning on Evolving Data. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021. Vol. 43, № 9. P. 3067–3078. DOI: https://doi.org/10.1109/TPAMI.2021.3062900 (date of access: 04.04.2025).
Derakhshan B., Rezaei Mahdiraji A., Rabl T., Markl V. Continuous Deployment of Machine Learning Pipelines. Proceedings of the International Conference on Extending Database Technology (EDBT). 2019. P. 397–408. URL: https://hpi.de/fileadmin/user_upload/fachgebiete/rabl/publications/2019/ContinousMLDeploymentEDBT2019.pdf (date of access: 04.04.2025).
Eastvedt D., Naterer G., Duan X. Detection of faults in subsea pipelines by flow monitoring with regression supervised machine learning. Process Safety and Environmental Protection. 2022. Vol. 161, № 1. P. 409–420. DOI: https://doi.org/10.1016/j.psep.2022.03.049 (date of access: 04.04.2025).
Evidently AI. Monitoring ML Models in Production. 2025. URL: https://evidentlyai.com (date of access: 03.04.2025).
Google Cloud. Vertex AI: Model Monitoring. 2025. URL: https://cloud.google.com/vertex-ai/docs/modelmonitoring/using-model-monitoring (date of access: 03.04.2025).
Google. TFX: A TensorFlow-Based Production-Scale Machine Learning Platform. 2025. URL: https://arxiv.org/abs/1707.01967 (date of access: 03.04.2025).
Hapke H., Nelson C. Building machine learning pipelines. O’Reilly Media. 2020. 389 p. URL: https://books.google.com.ua/books?id=H6_wDwAAQBAJ (date of access: 04.04.2025).
Jourdan N., Bayer T., Biegel T., Metternich J. Handling concept drift in deep learning applications for process monitoring. Procedia CIRP. 2023. Vol. 120, № 1. P. 33–38. DOI: https://doi.org/10.1016/j.procir.2023.08.007 (date of access: 04.04.2025).
Kourouklidis P., Kolovos D., Noppen J., Matragkas N. A Model-Driven Engineering Approach for Monitoring Machine Learning Models. MODELS-C: ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion. 2021. P. 160–164. DOI: https://doi.org/10.1109/MODELS-C53483.2021.00028 (date of access: 04.04.2025).
Meta AI. Fully Sharded Data Parallel: faster AI training with less GPU memory. 2025. URL: https://engineering.fb.com/2021/07/15/open-source/fsdp/ (date of access: 03.04.2025).
Microsoft Azure. Model management and deployment concepts. 2025. URL: https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment (date of access: 03.04.2025).
Microsoft. Responsible AI Standard – White Paper. 2025. URL: https://www.microsoft.com/en-us/ai/responsibleai? activetab=pivot:techresources-tab (date of access: 03.04.2025).
Prasad A. N. Monitoring and Maintaining Machine Learning Systems. In: Introduction to Data Governance for Machine Learning Systems. Apress, Berkeley, CA. 2024. P. 129–147. DOI: https://doi.org/10.1007/979-8-8688-1023-7_7 (date of access: 04.04.2025).
River. Online Machine Learning in Python. 2025. URL: https://riverml.xyz (date of access: 03.04.2025).
Shankar S., Parameswaran A. Towards observability for production machine learning pipelines. arXiv preprint. 2021. arXiv:2108.13557. DOI: https://doi.org/10.48550/arXiv.2108.13557 (date of access: 04.04.2025).
Vadde B. C., Munagandla V. B. DevOps in the Age of Machine Learning: Bridging the Gap Between Development and Data Science. International Journal of Machine Learning Research in Cybersecurity and Artificial Intelligence. 2024. Vol. 15, № 1. P. 530–544. URL: https://www.researchgate.net/publication/388526732 (date of access: 04.04.2025).
Wani D., Ackerman S., Farchi E., Liu X., Chang H. Data Drift Monitoring for Log Anomaly Detection Pipelines. arXiv preprint. 2023. arXiv:2310.14893. DOI: https://doi.org/10.48550/arXiv.2310.14893 (date of access: 04.04.2025).
Yadav P., Singh V. K., Joffre T., Rigo O., Arvieu C., Le Guen E., Lacoste E. Inline drift detection using monitoring systems and machine learning in selective laser melting. Advanced Engineering Materials. 2020. Vol. 22, № 12. Article 2000660. DOI: https://doi.org/10.1002/adem.202000660 (date of access: 04.04.2025).
Yang Y., Li Y., Zhang T., Zhou Y., Zhang H. Early safety warnings for long-distance pipelines: A distributed optical fiber sensor machine learning approach. Proceedings of the AAAI Conference on Artificial Intelligence. 2021. Vol. 35, № 17. P. 14991–14999. DOI: https://doi.org/10.1609/aaai.v35i17.17759 (date of access: 04.04.2025).
Zenisek J., Holzinger F., Affenzeller M. Machine learning based concept drift detection for predictive maintenance. Computers & Industrial Engineering. 2019. Vol. 137, № 1. Article 106031. DOI: https://doi.org/10.1016/j.cie.2019.106031 (date of access: 04.04.2025).