Proactive IT network monitoring through log analysis using ML and Open AI

Abstract

This research focused on a machine learning technique ( XGBoost – Extreme Gradient boosting), Transformer models (all-MiniLM-L6-v2 a sentence embedding model developed by Microsoft) based system for proactive network monitoring, performing log analysis for real-time anomaly detection and pattern analysis for root cause evaluation. This was done in order to address the challenge of reacting to problems only after they occur which leads to business revenue loss and increased idle time for workers when business operations are disrupted. The system makes use of the online NLP (natural language processing) model specifically (OPENAI or Cohere), which are inferred for intelligent problem explanation and solution recommendation. The methodology used was CRISP-DM for Data Science and incremental software methodology. The system enables network administrators to identify emerging problems within the network and address them pro-actively through system provided recommendations and anomaly evaluation insights before full negative impact on business operations.