Journal of Advances in Mathematics and Computer Science

This is an Original research article about the Real time intelligence at scale. Organizations have experienced exponential data growth in size and complexity, creating unprecedented challenges in Operational management and Quality Insights. Traditional Data Consumption and Processing approaches struggle with efficiently managing the scale, resource requirements, and dynamic nature and variety, highlighting the need for more efficient and advanced solutions. To develop and validate a data processing framework leveraging Edge AI and Dynamic Tables for optimizing data pipelines specifically for continuous and NRT/Real Time insights, focusing on automating business decisions, resource allocation, and pipeline optimization while maintaining performance and cost efficiency. We implemented a multi-stream medallion architecture for processing heterogeneous operational data, coupled with an adaptive resource allocation system and sophisticated data ingress orchestration mechanism. The framework is tested and evaluated across multiple cloud environments using production workloads from various organizations, testing its performance in multi-cloud environments, high-throughput production processes and time-sensitive deployments. Our framework demonstrated significant improvements over traditional data ingress approaches, achieving 40% faster data insights, 18% reduction in data latency, and 24% decrease in operational costs. These improvements were validated through comprehensives scenarios across different data sources and workload patterns, demonstrating the framework’s ability to maintain performance stability while optimizing resource usage. The system’s ability to adapt to varying workloads and automatically optimize data processing strategies provides a robust solution for modern data needs and challenges, offering improved operational efficiency and cost management while maintaining system reliability.