Automated Information Aggregation for Scaling Scale-Resistant Services

Abstract

Machine learning provides techniques to monitor system behavior and predict failures from sensor data. However, such algorithms are "scale resistant" $high computational complexity and not parallelizable. The problem then becomes identifying and delivering the relevant subset of the vast amount of sensor data to each monitoring node, despite the lack of explicit "relevance" labels. The simplest solution is to deliver only the "closest" data items under some distance metric. We demonstrate a better approach using a more sophisticated architecture: a scalable data aggregation and dissemination overlay network uses an influence metric reflecting the relative influence of one node's data on another, to efficiently deliver a mix of raw and aggregated data to the monitoring components, enabling the application of machine learning tools on real-world problems. We term our architecture level of detail after an analogous computer graphics technique