Abstract
Distributed stream processing systems (DSPSs) such as Apache Flink and Apache Spark Streaming serve as foundational infrastructure for real-time analytics in modern cloud environments. However, diagnosing intermittent performance bottlenecks—such as transient backpressure cascades, JVM garbage collection pauses, and network socket contention—remains notoriously challenging due to non-linear cross-layer interactions. Traditional machine learning models often detect anomalies based on statistical correlations, yielding uninterpretable alarms and misleading causal attributions. In this paper, we propose CausalTrace, an explainable artificial intelligence framework that integrates lightweight extended Berkeley Packet Filter (eBPF) system call tracing with structural causal modeling to accurately isolate and explain root causes of DSPS degradation. By capturing kernel-level thread scheduling, memory allocation, and socket I/O events with sub-2% runtime overhead, CausalTrace constructs dynamic causal directed acyclic graphs using a time-series constraint-based discovery algorithm. It then applies Shapley-valued causal attribution to produce human-interpretable diagnostic reports explaining performance anomalies. We evaluate CausalTrace on a multi-node Kubernetes cluster running representative stateful streaming benchmarks under diverse synthetic and real-world failure injection scenarios. Experimental results demonstrate that CausalTrace achieves a 94.6% root-cause localization accuracy, outperforming state-of-the-art correlation-based and tracing baselines by at least 18.2%, while drastically reducing mean time to diagnosis from hours to seconds.
