Due to the ever-increasing variety of network threats and the need to store and query data in real time, conventional network traffic monitoring systems are becoming inadequate. As a result, the task of figuring out how to properly monitor massive amounts of network traffic has risen to the forefront of the field of network security management. In order to achieve this goal, we have proposed a new network monitoring system that uses big data technology and uses Netflow as the monitoring object. This system has four primary functions: it can utilize Filebeat to collect Netflow in real time; it transmits the data safely based on Logstash; it stores the data in ElasticSearch; and it analyzes and presents the data in real time using Kibana. The results of our experiments demonstrate that our system can provide millisecond-level replies to 100 million Netflows. It’s able to give the foundation for network security management and fulfill the need for real-time monitoring of massive amounts of network traffic.

Du Zhongxing. Network Traffic Monitoring Algorithm Based on Big Data Analysis. Academic Journal of Computing & Information Science (2023), Vol. 6, Issue 5: 56-67. https://doi.org/10.25236/AJCIS.2023.060508.

[1] Al-Nafjan K., Al-Hussein M. A., Alghamdi A. S., Haque M. A., & Ahmad I. (2012). Intrusion detection using PCA based modular neural network. International Journal of Machine Learning and Computing, 2(5), 583.

[2] Chabaa S., Zeroual A., & Antari J. (2009, November). ANFIS method for forecasting internet traffic time series. In 2009 Mediterrannean microwave symposium (mms) (pp. 1-4).

[3] Datti R., & Verma B. (2010). Feature reduction for intrusion detection using linear discriminant analysis. International Journal on Engineering Science and Technology, 2(4), 1072-1078.

[4] Sivakumar R., Kumar E. A., & Sivaradje G. (2011, July). Prediction of traffic load in wireless network using time series model. In 2011 International Conference on Process Automation, Control and Computing (pp. 1-6).

[5] Vijayakumar M., & Parvathi R. M. S. (2010). Concept mining of high volume data streams in network traffic using hierarchical clustering. European Journal of Scientific Research, 39(2), 234-242.

[6] R. M. Vijayakumar, Concept Mining of High Volume Data Streams in Network Traffic Using Hierarchical Clustering, Academic Journal, 2010. 

[7] Muda Z., Yassin W., Sulaiman M. N., & Udzir N. I. (2011, July). Intrusion detection based on K-Means clustering and Naïve Bayes classification. In 2011 7th international conference on information technology in Asia (pp. 1-6).

[8] Yu B., & Fei H. (2008, October). Performance impact of wireless mesh networks with mining traffic patterns. In 2008 Fifth International Conference on Fuzzy Systems and Knowledge Discovery (Vol. 2, pp. 493-497).

[9] Erman J., Arlitt M., & Mahanti A. (2006, September). Traffic classification using clustering algorithms. In Proceedings of the 2006 SIGCOMM workshop on Mining network data (pp. 281-286). 

[10] Du X., Yang Y., & Kang X. (2008, December). Research of applying information entropy and clustering technique on network traffic analysis. In 2008 International Conference on Computational Intelligence and Security (Vol. 2, pp. 472-476).

[11] Datti R., & Verma B. (2010). Feature reduction for intrusion detection using linear discriminant analysis. International Journal on Engineering Science and Technology, 2(4), 1072-1078.

[12] Sadek R. A., Soliman M. S., & Elsayed H. S. (2013). Effective anomaly intrusion detection system based on neural network with indicator variable and rough set reduction. International Journal of Computer Science Issues (IJCSI), 10(6), 227.