THE USE OF SPARK FOR NEAREST NEIGHBOR CLASSIFICATION FOR HIGH-SPEED BIG DATA STREAMING

Computer science and Engineering

One of the most complicated and difficult aspects of machine learning and Artificial Intelligence (AI) application is data mining and parallel data transmission. This necessitates the adoption of methods that demonstrate high process effectiveness and efficiency, as well as the ability to change their structure and manage a large variety of data that arrives on a regular basis. This should lead to the presentation of a new incremental and distributed classifier based on the preferred closest neighbor algorithmic rule. The research employs the K-NN machine learning classifier, which is well-suited to such a demanding setting. This approach, which is implemented in Apache Spark, uses a distributed metric-space ordering to speed up searches. A great deal of live data transmission containing useful data, referred to as big data, is created frequently. For handling such large volume of data, there is a necessity of big data structures, which the Apache Spark resolves in this work. The model will performs up to one hundred circumstances speedier than ancient systems like Hadoop Map reduce and the system will be used for online payment processing and big data search pattern.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.