AUTOMATED DATASET PREPROCESSING USING WINSORIZARTION AND VISUALIZATION

Computer Engineering

Data preprocessing is a crucial step in Machine Learning, as it directly affects the quality of data and the information obtained from it. However, the numerous options available for each transformation task can overwhelm inexperienced users. To address this, we have developed a simple Python-based Auto-preprocessing architecture for Automated Machine Learning. This architecture provides automated, interactive, and data-driven support to help users efficiently perform data preprocessing tasks. Our method offers valuable insights into datasets and effectively handles standard data preprocessing tasks. It begins by detecting any issues in the data and presents them to the user through engaging visualizations. Then, it recommends the most effective data cleaning and preparation techniques by evaluating state-of-the-art methods. To evaluate the effectiveness of our approach, we applied the architecture to ten diverse datasets for automatic data preprocessing before feeding them into a Machine Learning algorithm. We compared the results obtained using our approach with those achieved by manually preprocessed data using the same algorithm. The results demonstrated that our approach not only simplified and streamlined the entire process but also significantly improved the model's performance.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.