Achieving Fully Autonomous AI-Driven Data Pipelines to Exploring Zero-Touch Automation for Efficient and Scalable Data Engineering Solutions

Abstract

The paper discusses the methods of application of the Random Forest Regressor and Linear Regression model to predict the price of houses when using the California Housing Prices dataset. The performance of the two models and the Random Forest model is measured by Mean Squared Error (MSE) and R-squared (R^2), whereby the latter has a higher accuracy. The results highlight the importance of the designed machine learning algorithms to improve predictive accuracy and the processing of complex data. The paper presents the potential of self-directed models in automating data pipelines and offers an insight on how the prediction of housing prices can be optimized, and how research in data engineering may be performed later on.