Gen AI For ELT (Extract, Load, Transfer) in Streaming Application with Databricks/Snow Flakes

Abstract

This paper provides a system literature review of the implementation of Generative Artificial Intelligence (GenAI) in ELT (Extract, Load, Transform) pipelines to incoming applications, concentrating on the Databricks and Snowflake services. The review is based on the summary of the results of fifty chosen studies devoted to the study of GenAI-based automation, scalability and adaptive transformation in real-time data processing. It is shown that GenAI drastically increases the intelligence of the pipeline and its work efficiency and allows working with dynamic schemas and with customised analytics. Nevertheless, data quality, data governance, explainability, and human control are still largely on the agenda. The research suggests a pathway to hybrid ELT architectures to combine GenAI automation and sound governance procedures to establish reliability and responsible execution in the streaming setting.