A Comprehensive Review of Agentic Artificial Intelligence for Autonomous Data Pipeline Management: Advances, Challenges, and Future Directions

Shahnawaz, Md

doi:https://doi.org/10.55041/ijcope.v2i7.250

Volume 02, Issue 7

Published on: July 2026

A COMPREHENSIVE REVIEW OF AGENTIC ARTIFICIAL INTELLIGENCE FOR AUTONOMOUS DATA PIPELINE MANAGEMENT: ADVANCES, CHALLENGES, AND FUTURE DIRECTIONS

Md Shahnawaz

Dr. Jeetendra Singh Yadav

Computer Science & Engineering

Bhabha University, Bhopal

DOI:https://doi.org/10.55041/ijcope.v2i7.250

Article Status

Plagiarism Passed Peer Reviewed Open Access

Available Documents

Download PDF Review Report

Abstract

The rapid development of cloud technology, big data, and artificial intelligence has changed the way data pipelines are managed in today’s world. Conventional data pipeline management systems depend on rule-based automation and manual intervention making them less effective in managing dynamic workloads, detecting anomalies, recovering from faults, and optimizing resources. The recent advances in Agentic Artificial Intelligence, LLMs, reinforcement learning, and multi-agent systems have given rise to the autonomous systems that can perform monitoring, diagnosis, recovery, and optimization on their own. This review studies the state-of-the-art AI information pipeline management processes, intelligent work flow orchestration, autonomous decision making, and self-healing systems. The paper reviews the existing methodologies, including ETL automation, metadata-based data ingestion, optimization through reinforcement learning, and agent-based orchestration systems, and presents their advantages, limitations, and potential fields of application. Moreover, the review identifies main research problems faced by researchers and developers including explainability, interoperability, scalability, security, and adaptation in heterogeneous cloud environments.

Keywords: Agentic Artificial Intelligence, Autonomous Data Pipeline Management, Reinforcement Learning, Large Language Models, Multi-Agent Systems, Intelligent Workflow Orchestration.

How to Cite this Paper

Shahnawaz, M. (2026). A Comprehensive Review of Agentic Artificial Intelligence for Autonomous Data Pipeline Management: Advances, Challenges, and Future Directions. International Journal of Creative and Open Research in Engineering and Management, <i>02</i>(7), 1-9. https://doi.org/10.55041/ijcope.v2i7.250

Shahnawaz, Md. "A Comprehensive Review of Agentic Artificial Intelligence for Autonomous Data Pipeline Management: Advances, Challenges, and Future Directions." International Journal of Creative and Open Research in Engineering and Management, vol. 02, no. 7, 2026, pp. 1-9. doi:https://doi.org/10.55041/ijcope.v2i7.250.

Shahnawaz, Md. "A Comprehensive Review of Agentic Artificial Intelligence for Autonomous Data Pipeline Management: Advances, Challenges, and Future Directions." International Journal of Creative and Open Research in Engineering and Management 02, no. 7 (2026): 1-9. https://doi.org/https://doi.org/10.55041/ijcope.v2i7.250.

Search & Index

References

Chowdhury, Asim. "High-Speed Data Ingestion and In-Memory Processing." Oracle 23AI & ADBS in Action: Exploring New Features with Hands-On Case Studies. Berkeley, CA: Apress, 2026. 335-360.

Kotla, Mutha Ravi Tej. "AI-driven data integration for mergers and acquisitions: Automating entity resolution and system consolidation." International Journal of Engineering & Extended Technologies Research (IJEETR)1 (2026): 198-201.

Gaddipati, Soma Sekhar. "Hybrid data integration architectures: Combining Informatica and cloud-native services for scalable enterprise data systems." Preprint. (2026)

Mohanasundaram, S., et al. "6G Deep Federated Optimization for Intelligent Data Quantum‐Resistant Routing and Monitoring System Using Game Theory." International Journal of Communication Systems1 (2026): e70326.

Rahman, Mizanur, et al. "Enterprise AI driven data management framework for cross industry decision intelligence and operational optimization." Shamsun, Enterprise AI Driven Data Management Framework for Cross Industry Decision Intelligence and Operational Optimization (February 24, 2026)(2026).

Malhotra, Arjun. "Autonomous Data Products: Enabling AI-Driven Data Interoperability in Cloud Architectures." International Journal of Data Engineering and Intelligent Computing1 (2026): 22-30.

Alangaram, S., et al. "Sales guard AI-driven decision intelligence platform for business optimization." International Journal of Engineering & Extended Technologies Research (IJEETR)3 (2026): 5022-5031.

Khemka, Akshat, and Gaurav Raj. "Unstructured Data Ingestion: Best Practices for Acquiring, Storing, and Processing Data from 200+ External Sources." (2025).

Rucco, Chiara, Antonella Longo, and Motaz Saad. "MIND: A Metadata-driven INgestion Design pattern for efficient data ingestion." Big Data Research(2025): 100574.

Rucco, Chiara, Antonella Longo, and Motaz Saad. "Enhancing Data Ingestion Efficiency in Cloud-Based Systems: A Design Pattern Approach: C. Rucco et al." Data Science and Engineering(2025): 1-16.

Ethical Compliance & Review Process

•All submissions are screened under plagiarism detection.
•Review follows editorial policy.
•Authors retain copyright.
•Peer Review Type: Double-Blind Peer Review
•Published on: Jul 30 2026

CCBYNC

This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. You are free to share and adapt this work for non-commercial purposes with proper attribution.

View License

Back to Volume 02, Issue 7 View All Issues Next Article

← Previous Article

A Comparative Study on the Impact of Brand Endorsements by Sports Personalities and Movie Artists on Consumer Purchase Behavior: Evidence from Secondary Data

Next Article →

A Hybrid Explainable Artificial Intelligence Framework for Scalable and Intelligent Data Processing in Cloud-Based Distributed Computing Environments