Artificial intelligence (AI) systems are increasingly studied for tasks such as medical-image analysis, risk prediction, and clinical decision support. However, strong predictive performance does not by itself show why a model reached a particular conclusion or whether that conclusion is appropriate for a patient. This paper reviews explainable artificial intelligence (XAI) methods that can make healthcare predictions more understandable to clinicians and other stakeholders. A focused narrative review was conducted using foundational work on local explanations and Shapley-value-based feature attribution, a healthcare-focused review, and World Health Organization guidance on AI ethics. The paper compares model-specific and model-agnostic explanations, with particular attention to LIME, SHAP, and visual explanation methods. The review indicates that explanations can support error analysis, communication, and model auditing, but they should not be treated as proof that a prediction is correct, fair, or clinically safe. Explanation quality depends on data quality, model behaviour, the intended user, and the clinical context. The paper proposes a practical evaluation framework combining predictive performance, explanation fidelity, stability, clinical usefulness, fairness checks, and human oversight. It concludes that XAI should be integrated into the full lifecycle of clinical AI systems rather than added only after model development.

Index Terms—Explainable AI, healthcare, clinical decision support, interpretability, SHAP, LIME.