The rapid expansion of online education following the COVID-19 pandemic has elevated the demand for robust, automated mechanisms to safeguard academic integrity. Traditional invigilation methods fail to scale in distributed learning environments, while existing commercial proctoring platforms suffer from high costs, opaque proprietary architectures, algorithmic bias, and significant privacy concerns. This paper presents an AI-based online examination proctoring system that integrates computer vision, deep learning-based object detection, and browser-based behavioral analysis to detect cheating behaviors in real time. The proposed framework employs MediaPipe for face detection and facial landmark extraction, YOLOv8 Nano for detecting prohibited objects such as mobile phones, and a Perspective-n-Point (PnP) method for three-dimensional head-pose estimation. Browser tab-switching behavior is monitored using the W3C Page Visibility API. Generated alerts are persisted in a MongoDB database and propagated to an administrative dashboard through WebSocket-based Socket.IO communication with sub-500 ms latency. Experimental evaluation across 47 participants and six simulated examination sessions yielded a weighted precision of 93.7%, recall of 91.2%, and F1-score of 92.4%. The system provides a scalable, transparent, and cost-effective open-source alternative for secure online examinations.