Tooth cavity detection is a critical aspect of orthodontic diagnosis, facilitating timely intervention to prevent further dental deterioration. Traditional methods for cavity detection, such as visual inspection and radiographic analysis, often suffer from limitations in accuracy and consistency, highlighting the need for advanced computational techniques.Therefore, in this study, we propose a novel approach combining an optimized Fuzzy Neural Network (OptFuzNet) with a modified Generative Adversarial Network (m-GAN) for accurate and efficient cavity detection. First, we pre-process dental images to reduce noise and improve image quality. The m-GAN is then employed to generate synthetic cavity images, augmenting the training dataset and improving model robustness. Then, the GLCM features such as contrast, Homogeneity, Energy, Correlation, and Dissimilarity are extracted from each image. Subsequently, the selected features are given to the OptFuzNet classifier to classify an image as a tooth or cavity. To improve the efficiency of the FuzNet classifier, the weight values are optimally selected using The Lyrebird Optimization Algorithm (LyOA). Our experimental results, validated on a benchmark dataset, demonstrate the superior performance of the proposed approach compared to traditional methods, achieving a high accuracy of 94.29% in cavity detection. The proposed framework offers a promising solution for accurate tooth cavity detection, thereby facilitating early diagnosis and effective orthodontic treatment planning.