Knee Osteoarthritis (OA) is a fatal joint condition affecting the lives of many people around the world. This deadly joint disorder is determined by inflexibility of joints, severity and poor functioning of joints. One of the common risk factors of the disability include age. And these are detected by assessing visible syndromes, clinical data and also through many joint examining reports such as Magnetic Resonance Imaging (MRI), radiography reports and computed tomography images. These imaging tests show the disease progression and provide information about the internal structure of our body. However, the traditional way of treatment is difficult as identifying the disorder at an early stage is impractical. Handling this dispute, a deep learning-based model is proposed to assess the severity of Osteoarthritis in scales of Kellgren and Lawrence (KL) grades automatically. The main novelty of the proposed model is that it is based on  the combination of a top-most current deep learning models which is the called High Resolution Network ( HRNet ) and an attention module called CBAM(Convolutional Block Attention Module), which records multiple resolution representations of knee joint X-ray images. This proposed model is referred to as OsteoHRNet in this paper. This model is a convolutional neural network (CNN) which maintains high-scale representational features. Our proposed framework attained an accuracy of 71.74% and mean absolute error (MAE) of 0.31 with the Osteoarthritis Initiative (OAI) dataset which contains train, test and validation sets of five grades ranging from 0 to 4. This proposed framework gains best remarks over the existing models.