Unable to discriminate between two classes while using HRNet model

[sparshgarg23]

So,I was experimenting with the HRNet model and I trained it on my dataset (which consists of 8 classes).
Dataset distribution is as follows
{'F_AF': 480, 'F_AS': 444, 'F_CA': 348, 'F_LA': 479, 'M_AF': 456, 'M_AS': 372, 'M_CA': 468, 'M_LA': 396
And I am using weighted sample in the dataloader with weights as follows
[0.0021, 0.0023, 0.0029, 0.0021, 0.0022, 0.0027, 0.0021, 0.0025
Although the model is able to perform well with a validation accuracy of approximately 83%,I observed that it ends up getting confused between two of the classes .

Some sample images belonging to the two classes are enclosed

I am resizing my images to 224x224,since HRnet is able to focus on high resolution features,I decided not to resize the images,but that ended up huriting the performance.
When I run the model on class 3 image,the predicted class comes out to be C1 with score 67%,the second top class is C3 with score 33%
the classifier which I am using on top of the HRNet features is shown below

(classifier): Sequential(
    (0): Linear(in_features=2048, out_features=625, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.3, inplace=False)
    (3): Linear(in_features=625, out_features=256, bias=True)
    (4): ReLU()
    (5): Linear(in_features=256, out_features=128, bias=True)
    (6): ReLU()
    (7): Linear(in_features=128, out_features=8, bias=True)

Any reason as to why the model is getting confused between these 2 classes,and what steps can I take to ensure the model doesn't get confused between these two classes