There is another possibility as well — when your data has not been labelled correctly. :)

I have been playing with the chexnet (cxr8/cxr14) dataset recently. The classifiers perform the worst for the class which has the highest number of images!