Cognitive radio is regulated to permit efficient access to leftover licensed bands by unlicensed users in an opportunistic manner. Spectrum sensing is aimed to protect the primary users from interferences of the secondary users. Energy detection constitutes the most dominant method for spectrum sensing in cognitive radio and many other systems owing to its non-coherent structure as well as its simplicity and applicability. The sensing capability of wireless detectors in unreliable environments is challenged by adverse effects of multipath fading. Motivated by this, in the presented work the performance analysis of the energy detector is investigated over η-μ and α-μ fading channels. Novel closed-form expressions are derived for the detector using the area under the receiver operating characteristic curve (AUC) measure. The derived analytical results are verified by numerical computations and Monte Carlo simulations.