Radiation losses in the interaction of superintense circularly polarized laser pulses with high-density plasmas can lead to the generation of strong quasistatic magnetic fields via absorption of the photon angular momentum (so-called inverse Faraday effect). To achieve the magnetic field strength of several Giga Gauss, laser intensities ≃1024W/cm2 are required which brings the interaction to the border between the classical and the quantum regimes. We improve the classical modeling of the laser interaction with overcritical plasma in the “hole boring” regime by using a modified radiation friction force accounting for quantum recoil and spectral cut-off at high energies. The results of analytical calculations and three-dimensional particle-in-cell simulations show that, in foreseeable scenarios, the quantum effects may lead to a decrease in the conversion rate of laser radiation into high-energy photons by a factor 2–3. The magnetic field amplitude is suppressed accordingly, and the magnetic field energy—by more than one order in magnitude. This quantum suppression is shown to reach a maximum at a certain value of intensity and does not grow with the further increase in intensities. The non-monotonic behavior of the quantum suppression factor results from the joint effect of the longitudinal plasma acceleration and the radiation reaction force. The predicted features could serve as a suitable diagnostic for radiation friction theories. © 2021, The Author(s).