Abstract: Electromagnetic waves propagating in a waveguide with a constant simply connected cross section S are considered under the condition that the material filling the waveguide is characterized by permittivity and permeability varying smoothly over the cross section S but constant along the waveguide axis. On the walls of the waveguide, the perfect conductivity conditions are imposed. It is shown that any electromagnetic field in such a waveguide can be represented via four scalar functions: two electric and two magnetic potentials. If the permittivity and permeability are constant, then the electric potentials coincide with each other up to a multiplicative constant, as do the magnetic potentials. Maxwell’s equations are written in the potentials and then in the longitudinal field components as a pair of integro-differential equations splitting into two uncoupled wave equations in the optically homogeneous case. The general theory is applied to the problem of finding the normal modes of the waveguide, which can be formulated as an eigenvalue problem for a self-adjoint quadratic pencil. At small perturbations of the optically homogeneous filling of the waveguide, the linear term of the pencil becomes small. In this case, mode hybridization occurs already in the first order and the phase deceleration indices of normal modes leave the real and imaginary axes only in the second order. © 2022, Pleiades Publishing, Ltd.