The investigation of the electromagnetic field in a regular waveguide filled with a homogeneous substance reduces to the study of two independent boundary value problems for the Helmholtz equation. In the case of a waveguide filled with an inhomogeneous substance, a relationship arises between the modes of these two problems, which in numerical experiments can not always be fully taken into account. In this paper, we will show how to rewrite the Helmholtz equations in the “Hamiltonian form” to express this connection explicitly. In this case, the problem of finding monochromatic waves in a waveguide with arbitrary filling will be reduced to an infinite system of ordinary differential equations in a properly constructed Hilbert space. The results of numerical experiments on finding normal waves, realized in the computer algebra system Sage, are presented. © 2017 Informa UK Limited, trading as Taylor & Francis Group.