In this paper the algorithm of finding eigenvalues and eigenfunctions for the leaky modes in a three-layer planar dielectric waveguide is considered. The problem on the eigenmodes of open three-layer waveguides is formulated as the Sturm-Liouville problem with the corresponding boundary and asymptotic conditions. In the case of guided and radiation modes of open waveguides, the Sturm-Liouville problem is formulated for self-adjoint second-order operators on the axis and the corresponding eigenvalues are real quantities for dielectric media. The search for eigenvalues and eigenfunctions corresponding to the leaky modes involves a number of difficulties: the boundary conditions for the leaky modes are not self-adjoint, so that the eigenvalues can turn out to be complex quantities. The problem of finding eigenvalues and eigenfunctions will be associated with finding the complex roots of the nonlinear dispersion equation. In the present paper, an original scheme based on the method of finding the minimum of a function of several variables is used to find the eigenvalues. The paper describes the algorithm for searching for eigenvalues, the algorithm uses both symbolic transformations and numerical calculations. On the basis of the developed algorithm, the dispersion relation for the weakly flowing mode of a three-layer open waveguide was calculated in the Maple computer algebra system. © The Authors, published by EDP Sciences, 2018.