This paper describes the algorithm for the numerical solution of the diffraction problem of waveguide modes at the joint of two open planar waveguides. For planar structures under consideration, we can formulate a scalar diffraction problem, which is a boundary value problem for the Helmholtz equation with a variable coefficient in two-dimensional space. The eigenmode problem for an open three-layer waveguide is the Sturm-Liouville problem for a second-order operator with piecewise constant potential on the axis, where the potential is proportional to the refractive index. The described problem is singular and has a mixed spectrum and therefore the Galerkin method can not be used in this definition. One way to adapt the Galerkin method for the problem solution is to artificially limit the area, which is equivalent to placing the open waveguide in question in a hollow closed waveguide whose boundaries are remote from the real boundaries of the waveguide layer of the open waveguide. Thus, we obtain a diffraction problem on a finite interval and with a discrete spectrum, which can be solved by the projection method, as done in this paper. © The Authors, published by EDP Sciences, 2018.