In the paper we construct a method for approximate solution of the waveguide problem for guided modes of an open irregular waveguide transition. The method is based on straightening of the curved waveguide boundaries by introducing new variables and applying the Kantorovich method to the problem formulated in the new variables to get a system of ordinary second-order differential equations. In the method, the boundary conditions are formulated by analogy with the partial radiation conditions in the similar problem for closed waveguide transitions. The method is implemented in the symbolic-numeric form using the Maple computer algebra system. The coefficient matrices of the system of differential equations and boundary conditions are calculated symbolically, and then the obtained boundary-value problem is solved numerically using the finite difference method. The chosen coordinate functions of Kantorovich expansions provide good conditionality of the coefficient matrices. The numerical experiment simulating the propagation of guided modes in the open waveguide transition confirms the validity of the method proposed to solve the problem. © 2016 SPIE.