The problem of directed waveguide mode scattering in an irregular planar optical waveguide (PWG) is solved using the theory of perturbations. The solution of the inverse waveguide scattering problem consists of restoring the autocovariance function and determining irregularity parameters by the measuring data of a scattering diagram in the far zone. The new complex solution algorithm for the described inverse waveguide scattering problem is a combination of classic regularization and the quasi-optimal filtration (smoothing) of the measured data. The computer simulation enables us to show that our method makes it possible to receive an approximately correct solution of the inverse problem with a root mean square (rms) error in restoring the given autocovariance functions of no more than 35% in the presence of high real noise (SNR≥1). The statistical parameters of the irregularities in this case can be determined with an error less than 15 to 30%. The roughness parameters determined in the experiment are in good agreement with the parameters obtained by a scanning electron microscope (SEM) and by a standard mechanical profilometer. © 2005 Society of Photo-Optical Instrumentation Engineers.