One of the challenges of machining process is to improve the quality of machined surface by reducing the vibration of cutting tools. The research aims to suppress vibration using composite boring bars with an enhanced damping capacity. A new design of boring bars with different cross-sections is considered. Static and dynamic behavior of the proposed tools is investigated. A mathematical model for determining the eigenfrequency is proposed, and it is compared with computer simulation and experimental results. The validity of the proposed models is verified by conducting experimental machining tests in order to study the changes in vibro-acoustic signals depending on the cross-sections of the toolholder. The results show that the composite material significantly improves damping of boring bars, which leads to a reduction in the vibration compared to conventional boring bars. © 2019, Springer-Verlag London Ltd., part of Springer Nature.