The purpose of this article is to analyze mechanisms of Loss-based Overload Control, LBOC and Rate-based Overload Control, RBOC for server overload control of the Session Initiation Protocol (SIP). Overloading occurs when a server is unable to process an entire incoming message flow due to a lack of resources. Standards of IETF SOC recommend several overload control mechanisms, including most important LBOC and RBOC. This article proposes the mechanisms of LBOC and RBOC and describes the results of the comparative analysis of these mechanisms based on the hysteresis control over incoming stream of signaling messages. The system resides in one of three modes (normal, overload, discard) based on thresholds and a size of an input queue. A signal message source implements the Markov-modulated Poisson process, MMPP-2 model. The leaky bucket algorithm is applied to limit the number of incoming messages in the implementation of RBOC mechanism. The comparison of results showed that RBOC mechanism based on the hysteresis control over incoming stream of signaling messages demonstrates higher effectiveness of the congestion control, as a result of which the average time in the overload mode is less than for LBOC mechanism. However, LBOC mechanism has the ability to maintain its high efficiency for all RTT values on the same thresholds, while RBOC mechanism needs to have its own threshold dynamic control mechanism for different RTT values to supply maximum efficiency. © Springer Nature Switzerland AG 2018.