By examining the loop gain βA as a function of frequency, we can determine whether the feedback amplifier is stable or not. The simplest and most effective means for doing this is through the use of a Bode plot for βA.
The difference between unity and the value of |βA| at the frequency of -180° shift, called the gain margin, is usually expressed in dB.
The difference between the phase angle at the frequency where |βA| =0dB and -180° is termed the phase margin.
If the β is reduced (less feedback is applied) shown in the figure below, then the magnitude plot is shifted down, phase margin increases. Thus, the worst case stability corresponds to β=1. Phase margin in an op-amp datasheet describes the stability of a unity gain buffer; other gains will have better phase margin. Loop gain must drop to unity before the non-inverting input phase shift reaches -180°. In other words, the greater the phase margin, the more stable the feedback system. There are designs (e.g., photo-diode trans-impedance amplifier and heavy capacitive loads) that will need a special compensation network to achieve reasonable stability.
Let's use an example below to demonstrate gain and phase margin in op-amp stability. The figure below shows various phase margins (30°, 45°, 65°, 90°) vs frequency plot when the closed-loop gain is 1. At 30°, 45°, the frequency step response shows a large peaking and ringing indicate potential oscillations. At 65°, the step response is negligible. At 90°, the op-amp is the most stable with the trade-off of slow timing response. Phase margin needs to be specified at a closed-loop gain based on the the combination of the op-amp and its feedback components.