Feel it easing away

When to understand that you have just started drifting away from someone ? It is when all their sarcasms and mockery are not just fun to you anymore, but just mere disturbance.