Kalmadi, Bhanot may be forced out