Forced into legal action, says Swamy