More athletes could pull out, say Aussies