Speaker miffed on frequent by-polls in Andhra