State of infrastructure not good enough, need to do more: PM