The brothers have hired engineers from top hedge funds, enlisted a bank and engaged regulators with the aim of opening their exchange in the coming months.