Nasdaq OMX Group is testing a new use of the technology that underpins the digital currency bitcoin, in a bid to transform the trading of shares in private companies.