Financial firms are creating a murky, ad hoc market where the red-hot stocks of closely held technology companies trade largely out of sight of regulators, other investors and the companies themselves.