This intriguing study looks at a time when the very act of smiling was improper and radical – and the moment when this changed. By Kathryn Hughes