A steady increase in study retractions has alarmed journals and scientists, raising difficult questions about how research is conducted.