This post is by Chou-han Yang, principal engineer at BloomReach. At BloomReach, we are constantly crawling our customers’ websites to ensure their quality and to obtain the information we need to r...