Basement hackers and amateur mathematicians are competing to improve the program that Netflix uses to recommend DVDs — and to win $1 million in the process.