There was little excited bidding on the art collection of the Riggio family, who built their fortune on the Barnes & Noble bookstore chain — a caution flag for the art market.