This paper proposes a search-based face annotation framework that mines weakly labeled facial images from the web. It introduces an unsupervised label refinement approach using machine learning to improve the noisy and incomplete labels of web images. A clustering-based approximation algorithm is also proposed to speed up the labeling process and improve scalability. Experimental results on a large test dataset show the label refinement algorithms significantly boost performance of the search-based face annotation scheme.