This paper addresses semantic segmentation by using an augmented hypothesis graph with contextual information. It mines background information from a training set and applies it to the unary terms of foreground regions in a fully connected conditional random field model over overlapping segment hypotheses. The final segmentation is obtained through maximum a posteriori inference followed by post-processing steps. Experimental results on PASCAL VOC 2012 and MSRC-21 data sets show the approach achieves state-of-the-art performance by incorporating contextual cues such as image classification and object detection.