This paper attempts to mine semantic context information from video surveillance of traffic scenes to build an intelligent system. It learns object-specific context about pedestrians and vehicles to classify them with a co-trained classifier. It also learns scene-specific context for each object type, like motion patterns, paths, and entry/exit points. This semantic context information improves object detection, tracking, and abnormal event detection. Experimental results show the effectiveness of these semantic context features for multiple traffic scenes.