The Weiler-Atherton algorithm clips concave polygons by tracing intersection points between the subject and clip polygons. It processes vertices clockwise, following the subject polygon boundary for outside-inside vertex pairs and the clip polygon boundary for inside-outside pairs. The algorithm traces the polygons, records intersection points, and clips the subject polygon accordingly until it reaches the starting vertex.