Mildewobscured brush inscriptions on ancient architectural objects are often illegible and not easy to decipher. The traditional character image segmentation algorithm does not take full account of pollution and character feature information, and can lead to incorrect interpretations. This paper presents a multifeatureguided “Grab Cut” segmentation algorithm used in combination with image polarization information. First, stokes solver was applied to the polarized images at four different angles (0°, 45°, 90° and 135°) to obtain the feature graphs of the degree of polarization. Then SLIC was used to make super pixel division of images of the inscriptions captured by a visible light camera, thus making it possible to extract super pixel feature distances of color and texture. Finally, “Grab Cut” segmentation was performed according to regional items (restrained by the degrees of polarization) and feature distances (guided by boundary items) to get segmentation results for the characters. Compared with other segmentation algorithms, which do not take into account degrees of polarization or feature distances, this method gave greatly improved segmentation.