Characters recorded in ancient calligraphy and painting works are precious historical information, and the extraction and recognition of faded characters is the basis for the excavation and display of historical value. This paper proposes for the first time a method for the automatic extraction and recognition of characters of faded texts in ancient calligraphy and painting works based on the combination of the spectral enhancement index and LeNet-5:utilizing the advantages of nondestructive hyperspectral-imaging detection and a wide spectral range to acquire hyperspectral data from ancient calligraphy and painting works, analyze the spectral characteristics of texts and backgrounds and construct handwriting enhancement indexes, so as to realize the enhancement of faded texts in ancient calligraphy and painting works; constructing the LeNet-5 convolution neural network which uses a handwritten Chinese character set for training to automatically recognize the binary image of the extracted text. Finally, some faded characters in the painting, Lun Dao Tu, by Zhang Shibao (1805—1878), a famous artist in the Qing Dynasty, were used as examples to verify the proposed approach. The accuracy of character recognition is 70.8%. The results show that the method proposed in this paper can effectively improve the intelligentization of extraction and recognition of faded characters in ancient calligraphy and painting works.