Under the persistent action of weathering factors, the microstructure of earthen sites continues to deteriorate, leading eventually to the development of macro diseases. The variation of microstructural characteristics reflects the inherent weathering degree of soil materials. In order to study the correlation between deterioration and the micro-pore structure of the Suoyang earthen site, a low temperature-liquid nitrogen adsorption test was conducted on soil specimens sampled from the city wall in different directions at the site for the determination of specific surface area, pore volume and pore size distribution characteristics, and the fractal dimension of the specimens was calculated based on the FHH fractal model. The research data show that Suoyang soils demonstrated a high deviation rate in specific surface area, and obvious difference in specific surface area of the specimens in different directions. The soils were mainly the micropores with a high coefficient of variation in pore volume and large dispersion degree, corresponding to weathering difference between them. The pore morphology included slit, wedge, “ink bottle”, tubular hole, and other complex forms. The pores less than 2.765 nm were mostly closed pores. In combination with the fractal dimension, it is found the soil specimens featured obvious fractal characteristics, irregular pore space structure, strong heterogeneity and large surface roughness. The fractal dimension had a certain correlation with the weathering or deterioration degree. The quantitative characterization of soil pores is of importance for the evaluation of weathering and deterioration of earthen sites.