Shoreline is described as an intersection of coastal land and water surface indicating water edge movements as the tides rise and fall. Remote sensing technology can provide a wide range dynamicmonitoring of the shoreline. However, traditional hard classification methods are mainly used to extract coastal waterline at the pixel level, and achieve the low accuracy. Whereas sub-pixel coastal waterline extraction is an attractive and challenging task due to the complex features in the coastal region. Therefore, an improved sub-pixel coastal waterline extraction method (ISPCW) is presented to achieve the higher accuracy of coastal waterline extraction. Firstly, a Water-Vegetation-Impervious-Soil (W-V-I-S) model is presented to detect W-V-I-S mixed pixels and determine endmember spectrum in the coastal region. Secondly, the linear spectral mixture unmixing technique based on Fully Constrained Least Squares (FCLS) is applied to the W-V-I-S mixed pixels for water abundance estimation; and finally, spatial attraction model is used to extract coastal waterline. In the experiment performed on EO-1 Hyperion data of Shanghai study area, Multiple Endmember Spectral Mixture Analysis (MESMA), Mixture Tuned Matched Filtering (MTMF), Sequential Maximum Angle Convex Cone (SMACC), and Constrained Energy Minimization (CEM), and classical Normalized Difference Water Index (NDWI) methods are chosen for the coastal waterline extraction comparison. The results indicate that the proposed ISPCW method achieved the best accuracy of 0.38 pixels in the experiment, and the accuracy of ISPCW method improved by 22.4%,33.3%,42.4%, 43.2%, and 51.3%compared with MESMA, MTMF, SMACC, CEM, and NDWI methods, respectively. Therefore, from these results, the ISPCW method exhibits better performance for coastal waterline extraction than the traditional pixel level method and sub-pixel level method, and can be effectively applied to coastal waterline extraction in the coastal region.