Most Japanese water utilities are public entities. They operate their services independently at the municipal level. There is no regulation to promote cost efficiency in their performance. The water price is set to obtain revenue to cover costs (i.e., operating and capital costs). Therefore, it is possible that some water utilities do not try to minimize costs during their operation and inefficiencies accrue. We used the stochastic cost frontier analysis, with a time-varying decay inefficiency model, on data from 1184 Japanese water utilities from 2014 to 2018 to estimate their cost efficiency. The results show that their average cost inefficiency is more than 82%, and the degree of inefficiency has increased over time.