We examine the nuclear medium effect on the charge symmetry breaking (CSB) caused by isospin mixing of two neutral vector mesons interacting with nucleons in neutron matter. It is found that isospin mixing is strongly enhanced in neutron matter as the neutron density increases and the charge symmetry is broken to a large extent. We illustrate the influence of the enhanced CSB for bulk quantities by calculating the mass of a fictitious neutron star composed of only neutrons. For the stars with central densities around the normal density of nuclear matter, the masses are reduced by as much as 35% in comparison with those emerging from an ordinary relativistic nuclear model with no CSB considered. The effect of CSB is attenuated for stars with higher central densities which actual neutron stars are supposed to have, although the charge symmetry remains broken to a large extent.