Understanding of the role of focal inflammation, a treatable feature, on neuro-axonal injury, is paramount to optimize neuroprotective strategy in MS. To quantify the impact of focal inflammatory activity on the rate of neuro-axonal injury over the MS course. We quantified the annualized rates of change in peripapillary retinal nerve fiber layer, ganglion cell plus inner plexiform layer (GCIPL), whole-brain, gray matter and thalamic volumes in patients with and without focal inflammatory activity in 161 patients followed over 5 years. We used mixed models including focal inflammatory activity (the presence of at least one relapse or a new/enlarging T2-FLAIR or gadolinium- enhancing lesion), and its interaction with time adjusted by age, sex, use of disease-modifying therapies and steroids, and prior optic neuritis. The increased rate of neuro-axonal injury during the first five years after onset was more prominent among active patients, as reflected by the changes in GCIPL thickness (p = 0.02), whole brain (p = 0.002) and thalamic volumes (p < 0.001). Thereafter, rates of retinal and brain changes stabilized and were similar in active and stable patients. Focal inflammatory activity is associated with neurodegeneration early in MS which reinforces the use of an early intensive anti-inflammatory therapy to prevent neurodegeneration in MS.