Zn1-xMgxO (x=0, 0.05, 0.1, 0.2 and 0.3, respectively) thin films have been synthesized by sol-gel method on glass substrates. The structure, morphology and optical properties of the samples have been studied by X-ray diffractonmeter (XRD), scanning probe microscope, UV-visible spectrophotometer, fluorescence spectrophotometer and spectroscopic ellipsometer, respectively. The XRD result shows that all the films have hexagonal wurtzite structure; no phase segregation is observed. The surfaces of Zn1-xMgxO thin films are smooth and the root mean square (RMS) roughness of the samples is only several nanometers. The transmittance spectra reveal that all samples have high transmittance above 90%, with Mg doping content increase, the optical band gap increases from 3.27eV to 3.77eV. The photoluminescence spectra show that all samples have two emission peaks in ultraviolet and violet region, a blue shift of ultraviolet emission is observed. The refractive indexes of all samples decrease with the increase of wavelength ranging from 350nm to 900nm. The refractive index changes apparently by varying Mg content, which has potential application in research of optical materials and the design of optical devices.