Previous neuroimaging studies suggest that childhood maltreatment (CM) can affect brain function and structure and constitutes a potent risk of developing depression. The present study attempts to differentiate the effect of CM from the impact of depression diagnosis on brain structure. Eighty-four patients with first-episode depression and 84 controls participated in this study. All subjects underwent a structural magnetic resonance imaging (MRI) scan. Voxel-based morphometric analysis of variance (ANOVA), between-group comparison, and regression analyses were performed on a whole-brain level to identify the unbiased effects of CM and depression on brain structure. Our data showed different main effects of CM and depression on brain structure separately: the significant main effect of CM on fronto-limbic areas, visual cortex and cerebellum, and the main effect of depression on orbitofrontal regions and bilateral temporal lobes. The maltreatment-by-diagnosis interaction effect was found located on the bilateral prefrontal cortex. The CM severity was negatively related with the inferior occipital volume and positively with volumes in the left dorsolateral prefrontal cortex, right caudate, and left middle temporal gyrus. This study suggested that early life stress like CM is associated with structural abnormalities of the fronto-limbic regions that are commonly regarded as the psychopathological consequence of depression. The effect of CM on brain structure should be considered in future neuroimaging research of depression.