Objectives: Establishing objective and quantitative imaging markers at individual level can assist in accurate diagnosis of Major Depressive Disorder (MDD). However, the clinical heterogeneity of MDD leads to a decrease in recognition accuracy, to address this issue, we propose the Windowed Attention Aggregation Network (WAAN) for a medium-sized functional Magnetic Resonance Imaging (fMRI) dataset comprising 111 MDD and 106 Healthy Controls (HC). Methods: The proposed WAAN model is a dynamic temporal model that contains two important components, Inner-Window Self-Attention (IWSA) and Cross-Window Self-Attention (CWSA), to characterize the MDD-fMRI data at a fine-grained level and fuse global temporal information. In addition, to optimize WAAN, a new Point to Domain Loss (p2d Loss) function is proposed, which intermediate guides the model to learn class centers with smaller class deviations, thus improving the intra-class feature density. Results: The proposed WAAN achieved an accuracy of 83.8 % (+/- 1.4 %) in MDD identification task in mediumsized site. The right superior orbitofrontal gyrus and right superior temporal gyrus (pole) were found to be categorically highly attributable brain regions in MDD patients, and the hippocampus had stable categorical attributions. The effect of temporal parameters on classification was also explored and time window parameters for high categorical attributions were obtained. Significance: The proposed WAAN is expected to improve the accuracy of personalized identification of MDD. This study helps to find the target brain regions for treatment or intervention of MDD, and provides better scanning time window parameters for MDD-fMRI analysis.