The standard clinical approach to assess the radiotherapy outcome in brain metastasis is through monitoring the changes in tumour size on longitudinal MRI. This assessment requires contouring the tumour on many volumetric images acquired before and at several follow-up scans after the treatment that is routinely done manually by oncologists with a substantial burden on the clinical workflow. In this work, we introduce a novel system for automatic assessment of stereotactic radiation therapy (SRT) outcome in brain metastasis using standard serial MRI. At the heart of the proposed system is a deep learning-based segmentation framework to delineate tumours longitudinally on serial MRI with high precision. Longitudinal changes in tumour size are then analyzed automatically to assess the local response and detect possible adverse radiation effects (ARE) after SRT. The system was trained and optimized using the data acquired from 96 patients (130 tumours) and evaluated on an independent test set of 20 patients (22 tumours; 95 MRI scans). The comparison between automatic therapy outcome evaluation and manual assessments by expert oncologists demonstrates a good agreement with an accuracy, sensitivity, and specificity of 91%, 89%, and 92%, respectively, in detecting local control/failure and 91%, 100%, and 89% in detecting ARE on the independent test set. This study is a step forward towards automatic monitoring and evaluation of radiotherapy outcome in brain tumours that can streamline the radio-oncology workflow substantially.