Chest X-ray (CXR) is commonly performed as an initial investigation in COVID-19, whose fast and accurate diagnosis is critical. Recently, deep learning has a great potential in detecting people who are suspected to be infected with COVID-19. However, deep learning resulting with black-box models, which often breaks down when forced to make predictions about data for which limited supervised information is available and lack inter-pretability, still is a major barrier for clinical integration. In this work, we hereby propose a semantic-powered explainable model-free few-shot learning scheme to quickly and precisely diagnose COVID-19 with higher reliability and transparency. Specifically, we design a Report Image Explanation Cell (RIEC) to exploit clinically indicators derived from radiology reports as interpretable driver to introduce prior knowledge at training. Meanwhile, multi-task collaborative diagnosis strategy (MCDS) is developed to construct N-way K-shot tasks, which adopts a cyclic and collaborative training approach for producing better generalization performance on new tasks. Extensive experiments demonstrate that the proposed scheme achieves competitive results (accuracy of 98.91%, precision of 98.95%, recall of 97.94% and F1-score of 98.57%) to diagnose COVID-19 and other pneumonia infected categories, even with only 200 paired CXR images and radiology reports for training. Furthermore, statistical results of comparative experiments show that our scheme provides an interpretable window into the COVID-19 diagnosis to improve the performance of the small sample size, the reliability and transparency of black-box deep learning models. Our source codes will be released on https://github.com/AI-medical-diagnosis-team-of-JNU/SPEMFSL-Diagnosis-COVID-19.