The unique properties of graphene-based materials (GBMs) placed them among the most exciting nanomaterials of the past decade. Scientists and industry are looking forward to working with not only efficient but also safe, sustainable GBMs. Designing a safer-by-design GBM implies to acquire the knowledge of which physicochemical characteristics (PCCs) can increase toxicity. In this systematic review, we extracted data from the literature to provide the available information about the structure-activity relationship of GBMs. 93 papers studying a total of 185 GBMs are included. Graphene oxides (GOs) and few-layer graphenes (FLGs) are the most studied GBMs. While reduced graphene oxides were often classified as poorly oxidant and weakly cytotoxic, graphene quantum dots were mostly moderately or highly cytotoxic. FLGs demonstrated relationships between median size and oxidative stress, between lateral size and both cytotoxicity and oxidative stress, and between thickness and cytotoxicity. We also underline relationships between median size, lateral size, and thickness of GOs and oxidative stress. However, it appears difficult to highlight clear structure-activity relationships for most PCCs and biological end points because despite a large amount of available data, the GBMs are often too poorly characterized in terms of PCCs descriptors and the biological end points investigation is not standardized enough. There is an urgent need for a better standardization of the experimental investigation of both PCCs and biological end points to allow research teams to play a part in the collaborative work toward the construction of a safer-by-design GBM through a better understanding of their key toxicity drivers.