We investigate modified theories of gravity in the context of teleparallel geometries with possible Gauss-Bonnet contributions. The possible coupling of gravity with the trace of the energy-momentum tensor is also taken into account. This is motivated by the various different theories formulated in the teleparallel approach and the metric approach without discussing the exact relationship between them. Our formulation clarifies the connections between different well-known theories. For instance, we are able to formulate the correct teleparallel equivalent of Gauss-Bonnet modified general relativity, amongst other results. Finally, we are able to identify modified gravity models which have not been studied in the past. These appear naturally within our setup and would make a interesting starting point for further studies.