Graph-based neural networks have promising perspectives but are limited by electronic bottlenecks. Our work explores the advantages of optical neural networks in the graph domain. We propose an optical graph neural network (OGNN) based on inverse-designed optical processing units (OPUs) to classify graphs with optics. The OPUs, combined with two types of optical components, can perform multiply-accumulate, matrix-vector multiplication, and matrix-matrix multiplication operations. The proposed OGNN can classify typical non-Euclidean MiniGCDataset graphs and successfully predict 1000 test graphs with 100% accuracy. The OPU-formed optical-electrical graph attention network is also scalable to handle more complex graph data, such as the Cora dataset, with 89.0% accuracy.