Modeling and understanding properties of materials from first principles require knowledge of the underlying atomistic structure. This entails knowing the individual chemical identity and position of all atoms involved. Obtaining such information for macro-molecules, nano-particles, and clusters and for the surface, interface, and bulk phases of amorphous and solid materials represents a difficult high-dimensional global optimization problem. The rise of machine learning techniques in materials science has, however, led to many compelling developments that may speed up structure searches. The complexity of such new methods has prompted a need for an efficient way of assembling them into global optimization algorithms that can be experimented with. In this paper, we introduce the Atomistic Global Optimization X (AGOX) framework and code as a customizable approach that enables efficient building and testing of global optimization algorithms. A modular way of expressing global optimization algorithms is described, and modern programming practices are used to enable that modularity in the freely available AGOX Python package. A number of examples of global optimization approaches are implemented and analyzed. This ranges from random search and basin-hopping to machine learning aided approaches with on-the-fly learnt surrogate energy landscapes. The methods are applied to problems ranging from supported clusters over surface reconstructions to large carbon clusters and metal-nitride clusters incorporated into graphene sheets.