Searches are performed for resonant and nonresonant Higgs boson pair production in the γγbb[over ¯] final state using 20 fb^{-1} of proton-proton collisions at a center-of-mass energy of 8 TeV recorded with the ATLAS detector at the CERN Large Hadron Collider. A 95% confidence level upper limit on the cross section times branching ratio of nonresonant production is set at 2.2 pb, while the expected limit is 1.0 pb. The difference derives from a modest excess of events, corresponding to 2.4 standard deviations from the background-only hypothesis. The limit observed in the search for a narrow X→hh resonance ranges between 0.7 and 3.5 pb as a function of the resonance mass.