Relating small-scale structures to large-scale appearance is a key element in material appearance design. Bi-scale material design requires finding small-scale structures - meso-scale geometry and micro-scale BRDFs - that produce a desired large-scale appearance expressed as a macro-scale BRDF. The adjustment of small-scale geometry and reflectances to achieve a desired appearance can become a tedious trial-and-error process. We present a learning-based solution to fit a target macro-scale BRDF with a combination of a meso-scale geometry and micro-scale BRDF. We confront challenges in representation at both scales. At the large scale we need macro-scale BRDFs that are both compact and expressive. At the small scale we need diverse combinations of geometric patterns and potentially spatially varying micro-BRDFs. For large-scale macro-BRDFs, we propose a novel 2D subset of a tabular BRDF representation that well preserves important appearance features for learning. For small-scale details, we represent geometries and BRDFs in different categories with different physical parameters to define multiple independent continuous search spaces. To build the mapping between large-scale macro-BRDFs and small-scale details, we propose an end-to-end model that takes the subset BRDF as input and performs classification and parameter estimation on small-scale details to find an accurate reconstruction. Compared with other fitting methods, our learning-based solution provides higher reconstruction accuracy and covers a wider gamut of appearance.