Simultaneous testing of rocks from which aggregate was derived (basalt, granite, dolomite, quartzite, gravel) and of 20 high-performance and ordinary concretes containing this aggregate revealed the factors that affect stress–strain properties of concrete. The mechanical property-related items addressed in this paper include rock mineral compositions and, excluding gravel, moduli of elasticity, stress–strain relationships, compressive strains at peak stress, and their compressive and splitting tensile strengths. The same parameters were determined for the concretes with the w/c ratio of 0.70, 0.58, 0.45 and 0.28, for which Poisson’s ratios were additionally found. The tests demonstrated that the suitability of aggregates for different concrete applications can be best predicted through the mineral composition, elastic modulus and stress–strain response of rocks from which the aggregate was sourced, while the compressive strength of the rocks was found to be a secondary factor governing the elastic properties of the concretes. The best stress–strain qualities were observed in the dolomite and basalt concretes and the worst in the granite concrete. The results obtained for the concretes and the data reported in the literature indicate that the values of elastic moduli in EN 1992-1-1 may be overestimated.