Subject: Re: [ublas] Matrix multiplication performance
From: Oswin Krause (Oswin.Krause_at_[hidden])
Date: 2016-01-29 02:50:44
Hi,
I would like to contribute with some Benchmarks as well. Is the code
available for testing?
Best,
Oswin
On 2016-01-28 19:49, Michael Lehn wrote:
> In the meantime some results from my Haswell machine. It has 4 quad
> cores. But there are
> other jobs running so I went up to 8 threads. But anyway, the
> parallelisation is simple for
> the maximal matrix dimension N=M=K=4000 it reaches
>
> 1) 32.9 GFLOPS with 1 thread
> 2) 63 GFLOPS with 2 threads
> 3) 104.6 GFLOPS with 4 threads
> 4) 180.5 GFLOPS with 8 threads
>
> that is ok for a simple implementation but can be done better. Most
> of all it takes much too long (or
> much to big problem sizes to scale well). But for the moment we
> should focus on a good single threaded
> implementation and do the parallel stuff the right way later. As this
> will require more than just a single
> #pragma omp parallel for
>
>
>
> [lehn_at_node042 session4]$ g++ -Ofast -Wall -std=c++11 -DNDEBUG
> -DHAVE_FMA -I ../boost_1_60_0/ -fopenmp matprod.cc
> [lehn_at_node042 session4]$ export OMP_NUM_THREADS=1; ./a.out
> # m n k uBLAS: t1 MFLOPS Blocked: t2
> MFLOPS Diff nrm1
> 100 100 100 0.00119632 1671.79 0.00089036
> 2246.28 3.90562e-14
> 200 200 200 0.00322943 4954.44 0.00082579
> 19375.4 1.50135e-15
> 300 300 300 0.0108177 4991.81 0.00221283
> 24403.1 2.18434e-16
> 400 400 400 0.0247278 5176.35 0.00429661
> 29790.9 5.58593e-17
> 500 500 500 0.053677 4657.49 0.00822185
> 30406.8 1.94899e-17
> 600 600 600 0.0820133 5267.44 0.0136631
> 31617.9 1.16524e-17
> 700 700 700 0.129231 5308.34 0.0208619
> 32882.9 6.82385e-18
> 800 800 800 0.19206 5331.67 0.0309358
> 33100.8 4.08617e-18
> 900 900 900 0.272354 5353.34 0.0430091
> 33899.8 2.54117e-18
> 1000 1000 1000 0.372831 5364.36 0.0582482
> 34335.8 1.64011e-18
> 1100 1100 1100 0.494906 5378.8 0.0796676
> 33413.8 1.08587e-18
> 1200 1200 1200 0.642926 5375.43 0.098814
> 34974.8 7.43828e-19
> 1300 1300 1300 0.815164 5390.32 0.125541
> 35000.5 5.26152e-19
> 1400 1400 1400 1.04147 5269.48 0.154808
> 35450.4 3.81507e-19
> 1500 1500 1500 1.24516 5420.99 0.187327
> 36033.2 2.82388e-19
> 1600 1600 1600 1.5581 5257.68 0.236257
> 34674 2.12031e-19
> 1700 1700 1700 2.57574 3814.82 0.273446
> 35933.9 1.61384e-19
> 1800 1800 1800 3.24948 3589.5 0.319974
> 36453 1.25033e-19
> 1900 1900 1900 4.01719 3414.82 0.378235
> 36268.4 9.86666e-20
> 2000 2000 2000 4.82997 3312.65 0.438886
> 36456 7.86863e-20
> 2100 2100 2100 5.88206 3148.89 0.517821
> 35769.1 6.31726e-20
> 2200 2200 2200 6.87358 3098.24 0.590235
> 36080.6 5.1152e-20
> 2300 2300 2300 8.08021 3011.55 0.659934
> 36873.4 4.19219e-20
> 2400 2400 2400 9.31063 2969.51 0.748285
> 36948.5 3.46865e-20
> 2500 2500 2500 10.5343 2966.51 0.84448
> 37005 2.88942e-20
> 2600 2600 2600 11.8768 2959.71 0.984227
> 35715.3 2.42294e-20
> 2700 2700 2700 13.3378 2951.45 1.06838
> 36846.5 2.04036e-20
> 2800 2800 2800 14.9304 2940.57 1.18762
> 36968.2 1.73201e-20
> 2900 2900 2900 16.8965 2886.87 1.33445
> 36552.8 1.47904e-20
> 3000 3000 3000 18.7376 2881.9 1.49449
> 36132.7 1.27205e-20
> 3100 3100 3100 20.8439 2858.48 1.66163
> 35857.5 1.09759e-20
> 3200 3200 3200 22.9032 2861.44 1.82771
> 35856.9 9.49415e-21
> 3300 3300 3300 28.2407 2545.05 2.08438
> 34482.2 8.25868e-21
> 3400 3400 3400 27.5374 2854.6 2.18449
> 35984.7 7.22064e-21
> 3500 3500 3500 29.925 2865.5 2.34372
> 36587.1 6.34137e-21
> 3600 3600 3600 32.6588 2857.17 2.56586
> 36366.7 5.5874e-21
> 3700 3700 3700 34.5032 2936.14 2.77154
> 36552.2 4.92873e-21
> 3800 3800 3800 36.9099 2973.29 2.97732
> 36860.1 4.36811e-21
> 3900 3900 3900 44.6497 2657.09 3.24271
> 36586.1 3.88313e-21
> 4000 4000 4000 56.9767 2246.53 3.88046
> 32985.8 3.46672e-21
> [lehn_at_node042 session4]$ export OMP_NUM_THREADS=2; ./a.out
> # m n k uBLAS: t1 MFLOPS Blocked: t2
> MFLOPS Diff nrm1
> 100 100 100 0.00120386 1661.33 0.000876976
> 2280.56 3.95867e-14
> 200 200 200 0.00323702 4942.82 0.00099518
> 16077.5 1.50256e-15
> 300 300 300 0.0106352 5077.5 0.00286667
> 18837.2 2.19644e-16
> 400 400 400 0.0247765 5166.19 0.00610925
> 20951.8 5.61969e-17
> 500 500 500 0.0478359 5226.2 0.00707235
> 35348.9 1.94268e-17
> 600 600 600 0.082058 5264.57 0.0108406
> 39850.3 1.16982e-17
> 700 700 700 0.129637 5291.71 0.0170924
> 40134.8 6.8281e-18
> 800 800 800 0.1925 5319.48 0.0214161
> 47814.4 4.09348e-18
> 900 900 900 0.273022 5340.22 0.0298684
> 48814.2 2.54562e-18
> 1000 1000 1000 0.373113 5360.3 0.0417747
> 47875.9 1.64027e-18
> 1100 1100 1100 0.499034 5334.3 0.0527302
> 50483.4 1.08356e-18
> 1200 1200 1200 0.64351 5370.55 0.0624654
> 55326.6 7.44302e-19
> 1300 1300 1300 0.829601 5296.52 0.0793488
> 55375.8 5.25547e-19
> 1400 1400 1400 1.13615 4830.35 0.0937135
> 58561.5 3.8117e-19
> 1500 1500 1500 1.38215 4883.71 0.11078
> 60931.4 2.82628e-19
> 1600 1600 1600 2.34569 3492.37 0.148535
> 55152.1 2.11636e-19
> 1700 1700 1700 2.80764 3499.73 0.166754
> 58925.2 1.61617e-19
> 1800 1800 1800 3.65597 3190.4 0.183227
> 63658.6 1.25225e-19
> 1900 1900 1900 6.04791 2268.22 0.229272
> 59832.8 9.8624e-20
> 2000 2000 2000 5.41562 2954.41 0.244907
> 65331 7.8709e-20
> 2100 2100 2100 5.79329 3197.15 0.320638
> 57766.1 6.31124e-20
> 2200 2200 2200 10.1105 2106.32 0.348126
> 61173.2 5.11424e-20
> 2300 2300 2300 11.746 2071.68 0.385373
> 63144 4.18844e-20
> 2400 2400 2400 13.4099 2061.77 0.438608
> 63035.8 3.46829e-20
> 2500 2500 2500 14.8645 2102.32 0.491434
> 63589.4 2.88839e-20
> 2600 2600 2600 17.1602 2048.46 0.550163
> 63893.8 2.42378e-20
> 2700 2700 2700 19.24 2046.05 0.616314
> 63873.3 2.03993e-20
> 2800 2800 2800 14.8633 2953.85 0.675975
> 64949.2 1.73082e-20
> 2900 2900 2900 18.533 2631.96 0.72636
> 67154.1 1.47984e-20
> 3000 3000 3000 18.2701 2955.64 0.804625
> 67112 1.27211e-20
> 3100 3100 3100 20.2371 2944.19 0.938507
> 63485.9 1.09831e-20
> 3200 3200 3200 22.6838 2889.11 1.07581
> 60918.1 9.49232e-21
> 3300 3300 3300 25.0228 2872.33 1.06473
> 67504.6 8.25942e-21
> 3400 3400 3400 27.3561 2873.51 1.16247
> 67621.6 7.21511e-21
> 3500 3500 3500 29.7889 2878.59 1.32098
> 64913.8 6.3385e-21
> 3600 3600 3600 34.8098 2680.62 1.37908
> 67662.7 5.58738e-21
> 3700 3700 3700 37.6151 2693.23 1.52253
> 66538.1 4.92976e-21
> 3800 3800 3800 38.99 2814.67 1.63282
> 67211.5 4.36537e-21
> 3900 3900 3900 57.5765 2060.53 1.75221
> 67707.4 3.88246e-21
> 4000 4000 4000 51.1335 2503.25 2.03062
> 63035.1 3.46549e-21
> [lehn_at_node042 session4]$ export OMP_NUM_THREADS=4; ./a.out
> # m n k uBLAS: t1 MFLOPS Blocked: t2
> MFLOPS Diff nrm1
> 100 100 100 0.00119733 1670.39 0.00124331
> 1608.61 3.84618e-14
> 200 200 200 0.00427996 3738.35 0.000965206
> 16576.8 1.47604e-15
> 300 300 300 0.0146617 3683.06 0.00235442
> 22935.6 2.18643e-16
> 400 400 400 0.0301558 4244.62 0.00431311
> 29677 5.57089e-17
> 500 500 500 0.0509763 4904.24 0.00541684
> 46152.4 1.94817e-17
> 600 600 600 0.0823676 5244.78 0.00815973
> 52943 1.16851e-17
> 700 700 700 0.131064 5234.07 0.0133055
> 51557.7 6.81692e-18
> 800 800 800 0.198438 5160.3 0.0208701
> 49065.4 4.09087e-18
> 900 900 900 0.273346 5333.91 0.0244156
> 59716 2.53963e-18
> 1000 1000 1000 0.374021 5347.3 0.0252625
> 79168.7 1.64654e-18
> 1100 1100 1100 0.502426 5298.29 0.05022
> 53006.7 1.08395e-18
> 1200 1200 1200 0.865696 3992.16 0.0443738
> 77883.9 7.44661e-19
> 1300 1300 1300 1.00063 4391.23 0.0544683
> 80670.8 5.25559e-19
> 1400 1400 1400 1.26828 4327.13 0.0599685
> 91514.7 3.80933e-19
> 1500 1500 1500 1.3623 4954.86 0.0826977
> 81622.6 2.8281e-19
> 1600 1600 1600 2.14419 3820.56 0.0940622
> 87091.3 2.11718e-19
> 1700 1700 1700 2.98106 3296.14 0.104828
> 93734.3 1.61252e-19
> 1800 1800 1800 4.10679 2840.17 0.125856
> 92677.2 1.25247e-19
> 1900 1900 1900 7.25737 1890.22 0.137977
> 99422.2 9.85647e-20
> 2000 2000 2000 9.0378 1770.34 0.195959
> 81649.8 7.86877e-20
> 2100 2100 2100 7.43091 2492.56 0.205205
> 90261 6.31814e-20
> 2200 2200 2200 8.01552 2656.84 0.229878
> 92640.5 5.11206e-20
> 2300 2300 2300 11.3209 2149.47 0.242479
> 100355 4.19281e-20
> 2400 2400 2400 11.7655 2349.91 0.267819
> 103234 3.4696e-20
> 2500 2500 2500 14.75 2118.65 0.318302
> 98177.1 2.89065e-20
> 2600 2600 2600 16.1598 2175.27 0.349963
> 100445 2.42432e-20
> 2700 2700 2700 19.6465 2003.72 0.384713
> 102326 2.04284e-20
> 2800 2800 2800 18.5487 2366.95 0.422473
> 103922 1.73051e-20
> 2900 2900 2900 18.4844 2638.87 0.431616
> 113012 1.48037e-20
> 3000 3000 3000 18.3601 2941.16 0.487947
> 110668 1.27205e-20
> 3100 3100 3100 20.1449 2957.67 0.555138
> 107328 1.09745e-20
> 3200 3200 3200 22.2403 2946.72 0.597566
> 109672 9.49034e-21
> 3300 3300 3300 24.3526 2951.39 0.635492
> 113100 8.25459e-21
> 3400 3400 3400 26.5834 2957.04 0.693353
> 113374 7.22134e-21
> 3500 3500 3500 28.9996 2956.93 0.753307
> 113831 6.33808e-21
> 3600 3600 3600 31.4492 2967.07 0.793409
> 117609 5.58761e-21
> 3700 3700 3700 34.9533 2898.33 0.959263
> 105608 4.93129e-21
> 3800 3800 3800 38.2463 2869.4 1.01686
> 107924 4.36735e-21
> 3900 3900 3900 42.3957 2798.35 1.08582
> 109262 3.88282e-21
> 4000 4000 4000 44.7076 2863.05 1.22383
> 104590 3.469e-21
> [lehn_at_node042 session4]$ export OMP_NUM_THREADS=8; ./a.out
> # m n k uBLAS: t1 MFLOPS Blocked: t2
> MFLOPS Diff nrm1
> 100 100 100 0.00120762 1656.15 0.001279
> 1563.72 3.8463e-14
> 200 200 200 0.0036143 4426.86 0.000631185
> 25349.1 1.48858e-15
> 300 300 300 0.0108139 4993.56 0.00204664
> 26384.7 2.20015e-16
> 400 400 400 0.0251417 5091.13 0.00316074
> 40496.9 5.58204e-17
> 500 500 500 0.0482996 5176.03 0.00479854
> 52099.2 1.9429e-17
> 600 600 600 0.0830052 5204.49 0.0074349
> 58104.3 1.16567e-17
> 700 700 700 0.13281 5165.28 0.0134778
> 50898.6 6.82167e-18
> 800 800 800 0.19639 5214.12 0.0143988
> 71117.2 4.08235e-18
> 900 900 900 0.279542 5215.68 0.0186552
> 78155 2.54218e-18
> 1000 1000 1000 0.381906 5236.89 0.020541
> 97366.2 1.63963e-18
> 1100 1100 1100 0.509376 5226 0.0338259
> 78697.1 1.08399e-18
> 1200 1200 1200 0.760565 4543.99 0.0317094
> 108990 7.44215e-19
> 1300 1300 1300 1.04442 4207.14 0.0419104
> 104843 5.25101e-19
> 1400 1400 1400 1.47537 3719.75 0.0450985
> 121689 3.81236e-19
> 1500 1500 1500 1.90994 3534.15 0.0514728
> 131137 2.82394e-19
> 1600 1600 1600 1.56705 5227.67 0.0599189
> 136718 2.11847e-19
> 1700 1700 1700 2.62892 3737.66 0.0756787
> 129838 1.61316e-19
> 1800 1800 1800 3.29831 3536.35 0.0827417
> 140969 1.25087e-19
> 1900 1900 1900 4.03473 3399.98 0.0915113
> 149905 9.85857e-20
> 2000 2000 2000 4.87315 3283.3 0.105251
> 152017 7.86417e-20
> 2100 2100 2100 5.87975 3150.13 0.123634
> 149813 6.31281e-20
> 2200 2200 2200 7.06021 3016.34 0.134536
> 158293 5.11845e-20
> 2300 2300 2300 10.6045 2294.69 0.162671
> 149590 4.19035e-20
> 2400 2400 2400 9.31785 2967.21 0.160164
> 172623 3.46453e-20
> 2500 2500 2500 10.4852 2980.38 0.181067
> 172588 2.89024e-20
> 2600 2600 2600 11.8263 2972.35 0.208792
> 168359 2.42313e-20
> 2700 2700 2700 13.2755 2965.32 0.226646
> 173690 2.04063e-20
> 2800 2800 2800 14.8042 2965.65 0.24966
> 175855 1.73142e-20
> 2900 2900 2900 16.9983 2869.58 0.287892
> 169432 1.47875e-20
> 3000 3000 3000 19.7129 2739.32 0.330801
> 163240 1.27204e-20
> 3100 3100 3100 21.4476 2778.02 0.382773
> 155659 1.09704e-20
> 3200 3200 3200 22.7482 2880.93 0.440904
> 148640 9.49823e-21
> 3300 3300 3300 25.1449 2858.39 0.416183
> 172698 8.25712e-21
> 3400 3400 3400 27.4412 2864.6 0.497616
> 157969 7.2164e-21
> 3500 3500 3500 30.5976 2802.51 0.49974
> 171589 6.33781e-21
> 3600 3600 3600 32.5102 2870.24 0.558002
> 167225 5.59021e-21
> 3700 3700 3700 35.3566 2865.26 0.571003
> 177418 4.93007e-21
> 3800 3800 3800 37.321 2940.55 0.578064
> 189847 4.36563e-21
> 3900 3900 3900 40.1645 2953.8 0.623894
> 190157 3.8876e-21
> 4000 4000 4000 43.1753 2964.66 0.709328
> 180452 3.46575e-21
>
>
> On 28 Jan 2016, at 18:41, Michael Lehn <michael.lehn_at_[hidden]> wrote:
>
>> Also the parallelisation with openmp is done pretty cheap and simple
>> at the moment. So you also
>> might want to check how it scales by
>>
>> export OMP_NUM_THREADS=2; ./a.out
>> export OMP_NUM_THREADS=4; ./a.out
>> export OMP_NUM_THREADS=6; ./a.out
>> ...
>
> _______________________________________________
> ublas mailing list
> ublas_at_[hidden]
> http://listarchives.boost.org/mailman/listinfo.cgi/ublas
> Sent to: Oswin.Krause_at_[hidden]