Test setup: * Cluster idle except for indexing traffic * Run query normalization routine from MjoLniR to generate load from 50 hadoop executors * Routine limited to enwiki_content * Choose two machines serving replicas of the same shard (elastic1017 and elastic1043) from the two separate groups of servers. A few (relatively representative) memory usage reports from elastic1043: ``` |---------------------------------------||---------------------------------------| |-- Socket 0 --||-- Socket 1 --| |---------------------------------------||---------------------------------------| |-- Memory Channel Monitoring --||-- Memory Channel Monitoring --| |---------------------------------------||---------------------------------------| |-- Mem Ch 0: Reads (MB/s): 1243.39 --||-- Mem Ch 0: Reads (MB/s): 249.55 --| |-- Writes(MB/s): 1061.45 --||-- Writes(MB/s): 17.09 --| |-- Mem Ch 1: Reads (MB/s): 1241.51 --||-- Mem Ch 1: Reads (MB/s): 241.42 --| |-- Writes(MB/s): 1061.21 --||-- Writes(MB/s): 9.76 --| |-- Mem Ch 2: Reads (MB/s): 1242.28 --||-- Mem Ch 2: Reads (MB/s): 241.36 --| |-- Writes(MB/s): 1061.07 --||-- Writes(MB/s): 9.79 --| |-- Mem Ch 3: Reads (MB/s): 1253.14 --||-- Mem Ch 3: Reads (MB/s): 241.45 --| |-- Writes(MB/s): 1070.09 --||-- Writes(MB/s): 9.81 --| |-- NODE 0 Mem Read (MB/s) : 4980.32 --||-- NODE 1 Mem Read (MB/s) : 973.77 --| |-- NODE 0 Mem Write(MB/s) : 4253.82 --||-- NODE 1 Mem Write(MB/s) : 46.45 --| |-- NODE 0 P. Write (T/s): 131039 --||-- NODE 1 P. Write (T/s): 114837 --| |-- NODE 0 Memory (MB/s): 9234.14 --||-- NODE 1 Memory (MB/s): 1020.23 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- System Read Throughput(MB/s): 5954.09 --| |-- System Write Throughput(MB/s): 4300.27 --| |-- System Memory Throughput(MB/s): 10254.36 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- Socket 0 --||-- Socket 1 --| |---------------------------------------||---------------------------------------| |-- Memory Channel Monitoring --||-- Memory Channel Monitoring --| |---------------------------------------||---------------------------------------| |-- Mem Ch 0: Reads (MB/s): 1561.84 --||-- Mem Ch 0: Reads (MB/s): 282.63 --| |-- Writes(MB/s): 1253.94 --||-- Writes(MB/s): 18.36 --| |-- Mem Ch 1: Reads (MB/s): 1555.15 --||-- Mem Ch 1: Reads (MB/s): 273.58 --| |-- Writes(MB/s): 1252.34 --||-- Writes(MB/s): 10.62 --| |-- Mem Ch 2: Reads (MB/s): 1556.21 --||-- Mem Ch 2: Reads (MB/s): 273.93 --| |-- Writes(MB/s): 1252.17 --||-- Writes(MB/s): 10.65 --| |-- Mem Ch 3: Reads (MB/s): 1567.87 --||-- Mem Ch 3: Reads (MB/s): 274.37 --| |-- Writes(MB/s): 1260.93 --||-- Writes(MB/s): 10.67 --| |-- NODE 0 Mem Read (MB/s) : 6241.07 --||-- NODE 1 Mem Read (MB/s) : 1104.51 --| |-- NODE 0 Mem Write(MB/s) : 5019.39 --||-- NODE 1 Mem Write(MB/s) : 50.30 --| |-- NODE 0 P. Write (T/s): 135793 --||-- NODE 1 P. Write (T/s): 121339 --| |-- NODE 0 Memory (MB/s): 11260.46 --||-- NODE 1 Memory (MB/s): 1154.81 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- System Read Throughput(MB/s): 7345.58 --| |-- System Write Throughput(MB/s): 5069.69 --| |-- System Memory Throughput(MB/s): 12415.27 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- Socket 0 --||-- Socket 1 --| |---------------------------------------||---------------------------------------| |-- Memory Channel Monitoring --||-- Memory Channel Monitoring --| |---------------------------------------||---------------------------------------| |-- Mem Ch 0: Reads (MB/s): 1451.99 --||-- Mem Ch 0: Reads (MB/s): 206.95 --| |-- Writes(MB/s): 1257.25 --||-- Writes(MB/s): 16.61 --| |-- Mem Ch 1: Reads (MB/s): 1450.56 --||-- Mem Ch 1: Reads (MB/s): 198.48 --| |-- Writes(MB/s): 1257.02 --||-- Writes(MB/s): 9.42 --| |-- Mem Ch 2: Reads (MB/s): 1451.96 --||-- Mem Ch 2: Reads (MB/s): 198.46 --| |-- Writes(MB/s): 1256.86 --||-- Writes(MB/s): 9.41 --| |-- Mem Ch 3: Reads (MB/s): 1462.16 --||-- Mem Ch 3: Reads (MB/s): 198.59 --| |-- Writes(MB/s): 1265.56 --||-- Writes(MB/s): 9.45 --| |-- NODE 0 Mem Read (MB/s) : 5816.67 --||-- NODE 1 Mem Read (MB/s) : 802.48 --| |-- NODE 0 Mem Write(MB/s) : 5036.69 --||-- NODE 1 Mem Write(MB/s) : 44.89 --| |-- NODE 0 P. Write (T/s): 133682 --||-- NODE 1 P. Write (T/s): 111961 --| |-- NODE 0 Memory (MB/s): 10853.36 --||-- NODE 1 Memory (MB/s): 847.37 --| |---------------------------------------||---------------------------------------| ``` Same thing for elastic1017: ``` |---------------------------------------||---------------------------------------| |-- Socket 0 --||-- Socket 1 --| |---------------------------------------||---------------------------------------| |-- Memory Channel Monitoring --||-- Memory Channel Monitoring --| |---------------------------------------||---------------------------------------| |-- Mem Ch 1: Reads (MB/s): 3230.91 --||-- Mem Ch 1: Reads (MB/s): 387.15 --| |-- Writes(MB/s): 3521.80 --||-- Writes(MB/s): 51.89 --| |-- Mem Ch 2: Reads (MB/s): 713.36 --||-- Mem Ch 2: Reads (MB/s): 183.63 --| |-- Writes(MB/s): 726.49 --||-- Writes(MB/s): 4.03 --| |-- Mem Ch 3: Reads (MB/s): 712.75 --||-- Mem Ch 3: Reads (MB/s): 183.74 --| |-- Writes(MB/s): 726.22 --||-- Writes(MB/s): 4.08 --| |-- NODE 0 Mem Read (MB/s) : 4657.02 --||-- NODE 1 Mem Read (MB/s) : 754.51 --| |-- NODE 0 Mem Write(MB/s) : 4974.51 --||-- NODE 1 Mem Write(MB/s) : 60.00 --| |-- NODE 0 P. Write (T/s): 282753 --||-- NODE 1 P. Write (T/s): 130817 --| |-- NODE 0 Memory (MB/s): 9631.53 --||-- NODE 1 Memory (MB/s): 814.51 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- System Read Throughput(MB/s): 5411.53 --| |-- System Write Throughput(MB/s): 5034.51 --| |-- System Memory Throughput(MB/s): 10446.04 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- Socket 0 --||-- Socket 1 --| |---------------------------------------||---------------------------------------| |-- Memory Channel Monitoring --||-- Memory Channel Monitoring --| |---------------------------------------||---------------------------------------| |-- Mem Ch 1: Reads (MB/s): 4239.13 --||-- Mem Ch 1: Reads (MB/s): 866.91 --| |-- Writes(MB/s): 4404.82 --||-- Writes(MB/s): 467.04 --| |-- Mem Ch 2: Reads (MB/s): 175.46 --||-- Mem Ch 2: Reads (MB/s): 458.88 --| |-- Writes(MB/s): 12.40 --||-- Writes(MB/s): 250.83 --| |-- Mem Ch 3: Reads (MB/s): 174.79 --||-- Mem Ch 3: Reads (MB/s): 458.59 --| |-- Writes(MB/s): 12.09 --||-- Writes(MB/s): 250.71 --| |-- NODE 0 Mem Read (MB/s) : 4589.39 --||-- NODE 1 Mem Read (MB/s) : 1784.39 --| |-- NODE 0 Mem Write(MB/s) : 4429.31 --||-- NODE 1 Mem Write(MB/s) : 968.58 --| |-- NODE 0 P. Write (T/s): 336267 --||-- NODE 1 P. Write (T/s): 145569 --| |-- NODE 0 Memory (MB/s): 9018.70 --||-- NODE 1 Memory (MB/s): 2752.96 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- System Read Throughput(MB/s): 6373.78 --| |-- System Write Throughput(MB/s): 5397.89 --| |-- System Memory Throughput(MB/s): 11771.67 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- Socket 0 --||-- Socket 1 --| |---------------------------------------||---------------------------------------| |-- Memory Channel Monitoring --||-- Memory Channel Monitoring --| |---------------------------------------||---------------------------------------| |-- Mem Ch 1: Reads (MB/s): 1065.14 --||-- Mem Ch 1: Reads (MB/s): 2136.34 --| |-- Writes(MB/s): 867.09 --||-- Writes(MB/s): 2027.39 --| |-- Mem Ch 2: Reads (MB/s): 644.22 --||-- Mem Ch 2: Reads (MB/s): 667.31 --| |-- Writes(MB/s): 621.00 --||-- Writes(MB/s): 529.47 --| |-- Mem Ch 3: Reads (MB/s): 643.64 --||-- Mem Ch 3: Reads (MB/s): 667.26 --| |-- Writes(MB/s): 620.69 --||-- Writes(MB/s): 529.44 --| |-- NODE 0 Mem Read (MB/s) : 2353.00 --||-- NODE 1 Mem Read (MB/s) : 3470.91 --| |-- NODE 0 Mem Write(MB/s) : 2108.78 --||-- NODE 1 Mem Write(MB/s) : 3086.30 --| |-- NODE 0 P. Write (T/s): 171676 --||-- NODE 1 P. Write (T/s): 196425 --| |-- NODE 0 Memory (MB/s): 4461.78 --||-- NODE 1 Memory (MB/s): 6557.21 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- System Read Throughput(MB/s): 5823.91 --| |-- System Write Throughput(MB/s): 5195.08 --| |-- System Memory Throughput(MB/s): 11018.99 --| |---------------------------------------||---------------------------------------| ``` Open questions: * Why is 1017 so heavily hitting Mem Ch 1, when 1043 is spreading out the reads between channels * Why does socket 1 have such low memory bandwidth compared to socket 0 on both machines?