How to - ECFS Performance test

Basic guideline in order to maximize the  performance from Elastifile cluster.


In order to run the erun performance tool, install the following rpm in a Centos7 client. 


https://drive.google.com/file/d/15ze9qjQjr5eJexlH8ETv-yRYNZ21HOv0/view?usp=sharing


For Erun tests:

  • use the same number of clients/loaders/machines, as the number of ECFS nodes, for example, 3x nodes should be tested with 3 erun clients/loaders.
  • in the erun command:
    • clients (not the number of clients/loaders.machines...) = number of cores / 2
    • files (nr-files)= number of cores 
    • Queue = should be tuned to suit the latency demands:
      • Latency too high --> decrease Queue size
      • Latency too low --> Increase Queue size.
    • When starting a new erun test, use --initial-write-phase flag, to create new data. this will first build the working set (perform only writes...) and only then start with the requested workload.
    • Once the data is available, and there is a need to rerun a new test on the same data, using different options (such as: different queue size or read/write ratio), use --reuse-existing-files instead.
    • erun example, OLTP workload for 4 cores per node, testing 4k block size, 70/30:
      • erun --profile io  --data-payload --max-file-size 100M --clients 2 --nr-files 4 --queue-size 8 --readwrites 70 --min-io-size 4K --max-io-size 4K --duration 12000 --erun-dir `hostname` 10.99.0.2:dc/root --initial-write-phase 
    • erun example, BW tests for 4 cores per node, testing 64k block size, 70/30:
      • erun --profile io  --data-payload --max-file-size 100M --clients 2 --nr-files 4 --queue-size 4 --readwrites 70 --min-io-size 64K --max-io-size 64K --duration 12000 --erun-dir `hostname` 10.99.0.2:dc/root --io-alignment 32768 --initial-write-phase 


For Any other testing tool on linux machine, the rule of thumb is:


  • Clients = half the number of the cluster cores. i.e. 3x nodes with 4x cores, should be tested with 6x clients
  • Total number of Files = as the same number the cluster cores. i.e. 3x nodes with 4x cores, should be tested with 12x clients
  • Reaching max number of IOPS, with low latency (~2ms) - using 4k or 8k block sizes.
  • Reaching max BW, where latency is less crucial (can be ~ 10-20 ms) - using 32k, 64k or 256k block sizes.


The current Elastifile configurations in GCP:


ECFS Configuration TypeCluster Info
SSD Persistent Disks - SMALL4 Cores, 32GB RAM, 4 x 175GB PD SSD
SSD Persistent Disks - MEDIUM4 Cores, 42GB RAM, 4 x 1TB PD SSD
SSD Persistent Disks - LARGE16 Cores, 96GB RAM, 4 X 5TB PD SSD
Local SSD16 Cores, 96GB RAM, 8 X 375GB Local SSD
Standard Persistent Disks (not under 2ms, due to standard drives..)4 Cores, 64GB RAM, 4 x 1TB Standard PD
MAX Configuration - Local SSD


Some examples of the expected performance results from different GCP configurations:


Maximum sustained IOPS (under 2ms)


ECFS Configuration TypeRead IOPSWrite IOPSMixed Read/Write IOPS (70/30 Ratio)
Per SystemPer NodePer SystemPer NodePer SystemPer Node
SSD Persistent Disks - SMALL - 3 nodes40,00013,00010,0003,30020,0006,600
SSD Persistent Disks - MEDIUM - 3 nodes40,00013,00010,0003,30020,0006,600
SSD Persistent Disks - MEDIUM - 3 nodes - Single Replication
42,00014,00024,3008,10030,00010,000
SSD Persistent Disks - LARGE - 3 nodes
74,00024,00019,0006,30045,00015,000
SSD Persistent Disks - LARGE - 3 nodes - Single Replication
74,00024,000
52,000
17,300
64,000
21,300
Local SSD - 3 node178,00059,00051,00017,000105,00035,000
Standard Persistent Disks (not under 2ms, due to standard drives..) - 6 nodes
18,0003,00011,5001,90014,0002,300
MAX Configuration - Local SSD







Maximum sustained throughput (MB/s)


ECFS Configuration Type
Read Throughput (MB/s)Write Throughput (MB/s)

Per SystemPer NodePer SystemPer Node
SSD Persistent Disks - SMALL - 3 nodes4 Cores, 32GB RAM, 4 x 175GB PD SSD70023320066
SSD Persistent Disks - MEDIUM - 3 nodes4 Cores, 56GB RAM, 4 x 1TB PD SSD70023320066
SSD Persistent Disks - MEDIUM - 3 nodes - Single Replication
4 Cores, 56GB RAM, 4 x 1TB PD SSD
1100366395131
SSD Persistent Disks - LARGE - 3 nodes
16 Cores, 96GB RAM, 4 X 5TB PD SSD1,700566330110
SSD Persistent Disks - LARGE - 3 nodes - Single Replication
16 Cores, 96GB RAM, 4 X 5TB PD SSD
2000666910303
Local SSD - 3 nodes16 Cores, 96GB RAM, 8 X 375GB Local SSD3,50011671,100367
Standard Persistent Disks - 6 nodes - Default4 Cores, 64GB RAM, 4 x 1TB Standard PD4708021836
Standard Persistent Disks - 3 nodes4 Cores, 64GB RAM, 4 x 1TB Standard PD2408011237
Standard Persistent Disks - 6 nodes4 Cores, 64GB RAM, 4 x 3TB Standard PD5008328045
Standard Persistent Disks - 3 nodes4 Cores, 64GB RAM, 4 x 3TB Standard PD2428015050
Standard Persistent Disks - 3 nodes4 Cores, 32GB RAM, 4 x 175GB Standard PD75255017
MAX Configuration - Local SSD






Single Client comparison tests:


Centos 7 with nfs (erun)



















Test (100MB files...)ECFS Configuration Type
Read IOPSWrite IOPS70/30 IOPSRead Throughput (MB/s)Write Throughput (MB/s)

ClusterLatency-msPer NodePer ClientClusterLatency-msPer NodePer ClientClusterLatency-msPer NodePer ClientClusterPer NodePer ClientClusterPer NodePer Client
1 Client,1 Connection, 4 filesSSD Persistent Disks - MEDIUM - 3 nodes4 Cores, 56GB RAM, 4 x 1TB PD SSD

















1 Client,6 Connections, 4 filesSSD Persistent Disks - MEDIUM - 3 nodes4 Cores, 56GB RAM, 4 x 1TB PD SSD

















1 Client,1 Connection, 1 filesSSD Persistent Disks - MEDIUM - 3 nodes4 Cores, 56GB RAM, 4 x 1TB PD SSD

















1 Client,6 Connections, 1 filesSSD Persistent Disks - MEDIUM - 3 nodes4 Cores, 56GB RAM, 4 x 1TB PD SSD






































1 Client,1 Connection, 20 filesLocal SSD - 3 nodes20 Cores, 128GB RAM, 8 X 375GB Local SSD235001.77833
93001.93100
123001.74100
1000333
665222
1 Client,1 Connection, 1 filesLocal SSD - 3 nodes20 Cores, 128GB RAM, 8 X 375GB Local SSD146001.74867
57001.91900
135001.94500
1300433
24080
1 Client,30 Connections, 20 filesLocal SSD - 3 nodes20 Cores, 128GB RAM, 8 X 375GB Local SSD1800002.660000
600002.720000
1130002.737667
1600533
770257
1 Client,30 Connections, 1 filesLocal SSD - 3 nodes20 Cores, 128GB RAM, 8 X 375GB Local SSD1940002.364667
620002.620667
1170002.639000
1600533
810270





















Windows 2016R2 with nfs services (Latency from the GUI...)



















1 Client,1 Node, 20 filesLocal SSD - 4 nodes20 Cores, 128GB RAM, 8 X 375GB Local SSD240002.1600024000165002.8412516500200002.2500020000875218.7587532080320
1 Client,1 Nodes, 1 filesLocal SSD - 4 nodes20 Cores, 128GB RAM, 8 X 375GB Local SSD170001.842501700045001.911254500120001.9300012000950237.595016541.25165
10 Clients,1 Node, 20 files eachLocal SSD - 4 nodes20 Cores, 128GB RAM, 8 X 375GB Local SSD950003237509500470002.9117504700700002.8175007000180045018068017068
10 Clients,1 Node, 1 file eachLocal SSD - 4 nodes20 Cores, 128GB RAM, 8 X 375GB Local SSD910001.9227509100350002.387503500600001.91500060001800450180670167.567
10 Client,4 Nodes, 40 clients total 1 files eachLocal SSD - 4 nodes20 Cores, 128GB RAM, 8 X 375GB Local SSD

00

00

00
00
00
10 Client,4 Nodes, 40 clients total 20 files eachLocal SSD - 4 nodes20 Cores, 128GB RAM, 8 X 375GB Local SSD

00

00

00
00
00






















G
Guy is the author of this solution article.

Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.