Intel Optane DC Persistent Memory to the reuse
Intel's new Persistent Memory technology has three modes of operation. One as an extension of your current memory. Imagine extending your server with 9TBytes of Memory. The second is called AppDirect mode where you can use the Persistent Memory as a persistent segment of memory or as a high-speed SSD. The third mode is called mix mode. In this mode, a percentage of the persistent memory is used for AppDirect and the other to extend your standard DDR4 Memory. When exploring this new technology, I realized that I could take the persistent memory and use it as a high-speed SSD. If I did that could I increase the throughput of my ElasticSearch Server? So I set up a test suite to try this out.
Hardware Setup and configuration
1. Physically configure the memory
• 2-2-2 configuration is the faster configuration for using the Apache Pass Modules.
2. Upgrade the BIOS with the latest updates
3. Install supported OS
• ESXi6.7+
• Fedora29+
• SUSE15+
• WinRS5+
4. Configure the Persistent Memory for 100% AppDirect Mode to get maximum storage
# ipmctl create -goal MemoryMode=0 PersistentMemoryType=AppDirect
# reboot
# ipmctl show -region
# ndctl list -Ru
[
{
"dev":"region1",
"size":"576.00 GiB (618.48 GB)",
"available_size":"576.00 GiB (618.48 GB)",
"type":"ElasticSearch",
"numa_node":1,
"iset_id":"0xe7f04ac404a6f6f0",
"persistence_domain":"unknown"
},
{
"dev":"region0",
"size":"576.00 GiB (618.48 GB)",
"available_size":"576.00 GiB (618.48 GB)",
"type":"ElasticSearch",
"numa_node":0,
"iset_id":"0xf5484ac4dfaaf6f0",
"persistence_domain":"unknown"
}
]
# ndctl create-namespace -r region0 --mode=sector
# ndctl create-namespace -r region1 --mode=sector
5. Now create the filesystem Create the FileSystem
# mkfs.xfs /dev/ElasticSearch0s
# mkfs.xfs /dev/ElasticSearch1s
# mkdir /mnt/ElasticSearch0s
# mkdir /mnt/ElasticSearch1s
# mount /dev/ElasticSearch0s /mnt/ElasticSearch0s
# mount /dev/ElasticSearch1s /mnt/ElasticSearch1s
Now that you should be able to access the filesystems via /mnt/ElasticSearch0s and /mnt/ElasticSearch1s.
Setup and configuration
We chose to evaluate the performance and resiliency of ElasticSearch using off the shelf tools. One of the most used performance test suites for ElasticSearch is ESRally. ESRally was developed by the makers of ElasticSearch and is easy to set up and run. It comes with its nomenclature that is easy to pick up.
• Tracks - Test Cases are stored in this directory.
• Cars – configuration files to be used against the distributions
• Race – This contains the data, index, and log files for ElasticSearch run. Each run has a separate directory.
• Distributions – ElasticSearch installations
• Data – data used for the tests
• Logs – logs for ESRally not for ElasticSearch.
ESRally can attach to a specific ElasticSearch cluster, or it can be configured to install and run a standard release of ElasticSearch on the fly, stored in the distribution directory. When I was looking at what directories to move between a PMEM drive and a SATA drive, I looked at the race directory. I found out that I would be limited by the data directory and log directory as well. So I decided to move the complete .rally directory between the two drives.
ESRally Pre-requisites
• Python 3.5+ which includes pip3.
• Java JDK 1.8+
Software Installation
ESRally can be installed from GitHub repository. See http://ESRally.readthedocs.io for more information.
To install ESRally, use the pip3 utility. Pip3 is installed when you install python 3.5.
# pip3 install ESRally
Now you want to configure ESRally
# export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
# ESRally configure
ESRally config creates a “.rally” directory in your home directory. This “.rally” directory is used to store all of the tests, data, and indices.
Next, you need to install the eventdata track. https://github.com/elastic/rally-eventdata-track
This track generates about 1.8 TBytes of traffic through ElasticSearch. It is a good test because much is the data is generated and not read from a drive, which means you are not limited by another drive’s performance and you can test the raw performance of the different drives with ElasticSearch. Total storage used is about 200GBytes.
Running Tests (Races)
Next, you run the tests for the different configurations. First, I run the following test against the .rally directory running on the SATA drive. I run the same test with increasing number of test clients. The “track” includes ingestion and search tests.
# ESRally --distribution-version=7.3.1 --car=16gheap --track=eventdata \
--track-repository=eventdata –track-params="bulk_indexing_clients:8"
# ESRally --distribution-version=7.3.1 --car=16gheap --track=eventdata \
--track-repository=eventdata –track-params="bulk_indexing_clients:16"
# ESRally --distribution-version=7.3.1 --car=16gheap --track=eventdata \
--track-repository=eventdata –track-params="bulk_indexing_clients:32"
# ESRally --distribution-version=7.3.1 --car=16gheap --track=eventdata \
--track-repository=eventdata –track-params="bulk_indexing_clients:64"
PMEM Drive Testing
Now that we have collected the test results from the SATA drive. We need to make sure we have set up the DCPMM to AppDirect mode, create a PMEM drive and mount the filesystem.
Ipmctl command to configure your ElasticSearch in AppDirect mode
# ipmctl create -goal MemoryMode=0 PersistentMemoryType=AppDirect
Ndctl command to create a ElasticSearch device for mounting
# ndctl create-namespace -r region0 --mode=sector
# ndctl create-namespace -r region1 --mode=sector
Mount the ElasticSearch device
# mkfs.xfs /dev/pmem0s
# mkfs.xfs /dev/pmem1s
# mkdir /mnt/pmem0s
# mkdir /mnt/pmem1s
# mount /dev/pmem0s /mnt/pmem0s
# mount /dev/pmem1s /mnt/pmem1s
Now move your “.rally” directory to the PMEM drive
# mv ~/.rally /mnt/pmem0s/ESRally
# ln -s /mnt/pmem0s/ESRally ~/.rally
Now run the tests again
# ESRally --distribution-version=7.3.1 --car=16gheap --track=eventdata --track-repository=eventdata –track-params="bulk_indexing_clients:8"
# ESRally --distribution-version=7.3.1 --car=16gheap --track=eventdata --track-repository=eventdata –track-params="bulk_indexing_clients:16"
# ESRally --distribution-version=7.3.1 --car=16gheap --track=eventdata --track-repository=eventdata –track-params="bulk_indexing_clients:32"
# ESRally --distribution-version=7.3.1 --car=16gheap --track=eventdata --track-repository=eventdata –track-params="bulk_indexing_clients:64"
Testing methodology
To isolate the effect of using a faster drive on the ingestion and indexing of documents into ElasticSearch, we run the cluster on a single node without network getting in the way of the performance testing. To simplify testing ElasticSearch, a standard benchmark test for ElasticSearch is used, ESRally. “ESRally” was developed by the makers of ElasticSearch and has been opensource(ed). The tool has a set of test suites that can be run to test different configurations of ElasticSearch.
The next variable is to pick one of the many test suites that are available to run. In ESRally test suites are called tracks. After running several races on the different tracks, I found the track “eventdata” to test the system for ingestion, index, and search throughput. Mostly because the events are generated on the fly and do not bottleneck on the disk, as many of the other tracks do. The eventdata track is found here. https://github.com/elastic/ralley-eventdata-track
Different heap sizes were used to see what the different heap sizes did to ingestion throughput, indexing, and searches. Initial testing found the 16gHeap car gave the best results for both the SATA and the PMEM drives.
• 1g heap
• 2g heap
• 4g heap
• 8g heap
• 16g heap – Selected this car as it gave the best results.
• 24g heap
Another factor we looked at was to test total throughput was to increase the number of test clients. We tested with the following number of clients
• 8 clients
• 16 clients
• 32 clients
• 64 clients
The last variable to look at in testing is where the data, index, and logs are stored. Since we hypothesize that a fast filesystem should help with overall performance. We created two passes of the above configurations. One on a SATA drive and one on a PMEM drive.
The following shows the tests that were run.
16 GHeap
|
SATA TEST
|
PMEM TEST
|
|---|---|---|
| 8 Clients | X | X |
| 16 Clients | X | X |
| 32 Clients | X | X |
| 64 Clients | X | X |
Test results
The track “eventdata” creates several metrics that are used to evaluate the performance of the different configurations. The metrics we are most interested in are total throughput in doc/sec, and the service request time (How long it takes to return a query for example.)
The first set of tests were on a 16G Heap configuration with different runs with a different number of clients. In general the PMEM drive configuration ran about 2x over the SATA drive consistently. Thruput is measured as the number of docs/sec ingested.
The second metric we evaluated is the service response time for operations. This is the time between request and response from the test client asking ElasticSearch to perform a query/find. The lower the response time, the better. In this case, we see that the PMEM drive performed about 2x what the SATA drive performed.
Comments
Post a Comment