TPC-H Benchmarks Show Memory1 Improves Spark SQL Performance By As Much As 289% While Lowering Costs By Up To 51%

SAN JOSE, Calif., May 17, 2017 /PRNewswire/ -- Diablo Technologies today announced the release of TPC-H benchmark data showcasing the performance benefits of Memory1™ for Apache Spark SQL™ workloads. By increasing the cluster memory size with Memory1, Diablo was able to improve data processing times by as much as 289% while lowering the overall Total Cost of Ownership (TCO) by as much as 51%. The published data clearly demonstrates that increasing application memory using Diablo Memory1 results in each server achieving three times the performance at approximately half the overall cost.

Apache Spark is a powerful open-source platform that enables high-speed data processing for large and complex datasets. Spark SQL, an Apache Spark module for structured data processing, allows SQL-like queries to be run on Spark data. Its large In-Memory requirement makes it an ideal application for Memory1.

The TPC-H test suite was selected to measure the performance of Memory1 for Spark SQL workloads. Benchmarks were run on the following configurations with different anticipated results:

• 5 node DRAM cluster vs 5 node Memory1 cluster –illustrates significant performance increase at similar cluster size
• 7 node DRAM cluster vs 2 node Memory1 cluster – illustrates consolidation achievable at similar performance levels

All DRAM nodes used SSD drives and all Memory1 nodes used DMX RAM Disk for Spark local storage.

The results on Memory1 demonstrate that users can achieve more work per server and greatly reduce the time needed to process increasingly larger datasets than servers with DRAM alone. Customers can now improve performance, get more work done with existing resources and, in some instances, realize a lower TCO.

Behind the tests

The TPC-H test suite is comprised of 22 queries that retrieve data from various subsets of the eight source tables under varying query parameters to obtain an aggregated indication of performance. For each configuration, the full 22 query set was run seven times. The results were then reviewed and normalized across all seven sets as follows:

• Validate all queries ran successfully
• Individual query completion time (breakdown of query type and performance)
• Aggregate query completion time (summarized cumulative full query set completion time)

The servers were configured as follows:

In order to comparatively test a Memory1 cluster to a DRAM-only cluster, a common 1TB CSV dataset was used. The source data was then converted to Parquet format with one Parquet file generated for each of the eight source tables. This was utilized for all tests to ensure data uniformity. The tests were executed using the tpch-spark test repository as its foundation.

Using Spark v2.0.1, the servers were first configured to use only the installed 256GB of DRAM per server to process the dataset. Next, the cluster was set up to run the tests on the same datasets with 2TB of Memory1 per server.

Results

5 x Memory1 nodes with DMX RAM Disk vs. 5 x DRAM nodes with SSD

Looking at the five node cluster comparisons, the Memory1 cluster completed the seven sets faster than DRAM with an average time of 48.41 minutes vs. the DRAM average of 144.41 minutes. In this configuration, Memory1 delivers a 289% work per node advantage.

2 x Memory1 Node with DMX RAM Disk vs. 7 x DRAM Node with SSD

The performance results show that the Memory1 node outperformed the DRAM node, completing the TPC-H set in an average time of 90.47 minutes vs. the DRAM average of 100.09 minutes. That is 11% faster than DRAM with five fewer compute nodes. In this configuration, Memory1 delivers 389% more work per node at a 51% lower TCO than DRAM.

Read the whitepaper in full here: https://goo.gl/NIIrH6

"With dataset sizes increasing daily, the need for larger memory footprints while maintaining affordability has become a business imperative," said Maher Amer, Chief Technology Officer (CTO) at Diablo Technologies. "With Memory1, customers can achieve more work per server and greatly reduce the time needed to process increasingly larger datasets than servers with DRAM-alone. Memory1 not only improves Spark SQL performance but it also significantly lowers the Total Cost of Ownership, and that is a very real and tangible business advantage."

Diablo's Memory1 solution delivers high-capacity flash-as-memory DIMMs and intelligent memory management software to enable more work per server. Memory1 scales up memory resources, delivering up to 40TB of application memory in a single rack. More efficient and resource dense servers means improved real-time analytics, faster business decisions, and more transactions completed in a shorter amount of time. The net result provides users with the flexibility to address evolving business needs and technologies at a lower TCO. Memory 1 solutions are currently available from Inspur and Supermicro. Contact your authorized reseller for more information and pricing.

FOLLOW DIABLO

https://twitter.com/diablo_tech 

https://www.linkedin.com/company/diablo-technologies 

https://www.facebook.com/pages/Diablo-Technologies/369582183128064 

ABOUT DIABLO TECHNOLOGIES 

Diablo Technologies, a pioneer and innovator of memory expansion technology, leverages the benefits of Software-Defined-Memory and NAND flash to deliver up to 10 times more work per server, at a lower cost. Diablo's Memory1™ solutions provide terabytes of system memory in a single industry-standard, two-socket x86 server – unleashing the power of Big Memory for Big Data.

Memory1, through Diablo's Memory Expansion software (DMX), is the industry's first solution to enable NAND flash as standard application memory. It is uniquely designed for environments that require large memory footprints for workloads such as in-memory computing, big data analytics, caching, database, and complex web applications. DMX intelligently manages memory virtualization, tiering, and data access between DRAM and NAND Flash devices. It also includes a learning engine that dynamically profiles applications, predicting and prefetching data, achieving significantly more work per server. Memory1 interfaces seamlessly with existing x86 server architectures, and requires no changes to hardware, operating systems, or applications. Supported NAND Flash devices include 128GB JEDEC-compliant DDR4 Flash-DIMMs and industry standard PCIe-based NVMe SSDs.

Diablo Technologies is headquartered in Ottawa, Canada with customer support offices in San Jose, CA and China. In August 2016, Diablo announced the close of $37M in oversubscribed Series C funding and production availability of Memory1. For more information visit diablo-technologies.com. 

Media Contact:
Mark Smith
JPR Communications
818-798-1472
[email protected]

SOURCE Diablo Technologies

Related Links

http://www.diablo-technologies.com