DICE: Data Intensive Computing Environment

Real Testing * Real Data * Real Results

Parallel File System Comparison Project

Project Description

File system performance is a key component of user application efficiency in high performance computing (HPC) data centers. Storage administrators and data center directors must choose wisely to select the correct file system to meet their users’ requirements and properly adjust settings for optimal performance. But which file system is the best for each situation? Today, there is no direct correlation between parallel file systems and many of the decisions are based upon vendor marketing. The DICE Program, in partnership with the Department of Energy’s Sandia National Laboratories, is conducting a project to develop a comprehensive benchmarking framework for evaluating file systems. This project also seeks to develop a normalization factor between different file system architectures. HPC community involvement is key to providing a viable tool and understanding all the factors for an accurate comparison, especially for file systems at large scale.

The status of the project was discussed during a BOF at SC10. Notes from the BOF are available here.

Results

Parallel File System Survey
In support of this project, DICE surveyed 27 HPC data center representatives concerning parallel file systems, benchmarks and I/O measurement, and trace collection. The survey was a web-based survey conducted in late Spring 2010. An invitation to participate in the
survey was provided to:
- HPC Data Center Managers,
- HPC Administrators,
- Storage Administrators,
- Archive Administrators and
- HPC Technology Architects.

Click here for the results of the PFS Survey.

Project Core Technical Team

Dr. Paul Buerger, Avetec
Lee Ward, Sandia National Lab
Matt Leininger, Lawrence Livermore National Lab
Dr. Daniel Duffy, NASA Goddard

Project Goals

  1. Survey the HPC technical community concerning parallel file system, benchmarks and I/O measurement, and trace collection
  2. Collect trace and benchmark data for analysis
  3. Develop comprehensive benchmark framework for evaluating file systems
  4. Develop normalization factor for accurate comparison of file system
  5. Release final report concerning all goals and publish results

How You Can Participate

If you are interested in participating in this project or would like additional information, please contact Armen Ezekielian, DICE Systems Administrator at aezekielian@avetec.org or by phone 937.322.5000 x2044.

If you wish to provide trace or benchmark data, please register at our collection site: https://www.dicenetwork.org