MSFSS: A Storage System for Mass Small Files

Abstract

We designed and implemented MSFSS, a scalable and flexible distributed file system for storage and retrieval of mass small files. MSFSS is a platform built upon the existing commodity file systems. It automatically stores files onto the best-fit commodity file systems according their access patterns. To avoid central bottleneck, it optimizes metadata size, separates metadata operations from file data transfer, and implements batch metadata operations. The system provides data migration, hot file caching, and replication, which are essential for large scale, reliable storage systems. It has successfully been deployed as storage system for our Web application which has about 50 TB of small files. Experimental results show that MSFSS provides high scalability and throughput in file operation services.