Paper Details

Reference:

Siying Dong, Andrew Kryczka, Yanqin Jin, and Michael Stumm,
RocksDB: Evolution of Development Priorities in a Key-value Store Serving Large-scale Applications",
ACM Transactions on Storage, 17(4), October, 2021, pp. 26:1–26:32.

Download:

PDF

Abstract:

This article is an eight-year retrospective on development priorities for RocksDB, a key-value store developed at Facebook that targets large-scale distributed systems and that is optimized for Solid State Drives (SSDs). We describe how the priorities evolved over time as a result of hardware trends and extensive experiences running RocksDB at scale in production at a number of organizations: from optimizing write amplification, to space amplification, to CPU utilization. We describe lessons from running large-scale applications, including that resource allocation needs to be managed across different RocksDB instances, that data formats need to remain backward- and forward-compatible to allow incremental software rollouts, and that appropriate support for database replication and backups are needed. Lessons from failure handling taught us that data corruption errors needed to be detected earlier and that data integrity protection mechanisms are needed at every layer of the system. We describe improvements to the key-value interface. We describe a number of efforts that in retrospect proved to be misguided. Finally, we describe a number of open problems that could benefit from future research.

Keywords:

Key-value stores, large-scale applications, RocksDB, SSD, compaction, databases

Reference Info:

DOI: 10.1145/3483840
ISSN: 1553-3077

BibTeX:

@article(Dong-ACMToS21,
    author = {Siying Dong and Andrew Kryczka and Yanqin Jin and Michael Stumm},
    title = {RocksDB: Evolution of Development Priorities in a Key-value Store Serving Large-scale Applications},
    volume = {17},
    number = {4},
    month = {October},
    year = {2021},
    pages = {26:1--26:32},
    doi = {10.1145/3483840},
    issn = {1553-3077},
    keywords = {Key-value stores, large-scale applications, RocksDB, SSD, compaction, databases}
)