Realtime data processing at Facebook

July 11, 2016 ~ Adrian Colyer ~ 4 Comments

Realtime Data Processing at Facebook Chen et al. SIGMOD 2016 ‘Realtime Data Processing at Facebook’ provides us with a great high-level overview of the systems Facebook have built to support real-time workloads. At the heart of the paper is a set of five key design decisions for building such systems, together with an explanation of ... Continue Reading

Efficiently compiling efficient query plans for modern hardware

May 23, 2016 ~ Adrian Colyer ~ 4 Comments

Efficiently Compiling Efficient Query Plans for Modern Hardware- Neumann, VLDB 2011 Updated with direct links to Databricks blog post now that it is published. A couple of weeks ago I had a chance to chat with Reynold Xin and Richard Garris from Databricks / Spark at RedisConf, where we were both giving talks. Reynold and ... Continue Reading

BTrDB: Optimizing Storage System Design for Timeseries Processing

May 4, 2016 ~ Adrian Colyer ~ 4 Comments

BTrDB: Optimizing Storage System Design for Timeseries Processing - Anderson & Culler 2016 It turns out you can accomplish quite a lot with 4,709 lines of Go code! How about a full time-series database implementation, robust enough to be run in production for a year where it stored 2.1 trillion data points, and supporting 119M ... Continue Reading

Gorilla: A fast, scalable, in-memory time series database

May 3, 2016 ~ Adrian Colyer ~ 16 Comments

Gorilla: A fast, scalable, in-memory time series database - Pelkonen et al. 2015 Error rates across one of Facebook's sites were spiking. The problem had first shown up through an automated alert triggered by an in-memory time-series database called Gorilla a few minutes after the problem started. One set of engineers mitigated the immediate issue. ... Continue Reading

The RAMCloud Storage System

January 18, 2016 ~ Adrian Colyer ~ 11 Comments

The RAMCloud Storage System - Ousterhout et al. 2015 This paper is a comprehensive overview of RAMCloud, published in the ACM Transactions on Computer Systems in August 2015. It's a long read (55 pages), but there's a ton of great material here. The RAMCloud project started in 2009, so this is therefore an overview of ... Continue Reading

Granularity of Locks and Degree of Consistency in a Shared Data Base – Part II

January 6, 2016 ~ Adrian Colyer ~ 3 Comments

Granularity of Locks and Degree of Consistency in a Shared Data Base - Gray et al. 1975 This is part 3 of a 7 part series on (database) 'Techniques Everyone Should Know.' Today we'll look at the second part of this paper which introduces the notion of differing degrees of consistency, and how we can ... Continue Reading

Granularity of Locks and Degree of Consistency in a Shared Data Base – Part I

January 5, 2016 ~ Adrian Colyer ~ 3 Comments

Granularity of Locks and Degree of Consistency in a Shared Data Base - Gray et al. 1975 This is part 2 of a 7 part series on (database) 'Techniques Everyone Should Know.' This is a paper of two halves, connected by the common theme of locking. The first part of the paper examines the tradeoff ... Continue Reading

Access Path Selection in a Relational Database Management System

January 4, 2016 ~ Adrian Colyer ~ 3 Comments

Access Path Selection in a Relational Database Management System - Selinger et al. 1979 This is part 1 of a 7 part series on (database) 'Techniques Everyone Should Know.' System R was a very influential Relational Database Management System (RDBMS) built at the IBM San Jose Research Laboratory starting in 1975. This paper introduces the ... Continue Reading

(Database) Techiques Everyone Should Know

January 3, 2016 ~ Adrian Colyer ~ 12 Comments

Welcome to 2016! To kick things off for the New Year, I thought we'd dip into the newly updated Red Book. In particular, I'm going to the next few days looking at the papers from Chapter 3, "Techniques Everyone Should Know". From Peter Bailis' introduction to the chapter: In this chapter, we present primary and ... Continue Reading

A higher order estimate of the optimum checkpoint interval for restart dumps

June 10, 2015 ~ Adrian Colyer ~ Leave a comment

A higher order estimate of the optimum checkpoint interval for restart dumps - Daly 2004 TL;DR: if you know how long it takes your system to create a checkpoint/snapshot (δ), and you know the expected mean-time between failures (M), then set the checkpoint interval to be √(2δM) - δ. OK, I grant that today's paper ... Continue Reading

the morning paper

a random walk through Computer Science research, by Adrian Colyer
Made delightfully fast by strattic

Datastores