Month: December 2014

Tachyon: Reliable, Memory Speed Storage for Cluster Computing

~ adriancolyer ~ 3 Comments

Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks - Li et al. 2014 Data processing can often be naturally expressed as a sequence of steps in a pipeline. For example, the unix command line below that pipes a file through a series of transforms to ultimately generate some output. cat Fin.csv | a | … Continue reading Tachyon: Reliable, Memory Speed Storage for Cluster Computing

Photon: Fault-tolerant and scalable joining of continuous data streams

~ adriancolyer ~ 4 Comments

Photon: Fault-tolerant and scalable joining of continuous data streams - Google 2013 To the best of our knowledge, this is the first paper to formulate and solve the problem of joining multiple streams continuously under these system constraints: exactly-once semantics, fault-tolerance at datacenter-level, high scalability, low latency, unordered streams, and delayed primary stream. It's interesting … Continue reading Photon: Fault-tolerant and scalable joining of continuous data streams

The 8 Requirements of Real-Time Stream Processing

~ adriancolyer ~ 4 Comments

The 8 Requirements of Real-Time Stream Processing - Stonebraker et al. 2005 Applications that require real-time processing of high-volume data streams are pushing the limits of data processing infrastructures. Stonebraker et al. make the case in 2005 that stream processing is going to become increasingly important. Not just for the usual finance, fraud, and command-and-control … Continue reading The 8 Requirements of Real-Time Stream Processing

The case for distributed operating systems in the data center

~ adriancolyer ~ 4 Comments

New wine in old skins: the case for distributed operating systems in the data center - Schwarzkopf et al. 2013. I attended the New Directions in Operating Systems one-day event in London last week, and came away with the impression that the beginning of the end of the traditional operating system is in sight. Today's … Continue reading The case for distributed operating systems in the data center