The evolution of continuous experimentation in software product development

September 29, 2017 ~ Adrian Colyer ~ 14 Comments

The evolution of continuous experimentation in software product development Fabijan et al., ICSE'17 (Author personal version here) If you've been following along with the A/B testing related papers this week and thinking "we should probably do more of that in my company," then today's paper choice is for you. Anchored in experiences at Microsoft, the ... Continue Reading

Peeking at A/B tests: continuous monitoring without pain

September 28, 2017 ~ Adrian Colyer ~ 3 Comments

Peeking at A/B tests: why it matters, and what to do about it Johari et al., KDD'17 and Continuous monitoring of A/B tests without pain: optional stopping in Bayesian testing Deng, Lu, et al., CEUR'17 Today we have a double header: two papers addressing the challenge of monitoring ongoing experiments. Early stopping in traditional A/B ... Continue Reading

An efficient bandit algorithm for real-time multivariate optimization

September 27, 2017 ~ Adrian Colyer ~ 4 Comments

An efficient bandit algorithm for realtime multivariate optimization Hill et al., KDD'17 Aka, "How Amazon improved conversion by 21% in a single week!" Yesterday we saw the hard-won wisdom on display in 'seven myths' recommending that experiments be kept simple and only test one thing at a time, otherwise interpreting the results can get really ... Continue Reading

Seven rules of thumb for web site experimenters

September 26, 2017 ~ Adrian Colyer ~ 6 Comments

Seven rules of thumb for web site experimenters Kohavi et al., KDD'14 Following yesterday's 12 metric interpretation pitfalls, today we're looking at 7 rules of thumb for designing web site experiments. There's a little bit of duplication here, but the paper is packed with great real world examples, and there is some very useful new ... Continue Reading

A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments

September 25, 2017 ~ Adrian Colyer ~ 10 Comments

A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments Dmitriev et al., KDD 2017 Pure Gold! Here we have twelve wonderful lessons in how to avoid expensive mistakes in companies that are trying their best to be data-driven. A huge thank you to the team from Microsoft for sharing their hard-won experiences ... Continue Reading

Distributed deep neural networks over the cloud, the edge, and end devices

September 22, 2017 ~ Adrian Colyer ~ 6 Comments

Distributed deep neural networks over the cloud, the edge, and end devices Teerapittayanon et al., ICDCS 17 Earlier this year we looked at Neurosurgeon, in which the authors do a brilliant job of exploring the trade-offs when splitting a DNN such that some layers are processed on an edge device (e.g., mobile phone), and some ... Continue Reading

CLKSCREW: Exposing the perils of security-oblivious energy management

September 21, 2017 ~ Adrian Colyer ~ 21 Comments

CLKSCREW: Exposing the perils of security-oblivious energy management Tang et al., USENIX Security '17 This is brilliant and terrifying in equal measure. CLKSCREW demonstrably takes the Trust out of ARM's TrustZone, and it wouldn't be at all surprising if it took the Secure out of SGX too (though the researchers didn't investigate that). It's the ... Continue Reading

A longitudinal, end-to-end view of the DNSSEC ecosystem

September 20, 2017 ~ Adrian Colyer ~ 5 Comments

A longitudinal, end-to-end view of the DNSSEC ecosystem Chung et al., USENIX Security 2017 DNS, the Domain Name System, provides a vital function on the Internet, mapping names to values. Unprotected, it's also an attractive target for hackers with attack vectors such DNS spoofing and cache poisoning. Thus about two decades ago a set of ... Continue Reading

To type or not to type: quantifying detectable bugs in JavaScript

September 19, 2017 ~ Adrian Colyer ~ 50 Comments

To type or not to type: quantifying detectable bugs in JavaScript Gao et al., ICSE 2017 This is a terrific piece of work with immediate practical applications for many project teams. Is it worth the extra effort to add static type annotations to a JavaScript project? Should I use Facebook's Flow or Microsoft's TypeScript if ... Continue Reading

Bringing the web up to speed with WebAssembly

September 18, 2017 ~ Adrian Colyer ~ 7 Comments

Bringing the web up to speed with WebAssembly Haas et al., PLDI 2017 This is a joint paper from authors at Google, Mozilla, Microsoft and Apple describing the motivations for WebAssembly together with a very concise exposition of its core semantics. If you're following along with the paper, I found it helpful to dip into ... Continue Reading

the morning paper

a random walk through Computer Science research, by Adrian Colyer
Made delightfully fast by strattic

Month: September 2017