ROC Part 2: ACME benchmarks
Traditional benchmarks focus on performance
- ignore ACME goals
- assume perfect hardware, software, human operators
20th Century Winner: fastest on SPEC/TPC?
21st Century Winner: fastest to recover from failure?
New benchmarks needed to drive progress toward ACME, evaluate ROC success
- for example, availability and recovery benchmarks
- How else convince developers, customers to adopt new technology?
- How else convince researchers to find new challenges?