Perspector: Benchmarking Benchmark Suites

Introduction

The paper introduces a novel mechanism to estimate the qualities of a benchmark suites to evaluate a particular system/techniques. It aids the user in the following situation: Suppose the user has developed a new solution that impacts the TLB (or any other measurable parameter of the system). She has 3 benchmark suites at her disposal, say A, B, and C. She can chose to run all of them and report the results. However, this can be an overkill and doing so is not possible all the time due to resource and time constraints. Or, she just wants to know what is best suitable suite for the given system so that in the future she can just run that instead of running everything.

Question: How to measure efficacy of the available benchmark suites? How to say, in a quantifiable manner, that benchmark suite A is better than B and C? Perspector answers this by scoring the benchmark suites based on their diversity, phase changes,coverage, and spread.

Note that this is can be applied to any system where you have few measurable parameters.