BenchmarkRunner: save and load results

README.md

@@ -14,7 +14,9 @@ Table of contents
     * [Obtaining heuristic solutions](#obtaining-heuristic-solutions)
     * [Saving and loading solver state](#saving-and-loading-solver-state)
     * [Solving training instances in parallel](#solving-training-instances-in-parallel)
-* [Benchmark](#benchmark)
+* [Benchmarking](#benchmarking)
+    * [Using BenchmarkRunner](#using-benchmarkrunner)
+    * [Saving and loading benchmark results](#saving-and-loading-benchmark-results)
 * [Current Limitations](#current-limitations)
 * [References](#references)
 * [Authors](#authors)
@@ -137,8 +139,10 @@ solver.load("/tmp/data.bin")
 solver.solve(test_instance)
 ```
 
-Benchmark
+Benchmarking
----------
+------------
+
+### Using `BenchmarkRunner`
 
 MIPLearn provides the utility class `BenchmarkRunner`, which simplifies the task of comparing the performance of different solvers. The snippet below shows its basic usage:
 
@@ -167,7 +171,32 @@ benchmark.parallel_solve(test_instances, n_jobs=2)
 print(benchmark.raw_results())
 ```
 
-The method `load_fit` loads the saved training data into each one of the provided solvers and trains their respective ML models. The method `parallel_solve` solves the test instances in parallel, and collects solver statistics such as running time and optimal value. Finally, `raw_results` produces a Pandas DataFrame containing the results.
+The method `load_fit` loads the saved training data into each one of the provided solvers and trains their respective ML models. The method `parallel_solve` solves the test instances in parallel, and collects solver statistics such as running time and optimal value. Finally, `raw_results` produces a table of results (Pandas DataFrame) with the following columns:
+
+* **Solver,** the name of the solver.
+* **Instance,** the sequence number identifying the instance.
+* **Wallclock Time,** the wallclock running time (in seconds) spent by the solver;
+* **Obj Value,** the objective value of the solution found by the solver;
+* **Relative Wallclock Time,** a number indicating how many times slower this run was when compared to the best time achieved by any solver when processing this instance. For example, if this run took 10 seconds, but another solver took only 5 seconds to solve the same instance, the relative wallclock time would be 2.00.
+* **Relative Obj Value,** how many times better (or worse) this solution was in terms of objective value, when compared to the solutions produced by the other solvers for the same instance. For example, if this solver found a solution with objective value 100.0 on a minimization problem, and another solver found a solution with value 80.0, then the relative objective value would be 1.25.
+
+### Saving and loading benchmark results
+
+When iteratively exploring new formulations, encoding and solver parameters, it is often desirable to avoid repeating parts of the benchmark suite. For example, if the baseline solver has not been changed, there is no need to evaluate its performance again and again when making small changes to the remaining solvers. `BenchmarkRunner` provides the methods `save` and `load`, which can be used to avoid this repetition, as the next example shows:
+
+```python
+# Benchmark baseline solvers and save results to a file.
+benchmark = BenchmarkRunner(baseline_solvers)
+benchmark.load_fit("training_data.bin")
+benchmark.parallel_solve(test_instances)
+benchmark.save_results("baseline_results.csv")
+
+# Benchmark remaining solvers, loading baseline results from file.
+benchmark = BenchmarkRunner(alternative_solvers)
+benchmark.load_results("baseline_results.csv")
+benchmark.load_fit("training_data.bin")
+benchmark.parallel_solve(test_instances)
+```
 
 Current Limitations
 -------------------

miplearn/benchmark.py

@@ -19,10 +19,11 @@ class BenchmarkRunner:
             solver.fit()
             
     def parallel_solve(self, instances, n_jobs=1):
+        if self.results is None:
             self.results = pd.DataFrame(columns=["Solver",
                                                  "Instance",
                                                  "Wallclock Time",
-                                             "Optimal Value",
+                                                 "Obj Value",
                                                 ])
         for (name, solver) in self.solvers.items():
             results = solver.parallel_solve(instances, n_jobs=n_jobs, label=name)
@@ -38,8 +39,21 @@ class BenchmarkRunner:
                     "Solver": name,
                     "Instance": i,
                     "Wallclock Time": wallclock_time,
-                    "Optimal Value": results[i]["Problem"][0]["Lower bound"]
+                    "Obj Value": results[i]["Problem"][0]["Lower bound"]
                 }, ignore_index=True)
+                groups = self.results.groupby("Instance")
+                best_obj_value = groups["Obj Value"].transform("max")
+                best_wallclock_time = groups["Wallclock Time"].transform("min")
+                self.results["Relative Obj Value"] = \
+                        self.results["Obj Value"] / best_obj_value
+                self.results["Relative Wallclock Time"] = \
+                        self.results["Wallclock Time"] / best_wallclock_time
     
     def raw_results(self):
         return self.results
+    
+    def save_results(self, filename):
+        self.results.to_csv(filename)
+        
+    def load_results(self, filename):
+        self.results = pd.read_csv(filename, index_col=0)

miplearn/tests/test_benchmark.py

@@ -8,6 +8,7 @@ from miplearn.problems.stab import MaxStableSetInstance, MaxStableSetGenerator
 import networkx as nx
 import numpy as np
 import pyomo.environ as pe
+import os.path
 
 
 def test_benchmark():
@@ -38,4 +39,11 @@ def test_benchmark():
     benchmark = BenchmarkRunner(test_solvers)
     benchmark.load_fit("data.bin")
     benchmark.parallel_solve(test_instances, n_jobs=2)
-    print(benchmark.raw_results())
+    assert benchmark.raw_results().values.shape == (6,6)
+    
+    benchmark.save_results("/tmp/benchmark.csv")
+    assert os.path.isfile("/tmp/benchmark.csv")
+    
+    benchmark = BenchmarkRunner(test_solvers)
+    benchmark.load_results("/tmp/benchmark.csv")
+    assert benchmark.raw_results().values.shape == (6,6)