ANL-CEEESA/MIPLearn
1
0
Fork 0
mirror of https://github.com/ANL-CEEESA/MIPLearn.git synced 2025-12-06 01:18:52 -06:00
Code Issues Packages Projects Releases Wiki Activity

Simplify BenchmarkRunner; update docs

Browse Source
This commit is contained in:
Alinson S. Xavier
2021-01-22 07:22:19 -06:00
parent aa9cefb9c9
commit f755661fa6
11 changed files with 191 additions and 178 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
docs/benchmark.md
View File

@@ -1,61 +0,0 @@
# Benchmarks Utilities
### Using `BenchmarkRunner`
MIPLearn provides the utility class `BenchmarkRunner`, which simplifies the task of comparing the performance of different solvers. The snippet below shows its basic usage:
```python
from miplearn import BenchmarkRunner, LearningSolver
# Create train and test instances
train_instances = [...]
test_instances = [...]
# Training phase...
training_solver = LearningSolver(...)
training_solver.parallel_solve(train_instances, n_jobs=10)
# Test phase...
test_solvers = {
"Baseline": LearningSolver(...), # each solver may have different parameters
"Strategy A": LearningSolver(...),
"Strategy B": LearningSolver(...),
"Strategy C": LearningSolver(...),
}
benchmark = BenchmarkRunner(test_solvers)
benchmark.fit(train_instances)
benchmark.parallel_solve(test_instances, n_jobs=2)
print(benchmark.raw_results())
```
The method `fit` trains the ML models for each individual solver. The method `parallel_solve` solves the test instances in parallel, and collects solver statistics such as running time and optimal value. Finally, `raw_results` produces a table of results (Pandas DataFrame) with the following columns:
* **Solver,** the name of the solver.
* **Instance,** the sequence number identifying the instance.
* **Wallclock Time,** the wallclock running time (in seconds) spent by the solver;
* **Lower Bound,** the best lower bound obtained by the solver;
* **Upper Bound,** the best upper bound obtained by the solver;
* **Gap,** the relative MIP integrality gap at the end of the optimization;
* **Nodes,** the number of explored branch-and-bound nodes.
In addition to the above, there is also a "Relative" version of most columns, where the raw number is compared to the solver which provided the best performance. The *Relative Wallclock Time* for example, indicates how many times slower this run was when compared to the best time achieved by any solver when processing this instance. For example, if this run took 10 seconds, but the fastest solver took only 5 seconds to solve the same instance, the relative wallclock time would be 2.
### Saving and loading benchmark results
When iteratively exploring new formulations, encoding and solver parameters, it is often desirable to avoid repeating parts of the benchmark suite. For example, if the baseline solver has not been changed, there is no need to evaluate its performance again and again when making small changes to the remaining solvers. `BenchmarkRunner` provides the methods `save_results` and `load_results`, which can be used to avoid this repetition, as the next example shows:
```python
# Benchmark baseline solvers and save results to a file.
benchmark = BenchmarkRunner(baseline_solvers)
benchmark.parallel_solve(test_instances)
benchmark.save_results("baseline_results.csv")
# Benchmark remaining solvers, loading baseline results from file.
benchmark = BenchmarkRunner(alternative_solvers)
benchmark.load_results("baseline_results.csv")
benchmark.fit(training_instances)
benchmark.parallel_solve(test_instances)
```

49
docs/usage.md
View File

@@ -2,12 +2,12 @@
## 1. Installation
In these docs, we describe the Python/Pyomo version of the package, although a [Julia/JuMP version](https://github.com/ANL-CEEESA/MIPLearn.jl) is also available. A mixed-integer solver is also required and its Python bindings must be properly installed. Supported solvers are currently CPLEX and Gurobi.
In these docs, we describe the Python/Pyomo version of the package, although a [Julia/JuMP version](https://github.com/ANL-CEEESA/MIPLearn.jl) is also available. A mixed-integer solver is also required and its Python bindings must be properly installed. Supported solvers are currently CPLEX, Gurobi and XPRESS.
To install MIPLearn, run:
```bash
pip3 install miplearn
pip3 install --upgrade miplearn==0.2.*
```
After installation, the package `miplearn` should become available to Python. It can be imported
@@ -176,11 +176,12 @@ Instance files must be pickled instance objects. The method `solve` loads at mos
```python
import pickle
from miplearn import LearningSolver
# Construct and pickle 600 problem instances
for i in range(600):
instance = CustomInstance([...])
instance = MyProblemInstance([...])
with open("instance_%03d.pkl" % i, "w") as file:
pickle.dump(instance, obj)
@@ -202,22 +203,50 @@ solver.parallel_solve(test_instances, n_jobs=4)
```
By default, `solve` and `parallel_solve` modify files in place. That is, after the instances are loaded from disk and solved, MIPLearn writes them back to the disk, overwriting the original files. To write to an alternative file instead, the argument `output` may be used. In `solve`, this argument should be a single filename. In `parallel_solve`, it should be a list, containing exactly as many filenames as instances. If `output` is `None`, the modifications are simply discarded. This can be useful, for example, during benchmarks.
By default, `solve` and `parallel_solve` modify files in place. That is, after the instances are loaded from disk and solved, MIPLearn writes them back to the disk, overwriting the original files. To write to an alternative file instead, use the arguments `output_filename` (in `solve`) and `output_filenames` (in `parallel_solve`). To discard the modifications instead, use `discard_outputs=True`. This can be useful, for example, during benchmarks.
```python
# Solve a single instance file and store the output to another file
solver.solve("knapsack_1.orig.pkl", output="knapsack_1.solved.pkl")
# Solve a single instance file and write the output to another file
solver.solve("knapsack_1.orig.pkl", output_filename="knapsack_1.solved.pkl")
# Solve a list of instance files
instances = ["knapsack_%03d.orig.pkl" % i for i in range(100)]
output = ["knapsack_%03d.solved.pkl" % i for i in range(100)]
solver.parallel_solve(instances, output=output)
solver.parallel_solve(instances, output_filenames=output)
# Solve instances and discard solutions and training data
solver.parallel_solve(instances, output=None)
solver.parallel_solve(instances, discard_outputs=True)
```
## 7. Running benchmarks
## 7. Current Limitations
MIPLearn provides the utility class `BenchmarkRunner`, which simplifies the task of comparing the performance of different solvers. The snippet below shows its basic usage:
* Only binary and continuous decision variables are currently supported. General integer variables are not currently supported by all solver components.
```python
from miplearn import BenchmarkRunner, LearningSolver
# Create train and test instances
train_instances = [...]
test_instances = [...]
# Training phase...
training_solver = LearningSolver(...)
training_solver.parallel_solve(train_instances, n_jobs=10)
# Test phase...
benchmark = BenchmarkRunner({
"Baseline": LearningSolver(...),
"Strategy A": LearningSolver(...),
"Strategy B": LearningSolver(...),
"Strategy C": LearningSolver(...),
})
benchmark.fit(train_instances)
benchmark.parallel_solve(test_instances, n_jobs=5)
benchmark.write_csv("results.csv")
```
The method `fit` trains the ML models for each individual solver. The method `parallel_solve` solves the test instances in parallel, and collects solver statistics such as running time and optimal value. Finally, `write_csv` produces a table of results. The columns in the CSV file depend on the components added to the solver.
## 8. Current Limitations
* Only binary and continuous decision variables are currently supported. General integer variables are not currently supported by some solver components.