From a3794d97db8491ef4dbbff07de03442b6195a0eb Mon Sep 17 00:00:00 2001 From: Alinson S Xavier Date: Wed, 29 Jan 2020 20:41:39 -0600 Subject: [PATCH] Improve description of benchmark challenges and results --- docs/benchmark.md | 28 +------------------------ docs/index.md | 3 ++- docs/js/mathjax.js | 8 +++++++ docs/problems.md | 52 ++++++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 9 +++++++- requirements.txt | 2 +- 6 files changed, 72 insertions(+), 30 deletions(-) create mode 100644 docs/js/mathjax.js create mode 100644 docs/problems.md diff --git a/docs/benchmark.md b/docs/benchmark.md index bd83cc5..b68149f 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -1,4 +1,4 @@ -# Benchmarks +# Benchmarks Utilities ### Using `BenchmarkRunner` @@ -62,29 +62,3 @@ benchmark.load_results("baseline_results.csv") benchmark.parallel_solve(test_instances) ``` - -### Benchmark problems - -MIPLearn provides a selection of random instance generators for some fundamental discrete optimization problems, as well a baseline MIP and ML formulation for these problems. The included problems are the following: - -* **Maximum Weight Stable Set Problem:** Given a graph G=(V,E) with vertex weights, the problem is to find a maximum weight stable set of the graph, where a *stable set* is a subset of vertices, no two of which are adjacent. The class `MaxWeightStableSetGenerator` can generate random instances of this problem with specified probability distributions for number of vertices, edge probability and weights. - - -### Benchmark results - -To illustrate the performance benefits of MIPLearn, we present a small number of computational results for some of the included benchmark problems. For more detailed computational studies, see the [references](#references) below. We compare three solvers: - -* **baseline:** Gurobi 9.0 with default settings (a conventional state-of-the-art MIP solver) -* **ml-exact:** `LearningSolver` with default settings, using Gurobi 9.0 as internal MIP solver -* **ml-heuristic:** Same as above, but with `mode="heuristic"` - -The experiments were performed on a Linux server (Ubuntu Linux 18.04 LTS) with Intel Xeon Gold 6230s (2 processors, 40 cores, 80 threads) and 256 GB RAM (DDR4, 2933 MHz). All solvers were restricted to use 4 threads, with no time limits, and 10 instances were solved simultaneously at a time. - - -#### Maximum Weight Stable Set Problem - -* Fixed random graph (200 nodes, 5% edge probability) -* Random vertex weights ~ U(100, 150) -* 300 training instances, 50 test instances - -![alt](figures/mwss.png) diff --git a/docs/index.md b/docs/index.md index f990d05..fd4f555 100644 --- a/docs/index.md +++ b/docs/index.md @@ -15,7 +15,8 @@ ### Documentation * [Installation and typical usage](usage.md) -* [Benchmark problems and results](benchmark.md) +* [Benchmark utilities](benchmark.md) +* [Benchmark problems, challenges and results](problems.md) * [Customizing the solver](customization.md) * [License, authors, references and acknowledgements](about.md) diff --git a/docs/js/mathjax.js b/docs/js/mathjax.js new file mode 100644 index 0000000..bfc06b8 --- /dev/null +++ b/docs/js/mathjax.js @@ -0,0 +1,8 @@ +MathJax.Hub.Config({ + "tex2jax": { inlineMath: [ [ '$', '$' ] ] } +}); +MathJax.Hub.Config({ + config: ["MMLorHTML.js"], + jax: ["input/TeX", "output/HTML-CSS", "output/NativeMML"], + extensions: ["MathMenu.js", "MathZoom.js"] +}); \ No newline at end of file diff --git a/docs/problems.md b/docs/problems.md new file mode 100644 index 0000000..86565e1 --- /dev/null +++ b/docs/problems.md @@ -0,0 +1,52 @@ +# Benchmark Problems, Challenges and Results + +MIPLearn provides a selection of benchmark problems and random instance generators, covering applications from different fields, that can be used to evaluate new learning-enhanced MIP techniques in a measurable and reproducible way. In this page, we describe these problems, the included instance generators, and we present some benchmark results for `LearningSolver` with default parameters. + +## Preliminaries + +### Benchmark challenges + +When evaluating the performance of a conventional MIP solver, *benchmark sets*, such as MIPLIB and TSPLIB, are typically used. The performance of newly proposed solvers or solution techniques are typically measured as the average (or total) running time the solver takes to solve the entire benchmark set. For Learning-Enhanced MIP solvers, it is also necessary to specify what instances should the solver be trained on (the *training instances*) before solving the actual set of instances we are interested in (the *test instances*). If the training instances are very similar to the test instances, we would expect a Learning-Enhanced Solver to present stronger perfomance benefits. + +In MIPLearn, each optimization problem comes with a set of **benchmark challenges**, which specify how should the training and test instances be generated. The first challenges are typically easier, in the sense that training and test instances are very similar. Later challenges gradually make the sets more distinct, and therefore harder to learn from. + +### Baseline results + +To illustrate the performance of `LearningSolver`, and to set a baseline for newly proposed techniques, we present in this page, for each benchmark challenge, a small set of computational results measuring the solution speed of the solver and the solution quality with default parameters. For more detailed computational studies, see [references](about.md#references). We compare three solvers: + +* **baseline:** Gurobi 9.0 with default settings (a conventional state-of-the-art MIP solver) +* **ml-exact:** `LearningSolver` with default settings, using Gurobi 9.0 as internal MIP solver +* **ml-heuristic:** Same as above, but with `mode="heuristic"` + +All experiments presented here were performed on a Linux server (Ubuntu Linux 18.04 LTS) with Intel Xeon Gold 6230s (2 processors, 40 cores, 80 threads) and 256 GB RAM (DDR4, 2933 MHz). All solvers were restricted to use 4 threads, with no time limits, and 10 instances were solved simultaneously at a time. + +## Maximum Weight Stable Set Problem + +### Problem definition + +Given a simple undirected graph $G=(V,E)$ and weights $w \in \mathbb{R}^V$, the problem is to find a stable set $S \subseteq V$ that maximizes $ \sum_{v \in V} w_v$. We recall that a subset $S \subseteq V$ is a *stable set* if no two vertices of $S$ are adjacent. This is one of Karp's 21 NP-complete problems. + +### Random instance generators + +The class `MaxWeightStableSetGenerator` can be used to generate random instances of this problem, with user-specified probability distributions. When the constructor parameter `fix_graph=True` is provided, one random Erdős-Rényi graph $G_{n,p}$ is generated during the constructor, where $n$ and $p$ are sampled from user-provided probability distributions `n` and `p`. To generate each instance, the generator independently samples each $w_v$ from the user-provided probability distribution `w`. When `fix_graph=False`, a new random graph is generated for each instance, while the remaining parameters are sampled in the same way. + +### Benchmark challenges + +#### Challenge A + +* Fixed random Erdős-Rényi graph $G_{n,p}$ with $n=200$ and $p=5\%$ +* Random vertex weights $w_v \sim U(100, 150)$ +* 300 training instances, 50 test instances + +```python +MaxWeightStableSetGenerator(w=uniform(loc=100., scale=50.), + n=randint(low=200, high=201), + density=uniform(loc=0.05, scale=0.0), + fix_graph=True) +``` + +### Benchmark results + +#### Challenge A + +![alt](figures/mwss.png) diff --git a/mkdocs.yml b/mkdocs.yml index a7f8b03..b3e22aa 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,12 +1,19 @@ site_name: MIPLearn theme: cinder +copyright: Copyright (C) 2019-2020 Argonne National Laboratory. All rights reserved. +repo_url: https://github.com/iSoron/miplearn nav: - Home: index.md - Usage: usage.md - Benchmark: benchmark.md + - Problems: problems.md - Customization: customization.md - About: about.md plugins: - search markdown_extensions: - - admonition \ No newline at end of file + - admonition + - mdx_math +extra_javascript: + - https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML + - js/mathjax.js \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 127c973..51e8597 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,4 +4,4 @@ pytest sklearn networkx tqdm -pandas \ No newline at end of file +pandas