You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
291 lines
12 KiB
291 lines
12 KiB
<!DOCTYPE html>
|
|
<html lang="en">
|
|
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
|
|
|
|
|
|
<link rel="../img/favicon.ico">
|
|
|
|
|
|
<title>Benchmark - MIPLearn</title>
|
|
|
|
|
|
<link rel="stylesheet" href="//use.fontawesome.com/releases/v5.5.0/css/all.css" integrity="sha384-B4dIYHKNBt8Bc12p+WXckhzcICo0wtJAoU8YZTY5qE0Id1GSseTk6S+L3BlXeVIU" crossorigin="anonymous">
|
|
<link rel="stylesheet" href="//cdn.jsdelivr.net/npm/hack-font@3.3.0/build/web/hack.min.css">
|
|
<link href='//fonts.googleapis.com/css?family=PT+Sans:400,400italic,700,700italic&subset=latin-ext,latin' rel='stylesheet' type='text/css'>
|
|
<link href='//fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,700italic,400,300,600,700&subset=latin-ext,latin' rel='stylesheet' type='text/css'>
|
|
<link href="../css/bootstrap-custom.min.css" rel="stylesheet">
|
|
<link href="../css/base.min.css" rel="stylesheet">
|
|
<link href="../css/cinder.min.css" rel="stylesheet">
|
|
<link href="../css/highlight.min.css" rel="stylesheet">
|
|
<link href="../css/custom.css" rel="stylesheet">
|
|
|
|
<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
|
|
<!--[if lt IE 9]>
|
|
<script src="https://cdn.jsdelivr.net/npm/html5shiv@3.7.3/dist/html5shiv.min.js"></script>
|
|
<script src="https://cdn.jsdelivr.net/npm/respond.js@1.4.2/dest/respond.min.js"></script>
|
|
<![endif]-->
|
|
|
|
<script src="//ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js"></script>
|
|
<script>
|
|
WebFont.load({
|
|
google: {
|
|
families: ['Open Sans', 'PT Sans']
|
|
}
|
|
});
|
|
</script>
|
|
|
|
|
|
|
|
|
|
</head>
|
|
|
|
<body>
|
|
|
|
<div class="navbar navbar-default navbar-fixed-top" role="navigation">
|
|
<div class="container">
|
|
|
|
<!-- Collapsed navigation -->
|
|
<div class="navbar-header">
|
|
<!-- Expander button -->
|
|
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
|
|
<span class="sr-only">Toggle navigation</span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
</button>
|
|
|
|
|
|
<!-- Main title -->
|
|
|
|
<a class="navbar-brand" href="..">MIPLearn</a>
|
|
</div>
|
|
|
|
<!-- Expanded navigation -->
|
|
<div class="navbar-collapse collapse">
|
|
<!-- Main navigation -->
|
|
<ul class="nav navbar-nav">
|
|
|
|
|
|
<li >
|
|
<a href="..">Home</a>
|
|
</li>
|
|
|
|
|
|
|
|
<li >
|
|
<a href="../usage/">Usage</a>
|
|
</li>
|
|
|
|
|
|
|
|
<li class="active">
|
|
<a href="./">Benchmark</a>
|
|
</li>
|
|
|
|
|
|
|
|
<li >
|
|
<a href="../problems/">Problems</a>
|
|
</li>
|
|
|
|
|
|
|
|
<li >
|
|
<a href="../customization/">Customization</a>
|
|
</li>
|
|
|
|
|
|
|
|
<li >
|
|
<a href="../about/">About</a>
|
|
</li>
|
|
|
|
|
|
</ul>
|
|
|
|
<ul class="nav navbar-nav navbar-right">
|
|
<li>
|
|
<a href="#" data-toggle="modal" data-target="#mkdocs_search_modal">
|
|
<i class="fas fa-search"></i> Search
|
|
</a>
|
|
</li>
|
|
<li >
|
|
<a rel="prev" href="../usage/">
|
|
<i class="fas fa-arrow-left"></i> Previous
|
|
</a>
|
|
</li>
|
|
<li >
|
|
<a rel="next" href="../problems/">
|
|
Next <i class="fas fa-arrow-right"></i>
|
|
</a>
|
|
</li>
|
|
<li>
|
|
<a href="https://github.com/ANL-CEEESA/MIPLearn/edit/master/src/docs/benchmark.md"><i class="fab fa-github"></i> Edit on GitHub</a>
|
|
</li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="container">
|
|
|
|
|
|
<div class="col-md-3"><div class="bs-sidebar hidden-print affix well" role="complementary">
|
|
<ul class="nav bs-sidenav">
|
|
<li class="first-level active"><a href="#benchmarks-utilities">Benchmarks Utilities</a></li>
|
|
<li class="second-level"><a href="#using-benchmarkrunner">Using BenchmarkRunner</a></li>
|
|
|
|
<li class="second-level"><a href="#saving-and-loading-benchmark-results">Saving and loading benchmark results</a></li>
|
|
|
|
</ul>
|
|
</div></div>
|
|
<div class="col-md-9" role="main">
|
|
|
|
<h1 id="benchmarks-utilities">Benchmarks Utilities</h1>
|
|
<h3 id="using-benchmarkrunner">Using <code>BenchmarkRunner</code></h3>
|
|
<p>MIPLearn provides the utility class <code>BenchmarkRunner</code>, which simplifies the task of comparing the performance of different solvers. The snippet below shows its basic usage:</p>
|
|
<pre><code class="python">from miplearn import BenchmarkRunner, LearningSolver
|
|
|
|
# Create train and test instances
|
|
train_instances = [...]
|
|
test_instances = [...]
|
|
|
|
# Training phase...
|
|
training_solver = LearningSolver(...)
|
|
training_solver.parallel_solve(train_instances, n_jobs=10)
|
|
|
|
# Test phase...
|
|
test_solvers = {
|
|
"Baseline": LearningSolver(...), # each solver may have different parameters
|
|
"Strategy A": LearningSolver(...),
|
|
"Strategy B": LearningSolver(...),
|
|
"Strategy C": LearningSolver(...),
|
|
}
|
|
benchmark = BenchmarkRunner(test_solvers)
|
|
benchmark.fit(train_instances)
|
|
benchmark.parallel_solve(test_instances, n_jobs=2)
|
|
print(benchmark.raw_results())
|
|
</code></pre>
|
|
|
|
<p>The method <code>fit</code> trains the ML models for each individual solver. The method <code>parallel_solve</code> solves the test instances in parallel, and collects solver statistics such as running time and optimal value. Finally, <code>raw_results</code> produces a table of results (Pandas DataFrame) with the following columns:</p>
|
|
<ul>
|
|
<li><strong>Solver,</strong> the name of the solver.</li>
|
|
<li><strong>Instance,</strong> the sequence number identifying the instance.</li>
|
|
<li><strong>Wallclock Time,</strong> the wallclock running time (in seconds) spent by the solver;</li>
|
|
<li><strong>Lower Bound,</strong> the best lower bound obtained by the solver;</li>
|
|
<li><strong>Upper Bound,</strong> the best upper bound obtained by the solver;</li>
|
|
<li><strong>Gap,</strong> the relative MIP integrality gap at the end of the optimization;</li>
|
|
<li><strong>Nodes,</strong> the number of explored branch-and-bound nodes.</li>
|
|
</ul>
|
|
<p>In addition to the above, there is also a "Relative" version of most columns, where the raw number is compared to the solver which provided the best performance. The <em>Relative Wallclock Time</em> for example, indicates how many times slower this run was when compared to the best time achieved by any solver when processing this instance. For example, if this run took 10 seconds, but the fastest solver took only 5 seconds to solve the same instance, the relative wallclock time would be 2.</p>
|
|
<h3 id="saving-and-loading-benchmark-results">Saving and loading benchmark results</h3>
|
|
<p>When iteratively exploring new formulations, encoding and solver parameters, it is often desirable to avoid repeating parts of the benchmark suite. For example, if the baseline solver has not been changed, there is no need to evaluate its performance again and again when making small changes to the remaining solvers. <code>BenchmarkRunner</code> provides the methods <code>save_results</code> and <code>load_results</code>, which can be used to avoid this repetition, as the next example shows:</p>
|
|
<pre><code class="python"># Benchmark baseline solvers and save results to a file.
|
|
benchmark = BenchmarkRunner(baseline_solvers)
|
|
benchmark.parallel_solve(test_instances)
|
|
benchmark.save_results("baseline_results.csv")
|
|
|
|
# Benchmark remaining solvers, loading baseline results from file.
|
|
benchmark = BenchmarkRunner(alternative_solvers)
|
|
benchmark.load_results("baseline_results.csv")
|
|
benchmark.fit(training_instances)
|
|
benchmark.parallel_solve(test_instances)
|
|
</code></pre></div>
|
|
|
|
|
|
</div>
|
|
|
|
<footer class="col-md-12 text-center">
|
|
|
|
<hr>
|
|
<p>
|
|
<small>Copyright © 2020, UChicago Argonne, LLC. All Rights Reserved.<br></small>
|
|
|
|
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
|
|
|
|
|
|
|
|
</footer>
|
|
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script>
|
|
<script src="../js/bootstrap-3.0.3.min.js"></script>
|
|
<script src="../js/highlight.pack.js"></script>
|
|
<script>hljs.initHighlightingOnLoad();</script>
|
|
<script>var base_url = ".."</script>
|
|
|
|
<script src="../js/base.js"></script>
|
|
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
|
|
<script src="../js/mathjax.js"></script>
|
|
<script src="../search/main.js"></script>
|
|
|
|
<div class="modal" id="mkdocs_search_modal" tabindex="-1" role="dialog" aria-labelledby="searchModalLabel" aria-hidden="true">
|
|
<div class="modal-dialog modal-lg">
|
|
<div class="modal-content">
|
|
<div class="modal-header">
|
|
<h4 class="modal-title" id="searchModalLabel">Search</h4>
|
|
<button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">×</span><span class="sr-only">Close</span></button>
|
|
</div>
|
|
<div class="modal-body">
|
|
<p>
|
|
From here you can search these documents. Enter
|
|
your search terms below.
|
|
</p>
|
|
<form>
|
|
<div class="form-group">
|
|
<input type="text" class="form-control" placeholder="Search..." id="mkdocs-search-query" title="Type search term here">
|
|
</div>
|
|
</form>
|
|
<div id="mkdocs-search-results"></div>
|
|
</div>
|
|
<div class="modal-footer">
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div><div class="modal" id="mkdocs_keyboard_modal" tabindex="-1" role="dialog" aria-labelledby="keyboardModalLabel" aria-hidden="true">
|
|
<div class="modal-dialog">
|
|
<div class="modal-content">
|
|
<div class="modal-header">
|
|
<h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
|
|
<button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">×</span><span class="sr-only">Close</span></button>
|
|
</div>
|
|
<div class="modal-body">
|
|
<table class="table">
|
|
<thead>
|
|
<tr>
|
|
<th style="width: 20%;">Keys</th>
|
|
<th>Action</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td class="help shortcut"><kbd>?</kbd></td>
|
|
<td>Open this help</td>
|
|
</tr>
|
|
<tr>
|
|
<td class="next shortcut"><kbd>n</kbd></td>
|
|
<td>Next page</td>
|
|
</tr>
|
|
<tr>
|
|
<td class="prev shortcut"><kbd>p</kbd></td>
|
|
<td>Previous page</td>
|
|
</tr>
|
|
<tr>
|
|
<td class="search shortcut"><kbd>s</kbd></td>
|
|
<td>Search</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="modal-footer">
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</body>
|
|
|
|
</html>
|