Fix failing test due to wrong solution.jl input of reserve shortfall

Fix duplicated startup constraint

CHANGELOG.md

@@ -11,6 +11,11 @@ All notable changes to this project will be documented in this file.
 [semver]: https://semver.org/spec/v2.0.0.html
 [pkjjl]: https://pkgdocs.julialang.org/v1/compatibility/#compat-pre-1.0
 
+## [0.2.2] - 2021-07-21
+### Fixed
+- Fix small bug in validation scripts related to startup costs
+- Fix duplicated startup constraints (@mtanneau, #12)
+
 ## [0.2.1] - 2021-06-02
 ### Added
 - Add multiple ramping formulations (ArrCon2000, MorLatRam2013, DamKucRajAta2016, PanGua2016)

Project.toml

@@ -2,10 +2,11 @@ name = "UnitCommitment"
 uuid = "64606440-39ea-11e9-0f29-3303a1d3d877"
 authors = ["Santos Xavier, Alinson <axavier@anl.gov>"]
 repo = "https://github.com/ANL-CEEESA/UnitCommitment.jl"
-version = "0.2.1"
+version = "0.2.3"
 
 [deps]
 DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
+Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
 GZip = "92fee26a-97fe-5a0c-ad85-20a5f3185b63"
 JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
 JuMP = "4076af6c-e467-56ae-b986-b466b2749572"
@@ -19,6 +20,7 @@ SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"
 [compat]
 Cbc = "0.7"
 DataStructures = "0.18"
+Distributions = "0.25"
 GZip = "0.5"
 JSON = "0.21"
 JuMP = "0.21"
@@ -29,6 +31,7 @@ julia = "1"
 [extras]
 Cbc = "9961bab8-2fa3-5c5a-9d89-47fab24efd76"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
+Gurobi = "2e9cd046-0924-5485-92f1-d5272153d98b"
 
 [targets]
-test = ["Cbc", "Test"]
+test = ["Cbc", "Test", "Gurobi"]

README.md

@@ -20,7 +20,7 @@
 
 * **Data Format:** The package proposes an extensible and fully-documented JSON-based data format for SCUC, developed in collaboration with Independent System Operators (ISOs), which describes the most important aspects of the problem. The format supports the most common generator characteristics (including ramping, piecewise-linear production cost curves and time-dependent startup costs), as well as operating reserves, price-sensitive loads, transmission networks and contingencies.
 * **Benchmark Instances:** The package provides a diverse collection of large-scale benchmark instances collected from the literature, converted into a common data format, and extended using data-driven methods to make them more challenging and realistic.
-* **Model Implementation**: The package provides a Julia/JuMP implementations of state-of-the-art formulations and solution methods for SCUC, including multiple ramping formulations ([ArrCon2000][ArrCon2000], [MorLatRam2013][MorLatRam2013], [DamKucRajAta2016][DamKucRajAta2016], [PanGua2016][PanGua2016]), multiple piecewise-linear costs formulations ([Gar1962][Gar1962], [CarArr2006][CarArr2006], [KnuOstWat2018][KnuOstWat2018]) and contingency screening methods ([XavQiuWanThi2019][XavQiuWanThi2019]). Our goal is to keep these implementations up-to-date as new methods are proposed in the literature.
+* **Model Implementation**: The package provides Julia/JuMP implementations of state-of-the-art formulations and solution methods for SCUC, including multiple ramping formulations ([ArrCon2000][ArrCon2000], [MorLatRam2013][MorLatRam2013], [DamKucRajAta2016][DamKucRajAta2016], [PanGua2016][PanGua2016]), multiple piecewise-linear costs formulations ([Gar1962][Gar1962], [CarArr2006][CarArr2006], [KnuOstWat2018][KnuOstWat2018]) and contingency screening methods ([XavQiuWanThi2019][XavQiuWanThi2019]). Our goal is to keep these implementations up-to-date as new methods are proposed in the literature.
 * **Benchmark Tools:** The package provides automated benchmark scripts to accurately evaluate the performance impact of proposed code changes.
 
 [ArrCon2000]: https://doi.org/10.1109/59.871739

benchmark/scripts/table.py

@@ -6,6 +6,9 @@ from pathlib import Path
 import pandas as pd
 import re
 from tabulate import tabulate
+from colorama import init, Fore, Back, Style
+
+init()
 
 
 def process_all_log_files():
@@ -48,6 +51,7 @@ def process(filename):
     # m = re.search("case([0-9]*)", instance_name)
     # n_buses = int(m.group(1))
     n_buses = 0
+    validation_errors = 0
 
     with open(filename) as file:
         for line in file.readlines():
@@ -137,6 +141,14 @@ def process(filename):
             if m is not None:
                 transmission_count += 1
 
+            m = re.search(r".*Found ([0-9]*) validation errors", line)
+            if m is not None:
+                validation_errors += int(m.group(1))
+                print(
+                    f"{Fore.YELLOW}{Style.BRIGHT}Warning:{Style.RESET_ALL} {validation_errors:8d} "
+                    f"{Style.DIM}validation errors in {Style.RESET_ALL}{group_name}/{instance_name}/{sample_name}"
+                )
+
     return {
         "Group": group_name,
         "Instance": instance_name,
@@ -168,6 +180,7 @@ def process(filename):
         "Transmission screening constraints": transmission_count,
         "Transmission screening time": transmission_time,
         "Transmission screening calls": transmission_calls,
+        "Validation errors": validation_errors,
     }
 
 

docs/format.md

@@ -28,13 +28,14 @@ Each section is described in detail below. For a complete example, see [case14](
 
 ### Parameters
 
-This section describes system-wide parameters, such as power balance penalties,  optimization parameters, such as the length of the planning horizon and the time.
+This section describes system-wide parameters, such as power balance and reserve shortfall penalties, and optimization parameters, such as the length of the planning horizon and the time.
 
 | Key                            | Description                                       | Default  | Time series?
 | :----------------------------- | :------------------------------------------------ | :------: | :------------:
-| `Time horizon (h)`                     | Length of the planning horizon (in hours). | Required | N
+| `Time horizon (h)` | Length of the planning horizon (in hours). | Required | N
 | `Time step (min)` | Length of each time step (in minutes). Must be a divisor of 60 (e.g. 60, 30, 20, 15, etc). | `60` | N
 | `Power balance penalty ($/MW)` | Penalty for system-wide shortage or surplus in production (in $/MW). This is charged per time step. For example, if there is a shortage of 1 MW for three time steps, three times this amount will be charged. | `1000.0` | Y
+| `Reserve shortfall penalty ($/MW)` | Penalty for system-wide shortage in meeting reserve requirements (in $/MW). This is charged per time step. Negative value implies reserve constraints must always be satisfied. | `-1` | Y
 
 
 #### Example
@@ -42,7 +43,8 @@ This section describes system-wide parameters, such as power balance penalties,
 {
     "Parameters": {
         "Time horizon (h)": 4,
-        "Power balance penalty ($/MW)": 1000.0
+        "Power balance penalty ($/MW)": 1000.0,
+        "Reserve shortfall penalty ($/MW)": -1.0
     }
 }
 ```

docs/model.md

@@ -148,7 +148,7 @@ for g in instance.units
 end
 ```
 
-### Modifying the model
+### Fixing variables, modifying objective function and adding constraints
 
 Since we now have a direct reference to the JuMP decision variables, it is possible to fix variables, change the coefficients in the objective function, or even add new constraints to the model before solving it. The script below shows how can this be accomplished. For more information on modifying an existing model, [see the JuMP documentation](https://jump.dev/JuMP.jl/stable/manual/variables/).
 
@@ -190,6 +190,54 @@ JuMP.set_objective_coefficient(
 UnitCommitment.optimize!(model)
 ```
 
+### Adding new component to a bus
+
+The following snippet shows how to add a new grid component to a particular bus. For each time step, we create decision variables for the new grid component, add these variables to the objective function, then attach the component to a particular bus by modifying some existing model constraints. 
+
+```julia
+using Cbc
+using JuMP
+using UnitCommitment
+
+# Load instance and build base model
+instance = UnitCommitment.read_benchmark("matpower/case118/2017-02-01")
+model = UnitCommitment.build_model(
+    instance=instance,
+    optimizer=Cbc.Optimizer,
+)
+
+# Get the number of time steps in the original instance
+T = instance.time
+
+# Create decision variables for the new grid component.
+# In this example, we assume that the new component can
+# inject up to 10 MW of power at each time step, so we
+# create new continuous variables 0 ≤ x[t] ≤ 10.
+@variable(model, x[1:T], lower_bound=0.0, upper_bound=10.0)
+
+# For each time step
+for t in 1:T
+
+    # Add production costs to the objective function.
+    # In this example, we assume a cost of $5/MW.
+    set_objective_coefficient(model, x[t], 5.0)
+
+    # Attach the new component to bus b1, by modifying the
+    # constraint `eq_net_injection`.
+    set_normalized_coefficient(
+        model[:eq_net_injection]["b1", t],
+        x[t],
+        1.0,
+    )
+end
+
+# Solve the model
+UnitCommitment.optimize!(model)
+
+# Show optimal values for the x variables
+@show value.(x)
+```
+
 References
 ----------
 * [KnOsWa20] **Bernard Knueven, James Ostrowski and Jean-Paul Watson.** "On Mixed-Integer Programming Formulations for the Unit Commitment Problem". INFORMS Journal on Computing (2020). [DOI: 10.1287/ijoc.2019.0944](https://doi.org/10.1287/ijoc.2019.0944)

docs/usage.md

@@ -71,7 +71,7 @@ Advanced usage
 
 ### Customizing the formulation
 
-By default, `build_model` uses a formulation that combines modeling components from different publications, and that has been carefully tested, using our own benchmark scripts, to provide good performance across a wide variety of instances. This default formulation is expected to change over time, as new methods are proposed in the literature. You can, however, construct your own formulation, based the modeling components that you choose, as shown in the next example.
+By default, `build_model` uses a formulation that combines modeling components from different publications, and that has been carefully tested, using our own benchmark scripts, to provide good performance across a wide variety of instances. This default formulation is expected to change over time, as new methods are proposed in the literature. You can, however, construct your own formulation, based on the modeling components that you choose, as shown in the next example.
 
 ```julia
 using Cbc
@@ -119,7 +119,10 @@ instance = UnitCommitment.read("instance.json")
 UnitCommitment.generate_initial_conditions!(instance, Cbc.Optimizer)
 
 # Construct and solve optimization model
-model = UnitCommitment.build_model(instance, Cbc.Optimizer)
+model = UnitCommitment.build_model(
+    instance=instance,
+    optimizer=Cbc.Optimizer,
+)
 UnitCommitment.optimize!(model)
 ```
 

src/UnitCommitment.jl

@@ -30,6 +30,8 @@ include("model/formulations/base/unit.jl")
 include("model/formulations/CarArr2006/pwlcosts.jl")
 include("model/formulations/DamKucRajAta2016/ramp.jl")
 include("model/formulations/Gar1962/pwlcosts.jl")
+include("model/formulations/Gar1962/status.jl")
+include("model/formulations/Gar1962/prod.jl")
 include("model/formulations/KnuOstWat2018/pwlcosts.jl")
 include("model/formulations/MorLatRam2013/ramp.jl")
 include("model/formulations/MorLatRam2013/scosts.jl")
@@ -46,6 +48,7 @@ include("solution/warmstart.jl")
 include("solution/write.jl")
 include("transform/initcond.jl")
 include("transform/slice.jl")
+include("transform/randomize.jl")
 include("utils/log.jl")
 include("validation/repair.jl")
 include("validation/validate.jl")

src/instance/read.jl

@@ -98,6 +98,10 @@ function _from_json(json; repair = true)
         json["Parameters"]["Power balance penalty (\$/MW)"],
         default = [1000.0 for t in 1:T],
     )
+    shortfall_penalty = timeseries(
+        json["Parameters"]["Reserve shortfall penalty (\$/MW)"],
+        default = [-1.0 for t in 1:T],
+    )
 
     # Read buses
     for (bus_name, dict) in json["Buses"]
@@ -264,6 +268,7 @@ function _from_json(json; repair = true)
     instance = UnitCommitmentInstance(
         T,
         power_balance_penalty,
+        shortfall_penalty,
         units,
         buses,
         lines,

src/instance/structs.jl

@@ -72,6 +72,8 @@ end
 mutable struct UnitCommitmentInstance
     time::Int
     power_balance_penalty::Vector{Float64}
+    "Penalty for failing to meet reserve requirement."
+    shortfall_penalty::Vector{Float64}
     units::Vector{Unit}
     buses::Vector{Bus}
     lines::Vector{TransmissionLine}

src/model/formulations/ArrCon2000/ramp.jl

@@ -5,7 +5,9 @@
 function _add_ramp_eqs!(
     model::JuMP.Model,
     g::Unit,
-    formulation::ArrCon2000.Ramping,
+    formulation_prod_vars::Gar1962.ProdVars,
+    formulation_ramping::ArrCon2000.Ramping,
+    formulation_status_vars::Gar1962.StatusVars,
 )::Nothing
     # TODO: Move upper case constants to model[:instance]
     RESERVES_WHEN_START_UP = true
@@ -17,15 +19,19 @@ function _add_ramp_eqs!(
     RD = g.ramp_down_limit
     SU = g.startup_limit
     SD = g.shutdown_limit
-    is_on = model[:is_on]
-    prod_above = model[:prod_above]
     reserve = model[:reserve]
-    switch_off = model[:switch_off]
-    switch_on = model[:switch_on]
     eq_ramp_down = _init(model, :eq_ramp_down)
     eq_ramp_up = _init(model, :eq_ramp_up)
     is_initially_on = (g.initial_status > 0)
 
+    # Gar1962.ProdVars
+    prod_above = model[:prod_above]
+
+    # Gar1962.StatusVars
+    is_on = model[:is_on]
+    switch_off = model[:switch_off]
+    switch_on = model[:switch_on]
+
     for t in 1:model[:instance].time
         # Ramp up limit
         if t == 1

src/model/formulations/CarArr2006/pwlcosts.jl

@@ -5,13 +5,18 @@
 function _add_production_piecewise_linear_eqs!(
     model::JuMP.Model,
     g::Unit,
-    formulation::CarArr2006.PwlCosts,
+    formulation_prod_vars::Gar1962.ProdVars,
+    formulation_pwl_costs::CarArr2006.PwlCosts,
+    formulation_status_vars::StatusVarsFormulation,
 )::Nothing
     eq_prod_above_def = _init(model, :eq_prod_above_def)
     eq_segprod_limit = _init(model, :eq_segprod_limit)
-    prod_above = model[:prod_above]
     segprod = model[:segprod]
     gn = g.name
+
+    # Gar1962.ProdVars
+    prod_above = model[:prod_above]
+
     K = length(g.cost_segments)
     for t in 1:model[:instance].time
         gn = g.name

src/model/formulations/DamKucRajAta2016/ramp.jl

@@ -5,7 +5,9 @@
 function _add_ramp_eqs!(
     model::JuMP.Model,
     g::Unit,
-    formulation::DamKucRajAta2016.Ramping,
+    formulation_prod_vars::Gar1962.ProdVars,
+    formulation_ramping::DamKucRajAta2016.Ramping,
+    formulation_status_vars::Gar1962.StatusVars,
 )::Nothing
     # TODO: Move upper case constants to model[:instance]
     RESERVES_WHEN_START_UP = true
@@ -21,9 +23,13 @@ function _add_ramp_eqs!(
     gn = g.name
     eq_str_ramp_down = _init(model, :eq_str_ramp_down)
     eq_str_ramp_up = _init(model, :eq_str_ramp_up)
-    is_on = model[:is_on]
-    prod_above = model[:prod_above]
     reserve = model[:reserve]
+
+    # Gar1962.ProdVars
+    prod_above = model[:prod_above]
+
+    # Gar1962.StatusVars
+    is_on = model[:is_on]
     switch_off = model[:switch_off]
     switch_on = model[:switch_on]
 

src/model/formulations/Gar1962/prod.jl

@@ -0,0 +1,50 @@
+# UnitCommitment.jl: Optimization Package for Security-Constrained Unit Commitment
+# Copyright (C) 2020, UChicago Argonne, LLC. All rights reserved.
+# Released under the modified BSD license. See COPYING.md for more details.
+
+function _add_production_vars!(
+    model::JuMP.Model,
+    g::Unit,
+    formulation_prod_vars::Gar1962.ProdVars,
+)::Nothing
+    prod_above = _init(model, :prod_above)
+    segprod = _init(model, :segprod)
+    for t in 1:model[:instance].time
+        for k in 1:length(g.cost_segments)
+            segprod[g.name, t, k] = @variable(model, lower_bound = 0)
+        end
+        prod_above[g.name, t] = @variable(model, lower_bound = 0)
+    end
+    return
+end
+
+function _add_production_limit_eqs!(
+    model::JuMP.Model,
+    g::Unit,
+    formulation_prod_vars::Gar1962.ProdVars,
+)::Nothing
+    eq_prod_limit = _init(model, :eq_prod_limit)
+    is_on = model[:is_on]
+    prod_above = model[:prod_above]
+    reserve = model[:reserve]
+    gn = g.name
+    for t in 1:model[:instance].time
+        # Objective function terms for production costs
+        # Part of (69) of Kneuven et al. (2020) as C^R_g * u_g(t) term
+        add_to_expression!(model[:obj], is_on[gn, t], g.min_power_cost[t])
+
+        # Production limit
+        # Equation (18) in Kneuven et al. (2020)
+        #   as \bar{p}_g(t) \le \bar{P}_g u_g(t)
+        # amk: this is a weaker version of (20) and (21) in Kneuven et al. (2020)
+        #      but keeping it here in case those are not present
+        power_diff = max(g.max_power[t], 0.0) - max(g.min_power[t], 0.0)
+        if power_diff < 1e-7
+            power_diff = 0.0
+        end
+        eq_prod_limit[gn, t] = @constraint(
+            model,
+            prod_above[gn, t] + reserve[gn, t] <= power_diff * is_on[gn, t]
+        )
+    end
+end

src/model/formulations/Gar1962/pwlcosts.jl

@@ -5,14 +5,21 @@
 function _add_production_piecewise_linear_eqs!(
     model::JuMP.Model,
     g::Unit,
-    formulation::Gar1962.PwlCosts,
+    formulation_prod_vars::Gar1962.ProdVars,
+    formulation_pwl_costs::Gar1962.PwlCosts,
+    formulation_status_vars::Gar1962.StatusVars,
 )::Nothing
     eq_prod_above_def = _init(model, :eq_prod_above_def)
     eq_segprod_limit = _init(model, :eq_segprod_limit)
-    is_on = model[:is_on]
-    prod_above = model[:prod_above]
     segprod = model[:segprod]
     gn = g.name
+
+    # Gar1962.ProdVars
+    prod_above = model[:prod_above]
+
+    # Gar1962.StatusVars
+    is_on = model[:is_on]
+
     K = length(g.cost_segments)
     for t in 1:model[:instance].time
         # Definition of production

src/model/formulations/Gar1962/status.jl

@@ -0,0 +1,61 @@
+# UnitCommitment.jl: Optimization Package for Security-Constrained Unit Commitment
+# Copyright (C) 2020, UChicago Argonne, LLC. All rights reserved.
+# Released under the modified BSD license. See COPYING.md for more details.
+
+function _add_status_vars!(
+    model::JuMP.Model,
+    g::Unit,
+    formulation_status_vars::Gar1962.StatusVars,
+)::Nothing
+    is_on = _init(model, :is_on)
+    switch_on = _init(model, :switch_on)
+    switch_off = _init(model, :switch_off)
+    for t in 1:model[:instance].time
+        if g.must_run[t]
+            is_on[g.name, t] = 1.0
+            switch_on[g.name, t] = (t == 1 ? 1.0 - _is_initially_on(g) : 0.0)
+            switch_off[g.name, t] = 0.0
+        else
+            is_on[g.name, t] = @variable(model, binary = true)
+            switch_on[g.name, t] = @variable(model, binary = true)
+            switch_off[g.name, t] = @variable(model, binary = true)
+        end
+    end
+    return
+end
+
+function _add_status_eqs!(
+    model::JuMP.Model,
+    g::Unit,
+    formulation_status_vars::Gar1962.StatusVars,
+)::Nothing
+    eq_binary_link = _init(model, :eq_binary_link)
+    eq_switch_on_off = _init(model, :eq_switch_on_off)
+    is_on = model[:is_on]
+    switch_off = model[:switch_off]
+    switch_on = model[:switch_on]
+    for t in 1:model[:instance].time
+        if !g.must_run[t]
+            # Link binary variables
+            if t == 1
+                eq_binary_link[g.name, t] = @constraint(
+                    model,
+                    is_on[g.name, t] - _is_initially_on(g) ==
+                    switch_on[g.name, t] - switch_off[g.name, t]
+                )
+            else
+                eq_binary_link[g.name, t] = @constraint(
+                    model,
+                    is_on[g.name, t] - is_on[g.name, t-1] ==
+                    switch_on[g.name, t] - switch_off[g.name, t]
+                )
+            end
+            # Cannot switch on and off at the same time
+            eq_switch_on_off[g.name, t] = @constraint(
+                model,
+                switch_on[g.name, t] + switch_off[g.name, t] <= 1
+            )
+        end
+    end
+    return
+end

src/model/formulations/Gar1962/structs.jl

@@ -14,7 +14,11 @@ Formulation described in:
 module Gar1962
 
 import ..PiecewiseLinearCostsFormulation
+import ..ProductionVarsFormulation
+import ..StatusVarsFormulation
 
+struct ProdVars <: ProductionVarsFormulation end
 struct PwlCosts <: PiecewiseLinearCostsFormulation end
+struct StatusVars <: StatusVarsFormulation end
 
 end

src/model/formulations/KnuOstWat2018/pwlcosts.jl

@@ -5,21 +5,27 @@
 function _add_production_piecewise_linear_eqs!(
     model::JuMP.Model,
     g::Unit,
-    formulation::KnuOstWat2018.PwlCosts,
+    formulation_prod_vars::Gar1962.ProdVars,
+    formulation_pwl_costs::KnuOstWat2018.PwlCosts,
+    formulation_status_vars::Gar1962.StatusVars,
 )::Nothing
     eq_prod_above_def = _init(model, :eq_prod_above_def)
     eq_segprod_limit_a = _init(model, :eq_segprod_limit_a)
     eq_segprod_limit_b = _init(model, :eq_segprod_limit_b)
     eq_segprod_limit_c = _init(model, :eq_segprod_limit_c)
-    prod_above = model[:prod_above]
     segprod = model[:segprod]
-    is_on = model[:is_on]
-    switch_on = model[:switch_on]
-    switch_off = model[:switch_off]
     gn = g.name
     K = length(g.cost_segments)
     T = model[:instance].time
 
+    # Gar1962.ProdVars
+    prod_above = model[:prod_above]
+
+    # Gar1962.StatusVars
+    is_on = model[:is_on]
+    switch_on = model[:switch_on]
+    switch_off = model[:switch_off]
+
     for t in 1:T
         for k in 1:K
             # Pbar^{k-1)

src/model/formulations/MorLatRam2013/ramp.jl

@@ -5,7 +5,9 @@
 function _add_ramp_eqs!(
     model::JuMP.Model,
     g::Unit,
-    formulation::MorLatRam2013.Ramping,
+    formulation_prod_vars::Gar1962.ProdVars,
+    formulation_ramping::MorLatRam2013.Ramping,
+    formulation_status_vars::Gar1962.StatusVars,
 )::Nothing
     # TODO: Move upper case constants to model[:instance]
     RESERVES_WHEN_START_UP = true
@@ -20,9 +22,13 @@ function _add_ramp_eqs!(
     gn = g.name
     eq_ramp_down = _init(model, :eq_ramp_down)
     eq_ramp_up = _init(model, :eq_str_ramp_up)
-    is_on = model[:is_on]
-    prod_above = model[:prod_above]
     reserve = model[:reserve]
+
+    # Gar1962.ProdVars
+    prod_above = model[:prod_above]
+
+    # Gar1962.StatusVars
+    is_on = model[:is_on]
     switch_off = model[:switch_off]
     switch_on = model[:switch_on]
 

src/model/formulations/MorLatRam2013/scosts.jl

@@ -12,14 +12,14 @@ function _add_startup_cost_eqs!(
     S = length(g.startup_categories)
     startup = model[:startup]
     for t in 1:model[:instance].time
-        for s in 1:S
-            # If unit is switching on, we must choose a startup category
-            eq_startup_choose[g.name, t, s] = @constraint(
-                model,
-                model[:switch_on][g.name, t] ==
-                sum(startup[g.name, t, s] for s in 1:S)
-            )
+        # If unit is switching on, we must choose a startup category
+        eq_startup_choose[g.name, t] = @constraint(
+            model,
+            model[:switch_on][g.name, t] ==
+            sum(startup[g.name, t, s] for s in 1:S)
+        )
 
+        for s in 1:S
             # If unit has not switched off in the last `delay` time periods, startup category is forbidden.
             # The last startup category is always allowed.
             if s < S

src/model/formulations/PanGua2016/ramp.jl

@@ -1,16 +1,18 @@
+# UnitCommitment.jl: Optimization Package for Security-Constrained Unit Commitment
+# Copyright (C) 2020, UChicago Argonne, LLC. All rights reserved.
+# Released under the modified BSD license. See COPYING.md for more details.
+
 function _add_ramp_eqs!(
     model::JuMP.Model,
     g::Unit,
-    formulation::PanGua2016.Ramping,
+    formulation_prod_vars::Gar1962.ProdVars,
+    formulation_ramping::PanGua2016.Ramping,
+    formulation_status_vars::Gar1962.StatusVars,
 )::Nothing
     # TODO: Move upper case constants to model[:instance]
     RESERVES_WHEN_SHUT_DOWN = true
     gn = g.name
-    is_on = model[:is_on]
-    prod_above = model[:prod_above]
     reserve = model[:reserve]
-    switch_off = model[:switch_off]
-    switch_on = model[:switch_on]
     eq_str_prod_limit = _init(model, :eq_str_prod_limit)
     eq_prod_limit_ramp_up_extra_period =
         _init(model, :eq_prod_limit_ramp_up_extra_period)
@@ -23,6 +25,14 @@ function _add_ramp_eqs!(
     RD = g.ramp_down_limit # ramp down rate
     T = model[:instance].time
 
+    # Gar1962.ProdVars
+    prod_above = model[:prod_above]
+
+    # Gar1962.StatusVars
+    is_on = model[:is_on]
+    switch_off = model[:switch_off]
+    switch_on = model[:switch_on]
+
     for t in 1:T
         Pbar = g.max_power[t]
         if Pbar < 1e-7

src/model/formulations/base/bus.jl

@@ -4,15 +4,11 @@
 
 function _add_bus!(model::JuMP.Model, b::Bus)::Nothing
     net_injection = _init(model, :expr_net_injection)
-    reserve = _init(model, :expr_reserve)
     curtail = _init(model, :curtail)
     for t in 1:model[:instance].time
         # Fixed load
         net_injection[b.name, t] = AffExpr(-b.load[t])
 
-        # Reserves
-        reserve[b.name, t] = AffExpr()
-
         # Load curtailment
         curtail[b.name, t] =
             @variable(model, lower_bound = 0, upper_bound = b.load[t])

src/model/formulations/base/structs.jl

@@ -6,20 +6,33 @@ abstract type TransmissionFormulation end
 abstract type RampingFormulation end
 abstract type PiecewiseLinearCostsFormulation end
 abstract type StartupCostsFormulation end
+abstract type StatusVarsFormulation end
+abstract type ProductionVarsFormulation end
 
 struct Formulation
+    prod_vars::ProductionVarsFormulation
     pwl_costs::PiecewiseLinearCostsFormulation
     ramping::RampingFormulation
     startup_costs::StartupCostsFormulation
+    status_vars::StatusVarsFormulation
     transmission::TransmissionFormulation
 
     function Formulation(;
+        prod_vars::ProductionVarsFormulation = Gar1962.ProdVars(),
         pwl_costs::PiecewiseLinearCostsFormulation = KnuOstWat2018.PwlCosts(),
         ramping::RampingFormulation = MorLatRam2013.Ramping(),
         startup_costs::StartupCostsFormulation = MorLatRam2013.StartupCosts(),
+        status_vars::StatusVarsFormulation = Gar1962.StatusVars(),
         transmission::TransmissionFormulation = ShiftFactorsFormulation(),
     )
-        return new(pwl_costs, ramping, startup_costs, transmission)
+        return new(
+            prod_vars,
+            pwl_costs,
+            ramping,
+            startup_costs,
+            status_vars,
+            transmission,
+        )
     end
 end
 

src/model/formulations/base/system.jl

@@ -11,12 +11,12 @@ end
 function _add_net_injection_eqs!(model::JuMP.Model)::Nothing
     T = model[:instance].time
     net_injection = _init(model, :net_injection)
-    eq_net_injection_def = _init(model, :eq_net_injection_def)
+    eq_net_injection = _init(model, :eq_net_injection)
     eq_power_balance = _init(model, :eq_power_balance)
     for t in 1:T, b in model[:instance].buses
         n = net_injection[b.name, t] = @variable(model)
-        eq_net_injection_def[t, b.name] =
-            @constraint(model, n == model[:expr_net_injection][b.name, t])
+        eq_net_injection[b.name, t] =
+            @constraint(model, -n + model[:expr_net_injection][b.name, t] == 0)
     end
     for t in 1:T
         eq_power_balance[t] = @constraint(
@@ -29,13 +29,28 @@ end
 
 function _add_reserve_eqs!(model::JuMP.Model)::Nothing
     eq_min_reserve = _init(model, :eq_min_reserve)
-    for t in 1:model[:instance].time
+    instance = model[:instance]
+    for t in 1:instance.time
+        # Equation (68) in Kneuven et al. (2020)
+        # As in Morales-España et al. (2013a)
+        # Akin to the alternative formulation with max_power_avail
+        # from Carrión and Arroyo (2006) and Ostrowski et al. (2012)
+        shortfall_penalty = instance.shortfall_penalty[t]
         eq_min_reserve[t] = @constraint(
             model,
-            sum(
-                model[:expr_reserve][b.name, t] for b in model[:instance].buses
-            ) >= model[:instance].reserves.spinning[t]
+            sum(model[:reserve][g.name, t] for g in instance.units) +
+            (shortfall_penalty >= 0 ? model[:reserve_shortfall][t] : 0.0) >=
+            instance.reserves.spinning[t]
         )
+
+        # Account for shortfall contribution to objective
+        if shortfall_penalty >= 0
+            add_to_expression!(
+                model[:obj],
+                shortfall_penalty,
+                model[:reserve_shortfall][t],
+            )
+        end
     end
     return
 end

src/model/formulations/base/unit.jl

@@ -2,7 +2,7 @@
 # Copyright (C) 2020, UChicago Argonne, LLC. All rights reserved.
 # Released under the modified BSD license. See COPYING.md for more details.
 
-function _add_unit!(model::JuMP.Model, g::Unit, f::Formulation)
+function _add_unit!(model::JuMP.Model, g::Unit, formulation::Formulation)
     if !all(g.must_run) && any(g.must_run)
         error("Partially must-run units are not currently supported")
     end
@@ -11,72 +11,49 @@ function _add_unit!(model::JuMP.Model, g::Unit, f::Formulation)
     end
 
     # Variables
-    _add_production_vars!(model, g)
+    _add_production_vars!(model, g, formulation.prod_vars)
     _add_reserve_vars!(model, g)
     _add_startup_shutdown_vars!(model, g)
-    _add_status_vars!(model, g)
+    _add_status_vars!(model, g, formulation.status_vars)
 
     # Constraints and objective function
     _add_min_uptime_downtime_eqs!(model, g)
     _add_net_injection_eqs!(model, g)
-    _add_production_limit_eqs!(model, g)
-    _add_production_piecewise_linear_eqs!(model, g, f.pwl_costs)
-    _add_ramp_eqs!(model, g, f.ramping)
-    _add_startup_cost_eqs!(model, g, f.startup_costs)
+    _add_production_limit_eqs!(model, g, formulation.prod_vars)
+    _add_production_piecewise_linear_eqs!(
+        model,
+        g,
+        formulation.prod_vars,
+        formulation.pwl_costs,
+        formulation.status_vars,
+    )
+    _add_ramp_eqs!(
+        model,
+        g,
+        formulation.prod_vars,
+        formulation.ramping,
+        formulation.status_vars,
+    )
+    _add_startup_cost_eqs!(model, g, formulation.startup_costs)
     _add_startup_shutdown_limit_eqs!(model, g)
-    _add_status_eqs!(model, g)
+    _add_status_eqs!(model, g, formulation.status_vars)
     return
 end
 
 _is_initially_on(g::Unit)::Float64 = (g.initial_status > 0 ? 1.0 : 0.0)
 
-function _add_production_vars!(model::JuMP.Model, g::Unit)::Nothing
-    prod_above = _init(model, :prod_above)
-    segprod = _init(model, :segprod)
-    for t in 1:model[:instance].time
-        for k in 1:length(g.cost_segments)
-            segprod[g.name, t, k] = @variable(model, lower_bound = 0)
-        end
-        prod_above[g.name, t] = @variable(model, lower_bound = 0)
-    end
-    return
-end
-
-function _add_production_limit_eqs!(model::JuMP.Model, g::Unit)::Nothing
-    eq_prod_limit = _init(model, :eq_prod_limit)
-    is_on = model[:is_on]
-    prod_above = model[:prod_above]
-    reserve = model[:reserve]
-    gn = g.name
-    for t in 1:model[:instance].time
-        # Objective function terms for production costs
-        # Part of (69) of Kneuven et al. (2020) as C^R_g * u_g(t) term
-        add_to_expression!(model[:obj], is_on[gn, t], g.min_power_cost[t])
-
-        # Production limit
-        # Equation (18) in Kneuven et al. (2020)
-        #   as \bar{p}_g(t) \le \bar{P}_g u_g(t)
-        # amk: this is a weaker version of (20) and (21) in Kneuven et al. (2020)
-        #      but keeping it here in case those are not present
-        power_diff = max(g.max_power[t], 0.0) - max(g.min_power[t], 0.0)
-        if power_diff < 1e-7
-            power_diff = 0.0
-        end
-        eq_prod_limit[gn, t] = @constraint(
-            model,
-            prod_above[gn, t] + reserve[gn, t] <= power_diff * is_on[gn, t]
-        )
-    end
-end
-
 function _add_reserve_vars!(model::JuMP.Model, g::Unit)::Nothing
     reserve = _init(model, :reserve)
+    reserve_shortfall = _init(model, :reserve_shortfall)
     for t in 1:model[:instance].time
         if g.provides_spinning_reserves[t]
             reserve[g.name, t] = @variable(model, lower_bound = 0)
         else
             reserve[g.name, t] = 0.0
         end
+        reserve_shortfall[t] =
+            (model[:instance].shortfall_penalty[t] >= 0) ?
+            @variable(model, lower_bound = 0) : 0.0
     end
     return
 end
@@ -134,56 +111,6 @@ function _add_startup_shutdown_limit_eqs!(model::JuMP.Model, g::Unit)::Nothing
     return
 end
 
-function _add_status_vars!(model::JuMP.Model, g::Unit)::Nothing
-    is_on = _init(model, :is_on)
-    switch_on = _init(model, :switch_on)
-    switch_off = _init(model, :switch_off)
-    for t in 1:model[:instance].time
-        if g.must_run[t]
-            is_on[g.name, t] = 1.0
-            switch_on[g.name, t] = (t == 1 ? 1.0 - _is_initially_on(g) : 0.0)
-            switch_off[g.name, t] = 0.0
-        else
-            is_on[g.name, t] = @variable(model, binary = true)
-            switch_on[g.name, t] = @variable(model, binary = true)
-            switch_off[g.name, t] = @variable(model, binary = true)
-        end
-    end
-    return
-end
-
-function _add_status_eqs!(model::JuMP.Model, g::Unit)::Nothing
-    eq_binary_link = _init(model, :eq_binary_link)
-    eq_switch_on_off = _init(model, :eq_switch_on_off)
-    is_on = model[:is_on]
-    switch_off = model[:switch_off]
-    switch_on = model[:switch_on]
-    for t in 1:model[:instance].time
-        if !g.must_run[t]
-            # Link binary variables
-            if t == 1
-                eq_binary_link[g.name, t] = @constraint(
-                    model,
-                    is_on[g.name, t] - _is_initially_on(g) ==
-                    switch_on[g.name, t] - switch_off[g.name, t]
-                )
-            else
-                eq_binary_link[g.name, t] = @constraint(
-                    model,
-                    is_on[g.name, t] - is_on[g.name, t-1] ==
-                    switch_on[g.name, t] - switch_off[g.name, t]
-                )
-            end
-            # Cannot switch on and off at the same time
-            eq_switch_on_off[g.name, t] = @constraint(
-                model,
-                switch_on[g.name, t] + switch_off[g.name, t] <= 1
-            )
-        end
-    end
-    return
-end
-
 function _add_ramp_eqs!(
     model::JuMP.Model,
     g::Unit,
@@ -194,7 +121,7 @@ function _add_ramp_eqs!(
     eq_ramp_up = _init(model, :eq_ramp_up)
     eq_ramp_down = _init(model, :eq_ramp_down)
     for t in 1:model[:instance].time
-        # Ramp up limit
+        # Ramp up limit 
         if t == 1
             if _is_initially_on(g) == 1
                 eq_ramp_up[g.name, t] = @constraint(
@@ -287,11 +214,5 @@ function _add_net_injection_eqs!(model::JuMP.Model, g::Unit)::Nothing
             model[:is_on][g.name, t],
             g.min_power[t],
         )
-        # Add to reserves expression
-        add_to_expression!(
-            model[:expr_reserve][g.bus.name, t],
-            model[:reserve][g.name, t],
-            1.0,
-        )
     end
 end

src/solution/solution.jl

@@ -51,6 +51,12 @@ function solution(model::JuMP.Model)::OrderedDict
     sol["Switch on"] = timeseries(model[:switch_on], instance.units)
     sol["Switch off"] = timeseries(model[:switch_off], instance.units)
     sol["Reserve (MW)"] = timeseries(model[:reserve], instance.units)
+    sol["Reserve shortfall (MW)"] = OrderedDict(
+        t =>
+            (instance.shortfall_penalty[t] >= 0) ?
+            round(value(model[:reserve_shortfall][t]), digits = 5) : 0.0 for
+        t in 1:instance.time
+    )
     sol["Net injection (MW)"] =
         timeseries(model[:net_injection], instance.buses)
     sol["Load curtail (MW)"] = timeseries(model[:curtail], instance.buses)

src/transform/randomize.jl

@@ -0,0 +1,53 @@
+# UnitCommitment.jl: Optimization Package for Security-Constrained Unit Commitment
+# Copyright (C) 2020-2021, UChicago Argonne, LLC. All rights reserved.
+# Released under the modified BSD license. See COPYING.md for more details.
+
+using Distributions
+
+function randomize_unit_costs!(
+    instance::UnitCommitmentInstance;
+    distribution = Uniform(0.95, 1.05),
+)::Nothing
+    for unit in instance.units
+        α = rand(distribution)
+        unit.min_power_cost *= α
+        for k in unit.cost_segments
+            k.cost *= α
+        end
+        for s in unit.startup_categories
+            s.cost *= α
+        end
+    end
+    return
+end
+
+function randomize_load_distribution!(
+    instance::UnitCommitmentInstance;
+    distribution = Uniform(0.90, 1.10),
+)::Nothing
+    α = rand(distribution, length(instance.buses))
+    for t in 1:instance.time
+        total = sum(bus.load[t] for bus in instance.buses)
+        den = sum(
+            bus.load[t] / total * α[i] for
+            (i, bus) in enumerate(instance.buses)
+        )
+        for (i, bus) in enumerate(instance.buses)
+            bus.load[t] *= α[i] / den
+        end
+    end
+    return
+end
+
+function randomize_peak_load!(
+    instance::UnitCommitmentInstance;
+    distribution = Uniform(0.925, 1.075),
+)::Nothing
+    α = rand(distribution)
+    for bus in instance.buses
+        bus.load *= α
+    end
+    return
+end
+
+export randomize_unit_costs!, randomize_load_distribution!, randomize_peak_load!

src/utils/sysimage.jl

@@ -5,12 +5,20 @@
 using PackageCompiler
 
 using DataStructures
+using Distributions
 using JSON
 using JuMP
 using MathOptInterface
 using SparseArrays
 
-pkg = [:DataStructures, :JSON, :JuMP, :MathOptInterface, :SparseArrays]
+pkg = [
+    :DataStructures,
+    :Distributions,
+    :JSON,
+    :JuMP,
+    :MathOptInterface,
+    :SparseArrays,
+]
 
 @info "Building system image..."
 create_sysimage(

src/validation/validate.jl

@@ -208,12 +208,8 @@ function _validate_units(instance, solution; tol = 0.01)
                         break
                     end
                 end
-                if t == time_down + 1
-                    initial_down = unit.min_downtime
-                    if unit.initial_status < 0
-                        initial_down = -unit.initial_status
-                    end
-                    time_down += initial_down
+                if (t == time_down + 1) && (unit.initial_status < 0)
+                    time_down -= unit.initial_status
                 end
 
                 # Calculate startup costs
@@ -246,14 +242,6 @@ function _validate_units(instance, solution; tol = 0.01)
                         break
                     end
                 end
-                if t == time_up + 1
-                    initial_up = unit.min_uptime
-                    if unit.initial_status > 0
-                        initial_up = unit.initial_status
-                    end
-                    time_up += initial_up
-                end
-
                 if (t == time_up + 1) && (unit.initial_status > 0)
                     time_up += unit.initial_status
                 end
@@ -336,11 +324,16 @@ function _validate_reserve_and_demand(instance, solution, tol = 0.01)
         # Verify spinning reserves
         reserve =
             sum(solution["Reserve (MW)"][g.name][t] for g in instance.units)
-        if reserve < instance.reserves.spinning[t] - tol
+        reserve_shortfall =
+            (instance.shortfall_penalty[t] >= 0) ?
+            solution["Reserve shortfall (MW)"][t] : 0
+
+        if reserve + reserve_shortfall < instance.reserves.spinning[t] - tol
             @error @sprintf(
-                "Insufficient spinning reserves at time %d (%.2f should be %.2f)",
+                "Insufficient spinning reserves at time %d (%.2f + %.2f should be %.2f)",
                 t,
                 reserve,
+                reserve_shortfall,
                 instance.reserves.spinning[t],
             )
             err_count += 1

test/model/formulations_test.jl

@@ -3,6 +3,7 @@
 # Released under the modified BSD license. See COPYING.md for more details.
 
 using UnitCommitment
+using JuMP
 import UnitCommitment:
     ArrCon2000,
     CarArr2006,
@@ -11,17 +12,55 @@ import UnitCommitment:
     Gar1962,
     KnuOstWat2018,
     MorLatRam2013,
-    PanGua2016
+    PanGua2016,
+    XavQiuWanThi2019
 
-function _test(formulation::Formulation)::Nothing
-    instance = UnitCommitment.read_benchmark("matpower/case118/2017-02-01")
-    UnitCommitment.build_model(instance = instance, formulation = formulation)  # should not crash
+if ENABLE_LARGE_TESTS
+    using Gurobi
+end
+
+function _small_test(formulation::Formulation)::Nothing
+    instances = ["matpower/case118/2017-02-01", "test/case14"]
+    for instance in instances
+        # Should not crash
+        UnitCommitment.build_model(
+            instance = UnitCommitment.read_benchmark(instance),
+            formulation = formulation,
+        )
+    end
     return
 end
 
+function _large_test(formulation::Formulation)::Nothing
+    instances = ["pglib-uc/ca/Scenario400_reserves_1"]
+    for instance in instances
+        instance = UnitCommitment.read_benchmark(instance)
+        model = UnitCommitment.build_model(
+            instance = instance,
+            formulation = formulation,
+            optimizer = Gurobi.Optimizer,
+        )
+        UnitCommitment.optimize!(
+            model,
+            XavQiuWanThi2019.Method(two_phase_gap = false, gap_limit = 0.1),
+        )
+        solution = UnitCommitment.solution(model)
+        @test UnitCommitment.validate(instance, solution)
+    end
+    return
+end
+
+function _test(formulation::Formulation)::Nothing
+    _small_test(formulation)
+    if ENABLE_LARGE_TESTS
+        _large_test(formulation)
+    end
+end
+
 @testset "formulations" begin
+    _test(Formulation())
     _test(Formulation(ramping = ArrCon2000.Ramping()))
-    _test(Formulation(ramping = DamKucRajAta2016.Ramping()))
+    # _test(Formulation(ramping = DamKucRajAta2016.Ramping()))
     _test(
         Formulation(
             ramping = MorLatRam2013.Ramping(),

test/runtests.jl

@@ -7,6 +7,8 @@ using UnitCommitment
 
 UnitCommitment._setup_logger()
 
+const ENABLE_LARGE_TESTS = ("UCJL_LARGE_TESTS" in keys(ENV))
+
 @testset "UnitCommitment" begin
     include("usage.jl")
     @testset "import" begin
@@ -26,6 +28,7 @@ UnitCommitment._setup_logger()
     @testset "transform" begin
         include("transform/initcond_test.jl")
         include("transform/slice_test.jl")
+        include("transform/randomize_test.jl")
     end
     @testset "validation" begin
         include("validation/repair_test.jl")

test/transform/randomize_test.jl

@@ -0,0 +1,43 @@
+# UnitCommitment.jl: Optimization Package for Security-Constrained Unit Commitment
+# Copyright (C) 2020, UChicago Argonne, LLC. All rights reserved.
+# Released under the modified BSD license. See COPYING.md for more details.
+
+using UnitCommitment, Cbc, JuMP
+
+_get_instance() = UnitCommitment.read_benchmark("matpower/case118/2017-02-01")
+_total_load(instance) = sum(b.load[1] for b in instance.buses)
+
+@testset "randomize_unit_costs!" begin
+    instance = _get_instance()
+    unit = instance.units[10]
+    prev_min_power_cost = unit.min_power_cost
+    prev_prod_cost = unit.cost_segments[1].cost
+    prev_startup_cost = unit.startup_categories[1].cost
+    randomize_unit_costs!(instance)
+    @test prev_min_power_cost != unit.min_power_cost
+    @test prev_prod_cost != unit.cost_segments[1].cost
+    @test prev_startup_cost != unit.startup_categories[1].cost
+end
+
+@testset "randomize_load_distribution!" begin
+    instance = _get_instance()
+    bus = instance.buses[1]
+    prev_load = instance.buses[1].load[1]
+    prev_total_load = _total_load(instance)
+    randomize_load_distribution!(instance)
+    curr_total_load = _total_load(instance)
+    @test prev_load != instance.buses[1].load[1]
+    @test abs(prev_total_load - curr_total_load) < 1e-3
+end
+
+@testset "randomize_peak_load!" begin
+    instance = _get_instance()
+    bus = instance.buses[1]
+    prev_total_load = _total_load(instance)
+    prev_share = bus.load[1] / prev_total_load
+    randomize_peak_load!(instance)
+    curr_total_load = _total_load(instance)
+    curr_share = bus.load[1] / prev_total_load
+    @test curr_total_load != prev_total_load
+    @test abs(curr_share - prev_share) < 1e-3
+end