Title: Mechanizing Optimization and Statistics
Abstract:

Scientific and engineering investigations are formalized most often in the language of numerical mathematics. The tools supporting this are numerous but disparate, leading to sub-optimal use of existing mathematical theory. We present a unifying framework by taking a programming languages based approach to this problem. Our richly typed language allows naturally declaring optimization and statistics problems, and a library of transformations allows users to interactively compile input problems to solvable forms. We implement our system as a domain specific language embedded in OCaml. Here, we focus on three features: disjunctive constraints, measure types and random variables, and indexing.

By disjunctive constraints, we mean disjunctions over propositions on reals, e.g. \(x \leq w \vee x \geq w + 4.0\). The usual solution strategy involves converting these into mixed-integer linear programming (MILP) constraints using the big-M, convex-hull, or other methods. Automation is clearly needed because these are algebraically tedious and manual application limits them to experts. We provide the first robust implementations and compare our results with that of ILOG CPLEX.

Statistics is increasingly important due to the increasing amount of data generated in the sciences. We introduce language features that enable declarative expression of statistical models and estimation problems. A type ‘prob T’ characterizes probability measures over type T, a special let binding introduces random variables, and some standard measures (e.g. Normal, Gaussian) can be used to construct more complex ones. We demonstrate with an example how our software facilitates exploring the large space of algorithms for solving statistical problems.

Finally, matrices are accepted canonical forms in mathematics, but practitioners employ a more flexible indexing notation: e.g. \(\forall i \in \{A,B,C\} \quad x_i \leq w_i\). Especially in optimization, this need is so critical that virtually every tool supports it. However, indexing has been treated as a mere syntactic convenience and is eliminated at parse time. We present a dependently typed theory that enables far richer index sets to be expressed. Importantly, our theory brings indexing into the formal realm, providing an O(n) to O(1) reduction in memory requirements and the potential for a corresponding computational improvement.

Download slides

When solving machine learning problems, there is currently little automated support for easily experimenting with alternative statistical models or solution strategies. This is because this activity often requires expertise from several different fields (e.g., statistics, optimization, linear algebra), and the level of formalism required for automation is much higher than for a human solving problems on paper. We present a system toward addressing these issues, which we achieve by (1) formalizing a type theory for probability and optimization, and (2) providing an interactive rewrite system for applying problem reformulation theorems. Automating solution strategies this way enables not only manual experimentation but also higher-level, automated activities, such as autotuning.

Download from publisher
Presentation slides

Citation
Sooraj Bhat, Ashish Agarwal, Alexander Gray, Richard Vuduc (2010). Toward Interactive Statistical Modeling, In Procedia Computer Science, International Conference on Computational Science ICCS 2010, 1(1): 1892-1838.

Mathematical programs (MPs) are a class of constrained optimization problems that include linear, mixed-integer, and disjunctive programs. Strategies for solving MPs rely heavily on various transformations between these subclasses, but most are not automated because MP theory does not presently treat programs as syntactic objects. In this work, we present the first syntactic definition of MP and of some widely used MP transformations, most notably the big-M and convex hull methods for converting disjunctive constraints. We use an embedded OCaml DSL on problems from chemical process engineering and operations research to compare our automated transformations to existing technology—finding that no one technique is always best—and also to manual reformulations—finding that our mechanizations are comparable to human experts. This work enables higher-level solution strategies that can use these transformations as subroutines.

Download from publisher
Download preprint
Associated code
Presentation slides

Citation
Ashish Agarwal, Sooraj Bhat, Alexander Gray, Ignacio E. Grossmann (2010). Automating Mathematical Program Transformations, in Proceedings of the 12th International Symposium on Practical Aspects of Declarative Languages, PADL 2010, Vol 5937 of Lecture Notes in Computer Science, pp. 134-148.

We first introduce a novel modeling framework, called linear coupled component automata (LCCA), to facilitate the modeling of discrete-continuous dynamical systems with piecewise constant derivatives. Second, we provide a procedure for transforming models in this framework to mixed-integer linear programming (MILP) constraints. Traditionally, such systems have been modeled directly with MILP constraints. We show with an example that our framework significantly simplifies model formulation and allows the complex MILP constraints to be produced systematically.

Preprint version
Published version

Citation
Ashish Agarwal, Ignacio E. Grossmann (2009). Linear coupled component automata for MILP modeling of hybrid systems, Computers & Chemical Engineering 33(1): 162-175.

Often, it is very difficult to pose a model for a system even after the system is conceptually understood. The reason is the mathematical languages we employ have few forms of expression. We define more expressive languages, first for dynamical discrete-continuous systems, and then more rigorously for mathematical programs (MP). Our approach provides theoretical basis for designing MP software.

The first framework we define is called linear coupled component automata (LCCA). It supports finite domain constraints, explicitly handles dynamics, and enforces modular modeling. We show how LCCA models can be mechanically converted into mathematical programming (MP) constraints. Currently, chemical process systems are usually modeled directly with MP constraints. We show with an example that it is much easier to model hybrid systems in our LCCA framework.

We then pursue a more rigorous approach for the MP part of our work, for the purposes of providing a computer implementation of an MP framework. There are two main results: a rich computer language for declaring MPs and automation of certain model transformations. Mathematically, these correspond to defining a set p of MPs and defining a binary relation on p.

The set p contains programs as one would want to write in practice, not just canonical matrix forms. Complex index sets can be defined in intuitive ways, and they are first-class entities in our theory, not mere notational conveniences eliminated at parse time. This has many benefits: it retains knowledge of the problem structure, keeps the program size to a minimum, and speeds up certain operations. Our definition of the semantics elucidates the nature of MP algorithms and explains the information sought from a solution.

The binary relation on programs p can be defined because a logical formulation allows treating constraints and programs as mathematical objects. Principally, our definition includes: a procedure for putting Boolean expressions into conjunctive normal form, and a procedure for converting disjunctive constraints into mixed-integer inequalities. Neither has been defined previously for a language as expressive as ours, and the latter has not been defined as a formal mapping on constraint spaces. Overall this leads to a procedure for converting general MPs to pure mixed-integer programs (MIPs).

Sets and set relations are defined using the methods of type theory, which espouses a close relation between mathematics and computation. As a result, the set p can be viewed simultaneously as a novel definition of MP and as the software architecture for implementing an MP language. Similarly, the binary relation on p can be directly implemented on a computer. Several examples of our software’s input and output are provided.

Download thesis
Download defense slides

Citation
Ashish Agarwal (2006). Logical Modeling Frameworks for the Optimization of Discrete-Continuous Systems, PhD Dissertation, Carnegie Mellon University.