Church of Optimization
I once attended a physics colloquium given by an unusual speaker: Lawrence Bacow, the current President of Harvard University. His talk was about the logical and philosophical principles that guide his everyday decision-making in leading his institution. He opened with, "I am trained as an economist, and I treat this as an optimization problem."
Bacow divided the resources available to him into four classes of capital: financial (money), physical (buildings), human, and reputational. At his disposal are several levers that convert the forms of capital into each other. For example, he can spend money to construct a new campus, or use the Harvard clout to lure a top researcher. Of course, the conversion fees are steep and the process often takes a while, so the president needs to think on a long time horizon. As stated by Bacow, the primary purpose of Harvard University is "educational experience of our students", the secondary is "research production". The job of the leader is then to maneuver in the space spanned by the dimensions of capital in order to optimize the purported objectives.
Recently I encountered a tantalizing phrase "neoliberal university" that might ring a bell with the above attitude. However, for the purposes of an exercise, I want to take Bacow here at the face value. He wants to run the creative problem of leading and organizing a major university as an optimization problem: if not a literal one, then at least a metaphorical one. As part of his job, Bacow regularly meets local leaders of the university, such as deans and department chairs, to discuss the problems they face. He asks, "What are the three things you would like to see changed? You cannot ask for more money, as that would relax the optimization problem, and that would not be interesting."
Optimization is a valid selection principle in many systems that are assigned a particular state by humans, or choose one themselves. Scientists took optimization as an axiom and got the principle of least action for classical mechanics, and the method of maximum likelihood for statistical analysis. Optimization can be a physical process: a ball rolls down the hill and stays there, having optimized its potential energy by losing the excess to friction. Optimization can be a biological process: E. coli builds copies of itself with efficiency close to thermodynamic limits, having arrived at its genetic and phenotypical architecture through long evolution. We can conjure up arguments from optimality when we know two things: the criteria for comparing system states, and the mechanism for discarding the states that are not optimal.
Where optimization starts becoming a problem is when it becomes a design principle, or becomes grounds for human decisions. In designing systems, we want, for good reasons, to circumvent the tortuous process of friction or evolution and skip to the place where the system gets good. This skipping can be performed, fast and at scale, with mathematical optimization run on a computer. This is the entire premise of operations research and a lot of design research: using optimization to make not merely better but best possible decisions. One popular design textbook never blinks an eye in identifying the "optimal" solution with "the one we want". Optimization has become the main operational metaphor in the design of industrial world: we want to maximize profit margins, maximize fuel efficiency, maximize client retention, and at times even optimize our lives for maximal happiness. All of those problems can be solved straightforwardly: you write down the objective function, add a few constraints, and throw it into a fancy optimizer program.
One prominent operations researcher just stared blankly as we tried to question this central metaphor, as if we were saying something blasphemous. But a few buildings away we found a very different and more sympathetic ear. The naval engineering researchers were getting jaded with optimization. They realized that hard part of ship design is not to optimize the solution for a given design objective, but to formulate that design objective. When ship design fails, the product is not just somewhat suboptimal; instead, it is unusable, extremely expensive, and nobody knows how that happened. The optimization could have picked a better solution, if only we knew the right objective function, if only we already solved the hard, qualitative part of the problem. Trying to design complex systems via optimization is an endless exercise in shifting goal posts.
The ideology of optimization is one of exclusion. It prescribes you to stop considering solutions that are not optimal under the selection pressure that you currently believe in. If the selection pressure changes, the solutions you just found might be suddenly invalidated - but you don't have other ones! In this way, optimization is inherently contrary to robustness and flexibility. Of course, if the objective suddenly changes, it is most tempting to pick up optimization again, making it chase its own tail.
But the exclusion of "suboptimal" solutions has real consequences, especially when the design choices have to do with humans. Earlier this week, a study got under a lot of heat for a claim of predicting scientific impact of papers. The last sentence of the abstract unironically promises to "optimize" the funding portfolios for impact, or, to put it differently, prevent "non-optimal" science from existing by cutting the funding. In context of science of science, this attitude has been thoroughly roasted four years earlier. However, more broadly, optimization keeps standing as a hegemonic ideology in a lot of design - so pervasive that it is near impossible to imagine anything outside of it.
My own research on design, in the scope of a few toy problems, doesn't shy away from quantitative design objectives, yet aims to navigate the design space in a different, less exclusionary manner. While it is not up to me (or another lone researcher) to build up an alternative framework for quantitative design, I can at least raise the question and offer some, if puny, alternative. By now, when I see someone approach a design problem as an optimization problem, I picture a dude, yelling at me, "Why would I want a system that is anything less than optimal?", spit flying in my face. But my dude, what do you want to optimize for?