19.3. Why is it impossible to infer the emergent properties of a complex system from the properties of the system components?
An emergent property is any property that a system has which its components do not. In a component-based software, the core intended property is itself emergent. But other properties, such as unintended functionalities or very unintended failure states, can also be deemed emergent if they are a result of the combination of components. Because of how broad this definition is, therefore, there are many kinds of emergent properties, most of which share only one trait; unpredictability.
The issue with the emergent properties of a system is that they are often fundamentally different than the properties of any of that system’s components. The textbook uses a perfect example; the properties of a bike (transportation) are fundamentally different than those of its parts (gears, chains, wheels). An emergent property can be in many categories, and affected by many factors. For instance, there is no bug-free hardware, software, or user. As such, when looking at complex systems, we are frequently not looking at a single instance of error in one location, but a chain; perhaps this error state in component 1 causes an error state in component 2? Perhaps it doesn’t, but instead reaches an unusual state that was not supposed to be accessible. Thus, unexpected emergent states occur, not all of which result in complete software failure.
The first limitation of inference, therefore, is that any individual component only supplies a tool to be utilized by the system; one which can be used in any number of unpredictable ways. Take pandas, a data-science component which allows for complex table-based operations and calculations. Alone, all it really does is allow the software to organize and generate data; but how that data is received, stored, interpreted; these pandas doesn’t know or care about. This could be part of a software which analyses census data to make voting predictions, or part of a security software which does table-based calculations to compare data extrapolated from a photograph of an individual against that of the ‘key’ user. All of these uses are emergent; the functionality of the part enables the functionality of the whole, but is otherwise almost entirely unrelated. Because of this, simply knowing that a software utilizes Pandas gives very little actual insight into what the software actually does; its emergent properties are impossible to guess.
Another implication of this is a topic which the sci-fi community loves; the emergence of true A.I.; the singularity. What happens when you combine a learning machine with time, self-evolution, and unlimited computational resources? Presumably, something which is more than the sum of its parts; something that could not have been deliberately designed, but which might create itself if placed in the correct environment. Here, the potential emergent property is intelligence… but when/where/if/how it happens is nonetheless uncertain. In a less lofty example, a database is not intelligent, nor are a series of if-statements; but combine them in the right way, and they could pass the Turing test. The emergent property now is the illusion of intelligence… or is it the real thing? Is our emergent property more than we intended, or less?
Another inference issue is that large component combinations can lead to another category of unintended emergent properties; unusual and unpredictable bugs. A system comprised of other complex systems has an exponentially greater complexity than that of the individuals; this is basic combinatorics. In software, many ‘components’ are themselves softwares. The possible combinations of subsystem ‘A’ states alongside subsystem ‘B’ states all the way through the possible states of subsystem ‘N’; these combinations become computationally intensive to even estimate, let alone try to make a single software explicitly account for every one of them. A component software will instead seek to limit the states to well-understood ones that serve the purpose of the system; and otherwise, mostly hope that things don’t go in an unpredictable direction. The combination of failure propagation and the potentiality for non-deterministic results from a black-box component means that any failure could have any outcome; but likely further failures. These failures can be mitigated, but not entirely prevented. Such a system could accurately be described as “putting bees in a box and waiting for honey”, as you can’t control any individual element, but if you set up the environment correctly you will still likely get the intended result.
This uncertainty is further exacerbated by the black-box mentality. On the one hand, trying to understand every component is a complete waste of time and energy. On the other, not knowing how a component works (and doesn’t) might lead to finding a flaw at the worst possible time; possibly after deployment, when one of the aforementioned unexpected state combinations occur. At the same time, as softwares grow in complexity, so too do components; any individual component utilized by your software might itself be made up of many more other components which you know nothing about. The potential depth of this recursion has no limit. This, combined with 7+-2, means that trying to understand every component, state, combination, and sub-component is a fool’s errand. Ultimately, the best you can do is test the things that you intend to use so that you have a hope of understanding some of the most predictable unusual states, and hope that the ones you have missed don’t occur; or at least don’t occur too often.