Theory vs. observation

I wrote the following for another context, but I think it might be of interest here.

What it's all about is a debate between people with two different views of how the scientific process operates. One group claims that science is based, first, on careful observation of the world, followed by construction of a theory to account for the data. The other group claims that theories and hypotheses are constructed first, followed by collection of data to provide evidence or refutation for the theory or hypothesis.

My opinion's different from that of either group...

Science relies on both observation and theory. Neither alone is sufficient, but the mixture that any particular science or scientist uses can vary a lot from case to case. Kind of like blind men describing the elephant.

It's an iterative process. Scientists use theory to guide observation, and observation to guide theory. In working on any particular problem, one can enter at different phases of the process. Sometimes one starts with puzzling observations in need of a better theory. (Quote: "The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!' but 'That's funny...'" -- Isaac Asimov) And sometimes one starts with theoretical ideas in need of observational support.

Which comes first, theory or observation? That's a less important question than may be apparent. As noted, any particular individual enters the process at a specific point, which may more heavily involve theory or observation. In either case, it's always (nowadays) true that every investigator is standing "on the shoulders of giants". (A large topic in itself. The metaphor, supposedly, is due to Newton. Famous book on the subject by Robert K. Merton. Basic idea: there are antecedents to everything, including the metaphor.)

But which is the absolute first? Sure, it has to be observation, but only in a somewhat trivial sense, in that all "knowledge" ultimately comes in through the physical senses. Or you could say that it's "only a theory" that your observations have a direct relationship to reality. Now we're in the territory of epistemology, which is generally not the concern of working scientists.

However, when one is thinking about the philosophy of science, one has to take into account the idea that theory determines what can be observed, and in fact what the "meaning" of observations can be. This leads into the realm of Thomas Kuhn and "paradigms" that control what is observed and how it is interpreted. This can be, and has been, taken to the extreme relativist position that science is meaningful only in terms of somewhat arbitrary cultural constructs. Almost all working scientists, of course, think that's going way too far.

Nevertheless, there are plenty of cases where theory has run far ahead of observation. Example just in physics include quantum mechanics, the big bang theory, cosmic inflation, and black holes. Indeed, the gold standard of theory is to make correct predictions of observations that have NOT already been made. A theory that merely accommodates existing observations is suspect of being fudged to fit the facts. Yet that's the right way to go in some cases, where the theory has "free parameters", like the Standard Model of particle physics. (Physicists still want to find a theory that predicts the parameters, and that goal remains quite elusive.) Climate models are the same way. They are adjusted to fit what has been observed in the past, with the hope that forward predictions will also be correct.

And that brings us back to relativity, in the Einsteinian, not cultural, sense (which have very little to do with each other).

The foundation of special relativity is Einstein's rather unorthodox (at the time) idea that the speed of light is the same in all reference frames. If one takes that to be axiomatic, then some quite surprising consequences inevitably follow, such as the equivalence of mass and energy (E=mc²). Nobody was expecting that, or had any observations to even suggest it. Two of Einstein's (five) amazing papers of 1905 resulted from following the axiom to its logical conclusion.

Now, one might think that the Michelson-Morley experiment of 1887 gave the observational basis for Einstein's special theory. But the evidence for this is very unclear. Einstein himself was quite vague about the issue. Pais' biography devotes more than 10 pages to the topic. One thing is clear: Einstein didn't cite the experiment in his 1905 paper, even though it would have bolstered his case. But at various times he acknowledged having been aware of it in 1905. In any event, the experiment doesn't seem to have been anything like the key motivation for special relativity.

General relativity (1916) is an even more interesting case. One of the foundations of GR was special relativity, of course. Another key insight was Einstein's "equivalence principle", which posited that the behavior of a moving object in a gravitational field was the same as the observed behavior of the object in a reference frame that is accelerating with respect to the object.

Again, Einstein took theoretical principles as axioms. He worked for about 10 years to figure out what the consequences had to be. While some observation obviously supported his principles, there was no other observational input after making them axioms. Interestingly, Einstein was not a strong mathematician, which may be why it took him 10 years after 1905 to come up with GR. He had to rely on a friend, Marcel Grossman, who was much better at math. (Of course, what they needed was very cutting edge math at the time.) Einstein also obtained the help of other eminent mathematicians, like Tullio Levi-Civita.

Out of this collaboration emerged the theoretical idea that gravity should not be regarded as a traditional Newtonian force, but instead as a phenomenon due to curvature of space itself. There was nothing particularly observational about this idea. It was simply a beautiful theoretical idea. Indeed, people still have a tough time conceptualizing what it means for space to be curved. Just as people have a hard time conceptualizing the 4 dimensions of spacetime. These kinds of ideas simply do NOT come out of everyday observation.

The story gets even better. Einstein and his collaborators decided that the right equation to describe gravity should have certain very technical, theoretical properties. The equation had to have a "covariant tensor" form. It should describe the geometry of space in terms of a mathematical construct called a "metric". And in the boundary case where no gravitational mass is present, the metric should be, specifically, the "Lorentz metric" used for spacetime in special relativity. From these theoretical considerations, rather than from any specific observations, the collaborators came up with a tensor equation, which is the essential part of GR.

From that equation it was possible to predict that light has to bend in the presence of (large) masses. Nobody had ever observed that, or even suspected it. Not only was the fact of bending correct, but the equation even correctly predicted the amount of bending. This is why Eddington's measurement in 1919 of the bending of light during a solar eclipse caused quite a sensation, including headlines in the NYT. It's part of the reason Einstein acquired his "genius" reputation. (Few ordinary people knew anything about the 1905 papers.)

And the story goes on. Einstein was, in fact, misled by observations to modify his GR equation. He inserted into it what he called a "cosmological constant", so that the equation would predict what observations at the time (around 1920) seemed to indicate - namely that the universe was not collapsing under the force of gravity, but appeared to be static. At times, it is actually better to rely on theory than observation.

Subsequent observations by Hubble (later 1920s) indicated that the universe was in fact expanding. (Even those observations turned out to be quite inaccurate, though qualitatively correct.) So Einstein tossed out the cosmological constant in disgust. That was (apparently) a mistake, as in 1997 new observations indicated that the universe was not only expanding, but actually doing so at an accelerated rate. The cosmological constant - if chosen correctly - in fact predicts that.

Now, the actual value of the constant does depend on observations. It has to have the value that gives the correct amount of observed acceleration. All attempts to use theory to compute this value a priori have been miserable failures... so far.

And that view of the cosmological constant depends on other theoretical assumptions (such as the near perfect flatness of spacetime due to inflation) which have conceptual appeal, but (at least until fairly recently) little independent observational support. Indeed, much of modern cosmology itself depends largely on theoretical assumptions (isotropy and homogeneity) that observationally are only approximations, and could be substantially wrong.

Bottom line: theory and observation in the scientific process cannot be separated. It's kind of like trying to imagine one hand clapping.

Tags: relativity, scientific method