On my view of descent with modification, the origin of rudimentary organs is simple. We have plenty of cases of rudimentary organs in our domestic productions,—as the stump of a tail in tailless breeds,—the vestige of an ear in earless breeds,—the reappearance of minute dangling horns in hornless breeds of cattle, more especially, according to Youatt, in young animals,—and the state of the whole flower in the cauliflower. We often see rudiments of various parts in monsters. But I doubt whether any of these cases throw light on the origin of rudimentary organs in a state of nature, further than by showing that rudiments can be produced; for I doubt whether species under nature ever undergo abrupt changes. I believe that disuse has been the main agency; that it has led in successive generations to the gradual reduction of various organs, until they have become rudimentary,—as in the case of the eyes of animals inhabiting dark caverns, and of the wings of birds inhabiting oceanic islands, which have seldom been forced to take flight, and have ultimately lost the power of flying. Again, an organ useful under certain conditions, might become injurious under others, as with the wings of beetles living on small and exposed islands; and in this case natural selection would continue slowly to reduce the organ, until it was rendered harmless and rudimentary.
Any change in function, which can be effected by insensibly small steps, is within the power of natural selection; so that an organ rendered, during changed habits of life, useless or injurious for one purpose, might easily be modified and used for another purpose. Or an organ might be retained for one alone of its former functions. An organ, when rendered useless, may well be variable, for its variations cannot be checked by natural selection. – Charles Darwin, On the Origin of Species
There is also a third paper on the subject Relaxed selection in the wild – published in Trends in Ecology and Evolution. This is a really interesting overview of current thinking on the subject.
As the above quote shows, Darwin was quite familiar with vestigial – or as he called them rudimentary – organs. He mentions them in On the Origin of Species, The Descent of Man, and The Variation of Animals and Plants Under Domestication. The later two of which are elaborations of what Darwin said in The Origin (with, however, a wider range of examples). Scientists coming after Darwin followed several main trends in researching the issue further. First, more examples of vestigial organs were identified. Second, the idea that vestigial organs were more variable than fully functioning organs was followed up on. A good example of this is a 1997 article by Tague on the thumb in Colobus guereza and Ateles geoffroyi. In that paper Tague argued that because these two species were only distantly related the reduction of the thumb is a result of convergent evolution therefore if the thumb is highly variable in the two species the hypothesis that vestigial organs are accompanied by high variability is supported (this is, in fact, what Tague found).
A third strand is the concern of the Lahti et al paper, mainly the effects of reduced selection on an organ or trait. Lahti et al divide the issue up into three broad categories. First, they look at the direct fitness costs and benefits. Two types of cost are identified. First, constitutive costs are costs that are automatically incurred when a cost is expressed or maintained. Second, contingent costs are costs that arise because of a particular feature of the environment where the traits are expressed. Phrased more technically costs occur whenever a fitness curve is not maximized. If the effect on the curve is the same in all environments it is constitutive, otherwise it is contingent. Constitutive costs will actually be the subject of a future post, so I will press on. The second category of Lahti et al has to do with the effect of indirect costs and benefits on correlated traits. Two factors govern whether a cost or a benefit is incurred with correlated traits. These are the polarity of the correlation and the fitness function of the second trait. There are a number of possibilities, of which I will mention one. If there is a negative correlation a situation of antagonistic pleiotropy arises. In Astyanax cave fish eye degeneration is caused by increased sonic hedgehog signaling. Enhanced signaling also leads to increased numbers of taste buds so if selection favors more taste buds this leads to increased eye degeneration. The third category is that of neutral factors – mainly the effect of the accumulation of neutral mutations and founder efffect. Lahti et al then look at how all this effects trait persistence, vestigialization, or complete loss. They also point out the implications for health (evolution of various types of resistance in disease carrying vectors) and conservation biology. here is part of their concluding remarks:
The major issues in the study of relaxed selection are whether and how quickly a trait will decay when a source of selection is relaxed, and how the answers to these questions relate to the particular traits and mechanisms involved. Future research will be helpful in addressing these questions if it aims to identify and explain phenomena that are still poorly known, such as trait costs and correlations, cases of trait vestigialization or persistence, and neutral evolutionary mechanisms. In addition, replicate events are especially valuable; comparing genetic, developmental and phenotypic changes in different populations or species is probably the best way to assess the generality of existing hypotheses and observations. Finally, theoretical work is needed to integrate the various factors influencing trait evolution following the relaxation of a source of selection, and to flesh out and predict relative probabilities for the pathways and mechanisms outlined here. For instance, existing population genetic theory could be developed into models that integrate the influences on traits and produce predictions of waiting time to trait loss versus the likelihood of new functions arising. Adaptive landscape models could also be used to represent the cost–benefit structures of traits and how these structures change when a source of selection is removed.
Lahti et al (2009) Relaxed Selection In The Wild. Trends in Ecology and Evolution. Volume 24, Issue 9, 487-496, 04 June 2009
Tague (1997) Variability of a Vestigial Structure: First Metacarpal in Colobus guereza and Ateles geoffroyi. Evolution, Vol. 51, No. 2. (Apr., 1997), pp. 595-605.