Patuxent Wildlife Research Center
NAAMP III Archive - statistical issues
Home | Archive by Alphabetical Order | Archive by Category
Paul H. Geissler
Biological Resources Division
Office of Associate Chief Biologist
12100 Beech Forest Road
Laurel, Maryland 20708-4038
301-497-5780, FAX 301-497-5784
The need for randomly selected routes and other sampling issues have been debated by the Amphibian electronic discussion group. Many excellent comments have been made, pro and con, but we have not reached a consensus yet. This paper brings those comments together and attempts a synthesis. I hope that the resulting discussion will bring us closer to a consensus.
I will quote selections from the discussions to illustrate the points that have been made. Click on the links to see the text of the comments. The archive of the amphibian discussions can be downloaded from ftp://ftp.im.nbs.gov/pub/naamp/archive396.txt. I have removed much of the header information and saved cleaned versions as archamp.txt and archfcall.txt. Please let me know if you think other comments should be included in this discussion or if you think the selections that I quoted do not adequately capture the discussion.
Need for random routes
In Appendix 1 to last year's protocols and strategies, Paul Geissler said: "The selection of sites for surveying amphibians is very important. For example, Mossman et al. (1995) reported that 10 of 11 Wisconsin amphibian species had negative trends (3 "significant") on the surveyed sites. However, that result does *NOT* imply a decline in the Wisconsin populations unless it can be demonstrated that the sites are representative of amphibian habitat in Wisconsin. Observers subjectively selected the wetlands to be sampled, and they are likely to select good habitat where amphibians are present. With normal variation, good sites are expected to decrease and fair sites are expected to increase. Thus the Wisconsin results are expected because of the method of site selection and should not be used as evidence for declines in amphibian populations."
He quoted several statistics texts to demonstrate the importance of a random sample and cited the example of the Literary Digest public opinion poll about repealing prohibition. Although 4,806,464 questionnaires were returned, Willcox (1931) concluded that "In my opinion this [wage-earner] bias of the Literary Digest samples in favor of the drys outweighs the other biases in favor of the wets. . . . But if others think differently I can see no way to prove that my opinion is better than theirs." The take-home message is that even with enormous samples, valid conclusions cannot be made unless probability sampling is used. It may be hard to relate public opinion polls to amphibian surveys, although the statistical sampling principles are the same. To my knowledge, no one has made an extensive study of the biases associated with allowing observers to select their own routes, but I am confident that the results would be similar to Willcox's conclusions. When the Literary Digest predicted that Landon would defeat Roosevelt in the 1936 presidential election, their error was soon obvious. However, we do not have a similar check on our estimates of amphibian trends.
Because nonrandom public opinion polls gave false election predictions, the importance of random sampling was established in the 1930s. The problems with judgement samples were not as obvious in wildlife surveys. However, the problems with the nonrandom "management" mourning dove call-count routes became apparent in the 1950s. Foote et al. (1958) concluded that "while the management-route data reflect calling-population changes on those routes from year to year, these data cannot be shown to reflect actual population change from year to year within the 7-state sampling area because of the relatively large bias and because the year-to-year harmony between original management-route data and those from a random sampling is unknown." Because of the problems with allowing observers to select their own routes, those routes were all replaced by random routes.
Christine Bishop expressed the view that "So our methods are not as rigorous statistically in terms of choosing the locations of the survey routes as has been suggested is necessary. But I think they are providing very useful information which is standardized over time, a lot better than anything in the past. The value of the information might be even better if the routes were more random and more stratified. On the other hand, I have discussed this with several statisticians and all but one says that our approach is statistically valid and does not need to change."
Brian Smith added that "I think an important consideration in the design of volunteer monitoring programs is clearly the volunteers themselves. I agree that monitoring programs should be statistically defensible, but it is necessary to keep the interests of the volunteers in mind, also. Some people are not going to want to go to certain areas, or at certain times, or may not be interested in certain types of activities, but may want to be involved anyway. If they can be drawn into a program that uses a variety of approaches, such as that described for Ontario, then so much the better. The program which has been developed in Ontario seems like a nice middle ground to me, and if it is statistically defensible as well, then we should be getting some good information."
Sam Droege responded that "It all depends on your goal. If your goal is to track trends you need to use random surveys, if you have other goals (e.g., educational, phenology) then maybe you don't. However, if you want to make statistically defensible statements about trends in amphibian populations in a region, then all statisticians will agree that you cannot use observer chosen sampling programs without biasing the results. This was hammered out in the 20's and again in the game biology field in the 50's, at great loss of information. The earlier years of observer chosen wildlife surveys for waterfowl and upland game birds have all had to be dropped from the long-term data sets. They could not be corrected for and effectively those groups lost many years of information on population trends and had to start completely over. Please, please, please, don't lets make the mistakes of the past. That ancient but relevant literature is reviewed in Appendix 1 of the AMP document ()."
Lisa Gelvin-Innvaer commented that "The upshot is that decisions (conservation, policy, etc.) are being made with our data and/or it may be legally challenged. It had better stand up." Chris Phillips added that "The possibility that volunteers won't be completely satisfied with some random routes should not prevent us from doing this the right way. That's a poor excuse for adopting a sampling scheme that will not provide reliable data. A previous posting suggested that the need for random samples could be considered opinion. I think Appendix 1 points out that this is not so. If the question is phrased correctly, I don't think you could find many statisticians who would agree that judgment samples are valid in this endeavor." Jim Harding responded "however, those people who are trying organize frog calling surveys have a dilemma-- if they insist entirely on assigned routes, then the numbers of available people to do the survey will be far fewer, the number of sites covered will also fewer, and the amount of data will be much less. We should try to set up surveys to maximize statistical relevance wherever possible, but we must also be careful not to discourage enthusiastic volunteers willing to make good observations over time."
After some comments from Natalie Helferty and Jim Harding, Charles Francis responded "your arguments that statistically uncertain data are better than no data are certainly valid. However, statistically valid sampling data would be even better. Even with a volunteer based scheme, I am sure that sampling designs can be selected that will meet the requirements of being statistically valid, without losing most volunteers. The major requirement is to select an appropriate sampling scheme. It is not necessary that all parts of a state or province be covered evenly, nor that all marshes have equal probabilities of coverage. However, it is important that the sampling scheme be known. . . . If we have a chance to devise a good sampling scheme for amphibians, we should do so. I think the problems arise not with getting volunteers to carry out sound surveys (and we are not asking them to survey routes with no frogs--we all agree that is useless), but rather working with statisticians to devise a sampling scheme that meets the needs of both volunteer surveyors and biologists. Such schemes most certainly exist, but further work may be required to find the best ways to implement them."
Selection of routes
Paul Geissler explained the selection a random sample and described the procedures used for Mourning Dove Call-Count and Breeding Bird Survey routes in Appendix 1 of last year's protocols and strategies. The protocols for calling-amphibian surveys describe the suggested sampling frame and route selection procedures. Charles Francis, James Stuart, Paul Geissler, and Charles Francis discussed alternative sampling procedures.
On a related issue, Jim Andrews suggested that "the route needed to be surveyed at least three times during the calling period of each species" and "that anything less than this . . . will allow you to see changes along a given route." However Sam Droege and Paul Geissler pointed out that the best way to detect changes in a State, Province, or other large area, is to survey a route only once during the calling period of each species and to add more routes.
Mike Duran posed a question about the best way to achieve the statistical randomization of calling-frog surveys to some of the sci.stat.* newsgroups. The resulting comments were useful additions to our discussions. I was especially interested in Alan Zaslavsky's suggestion for using nonrandom route data. He noted that "there is no explicit random sampling plan, but you can proceed with this type of analysis if you can describe what you think were the appropriate probabilities that your volunteers would show up in a particular area. I see no alternative to estimating these probabilities by using a model. . . . In order to be believable, this model has to be fairly rich, i.e., it should include 'all' variables that are likely to affect the behavior of the volunteers." I think his suggestion would work, but I think the effort required to estimate the probabilities and justify the heroic assumptions would far exceed the effort required to use random routes. His approach will be extremely useful for salvage operations where only nonrandom data are available for an endangered species. Remember Willcox's unsuccessful efforts to salvage the Literary Digest poll about repealing prohibition with 4,806,464 respondents.
As a starting point for the discussions, I will list the points and my understanding of the agreement. Please respond with your suggestions and opinions.
U.S. Department of the Interior
U.S. Geological Survey
Patuxent Wildlife Research Center
Laurel, MD, USA 20708-4038
Contact: Sam Droege, email: Sam_Droege@usgs.gov
Last Modified: June 2002