A Look at Pew's (2013) "The World's Muslims" Survey Report and Data Set

On April 30, 2013, the Pew Research Center for Religion & Public Life published an extensive report of findings from surveys that explored the opinions of Muslims living in various regions of the world. Report materials published April 30, 2013 with the main report include the Complete Report, Topline Questionnaire, and Infographic, whereas the Data Set was published June 4, 2014. The Data Set includes data collected in 2011-2012 and some supporting files, including a Codebook and detailed questionnaire. The April 2013 report compiles results from surveys conducted in 2008-2009 in sub-Saharan Africa and in 2011-2012 in various regions. A report of the 2008-2009 sub-Saharan African results (published in 2010) can be found here and the Data Set for it is here. An earlier report of some of the 2011-2012 data can be found here, though the data for it are in the June 4, 2014 Data Set cited above. I describe how to do a simple analysis of the data below.

The study is vast in terms of the number of countries surveyed. According to Pew:

"The survey involved a total of more than 38,000 face-to-face interviews in 80-plus languages. It covered Muslims in 39 countries, which are divided into six regions in this report – Southern and Eastern Europe (Russia and the Balkans), Central Asia, Southeast Asia, South Asia, the Middle East and North Africa, and sub-Saharan Africa." (p. 15, Complete Report).

Despite this broad coverage, some countries with at least fairly large numbers of Muslims were not included in the study's report:

"This report includes data on every nation with a Muslim population of more than 10 million except Algeria, China, India, Iran, Saudi Arabia, Sudan, Syria and Yemen. Together, the 39 countries and territories included in the survey are home to about two-thirds of all Muslims in the world." (p. 37, Complete Report).

I will add that while the data for Algeria and Iran are not presented in the (April 30, 2013) report, they are included in the (June 4, 2014) data file. Some of the Iranian data were presented in another report on June 11, 2013. It appears to me that the Algerian data may have been excluded from the report due to an administrative error that resulted in an unacceptably low level of national representativeness. In the Survey Methodology section of the Codebook (one of several files in the Data Set folder) there is this entry with the Algerian sample information:

"Nationally representative of 75% of the adult population. Interviews conducted in the Western region were excluded due to an administrative error." (p. 3, Background and Codebook).

In the sub-Saharan African study, results for Muslims in four countries with small Muslim minority populations, namely Botswana, Rwanda, South Africa, and Zambia, were excluded due to inadequate sample sizes (ns < 100; see p. 66 of the full report). Overall though, taking both the 2008-2009 and 2011-2012 data sets into account, the Pew researchers achieved a very broad coverage of the culturally and geographically diverse Muslim populations, despite many obstacles.

 Pew covered a broad range of subject matter in its questionnaires in these studies, though the main focus was on religion. There were over 100 questions, dealing with topics such as sharia, "religion and politics," "morality," "women in society," "relations among Muslims," "interfaith relations," and "religion, science, and popular culture" (pp. 6-7, Complete Report 2013). There were some differences between the standard questionnaire fielded in the sub-Saharan countries in 2008-2009 and the standard one fielded in the other regions in 2011-2012. For example, in the sub-Saharan African survey but not in the later survey, respondents were asked whether they thought AIDS was God's punishment for immoral sexual behavior (Q58a), whether they'd had their daughters circumcised (Q72d), and whether the use of arms and violence against civilians in defense of religion was justified (Q88).  In comparison with that latter question, in the 2011-2012 survey respondents were asked a somewhat similar question but which specifically referred to suicide bombing (Q89) as the method of violence. There was also considerable overlap in content between the two questionnaires. For example, both questionnaires asked about basic articles of faith such as heaven, hell, angels, and belief in one God and his prophet. Both asked about basic aspects of religious practice such as prayer, pilgrimage, fasting, giving zakat, and reading scripture. Both also asked about sharia, including specific harsh traditional aspects such as the death penalty for apostasy, stoning of adulterers, and whipping and cutting off of hands for crimes like theft.

Despite the challenges in conducting surveys in many of the countries of interest, I believe these Pew studies are for the most part of a reasonably good methodological standard for this type of research. They were conducted by reputable research organizations. The questionnaire was administered in face-to-face interviews to adult respondents 18+ years of age in their place of residence. Stratified random sampling was used in almost all countries except for Uganda and Russia, where, nevertheless, other types of probability samples were used and high representativeness attained. In addition, for many of the countries, the Pew researchers applied a weight variable to their analyses to further increase the national representativeness of the results. One cause for concern is that there were small sample sizes for several of the sub-Saharan African countries included in the report that have Muslim minorities, with margins of error in many of those cases equal to or greater than ±6, and up to ±10 (see p. 66, full report, 2010). This limitation should be kept in mind when considering the results for those individual countries. 

In terms of consistency of results, for questions about sharia and particular elements of sharia, the Pew results have been mixed. Some changes in the numbers may be based on real changes in opinion, but other changes seem too large and too abrupt to be based entirely on real changes in opinion. For example, comparing the results in a Pew 2010 Global Attitudes & Trends report (data collected in the spring of 2010) with those of the Pew 2013 Complete Report (most data collected in 2011-2012) on the level of support among Muslims for the death penalty for apostasy, shows relative consistency in countries such as Pakistan (76% vs 75%), Egypt (84% vs 88%), Jordan (86% vs 83%), and Turkey (5% vs 8%), but less consistency in Indonesia (30% vs 16%), Nigeria (51% vs 29%*), and Lebanon (6% vs 17%). *The Nigerian data shown on p. 219 of the (2013) Complete Report are from 2008-2009. These findings highlight the need to compare results from different studies.

Of course, one should not rely excessively on survey results. One should also use a variety of research methods to examine any phenomenon to get a more complete picture. For example, survey research can be supplemented by more detailed ethnographic investigations, first-person experiential accounts, and so on. Study of the relevant social, legal, religious, political, and historical contexts is also important--and indeed the authors of the Pew (2013) Complete Report consider some of these factors.

How to do a Simple Analysis of Pew's Data

The data sets for these Pew studies are available for the general public to download and explore, though Pew cautions that only those with expertise with such data sets should do so. I agree with Pew that those who wish to attempt statistical analyses of the data with the intention of reporting the results should either be experts or should have expert guidance or supervision. (I have academic training in statistics and research methods, so I haven't had any problems analyzing Pew's data). It is possible however to do some simple analyses that do not require statistical expertise. For example, one can readily use the data file to confirm results from the Pew reports that may be disputed or misunderstood among non-experts. As I will discuss in my next blog posting, there has been a widespread misunderstanding in the popular media about Pew's (2013) apostasy data. I will show the reader, below, how to do a simple analysis of the apostasy data. The data can be analyzed using IBM SPSS, or the open source freely-available (but very basic) PSPP, or other major statistical packages that can read SPSS .sav files. If you can use a spreadsheet and know how to download software, you can probably follow my instructions below for using PSPP to do a simple analysis.

1. Download PSPP to your computer if it's appropriate for your system.

2. Download to your computer the Data Set for the Pew 2013 report from here if you agree to the terms and fill out the form.

3. Open The World's Muslims Data Set folder, and click on the 2012 dataset sav (spss format) file. It should open automatically in the PSPP data editor window. You should see a large array of data. If you scroll all the way to the bottom there should be 32604 cases. If you scroll all the way out to the right you should see Q133 at the top of the data column. Don't do anything to alter the data. Go to the File menu and save at least one extra copy of the data file. I simply added a "2" to the end of the file name when I saved the copy that I'm using in this example.

4. Go to the Data menu and click on it. Here's how that menu should appear (current version of PSPP):

5. Click on Weight Cases. A dialogue box will appear. Select the "Base Weight..." variable.

6. Enter the weight variable into the frequency variable slot. Then click OK. The weight variable is now on and will be used automatically in the analyses as long as you leave it on.

7. Next click on the Analyze menu, select Descriptive Statistics, then select Crosstabs and click on it.

8. The Crosstabs dialogue box will appear. From the variables list select Country and add it to the Rows box. Then from the variables list scroll way down and select Q92b, the apostasy question data, and add it to the Columns box. Then click on the Cells tab, opening the dialogue box, and leave only the Row output option with a check mark beside it (see image below). Then click Continue to close the Cells dialogue box. Then click OK, which triggers the analysis.

9. Check the Output window. The Output window icon flashes whenever you've just added new output. Here are the commands for the above-described analysis:

 Note that the Chi-Square results don't concern us here.


10. Here are the apostasy results for this data set:

The next step is to compare these results with the rounded results given in the Pew (2013) Complete Report, page 219. Note that Algeria is not included in the report, probably for the reason I cited above: The national representativeness of the Algerian data here is only 75%. Thailand is included, but keep in mind, as Pew (2013, Complete Report) indicates on p. 156 and elsewhere, these Thai data are only generalizable to Muslims in the five southernmost regions of that country, not to Muslims in Thailand generally. Niger is listed with the sub-Saharan African countries on p. 219, but it's in this data file because it was surveyed during the 2011-2012 period. Comparing the favor and oppose results with those on p. 219, we find they all match up for the included countries. Note that "Don't know" and "Refused" responses are separated in the data file but are combined for presentation purposes in the Complete Report.

A few other notes should be made.

The Total shows the averages of the above data, but these are questionable because they don't take the country population sizes into account.

The number of cases in the analysis is less than 32604 because the apostasy question Q92b was not asked in three countries, namely Iran, Morocco, and Uzbekistan.

The number of cases in the analysis is of weighted cases. The weight variable adjusts the counts slightly, and produces numbers with decimal values instead of whole round numbers. To check if the raw (unweighted) sample sizes for each country match up with those given in the report, it is necessary to remove the weight variable from the analysis.

The same instructions above (2-10) can be replicated for obtaining and checking the apostasy results from the sub-Saharan African data set. [Note that in the sub-Saharan data file the apostasy question is labeled Q95c. Data for Christians and others are also included in the sub-Saharan data file, but only Muslims were asked the apostasy question].

The same instructions above (4-10) can be replicated for simple analyses of results for other such questions, though be careful to note any dependencies between questions in the wording that indicate that only a subset of respondents were asked a particular question (e.g., see the wording of Q11 on p. 173 of the Complete Report).

To do more complex analyses, I recommend using SPSS, Systat, JMP (SAS), Statistica, or other software that is more sophisticated than PSPP. Note: If you use Statistica 12, when applying the weight variable, use the "weighted moments" option to obtain results consistent with those reported by Pew. Selecting the appropriate weighting option may be an issue in other packages I haven't tried.