Principles of Scholarly Review

The Anna Karenina Principle

“All happy families are all alike; every unhappy family is unhappy in its own way.” Thus begins Leo Tolstoy’s classic novel Anna Karenina. By this Tolstoy meant that several key qualities were necessary for a marriage to be happy. The Anna Karenina Principle has been applied to other areas to indicate that multiple specific qualities must be present for an enterprise or process to succeed.

Much as a skilled athlete or proficient musician makes difficult tasks look natural and almost effortless to the casual observer, the processes of a quality scholarly review are may escape the casual reader. As we learn from the Anna Karenina Principle, a review that does not meet the required standards does not function as a piece of scholarship. Merely completing a university education or publishing scholarly articles does not qualify one as a competent reviewer. Even established scholars sometimes underestimate the skill and competency needed to perform a proper review.

Here we will address the multiple factors needed for a successful scholarly review. We will also learn from negative examples to understand where things went wrong and how to avoid their errors.

Obligations of Reviewers and Editors

Scholarly reviewers are charged with assessing the scholarly merit, rigor, and contribution of a work. Competent scholarly reviewers provide a valuable service in offering context and perspective. Much is at stake for the authors of a work being reviewed. A work’s academic reputation, citations, and purchase decisions often hinge heavily on reviewer feedback. 

When a scholarly journal publishes a review, the editors and reviewers represent that the review adheres to the principles and ethics of scholarship. Reviewers have an obligation for accuracy, fairness, and scholarly rigor. The editors have an obligation to ensure that the reviewers are in fact qualified in the subject matter, providing a basis for them to opine. The editors represent that any criticisms in the piece are permissible by the rules of scholarship and that basic steps have been taken to ensure the accuracy of any claims. Lay readers, who may lack the opportunity and expertise to validate all of the facts for themselves, trust that the reviewers and editors have adhered to principles of scholarship as they claim.

Reviewers, as well as authors, can be held accountable to these principles of scholarship; their credibility is also at stake. Even competent reviewers may occasionally make an inadvertent error, which reputable journals will promptly acknowledge and correct. Those who make derogatory claims should be prepared to accept accountability and make corrections or retractions if their claims are in error.

If we can learn much by the studying successful works, the analysis of reviews that fall short of scholarly standards can memorably point out pitfalls.

Reaching the Nations

Our work Reaching the Nations: International LDS Church Growth Almanac (Cumorah Foundation, 2013), has been positively received by leading sociologists of religion, including Armand Mauss, Henri Gooren, and Ronald Lawson, and was featured by Peggy Fletcher Stack in the Salt Lake Tribune.

Reaching the Nations is a two-volume work of nearly a million words containing detailed country profiles on the LDS Church in 219 countries and territories. It contains 4,715 footnotes and extensive original research. For Matt Martinich and myself, RTN was the culmination of over a decade of intensive research. As part of this research, I traveled to 56 nations to conduct field work. Using a systematic methodology, we gathered original data from an extensive network of hundreds of contacts worldwide and from surveys of over 6,000 Latter-day Saints and returned missionaries. 

We worked diligently to collect information for which reliable current data was not available from any publicly available source, ranging from member activity rates and convert retention data to seminary and institute attendance statistics and the locations and years of organization of all congregations in the country. Anyone who has attempted to collect data of this nature appreciates the difficulty of the task. Even individuals who have lived in a country for many years typically have only fragmentary awareness of specific data. Obtaining reliable information on these points requires a network of both administrative and lay contacts, a standardized research methodology, careful fact-checking and verification, and a method for resolving discrepancies. Obtaining these data for even one nation is a challenging task, yet we collected this data and provided broad original research and analysis of The Church of Jesus Christ of Latter-day Saints in 219 countries and territories worldwide.

Reaching the Nations was reviewed in the 2016 issue (#3) of Mormon Studies Review. The reviewers offered many complementary comments about our work and acknowledged that our estimates of member activity rates and other original data were accurate and consistent with their knowledge of specific countries. However, the review also contained numerous inaccuracies, misunderstandings, and material misrepresentations of our work. It appeared to us that basic fact and reference checking had not occurred.

Of the many thousands of articles and reviews in scholarly journals I have read in my career, the MSR review stands out its divergences from normative scholarship in both the magnitude and quantity of its errors. I suspect that if the reviewers were to write to the Encyclopedia Britannica to berate them for printing Maori without a macron and to advise them that their histories are wrong, to the French National Institute of Statistics INSEE to advise them that the data they publish on religion and ethnicity is allegedly illegal or invalid, and to the CIA World Factbook to dismiss their demographic data, their behavior would be viewed negatively. Yet these claims were presented as constituting valid criticisms of RTN in the MSR piece, and represent only a small sampling among numerous inappropriate, bizarre, and demonstrably false criticisms leveled. 

Our response was printed as David G. Stewart, Jr. and Matthew Martinich (2017), "Letter to the Editor," Mormon Studies Review: Vol. 4 : No. 1, Article 24, online here. This post will not deal with the full range of errors in the review, which we addressed in our published response. My central aim is not to embarrass individuals. I will not call out reviewers by name, although initials may be used at times for substantiation.  Confucius noted: “If I am walking with two other men, each of them will serve as my teacher. I will pick out the good points of the one and imitate them and the bad points of the other and correct them in myself.”  

This posting will address process errors. Went wrong with the MSR review, what rules and processes of scholarship were transgressed that ostensibly educated individuals engaged in a litany of errors? Readers who can understand what went wrong with the MSR review and how not to make similar errors will be enhanced in their personal scholarship.

Qualifications for Peer Review

Peer review consists in review by individuals ("peers") qualified in the work's field. The reviewer must possess adequate knowledge, judgment, and perspective to be able offer accurate and meaningful observations and critique. Familiarity with current literature in the field is a prerequisite to understand any novel contribution of the work and how it measures up to extant standards.

An economist who has studied world economies with local knowledge of China, for instance, could potentially be qualified to review a book on the modern Chinese economic system. A Chinese language teacher, a historian who visited China to record oral histories, or a foreign exchange student who spent a semester abroad would not. Such individuals are welcome to their own perspectives. But their assessments would not constitute peer review.

The extensive original research in Reaching the Nations pertains to the missiology and sociology of international church growth, with emphasis on original quantitative and qualitative research and analysis. Numerous sociologists with expertise in the LDS Church and local knowledge of various countries would have been well-qualified to review RTN. MSR chose to turn instead to scholars with no expertise in the field.

The reviewers' background presented in the MSR review and in other publicly available sources offers no evidence of qualifications in the field of our work. A language instructor or religion teacher is not an expert on sociology or missiology. This should have posed an immediate red flag for the editor. The umbrella of diverse secular disciplines sometimes considered under Mormon Studies does not offer license for a "one size fits all approach." Yet that is precisely what occurred in the MSR review. 

Having written on church growth topics for over twenty years, I am closely familiar with the literature. Only one reviewer had previously published on international church growth in the English-language literature. His contribution consists of two published articles now written over twenty years ago (which I had cited ten times in a prior work). These articles were anecdotal-observational, as their titles "Reflections" and "Issues" suggest. The reviewer's background is as an educator and language teacher. His essays offer no indication of any formal research methodology or study design. No qualifications are evident regarding broader social research or missiology. 

Another reviewer (CC) had written a short paper in French addressing general church growth figures and issues of activity and retention for a general audience.  The other reviewers were "dark horses" with no publications we could identify on international church growth. We knew of a couple as graduate students or up-and-coming scholars in other fields, but found nothing that would qualify them as experts on missiology or sociology.  Going down the list, we find one historian (who now teaches religion), a graduate student at the time (now a religion instructor with an interest in history), an instructor in East Asian Languages and Civilizations, and so on.  Beyond a general interest in the Church in specific countries and some affiliation with academics, one finds no basis that would support their selection as reviewers.

I am not claiming that these individuals are not competent scholars within their own fields. However, the field of our work is outside of the reviewers' respective areas of scholarship and expertise. Such forays into unfamiliar fields seldom end well, especially when not approached with humility and caution.

Most MSR designees appear to have had little or no prior experience as scholarly reviewers. The nature and magnitude of errors made by the MSR reviewers demonstrate a lack of understanding of the rules and processes of a scholarly review. The review appears to have been approached as an anything-goes free-for-all, and not as an academic process governed by specific principles and standards.

As the MSR reviewers were not qualified in the field of study, the review was not a peer review, and should not have been eligible for inclusion in a reputable scholarly publication.

Editorial Responsibility

The editors of reputable specialty journals are experts in the journal’s field and typically also carefully review, referee, and offer insights and perspective.  We wondered where the MSR editors were. Many reviewer errors could have been caught with basic fact and reference checking even by a secretary with no knowledge of the subject material. No perspective or moderation was provided of reviewer comments which were not consistent with the extant scholarly literature and not valid under the rules of scholarship. The editors were AWOL.

Upon inquiring regarding numerous factual errors in the review, we were initially advised that the reviewers were the "experts" and so the editors deferred to them. The reviewers were essentially given a blank check with little or no editorial supervision or oversight.

In some areas of Mormon studies, when a journal addresses a topic far afield from the editor’s training and expertise, the editor will defer entirely to the reviewers.  To the editor, with no qualifications in the subject matter at all, a reviewer with sketchy credentials may appear well-qualified.  In the land of the blind, the one-eyed is king.  Reviewers appeared to know just enough to make erudite-sounding criticisms, but not enough to responsibly resolve or investigate their own concerns. The editors also were unqualified and failed to fulfill their obligation to ensure that the steps of the scholarly process had been followed. 

Bias and Conflicts of Interest

Reviewers, like other humans, are subject to bias. The LDS scholarly community is small and scholarship in some fields is only emerging. Religious topics tend to be more disputed with a thinner line between scholarship, apologetics, and polemics than in other disciplines. These may be some of the reasons contributing to my observation that bias and conflict of interest seem to be more frequently encountered in LDS scholarship than in well-established secular disciplines.

A survey of the LDS review literature suggests that reviewers who are known to have close personal or professional ties with the authors being reviewed may be likely to offer overwhelmingly positive comments. In some cases, dubious assumptions and glaring shortfalls of a favored author or viewpoint may not be identified and questioned by the reviewers. Criticisms, if any, are likely to be gently stated. 

The work of prominent individuals in a field is less likely to be challenged due to the perceived risk individuals may face to their own professional reputation. Those who review the work of unfamiliar authors often take a harsher and less careful tone. 

Bias also occurs in favor of authors who share similar opinions, and against those with divergent views. Reviewers perceive viewpoints closer to their own as more meritorious, even when the matter is one of opinion and other viewpoints may be equally or better supported by evidence. 

None of these issues have anything to do with the scholarly merit of the work being reviewed. These issues represent biases or conflicts of interest on the part of the reviewer. The reviewer who acts in this fashion may be blind to his or her biases, and sincerely believe that he or she is acting in a fair and even-handed manner.

The potential for bias requires specific countermeasures (blinding, disclosure of conflicts of interest, and so forth) to minimize. Reputable journals conduct "blinded reviews" in which papers submitted for publication are sent to reviewers who are unaware of the authors identities. This can assist with more balanced reviews of works submitted for publication, although the identity of authors typically cannot be concealed in the review of already-published works. Reviewers may write more carefully and fairly when the identity of the authors is unknown. Blinding is therefore very helpful, and is considered obligatory in publications which strive for fairness and impartiality. However, blinding does not remove issues of potential bias related to the reviewers' personal opinions and viewpoints.

Responsibility for Fairness

Reviewers have an ethical responsibility to provide a fair, impartial and accurate assessment of the work being reviewed. When controversies or differences of opinion exist, reviewers have an obligation to discuss both sides of an issue in a balanced and even-handed fashion, and to disclose where their own perspectives may diverge from the scholarly mainstream. Reviewers are expected to adhere to accepted middle-of-the-road scholarship, rather than using the review as a platform to advocate their own marginal views and agendas.

Several items regarding the MSR reviewers raised concerns. The single reviewer with prior publications on church growth had been involved in a controversy with another scholar that resulted in a letter, a response, a counter-response, and a counter-counter response (see Journal of Mormon History, Vol. 24, No. 1 (Spring 1998), pp. xix-xxxvi, il-lv, online here). The other scholar (not the reviewer) had been accused of plagiarism. He acknowledged those criticisms and offered an apology consistent with the seriousness of the allegations which deeply impacted his scholarly standing and credibility. The reviewer, however, engaged in numerous personal attacks and negative assertions not directly related to the topic of criticism and without apparent evidence.

When I first read the JMH exchange, years before the MSR review, I recall being struck by elements which seemed inappropriate. I would welcome feedback from others who have read the letters. In more than twenty years of academics, I don't recall ever seeing another published exchange involving four letters between two parties. The reviewer's repeated personal attacks beyond the limits of the evidence raised concerns regarding his fairness and adherence to the scholarly process. I deplore the other scholar's conduct in utilizing others' material without attribution. Yet his objection to the reviewer's sweeping personal attacks going beyond the evidence is not without merit. I wondered why the editors failed to referee the exchange.

The reviewer was also a celebrated blogger. Some behaviors that may offer entertainment value in a popular blog, including the use of sarcasm, caricaturing of other viewpoints, and repeated assertions without valid documentation, are inappropriate in a scholarly review.

It seems unsurprising that this individual was our chief critic in the MSR piece. Whereas most reviewers alleged few or no errors, this reviewer presented a laundry list. His behaviors demonstrated close parallels to the JMH letters. We demonstrated in our response that his criticisms were overwhelmingly inaccurate and unsupported. Out of more than ten alleged errors, fact-checking confirmed only one actual error (carried over from an inaccurate heading in a source) and one omission. He repeatedly presented his own erroneous, dubious, or controversial claims as fact. Our work was misquoted, extraordinary claims contrary to the mainstream literature were made with no support beyond his personal authority, defunct links were presented without academic referencing, and novel historical revisionism was set forth. He went on to draw sweeping negative conclusions, dismissing our research and analysis without engaging it. The reviewer's conduct was incongruous with his own recklessness and lack of self-insight. 

There were significant "red flags" which should have disqualified him as a reviewer. We wonder if the editors were unaware of the reviewer's prior history, or were aware but failed to provide editorial oversight. At a minimum, the editor should have fulfilled his obligations as a referee of the scholarly process in enforcing standards of fairness, objectivity, and documentation.

Avoid Bias
Adherents of different faiths, as well as atheists, agnostics, and disaffiliated individuals, are capable of writing fair-minded scholarly reviews. Fairness and impartiality, however, are compromised when an individual adopts a critical agenda and is principally interested in attacking a faith rather than treating it in an even-handed fashion. This is the case for at least two of the MSR reviewers.  

We wonder why the Maxwell Institute selected two ex-Mormon critics who have demonstrated hostility to the growth and activities of the LDS Church as reviewers of mission studies research intended to inform and empower LDS growth.  The MSR reviewer of the Brazil section (MJdO) is a contributor to the Ex-Mormon Foundation's newsletter, who explained why he "resigned [his] membership in protest and shame." 

The New Zealand section reviewer for MSR (GC) is a prominent critic and disaffiliated member of the LDS Church who acknowledged that her "relationship with the Church is troubled."  The tone of her personal writings demonstrates considerable anger and resentment against the LDS Church.  One might characterize her as not merely an ex-Mormon but, to a considerable degree, an anti-Mormon who has repeatedly attacked the LDS Church, its teachings, leadership, and members.

In her same post of 11 March 2019, she wondered why the LDS Church did not thank disaffiliated members like her for their service. She demands for active LDS members to apologize to her and other ex-Mormons for reasons that seem unclear beyond her broadly-painted notions of communal guilt for supposed offenses. She appears prone to exaggeration, accusing LDS members of "shunning" her (in fact, the LDS Church has no institutional practice of shunning, in contrast to the JW faith. Her own inclusion in numerous LDS-oriented scholarly venues for which her qualifications are doubtful suggests the opposite). All the while, she is seemingly oblivious to the hurtful nature of her own sharp attacks on the LDS Church and its members. If her association with other local members was based on the Church, and she not merely repudiates but actively attacks its belief system, it might not come as a great surprise that some active church members might find little basis to maintain fellowship with her. Perhaps offering the same courtesy and respect to the LDS faith that she demands for herself would be a better starting point. 

Her further writings convey a pervasive lack of self-insight and raise numerous questions of "double standards."  GC's critiques run far ahead of her scholarship. She repeatedly indulges in sweeping critical over-generalizations with no support beyond personal anecdote, while omitting positive data that would undermine her narrative and agenda. Her own hostile and hypercritical tone towards Mormonism is incongruous with her incapacity to bear any perceived slight toward her by active Mormons.  Her conduct raises larger questions regarding her identity as a scholar as opposed to merely another disaffected critic seeking to gratify feelings of bitterness and spite.  While she is welcome to her personal opinions, a line must be drawn distinguishing opinion and anecdote from evidence-based scholarship.

This ex-Mormon reviewer (GC) served as the first editor and chapter author of Decolonizing Mormonism, published in 2018 and apparently in progress at the time of the MSR review. Decolonizing Mormonism describes itself as "a critical reflect[ion] on... American Mormon Cultural Imperialism." Her co-editor is the first editor of Mormon Feminism: Essential Writings. 

A survey of these works confirms a strong activist slant. I am sympathetic to some viewpoints in Decolonizing Mormonism. However, some authors, like GC, appear to be more interested in deconstructing Mormonism than decolonizing it, and the work suffers thereby.  DC offers a small number of perspectives which do not well represent my experience with international Mormons. It emphasizes only a few areas of the international Church while failing to represent many others entirely. The work's principal basis, as the publisher's description acknowledges, is personal reflection and anecdote, rather than original research. Some authors came across as more successful or sympathetic than others, yet large portions consisted of more activism than scholarship.  Dubious assertions based on little more than anecdote and personal opinion are repeatedly set forth as fact. Controversial viewpoints are repeatedly presented without balance or counter-points. Population-based sociological data is minimally engaged if at all by most of the chapter authors. Multiple authors made no apparent effort to provide a broad-based picture beyond their few selected areas and perspectives, or to ensure that their views and experiences are representatives of larger populations. Some DC authors were quick to level criticism at those who do not share their background and perspectives, but notably deficient at offering balanced perspectives or recognizing the limitations of their own views.

Although the group of activist writers assembled by these two anti-Mormon feminists is far from the mainstream of international LDS membership, GC's essay represents the anthology's low point.  In contrast to all of Decolonizing Mormonism's other authors, GC's own chapter astonishingly fails to cite a single reference, scholarly or otherwise, in the endnotes (see pp. 275-76). The bibliography contains little relevant to her chapter beyond a citation of an article from the New Era magazine and a reference to one of her own opinion pieces on Patheos.  Her essay might be viewed as an opinion piece, a personal narrative, or a polemic, but it is difficult to determine on what basis an essay which engages no external data at all can be considered scholarship, or how her experiences can be utilized to draw general conclusions. Her repeated breaches of the normative academic processes leave her credentials as a scholar, let alone as a reviewer or editor, very much in doubt. We wonder how such "reviewers" can pontificate to us about scholarly rigor.

This Decolonizing Mormonism author/editor who reviewed RTN appears to have been unable to separate her activist viewpoints from the responsibilities of a scholarly reviewer in MSR. She represented RTN's use of Maori without a macron as a violation of universal standard, and imputed from this our lack of local knowledge and contacts. She attempted to impose a rule that foreign words should "always" be written internationally with local diacritics, and further represented her insights and perspectives as superior to our own.

None of these criticisms are valid. We were merely following the standards of extant literature on the international Church, as well as standard English. It is also troubling that the editors failed to catch and remedy these obvious falsehoods. We could have written place names like Санкт-Петербуг and Київ in their native renditions throughout the text, but that is not the standard, and produces considerable difficulties for search and portability.

Our goal in RTN was for standard, accepted, even-handed scholarship. Had the reviewer (GC) read our work thoughtfully instead of hyperventilating at the absence of a foreign-language diacritical, she surely could have identified many original data points and items of insight in RTN that represented new contributions to the literature. Her assessment appears to have been dominated by activist "hot-button issues" that precluded her from engaging our work in a balanced and even-handed fashion. Her activist issues would have been out of place in the composition of our work. In presenting baseless criticisms of RTN, she failed to recognize and reign in her own substantial biases.

I would appreciate a formal written response from the Maxwell Institute regarding on what basis they believed these reviewers to be qualified and impartial, or whether they were totally unaware of the reviewers' vocal stances against the LDS Church. Whatever one's opinions of these reviewers' works, their writings are remote from the centrism, fairness and impartiality expected of scholarly reviewers.

Understand the Authors' Purpose and Audience

A reviewer should seek to understand the authors' methods, purpose, and audience.  For example, Decolonizing Mormonism is not a work of population-based sociology. Its deficiencies by sociological standards should surprise no one. It includes elements of historiography, personal narrative, and reflection, and deserves to be evaluated as such from the standpoint of scholarly review.  My interest in the larger sociological picture, and being unimpressed by anecdotes, observations, and philosophical reflections, diverges from its authors' methods and purposes.

Imposing the expectations of another field, while neglecting the work's contributions and strengths by the standards of its own field, would be inappropriate for a scholarly review. Yet this is precisely what the MSR reviewers of RTN did repeatedly. As outsiders without expertise in sociology and missiology, they reviewed the work through the lens of their own agendas and interests. They ignored the field of the work and imposed invalid criteria and arbitrary demands.

Are the Criticisms Permissible by Rules of Scholarship?

A scholarly review is not a free-for-all where anything goes. Reputable reviews are governed by specific rules of scholarship. The reviewer's mantle does not offer a license for gratuitous dagger-throwing. To be permissible, a criticism must be valid by the rules of scholarship to be permissible. We will examine several items below.

Comparable Works

The standards of review derive from accepted, mainstream scholarly principles and understanding, as well as the extant literature. The extant literature, both scholarly and lay, serves as a basis for evaluating a work's contribution.

Every criticism involves a comparison. When a reviewer claims that a work is deficient because it does not include such-and-such, he is implying that this is the standard in the field. When reviewer alleges that the authors have made an inaccurate statement, he is asserting that his own claims and statements are accurate. When a reviewer chides an author for an alleged lack of scholarly rigor, he is asserting that his own work conforms to principles of rigorous scholarship.

As we read the MSR reviewers' litany of pie-in-the-sky criticisms with no basis in the comparable literature, and that had no basis in the accepted standards in the field, our repeated question was: compared to what?

An author should not be criticized for adhering to accepted standards, norms and conventions of scholarship and comparable works in the field. Yet the MSR reviewers did so repeatedly, imposing their own arbitrary criteria which do not reflect the accepted norms of extant scholarship.  

Consider Logistics

Reaching the Nations was an epic undertaking, containing detailed data on the Church in 219 nations and territories. We traveled to 56 nations for on-site fieldwork, and completed the largest independent survey to date of international Mormons, with over 6,500 responses. We spent many thousands of hours in original research over more than a decade, recruited hundreds of local contacts, and incurred hundreds of thousands of dollars in expenses. The final work weighs in at just under a million words and contains 4,715 footnotes. No published work on international LDS Church growth, to our knowledge, comes close to RTN in scope or contribution of original data.

Responsible reviewers measure from the context of extant works. Yet rather than appreciating the vast scope of the work, the MSR reviewers levied criticisms based on arbitrary pie-in-the-sky demands disconnected from logistical realities and the standards of extant literature.

One reviewer insisted that we should have referenced and combed through 130 years (1840-1970) of Millennial Star magazines for our entry on the United Kingdom. Such a criticism implies a lack of awareness of RTN as an almanac-style work for 219 countries and territories, rather than having unlimited time and resources to engage in historiography of only one. Despite making a strong recommendation for inclusion of the periodical, he was unable to offer any specific examples of how it would meaningfully enhance RTN.  Our prior review of Millennial Star archives found little we would potentially use, due to its different focus and remote dates. The Star contains a wealth of historical data, but that is a project for a historian, not a world almanac with limited space to devote to each country. 

Some of the reviewers' critiques were contradictory. While our 2012 and 2013 data were criticized as somehow not being adequately "current" in a work with a 2013 publication date (MSR reviewed in 2016), the reviewers simultaneously criticized us for not including certain very old items which offered little potentially relevant data. The Millennial Star was discontinued in 1970, nearly fifty years ago, and offers little pertinent to our work across its many volumes. The reviewers also criticized us for numerous citations to the LDS Church News, which contained the largest number of recent references to church growth. How is the most recent available data from RTN's year of publication, the prior year, and the leading publication of international LDS news "out of date" whereas the content of a 50-130 year old periodical is apparently essential?

Another reviewer (WD) criticized our work as not even representing a "serious approach" for not citing dated and eclectic scholarship. We had surveyed this literature before composing RTN (and have referenced it extensively in other publications), but elected not to engage it found that it offered only sparse and dated data points on a few of the 219 nations and territories covered in our work.

Such problems were pervasive in the RTN review. Reviewers imposed arbitrary pie-in-the-sky demands disconnected from logistical realities and the available literature, while failing to understand or appreciate the work's significant contribution. Most of the reviewers have not even published a short book of their own. Several have published articles, book chapters, or short encyclopedia entries. The reviewers appear to have been utterly unaware of the logistical considerations of publishing a book of this magnitude.

Next, I will address several ethical responsibilities of reviewers, including the responsibility for fairness, freedom from bias, and avoidance of conflicts of interest.

English-language Scholarship

One reviewer (WD) claims that our information regarding the date of Spanish control over the Netherlands, while widely cited in the English literature, is false, and is disproved in the Dutch-language literature. We noted in our response that there are good reasons for the date we cited as widely acknowledged in the English literature. Even if the matter of date were meritorious, authors of English-language scholarship cannot be criticized on the basis of claims made in the non-English literature to which they have no reasonable access.

This critique violates a principle of scholarship: Authors of English-language works are only responsible for knowing the mainstream English-language literature. While the reviewer's access to alternative viewpoints may be of interest to local historians, it does not represent a valid critique in a mainstream English-language scholarly review, and has no place here.

Assess By Existing Standards

Scholarly standards require for reviewed works to be assessed by the standards of extant works. Although a reviewer may express an opinion regarding items he or she feels might be desirable, a reviewer is out of line to convey that a work is deficient because it does not meet made-up requirements not consistent with the extant literature.

One of MSR's anti-Mormon reviewers (GC) conveyed that RTN was deficient and out of the scholarly mainstream because we wrote Maori without a macron (Māori). We are well aware that Māori is written with a macron in New Zealand, but this is not the case in standard English. The Maxwell Institute itself, official Church magazines, The Church News, and the Desert News Church Almanac all use Maori without a macron. None of the dozens of articles Maori in The Church News (formerly LDS Church News) written before our date of writing appear to use a macron; one finds it in a 2017 piece published four years after RTN for which key content was provided in New Zealand. In the Journal of Mormon History, most papers omit the macron whereas a recent piece written by a New Zealander after RTN's publication date contains it. The macron is used primarily in publications originating from New Zealand. 

Reputable works like the Encyclopedia Britannica render Maori without a macron. A macron in Wikipedia appears to reflect New Zealander influence as because it is editable by anyone, but this is not standard English. The use of local words without diacritics is typical in standard English, and aids greatly in search and portability.  We wonder where the editors were, as neither the reviewer nor the editors could have been unaware that the use of Maori without the macron is standard in both popular works and scholarship of the international LDS Church. 

The reviewer invented an arbitrary criterion not consistent with the extant literature and usage. She then misrepresented it as a universal standard and used it to criticize our work. Why was our work singled out for criticisms that are demonstrably "pants on fire" false? While we understand the reviewer's desire to advocate for wider use of Māori with a macron, academic standards require scholarly review to be based in accepted norms. This claim is not admissible and has no place in a scholarly review.

Another reviewer criticized us for using standard English-language reference sources for country data, dismissing them as wrong without providing any examples, substantiation, or alternative data that he believed to be more accurate. Such "hand waving" is also not admissible in published scholarship.

Where controversies or reasonable differences of opinion exist, a scholarly reviewer is required to disclose this and address the topic in a balanced and even-handed fashion. Instead, the reviewers repeatedly represent their own marginal and idiosyncratic viewpoints as fact, while attacking RTN as inaccurate for presenting mainstream understandings.

As we finally arrive at criticisms that, if true, could potentially be admissible by the principles of scholarship, we are confronted with the question of their veracity.

Read the Text

It seems obvious that the work being reviewed needs to be read. Unfortunately, it appears that several of the RTN reviewers merely skimmed, or at best, did not read attentively even their short assigned sections. This is particularly problematic when the reviewer goes on to make derogatory claims that arise from his or her own inattention and erroneous assumptions rather than from any actual defect in the work.

Reviewers may be pressed for time, and may not always be able to comprehensively study a work being reviewed. However, reviewers have an ethical obligation to recognize the limitations of their acquaintance with the work, and to disclose when their claims are based on something other than a full reading. If one claims in a printed scholarly publication that the reviewed work doesn't source certain data, one had better make certain that it actually isn't there, and not that one failed to review the bibliography and source notes. If one claims that a work misstates certain items, it had better be the case that the work is actually wrong, and not that the reviewer has misquoted the work and the authors are vindicated merely by checking the work itself. Such conduct is not merely careless, but unethical.

Understand the Authors' Methodology

Awareness of the authors' methods and approach is important to a proper understanding and appraisal of their work. What was their data? Was original research or inquiry conducted? If so, what was the methodology?

Two MSR reviewers (GC and TR) astonishingly misunderstood the entire nature of our Reaching the Nations. One reviewer (GC) alleged that we had not engaged local contacts or researchers. The other (TR) claimed that Reaching the Nations "should be seen as no more than what it is: an encyclopedic reference." Encyclopedias collect and summarize information available in other public sources. They do not conduct original research, collect new data, or offer novel insights or analysis. 

Her statement thus alleges that RTN only regurgitates publicly known facts while offering no new data or insight to the scholarly corpus. The review goes on to chide us for allegedly failing to recruit local contacts. According to their claims, RTN apparently could have been composed by two guys sitting in a library. In fact, we worked with local contacts extensively.

With mere hand-waving, the reviewers delegitimized and dismissed our entire research methodology, our network of hundreds of local contacts worldwide and thousands of survey responses, and our on-site fieldwork in 56 nations. The many thousands of hours we spent engaged in original research over more than a decade, the numerous local contacts we recruited, and the hundreds of thousands of dollars in expenses we incurred, are apparently figments of our imagination.

The reviewers' claim is astonishing to anyone who has read even brief selections of our work and has any familiarity with the extant literature. The India entry of RTN which the second reviewer appraised (V2 pp. 900-911) contains considerable information which, to my knowledge, is not found in any other publicly available source, including but not limited to the following:

1. LDS member activity rates and weekly church attendance

2. Convert retention data

3. The locations of all branches and extent of national outreach

4. The years of organization of new congregations

5. Seminary and institute attendance statistics

6. Local missionary service rates

7. Information regarding local branch leadership

8. Local church events not elsewhere announced. 

9. Detailed insight and analysis into local challenges and opportunities.

I am aware of no published or publicly accessible sources (other than our own research and citations thereof) available at or before RTN’s 2013 print date we could have obtained the following data we presented in the India chapter (as well as for each other country and territory). If indeed the reviewer is aware of other public sources where this data was available, we could have saved a great deal of time, effort and expense.

Our analysis also contains extensive insights information reflecting a current awareness of the situation from local sources. Even if one knew nothing at all about the extant literature, the India entry makes specific reference to information being obtained directly from local LDS sources. The introduction to RTN also notes that "we have sought to find the most accurate and current information available, including hundreds of member, returned missionary, and mission president reports, and our own research."

We are therefore puzzled as to how at least two MSR reviewers apparently failed to recognize that RTN contained original research at all. Did they actually read their assigned sections of the RTN text? Were they completely unaware of the extant literature on their respective countries that they failed to recognize that numerous data points and analysis were nowhere else publicly available? Did they also fail to conduct any due diligence on their negative claims before putting them in print?

Conduct Due Diligence

One reviewer (CC) challenged the provenance of our ethnic and religious data for France, claiming that collection of such data is illegal. He thus implied that our data are either illicit or fabricated. As we noted in our reply, the regulation he refers to applies only to national censuses and includes exceptions for social surveys and data collection by the French bureau of statistics, which publishes data on this topic. The reviewer's allegation demonstrates his lack of knowledge of basic demography and statistical data collection in his own country, as such data is abundantly cited and utilized by demographers and sociologists. What is even more exceptional is that he sets forth a false and defamatory charge in print without taking any apparent steps to verify his claim. Whatever the case, he appears to lack any curiosity to educate himself about the topic of his criticism. Simply looking up demographic statistics to learn about their source or reviewing the provisions of the law itself would have dispelled his misconception. Unfortunately, the reviewer found it easier to levy false charges against our work than to conduct due diligence. 

It is difficult to determine how an individual without knowledge of basic demographic methods was deemed qualified to serve as a reviewer of a work of social science research. Oddly, this reviewer lists his participation in the review as one of two publications on his CV for 2016. We view it as an affront to credibility rather than a scholarly achievement.

We were alarmed by the large number of criticisms in which the reviewer appeared to be speaking off the cuff, apparently without performing any due diligence. Some misquoted our work, others made extraordinary claims while offering no documentation. Of the few references reviewers provided, some did not support the reviewers' claims; others involved inaccurate or defunct references.

We also wonder how the editors printed such baseless criticisms without any apparent fact or reference checking. Fact and reference checking is always done by reputable academic journals.  Admonitions to scholarly rigor, it seems, were merely polemics for the reviewers to trot out in gratuitous criticism, and not a principle which they viewed as applying to themselves. 

I have no basis to understand how an ostensibly scholarly journal could omit required fact and reference checking. In my own journal publications, including an article I was asked to submit to the predecessor of MSR, I have had to submit extensive referencing, including copies of title pages and the page showing each item cited, to document that references were correct and that they were appropriately cited and utilized. My citations went through with no errors or inaccuracies identified. In contrast, the MSR reviewers engaged in extensive factual errors with no apparent due diligence.

Check the Dates

One of the ex-Mormon reviewers claims that our statistics on Brazil are "outdated and wrong." He further goes on to chide us for failing to include data from the Brazilian census, oblivious to the fact that census results were unavailable until after our publication date, as we documented in our response. His claim that our statistics are simultaneously "outdated and wrong" is a logical contradiction: either they are outdated (in which case they were previously accurate) or wrong (never correct). The double criticism appears to be a polemical way of hitting us twice for statistics outdated. Both of these criticisms are false. As we noted in our MSR response, the reviewer (who wrote in 2016) apparently failed to note RTN's 2013 publication date. RTN's statistics reflected the most recent available data at the time of publication.

Another reviewer (TR) noted that our work was lacking in credibility because it failed to mention the formation of the Hyderabad India stake in June 2012. In fact, the India Hyderabad Stake was announced on Matt’s church growth blog on April 14, 2012, six weeks before its announcement in the LDS Church News. The announcement did not make it in to RTN due to our editorial submission deadlines in mid and early 2012 for the work to be printed in 2013 (we were permitted to make only some statistical updates to the printer’s copy in early 2013). I do not expect the reviewer to be aware of the logistics of publishing a work of nearly a million words, as to my knowledge she has not yet published even a short book, but her claim demonstrates a lack of care and awareness.

These two reviewers, and possibly a third, appear to have assumed that RTN was published in 2016 (or perhaps late 2015), but failed even to check the date on the title page.

Acknowledge One's Limitations

Socrates was declared by the Oracle of Delphi to be the wisest of the Greeks. Yet Socrates stated, "I know nothing." All were ignorant; yet Socrates alone was aware of his ignorance. Others, he observed, lived lives of presumption, claiming to know that which they did not know, and lacking curiosity to investigate for themselves.

Competent scholars work hard to avoid erroneous assumptions, remain open-minded, and always try to refine their understandings. Good scholars are intellectually honest. If they don't know something, they will take steps to find out or defer judgment. I have had reviewers message me for clarification or additional information. One was a professor emeritus of sociology at a university in the Eastern U.S. He was eminently more qualified than any of the MSR reviewers, yet took steps to make sure he correctly understood certain items (he had) before opining in print.

Several MSR reviewers, in contrast, jumped to unfounded negative conclusions based on their own false assumptions and misunderstandings, and spoke beyond the bounds of their knowledge and expertise while failing to perform required due diligence.

Negative Comments Don't Provide Balance if They Aren't True

I wondered why the MSR review contained so many derogatory comments and negative claims. I was advised that these items were included to provide "balance" and "objectivity." Criticism, it seems, is also a reviewer's way of asserting his or her authority and sophistication.

Reviewers can and should bring new perspectives.  Impartial reviewers will rarely if ever agree with the authors on everything. Honest, impartial, and accurate feedback should be offered in good faith. Reviewers may offer original observations or alternative ways of thinking that the authors may not have fully considered or explored. Independent review by ostensibly impartial experts, who may be able to perceive defects or shortcomings not apparent to the authors, is an essential part of the scholarly process. 

Nor is parity between negative and positive items an indicator of an impartial review. Research clearly demonstrates that negative items carry much more weight than positive ones and that multiple positive comments are needed to offset a single negative one. Developmental research shows that humans remember negative stimuli significantly more than positive ones (see for instance Vaish A, Grossman T, Woodward A, "Not all emotions are created equal: the negativity bias in social-emotional development," Psychological Bulletin 2008 May 134(3): 383-403, online in html and pdf format). Similar research across a range of disciplines has confirmed similar findings. People tend to remember and be more influenced by negativity than positivity. Harvard Business Review cites "The Ideal Praise-to-Criticism Ratio" (J. Zenger and J. Folkman, 15 March 2013, online here) at five positive comments to one negative one in offering interpersonal feedback and constructive criticism (see also Losada M, Heaphy E, "The Role of Positivity and Connectivity in the Performance of Business Teams: A Nonlinear Dynamics Model," American Behavioral Scientist, 1 February 2004, online here). 

This awareness extends to other fields. For Uber drivers, ratings near five stars are the norm. Drivers with middling ratings are removed, and even a rating of 4.6 has been described as a driver's "death zone." Politicians tend to use negative advertisements about their opponent even though the public dislikes negativity because such ads are memorable and impactful. 

I am in no way suggesting that reviewers should adhere to a specified "ratio" of positive to negative comments.  Reviewers are obligated to be accurate impartial. The positives and negatives will depend on the merit of the work. But a review with similar numbers of positive and negative items is not a "balanced" review: it is perceived as significantly negative.

No criticism that is false can provide "balance" or "objectivity." Reviewers who make derogatory comments have both a scholarly and an ethical obligation to check their facts and ensure that derogatory comments they make are accurate, fair, and valid. If a reviewer is going to claim that an author bungled a fact, he had better make sure that the author is in fact incorrect and that his reading is accurate. If a reviewer claims that a historical item is inaccurate, he had better be sure that his view represents a mainstream consensus and that the author's statement is out of the accepted scholarly mainstream, instead of the reverse. If a reviewer claims that work is deficient because it fails to meet an alleged "standard," this had better be an accepted standard of the extant scholarly literature rather than an arbitrary imposition invented by the reviewer. And a reviewer who pontificates on pie-in-the-sky suggestions and offers an expansive "wish list" should, at a minimum, make sure that he has fully acknowledged and appreciated the work's contribution to the extant literature before indulging in expansive flights of fancy. The MSR review failed extensively on all of these points.

Conclusion

We were willing to consider other perspectives and were eager to find some thoughtful insights in the MSR review that would enhance our work. Sadly, beyond the correction of a single factual error perpetuated from one of our sources, the MSR review offered almost no relevant or applicable insights. Instead, our time was spent addressing criticisms of various reviewers who muddled their own facts, cited invalid references, misread and misquoted our work, misquoted data regarding their own ostensible countries of expertise, and spectacularly failed to grasp the nature and scope of our research. We would not be able to utilize the work of research assistants or ancillary authors who conducted themselves in such a fashion.

We were repeatedly disappointed by the reviewers' lack of scholarship, fairness, and incapacity to offer reasonable context and perspective. We wondered why they were so eager to discredit us, and why they repeatedly ignored core principles of scholarship to do so. We did not have the luxury of being able to narrow our coverage to only one nation and territory as the reviewers: we had to provide similar data and analysis for 218 others. Notwithstanding this, our data appear to be significantly more accurate and reliable than the reviewers’ claims, even for their supposed areas of expertise.

While I am not a psychologist, such behaviors appear to convey vast arrogance. The reviewers seem to have believed that they were such enlightened experts (and that the authors, conversely, were bumbling incompetents) that they did not need to bother with fact-checking or academic referencing. Some of the authors may have been engaged in self-affirmation, attempting to establish themselves as "experts" by putting us down. We wondered if some felt threatened that someone else had written about “their” country without first seeking their permission and input.  Whatever the case, numerous behaviors in the "review" run contrary to normative scholarship.

Gratuitous criticism is not inconsequential. Our concern about the MSR piece led us to discontinue work on a second edition of RTN. We welcome serious scholarly discussion in good faith. However, we have neither time nor interest to engage those who present bizarre alternative histories, who levy extensive criticisms while misquoting our work, bungle their own facts, demonstrate a lack of expertise in the field and lack of knowledge of the extant literature, and engage in sweeping delegitimization of our entire research endeavor without conducting rudimentary due diligence.

The MSR Review of RTN is not a peer review, as the reviewers were not qualified in the field of the work. It is not a work of scholarship, as the reviewers extensively contravened core principles of scholarship. Its primary value appears to be as a teaching tool so that others can understand what not to do in a scholarly review by examining its flaws, and improve their own scholarship thereby.