Five Axioms for Writing Powerful Delphi Statements

The most critical skill for creating a high-impact Delphi study is the ability to write powerful statements. Taking a few moments to learn how to write statements correctly will save hours of time and can increase the quality of your research remarkably. It can also save your project from the horrible outcome of spending months recruiting experts and giving survey rounds only to find that the final data obtained is useless.

Although the Delphi methodology has changed little in the 70 years since its invention, it remains one of the best methods of forecasting and gaining consensus. As the COVID-19 pandemic continues, and the opportunity for personal meetings in the form of focus groups or expert panels has been reduced, the Delphi method is an increasingly attractive method for research in innovative fields.

Sadly, the logistics of doing a Delphi study are not without hurdles. Many design decisions must be made by the study investigators before the project even begins. Getting these design decisions wrong may completely invalidate a study. The result: wasted time and effort.

As a statistician, I am often consulted to help with the design of Delphi studies. And, I usually comment that the statistics behind a Delphi study are not difficult, but merely tedious. The most common mistake I see is failure to properly construct the statements for the Delphi study. Many researchers approach the writing of these statements as if they were part of a survey or questionnaire. Regrettably, Delphi statements are not the same as survey questions: they are a unique breed. Furthermore, if the design of the statements and the rating scale is poor at the outset of the project, no sophisticated statistics or clever analysis can save the study.

Fortunately, while the requirements for a great Delphi statement are not the same as a survey question, writing the statements need not be difficult. Researchers should follow these five simple axioms to create powerful statements.

Axiom 1: Statements Over Questions

The best Delphi studies present a series of statements, and ask the experts to rate their agreement with each statement. This allows the project team the ability to present an organized and actionable report. Traditionally, the report will provide a list of all statements that have reached consensus in the order of the strength of agreement.

For example, imagine a startup business that is looking into the possibility of developing a series of hotels and restaurants dedicated towards quality experience for pets. The experts are a group of restaurant and hotel owners who have experience in the hospitality industry. A traditional survey might ask a question such as "would you be interested in managing a restaurant that allowed cats?" A Delphi statement for the same topic would be: "Restaurants should be developed that cater to the unique needs of cats." Statements have a number of unique features that are critical for the reporting of Delphi studies - more on that to come.

Axiom 2: One Scale to Rule them All

Because Delphi reporting involves assessing consensus and then ranking the statements, it is best to have all statement use the same rating scale. My experience is that the best rating scales for Delphi studies is the 7 point numeric scale, from strongly disagree to strongly agree. A previous blog post Three Ways to Analyze Likert Scales - Avoiding the Likert Crush describes how this simple scale can drastically simplify your research. In practice, almost any survey question can be rephrased to a statement rated with the 7-point agreement scale.

Using the 7-point agreement scale dramatically simplifies the analysis of the final data. Analysis involves simply obtaining the standard deviation of the ratings for each statement - a value of 1.0 or less is usually considered to be consensus. Then, the mean value on the 7-point agreement scale is taken as the group's overall agreement. Finally, in the report, the statements are ranked by this mean value.

Axiom 3: Statements Must Stand on Their Own

It comes as a surprise to many researchers on their first Delphi study that all statements do not necessarily reach consensus by the end of the study. I consider this the mark of a good Delphi study - it shows that the researchers are thinking in an innovative and progressive manner. Delphi studies are often about forecasting the future or about developing guidelines for future events. Innovative studies often venture into new and uncharted territory, and it is normal that there will be topics where the experts simply do not agree. Statements that do not meet reach consensus are dropped from the report of recommendations.

Because of the need for consensus, researchers will not know in advance which statements will be in the report, and in what order they will be, the statements must be independent.

For example, a typical survey or questionnaire might use this format:

Which of the following pets should be allowed to swim in the facilities pet pool?

1. Cats
2. Dogs
3. Ferrets
4. Snakes

Unfortunately, this type of survey question does not easily lend itself to analysis in Delphi studies. For instance, the expert group may have weak consensus about cats, strong consensus about dogs, and no consensus about ferrets and snakes. If this occurs, cats and dogs will both appear in the final report, but not together, one at the top of the results table the other at the bottom. Making it difficult for the readers of the report to understand.

Instead, this question should be divided into four independent statements rated on the 7-point agreement scale. For example, one statement would be "cats should be allowed to swim in the facilities pet pool," and another would be "dogs should be allowed to swim in the facilities pet pool," and so on. In the end, there are four independent statements that are easy to order and report.

Axiom 4: One Statement for One Fact

In the effort to quickly get the most amount of information many researchers make the error of developing statements that are too complex and introduce bias by containing several concepts in one statement. Consider, for example, the statement, "the facility should have a pet pool to allow dogs and cats to swim for recreation and rehabilitation." This type of statement introduces a huge amount of bias as the statement is really several concepts contained in one. For instance, what if an expert agrees that dogs should have a pet pool but not cats? What if the expert feels strongly that the pool should only be used for rehabilitation and not recreation? It is a useful exercise for the researcher to look carefully at the statements and try to think of cases where they may break down.

It would be far more effective to break this statement into several more concise statements, for example:

• The facility should have a pet pool for recreation
• The facility should have a pet pool for rehabilitation
• The facility should have a pool for dogs
• The facility should have a pool for cats

Although breaking the statements up in this way may seem tedious for both the investigators and the participants, it is the most effective way of avoiding bias and assuring that useful data results.

Axiom 5: Use the Whole Scale

One of the major benefits of the 7-point agreement scale is the huge amount of information it can provide - if the whole scale is effectively used. The statements should be arranged so that no matter what the expert consensus (agreement, disagreement, or in-between) valuable information results.

For instance, imagine the statement: "cats are the most important pets to the facility." Depending on the consensus of the experts, the results of the statement may be hard to interpret. For instance, if the group consensus is strong agreement, then the facility should clearly allow cats. However, if the group consensus is not strong agreement, what is the conclusion? Do the experts think that all pets are equal? Maybe they believe that dogs are actually more important than cats, but cats should still be allowed. Maybe they believe that cats should be prohibited from the facility. When the design of the statement is poor, poor quality information may result.

Conversely, consider the statement "the facility should allow cats." In this case, considerably more actionable information is likely to result. If the group consensus is strong agreement with the statement, it makes sense to pursue a facility that allows cats. If the group consensus is strong disagreement with the statement, then it suggests that the facility should not allow cats. And, if the group consensus is in the middle of the 7-point scale, then facilities can probably allow cats or not at their discretion.

Statistically, use of the whole scale is also important. Ideally, we like to see a normal distribution of the individual rating results, with at least one participant choosing the 1 (strongly disagree) and at least one participant choosing the 7 (strongly agree). How can researchers be sure that the experts will use the whole scale? While it is not always possible to predict what the experts will say, it can be a useful exercise to imagine what may happen. If the mean result is 1 on the 7-point scale, what would that signify to us? What if the result was 7? What about 4? If all values will provide useful information, then it very likely that the whole scale is being used effectively.

The Perfect Delphi Statement

In a previous blog post These 10 Things are Killing Your Survey we wrote about the deceptive simplicity of survey design. In essence, it is easy to write a survey, but hard to write a good survey. Writing Delphi statements also seems deceptively simple. While the Delphi method can be an excellent tool for forecasting and gaining consensus, it is exquisitely sensitive to the quality of the statement written by the investigators. Getting high-quality results demands high-quality statements. And writing high-quality statements is not easy.

In many ways the perfect Delphi statement follows the rules of Antoine de Saint-Exupéry: “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” How do you attain this perfection? Follow the five simple axioms. Make statements, not questions. Use a single scale for all statements. Make sure statements stand on their own. Use one statement for each fact. Use the whole scale. And, if you are not confident, ask for help from an experienced statistician - before you perform your study instead of after.

Download the infographic checklist if you would like a quick concise summary of the Five Axioms for Powerful Delphi Statements.