Translate A Survey in Six Easy Steps

Those of us who do research in more than one language are often at a crossroad when we find the right survey tool in the wrong language. Should we translate the tool to the language we need? Or, is it better to just create our own tool? Whether you are developing a questionnaire, a rating scale, or a test instrument, translating a survey can be a great way to quickly obtain a valid and reliable tool. However, translating a survey is not a trivial task.

There are many reasons why translating a survey is often preferable to creating a new one. In a previous blog post we gave our cardinal rule for designing a new survey: don't design a new survey. It is often much easier to use a previously published tool, and it usually makes sense to leverage the work of other researchers. Especially if the published tool has been validated and shows good reliability. Since there are many reliable and validated survey tools that have been published in many languages, it often makes sense to translate a tool rather than reinvent the wheel.

When I was working with the researchers at SIMNOVA to develop an  Italian language evaluation tool for medical simulations at the Italian SIMCUP, we faced this exact dilemma. Was it best to translate a published simulation evaluation tool? Or, create our own?

Junior researchers often feel that the translation of survey tools is a simple administrative task. I have seen researchers who simply sit down at a computer, translate the research tool using their personal language skills, and then set out to use the tool. They trust that the new translation will maintain the validity and reliability of the original tool. This however is a fallacy. There is no guarantee that a translated survey will maintain the same characteristics as the original tool. Like most important research topics, it needs to be proven. How? By following these six simple steps.

Step 1: Find the Tool

The decision between translating an existing tool or developing a new one begins with a careful search for the right tool. There is little point in attempting to translate a tool that doesn't work.

What makes a tool a good candidate for translation? Firstly, the tool should ask the right questions and have been used in the right population. Do the questions focus on the topic you are researching? Is the survey approximately the right length? Is the format acceptable? Secondly, the tool should have the right survey characteristics. The tool should have published proof of validity and reliability. The whole point of translating an existing tool is to avoid having to directly prove its test characteristics. Be harsh in judging the tool, and only translate a validated and reliable tool.

Want a simple rule to decide if translation makes sense? Do a careful literature search to see how the published test has been validated and how often it is used. How would you feel about trying to reproduce all that research? If the number of publications is trivial, and the work done to assess the tool is scant, it may make sense to just design and validate your own tool. If the literature is extensive - more than you think your team could accomplish - it is probably best to translate.

Where do you find the tool? Often, asking the content experts is the first step. Find out who the leaders are in the field and ask them to suggest their favorite tools. Particularly if there is a de facto gold-standard test already. For instance, if you were creating an Italian language tool for the prognosis of coma victims, the Glasgow Coma Scale is the practical choice. After talking to the experts, do a literature search to make sure that publication of the validity and reliability of the tool actually exists.

When we were developing the simulation evaluation tool for the SIMCUP, we started by asking the experts in simulation in Canada which evaluation tool they used. We were told the Ottawa Crisis Resource Management Global Rating Scale was commonly used. Our research team looked at the tool and felt it was applicable to the population of medical residents who would be competing at SIMCUP. We also very much liked the format of the tool. A quick literature search revealed that the team that created the tool had thoroughly evaluated it for validity and reliability. In addition, the tool had been used in many other publications. After a cursory review of these publications, our team concluded that translating the English Ottawa GRS to Italian would be far easier than developing our own tool.

Step 2: Translate it

Translating any survey should never be a trivial task. The correct translation ensures the content validity of the tool. As many of us who are multilingual will attest, being able to speak several languages does not guarantee you are a good translator.

This is a good time to be very honest with yourself regarding your language skills. Don't overestimate your language skills: get help instead. It is mandatory at this point to think about the purpose of the survey including the level of language proficiency among those using the survey. For instance, if the survey tool is going to be given to participants with university-level education, it should be written in a university-level language. If you are not comfortable writing at the university level - get help.

Working as a team is always the best plan for translation. The best teams are often made of a mix of native language speakers from both the original and target language. Language is subtle, and subtle changes are often noted best by native language speakers.

Cultural congruence is also essential to the correct interpretation of the original tool.  I spoke to Dr. Manuela Verde, COO of Stat59 and co-author of the Italian Translation of the Ottawa GRS, about this, and her point is quite straightforward: “Surveys by design reflect the target population’s cultural code, their behavioral norms and local practices. Unless the translator has familiarity with the original context, it will be impossible to ‘crack the code’ and make sense of the cognitive diversity, i.e. the variety of meanings, perspectives and conventions. A multicultural team is the only way to foster the cultural ambidexterity required to access and convey these intrinsic values and insights.”

Why worry so much about the subtleties of the language? Isn't it adequate to just make sure everybody understands? Unfortunately, small changes in language can make huge changes in the validity and reliability of the instrument. In the worst-case scenario, bad translations can lead to a new instrument that does not have adequate validity and reliability. Sometimes, the poor performance of the tool is not noticed until it is actually used. For instance, imagine a new survey tool that is designed to assess students to see if a new curriculum in calculus results in better knowledge of integration. If the evaluation tool is based on a bad translation, the researchers may only notice at the end that the tool did not find a difference between the old and new curriculum. At this point, the researchers are left scratching their heads: is the lack of difference due to the bad curriculum or due to a bad evaluation tool? Remember, if you are doing high-quality research your translated tool may become the new gold standard in the target language, and other researchers will be relying on your tool for their research. It makes sense to spend a little extra time getting the translation right so that the work is of the highest quality.

When we created the Italian translation of the Ottawa GRS, we used a team of four translators. All were comfortable speaking and writing in both languages. Three were native Italian speakers; one (me) was a native English speaker. We created the translation in an iterative fashion, by carefully reviewing the original tool, translating it, and then sharing it. Each team member had a chance to translate, revise, and ask questions of the other team members. In addition, we tried very hard to replicate the format of the original tool, including using the same rating scale, replicating the format of the page, and translating the instructions carefully. The goal of the translation was to replicate as exactly as possible the English language tool. After translation, we distributed the tool to several native Italian language experts who would be potential users of the tool. We listened carefully to their comments, integrated their suggestions, and again redistributed the new translation to our team. The same iteration was followed several times until no further changes were suggested by either the experts or the translation team. As we were hoping that the new tool would be adequately valid and reliable to serve as the test instrument for the Italian national SIMCUP, we knew the translation needed to be of the highest possibile quality. It would have been unfair to the students in the competition to evaluate them using a poor-quality tool. Only after a careful translation was the survey ready for its first use.

Step 3: Check the Reliability

What is reliability? In the book "Assessment in Health Professions Education," Steven Downing explains that "test or assessment data are reliable to the degree to which they can be replicated or reproduced." Usually, proving a survey’s reliability is a laborious process. Fortunately, translating an existing survey which has already been shown to be reliable is a simple method to get a reliable tool. Simply prove that the translated survey gives the same results as the original survey. This often called "alternate form reliability."

The best way to assess reliability of a translated tool is to use the two tools simultaneously and independently on the same subject. For instance, imagine translating an airplane pre-flight checklist from English to Spanish. Reliability of the test could be assessed by having an English language speaker using the English language tool and a Spanish language speaker using the Spanish language tool on the same airplane. Doing this test multiple times and calculating a correlation coefficient between the two tools is a good way to assess reliability.

Sometimes, it can require some imagination for the researchers to develop a way to compare the two tools. Sometimes, the survey may need to be given once in one language, then repeated in the other language at another date. Or, the participants can be randomized to one tool or the other using the "split-halves technique." Whatever design you choose, make sure to get your statistician involved early when designing the study to ensure the best results.

For our Italian language translation of the Ottawa GRS, we used direct independent simultaneous evaluation of the two tools. We started with a group of bilingual examiners. Examiners were randomly assigned to the English or Italian tool. Examiners rated several simulations using the two tools - the examiners sat in the same room watching the same simulation but independently scoring the participants. Statistical analysis showed a strong correlation between the scores given by the two examiners: giving strong evidence to the reliability of the new tool.

Step 4: Assess Validity

Validity of questionnaires, surveys, or testing tools is vitally important. What is validity? The simple definition is given by Mark Litwin in his short book "How to Measure Survey Reliability and Validity": "how well [a tool] measures what it sets out to measure." Assessing validity is not easy. It often takes years of work and multiple publications to prove the validity of a tool. Fortunately, translating an existing tool is a great way to short circuit this long process. In step 2 we showed that our new translated survey gave the same results as the original tool. Thus, we can conclude that the validity of our new tool should be equal to that of the original tool.

Nonetheless, it is also helpful to look more directly at our new tool to show its validity. How can we assess validity? One very practical way is to assess the concurrent validity of our new tool against another tool. Ideally, we would like to assess the correlation between our new tool and an established "gold-standard" tool. In the case of creating a translation, this reference standard can be in either language.

What if a reference standard doesn't exist? Using a correlation with another test can still be helpful.

In the case of the Italian language translation of the Ottawa GRS, we could find no established reference standard. Instead, we assessed the concurrent validity of the global rating tool (measuring team dynamics) versus the standard SIMCUP technical skills rating tool. We hypothesized that teams showing good communication skills and problem-solving skills would be more likely to have good technical skills. Our study showed quite a strong correlation between the new translated tool and the established technical skills tool. This helped prove the validity of our new tool.

Step 5: Calculate Chronbach's Alpha

No discussion of survey reliability and validity would be complete without talking about Chronbach's alpha.

What is this mythical test statistic? Chronbach's alpha measures the internal consistency of a test - another form of reliability. Let's consider a theoretical survey we give to people in our neighborhood to test their knowledge on how to keep their cat healthy. Our survey contains 20 questions about cat health, including how often to visit the vet, when to get vaccinations, how often to brush their teeth, the best ways to avoid hairballs, and how to keep your cat safe from traffic. It is highly likely that those who know about cat vaccinations will also know about tooth brushing. Chronbach's alpha measures the strength of this association.

Luckily, calculating Chronbach's alpha is not difficult. It does not require obtaining any additional data other than that from steps 3 and 4 above. It is a simple statistical test that can be calculated by any statistics software.

For our translated simulation rating tool, Chronbach's alpha was 0.97. Indicating a high degree of internal consistency.

Step 6: Get Out and Use it

The final step in any survey, questionnaire, or evaluation tool is to get it out into the real world and use it. The measure of how well the tool performs in the real world is called construct validity. And, unlike the other measures of validity, there is no single statistical test to measure it. How do you show construct validity? Encourage other researchers to use your tool. Trial the tool in different settings.

Following our development of the Italian translation of the Ottawa GRS, we worked hard to encourage the use of our tool. The translated tool was published in the official journal of the Italian Society of Internal Medicine. Our goal was to quickly and effectively distribute the tool to Italian language researchers. Since its inception, the tool has been used yearly for the annual Italian national SIMCUP. This provides further proof of the usability of the tool. In addition, we encouraged other researchers working in Italian to use the tool for their simulations.

Putting the Six Steps into Action

Unfortunately, translation of a survey tool is not a one-step one-person process.

Translating a survey tool is a team sport. The six steps of translation set out a simple roadmap for teams to create a high-quality translation. Find a well-validated published tool using expert opinion and literature search. Translate the tool in an iterative fashion with a multilingual team in an iterative fashion. Assess the reliability by comparing your tool to the original tool. Assess validity by comparing your tool to an established reference standard. Calculate the internal consistency of your results. And, finally, publish your translation and get it into use.

As the crisis of reproducibility continues to haunt us as researchers, all of us are under increasing pressure to ensure that our research work is of the highest possible quality. While in the past it may have been considered a normal practice for a single researcher to translate a tool and put it directly into practice, this should no longer be considered adequate. Instead, translation of a tool should be a careful and deliberate process. And, the resulting translation should be considered an academic work worthy of publication on its own.

Looking for a quick summary of How to Translate a Survey in Six Easy Steps? Download the infographic checklist