Language proficient pilots and controllers spend much of their working lives engaged in the dynamic exchange of information and negotiation over the radio, moving seamlessly between standard phraseology and plain language when phraseology does not suffice. This skill, known as ‘code-switching’, requires deep operational knowledge intertwined with proficiency in both phraseology and plain language. Often, the two are mixed together in single transmissions, blended to the point that two becomes one, bound together in communicative matrimony. This discourse of radiotelephony (RT) is at very heart of the ICAO Language Proficiency Requirements (LPRs), and is baked into the Rating Scale at ICAO Level 4 where in the ‘fluency’ criterion, the descriptors refer explicitly to the ‘transition from rehearsed or formulaic speech to spontaneous interaction’. Safe and efficient RT relies on proficiency in both plain language and phraseology. Inaccurate or inappropriate use of either leads to misunderstanding which leads to wasted time on increasingly busy frequencies, or worse still, loss of situational awareness and the erosion of safety.
A basic principle of specific-purpose language assessment holds that in order to make valid inferences about a test taker’s ability to perform communicative tasks in real life, we should provide test tasks that capture the characteristics of real-life communicative conditions as closely as possible. In the context of assessment of aviation English in accordance with the ICAO LPRs, this means presenting content and tasks that engage pilots and controllers as participants in RT communications, both in terms of what test takers hear, and in terms of what they say.
The closer we get to the language of the radio, the more valid our inferences will be. Likewise, the further we get from the radio, the more indirect the observation, the less valid the inference. Simply put, if we want someone to demonstrate that they can do something, we ask them to do it, and observe their performance. For example, if we want to know if someone can ride a bicycle, we give them a bicycle and ask them to ride it. We may want to ask the person some questions about riding bicycles too, but answering questions about riding bicycles doesn’t reveal whether the person can ride one (they could answer the questions perfectly, but still fall off!) The same is true with aviation language testing. Asking a test taker questions about aviation is not enough. If we really want to know whether someone can communicate on the radio, we need to present tasks that require their active participation in radio communications. Why, then, do so few aviation English tests actually do this?
From our experience designing aviation English test tasks for airline pilots, we know it’s not easy. It requires very careful scenario design that demands both aviation and language testing expertise. We need to think about the phase of flight, weather, time of day, airspace and traffic situation and other considerations such as aircraft type, nature of operation and flight plan. We need to consider the routine and non-routine events around which we intend to elicit plain language, and to build these into a ‘storyboard’, and to construct prompts which trigger code-switching and which tease out more extended plain language performance from the test taker within the tightly restricted parameters of operations and communications protocols. The storyboard needs to be scripted with the phraseologies and plain language for the scenario, and visual prompts need to be developed along with other supporting content such as aerodrome charts, SIDS and STARS to provide operational context. Once drafted, we take the tasks through a cycle of trialling and revision to ensure that they are technically plausible and that they elicit the target language from the test taker. Then, once introduced into the live test battery, we need an aviation subject matter expert interlocutor, ideally one with operational experience in the role (ATC if testing pilots, pilots if testing ATC). In short, developing and administering test tasks that engage the wide range of pilots in RT communications is challenging, time-consuming and expensive.
Allied to these design and administration challenges, and sometimes referred to as a reason for aviation English tests to steer clear of RT communications, is the fact that such tasks do not actually produce much plain language, especially when compared to other types of task such as picture description or Oral Proficiency Interview (OPI). On reflection, one might reasonably argue that the resource required to develop and administer such tasks is disproportionate to the quantity of plain language elicited by them. This may be true. But just like RT communications, in aviation language tests, it’s quality, not quantity, that matters. To observe test takers code switch around routine and non-routine events in an operational context is to tap the very kernel of the ICAO LPRs, providing the evidence we need to make valid language assessments. Without that evidence, a test cannot claim to measure language proficiency for aeronautical communication.
What’s the industry guidance on the matter? For better or for worse, ICAO stipulates a clear separation of phraseology and plain language: ‘Teaching and testing standardized phraseology are operational issues, not a language proficiency issue and … a test designed to evaluate knowledge or use of standardized phraseology cannot be used to assess plain language proficiency’.1 Clearly, judgements about the accuracy and appropriacy of a pilot’s or controller’s use of phraseology is the preserve of operational experts, not linguists. And clearly, tests developed to assess only phraseology (for example, the UK CAA FRTOL2 or the Australian CASA AROC3) are, by design, not suitable for the assessment of plain language. However, we question the logic of separating phraseology from its partner, plain language. In fact, we would argue that, with the necessary skill and care, it’s not only possible but desirable to construct tests that address phraseology and plain language together. On theoretical grounds, the two are co-dependent: Language proficiency means using phraseology at all times but knowing its limits, knowing when and equally when not to use plain language. The two go hand-in-hand, and if they co-occur in real life, tearing them apart for the purposes of assessment creates a communicative artifact. This could be the fundamental reason why so many aviation English tests are wide of the mark.
With the guidance on the assessment of phraseology as it is, what role, if any, can phraseology play in a test of aeronautical communication? ICAO states that ‘it is acceptable that a test could contain a scripted test task ... in which standardized phraseology is included ... as a means of setting a radiotelephony context in which to elicit plain language responses from the test-taker’4. For test designers seeking to get closer to the true target of the LPRs, this is the green light required to address plain language as it occurs on the radio, what ICAO calls a ‘narrow’ interpretation of the Rating Scale and Holistic Descriptors5. However, ICAO goes on to define what it considers to be an equally valid ‘broad interpretation’ of the ICAO Rating Scale on Holistic Descriptors for the purposes of test design:
To elicit plain language … without replicating radiotelephony communications specifically. Examples may include question and answer routines, problem-solving exchanges, briefings, simulations and role-plays6.
This guidance gives test developers carte blanche with the design of test tasks for pilots and controllers, effectively permitting the weaker ‘describe the picture’ tasks, or ‘why did you choose to be a pilot?’ OPIs, or ‘listen to this transmission and tell me what you heard’ tasks so common in aviation language testing. This is very convenient for the aviation English test developer. Not only are they relatively quick and easy to construct and administer, requiring minimal aviation subject matter expertise, but they elicit plenty of plain language from the test taker. The fact that tasks of this nature feature so heavily in the ICAO Rated Speech Samples Training Aid has undoubtedly strengthened the perception that they are acceptable, if not the perception of endorsement7, which has helped to create the echo chamber of construct-weak test tasks that we find ourselves in today. The question is, what can we infer from performance on such tasks? Does successful performance allow us to infer that a test taker can communicate on the radio in real life? Our view is that such tasks elicit performances that are too far from RT communications to make this bold leap of faith.
It’s strange to us that while ‘the sole object of the ICAO LPRs is aeronautical communications’8, tests which actually address RT communications are the rare exception rather than the rule. Professor Charles Alderson famously stated in 2010 that ‘we can have little confidence in the meaningfulness, reliability, and validity of several of the aviation language tests currently available for licensure’9. Given that radio communication is conspicuously absent in so many of today’s aviation language tests, maybe this damning indictment of the state of aviation language testing is still true.
But there is a glimmer of hope. The International Civil Aviation English Association recently published Test Design Guidelines ‘developed to help civil aviation authorities and organisations involved in the design of LPR tests recognise and understand key issues related to the design of LPR tests and their impact on overall LPR testing practices’10. There are eight criteria in total. Criterion number 1:
Test instruments need to include appropriate tasks that directly assess how test-takers use language in radiotelephony communication contexts.
If the Test Design Guidelines have their desired effect, as we hope they do, maybe in time we will see radio communication take its rightful place as the centrepiece in aviation language tests the world over. Maybe we will be able to rip up the divorce papers and see tasks in which plain language is happily reunited with its long-lost partner, phraseology, and the two will ride off into the sunset, hand-in-hand in matrimonial harmony.
To find out more about how Latitude can help you with aviation language test development, email info@latitude-aes.aero.
1ICAO (2010) Document 9835, Manual on the implementation of language proficiency requirements, section 6.2.8.6
2Flight Radio Telephony Operator Licence (FRTOL)
3Aeronautical Radio Operator Certificate (AROC)
4ibid. section 6.2.8.8.
5ibid. section 6.2.8.9.
6ibid. section 6.2.8.9
7From the Rated Speech Samples Training Aid: ‘Inclusion of any speech samples should in no way be interpreted as an approval, endorsement or otherwise’.
8ibid. section 6.2.8.9.
9Alderson, J.C. (2010) A survey of aviation English tests Language testing 27 (1) pp. 51–72
10https://www.icaea.aero/icao-lpr-tdg/guidelines/