The week of 15-18 September 2015 saw the 27th annual conference of the European Association for International Education (EAIE) held in Glasgow, UK. As part of our ongoing investment in new methods and strategies for understanding the outcomes of scholarships, Dr Matt Mawer from the CSC’s Evaluation team joined a panel of colleagues from EP-Nuffic and DAAD to discuss ‘Tracing the outcomes of study abroad scholarships’.

For his contribution to the panel, Matt highlighted three main types of challenges facing evaluators:

  1. Tracing or tracking challenges
  2. Measurement challenges
  3. Analytic challenges

He also presented several possibilities for working across academic, professional, governmental, and non-governmental communities to overcome or circumnavigate some of the more difficult facets of understanding scholarship outcomes. Below are a few highlights from his contribution.

Tracking challenges

To get an appreciation of the timescale involved in tracking scholarship recipients, it is useful to reflect that if one were interested in the early career experiences of doctoral students then the course of an evaluation may run for 15 years.

eaie-ppt-phd-early-career

This is a long time to maintain contact with alumni and engage them to participate in evaluation research. Additionally, evaluators want to track outcomes as they happen and inform policy in response, not to react to policy decisions ten years or more after their effects are felt.

As fellow panel member Dr Martin Bruder (DAAD) noted, longitudinal studies – rather than retrospective snapshots – are becoming increasingly important in scholarship evaluation because they allow tracking over time and not only ex-post-facto evaluation.

Measurement challenges

Scholarship programmes frequently use cross-national surveys, an instrument designed for collecting information from alumni in different countries. Often the context is very wide – sometimes globally – and frequently concerned with processes like employment trajectories that might work differently in each location.

eaie-ppt-common-topicsOne challenge faced by evaluators is that, while these tools suit their particular evaluation, they often produce largely incomparable data if they are not coordinated internationally to be used in the same way by different evaluators. In many cases, evaluations tend to measure at least some of the same variables: employment outcomes, contacts maintained with host countries, and research activity, among other topics.

With these convergences, it should be possible to develop several core areas in which outcomes can be tracked across many scholarship programmes – a ‘post-scholarship trajectory survey’ – and thus produce useful combined data for wider analysis.

Analytic challenges

Sometimes in analysis of scholarship outcomes there has not been close examination of how the academic and non-academic components of scholarship programmes might lead to particular outcomes.

eaie-ppt-scholarship-outcomes

What features of courses help to promote future civic involvement? What features of courses promote return home, and what features encourage staying abroad? What role do institutions play in preparation for returning home or in the reintegration of students when they do return?

Answering these kinds of questions will require close collaboration between scholarship funders, evaluators, and academic institutions to link up data on post-scholarship outcomes to data on study experience. Understanding the wider consequences of particular programme designs for the long-term trajectories of scholarship recipients is an exciting project for scholarship providers, evaluators, and host institutions to take up together.

The need for collaboration

Working in collaboration to ‘trace the outcomes of study abroad scholarships’ has great potential to improve our understanding.

There are topics we can investigate better together, because we have more data, more experts giving collective input, and more alternative scenarios for how scholarships may operate. And there are topics we can only investigate together: such as whether there are any synergy effects or interference effects between programmes operating in the same disciplinary and geographical space.

In addition to their other collaborative work, the CSC will also be contributing to the Association of Commonwealth Universities’ (ACU) new blog series, Measuring Success?, which invites guest contributions on the outcomes of international higher education scholarships and of evidence-informed policymaking.