The Holy Grail of Learning Gain

There is no great enthusiasm for the proposed new framework for measuring teaching excellence across England’s universities. Most Vice-Chancellors who have commented have been critical and the first survey of university staff to come out since the government’s green paper was published is equivocal. But it’s worth a close look at the detail of what is being proposed. At its heart, and still beyond reach, is the holy grail of “learning gain”. If this can be measured, in a valid way that is credible with students, teachers and employers, then real improvements in quality will follow.

The early form of the Teaching Excellence Framework will only apply to English universities (because of devolved responsibilities in Wales, Scotland and Northern Ireland) and it will be launched using existing mechanisms and metrics; existing evaluations by the Quality Assurance Agency, and tried and tired metrics such as student retention, the National Student Survey and crude measures of graduate employment. The TEF has been roundly condemned for this, but such criticisms are based on a shallow reading. Here’s how the green paper sketches out the wider intentions:

As TEF develops we will incorporate new common metrics on engagement with study (including teaching intensity) and learning gain, once they are sufficiently robust and available on a comparable basis. We are also conscious that there are other possible proxies of teaching excellence. … However, we recognise that these metrics are largely proxies rather than direct measures of quality and learning gain and there are issues around how robust they are. To balance this we propose that the TEF assessment will consider institutional evidence, setting out their evidence for their excellent teaching.

Part of the reason for the lukewarm response is suspicion about the government’s ambitions for privatization. These proposals, if implemented, will make it much easier for new, for-profit, universities to be established and for existing universities to be taken over by commercial interests, in the same way that public facilities from water utilities to prisons have been sold off. But in amongst this neo-Thatcherite agenda are proposals that would fit comfortably in a Labour Party manifesto. Two are particularly relevant to the search for measuring learning gain.

First, the green paper proposes that measurements of teaching excellence should be matched with measures of social mobility. This is essential in the context of socioeconomic inequality. Study after study has shown persistent correlations between household income, the quality of schooling and access to Higher Education. Where you live matters, and the universities that do the heavy lifting are those that provide for large numbers of students who are first-in-family, from low income neighbourhoods or other localities with weak traditions of university attendance. The approach proposed for the TEF will make each university’s contribution to social mobility apparent in it’s public profile of quality.

Secondly, the green paper exposes the current conventions for grading degrees as the sham that they are:

The Higher Education Academy found that nearly half of institutions had changed their degree algorithms to; ‘ensure that their students were not disadvantaged compared to those in other institutions’. …. Over 70% of graduates now get a First Class or 2:1 degree, compared with just 47% in the mid-1990s. In 2013/14, over 50% of students were awarded a 2:1, suggesting that this grade band not only disguises considerable variation in attainment, but also permits some to coast.

In response, the green paper is proposing a planned transition to a Grade Point Average system, that can incorporate authentic measures of what learners actually achieve in their time in Higher Education. The push for this will, increasingly, be from both students and employers.

GPA measures have, of course, long been an accepted currency in the US, and are known to have a strong correlation with student progression, graduation and subsequent success. Indeed, the US is now edging beyond the use of GPA measures and towards the acceptance of competency measures. Degree classifications are a debased currency, and need to go.

But the challenge of measuring learning gain still remains.

As the green paper notes,

Our expectation is that effective metrics will be:

  • valid: the metric provides a useable measure of or proxy for teaching quality

  • robust: the metric is based on accurate data that has been subject to rigorous quality assurance

  • comprehensive: the metric provides wide coverage (except in the case of some additional metrics) that enables institutional and subject level comparisons

  • credible: the metric is established and has gained the confidence of the sector

  • current: the metric has been collected in the last 3 years.

The rich digital footprint that every student leaves now offers the possibility of meeting these standards in developing acceptable proxies for learning gain.

For example, pilot studies have shown how the language used by learners in online engagement can be analysed for semantic development, tracking whether or not students move towards the use of more abstract conceptualization. In a second example, network analysis shows how those on a course move from dependency on a one-to-one relationship with an instructor and towards forming independent, peer-to-peer learning initiatives.

These are early days. But the timeline that’s been set out for the Teaching Excellence Framework allows for the development and applications of new measures that will be reasonable proxies for the value that is added by completing a university-level programme. This must surely be a prize worth pursuing.


BIS: “Fulfilling our Potential: Teaching Excellence, Social Mobility and Student Choice”. Green Paper, November 2015.

Jack Grove, “Lukewarm support for TEF from university staff, survey shows”. Times Higher Education. November 11 2015

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s