Potholes and fallen trees – the road ahead

Earlier this year (May 2015) we had an election here in the UK so it seems appropriate to try and scout out what may lay ahead, based on the budget statement, the business secretary Sajid Javid’s speech on productivity, and Jo Johnson’s speech to universities UK.

While there are some aspects that I agree with there are others where their impact very much depends on the detail of its implementation. Overall, there is a lack of coherence and a number of contradictions, although few are as blatant as this example from the energy section of the productivity document:

I'm pretty sure one of these isn't going to get done

I’m pretty sure one of these isn’t going to get done

or the proposal to double the proportion of disadvantaged students entering university. That sounds great, until you realise this is the government that tripled tuition fees, removed Education Maintenance Allowance that supported 16-18 year olds in study prior to university, and is removing maintenance grants aimed at the poorest students only to replace them with more loans.

An overriding theme throughout all of these speeches and documents is that of monetary value being the only measure of importance. Johnson says that only half of students polled thought that their course had provided good value for money[1], but a huge amount hangs around how we choose to define value. Is it simply a business measure of return on investment – the improvement in our salaries for the cost incurred? Is it that our path to a ‘good’ job (another loaded word) is eased? Or is it the knowledge that we were exposed to, our lecturer’s enthusiasm for their specialism that inspired us to change direction? Or simply watching the sun come up with friends after the final summer ball with that bittersweet mix of relief at having succeeded and regret that our time is ending? Do we measure ‘value’ in the ache of intellectual exertion and growth or in the frame we hang on the wall afterwards? I remember my eldest daughter wanting help with her English essays (i.e. ‘tell me what to write’) and us having the same conversation a number of times that the homework was the research, analysis and synthesis of the raw material. The homework was the process and the piece of paper handed into the teacher was just the proof that you’d done it.

There is a direct link to earnings made[2] and the phrase ‘… enable education providers to assess their effectiveness in delivering positive labour market outcomes for their students’ appears so there is little doubt what definition of value is being used. It’s obvious no one actively involved in the process of education wrote this. Is this really how we want to gauge higher education, in “positive labour market outcomes”? Yes, we want our students to be successful in whatever field they choose, but that should naturally follow on from quality education anyway. Having those outcomes may be appropriate for vocational courses (of whatever level) where that’s an explicit objective of the course anyway. That’s not a dig at vocational education, by the way. We have a very biased attitude towards vocational education in the UK, seeing it as the poor relative of higher education and a refuge for those lacking the ability to do a degree. That’s profoundly wrong. We only have to look across the channel to Germany to see what a less prejudiced approach to vocational education can achieve for individuals and the society they are part of.

As part of the push towards transparency and ‘more informed choices’ elements of the US version of the National Student Survey (NSS) are to be piloted. This is good because the American survey is much more fit for purpose in terms of measuring what’s important for teaching and learning rather than the ‘customer satisfaction’ surveys we use in the UK, but it does depend on how those elements are integrated with the current NSS. Jo Johnson’s speech talks about using these elements to measure ‘learning gain’. My big fear here is that the metric we use is an easy-to-measure proxy rather than a true representation of the thing we’re interested in. The speech defines learning gain as ‘the improvement in knowledge, skills and work-readiness’. Leaving the issue of work-readiness to later, these are all graduate properties that are emergent and as such at this level will be relatively difficult to measure with reliability and validity. What this will mean, as has been the case with the ‘value-added’ methodology in schools, is a push towards a proxy that may be only loosely representative of the true quality we’re trying to measure.

Another element to this danger of proxy worship is the next paragraph in the speech which, while acknowledging that “independent learning is vital”, then states that universities should be more transparent about how student time is split between lectures, seminars and tutorials, as well as who is doing the teaching. This information is valuable, but is not the whole story. I’d certainly welcome information on who is doing the teaching being more available, even if only to avoid the explosion in the use of adjuncts that has happened in the USA. Contact hours, however spent, will vary according to discipline, with STEM subjects typically having higher contact hours than the humanities, and again I can see no issue with this information being available in principle. The issue is that without some more qualitative metric to counterbalance them these numerical measures will distort how higher education is provided. Independent study will be sidelined in favour of contact hours since that will be the measure that importance is attached to, but how are those contact hours to be delivered, and by whom? Staff time (already stretched) costs money and student numbers are increasing, so if there are to be no additional resources provided and we can’t (and shouldn’t) reduce costs by using adjuncts or post-graduates to do the teaching, how on earth is it all expected to come together in some sort of coherent whole?

There is a worrying mention of the use of competition to drive up standards. It’s worrying in two ways. First, the practice of education is a collegiate activity and we succeed, as Newton put it, by “standing on the shoulders of giants”. Secondly, the productivity plan[3] makes clear what form this will take place by saying “widening the range of high quality higher education providers” and “remov[ing] barriers preventing alternative providers from entering and growing in the market”. To me, this sounds very much like fast-tracking a large-scale expansion of the for-profit sector in UK higher education. Given the current high standing with which UK higher education is held and widespread concerns over some of the for-profit colleges in the United States this seems a very risky strategy to pursue.

The productivity plan then goes on to say the government “will explore options to allow the best providers to offer degrees independently of existing institutions before they obtain degree awarding powers”[4]. I’m a little confused by the inclusion of this sentence. Courses can be already taken at institutions that don’t have their own degree-awarding powers provided that they are accredited by an institution that does – my own post-graduate teaching qualification was taken at an further education college but validated by a university – so why the need to ‘explore options’? Also, how is ‘best’ to be measured, and by whom?

The Jo Johnson speech mentions ‘innovation in terms of course length and design’[5], with an explicit mention of degree apprenticeships. This is a welcome acknowledgement of the importance of vocational education. Unfortunately, Johnson also mentions two-year courses on the grounds that they offer a more accessible route to higher education and, together with degree apprenticeships, offer a ‘faster route to productive employment’. I think it’s much more likely that accessibility to higher education is more adversely affected by the level of debt students will have on graduation rather than the length of the course. And why should a faster route to employment even be a goal? What evidence is there that a shorter course would produce graduates with comparable skills and attributes to those that graduate after three years?

Two-year courses would be attractive to those new for-profit providers since they can produce more graduates than a conventional university in the same time, particularly if the Teaching Excellence Framework allows some institutions to self-certify their level of performance (as has been done in schools where those that received the higher grades on inspection were re-inspected on a longer timescale and effectively inspected themselves in the intervening period). Together with the uncertainty around the degree-awarding powers for new providers these ‘more accessible’ pathways to HE and shorter routes to employment seem to be a recipe for dumbing down, slipping standards, and increasing ‘shareholder value’.

Johnson also talks about course choice and curriculum design[6], saying that not enough is being done to ensure graduates are prepared for work, and quotes a CBI/Pearson Education and Skills survey that nearly half of employers (47%) felt that universities weren’t doing enough to make graduates job-ready. Well, that would depend on how you chose to define ‘prepared’ and ‘job-ready’. My own response to that 47% of employers is: do your own training. We’re not your HR department. Graduates should have graduate-level, transferable skills – that’s the contribution from higher education to being job-ready. That means graduates can adapt to the specifics of a particular role or industry, but it’s the employer’s job to mediate and support that adaptation. If business has any evidence that we’re not producing that standard of graduate (beyond vague complaints that graduates are not ‘job-ready’) then they should produce it.

Johnson then says that businesses should not just be customers of universities, but “active partners”, and says that business outreach should be a core function with influence over curriculum design. Absolutely not – I repeat, universities are not part of a business’ HR department. Education is not the same as job training. Should businesses have some involvement in curriculum design? Yes, of course – collaborations with industry can be very productive for both sides – but decisions around the curriculum should rest with the university except in rare cases.

I’m not anti-business – trade and enterprise have always been a part of human societies and business is integral to modern societies – but I am against the fetishistic worship of business and entrepreneurs as having all answers to all questions. Increasingly, mainstream economics and politics seem to revolve around the mantra ‘the answer is a free-market. Now, what was the question?’ although there are others such as Naomi Klein and Paul Mason putting forward alternative viewpoints. Education has huge societal value beyond servicing the needs of business, but this seems to be increasingly neglected. A telling point is that higher education in the UK is administered not from the Department of Education, but from the Department of Business, Innovation and Skills.

Overall, the direction of travel for higher education in the UK does not look good. Statements and aims often explicitly contradict each other, such as wanting to widen access while simultaneously constructing more disincentives. The language of business dominates with the focus and metrics primarily targeted towards financial measures rather than anything innate in the process and practice of education. These are not the characteristics of a healthy higher education system, unless the function of that system is to be a subsidiary of business. These are not the characteristics of a system that Johnson himself acknowledges is a “world leader in higher education … and as number one in the world for the impact of our research”.

footnotes

1Johnson – Meeting value for money expectations.
2Johnson – More informed choices.
3Productivity plan – section 4.10
4Productivity plan – sections 4.9 and 4.10
5Johnson – Widening Participation.
6Johnson – Course choice and curriculum design.

Advertisements

Fall seven times, stand up eight

Recently I attended my university’s Teaching and Learning Conference where the theme was on employer engagement: “Beyond the Placement: Where Next?”. The conference didn’t start with the usual long keynote speech from a single speaker, but was more of a joint effort with four speakers giving their perspectives on the theme. First was Professor Carol Costley from Middlesex University talking from the perspective of the academic as researcher, followed by Professor Richard Sandell from the School of Museum Studies here at Leicester, who spoke about how curriculum design could be used to increase the range and diversity of contact between students and employers. The third speaker was Jennifer Williams from Teach First who spoke about the core competencies and values they looked for in their graduate recruits, and the fourth speaker was Dabean Faraj, a medical student, who gave an excellent and humorous presentation that brought the differing strands together from the perspective of a student.

Resilience chalked on road

Photo: miz_ginevra (CC BY 2.0)

The keynotes over we departed to various parallel sessions before coming together later for discussions. The discussion highlighted the difference (and potential conflict of interest) between education and training, and as one of my colleagues put it we were veering “dangerously close to discussing the purpose of a university.” I’m firmly in the education in its widest sense camp rather than education as training. That’s not to say that employers shouldn’t be involved just that that involvement needs to be carefully considered. Employers can bring a great deal to the table, but their power needs to be kept in check. Unfortunately, a recent speech by our new Minister of State for Universities and Science, Jo Johnson, seems to be indicating a different agenda “Businesses should not just be seen as customers of universities … but as active partners”. A lot depends on that definition of ‘partner’, but that’s the subject for another post.

Jennifer Williams mentioned that resilience was one of the core competencies that applicants tended to score lower on. So what do we mean by ‘resilience’? One of the other attendees described it as ‘bouncebackability’, the ability to pursue a goal or objective with flexibility and determination in spite of difficulties and setbacks along the way. It’s a small part of a person’s personality but one that can have a big effect.

So how do we develop it? One suggestion was for failure to be a low cost option, with an opportunity to succeed afterwards. This may happen, and our response to it in conjunction with our students can help them to become more resilient, but how could we possibly design for that? Are we really going to deliberately engineer situations that will cause our students to fail in order that we can teach them to be resilient?

An alternative, with apologies to Vygotsky, is to use the ‘zone of proximal discomfort’. We learn resilience not in the act of failing, but in our response to it, in the act of striving to overcome those challenges that we think are beyond us. In the words of Tennyson, “ … To strive, to seek, to find, and not to yield.” An example of this is military training where a recruit is taken out of their mental and physical comfort zone, but not beyond their ability (although they might not think so, particularly in the case of the military where it’s less stepping outside the comfort zone and more of an emigration). The end result is an increase in confidence, ability and the resilience and character to go forward when all your senses are telling you to take cover instead. Schools are increasingly starting to recognise the value of resilience as part of the focus on character education, precisely because of the impact it can have on learning and the students’ lives as a whole. Which of course brings us back full circle: education as a holistic practice, not job training.

Assessment with portfolios – a neglected opportunity?

I finally got around to reading Tony Bates’ blog post on the outlook for online learning in the short and medium term. It’s an in-depth post, but I want to concentrate on one particular aspect. Tony puts forward the idea that the traditional lecture-based course will gradually disappear as learning shifts from the transmission of information to knowledge management. Later, he talks about an increase in the use of portfolios for assessment, and to my mind these two are a natural fit. This is because as teaching shifts from the presentation of a package of content to students meeting a common set of criteria traditional written assessment becomes increasingly less fit for purpose.

Clipboard and pen

Photo credit: Dave Crosby (CC BY-SA)

The idea of assessment by portfolio in higher education isn’t new. A few years ago, I saw Lewis Elton give the plenary at the teaching and learning conference at the University of Birmingham, where he suggested using portfolios as a replacement for the UK system of degree classifications. The plenary covered much of the ground in his 2004 paper ‘Should classification of the UK honours degree have a future?’ and provoked much discussion. I believe there is a case to be made for portfolio assessment to have a greater role in higher education. As Elton says, the “measurement of achievement has become more important than achievement itself”. In a recent article discussing competition in education, Natalie Bennett (the leader of the Green Party here in the UK) quoted the Education Reform Act 1988, which said that education should: “promote the spiritual, moral, cultural, mental and physical development of pupils … and prepare pupils … for the opportunities, responsibilities and experiences of adult life.” The quote relates specifically to the context of secondary education (11-18 years old), but it’s still a pretty powerful quote. And if that’s the case for school children then surely it should hold for university and higher-level study when their capacity to engage in activities that promote those values is arguably higher? We may want assessment that is both valid (measures what it is designed to measure) and reliable (gives consistent results) for very good reasons (such as quality control and comparisons across institutions), but that restricts the types of assessment we can do. What we get (across the education system as a whole) is assessment that comes down to a number or grade, and that steers the testing towards that which is easily measured, for example, a 1,500 word essay or a three hour exam.

So what does assessment by portfolio give us that conventional assessment doesn’t? First, it’s an active process and in a well designed assessment the learner has to engage over a period of time, evaluating and reflecting on their learning. Students can revisit their work and re-purpose it, for example, for use within a presentation to a potential employer. This does bring in issues of ownership and access after the end of the course, but I’ll leave those aside for now, and in any case those criticisms isn’t unique to portfolios.

Secondly, higher order and transferable skills can be assessed more effectively than they could through a conventional written assessment. For example, selecting items for inclusion will involve evaluating individual items, reflecting on their purpose and value, and synthesising the collection into a coherent whole. This does mean that the learners need to have a high level of independent learning skills, which may not be the case. A supportive pedagogical design with clear scaffolding and direction can help develop these skills. Another point is that this form of assessment is authentic – it assesses in a direct analogue of how use these higher order skills within a workplace. Support and direction need to be explicit, not only on the process of the assessment, but also on its purpose. My daughter used an eportfolio as a record of achievement during her degree, but didn’t see the point and questioned why they couldn’t simply submit their assignments and leave it at that. Another acquaintance is studying for a primary PGCE on a course that uses the Mahara eportfolio and said that it’s almost universally hated, mainly it seems because they find the user interface unintuitive.

The portfolio can be an integrating influence that draws the rest of the course together for students. They have been used very successfully on a Master’s level distance learning course by the Open University (Mason et al, 2004). The course consisted of four modules with two items being selected from each module for inclusion. Two thirds of students were positive about the role of the portfolio as an integrating element in the course.

So what are the downsides? Well, most criticisms centre around the issues of reliability and validity, but that brings us back to Elton’s statement that achievement is second place to the measurement of achievement. Elton also said that the “prime purpose of assessment should be to encourage good learning” (Elton, 2004), and that there should be a bias towards success and not failure. He’s not referring to grade inflation, but that we should move away from the deficiency model of traditional assessment. This brings to mind the perennial debate around standards, ‘dumbing down’, and whether assessments should act as a gate-keeper (norm-referenced) or as a marker of achievement (criteria-referenced). Should everyone on a course be able to get a first class degree? Absolutely, if they’ve met the standards for a first-class degree, but I can imagine the outcry if a department were to award first class honours to an entire cohort, supposing they were lucky enough to get such a cohort in the first place.

Elton recommends that if something can be assessed reliably and validly then grade it conventionally. If something can be assessed validly but not reliably then it should be graded pass/fail using experienced examiners. If it can’t be assessed either reliably or validly then it should be reported on (again by experienced examiners) rather than graded. Knight (2002) had a similar argument, stating that summative assessment should be restricted to “that which can be reliably, affordably and fairly assessed”. Skills development and other aspects of the curriculum should be formatively assessed. Portfolios, then, blur the lines between formative and summative assessment.

Portfolios have great potential for assessment, provided that they are used wisely and that significant effort is made to change the students’ focus away from the product. It’s like getting an essay for homework at school – the essay isn’t the homework, the homework is the process of research, drafting, revision and synthesis, and the printed essay is just the evidence you did it. The sticking point is portfolios they exist within an educational ecosystem that functions to support and validate conventional assessment, and that is likely to change only slowly.

References

Bates, T. (2014). 2020 Vision: Outlook for online learning in 2014 and way beyond.

Bennett, N. (2014) Let’s Get Heretical on Education: Competition Has Failed.

Elton, L. (2004). Should classification of the UK honours degree have a future? Assessment and Evaluation in Higher Education, 29(4), 415–422.

Knight, P. (2002). Summative assessment in higher education: practices in disarray. Studies in Higher Education, 27, 275–286.

Mason, R. Pegler, C. and Weller, M. (2004). E-portfolios: an assessment tool for online courses. British Journal of Educational Technology, 35(6), 717–727.