Evaluating practices in relation to outcomes
Leaders are skilled at evaluating the organisation’s collective and individual staff practices in relation to learning outcomes and wellbeing.
They use high levels of quantitative and qualitative data literacy. They are curious about patterns and practices.
They can describe and identify problems or challenges in ways that open up real discussion and identification of needs, and solutions.
Educational leadership capability framework, Education Council, 2018
This page summarises some thinking and research related to this capability. It is not a comprehensive or definitive background to the capability.
Evaluating practices in relation to outcomes underpins other leadership capabilities, for example, strategic thinking and planning, and adept management of resources.
Assessment, evaluation and inquiry
Systems that perform better are systems in which assessment, inquiry and evaluation information is used effectively to inform learning.
(OECD, 2013)
In New Zealand, the Education Review Office's School evaluation indicators include Domain 6: Evaluation, inquiry and knowledge building for improvement and innovation – see pp 40-41.
Evaluation means making a judgment about the quality, effectiveness or value of a policy, programme or practice in terms of its contribution to the desired outcomes. (ERO, 2015a)
Professional inquiry is an integral part of the evaluation process. Evaluation, inquiry and knowledge building processes work together to foster the use of relevant information at student, classroom, teacher and school levels to promote improvement. (ERO, 2015a)
Evaluative thinking
Evaluating practices involves thinking about:
- what evidence will be useful to demonstrate success
- what range of objectives and targets make sense to determine progress
- what processes will build knowledge and develop practical uses for it.
At the core of evaluative thinking is a process of considering the evidence, in relation to the questions that prompted its collection, and engaging in careful inquiry and interpretation.
(Earl and Timperley, 2015)
Using evidence
School effectiveness is associated with effective use of evidence at student, teacher, classroom and school levels.
Leaders play a key role in this. Where leaders do not make evidence use a priority, teachers are less likely to use it (Louis et al, 2010, in Ontario Ministry of Education, 2013).
To become competent and confident at interpreting and using evidence, leaders need to:
- develop an inquiry habit of mind
- become data literate
- create a culture of inquiry in the school.
Earl and Katz (2003)
An inquiry habit of mind
Earl and Katz (2003) describe the inquiry habit of mind as:
- valuing deep understanding
- reserving judgement
- tolerating ambiguity
- taking a range of perspectives
- posing systematically more focused questions.
Data literacy
Data literacy is not just about numbers. It is a process of:
- standing back and deciding what you need to know and why
- collecting or locating the necessary data
- finding ways to link key data sources, ensuring that the data are worth considering
- being aware of the limitations of the data
- thinking about what the results mean
- systematically considering an issue from a range of perspectives so that you feel you have sufficient evidence to explain, support, and also challenge your point of view.
(Earl and Katz, 2006)
A culture of inquiry and evaluation
School leaders create an inquiry culture when they:
- make evidence use a priority
- establish the purpose(s) for collecting and using evidence
- use data in non-threatening ways
- provide planned opportunities and time for working with evidence
- engage staff in collaborative interpretation and decision making.
(Ontario Ministry of Education, 2013)
Developing school-wide inquiry
Wendy Kofoed shares key messages from her experience building inquiry approaches into school practices.
Making sense of evidence
While data provides lenses for looking at a situation, it doesn't generate questions, answers or solutions.
Leaders and teachers need to interpret what the data might be signalling in their context. They need to learn to uncover patterns, generate hypotheses and systematically test these in a range of ways with a range of audiences.
It is not an easy job and involving a range of voices and expertise helps avoids the cognitive biases that can limit how well people respond to new information.
Cognitive biases include:
- not thinking through all possibilities
- focusing on confirming existing hypotheses, not challenging them
- paying too much attention to things that are vivid
- considering the information to be an exception or an anomaly
- hesitating to take action in a new direction
- not wanting to expose vulnerabilities.
(Katz and Dack, 2013, in Earl and Timperley, 2015)
Collaborative sense making also involves drawing on research and using suitable frameworks or indicators to analyse data. It requires that we know what good looks like, so we can recognise our strengths and areas for improvement. (ERO, 2015b)
Related pages
Evidence-based leadership – articles and stories
Spiral of inquiry: Leaders leading learning – guide
Theory for improvement – tool
Tū Rangatira: Māori-medium educational leadership
This capability features strongly in the whenu (key role of leadership):
- He kaiako – the teacher and learner
Tū rangatira (English)
Tū rangatira (te reo Māori)
Evaluation resource hub
An online guide to evaluation for Australian schools. The hub includes this introductory video on evaluative thinking.
Evaluation resource hub – NSW Department of Education
References
Earl, L. Katz, S. (2003). Leading schools in a data-rich world.
Earl, L. Katz, S. (2006). Leading schools in a data-rich world: Harnessing data for school improvement. Thousand Oaks, CA: Corwin Press.
Earl, L. Timperley, H. (2015). Evaluative thinking for successful educational innovation. OECD working paper.
Education Review Office. (2016). School evaluation indicators. See pp 40-41 Domain 6: Evaluation, inquiry and knowledge building for improvement and innovation.
Education Review Office. (2015b). Effective school evaluation: How to do and use internal evaluation for improvement
Education Review Office. (2016). Effective internal evaluation for improvement
OECD. (2013). Synergies for better learning: An international perspective on evaluation and assessment
Ontario Ministry of Education. (2013). Using data: Transforming potential into practice. Ideas into Action series.
Image credit: Markus Spiske, Color, pattern, magnifier and purple, on Unsplash