Evaluating practices in relation to outcomes

Leaders are skilled at evaluating the organisation’s collective and individual staff practices in relation to learning outcomes and wellbeing.

They use high levels of quantitative and qualitative data literacy. They are curious about patterns and practices.

They can describe and identify problems or challenges in ways that open up real discussion and identification of needs, and solutions.

Educational leadership capability framework, Education Council, 2018

This page summarises some thinking and research related to this capability. It is not a comprehensive or definitive background to the capability.

Evaluating practices in relation to outcomes underpins other leadership capabilities, for example, strategic thinking and planning, and adept management of resources.

Assessment, evaluation and inquiry 

    Systems that perform better are systems in which assessment, inquiry and evaluation information is used effectively to inform learning. 
    (OECD, 2013)

In New Zealand, the Education Review Office's School evaluation indicators include Domain 6: Evaluation, inquiry and knowledge building for improvement and innovation – see pp 40-41.

Evaluation means making a judgment about the quality, effectiveness or value of a policy, programme or practice in terms of its contribution to the desired outcomes. (ERO, 2015a)

Professional inquiry is an integral part of the evaluation process. Evaluation, inquiry and knowledge building processes work together to foster the use of relevant information at student, classroom, teacher and school levels to promote improvement. (ERO, 2015a)

Evaluative thinking

Evaluating practices involves thinking about:

  • what evidence will be useful to demonstrate success
  • what range of objectives and targets make sense to determine progress
  • what processes will build knowledge and develop practical uses for it.

At the core of evaluative thinking is a process of considering the evidence, in relation to the questions that prompted its collection, and engaging in careful inquiry and interpretation. 

(Earl and Timperley, 2015) 

Using evidence

    School effectiveness is associated with effective use of evidence at student, teacher, classroom and school levels.

Leaders play a key role in this. Where leaders do not make evidence use a priority, teachers are less likely to use it (Louis et al, 2010, in Ontario Ministry of Education, 2013).

To become competent and confident at interpreting and using evidence, leaders need to:

  • develop an inquiry habit of mind
  • become data literate
  • create a culture of inquiry in the school.

Earl and Katz (2003) 

An inquiry habit of mind

Earl and Katz (2003) describe the inquiry habit of mind as:

  • valuing deep understanding
  • reserving judgement 
  • tolerating ambiguity
  • taking a range of perspectives
  • posing systematically more focused questions.

Data literacy

Data literacy is not just about numbers. It is a process of:

  • standing back and deciding what you need to know and why
  • collecting or locating the necessary data
  • finding ways to link key data sources, ensuring that the data are worth considering
  • being aware of the limitations of the data
  • thinking about what the results mean
  • systematically considering an issue from a range of perspectives so that you feel you have sufficient evidence to explain, support, and also challenge your point of view.

(Earl and Katz, 2006)

A culture of inquiry and evaluation

School leaders create an inquiry culture when they:

  • make evidence use a priority
  • establish the purpose(s) for collecting and using evidence
  • use data in non-threatening ways
  • provide planned opportunities and time for working with evidence
  • engage staff in collaborative interpretation and decision making. 

(Ontario Ministry of Education, 2013)

Developing school-wide inquiry

Wendy Kofoed shares key messages from her experience building inquiry approaches into school practices.

Transcript

Making sense of evidence

    While data provides lenses for looking at a situation, it doesn't generate questions, answers or solutions.

Leaders and teachers need to interpret what the data might be signalling in their context. They need to learn to uncover patterns, generate hypotheses and systematically test these in a range of ways with a range of audiences.

It is not an easy job and involving a range of voices and expertise helps avoids the cognitive biases that can limit how well people respond to new information.

Cognitive biases include:

  • not thinking through all possibilities
  • focusing on confirming existing hypotheses, not challenging them
  • paying too much attention to things that are vivid
  • considering the information to be an exception or an anomaly
  • hesitating to take action in a new direction
  • not wanting to expose vulnerabilities.

(Katz and Dack, 2013, in Earl and Timperley, 2015) 

Collaborative sense making also involves drawing on research and using suitable frameworks or indicators to analyse data. It requires that we know what good looks like, so we can recognise our strengths and areas for improvement. (ERO, 2015b)

Related pages

Evidence-based leadership – articles and stories

Spiral of inquiry: Leaders leading learning – guide

Theory for improvement – tool

Tū Rangatira: Māori-medium educational leadership

This capability features strongly in the whenu (key role of leadership):

  • He kaiako – the teacher and learner

Tū rangatira (English)

Tū rangatira (te reo Māori)

Evaluation resource hub

An online guide to evaluation for Australian schools. The hub includes this introductory video on evaluative thinking. 

Evaluation resource hub – NSW Department of Education

References

Earl, L. Katz, S. (2003). Leading schools in a data-rich world

Earl, L. Katz, S. (2006). Leading schools in a data-rich world: Harnessing data for school improvement. Thousand Oaks, CA: Corwin Press.

Earl, L. Timperley, H. (2015). Evaluative thinking for successful educational innovation. OECD working paper.

Education Review Office. (2016). School evaluation indicators. See pp 40-41 Domain 6: Evaluation, inquiry and knowledge building for improvement and innovation.

Education Review Office. (2015b). Effective school evaluation: How to do and use internal evaluation for improvement

Education Review Office. (2016). Effective internal evaluation for improvement 

OECD. (2013). Synergies for better learning: An international perspective on evaluation and assessment

Ontario Ministry of Education. (2013). Using data: Transforming potential into practice. Ideas into Action series.

Image credit: Markus Spiske, Color, pattern, magnifier and purple, on Unsplash

Tell a colleague | Back to top