The Impact of the Research
I intend to start by exploring what impact might mean in the context of my research.
Evidence of impact implies change not positive value, it is possible that my research could have had impact without that being an improvement or implying progress.
Impact is not synonymous with effectiveness. My research could be effective in answering my research questions but fail to have significant impact if I or others saw no value in those answers.
Impact may be time sensitive. It is possible that this examination of impact may fail to capture longer term impacts. It is also possible that initial enthusiasm may wane and impact may fade over time.
It is difficult to ascribe simple causal relationships to actions in complex situations so it may be difficult to be sure of the impact
Impact may occur at different levels. There could be deep impact on my own practice but there may also be shallower impact beyond this on the practice of others and this may be hard to capture.
Impact may not only be due to my research actions and findings but due the the very act of conducting the research. This may be attributable to the Hawthorne effect(1) or to the influence of working in proximity to someone engaging in reflective practice (McNiff 2002(2)).
Impact studies are often undertaken by organisations to evaluate the effectiveness of implementing some organisational change. I have had experience of this wider sort of impact study in my job role as a research assistant to a network learning community.
However I am not an organisation and my research isn’t part of some change within an organisation. Although it might be said to have effects beyond those on me, these are essentially largely hidden. I have instead the shadows of possible effects. I have what people say they might do, (the survey results) little evidence of anything they’ve actually done.
There are things people have actually done and that I can be sure have been influenced by my actions. These might even be measurable effects. I am unconvinced that my qualitative data provides measurable effects in this distributed environment. I have to rely on ‘hits and contributions’ (Terrell, 2006, unpublished (3)) for measurable effects.
I have a small amount of evidence of people creating links to the blog and being positive about it. Some of these people have quite high status as innovators or policy influencers (Steven Downes, the TES, NACE, – for example). Eve Thirkle suggested that I am too ‘tentative’ about claiming impact for these links. She was impressed by them as ‘experts’ but I see the impact of their linking to the blog as being fairly low.
More directly influential, in my opinion, are sites popular with teachers (Primary Resources, Teachingideas.co.uk, Sparklebox.co.uk, Discovery Educators Network,) linking to it.
I know that this has influenced the numbers of people consulting the site as about 40% of my visitors now come from these links rather than google searches.
Then there are the people using the site, 600-800 (April 2006) hits a day. They tend to view multiple entries, often using the categories to explore. They stay for as much as 10 minutes (this is a lot for a web site!). The most common exit point is for them then to go the the Classroom Displays Group on Flickr. Visitors come from all over the world:
(Note – I have now got much more detailed stats as I was recently able to install Google Analytics on the site)
Other quantitative data
I know that:
2 TAs joined the TA Forum because they saw it in my research.
1 NQT has asked me for permission to point to one of her displays on the blog in her CV.
So these are the measurable impact.
There is more impact that isn’t easily measured.
Impact on me
“It’s made me look beyond the role I have in school. It’s made feel, at times, that I can take control of things, that I can make good things happen.”
(from a posting to the learning set)
Looking at the impact the research has had on myself was quite challenging. My initial response was very negative. I wrote in the learning set about exhaustion and confusion.
I began to think that I needed to approach this from a different angle, that I needed to use a reflection technique to help me access the deeper impact.
I wrote The woman who made quilts as a reflective blog post and this forms part of my impact study.
The story made sense of my choice of topic for my final year action research in a way that took me by surprise. By rediscovering my voice, or at least one of them, I felt energised and able to once more make progress. The feelings of being exhausted and overwhelmed by the task of defending and validating my research began to fade.
I needed to dig deeper, to draw out some of the learning from the reflection. I needed to do this gently though, I had no desire to rip apart my carefully crafted story.
In my on-line research journal I started by exploring the story of the making of the reflection.
So there it is – the story of the woman who made quilts. My authentic voice. In it I hear the echoes of Angela Carter and LeGuin, my heroines of old. Its not great Art. Its like the quilts, simple and effective, it will keep me warm on cold nights and protect my dreams
(From Acting to Improve,)
Two things became clear to me from this:
that the need to be creative is fundamental to me and making the Classroom Displays blog goes some way towards satisfying that need.
the actions of making and improving the Classroom Displays blog move me in the direction of my values as suggested as a criteria for judging success by McNiff (2002(2)).
Impact on Others
Wider impact in school and other on colleagues
I’m widening this beyond the context of my school and defining my workplace as the on-line environment in which the resource I provide sits. My colleagues in this context are also those people I meet on-line and who attended my on-line exhibition. The data was collected from an on-line exit questionnaire, blog comments on the entry page to the exhibition and interviews with two colleagues.
Using the analysis technique of emergent coding and grounded theory on my impact data from the exhibition I identified 4 categories of impact. There were:
The respondents said the research fulfilled a perceived need
(In 17 instances respondents mentioned a need for such a resource and/or research into this topic.)
The respondents mentioned that the research offered opportunities for improved outcomes for staff or children.
The respondents said the research showed the importance of sharing expertise
The respondents suggested the research had educated them about web 2.0 services and/or promoted their use of such services.
In a posting to the learning setI discussed my qualms about handling the data in this way. My worries were that it was mechanistic and too quantitative an approach. Discussing it further with my peer-review partner I concluded that it did have value although it did not tell the whole story.
Conclusions on impact
I believe that a small first person action research study such as mine can only make small claims of impact. The main impact has been on myself and on my own practice. This impact does not easily lend itself to measurement, these are subtle shifts in my attitudes and thinking. There is some potential impact on my professional development in that the research moves me nearer to needs identified in my mission statement in my personal development plan at the end of year 1.
I want to be able to be financially independant doing a job which allows me to use my creativity and have some control over my time. I want to be able to provide a service which encourages and enables people to express their own creativity. I want to move towards spending more of my time on creative activities. I want to use my display skills.I want to have the opportunity to work on the skills that I am improving at – in particular digital tools and computing. These skills are all transferable and may mean that I can move into environments other than schools.
(Personal Mission Statement – Hartley 2004)
I cannot easily assume cause and event and attribute impact much beyond this.
Some impact on others can be measured but it is mostly quantitative data that provides this. Again though even here these are small claims of impact, tiny shifts in people’s actions that can be observed by the mechanistic process of counting visitor numbers or links. The measuring of wider impact on others is harder to justify.
The data I collected after the exhibition shows some possible impact. Some of the people who viewed it may be more aware of web 2.0 because of the research. Some may be more willing to see the internet as a useful place for collaborative working as a result of the research. To measure these outcomes I would have to be able to see if this really did effect their actions and in the on-line context this is not possible. Even with those with whom I share a face to face context I cannot actually measure shifts in their on-line behaviour.
People who comment on the blog or join the group – these subtle shifts and nuances of behaviour, show the ‘evolution’ of relationships as a result of the changes in my practice.
“Remember that you are not trying to demonstrate a cause and effect relationship between you and other peoples actions. You are not saying, I brought about improvement or I made that happen. You are saying, I can show that certain changes took place as I changed my practice, particularly in myself, and different relationships evolved.”
(1) Franke, R.H. & Kaul, J.D. “The Hawthorne experiments: First statistical interpretation.” American Sociological Review, 1978, 43, 623-643. cited in Clarke D, (1999),Performance,Learning, Leadership and Knowledge Url: http://www.nwlink.com/~donclark/hrd/history/history.html Available. Last accessed 20/4/06
(2)McNiff , J.( 2002), Action research for professional development, Concise advice for new action researchers. Available,
URL: http://jeanmcniff.com/booklet1.html Last accessed 24/4/06
(3) Terrell, I. (2006) Ultraversity Hotseat, unpublished