Tag: methodology

Next stage: Analysing the data

OK, the interviews are in the can. 7 in all, and I’m really grateful to my colleaugues for participating.

I’ve taken detailed notes from each one, not quite full transcripts but enough for me to have an accurate, unvarnished written record of what was said. I’ve put timecodes at certain points so I know where to look in each audio file to get a direct quote should I need one.

I’m really pleased with the depth and variety of responses I got from my participants. Each interview was slightly different from the previous. This was down mainly to each person having a different set of experiences (so an equal emphasis on the questions wouldn’t have been appropriate) and that I let the conversations flow as much as I could. The wording of my questions was slightly different for each person as well but my initial take is that I managed to get comparable answers from everyone.

The next task is to identify the main themes. Guy’s advice was to start by creating a long list of possible themes and then progressively narrowing down to the definitive list.

I’m going to create a matrix with 5 main topic areas;

  • Nature of storytelling
  • Benefits within an organisation
  • Disbenefits
  • Team and personal development
  • Organisational conditions

…then cut and paste the relevant responses from each of the participants. I may even do this using paper, scissors and glue. There’s going to be a lot of info and a laptop screen may not be enough.

Straight out of the blocks, the thing which has interested me most about the responses was the lack of emphasis on technical skills and technologies. People mostly talked about storytelling and a great deal of that was about the organisational culture needing to be one that encourages openness. There’s an inherent vulnerability to being a storyteller and the conditions have to be right for that to these approaches to flourish it seems.

Interviews – so far, so good!

[Apologies for typos. I’m on a wobbly train so finger accuracy is suffering.]

I’m now 2 interviews into my data collection and it’s going really well.

Both have ended up being about 35-40 mins rather than the hour I’d anticipated  but even at that length they’re generating lots of useful and unexpected stuff that links very closely with the reading I’ve been doing.

I think longer interviews would be exhausting for the participants and it’s taking me about 90 mins to listen back to them and take notes. With 6 participants, that a lit of work even before I start analysing the data.

On a first listen there are some emerging themes that I’ll be able to use as slices but I won’t say what they are at this stage. Some of the participants read the blog and I don’t want to influence what they say in the interviews.

Guy had a great suggestion of “participant checking” which I’m going to do after I’ve done my initial analysis. It will mean probably getting all the participants together (probably using video conferencing) to discuss whether the themes I’m identifying actually reflect what they see as being accurate.

As for conducting the interviews, I hope I’m being a good researcher. I’m trying to use all the skills I developed as a coach at Siemens:

  • using open questions
  • reflecting back answers using the language the participant used
  • probing and asking for elaboration
  • leaving enough room for the participant to answer fully
  • maintaining eye contact and all that stuff
  • only taking the minimum of notes. It helps me to listen closely but means I can give the interviewee as much attention as possible.

I seem to be eliciting lots of extended, sophisticated responses so far so I’ll keep on with that approach.

I am detecting my greasy fingerprints in some of their answers; the outcomes of some of the sessions I’ve done, converstaions I’ve had and so on. I’ll need to make sure that I’ve written clearly about what my infuence on the answers might have been in my methodology section.

1 more interview this week then 3 next. I’m liking this feeling of momentum.

Case Study Methodology

I’ve been reading Yin’s Case Study Research (2003) to get some grounding in putting together my research. Here’s my initial take from it.

Yin lists the 5 main components of research design. This is how I think my work relates:

  1. A study’s questions – I’m happy that my research questions are relevant and useful and feedback from Guy is positive (note – need to post the revised proposal to the blog!)
  2. My propositions based on the questions – my initial direction is that storytelling as part of an  will have a positive qualitative impact on an organisation’s communictaion and internal working practices but the counter proposition is that any gains may be marginal or there will be no significant change in the state of affairs given the effort required to deveop skills and approaches. These propositions need sharpening up, they feel flabby. Yin says that there are valid reasons for not having valid propositions if the case study is more exploratory in nature
  3. The unit of analysis – bit trickier this one. The reseach is looking at Netskills (although there are some “dotted lines” to people in other organisations)but the study itself will mainly be focussing on the experiences of a subset of employees. Individual experiences are important to that but the case study is trying to look at the orgainisational level. It will be important to include key people not involved in the storytelling development activities  but who are stakeholders (e.g. the director, the training manager, “publicity officer”)
  4. The logic linking the data to the proposition – Again, problematic at this stage. I’m hoping that the interviews and observations will bring out some emergent themes; I certainly don’t want to dictate these, given my participant observer status. It’s hard to say at this stage what the logic will be for linking these data back to the propositions but I will need to make sure that my interview questions are closely linked to the research questions. 
  5. Criteria for interpreting findings – The data is going to be predominantly qualitative and narrative in nature (I’ll be asking for participants to describe their own journeys and add their own interpretations to events). As such there will be a significant subjective interpretation of the data and much emphasis on the participants’ opinions as to the value of the work they’ve undertaken. Another issue is timescales, which are not huge! In reality, the effects, positive or not, of this development work will be cumulatively visible over an extended period of months or years so this case study may just provide the basis for further research or form the template for case studies on other organisations

The role of theory

Yin quotes Sutton and Shaw (1995, p378) who describe a theory as “…a [hypothetical] story about why acts, events, structure and thoughts occur.” It’s nice to see the work story cropping up there but I’m not entirely sure what the difference between theory and proposition is.

I’ve tried to articulate my theory of story in this context to myself but I’m not coming up with anything that I find particularly satisfying yet. This is a question I want to take to Guy at our next meeting.

Perhaps the theory will emerge as a result of the research and trying to manufacture one now isn’t going to be helpful. Another thing I’ll need to bear in mind is how much I’ll be able to generalise from this cae study to a universal theory. I’ll have to be very caeful there.

Criteria for judging quality of reseach design

According to Yin there are 4 tests of quality

  1. Construct validity – can I demonstrate the data I’m collecting and measuring in the case study actually reflects the focus of the study. I’ll need to show this in my analysis but at this stage I think that looking for quantitative measures in things like social media activity (follows, mentions, retweets etc) or numbers of people booking on workshops for example wouldn’t reflect the focus of my study.
  2. Internal validity – If I’m establishing cause an effect between x and y am I able to show that it’s not something that has been caused by z? It will be important to look at the wider context at Netskills. There has been quite a lot of activity recently in team and personal development and there are also a number of staff taking on different roles in the team. This might have an impact on the study, as well as drivers coming from the changes that are happening to Jisc, from whom we receive a proportion of our funding.
  3. External validity – are the findings generalisable beyond the cae study. I anticipate that I’ll not be able to say definitively that what has happened in the Netskills context  will be directly applicable to other organisations. Although there are similar organisations to Netskills, even with Jisc, much of what we’ve been doing relates closely to individuals and the Netskills working culture. This isn’t to say it won’t be applicable, just that there will have to be clear and serious caveats written into the analysis.
  4. Reliability – would the results of the case study be replicable by another researcher following the same process? This links closely to point 3 but it’s a reminder to me to make sure that my approach is clearly documented and explained.

What type of case is Netskills’ storytelling development?

Going by Yin’s terminology I think that the Netskills case is representative, rather than a unique or revelatory case, in that I’m trying to establish what lessons could be learned for similar organisations. Given a longer time frame of years a longitudinal approach might yield more reliable results so again, this could form the basis for further study.

Lastly…

I need to be aware of Yin’s advice that multiple case studies are preferred over single as it means replication is possible. I’m not going to have that opportunity so that will necessarily colour my findings.

I also need to keep the case study design open and be willing to change the design as the study progresses.

Gathering data – some thoughts

I experimented on Will before Christmas by trialling my first stage interview questions on him. I felt it went well and Will’s responses will be very valuable when seen alongside my other research subjects.

I don’t want to post too much about the interviews here in case I influence the other participants.

These interviews won’t generate sufficient data by themsleves and I’m struggling to think how best to capture other aspects. This is one of my main aras of concern to take to Guy next time I speak to him.

How do I fit in? I’m part of the case study as well but do I need to answer my own questions? Does that risk me skewing the data? I’ve undertaken quite a lot of work in the form of presentations, workshops, blog posts and so on but I’m not sure how to represent that in the case study.

All being well, over the course of the study we will have produced our own digital stories. I want to include these but will also need to capture reflection from the people that have produced them as well as the rest of the peer group.

For the 2nd stage interviews I was considering doing a workshop to try and capture individual experiences but also some of the group dialogue. We have been learing about participatory approaches to workshops recently and it would be good to put some fo those into practice. This will allow participants to visualise a lot of their experiences, mapping their journeys. This will have to be done carefully to make sure that sufficient data is captured on the day (I’ll need to employ an observer/note taker) as well as giving participants a chance to explain and interpret in their own words what they have come up with.

The more I think about it the more nervous I get that time is ticking away. I want to get my initial round of interviews wrapped up in the next 2 weeks or so so I can start transcribing and coding the data and then planning the next stages.

Methodological tangle: phenomenology, ethnomethodology or symbolic interactionism?

Russian_dolls_zeta_

I’ve been reading up on different methodologies tonight and have been getting myself confused.

As I said in the last post I’m placing this research firmly in the “subjectivist” camp. Tonight I’ve been looking at what Cohen and Mannion (2007) say about naturalistic approaches to research (pp 19-26).

I want this research to be an immersive process, where I try to share the “frame of reference” of the people and groups I’m studying. The fact that I’m a participant in the field of project support means that trying to take a removed, objective perspective isn’t realistic. I’m trying to understand the situation from within. 

Cohen et al focus on 3 traditions within this naturalistic approach; Phenomenology, Ethnomethodology and Symbolic Interactionism.

At the moment I’m still trying to tell them apart. Here’s a summary of what I understand of each:

Phenomenology

The most important thing is the “subjective conscienciousness”, where and individual’s consciousness creates meaning based on experiences. It concerns understanding of the underlying meaning of “things”, stripping away assumtions based on how we perceive the world, influenced by cultural norms.

Ethnomethodology

The study of everyday activities, particularly social interactions. The focus is on how the sense that individuals have made of the world influences behaviours. The most important thing is how the individual perceives their reality. Research relies heavily on immersive fieldwork

Symbolic Interactionism

This is not something I’ve come across before so it’s the one I’m having most trouble understanding (It’s all relative, I’m not sure I understand the other two either!). SI seems to look more at how people’s actions are aligned to perceptions of how others might act but like the previous two is also concerned with the meanings that individuals ascribe to things. SI sees this process as something which is constantly in a state of flux.

Where do this leave me?

I’m not sure I’m able to draw clear lines between these 3 approaches. They all seem to hand on the idea of how meanings are constructed. This is especially relevant given my topic area. I said in a previous post that one of the functions of story is to help us ascribe meaning to experience so a methodology based on these naturalistic  approaches seems appropriate.

But even with this I’m getting myself tangled up. This dissertation is a story on the topic of the usefulness of story that relies on finding out the stories of the people that are involved in the process of telling stories. 

All I can think of is Russian dolls!

The fact is that I still want to want to take an ethnographic approach which will help me discover what it is like for teams undertaking these funded projects with a view to seeing how storytelling impacts on them. But I can also see that this idea of interactionism might be helpful. The teams exist and act in a context where they are trying secure funding, now and possibly in the future, from JISC so relations with programme managers will be important. They work for institutions that will be placing their own demands on them. And then there’s me, a proponent of storytelling who probably has a few ingrained assumptions about the value of storytelling and works for a company who has a strategic relationship with JISC.Innovation who is funding the projects.

I’m not sure I’m any the wiser!

Methodological Assumptions

Some initial thoughts about my research standpoint, based on Burrell and Morgan’s scheme for analysing assumptions about the nature of social science (1979, cited in Cohen et al (2007, p9))…

I don’t view the answers to my research questions existing in an objective form, independent of the actors involved. I’m dealing with story, something that is intrinsically subjective and human. The effectiveness of a story comes from its affectiveness; the emotional reponse that is created through narrative. Someone will respond to a story in one of a myriad ways, influenced by all sorts of psychological and sociological factors that are impossible to untangle.

There might be empirical studies into the physiological response to narrative but I don’t anticipate that they would be helpful in this context. Certainly the literature I’ve been reading doesn’t dwell on the scientific side of things.

So I take a strongly nominalist viewpoint.

As I’m assuming that knowledge is something “personal, subjective and unique” (Cohen et al, 2007 p7). To uncover people’s responses to stories and their attitudes towards them will mean me getting directly involved with the subjects. I won’t be able to gather data remotely or in a  way that avoids participation in some way. Exactly how, I’m not sure yet.

This is an anti-positivist approach.

People react differently to stories, as mentioned earlier. I’m working off the assumption that there is an unpredictable element to the use of story, no magic formula that will guarantee an automatic response in the audience. This doesn’t mean that you can’t make choices that influence how someone responds. Stories are cultural artefacts so involve aspects that will be recognisable to both creator and listener. The guidance I hope to provide in the project that this dissertation is linked to will be based on consciously using those aspects of story to achieve the desired effects.

My understanding of human nature is Voluntarist.

Lastly, this seems to call for an idiographic methodology. There will be a strong qualitative aspect to the research. My feeling is that although there may be some quantitative metrics that I could look at I would have to make some pretty large assumptions about a scientific basis for them. To give a simplistic example, I could look at the number of views a digital story posted on YouTube receives but I’m not sure I could use this to say there is a direct causal link between the way the elements of the story were compiled and its popularity. 

Notes of Caution

I don’t want to completely dismiss any notion of quantitative research method based on these assumptions. It may be that there will be useful quantitative data that I can use to help tell my story, I’m just not quite at the stage of deciding what that might look like yet.

Another warning bell going off is that it’s nice to have some big words to attach to my methodological approach, that doesn’t necessarily make them right. It feels right but that might be my own broader worldview and personal style at play. I need to keep this under scrutiny and reassess as I go. I’ve not addressed them critically yet. That’s my next step.

My colleague Will has been encouraging me to see this dissertation as telling a story. In this post I’ve talked about how project teams will be telling stories, but I’m part of that process too. This story will be “personal, subjective and unique” and that’s not something that I want to shy away from. But as I was saying to Will this evening, I don’t want to use that as an excuse for producing a piece of work that is ill-defined and full of woolly thinking!

Next steps

Read on to continue to critically examine these aspects of methodology and choose methods that will help me with the research.

Cohen L, Manion L & Morrson, K (2007) Research Methods in Education (6th edition), Routledge, Oxford.