How to write Quantitative Research Paper

how to critique quantitative research methodology and how to use quantitative research methods and how to write quantitative research proposal
Dr.JohnParker Profile Pic
Published Date:01-07-2017
Your Website URL(Optional)
DANIEL MUIJS DOING quantitative research IN EDUCATION9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 1 ■ ■ ■ Chapter 1 Introduction to quantitative research ■ ■ ■ What is quantitative research? Research methods in education (and the other social sciences) are often divided into two main types: quantitative and qualitative methods. This book will discuss one of these two main strands: quantitative methods. In this chapter we will have a look at what is meant by the term quantitative methods, and what distinguishes quantitative from qualitative methods. When you think of quantitative methods, you will probably have spe- cific things in mind. You will probably be thinking of statistics, numbers – many of you may be feeling somewhat apprehensive because you think quantitative methods are difficult. Apart from the last, all these thoughts capture some of the essence of quantitative methods. The following definition, taken from Aliaga and Gunderson (2002), describes what we mean by quantitative research methods very well: Quantitative research is ‘Explaining phenomena by collecting numeri- cal data that are analysed using mathematically based methods (in particular statistics).’ Let’s go through this definition step by step. The first element is explain- ing phenomena. This is a key element of all research, be it quantitative or qualitative. When we set out do some research, we are always looking to explain something. In education this could be questions like ‘why do teachers leave teaching?’, ‘what factors influence pupil achievement?’ and so on. The specificity of quantitative research lies in the next part of the defini- tion. In quantitative research we collect numerical data. This is closely connected to the final part of the definition: analysis using mathematically 19079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 2 2 ■ Doing Quantitative Research in Education based methods. In order to be able to use mathematically based methods our data have to be in numerical form. This is not the case for qualitative research. Qualitative data are not necessarily or usually numerical, and therefore cannot be analysed using statistics. Therefore, because quantitative research is essentially about collecting numerical data to explain a particular phenomenon, particular questions seem immediately suited to being answered using quantitative methods: ■ How many males get a first-class degree at university compared to females? ■ What percentage of teachers and school leaders belong to ethnic minority groups? ■ Has pupil achievement in English improved in our school district over time? These are all questions we can look at quantitatively, as the data we need to collect are already available to us in numerical form. However, does this not severely limit the usefulness of quantitative research? There are many phenomena we might want to look at, but which don’t seem to produce any quantitative data. In fact, relatively few phenomena in edu- cation actually occur in the form of ‘naturally’ quantitative data. Luckily, we are far less limited than might appear from the above. Many data that do not naturally appear in quantitative form can be col- lected in a quantitative way. We do this by designing research instruments aimed specifically at converting phenomena that don’t nat- urally exist in quantitative form into quantitative data, which we can analyse statistically. Examples of this are attitudes and beliefs. We might want to collect data on pupils’ attitudes to their school and their teach- ers. These attitudes obviously do not naturally exist in quantitative form (we don’t form our attitudes in the shape of numerical scales). Yet we can develop a questionnaire that asks pupils to rate a number of state- ments (for example, ‘I think school is boring’) as either agree strongly, agree, disagree or disagree strongly, and give the answers a number (e.g. 1 for disagree strongly, 4 for agree strongly). Now we have quantitative data on pupil attitudes to school. In the same way, we can collect data on a wide number of phenomena, and make them quantitative through data collection instruments like questionnaires or tests. In the next three chapters we will look at how we can develop instruments to do just that.9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 3 Introduction to quantitative research ■ 3 The number of phenomena we can study in this way is almost unlim- ited, making quantitative research quite flexible. However, not all phenomena are best studied using quantitative methods. As we will see, while quantitative methods have some notable advantages, they also have disadvantages, which means that some phenomena are better stud- ied using different (qualitative) methods. The last part of the definition refers to the use of mathematically based methods, in particular statistics, to analyse the data. This is what people usually think about when they think of quantitative research, and is often seen as the most important part of quantitative studies. This is a bit of a misconception. While it is important to use the right data analysis tools, it is even more important to use the right research design and data collection instruments. However, the use of statistics to analyse the data is the element that puts a lot of people off doing quantitative research, because the mathematics underlying the methods seem complicated and frightening. Nevertheless, as we will see later on in this book, most researchers do not really have to be particularly expert in the mathemat- ics underlying the methods, because computer software allows us to do the analyses quickly and (relatively) easily. ■ ■ ■ Foundations of quantitative research methods Realism, subjectivism and the ‘paradigm wars’ Now we have defined quantitative research, let’s compare it with qualita- tive research, against which it is usually contrasted. While quantitative research is based on numerical data analysed statistically, qualitative research uses non-numerical data. Qualitative research is actually an umbrella term encompassing a wide range of methods, such as inter- views, case studies, ethnographic research and discourse analysis, to name just a few examples. The difference between quantitative and qualitative research is often seen as quite fundamental, leading people to talk about ‘paradigm wars’ in which quantitative and qualitative research are seen as belligerent and incompatible factions (a bit like capitalism and communism). Many researchers define themselves as either quantitative or qualitative. Where does this idea come from?9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 4 4 ■ Doing Quantitative Research in Education This idea is linked to what are seen as the different underlying philosophies and worldviews of researchers in the two ‘paradigms’ (also called ‘epistemologies’). According to this view, two fundamentally dif- ferent worldviews underlie quantitative and qualitative research. The quantitative view is described as being ‘realist’ or sometimes ‘posi- tivist’, while the worldview underlying qualitative research is viewed as being ‘subjectivist’. What does this mean? Realists take the view that what research does is uncover an existing reality. ‘The truth is out there’ and it is the job of the researcher to use objective research methods to uncover that truth. This means that the researcher needs to be as detached from the research as possible, and use methods that maximise objectivity and minimise the involvement of the researcher in the research. This is best done using methods taken largely from the natural sciences (e.g. biology, physics, etc.), which are then transposed to social research settings (like educa- tion). Positivism is the most extreme form of this worldview. According to positivism, the world works according to fixed laws of cause and effect. Scientific thinking is used to test theories about these laws, and either reject or provisionally accept them. In this way, we will finally get to understand the truth about how the world works. By developing reliable measurement instruments, we can objectively study the physical world. However, this view, that there is a true reality out there that we can meas- ure completely objectively, is problematic. We are all part of the world we are observing, and cannot completely detach ourselves from what we are researching. Historical research has shown that what is studied and what findings are produced are influenced by the beliefs of the people doing the research and the political/social climate at the time the research is done. If one looks at research from a quantitative versus qualitative perspec- tive, qualitative researchers are subjectivists. In contrast to the realist view that the truth is out there and can be objectively measured and found through research, subjectivists point to the role of human subjectivity in the process of research. Reality is not ‘out there’ to be objectively and dis- passionately observed by us, but is at least in part constructed by us and by our observations. There is no pre-existing objective reality that can be observed. The process of our observing reality changes and transforms it, and therefore subjectivists are relativistic. All truth can only be relative and is never definitive as the positivists claim. The extreme relativist posi- tion is obviously as problematic as the extreme positivistic one, because,9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 5 Introduction to quantitative research ■ 5 for example, it would in theory deny that anything more than social con- sensus and power distinguishes witchcraft and modern science. If you look at the extreme forms of the two views we have set out here, it would seem that quantitative and qualitative research methods are pretty incompatible. These extremes are, however, a gross simplification of the views of both quantitative and qualitative researchers, and very few people in either ‘camp’ subscribe to them. I have included them here because they are frequently presented in only slightly less extreme forms as straw men with which critics of one method (qualitative for example) may attack users of different methods (for example quantitative). Qualitative methods is an umbrella term for a large number of different research methods (such as participant observation, interviews, case stud- ies, ethnographic research) which are quite different. They are used by researchers with quite different worldviews, some of which clearly lie towards the realistic end of the spectrum. To ascribe radical subjectivist views to all qualitative researchers is a fallacy. To label all quantitative researchers positivists is equally inaccurate. Quantitative researchers have taken up many criticisms of positivist views, and there are now a variety of epistemologies underlying theory and practice in quantitative research. I think it is true to say that very few quantitative researchers nowadays are radical positivists. Post-positivism, experiential realism and pragmatism Post-positivists accept the critique of traditional positivism that has been presented by the subjectivists, without going so far as to reject any notion of realism. Post-positivists accept that we cannot observe the world we are part of as totally objective and disinterested outsiders, and accept that the natural sciences do not provide the model for all social research. However, they do believe in the possibility of an objective real- ity. While we will never be able to totally uncover that reality through our research, post-positivists believe that we should try and approximate that reality as best we can, all the while realising that our own subjectiv- ity is shaping that reality. Rather than finding the truth, the post-positivist will try and represent reality as best he or she can. In contrast to positivists, post-positivists believe that research can never be certain. Rather than focusing on certainty and absolute truth,9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 6 6 ■ Doing Quantitative Research in Education post-positivist social science focuses on confidence – how much can we rely on our findings? how well do they predict certain outcomes? A second worldview or epistemology that underlies the work of some quantitative researchers is called experiential realism. Experiential realism claims, as do anti-positivist positions, that we cannot observe the world in a purely objective way, because our perception itself influences what we see and measure. In contrast to subjectivist positions, however, expe- riential realists believe that there is a limit to subjectivity. Humans are limited in their subjectivity by the fact that we use a limited number of schemas to formulate our views of the world. This is because our percep- tion is ‘embodied’. We don’t observe passively, but actively interact with the world through our bodies. Experiential realists see the use of metaphor as crucial to the way we make sense of the world around us. We use metaphors to understand our world. One of the main metaphors we use to do this is the subject/object schema, which divides the world up into objects (things) and subjects (people). This metaphor has its origins in the fact that in our dealings with the world we find that there is a distinction between an external world consisting of edges, surfaces and textures that are not us, and those things that are us, the actor. As we move around our world, the objects remain invariant. Science, according to this view, is an activity that is based on this subject/object schema (Mulaik, 1995). A lot of researchers, both quantitative and qualitative (the author included), take a pragmatist approach to research, using different methods depending on the research question they are trying to answer. In some cases this will lead them to quantitative research, for example when they need to give a quantitative answer to a question or generalise findings to a popula- tion, or are looking to test a theory mathematically; in other cases they will employ qualitative methods. Sometimes a mixed methods approach com- bining quantitative and qualitative methods will be the most appropriate. Philosophers like Peirce, Dewey and James developed pragmatism as a philosophy in the USA. One of the main contentions of this school of philosophy is that the meaning and the truth of any idea is a function of its practical outcome(s). Pragmatists strongly oppose the absolutism they see as a key part of most other philosophical beliefs, and put themselves in opposition to other philosophies (think of the positivist/subjectivist debate) which are totally rejected. 9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 7 Introduction to quantitative research ■ 7 As for the subjectivists, there is no definite truth in pragmatic philoso- phy. Truth is constantly changing and being updated through the process of human problem-solving. The key question for pragmatists is not ‘is it true?’ or ‘is it right?’ but ‘does it work?’ ■ ■ ■ When do we use quantitative methods? If we take a pragmatic approach to research methods, first of all we need to find out what kinds of questions are best answered using quantitative as opposed to qualitative methods. There are four main types of research question that quantitative research is particularly suited to find an answer to: 1. The first is when we want a quantitative answer. Examples are: ‘How many students choose to study education?’ or ‘How many mathe- matics teachers do we need and how many have we got in our school district?’ That we need to use quantitative research to answer this kind of question is obvious. Qualitative, non-numerical methods will obviously not provide us with the (numerical) answer we want. 2. Numerical change can likewise only accurately be studied using quantitative methods. Are the numbers of students in our univer- sity rising or falling? Is achievement going up or down? We would need to do a quantitative study to find out. 3. As well as wanting to find out about the state of something, we often want to explain phenomena. What factors predict the recruit- ment of mathematics teachers? What factors are related to changes in student achievement over time? As we will see later in this book, this kind of question can also be studied successfully using quanti- tative methods, and many statistical techniques have been developed that allow us to predict scores on one factor or variable (e.g. teacher recruitment) from scores on one or more other factors or variables (e.g. unemployment rates, pay, conditions). 4. The final activity for which quantitative research is especially suited is the testing of hypotheses. We might want to explain some- thing, for example whether there is a relationship between a pupil’s achievement and their self-esteem and social background.9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 8 8 ■ Doing Quantitative Research in Education We could look at the theory and come up with the hypothesis that lower social class background leads to low self-esteem, which would in turn be related to low achievement. Using quantitative research we can try and test this kind of model. ■ ■ ■ Units and variables When we collect data in quantitative educational research, we have to collect them from someone or something. The people or things (e.g. schools) we collect data on or from are known as units or cases. The data that we are collecting from these units are known as variables. Variables are any characteristic of the unit we are interested in and want to collect (e.g. gender, age, self-esteem). The name variable refers to the fact that this data will differ between units. For example, achievement will differ between pupils and schools, gender will differ between pupils, and so on. If there are no differences at all between units we want to study we probably aren’t going to be able to do any interesting research (for example, studying whether pupils are human would not yield interesting findings). ■ ■ ■ What is a hypothesis? A hypothesis is a tentative explanation that accounts for a set of facts and can be tested by further investigation. For example, one hypothesis we might want to test could be that poverty causes low achievement, or that there is a relationship between pupils’ self-esteem and the amount of time they spend watching television. Quantitative researchers will design studies that allow us to test these hypotheses. We will collect the relevant data (for example, parental income and school achievement) and use statistical techniques to decide whether or not to reject or provisionally accept the hypothesis. Accepting a hypothesis is always provisional, as new data may emerge that causes it to be rejected later on. 9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 9 Introduction to quantitative research ■ 9 The types of problem outlined in 1 and 2 opposite are called ‘descriptive’ – we are merely trying to describe a situation – while those in 3 and 4 are ‘inferential’ – we are trying to explain something rather than just describe it. As mentioned above, while quantitative methods are good at answer- ing these four types of questions, there are other types of question that are not well suited to quantitative methods: 1. The first situation where quantitative research will fail is when we want to explore a problem in depth. Quantitative research is good at providing information in breadth from a large number of units. But when we want to explore a problem or concept in depth quan- titative methods are too shallow. To really get under the skin of a phenomenon, we will need to go for ethnographic methods, inter- views, in-depth case studies and other qualitative techniques. 2. We saw above that quantitative research is well-suited for the test- ing of theories and hypotheses. What quantitative methods cannot do very well is develop hypotheses and theories. The hypotheses to be tested may come from a review of the literature or theory, but can also be developed using exploratory qualitative research. 3. If issues to be studied are particularly complex, an in-depth quali- tative study (a case study, for example) is more likely to pick up on this than a quantitative study. This is partly because there is a limit to how many variables can be looked at in any one quantitative study, and partly because in quantitative research it is the researcher who defines the variables to be studied. In qualitative research unexpected variables may emerge. 4. Finally, while quantitative methods are better at looking at cause and effect (causality, as it is known), qualitative methods are more suited to looking at the meaning of particular events or circumstances. What then do we do if we want to look at both breadth and depth, or at both causality and meaning? In these situations, it is best to use a so- called mixed methods design in which we use both quantitative (for example, a questionnaire) and qualitative (for example, a number of case studies) methods. Mixed methods research is a flexible approach where the research design is determined by what we want to find out rather than by any predetermined epistemological position. In mixed methods research, qualitative or quantitative components can predominate or both can have equal status. 9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 10 10 ■ Doing Quantitative Research in Education ■ ■ ■ Common misconceptions 1. I have to have an epistemology to do research, don’t I? No, not necessarily. While you may have strong epistemological and philosophical beliefs that determine what kind of research you want to do, you can also start out wanting to solve a particular problem or wanting to find out about a par- ticular phenomenon. In such a situation you will be able to pragmatically choose what methods are best suited to solving your research question. 2. Data has to be in a quantitative format to do quantitative research, doesn’t it? Not necessarily. If data are not naturally available as num- bers, you can try and turn non-quantitative data (like attitudes or opinions) into quantitative data by measuring them numerically (for example, by using a questionnaire rating scale). 3. Qualitative and quantitative research are incompatible, aren’t they? Not necessarily. Qualitative and quantitative research can be usefully com- bined in mixed methods designs, which often produce a lot of useful information. Also, depending on your research question, you might in one instance want to use quantitative and in another instance qualitative research. This is something I personally often do. 4. The most important thing about quantitative research is the statistics, isn’t it? Not at all. While the way in which you analyse your data mat- ters, if you haven’t designed your research well and collected the data in a valid and reliable way, you will not get valid results however sophisticated your analyses. 5. Qualitative research is purely subjective, isn’t it? Not necessarily. While some qualitative researchers might take a strong subjectivist stance, there is a wide variety of qualitative methods that can accommodate a variety of viewpoints. 6. We can never explain things using quantitative research. To do that we need to use qualitative methods. That is not strictly true. While qualita- tive research usually provides more depth and less breadth than quantitative research, a well-designed quantitative study will allow us not just to look at what happens, but to provide an explanation of why it happens as well. The key lies in your research design and what variables you collect.9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 11 Introduction to quantitative research ■ 11 ■ ■ ■ Summary In this chapter we have discussed what quantitative research is. We said quantitative research is about explaining phenomena by collecting quanti- tative data which are analysed using mathematically based methods. The fact that the data have to be quantitative does not mean that they have to be naturally available in quantitative form. Non-quantitative phenomena (such as teacher beliefs) can be turned into quantitative data through our measurement instruments. Quantitative research is often placed in opposition to qualitative research. This is often turned into a ‘paradigm war’ which is seen to result from apparently incompatible worldviews underlying the methods. When you look closer at researchers’ actual beliefs, it appears that the so-called subjec- tivist (qualitative) versus realist (quantitative) divide is not that clear-cut. Many researchers take a pragmatic approach to research and use quan- titative methods when they are looking for breadth, want to test a hypothesis or want to study something quantitative. If they are looking for depth and meaning, they will prefer to use qualitative methods. In many cases, mixed methods approaches will be appropriate. ■ ■ ■ Exercises 1. Gender (male/female) is not a quantitative variable. Can you think of any ways you could study gender in quantitative research? 2. Learning styles (e.g. visual, audio, kinaesthetic) are not a quantitative variable. Can you think of any ways you could study learning styles in quantitative research? 3. What is your worldview (epistemology) with regard to research? Do you think it is compatible with using quantitative methods? 4. Can you think of a research question you could study using quantitative methods? 5. What kind of research question would you study using a mixed methods design? 6. What are the main distinctions between post-positivism and positivism?9079 Chapter 01 (1-12) 24/2/04 12:10 pm Page 12 12 ■ Doing Quantitative Research in Education ■ ■ ■ Further reading If you want to know more about quantitative and qualitative research, a good overview of a range of methods is given in Cohen, L., Manion, L. and Morison, K. (2000) Research Methods in Education, 5th edn (Routledge Falmer). This also gives an introduction to the subjectivist–realist epistemological debate. An excellent introduction to mixed methodology research is Tashakkori, A. and Teddlie, C. (2000) Mixed Methodology (Sage Publications). A fascinating but tough work by a leading proponent of experiential realism is Lakoff, G. (1990) Women, Fire and Dangerous Things. What Categories of Thought Reveal About the Mind (University of Chicago Press). Menand, L. (ed.) (1998) Pragmatism (Random House), is a selection of writings by pragmatist philosophers, old and new, and is probably the best overview of this philosophy around.9079 Chapter 02 (13-33) 24/2/04 12:11 pm Page 13 ■ ■ ■ Chapter 2 Experimental and quasi- experimental research ■ ■ ■ Types of quantitative research Once you have taken the decision to do a quantitative study, you have to design it. There are two main types of quantitative research design, exper- imental designs and non-experimental designs. Experimental designs are sometimes known as ‘the scientific method’ due to their popularity in scientific research where they originated. Non-experimental research is sometimes (wrongly, as we will see in the next chapter) equated with survey research and is very common in the social sciences. When hearing the term experimental designs, most of us think back to school experiments in science. Experimental research in the social sciences follows the same basic pattern as those (natural) science experiments. The basis of the experimental method is the experiment, which can be defined as: a test under controlled conditions that is made to demonstrate a known truth or examine the validity of a hypothesis. The key element of this definition is control, and that is where experimental research differs from non-experimental quantitative research. When doing an experiment we want to control the environment as much as possible and only concen- trate on those variables that we want to study. This is why experiments traditionally take place in laboratories, environments where all extrane- ous influences can be shut out. In non-experimental research we will not be able to control out extraneous influences. Control is also increased by the fact that in an experiment the researcher manipulates the variable that is supposed to affect the outcome of the experiment, the so-called predictor variable, while in non-experimental research we have to use the variable ‘as it appears’ in practice. 139079 Chapter 02 (13-33) 24/2/04 12:11 pm Page 14 14 ■ Doing Quantitative Research in Education ■ ■ ■ Example 2.1 Violent attitudes and deferred academic aspirations: deleterious effects of exposure to rap music In this study, a team from the University of North Carolina (Johnson, Jackson and Gatto, 1995) sought to look at the effects of watching violent rap music videos (experimental group), compared to non-violent rap music videos (control group 1) or no music videos (control group 2) on adolescents’ atti- tudes to violence and deviant behaviour. Forty-six 11–16-year-old boys from a club in Wilmington were randomly assigned to one of the three condi- tions. In the violent condition, subjects were shown eight videos containing violent images. Those in the non-violent condition were shown eight non- violent videos. Following the viewing, subjects were asked to read a text passage in which a boy hit another boy who kissed his girlfriend and respond to a number of questions on the acceptability of this behaviour. After that, they read another text passage, featuring a discussion between two characters. One had a ‘nice car’ and ‘nice clothes’ through dodgy activ- ities while the other was completing college. Subjects were then asked to respond to a number of questions probing their views of these alternative career choices. The control (non-video) group also participated in this activ- ity. Subjects were told that the videos were part of a memory test and had been randomly chosen. Results showed that the subjects exposed to the videos were significantly more likely to approve of violent behaviour and of a deviant career path than those who viewed the non-violent video. The controls were least likely to approve of violence or a deviant career path. This experiment suggests a deleterious effect of watching violent videos. However, a number of caveats need to be taken into account. Firstly, as men- tioned above, this is clearly a somewhat contrived situation. It is not clear from this experiment how strong this effect is, and therefore whether it is practically significant within a real-life context in which many other factors may affect attitudes to violence. The sample is from a very specific group (black boys from a boys club), and the extent to which these findings gener- alise to other populations needs to be examined. The authors also did not provide any information on prior factors that could differ between the groups (e.g. age), notwithstanding random assignment to groups. This study there- fore would need replication in further experiments before we could say anything definitive, although the findings are clearly of great interest.9079 Chapter 02 (13-33) 24/2/04 12:11 pm Page 15 Experimental and quasi-experimental research ■ 15 ■ ■ ■ How to design an experimental study There are a number a number of steps to go through when doing experi- mental research. These are outlined in the following sections. Define your research objectives Any research design starts with formulating the research objectives. This step needs to be taken before you decide whether or not to do experimen- tal research, as the research objectives will determine what kind of research to do. Your research objectives describe what you want to study and how. You need to spell out clearly what the aims of your research are. Research objectives need to be realistic. It is important to understand that you can’t do everything. We have to limit ourselves to what is actu- ally researchable. For example, let’s say we want to look at the effects of different test conditions on examination performance. When we think this through, there are an almost unlimited number of conditions that could vary slightly and affect test performance, such as lighting levels, how many adults are present, seating arrangements, temperature and so on. To look at all of these in one study would be impractical and all but impossible. So we will need to set ourselves a more limited goal, by thinking about which aspects might really make a difference and choos- ing just one (or a small number), for example seating arrangements. Our research objective would then be to look at whether or not seating arrangements affect examination performance. We also need to be clear on what our population is. The population is the group of people we want to generalise to. For example, if we were to do this experiment, we would use, say, 40 students in two different seat- ing arrangements and see what effects we can find. Usually, we don’t just want to draw conclusions that are only applicable to that group of 40 students. What we want to do is say something about seating arrange- ments among students more generally. Many statistical methods that we will discuss in the following chapters have been designed to allow us to do just that. But before we can do this, we must be clear about which population we actually want to generalise to. All students of 18 and over? First-years only? This is important because it will affect who we get to take part in our experiment. If I did a study using only secondary school kids, I couldn’t then go out and generalise to primary age kids. 9079 Chapter 02 (13-33) 24/2/04 12:11 pm Page 16 16 ■ Doing Quantitative Research in Education Formulate hypotheses The research objectives you have developed now need to be refined into the form of a number of specific research hypotheses you want to test. A research hypothesis can be defined as ‘a tentative explanation that accounts for a set of facts and can be tested by further investigation’, as we men- tioned earlier. In experimental research, we traditionally look at two distinct types of hypotheses: the null hypothesis and the alternative hypothe- sis. The alternative hypothesis is the one we want to be true, the null hypothesis is the opposite. For example, I might want to know whether adding moving pictures to a presentation will improve pupils’ memory of the key content of the presentation. I would have two hypotheses: ■ Null hypothesis (H ): adding moving pictures will not improve 0 pupils’ retention of the content. ■ Alternative hypothesis (H ): adding moving pictures will improve 1 pupils’ retention of the content. This example presents the most simple case, where there is only one hypoth- esis to be tested. In many studies there will be several hypotheses, and one can also hypothesise mediating factors that influence the relationship between the variables. An additional hypothesis that includes as a mediating factor whether or not moving pictures are aligned to content could be: ■ H : adding moving pictures will improve pupils’ retention of con- 1 tent if the moving pictures are closely aligned to the content. ■ H : adding moving pictures will not improve pupils’ retention of 0 content if the moving pictures are not closely aligned to the content. While the terminology refers to a ‘null hypothesis’, this does not neces- sarily mean that the null hypothesis always has to specify that there is not going to be any effect while the alternative hypothesis specifies that there will be an effect. The null hypothesis can itself predict a specific value, for example: ■ H : the difference between boys and girls on a word retention test 1 will be more than 20 per cent. ■ H : the difference between boys and girls on a word retention will 0 be less than 20 per cent.9079 Chapter 02 (13-33) 24/2/04 12:11 pm Page 17 Experimental and quasi-experimental research ■ 17 or: ■ H : the mean score on a self-esteem inventory will be between 20 1 and 30. ■ H : the mean score on a self-esteem inventory will be between 10 0 and 20. In practice, most researchers test a null hypothesis of no difference because standard statistical tests are usually designed to test just that hypothesis. However, it is important to remember that other types of null hypothesis are possible, because a value or difference of zero might not be realistic for the research question you are looking at. ■ ■ ■ Example 2.2 How should verbal information be presented to students to enhance learning from animations: auditorily as speech or visually as on-screen text? This question was studied by Mayer and Moreno (1998), who con- ducted an experiment in which students were asked to view an animation showing the process of lightning either accompanied by con- current narration or on-screen text. The theory they wanted to test was that visual and auditory learning are processed in two different parts of the working memory: the visual working memory and the auditory working memory. That would mean that if narration is given alongside the animation, students will represent the narration and animation in two different parts of the working memory, while if on-screen text is pre- sented with animation, students will try to represent both the animation and text in the same part of memory (the visual auditory memory) which may then become overloaded. Better performance was therefore hypothesised for the text group. The experiment was conducted by randomly assigning students to the two groups, one viewing the narration with on-screen text, the other with narration. Following the presentation, students were given a retention, matching and transfer test. It was found that students in the animation– narration group did significantly better than those in the animation–text group on all three tests, supporting the experimenters’ hypothesis.9079 Chapter 02 (13-33) 24/2/04 12:11 pm Page 18 18 ■ Doing Quantitative Research in Education Set up your research design Once one or more hypotheses have been set up, you need to decide how to test these hypotheses. If an experimental methodology is chosen (the advantages and disadvantages will be discussed in the next section of this chapter), you will then have to decide what experimental design to use. The traditional experimental design, known as the pre-test post-test con- trol group design works as follows: participants (often known as ‘subjects’ in experimental research) are placed into two groups, the experimental and the control group. The experimental group will receive the ‘treat- ment’ (e.g. watching a violent music video as in Example 2.1 p.14), the control group will not. Both groups will receive a pre-test on whatever instrument is used to assess the effect of the experiment (e.g. a test) before the treatment is given, and a post-test, usually on the same instru- ment, after the treatment has been given. The sequence therefore is: 1. Pre-test 2. Treatment 3. Post-test Experimental group X X X Control group X X Following the post-test, statistical analyses are carried out to see whether the treatment has had an effect (see later). There are a number of variations on this basic design. As we have seen in Example 2.2, it is often desirable to have more than one treatment group. There can, for example, be variations in the treatment that we might want to study. In Example 2.2 (see box) we have two treatment groups and one control group. More control groups and treatment groups are also possible. The pre-test post-test design is also not always followed, as we can see in Example 2.1 where no pre-test is used. Usually it is better to use both a pre- and a post-test, though, because without pre-testing we can never be sure that any difference we find on the post- test is the result of the treatment and not the result of differences that already existed between the two groups before the treatment. Another decision you will have to take is whether or not to give the control group a placebo. This practice comes from medical research, where it is well-known that some patients show recovery as a result of a belief in the treatment rather than as a result of the treatment itself.9079 Chapter 02 (13-33) 24/2/04 12:11 pm Page 19 Experimental and quasi-experimental research ■ 19 Because of this, it is common practice in medical trials to provide the control group with a placebo treatment (for example, a sugar pill) rather than nothing at all. Often, a percentage of the group given a sugar pill will show recovery as a result of their belief that they are taking an effec- tive pill. This obviously means that if no placebo was given, we couldn’t say for certain whether any effect of the treatment was because it actu- ally worked or because some patients believed it works. This can be an issue in educational research as well. That individual behaviours may be altered because participants in the study know they are being studied was demonstrated in a research proj- ect (1927–32) which looked at raising worker productivity in a factory. This series of studies, first led by Harvard Business School professor Elton Mayo along with associates F. J. Roethlisberger and William J. Dickson, started out by examining the physical and environmental influences of the workplace (e.g. brightness of lights, humidity) and later moved on to the psychological aspects (e.g. breaks, group pressure, working hours, managerial leadership). One of the main findings was that productivity increased regardless of the innovation introduced. One explanation is that this is the result of the extra attention paid to the workers (by the researchers) which motivated them to work harder. The same effect could also occur in educational settings. An intervention, for example a programme to help improve pupils’ reading skills, could motivate pupils because of the additional attention they are receiving, leading to higher achievement. Likewise, when teachers engage in a new project, they may work harder and be more motivated simply because they are doing something new or because they know they are part of a research study. Selecting a placebo can be hard in educational experiments, though. It is not as simple as giving patients a sugar pill. Any placebo intervention has to be sufficiently plausible to have an effect, and therefore is often likely to become an intervention in itself. This causes two problems: firstly the addi- tional cost and effort involved in developing a plausible placebo, and secondly the fact that we are now measuring the effect of one treatment against that of another treatment rather than against a control Therefore, in these cases it can often be a good idea to have two control groups: a ‘placebo’ group (which receives a placebo intervention) and a ‘real’ control group (which doesn’t receive any intervention). In some cases schools have been given money to buy in any intervention they want rather than the researchers developing a second intervention themselves.