Educational Technology Research and Development

Multidisciplinary Methods in Educational Technology Research and Development and current trends in educational technology research | free pdf download
DrKateBesant Profile Pic
DrKateBesant,United States,Researcher
Published Date:07-07-2017
Your Website URL(Optional)
Comment
Randolph: Multidisciplinary Methods in Educational Technology Research and Development HAMK Multidisciplinary Methods in Educational Technology Research and Development Over the past thirty years, there has been much dialogue, and debate, about the conduct of educational technology research and development. In this brief volume, Justus Randolph helps clarify that dialogue by theoretically and empirically charting the research methods used in the field and provides much practical information on how to conduct educational technology rese- arch. Within this text, readers can expect to find answers to the following questions: • What are the methodological factors that need to be taken into consideration when designing and conducting educational technology research? • What types of research questions do educational technology researchers tend to ask? • How do educational technology researchers tend to conduct research? – What approaches do they use? What variables do they examine? What types of measures do they use? How do they report their research? • How can the state of educational technology research be improved? In addition to answering the questions above, the author, a research methodologist, provides practical information on how to conduct educational technology research – from formulating research questions, to collecting and analyzing data, to writing up the research report – in each of the major quantitative and qualitative traditions. Unlike other books of this kind, the author address some of research approaches used less commonly in educational technology research, but which, nonetheless, have much potential for creating new insights about educational pheno- mena – approaches such as single-participant research, quantitative content analysis, ethno- graphy, narrative research, phenomenology, and others. Multidisciplinary Methods in Educational Technology Research and Development is an excellent text for educational technology research methods courses, a useful guide for those conducting (or supervising) research, and a rich source of empirical information on the art and science of educational technology research. Justus J. Randolph ISBN 978-951-784-453-6 ISSN 1795-4231 HAMKin julkaisuja 12/20076 Multidisciplinary Methods in Educational Technology Research and Development Figure 1. Strategic framework for the Digital Learning Lab research project. The purpose of the eighth element (Multidisciplinary research methods in educa- tional technology), and the purpose of this book, is to chart the multidisciplinary methods used in educational technology research and, from that charting, to pro- duce information that can help foster improved research methods in educational technology. The improved research methods are, in turn, intended to bring about improved theoretical and empirical information about technologies for improving education. This book is based on numerous disciplines. They include education, psychology, sociology, media studies, computing, program and policy evaluation, software en- JLHQUHLJQE XVLHQVVD RGQ WGUHK LOVFSLLHQZV RKVPH HWRKGFV DLQ Q URIWP ¿HK RGOH I educational technology research and development. Organization 7KLVERRLVURDJQLHGLWQVHFSKD,QWHK¿UVFSKD,GLVFXVVWHKHPW - RKGVFLFRGHK DHWHED WGQ IHK FDURWVWLWKD FQQHXHPÀ HWRKGVFFLRHKLQHGDO GLHWHRJFYOKDIFDRUQHRHWKVWW\KVQDHHP SDHROHYUFGKQWWLGLKXFQO following: 1. The research problem; 2. The purposes and frameworks for the research; 3. The state of and types of the previous research; 4. The type of research act implied by the research question; 5. The level of generalizability needed; 6. The level of accuracy needed; 7. The feasibility of carrying out an investigation; 8. The propriety of an investigation; 9. The utility of an investigation; 10. Whether a quantitative, qualitative, or mixed-methods tradition is being adopted; 11.:HKWUHHUFKWUQRQRDRROL¿SUWPQLDHRVQLHHGG 12. The degree to which stakeholders participate in the research process; and 13. The degree to which the researcher becomes a participant in the intervention or setting of the investigation. G¿HHHHQ DG 7U,WK LRQDWXF WHUWVWHUYHQRNPreface 7 UDHURFGVORHHVLDLDHXPVTFDVUFLWXWQVRLK FHGX W QDQDRHFKQ WL O R - logy research and development. GXTDWLQR¿VMWFP,XUDXWSKVVDUKRWUHDLYHVHURSDDSDUFK - ches (survey research, causal-comparative and longitudinal research, correlational UHVHDUFKDGQHSUHWLQHDPOUHV HDUFK,QWHKVHFGQRSDUWIR&KDGLVFXV QDDUUHOUWU HVDWPHUHVDDSHKDSVDDUUWFKKJ\FK case study research, grounded theory research, and ethnography). U,QDGLQYIHWHKRJORKFLDOKDUDFWLVWFLFXVGD - tional technology research by synthesizing and analyzing the results of several pre- vious methodological reviews. The questions that the overview answers are listed below: 1. What are the meta-categories that can be used to subsume the research cate- gories in other methodological reviews of educational technology research? 2. What are the proportions of quantitative, qualitative, and mixed methods research that educational technology researchers have tended to use? 3. How do those proportions differ over time periods and publication forums? 4. ZGR+RW RKVHSUSUWRR QRVFL SDPURHWRW HSK USUWRR QRVLL QW H¿K GOHIRFHXGD - tion research proper? 5. In what proportions do educational technology researchers choose (a) rese- arch methods, (b) experimental research designs, and (c) measures? 6. How do educational technology researchers tend to report educational technology studies? 7. What suggestions are given for improving educational technology research? ,Q&KDGLVFXVWKRPVWFPQRRPOXVHGDWQWDDGQ TXD OWLWDGHLY WDD FORHOFWHPWGVLQWHKIRHGDOHWFJKQ\UHVHDUFK7HKVHHGLXFOQ WHKVVWQRQLDLUHVVWVWQYHLZQLUHFVWYQDRL ,U&SKQDDJGLYWHGRLZKLIHPWIGGWDVDQDO\VLV and refer the reader to the seminal books for each form of data analysis. I discuss the quantitative analysis of quantitative data, the quantitative analysis of qualita- tive data, the qualitative analysis of qualitative data, and the qualitative analysis of quantitative data. H,WVQQHU,IKHSQEWDPDWXDLJQDWXLDOWWLDL educational technology investigations. I provide information on writing up conven- tional quantitative and qualitative reports, discuss alternative styles of reporting, H PWRIXKOGRDSVLPQJHHUVXDUIGSHQW+VQHIV ¶ YHQGTYHDLWQWRQTLRQRXWDRU&USWHRULSU DRIKRHQHUWHIOHRYHUD,HQRYSWHUHU HUWREGGVDLHUVWHOHORJ¿HRITXHX RORLRQDWXF¿HOGKRLRQ LYHLWTX\HV,SWHU RIHHUFGRPZRLHWRYHUHD,S&KRYSWHU SKHQRPHQRORH¿YHLYHRDFLYHLWMRUTX V,SWHU WIVHW&KUIRWHUQSDHLWU, VHVK¿YHVMRRUWHJ,I8 Multidisciplinary Methods in Educational Technology Research and Development ,Q&KDUSGLHPVRQXFFROQVLRWXRHPWRKGVRFKLQHGDO technology research. In the Appendix, I have included a list of key questions to con- sider when planning educational technology research and development projects. The Target Audiences and How They Will Benefit from this Book The target audiences for this book are primarily educational technology students, their supervisors, and educational technology researchers. However, because of the multidisciplinarity of this text, I expect that students, instructors, and researchers LWIQURIEHKUYWFPQDRXGDRRLUDOIPUKWRR\ RWRUHHUHHWVGQH ,Q W H K SD U DJ K UDS V  ZEHR O , H ODS LQ ZR K W KLV ERRN LV H S HF HW G RWEH¿WDUHWJ groups. 7H¿KUVJWHSUXRSHFWHGWREHWDHQ¿UHVWWFHQIHXGRVWXQGDDRLWOHFKRJORQ\UHVHDUFK DQWPSR:SRHKWHK¿UVDQVHFGQRFKDZLOOKURWWRK - VVH UWWQHGX HVHDUUHFKV WRW ¿HK RGOH HI FXWQGDDRLWO HRJFOKRQU\ HVHDUFKDGQURYSHGL them with the basic information needed to make informed decisions about what methods to choose. The third chapter will help students understand what types of methodological choices practicing educational technology researchers tend to make. The remaining chapters are intended to help students make informed decisi- ons about data collection, analysis, and reporting and to refer them to the seminal resources for carrying out those activities. The appendix, Key Questions in Educa- WHQRJDRFOKLRQO\H7W0RKGVFLRH&KLVHGVLJHQGWRSHOKVWWHQGVXWKLQNWKUXJRKWHK issues that are critical when choosing methods in educational technology research and development. HJVHIWUXSFSR7EHQHHKDR¿UKWFERLRURKUYLDKUVRQ - ers of educational technology students. With hope, this book, in whole or in part, can serve as a course text in an educational technology research methods class and as a catalyst, focusing tool, and source of common vocabulary for academic dialo- gue between supervisors and their students. 7KWKLUJUXRHSHFHWWEHQHDUVHDHQVRFHXWQGDDRLHWF RJOKQR \ U H VH - DUFKW,L\PSRKWWKDWHKHPWRJORKFLDOIUVWKDKYDWGLLFD serve as a starting point for clarifying the methods-choice debates in educational technology. Also, because the third chapter of this book synthesizes the research about the practices of educational technology researchers, it is my hope that it can L U FLS OHDK O OH\QXVV WLSUUHZQRRPVL HRWLWKOVDFSWLLQQD U \ WGOHQHGWLHIK \ – questions such as: • How do educational technology researchers and developers tend to conduct VWFL¿QL"UHV • What methods do they tend to use? • What methods do they tend not to use? • How do the observed practices in educational technology research differ from what is suggested as best practice, and why? TXFLLHQ RLW¿ WG TD HPS QG¿HHQH,WWRDFGRHVVHU OGH¿WRGSGH VGDFWHGVHWHNHWRPHQVXWHGS LHQHOSVSWHUGWKLWYHOGHHQG VLWHQW SKILHHURWV¿HOGHQ¿WPH LRQDWXFHLFDEVHRY,SWHUPreface 9 • How do the research practices in educational technology research differ from WHKVHDUUFFDWHGVK In addition, the answering of these questions is intended to help educational technology researchers understand the prevalent epistemological and ontological WUQRLGK Using this Book in the Classroom As noted earlier, one of the purposes of this book is to serve as a tool for instruc- ting students in educational technology research methods. For example, having students read chapters one and two and then having them think through the list of key questions in educational technology methods choice at the end of this book would be a good way to familiarize students with the factors that are important in choosing appropriate methods. The key questions in methods choice at the end of this book might also serve as an intellectual organizer for supervisors and their students who are beginning to plan theses, dissertations, or projects. The third chapter can be used to introduce students to the practice of educational technolo- JUHVHDUFK7KXUWK¿IWKDQWFSKVDDUWXRGWDDFORHOFWQRL analysis, and reporting; can be used to familiarize students with those issues and to refer them to the essential texts in those areas. At the end of each chapter, I have included a Questions to Consider section that can be used as a catalyst for group or online discussions. Positioning Myself In the tradition of qualitative research, I think that it is worthwhile to provide the reader with some information about the author. For the past four years I have been involved with educational technology research and evaluation in various programs at the University of Joensuu and HAMK University of Applied Sciences. My main area of interest is how scientists, particularly educational technology researchers, conduct and report their research. I also am interested in research about and meta- analysis of educational interventions. My doctoral training; at Utah State Univer- WVL\68Z DLV HQ FXWGUQDRL HVHDUFDK SGQ URJUDHP YDWXQDORL , DHKYD PVDVUHW¶ JUHHLQHGDDQHGDOGDPLQLVWWUDFUHWLV\¶E0 degree was in English, art history, and philosophy. Although my doctoral studies were mostly quantitative in nature, I have spent the past few years developing my toolbox of qualitative methods. Besides working for HAMK University of Applied Sciences and the University of Joensuu, I have worked as an evaluator or researcher for organizations such as the &WURIOR3DQ3UUDDWXQDORL7KOURGL:,QVWX HIW U5R HVHDUFK DQDWXQDORLWKWQDRDL1O&HD+ULQVVHWVDGQ 0D HQPD W8DQDK6UHLGWWYWH8QWVLDH/K\RJDQ&WL\6FRO'KRLVWUFLW,DOHNVUDGHZRKRYDVDQ (JQOLUVOHDKRLYRFDKRQQDSHUFNHQULHZRLQJDVD school principal. DGHYHKGKQ3GDOLWHVDFUDWDSKWH JH QWVPHQJIRUHQWHUH(YG LWHGZH(YPRJG\LFHQHU LRQDFKHORUDW¿FLRQLRQDWXFQGLRQDWXFGH DEHKLFZKWHUKVLGIRH\ U¿HOHLQWVLLWDG HU¿HOQRW"VLLFKSHU10 Multidisciplinary Methods in Educational Technology Research and Development I was reared and educated in the United States, but I have been living in Europe for most of my adult life. I originally came to Finland, where I still reside, to do a Fulbright-sponsored research and evaluation internship. Key Sources Although I drew on many sources when putting together this book, I tended to draw QRPVRURHPWKDK&U HVVZ¶O Qualitative Inquiry and Research Design: Choosing Among Five Traditions was particularly useful for providing the information on the sections about the qualitative traditions and their reporting. KG6DLK&VNRRDV G&QDESOH¶PO Experimental and Quasi-Experimental De- signs for Generalized Causal Inference was the source of much of the information QRHSHUWLPQHDOHGVJLQVOVR)UHFKWOLQJ)UUVRHLQRRG+DGQ V JHK¶V The User Friendly Handbook for Project Evaluation was drawn on heavily for the summary of data collection methods. Some Notes on the Terms Used Before going on, I think that it is important to clarify what I mean by some of the terms I use, because different authors use them differently. I use the term research method to refer to the set of steps that a researcher goes through to conduct an investigation, from designing an investigation to reporting the results of that in- vestigation. By research approach I mean the approach that underlies the research method. Examples of the research approaches I discuss in this book are survey research, causal-comparative research, correlational research, experimental rese- arch, case-study research, narrative research, grounded theory research, ethno- graphic research, and phenomenological research. Finally by the term research act, I mean the types of intellectual activities that are characteristic of a certain rese- arch approach. For example, quantitative description is a characteristic intellectual activity of survey research; explanation is a characteristic activity of many types of DOWDHVHDUFKLVFXVKLI\SHHVHDUFKWD Acknowledgments 7K L VUHVH DUFKZ DVVSUSRX HWGE\ D JU DQW I PWK UR H(X USRH DQ 5HJLQR DOW'QHPSRHOHY )XGQGDP+0Q8\ROSS6FFHVWK6WD3UYRLLD 2IWXRUHKLQODGQ 7KDQNW5X.QWKU%QVWDGQP%DDPUD7NDWKFPR - ments on this text. Thanks to Ilkka Jormanainen for his work on recreating several IRWHKXUHUUHSLQKHOVRWKDQN/DHLDUQRRL9- PD6DD ULQH+QGLHL JLWLHODKRHWOWODUHJKHX\V GQ LODDUHVLGQODR.NUH/WHKDU/QDLQ support on this project. UHSYXRVLQHUYRVLIRWHUS&KDZDVURLJLQDOO\OELXHSVKGDVD)³FURW,FQLXÀ D7P%DR\HLRJF5OKQHRQV´H\F(DXW7LGQUDDRFHQFKLLWRHO0&KRKGV PUHDGLHWNGV DU¶ volume, Modern Approaches to Digital Learning: DLL Project’s Results, published QJHQV JEIRUUHL 36DPDRFW' HQWRVHUGWHV¿J UHLUIRU\RQHLHUDUED%DLJKWRV Q)HRI6¿F OQFWHHE\LHQGLHIVLWLYHU.WRGWHLW Q&KSWHUVLDFVRIUHQIHUWWHGVW,GLYHULWTX +X HOVVHURWQH13 Chapter 1 Methodological Factors in Educational Technology Research and Development The methods-choice debate is one that resurfaces with regular frequency in the education research community. This regular resurfacing is not surprising, though, HPWRKG'WXVHRFHFLPWLSFLRQUHKRRPVRLKWGWJRKDLFQH IFLRHKHGIQFW are affected by the political, economical, and social currents of the times (Greene, Lipsey, Schwandt, Smith, & Tharp, 2007). The methods-choice debate helps de- termine what the research community, the media, government agencies, program funders, and the public accept as convincing evidence. And, among many other UHDQVRHPWRKGVFRKLVDUHHÀFWQRLRUHVHDUFKFPXRQPWL\¶O\XQLLSHV - temological, ontological, and axiological positions. ,WKVWKHPWRKGVFLRFKWGLWKVRFLDVFFHÀDUHGQHZK&QUR - HUUD³HRISPF UWLFLL&DEHSOH¶PO DOHHYDYRWLHLVP - luation in which laboratory research methods were favored for informing policy. Z³DHHW&OQUURRKDSX 5OD-HQQHLIHIFQDLOWLQK QLHQW\WHKVHVWWKURRWKHGLF¿HXELWORWKHSLWDOSD´UP Ten years later the debate resurfaced as what are referred to now as the paradigm warsFQRÀEHWHZWRKVFWDLDWQWDHPWRKGDQWVHRGYD - cating qualitative methodsVHHW'D WDEDX6FU 7HKLYHQSDUD - digm wars waned as the mixed-methods paradigm gained increasing acceptance. In its latest form, the methodological debate has resurfaced in response to the U.S. JIRWLFXLG QVLJQ V'HSD¶U(IGRWUHVHDWUFKKD adopts formal random sampling and experimental designs. 7¿HK RGOH HI FXWGQDDRLWO HRJFOKRQU\ HVHDUFDK GGQ WLQHPSRHOHY VLXPPRW WHKVVLIFDWWKGKDVDQROKLUVRWIRHPWROKFLDGRWL ZRQ7 HHK DOUHLVDW U WLIFDRW IWKLRVLRJGQJQ LHWHED V&ODDNU GQVLQWL¶L DO HPWFOHXRWRHKWGGQDUFDRUOKHLHRDFQUROYDOHQLVH&D6LODFQUNHFUK DQDU V 6GZRQ ¶ WLFHOOWDH D VW LSUPH FDO L PHWRJLORRFDO KG UH Y HLZ V WK IR HH FDXWLGQDOR  FKWV\HDHXKUHSDDHRQ\WQHKVLV these reviews). Other high and low points in the history of the educational techno- RJO\HPWRKGVFRKKYDEHSKSL3DQ0LVWRL¶V GLLVVDR almost the entire body of previous research on distance learning because of its met- JDFUOÀVLVV&ODH PG¶LDSDFULVRVWVWWUHL - VHKHVVHHVHDHHU FORFW FUWLFLLIWVPRHK LYH OHK&YH¶5V G5Q%DLJH'VRIW DWKHLHXGQRPUNRIPLFDZLW LFGRORKR IOVP VHUGVHQHGHEDWHHLF RIIRUDVG GROWHQFQGHKYHEDXF5LWHUOQRORJWH ZTXLHLWG\IRJLFRJ 6QRZ  VIVHEDWHORJGR\J¿HOHQGHEDWHH QHQRW W\WRLWPHQWLRQDWXFLRUSULRQGHYH KRGVLYHLWTXQJDGYRHHQWLFD± LJDGPHQHUHQHIGRHR¿YH UJHLWQVGRJDWWHKEDFW ZHU ZHQ´HUIKEDFWGV  VV VLHQOHQHEDHHHHQ QJGHUVDIHLFV DFDYHQVHIVDE14 Multidisciplinary Methods in Educational Technology Research and Development L OGQOL1FDXHWURGJQPFHDDORKQVRLVRQHIHOX\DVWQUFRVKLV%DDN¶R R¿WKOWLLDWXKUR³FFOZWKXDLPLKDOV FHGX WHFKQ WQDRRD OLIR O - JVPWURDL\WWKRJOFLDKLOVDQVWL¿HGQRVWUFXWLYLVW USSFDKDLQQD UQLJQ FFUWGR HKLFXQXOU-HQWHUDPWQRK GVFRHK³LVHWGHED«WQR about the desirability of generating evidence or about the need to consider the re- lative value of different methodologies. Instead, the debate is primarily over when, or under what circumstances, various methodologies provide the most useful, or :W WQLRFHLXKYZPFGHLXLPVROG-HQVDGQ5VWHWKWZ forward in the methods-choice debate is not to try to resolve the controversy, be- cause the controversy involves deeply-rooted disagreements that are not likely to go  5 WKDUWK HZ\DD H \ VXJJH VW WKWZD \IDUR ZD UGL V ³RW FOD U L I \WKHL VHVXWV R\ GDOHLPURH UFSXRGW LLDRJOHX It is with that piece of advice in mind that I put forth the goal of this chapter: to clarify the issues, I identify and describe some of the factors that are particularly important to consider when choosing methods for educational technology research and development. To make these factors more easily understood I break them into two categories, both of which are critical to understanding methods choice: factors QFWILRVHTVW and FDIWQFH how a research question is answeredIFDUW7RLKWUFWWQKXXDRÀWPKOQDRL the research question are listed below: • The research problem. • The purposes of research and their corresponding traditions. • The state of the previous research. GQWHIK FDURWVWWKD LFQQHXHKÀ ZDR UHVHDUFKTHXVWQLRL V DQVZHUHGDUHOLVHWGEH - low: • The methods used in the previous research. • The research act implied in the research question. • The feasibility of the research. • Safeguards for propriety. • The degree of utility needed. • The degree of accuracy needed. • The degree and kind of generalizability needed. HHIHQIRVH LQÀXHWKDWVRULRQXHKDUFUHWKHIRQODWPXRUKHHLQÀXHWKDW ´ SYHG D\KHDWDWRJ S´HQKDEOHDF LF5RJGHQVDWRJ ´ SGOHJDKWRWHRDF E\DFHMXEO\FWGRKRPHLJYHO\OFSDL WP WKIHHQWHUHHOG GGHUQGHU TRWHD:Methodological Factors in Educational Technology Research and Development 15 • The degree of stakeholder participation in the research process. • The degree of researcher participation in the research setting. ,QRWWQLI\LJQWVHIFDWV,DOVRWQLI\W¿ PHYWURM D \ SH RV UI H VH - arch questions in educational technology research and development. With hope, identifying and describing these factors and research questions will help improve the productivity of the dialogue about methods choice in educational technology re- search within and between researchers, funders, policy makers, and practitioners. What I do not provide here is a concrete set of rules for determining what research approach to use, what data collection methods to use, what analysis methods, or what reporting methods to use over a large set of research situations. One reason is that what may constitute the best methods choices is somewhat subjective – hence, the deep-seated disagreements about methods choice that are not likely to go away. The other reason is that while I believe that there are probably some general guideli- nes that apply across cases, methodological choices are heavily context-dependent. The methods that bring about actionable evidence in one setting may not bring about actionable evidence in another. Methods choice involves a careful weighing of many factors to create the most actionable evidence possible. The methods choice factors I propose here are drawn primarily from my experien- ce conducting methodological reviews of the educational technology literature and IURHNUHGDLDQUHSYXRLUHVHDUFRHPWRKGFLRFK XO-HQV 5RJ KG6DL&RNR&EDSOHP O7HKDUDOVGZUDIU P R \P W U DLQLQJ as a research methodologist and from my experience teaching and supervising edu- cational technology students. Factors Influencing the Formulation of the Research Question Of primary importance in methods choice is the formulation of the research ques- WRLEHFXVDHPW³ORRKGLHWKVUHYDQREVXVWDFQHHQYWHKPDV´ (Greene et al., 2007). While the research question may be of primary importance in determining the right research methods, there are a variety of factors that are of primary importance in determining the right research question – (a) the research problem, (b) the research purpose and its associated tradition, and (c) the state of the previous research. So, by substitution, the factors that are of primary importan- ce in formulating the research question are the foundation on which methodologi- cal choices are made. (For the sake of simplicity, hereafter I use the term research question to refer to all of the following: scholarly research questions, evaluation questions, and development tasks.) %UHHPLQQRWGLVFXVWKIFDURWWWKDLFQXÀWUKXPWOIDRHG - nal technology research problems, it might be useful to make a distinction between the different kinds of research questions. One distinction I make is between kno- wledge-base questions and empirical research questions. The other distinction is between procedural research questions and structural research questions. LRDWLRQXFIRHHHQVHVRJRYIR WHUHUIWHYHUV\RJHQ QRH\ VKE HJHVQKVGVQJ\P KHLGHRUKHLGHLRQLWDGG16 Multidisciplinary Methods in Educational Technology Research and Development As the name suggests, knowledge-base questions are answered through an exa- mination of the knowledge base. The following are examples of knowledge-base questions: • What is known about best practices in user-centered design? • Across studies, what are the academic effects of tools that help students vi- sualize algorithms? • :YWKD DUHOLEDVD UNH ZRQWQ LR FQQHXWHÀ HHK IHIFWHQHLYVRV HI FXGDWDOLQ - terventions? On the other hand, empirical research questions are meant to be answered through ¿DUGVQWLSUPHDOUHVHDUFK7KOOLQHVWQRLDUHDPVIRUPHFLDO research questions; however, they might also be knowledge-base questions if they had already been answered in the previous research: • What are the effects of a new technological intervention on the long-term and short-term memory retention of vocabulary words? •JWKRVUDWWHQHVUFDUVHSUR7WWWKD\UDHWLVH¿ZG new intervention? • In what ways do teachers and students report that a new intervention can be improved? The utility of answering each of these types of research questions varies depending on the circumstances of the research context. Answering knowledge-base questi- VFDQIHDSPDUFSDWUDJUWLUWV¿ZWKD are the current views about best practice. Also, answering knowledge-base ques- tions by conducting a literature review can help clarify what are the unanswered questions that are of import to the research community. On the other hand, ans- wering empirical questions can help add to the knowledge base when there is no existing knowledge or when the existing knowledge is inadequate or in doubt. As I explain later, the choice of an empirical question is often predicated by the answer to a knowledge-base question. In addition to the distinction between knowledge- base questions and empirical research questions, a distinction can also be made between structural and procedural research questions (Stake, 1995). UHDJPDOWQDHDLHJKQOHFOFDLUF¿WLWL\R\7 DWQVLHHXDVWQURFLK be broken down into its structural components. For example, the general research HXTVW³QRL :WLKD VWHHK VFVQHRH WI HHK SUHFQHLHRIVQHVHRFI PR XQLPW \ LQ Q R OLQH FXRU"V´FHV XRGEO H QUGHNER ZR QLRWWQ HIK OZRRO LJVQ WUFXW XUDOVXEHXTVWQRVLE\WHK issues, or structures, that are implied in the general research question: • What do teachers experience in terms of the phenomenon of sense of com- munity in online learning? TF\DGGLWWRDEVSGHQHU RXWQGLRQHDFSURIRXSLRQHLWRUKHOSOHRURQ KDLWVDKHWQGKHXGHWHDGRZGH SLSOHHVTXJIRRZHLFK LRQMethodological Factors in Educational Technology Research and Development 17 • What do students experience in terms of the phenomenon of sense of com- munity in online learning? •:WKDHPGXULDHWFQRWVLFQXÀVWDOVRNHSFLHF - munity in online learning? In others words, structural subquestions unpack the salient issues in a general re- search question. In addition to unpacking research questions by issue, Stake (1995) suggested that research questions can also be unpacked by the research procedures to be used. For example, if one were using a phenomenological procedure to answer the general UHVHDUFKTVW³:KLVWHVVHHWHSUHHVQH VHRI FRXQL PP W \LQ QROLQXRU"VHOZRVKOLURFHXGUDEHXTVWQRLFLOJLQWK in phenomenological research, might be asked: •:WKDWVWVDHGVFUKVDQVWWGVXHSFLHIR VQHVHRI community? • What meanings can be inferred from these statements? • What themes emerge from these meanings? • What are the contexts of and thoughts about the experiences of sense of com- munity? In short, procedural subquestions unpack an overall research question by breaking it down into the research procedures that will be used. By breaking down the overall research question into procedural or structural sub- questions, the research process and research reporting process become clearer and more manageable. For example, the researcher can rely on the procedural subques- tions to naturalistically guide the research process. Also, one often-used method of structuring the discussion section is to organize it by research subquestions. In each section of the discussion, the author answers a research subquestion by refer- U W¿ RGLHQKJQL IHJHRQKVVX WO HVF UWOHWD L JQR KVGHQ L H RJWYQH KV XU RHVL - VH DUFK DQG H ODLQLQ S JZRKW RKVH ¿QG L JQ VSUOHK HVHWOYR HUK HVHD U±FPHSKOUWER HK topic of the following section. The Research Problem \0GLVVWWQDRLSVXYLVRZDGQRWKPDLD³OIRD LUVS LQ W ¶RHV G HQQ - FOH\SVS´VDWLUWQJKRHQQKGRDLHLFHPLDDVKWWDLO\VHUQWKKHD U HVRHDU W KLV SRLQWFK was that a lack of research does not necessarily mean that there is a need for rese- arch; research needs to be rationalized by both a need for and a lack of research on that topic. It is the research problem that demonstrates the need. NDF±PHIIRVUHUHU SU W ¿ WLRQ V UWW VHQHUHQ¶G¶DFHUWHHLEWHPHQ VHSHVKWLWZKDVZKOVXQJSHIR´WHF RILHQFKHRIQFKHDWLRQXH RPRIHQHUHK¶GHUHHQGHOPWH18 Multidisciplinary Methods in Educational Technology Research and Development In this section, I make a distinction among three types of research problems in educational technology: the scholarly research problem, the evaluation problem, and the development problem. These types of research problems correspond with the different purposes of educational technology research and their associated tra- ditions, which are discussed in the next section. 7KOVDFRKOUUHVHDUFKOUESR³WHKLWQFWXDDGQDUGLV\QDVRFQHURSHUHOS - WL\ ´ 2IF¿RH 5I HVHDUF6K UHYFLHV S G  LIUHIZV HKWUHK WLL VDQDOHSLSGRU basic research problem, as explained below: In applied research, the problem is based on a need, which may be based RHIEX¿OOWSRSXEGLFVRPHDWLQJ shortcoming in educational or psychological services. The need is not, ho- wever, the problem. Any one need may be the basis for a number of different research problems, depending upon the research evidence that is available and judgments about how to best address the need. For example, the need to avoid the erroneous placement of bilingual minority students in special education classes might lead to research on the sensitivity of school person- QHOFRWXOWUXDXOHÀQLQFHVWQRKHUGHFLLVQLRVDERXWVWXGHQWRQVWKHHYLGHQFH for the validity of the instruments used to classify bilingual students, or on FODQYROYHLLQPSHVUHDODHVQDWQWQW¿FILRWKHDQGDWHUHXWQWLRQGHFLQV,QLRV basic research, the assumed need is for adequate knowledge, and reference to public policy or needs data is usually not necessary. (p. 2) The most frequently seen types of educational technology research problems (or the needs upon which they are based), which are implied by the major educational technology research questions that I discuss later, include: • a disconnect between how educational theory informs technologies for edu- cation, and vice versa; • a need for information about the best methods for educational technology research and development; • a need for information about the best methods to implement and improve the utility of technological innovations; • a need for information about the effectiveness of certain kinds of technolo- gical interventions; and • a need for information about what factors moderate the effectiveness of cer- tain kinds of technological interventions. S&DPURHGWOVDFRKOUUHVHDUFKOUESRPVRKW\SHIRUHVHDUFKPHOVURES which I refer to here as evaluation problems, are local in scope. For example, an educational organization might have a need to respond to a local problem within their organization – perhaps there is a high degree of student attrition that needs to reduced, a need to determine if a certain distance education program should VHURWHVHP\R RQDOLF\OLFRGOHHRUGDPLQHDQGRUQGDWDLQ TXOOHHOHP\HMethodological Factors in Educational Technology Research and Development 19 be continued or abandoned, or a need to determine if a program had been imple- mented as promised. Evaluation problems are typically articulated by program sta- keholders. Development problems, as the names suggests, concern the development of inter- ventions or a lack of knowledge about how to best develop those interventions. For HHODSPFXPIRWHK¿IRFHXWGQDDRLOHWFOKRQDOZWLKHQZ or adapting existing technological interventions to solve current educational prob- lems. The purpose of educational technology research; whether it is scholarly, evaluative, or developmental; is to solve the types of problems mentioned above. In the next section, I discuss these different research purposes and the traditions with which they are usually associated. The Purposes and Traditions of Educational Technology Research Typically, research in educational technology is conducted for one or more of the following purposes: 1. to answer questions that are important for the development of an educatio- nal intervention; 2. to answer questions that are important to local stakeholders to improve, come to understand, or assign value to a program; or 3.QVHVWQRLKDUSURPWDWQFKLLXPRQPL  W\ :KLOLIRGLIF¿XWOWGZUDFDUOLHQVEHWWHKVHXS U S R VHV HGUHW P LQLQJ the primary reason for conducting research is helpful in understanding methods choice. The research traditions that correspond primarily with the purposes of re- search listed above are (1) design-based research, (2) evaluation research, and (3) education research, respectively. It is important to note that research traditions can easily overlap one another. For RFHXIDVWWHHQWHUGHDORKOGPQQUVHDHLRRODLKHNSJQPV UYR UHJGLWHVGKL LSURPWDWLQ QVJLWKIV UDR VWFQHLLFW¿ HUKR\V L¿PLODO\U GQLJQVIURUHVHDUF might serve as a starting point for the development of an educational intervention. In the sections below, I go into more detail about each of these research traditions. The design-based research tradition. A research tradition that has gained much credibility over the past few years and that works well for developing educational WLYWLHLVRORLVHGVVHUHVHDUFK³GES L U FDO L  FDHXWLGQDOR  UHVHDUFKZWKWHUKR\GULQRDUQLYLWUQRPVHVVH 5HVHDUFK&ORHOFF FWUGRLYHLSWRWJQ'HKHJVL%DQVHGHVHDUFKORFWL - KKDUFDWLVWFLVVHHVHDUFUH KDGUQEDLJVRIGHYH HUH¿YHFW OH&5 GQ%DLJ '´HQHQQJOHIVLJGHYHQHKLW HPVOHQKLFZKGQEDLJVWRUDF KVLFEDP PS J¿RWIVTO HQZHOHRWHQVLWH ¿FFHQHVWWRWHLWDVWZHUTXWRD YHORSLGHQJVGH\RJHOGK20 Multidisciplinary Methods in Educational Technology Research and Development First, the central goals of designing learning environments and develo- ping theories or “prototheories” of learning are intertwined. Second, de- velopment and research take place through continuous cycles of design, enactment, analysis, and redesign.... Third, research on designs must lead to sharable theories that help communicate relevant implications to prac- titioners and other educational designers. Fourth, research must account for how designs function in authentic settings. It must not only document VXFFHVVURLDXUHOIEXWDOVRRIFXVQRHLQWUDFWLRQVWKDWUH¿QHRXUXQGHUQDVW - ding of the learning issues involved. Fifth, the development of such accounts relies on methods that can document and connect processes of enactment to outcomes of interest. (p. 5) As shown above, design-based research has many characteristics, the most distin- FWELHQWFXQRRXL³QFFH\OVRQWWPDQDVLVDQ´UH,Q the traditional research framework, summative, generalizable, and rigorous stu- dies are valued; however, because those types of studies are long and resource in- tensive, they are not feasible for the initial development of an intervention. Instead, LQVWDOHWGLEQVLULQHUSVHVDXHXPDRQVUFKFXFGDQRUQRL to determine how to improve an intervention. After the intervention has been per- fected through many cycles of design and testing, only then does it makes sense to conduct a summative, large-scale, and resource-intensive study. What is more, design-based research is an exploratory sort of activity and, as such, can lead to L QHJVWLUHEKXWRRVKHHUVHWVLQWLOGRDVWJQKD¿DHUQ WPDURHPD\VXWU,HHQKV basic form of design-based research, no particular set of methods is prescribed; the appropriate method is the one that leads to the type of information that is needed H¿QWKQWQRL XSSRODPDQLHIVWWQDRLRQVHGUHVHDUFKLQQD5QO V¶,DQG Q - HWJWUDLYH/HDUQLJQ'HVLJ)LQ)UDJ N ,/XHUPRZUHZRVKVW'HSKKDVHVLWQ,/HK' framework and how they compare with the phases of other design traditions, such DVLQVWUFXWQDRLOHGV&DU H\UFSXRGQ 8OULS  XVDJHUHGHFWQHHGJVL&QQVWRD WQLQH /RFNRRGZ GLIIXVIRLQRQ - WYQDRVLHU 5RJV DGQFHXWGQDRLUHVHDUFK ,VDFD 0FHLKDO 7HK,/' framework begins with an informed exploration phase that includes problem iden- WLWQDRLOWLDXUHVXU\OUERSHG¿QWLQRLHQHGDQDO\VLVDQGXFLD characterization. The next phase, enactment, includes researching the initial in- tervention design, creating a prototype, and then developing a fully detailed in- WQRYL7KHQWKSDVHOLRYQWDFFH\OVRLSROW HVWLDJQ GUQ WHQ¿HPHQ IWR LHK UHWQWQ%QHRYL DQQD5QWLODGGQ HVFULEHVWHDK FWLYWLHLZV L W KLQ W KLV KSDVH DV DWWGDQOLSVUPWH\¿WVPVWDWLJQ\HRY1D LQ)XURKODVWHDJVFDQRORNWHDOUVWHDJVLQ W,HK /'I UDNHUPRZ For example, the results of an evaluation might indicate that the intervention needs to be redesigned. After another cycle of implementation and evaluation, it could be HUHGWPLHQZG HKWWUHK UHK RWHQH¿PHQWI LHK UHWQWKQQHRYL LGD WGV HUHHIFW7K GKLDPWUEWKHRYDWX,Q/DORL'¿RKSDQDVHO Z LWLHVKVKPHLWQDL - on, adoption, adaptation, and summative evaluation of the intervention. 7ISKXUHXVRUFHQVHGUHVHDUFKDUHWHK'HV%DVHG5HVHDUFK &ORHOFWLGDQ .OHO, WK'HVV HG5HVHDUF&K ORHOF - QRWFLGDZ¿ QOLQNVLQDOZUWLLQRHG VJQL EDVH G UHVHDUFK and links to various other design-based research resources. Kelly (2003) edited a VQJHPWRVGVLWHQHEQYHH  Q%DLJHQ\QGVLWHZHEYH¶ V QLJEDVLJGHIRUVOHOZR WGIIHRDGHUVSDFWRR HIHGVL LHUREDFSWHUZHLJ LRQDWOXWHRUKHLYHWHPQHPHQWHIRULRQHPHQ WILYHLWHUVYHHHQWHU HHQGVDHPYHWHUD¿F LRQ (SQJHU KLFVLJGHW NLF 'QLJ LW%DVEDVLJGHIU YHQHLHUHWWRU EFFWXDWW VHGWHHUGQEDGLJLJDGHÀHYH QVLJGHGO\HQDFHQVLJGHIVVLWJLYHMethodological Factors in Educational Technology Research and Development 21 Figure 2. Merging of design and research processes into the integrative learning design framework. From “The Role of Design in Research: The Integrative Learning Design Framework,” by B. Bannan-Ritland, 2003, Educational Re- searcher, 32(1), p. 22. Copyright 2003 by Sage Publications, Inc. Reprinted with permission. 22 Multidisciplinary Methods in Educational Technology Research and Development special issue of Educational Researcher that contains a selection of articles that provide a thorough overview of the design-based research tradition. The evaluation tradition. Three purposes are generally assigned to evaluation. Similar to design-based research, one purpose of evaluation research is to collect data that can be used to improve an intervention (formative evaluation). Another purpose is to collect data that can be used for decision-making or assigning va- lue to a program (summative evaluation). Yet another purpose is to make sense of DUSRJUDP 0DNUHQU+\ XO-HQV 5HJDUGHOVVIRWHKVSHFLF¿XSUSRVHIR evaluation, evaluation research answers questions that are primarily of interest to local stakeholders. There are a variety of evaluation traditions to choose from, but a standard method for conducting an evaluation consists of the following steps: • Develop a conceptual model of the program and identify key evalua- tion points, •'HYHORSHYDOXDWLRQTXHVWLRQVDQGGH¿QHPHDVFUDEOHRXWXPHRV • Develop an evaluation design, • Collect data, • Analyze data, and • Provide information to interested audiences (Frechtling, Frierson, Hood, & Hughes; 2002, p. 15). There are a many good resources for evaluation research. For example, the U.S. Na- tional Science Foundation has created a series of useful, free, and practitioner-ori- ented evaluation handbooks. The latest in the series is Frechtling, Frierson, Hood,  VXJHK¶V+DQ The User-Friendly Handbook for Program Evaluation. It pro- vides an overview of the types of evaluation, the steps involved in conducting an evaluation, an overview of quantitative and qualitative methods, and a section on strategies for culturally responsive evaluation. Other handbooks in this series in- clude The User-Friendly Handbook for Program Evaluation: Science, Mathema- tics, and Technology Education (Frechtling, Stevens, Lawrenz, & Sharp, 1993) and The User-Friendly Handbook for Mixed Methods Evaluation (Frechtling & Sharp, 1997). Seminal books in evaluation research include Herman (1987); Mark, Henry, UH URUHLOQWD XDOHQQ VQ The education research tradition. 7KH¿QDWOUDGWL,QGRHLDOZWLUHKKHLWVFHGHXKD - tion research tradition. While design-based research and evaluation research may indeed be types of research on education, I have chosen to use the term education research to refer to research that answers questions that are of interest to the edu- cation research community. Although design-based research and evaluation rese- DUFFK DGQ PR FWKX DR QVWUHZ THK HXVWQRRVL WI VHK WFQHLLFF¿ XPRPQWLW\WKDLVWRQ their primary function. G:V HLV G-WR3Q G7ODV VN3D G Methodological Factors in Educational Technology Research and Development 23 There is no shortage of high quality books and resources on the practice of educa- tion research. They are too numerous to describe here, but I do recommend Gall, Borg, and Gall (1996) as an introductory guide to the multifaceted literature on education research. The State of the Previous Research For many reasons, becoming familiar with the state of the previous knowledge on a UHVHDUFKSRWLOWDLXUHUHYLZFUDOIVLXIQWPODR of a research question. First, conducting a literature review or needs analysis ma- HVN  WSL R VHOVLRGEWHUHW PLQHKZDQVZ R VUH U HVLQH¶JHQDUR FKTHVX WZLOOFUL SRW UHHLVWLNJQ HZORHG7QJ HK UHPFLD(Q FXWG5QDRL HVHDUFK VVR FL suggests that research can contribute to knowledge in the following ways: • It can contribute to an already established theory or line of empirical research, • It can help establish a new theory, • It can meet a practical need, or • It can make up for a lack of needed information about a problem or issue. (p. 34) For example, the literature review should make it possible to determine whether there are established theories already and to what degree they have been substan- tiated. Or, from an empirical research point of view, a literature review can show LIWKWVRYDULKDEHWGLLGHZKWKWHKDVVRFLDWQRVLEHW - ween those elements are understood, and whether the causal mechanisms under- LQWKRQHPRQHKSKDEHHWGLLGDQUDWHKVDVSHFWVW state of the previous research will have considerable impacts on the focus of the current research. In some sense, the literature review is the mother of the research question. Second, the literature review provides a basis for comparing and contrasting cur- WU¿GQLJQVZUSHYXRVL¿GQLJQS&DPURLQDQWFQRUDVWLQF XU WUQH ¿ GQ LQJV ZWLUHSYXRL¿GQLSKXELODHYFGLEDVHWXSWKFXU VWUQH WG\LX Q FQR - WG¿OHGRUQSVKOLHWOWKEDDLWUHWVKQGLRIHXUI - WFSDLWLQVWUHWWDQHPVFWHPXRRVDGQVHWWLQJV&SDPURLJQDGQFRWQUDVWLQFXUU DUHQYXRLGQLJQOVKXUUGQHQLUJHQRDQLJQ )LQD¿OLO\JQEDWUSHYVLUHVHDUFKQRDFDQSOHKUHVHDUFKUHVRO - cate themselves in what I call a research family and get a clear picture of how their  RWQUHWVHVDUFK research lineage. By research family, I mean the individual rese- archers or groups of researchers that investigate the same topic. By understanding ZRK HQR W¿V RDLWQ UHVHDUFKILDPLO\W LHV DWUHVL RXUHGQVWDGQWZKD %HUHFK calls the tribes and territories GV¶HQIR research lineage, I mean the histo- rical line of research on a particular topic. By understanding the history of research R\¿HO% ¿LD WRSLFRXKHRXWRXWQG VPHPW¿HFLYHWRJVDV¿GS HQWJ HQHUWSDYHUHGKHQJRKLFGHJVZKDWHV HVHHQQGHOVQJVK JGJVKLWHQ KHRXWDEHWH\W¿HHQQYHQHJO\ HU¿HHQHQYHVDEOHUHOHPHQ\NHH LRQDW EXWHRQWLRQ LRQQH¶RUWRUDFLFLWDVLHWHUDQJGRE\LF24 Multidisciplinary Methods in Educational Technology Research and Development HFLVSHHLRHKZVDRUHUSHUDSWUFHFHKWLDVUQVDFRKQGLKWLUVWKDWQ\ - tify what is needed at the present, and predict what will be needed in the future. For WKLV &UHVXJHVJZHOVWOV PDNLJQD research map – a visual representation of UVRZRK ¶HQHVHDUFK¿WLV ZQ WLKWHSK UHYXRVOL WUHWLDXU±H WRXGQVWDQQUHRVH - arch lineage. An example of an outstanding research map, which was created by a GDJVLFFQWLLDXQSUFRIPRIR¿FWKLDH3KGQQGWLDHVLGHVWPD in Figure 3. It shows the relationships among the previous research studies and the UWOHQDRKVSLEVL HWHZWQH SHK UHYXRUVL HVHDUFKDWGQ YHK DUXRVQL HHGVWWKDWHKK'3 USRRFVHDHGQVGHWLDDUFHDQUHV Figure 3. Example of a research map. By F. Girardin, 2007. Copyright 2007 by F. Girardin. Reprinted with the permission of the author. The Five Major Categories of Educational Technology Research Questions In the sections above, I discussed, in general, the factors that go into choosing re- search questions. In the section below, I discuss what types of research questions WRKVFIDURWW\LLWK¿RFXHWQGDDRLHWRFOKRQJ\UHH+,XVHGDQ empirical approach to identify the major categories of research questions in educa- tional technology, between and within the design-based, evaluation, and education research traditions. With hope, identifying and describing the categories of ques- tions that are often seen in educational technology research will help add clarity to the debate about which methods are appropriate for answering these kinds of research questions. OIHOGHQHOGRWHQGVH VWWRDGGVPKLH¶GUVS LRQQWHSUH'HOGOWHUHLHQG H¶GVHU WRRHQ D¿LDFKWUWUMethodological Factors in Educational Technology Research and Development 25 Design-based research questions. 7HHKJVLDQVHHGVHDUFKFWHLY 5) has given some suggestions for the categories of research and development ques- tions that are of critical importance. The list below summarizes the major research and development questions mentioned there. 1.5HVHDUFKHTXVWQRVWLWGKHDDOZWLWIRWKWQHPHSHGUHKRKROHYHLVR - URDVUQL 2. 5H VUHHWHHQDXDVQUHUWWFIYFQDWQOKKRWVDLLWHQKRKVL HKGQWQQHRL authentic setting. 3.5HVHDUFKHVWQRLWWKDHGDZWLKKDLQWQRYLFXVDHWKUH outcomes. 4.P'SHRHVWQRLWWKDHGDZWLKKDLQWQRYLFDQEH LUSRP - ved. ,QWUHPVRI%DQQD5QV,WLO¶DGQ HWJQWUDH/LY HDUQLJ'Q HJVLQ)UDHPZNUR  WHKVURFHLDRUHVHDUFHXTVWQRLHWQQHRPLEDRWDNR VS HFL¿F HDQLQ P JV through the steps in the informed exploration stage. Those steps are problem iden- W¿FLHDWLWUODWVGH¿QLWLRQUXSUREOHQXUHHPHQDO\VDGLRQYHLV\V , and audience cha- racterization. Evaluation questions. EPWUHH5PH RWKD RHQ WI SHK ULPDU S\ XUSRVHRV HI YDWXQDORL research is to answer questions that are important to program stakeholders. So, it is no surprise that questions in evaluation research come from people who are invol- ved in a program or intervention. Typically, evaluation questions are generated in two phases – a divergent question phase and a convergent question phase. In the di- vergent question phase the evaluator collects an unedited list of research questions from the people involved in the program – for example, from the administrators, practitioners, and clients. In the convergent phase, the evaluator and sometimes the stakeholders decide which of the questions from the divergent list need to be DQVHUVQFLDQVHWD Because there has been no review of the questions in educational technology evalu- ation reports, a lateral review of the questions in computer science education evalu- ations might provide some insight into the categories of questions that educational technology evaluators strive to answer. I make the assumption here that the body of computer science education research is more or less generalizable to the body of FHXWGQDDRLOWHRJFOKRQ\ UHVHDUFKIUWR RUZ HD¿QVRV UVWEHFXVDHVRKPDVLV LSV RWX HQ FXWGQDDRLWO HRJFOKRQ\UHVHDUFKDGGQWLQHPSRHOHY QWKHUF VFFFXHWGDRLDGQVHGFQREHFXVDWKWZGHKLLEPDVLPLODULQ WKDWQWLQDOHVHDUFHWRKVGHVKHSDOQ  th ,, 5QFFE KXSHGWOQDDGRRGQ UHY HLRZ I . LUHGDJQU WQHW KXJURK grade evaluation reports of computer science education programs that had been publis- hed before March 2005. I inferred the evaluation questions from those evaluation reports. For example, if an evaluator had examined student achievement as an out- come, then I assumed that at least one of the evaluation questions had to do with DGRG VXH5KP\RIULWGTX\DHTX VLHLWQ\WV¿HORHHQHLHQ RPSXWH¿HOG HPSKXF GOHUZHUHDQEKFGZKWDG¿ZHU QHYHGVKIVWHJH HQWHUQRZOVTXYHOHQW GVLGHHVHQWHUQRZOVTX DW RWLWTZGDL QJ´RLHIOH KH³SURWRWU  OHSRO&5%'26 Multidisciplinary Methods in Educational Technology Research and Development the ability of the program to bring about student achievement. The factors that were examined are equally telling. For example, if gender had been examined as a factor, then I assumed that there was an evaluation question about whether the program had a differential effect for male or female participants. At any rate, Table 1 shows that the outcomes that the evaluation questions most often dealt with, in decreasing order of frequency, were attitudes, enrollment, and achievement in core courses and that the interaction factors that were examined most often were gender, ap- titude, and UHLH . Table 1. The Question Topics in K-12 Computing Education Program Evaluations. Question Question topic Frequency (%) Outcome (out of 67 outcomes in 19 cases) 1 Stakeholder attitudes 17 (25.4) 2 Enrollment 13 (19.4) 3 Achievement in core subjects 14 (20.9) 4 Computer science achievement 9 (13.4) 6 Teaching practices 5 (7.5) 7 Intentions for future CS jobs/courses 3 (4.5) 8 Program implementation 2 (3.0) 9 Costs and benefits 2 (3.0) 10 Socialization 1 (1.5) 11 Computer use 1 (1.5) Factors (from 19 cases) 12 Gender 3 (15.8) 13 Aptitude 3 (15.8) 14 Race/ethnic origin 5 (26.3) VUDHFPLQUH A Methodological Review of Program Evaluations in K-12 Computer Science EducationHLRULFDWWRGVXESXEOWLDVFULSWIQXPOSK-5DQGRP´E\-Q Education research questions. In this section, I present the categories of research questions that have been of import to the educational technology community over W HK OD V W QHW U R ¿I HWQH HDUV \  :LW K SRHK H DPLQLQJ W HVK H HVXT W LQRVIRWHKSDVWFDQ help give more meaning to the research questions of the present. 0RWKDQRHIRUZDVSDFWRVVEOHSH)UR³ UDFJLQWKQLFR

Advise: Why You Wasting Money in Costly SEO Tools, Use World's Best Free SEO Tool Ubersuggest.