The Open Collections website will be unavailable July 27 from 2100-2200 PST ahead of planned usability and performance enhancements on July 28. More information here.

UBC Community, Partners, and Alumni Publications

UBC Mooc pilot : design and delivery overview Engle, Will

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata


52387-UBC_MOOC_Pilot_Report.pdf [ 1MB ]
JSON: 52387-1.0107914.json
JSON-LD: 52387-1.0107914-ld.json
RDF/XML (Pretty): 52387-1.0107914-rdf.xml
RDF/JSON: 52387-1.0107914-rdf.json
Turtle: 52387-1.0107914-turtle.txt
N-Triples: 52387-1.0107914-rdf-ntriples.txt
Original Record: 52387-1.0107914-source.json
Full Text

Full Text

DESIGN AND DELIVERY OVERVIEWUBC MOOC PILOTEXECUTIVE SUMMARY 1Key development and delivery lessons learned from the pilot included:It takes a village to develop a MOOCCourse development and delivery involved teams with a diverse set of roles and expertise.  In addition to the course instructor, instructional designers, media producers, web programmers, and student academic assistants were involved in the success of the pilot. Partner relationship management at both strategic and operational levels was also important.There was inherent tension between scalability and functional capability in MOOC deliveryInstructors in the pilot scaled back the design of their courses as the Coursera platform did not natively support a way to assess or deliver the types of activities they originally envisioned. Formally assessing activi-ties that were conducted using tools external to the Coursera platform was technically challenging and often relied upon using work around strategies based on Coursera’s peer assessment functionality. MOOCs represented a way for instructors to “publish” their teachingThe ability to expose their subject knowledge and expertise, as well as their teaching ability and practices, to a broad audience was a motivating factor for instructors.UBC successfully delivered five massive open online courses (MOOCs) in the spring and summer of 2013. Individual MOOCs incorporated different pedagogical design strategies to achieve their desired learning outcomes and course objectives. The pilot MOOCs lasted from five to eleven weeks and provided tens of thousands of learners worldwide, including over 8,100 students who earned certificates for completing the courses, with an opportunity to engage with UBC instructors and learning materials. Development of these courses involved the creation of large amounts of new learning material, including more than 60 hours of video-based lectures, 98 text-based module pages, 1,040 quiz questions, the use of tools outside of the Coursera platform, and multiple innovative learning activities. Learning materials from the MOOCs have been, or will soon be, used by hundreds of UBC students in credit-bearing courses.  Addition-ally, instructors made efforts to facilitate the reuse of their learning mate-rials through the use of Creative Commons licenses and the transferring of content to additional platforms beyond the Coursera platform, such as external YouTube channels.The MOOC pilot supported UBC’s learner-centred focus in the classroom by providing a rich set of resources and strategies to support flexible learning options for students who are registered in UBC courses.  These strategies include the processes and best practices for the development of media-rich learning materials, agile approaches to course delivery, and instructional design strategies to better scaffold self-based, peer-based, and open learning efforts.EXECUTIVE SUMMARYTABLE OF CONTENTS1 EXECUTIVE SUMMARY3 INTRODUCTION3 Table 1: Courses Delivered during MOOC Pilot4 Table 2: Overview of Enagagement with Coursera-based Pilot Offerings5 COURSE DESIGN AND PRODUCTION5 Table 3: Course Elements6 Table 4: Media Production Strategies8 COURSE DELIVERY9 DEVELOPMENT AND DELIVERY COSTS9 Figure 1: Overview of MOOC Pilot Costs by Category10 DEVELOPMENT AND DELIVERY CHALLENGES11 DEVELOPMENT AND DELIVERY LESSONS13 LEARNER DEMOGRAPHICS14-15 Table 5: Comparison of Learner Demographics16 COURSE ACTIVITY16 Table 6: Aggregate Course Activities17 Figure 2: Number of Students Who Watched Lectures Per Week18 Figure 3: Number of Students Who Submitted Quizzes Per Week19 MOOCs AS OPEN EDUCATION RESOURCES ANDON-CAMPUS REUSE 20 CONCLUSION21 APPENDIX A: SPECIFIC COURSE OVERVIEWS25 APPENDIX B: COURSE EXPENDITURE COMPARISONS26 APPENDIX C: POSITIVE STUDENT FEEDBACK30 ACKNOWLEDGEMENTS3Table 1: Courses Delivered during MOOC PilotCourse Name Instructor(s)  (all from UBC unless specified) Start Date Course Length Game Theory   Dr. Kevin Leyton-Brown, Dept of Computer Science  Dr. Matt Jackson (Stanford University)Dr. Yoav Shoham (Stanford University) January 7, 2013 7 Weeks Useful Genetics  Dr. Rosie Redfield, Dept of Zoology May 1, 2013 11 Weeks Climate Literacy: Navigating Climate Change Conversations  Dr. Sara Harris, Dept of Earth, Ocean & Atmospheric Sciences Dr. Sarah Burch, UBC Continuing Studies Centre for Sustainability May 17, 2013 10 Weeks Game Theory II: Advanced Application  Dr. Kevin Leyton-Brown, Dept of Computer Science  Dr. Matt Jackson (Stanford University) Dr. Yoav Shoham (Stanford University) May 27, 2013 5 Weeks Introduction to Systematic Program Design 1 Prof.   Gregor Kiczales,      Dept of Computer Science June 3, 2013 8 Weeks  INTRODUCTIONIn response to requests from UBC faculty members in the early summer of 2012, UBC contracted with Cours-era (a US-based platform provider) in September of that year to pilot a limited number of Massive Open Online Courses (MOOCs). MOOCs are non-credit offerings that may be taken by large numbers of learners from around the world for free.  UBC’s participation in MOOC development represented an opportunity to explore a rapidly evolving on-line teaching and learning space and bring the lessons learned into our on-campus and online courses. UBC’s initial announcement about the pilot project is available at  In total, five courses (Table 1) were developed as part UBC’s MOOC pilot; Game Theory I and II courses were developed and delivered in partnership with Stanford University.  The first MOOC was offered in January 2013 and all four other initial offerings were completed by September 2013. Four courses leveraged Cours-era ( for delivery, whereas Game Theory II was delivered (by faculty election) on Google’s open course platform, Course Builder. For the purposes of this report, we focus on the four Coursera-based courses, where institutional support efforts were concentrated. INTRODUCTIONStatistics Category Useful Genetics Climate Literacy Systematic Program Design 1 Game Theory 1 Pilot Total Total sign-ups 37,721 24,303 78,704 189,423 330,151 Total active students* 17,411 13,553 49,006 84,965 164,935 Signature Track sign-ups 98 236 601 N/A 935 Number of final exams taken 705 1,161 1,683 8,482 12,031 Number of certificates earned  590 750 1,749 5,085 8,174 Normal 407 435 509 3,016 4,367 Distinction 183 315 1,240 2,069 3,807  Table 2: Overview of Engagement with Coursera-based Pilot OfferingsBasic statistics for the Coursera-based courses are included in Table 2. Approximately 330,150 people signed up for the initial UBC MOOC offerings and, of these total sign-ups, 164,935 learners logged into a course at least once.  In aggregate, 5% percent of these learners, 8,174 in total, earned course certificates.  *Defined as logging into the course at least once during the delivery phase .  This pilot project aligned with UBC’s value as stated in Place and Promise (2009): “The University supports scholarly pursuits that contribute to knowledge and understanding within and across disciplines, and seeks every opportunity to share them broadly.” Furthermore, the pilot served as a rich information source and innovation test bed for UBC’s Flexible Learn-ing Initiative (  330,150 learners logged into a course at least once people signed up for the initial UBC MOOC offerings 164,935learners earned course certificates8,1744INTRODUCTIONSyste atic ProgramDesign 1 r  iil tTotalCourse Elements Useful Genetics Climate Literacy Systematic  Program Design 1 Game Theory  Game Theory II Pilot Total Course Length (weeks) 11 7 8 7 5 38 Total Number of Video Lectures 107 38 86 48 26 305 Total Minutes of Video Produced 1,802 405 857 458 343 3,865  Total Number of Quiz Questions 480 201 175 93 100 1,049     Graded 170 165 95 60 70 560     Un-graded (practice or in-      video) 310 36 80 33 30 489 Peer Assessments  1 2 1 - - 4 HTML Module Pages 29 21 38 4 6 98 Other Exercises 1 - - 8 n/a 9  COURSE DESIGN AND PRODUCTIONMOOCs are an emerging delivery model predicated on the provision of free and open access to courses to large numbers of learners.  Course design for this pilot was primarily the responsibility of the instructors but took place in a context of course and support teams with a variety of roles and expertise. None of the pilot instructors had taught a fully online distance course before. The UBC Centre for Teaching, Learning & Tech-nology (CTLT), the Faculty of Science, and UBC Continuing Studies provided instructional design and project management support, UBC IT Creative Services provided media development support, the Office of the University Counsel provided contract support, and department-based graduate and undergraduate academic assistants provided in-depth production and delivery assistance. Coursera-based courses consist of instructor-centric video lectures, text-based html pages, automated or peer calibrated assessments set at regular intervals, and peer supported learning though discussion forum engagement and peer assessment activities.  Instructors had different expectations and approaches to the pedagogical design and development of their courses. Course teams worked with instructional design staff to implement different design and delivery strategies based upon their desired learning outcomes, course objectives and platform functionality.  Table 3 provides an overview of some of the primary course elements.  Please see Appendix A for specific course level information.Table 3: Course Elements5COURSE DESIGN AND PRODUCTIONrs  l Ga eTheoryli teit r c r  IIil ttalTable 4: Media Production StrategiesAlthough they were not the only format of learning material, video resources did constitute a major compo-nent of the pilot courses. Each of the courses developed different models for their media production.  Table 4 provides an overview of the different media development strategies employed by the individual courses.6COURSE DESIGN AND PRODUCTIONType Description Advantages Disadvantages Intensive Studio Production Model  (Climate Literacy) Authors worked with a video director/producer for filming; studio post-production team created animations, graphics, and other dynamic post-production effects. ! Professionally produced videos with little media production expertise needed by the instructors. ! More expensive ! Less instructor control over process ! Production and post-production processes (including corrections and edits) can be slow. Hybrid Instructor/Studio Produced Model  (Systematic Program Design) Studio filmed “head shot” introduction and conclusion per lecture; used desktop model to record the lecture material segments, which were voice-over screen captures of a programming environment. Instructor does all editing and post-production, including annotations. ! Flexible, allows spectrum of video types to be produced;  ! Technical parameters (e.g., lighting, sound) handled by the studio while instructor retains control over aspects of production; ! Instructor has complete control over materials and format, enabling strong instructor presence. ! Instructors need time to gain media production expertise. ! Overall instructor time and resource needs can be high and challenging to track Light Studio Production Model (Game Theory I & II) Instructor filmed lecturing in front of a green screen, slides captured from tablet. Slide and instructor composited in post production. No graphics and animations; minimal post-production editing. ! Professionally filmed videos with no media production expertise needed by the instructors;  ! Little to no studio pre-production needed; ! Full process is fast and minimal time is needed for post-production phase. ! Rigid format with low customization capability.  ! Production and post-production processes (including corrections and edits) can be slow. Instructor-Produced Model  (Useful Genetics) Instructor and course team produced all video content using desktop equipment (external HD webcam, Wacom tablet for slide annotation, and lighting and sound equipment). Instructor and course team did all pre- and post-production for the videos using screen capture software (e.g. Camtasia). ! Instructors gain media production expertise and can create videos to fit their specific needs on their own schedule;  ! Studio costs low ! Instructor has complete control over materials and format, enabling strong instructor presence; ! Agile process, the videos can be created, edited, and refreshed quickly and in-expensively ! Instructors need time and support to gain media production expertise ! Technical details (lighting, sound, etc.) can be very hard to control outside of studio; ! Inconsistencies across different devices can make technical support and trouble-shooting difficult; ! Overall instructor time and resource needs can be high and challenging to track.  e s ription ntages is vantagesNon-video learning content included text-based module pages, quizzes, peer assess-ments, external resources, and program-ming files. Strategies for creating and imple-menting these materials varied across the courses, with some courses depending upon their instructors and graduate or undergraduate academic assistants to com-pletely manage this process while others relied upon CTLT instructional design and programming support staff. For example, the Climate Literacy instructors and gradu-ate academic assistants developed the content module pages and quizzes in a Word-documents format, which was then sent to the CTLT to be developed on the Coursera platform. After materials were implemented in the course shell, quality assurance testing involving both the CTLT and instructors was performed.  Additional quality assurance testing happened when the module opened.With the exception of Game Theory, instruc-tors chose to structure their courses through the creation of module pages for each week.  Although Coursera supported this approach, it was not a default of method of course organization; instead the platform organization default was based on separate auto-generated indexes of video lectures, quizzes, and assignments.  The use of module pages allowed instructors to better scaffold learning by providing clear path-ways and learning goals for each week.  Module pages included information on the amount of time needed to complete the module, learning goals, overview and infor-mation about the activities for that module, lecture videos, external readings and exer-cises, and additional quiz or assignment help.  7COURSE DESIGN AND PRODUCTIONCOURSE DELIVERYt.POJUPSJOHDPVSTFGPSVNTt3FTQPOEJOHUPTUVEFOURVFTUJPOTBOEDPNNFOUT        in forumst)PTUJOHMJWFXFFLMZiPóDFIPVSTwVTJOH(PPHMF        Hang-outst3FWJTJOHBOESFöOJOHDPVSTFNBUFSJBMTCBTFEPO        student feedbackt5SPVCMFTIPPUJOHDPVSTFNBUFSJBMJTTVFTSFQPSUFE        by learnerst&TDBMBUJOHQMBUGPSNBOEUFDIOJDBMJTTVFTUP        Coursera stafft0OHPJOHQSPEVDUJPOBOEJNQMFNFOUBUJPOPG        upcoming course modulesGraduate and undergraduate academic assistants were a crucial component in all courses. The “just in time” produc-tion model for these courses created a greater reliance upon the academic assistants in the design, development and delivery phases of all pilot courses. The academic assistants, under the direction of the instructor, supported the teach-ing of these courses by developing course materials, directly assisting learners, troubleshooting platform settings, performing QA testing, and many other activities.The scale of the pilot courses allowed for very little one-to-one interaction between instructors and learners; almost all course interaction happened in the course forums.  One of the major activities that the academics assistants performed was forum monitoring and moderating.  Across the pilot courses, the academic assistants had subject expertise and were generally encouraged to respond directly to learners in the forums or to bring activity to the attention of the instructor. 8COURSE DELIVERYAll courses produced approximately two to three weeks of complete course materials by the time the individual courses opened.  Thus the course teams continued to be in a development phase during the delivery of the courses, necessitating a “just-in-time” approach to the course devel-opment, production, and delivery.During delivery of the courses, the instructors, academic assistants, and CTLT support staff were involved in the following activities:Production and delivery costs for these courses fell broadly into the categories of video production (44%), academic assistance (24%), instructional design and web programming (15%), and content licens-ing (16%). Figure 1 provides an overview of the costs by category for all courses involved in the pilot.  Please see Appendix B for the specific costs per course. Please note, we are capturing only the direct costs; faculty time and strategic coordination support are not included. Figure 1: Overview of MOOC Pilot Costs by CategoryDEVELOPMENT AND DELIVERY COSTSAcademic Assistance,Development Phase11%Academic AssistanceDelivery Phase13%InstructionalDesign/ProjectManagement Support11%Course VideoProduction13%Course Video Pre-Production10%Promotional Videos3% Other1% Web ProgrammingSupport4%Course Video Post Production18%Content Licensing16%9DEVELOPMENT AND DELIVERY COSTSDEVELOPMENT ANDDELIVERY CHALLENGESChallenges encountered during the development and delivery of the pilot courses included:Partner Relationship ManagementThe pilot involved course teams working with various groups both within (e.g. UBC studio, CTLT) and external to UBC (e.g. Coursera, Stanford). There was a need to ensure that external contracts with Coursera and partner institutions protected and supported UBC instructors and the University. Addi-tionally, there were learning curves associated with adapting to partner workflows, which lead to increased development time.CopyrightCopyright posed a challenge in the development of pilot courses due to an inability to incorporate Library-licensed materials such as links to subscription-based journal articles, the lengthy duration involved in the permission seeking process, and a conservative approach to relying upon fair dealing exceptions. Instructors instead shifted to using self-created materials as well as open-access readings, textbooks, and resources. The process for finding appropriate and useful materials increased produc-tion times.Ongoing Platform Learning CurveIterative upgrades of the Coursera platform throughout the pilot resulted in shifts in the interface and functionality. Instructors and course teams had different levels of comfort with ambiguity around platform functionality.Time Costs and Need for AgilityThe pilot involved higher than expected time demands on all people involved.  The compressed time frame prompted a need to build out an agile development-support model in order to be able to respond to development needs rapidly.Academic Assistant Instructional SupportActivity in the MOOCs happened 24 hours a day and it was important for course teams to set clear schedules and expectations on academic assistant time.  Additionally, there was a need for scaffold-ing better academic assistant support through the development of support resources and training on core MOOC activities, such as forum moderation.10DEVELOPMENT AND DELIVERY CHALLENGESDEVELOPMENT AND DELIVERY LESSONSUBC’s past experience in developing traditional online courses informed the ability to develop pedagogically well-designed MOOCsUBC’s experience in instructional design for distance and blended learning provided a grounding for designing well structured courses that went beyond default MOOC models. Careful articulation of learning goals, well designed course activities, and structured course modules were especially impor-tant for with the diverse range of learners with different motivations, expectations, and learning styles. There are still opportunities to explore and implement instructional design models to better engage learners in the critical first two weeks of a course and beyond. Peer to peer support is also an important aspect of MOOCs and can be scaffolded through learner engagement and community building strategies.MOOC students were motivated by gradesEven though none of the courses in the pilot could be taken for credit, grades were an important motivating factor for many learners. Ungraded learning activities had lower participation rates than graded activities.Observations from the pilot included:Students appear to respond positively to instructor engagementWhile the scale of the pilot courses did not allow for direct one-to-one communication between instructors and individual learners, learners almost always responded positively to instructor engage-ment in the discussion forums and course announcements.  This engagement, although time consuming when done at scale, created real instructor presence within the course which supported the MOOC’s learning community development.  There was an inherent tension between scalability and rich assessment functionality in online course deliveryAlmost all instructors in the pilot scaled back the design of their courses as the Coursera platform did not support a way to assess the types of activities they wished to use. Formally assessing activities outside the platform was technically challenging and often relied upon using work around strategies based on Coursera’s peer assessment functionality. DEVELOPMENT AND DELIVERY LESSONS 11A motivation for instructors to develop MOOCs was the opportunity for instructors to “publish” their teachingInstructors in the MOOC pilot were interested in reaching a larger audience on important topics. Additionally, the instructors were deeply engaged in the pedagogical process of teaching. The oppor-tunity to expose their subject knowledge and expertise, as well as their teaching ability and practices,  to a greater audience was a motivating factor for instructors.MOOC instructors faced open discussion and critiques of their subject knowledge and teaching strategiesLearners in the MOOC pilot felt comfortable critiquing all aspects of course content, learning activi-ties and instructor performance. Additionally, MOOC participant demographics different from those of on-campus students and the pilot MOOCs included learners with subject knowledge and teaching expertise. Critical feedback came from both learners and peers (which could be the same individual).MOOC platforms are rapidly evolvingCoursera, as a platform, was in a state of constant, iterative change throughout the pilot, which required a moderately-high threshold for technological ambiguity and agility as features and processes are added or changed.  MOOCs require significant investmentsThese investments include faculty time, staff expertise, relationship building, and dollars that are difficult to capture.  In general, everyone involved with the MOOC pilot contributed significantly greater efforts and time to the project than was anticipated.DEVELOPMENT AND DELIVERY LESSONS 12LEARNER DEMOGRAPHICSThe profile of learners who engaged with the Coursera-based courses in the MOOC pilot mostly matched the overall profile trend for all Coursera users.  The learners were generally in their mid-to late 30’s, educated (60-70 percent having a bachelors degree or higher), and Caucasian (65-70 percent).  Approximately 30 to 40 percent of learners lived in North America, with the rest being from all parts of the world:181Useful Genetics had learners from countries186Climate Literacy had learners from countries191Systematic Program Design 1 had learners from countries202Game Theory had learners from countriesTable 5 provides an in-depth comparison of learner demographics in the Useful Genetics and Climate Literacy courses to Coursera’s averages.  Please note that in-depth demographic comparisons were not available for all courses due to this Coursera function-ality still being in a development phase during the delivery of the courses.LEARNER DEMOGRAPHICS 13Table 5: Comparison of Learning DemographicsLEARNER DEMOGRAPHICS 14Statistics Category Useful Genetics Percent Climate Literacy Percent Systematic Program Design 1  Percent Game Theory Percent Coursera Average Percent Gender      Female 55 50 21 22 40 Male 45 50 79 78 60       Age (in years)      13-19 4 2 3 3 n/a 20-29 42 34 43 48  30-39 26 26 32 30  40-49 11 13 14 11  50-59 8 11 6 5  60-69 6 9 2 2  70+ 3 3 1 1        Residence (location currently living)      United States 33 33 33 29 27 India  7 6 8 11 5 Canada 5 6 4 3 4 United Kingdom 4 5 4 4 4 Spain 3 2 3 2 4 Brazil 3 3 2 4 5 Russian Federation 3 1 3 3 2 Germany  2 3 3 2 2       Race      White or Caucasian 68 69 n/a n/a 67 South Asian 11 9   8 East Asian 6 6   7 Decline to State 5 7   8 Other Asian 5 5   5 Black or African American 3 4   5  Statistics Category f lticstateracytSyst iPro sign 1Percerytrsera AveragetTable 5: Comparison of Learning Demographics (cont’d)LEARNER DEMOGRAPHICS 15Statistics Category Useful Genetics Percent Climate Literacy Percent Systematic Program Design 1  Percent Game Theory Percent Coursera Average Percent Education      Bachelor’s Degree 31 35 36 35 34 Master’s Degree 29 33 30 35 31 Some College But No Degree 11 9 11 8 9 High School Diploma or Equivalent 8 5 8 8 7 Doctorate Degree 8 7 5 6 7 Professional School Degree 5 4 2 3 5 Some High School 4 2 3 2 2       Student Status      Not a student 62 69 66 61 68 Full Time Student 28 10 25 29 20 Part Time Student 9 20 10 10 11       Employment Status      Employed Full Time 41 45 55 55 54 Unemployed and Looking For Work 14 14 15 13 11 Employed Part Time 12 11 8 9 10 Unemployed and Not Looking for Work 11 8 8 10 6 Self Employed Part Time 5 6 4 4 5 Self Employed Full Time 4 5 5 5 6  Statistics Category f leticstateracytSyst iPr sign 1Pererytrsera AveragetCOURSE ACTIVITYTable 6: Aggregate Course ActivitiesCourse Activities Useful Genetics Climate Literacy Systematic  Program Design 1 Game  Theory  Total Videos Available 107 38 86 48 Total Video Lectures Streamed 258,666 173,870 689,935 1,169,252 Total Video Lectures Downloaded 438,760 110,071 735,645 1,139,415 Unique Video Lectures Watched 358,980 148,174 685,535 1,292,015 Total Quiz Questions Available 480 201 175 93 Total Module Quiz Submissions  49,537 33,335 56,524 241,558 Total Forum Activity (# of Threads, Posts, and Comments made) 14,078 33,055 21,059 12,403  Activities that took place outside of the Coursera platform included links to open-access readings and online activities, the use of a third-party only mapping platform to create a student produced global map of climate change impacts, the use of a third party tool to support learners in the creation and discussion of new genetics assessment questions, and a UBC hosted online game based environment to illustrate game theory concepts.Across the pilot course, there were higher levels of engagement with activities that were graded than optional or non-graded activities.  For example, Climate Literacy was the only course in the pilot in which forum participation was part of the grading policy, which resulted in a much higher rate of forum activity than the other courses.There was an inherent tension between some desired course func-tionality and the ability to both scale that functionality and to assess the related activity with large numbers of learners. For this reason, courses in the pilot mostly relied upon the default assess-ment and content delivery functionality available through Cours-era, with some optional, non-assessed activities taking place outside the system. Table 6 provides an overview of the amount of activity that took place within the Coursera framework:COURSE ACTIVITY 16ticsliiAll courses in the pilot followed the same general trend for student engagement over time:t A high percentage (45-60%) of the people who enrolled in a course before it started never logged         into or engaged with the course once it launched. t There were steep declines in engagement with course activities during the first two weeks of the         course. t By the third week of a course, these declines became more gradual and flattened during the final         weeks of delivery.These figures can be seen in Figures 2 and 3, which show the amounts of students who watched lectures and took quizzes per week on the Coursera platform.Figure 2: Number of Students Who Watched Lectures Per Week125,00025,0005,0001,000Week 1      Week 2     Week 3      Week 4      Week 5      Week 6      Week 7     Week 8     Week 9     Week 10         Useful Genetics Climate Literacy Systematic Program Design Game TheoryCOURSE ACTIVITY 17Figure 3: Number of Students Who Submitted Quizzes Per WeekWeek 1        Week 2         Week 3      Week 4          Week 5        Week 6        Week 7        Week 8        Week 9       Week 10         Useful Genetics Climate Literacy5002,50012,50062,500Systematic Program Design Game TheoryCOURSE ACTIVITY 18* The week 6 quiz for Systematic Program Design was not formally assessed.*MOOCs AS OPEN EDUCATION RESOURCES AND ON-CAMPUS REUSEThe instructors in the pilot valued the potential for reuse of their MOOC-based learning materials produced at UBC and beyond. Four of the courses (Game Theory I and II, Systematic Program Design, and Useful Genetics) have already, or will soon, incorporate their MOOC materials into their UBC credit bearing classes in a flipped classroom model.  These materials will be, or have already been, used by hundreds of UBC students across multiple sections of the credit-bearing courses.All instructors incorporated an open access approach to their courses.  This was accomplished by:MOOCs AS OPEN EDUCATION RESOURCES AND ON-CAMPUS REUSELeaving their courses active and open for enrollment on CourseraEven though these courses have ended, the instructors have set their courses to be continually accessible. Individuals, who have signed up for a Coursera account, may still enter the course and engage with the materials and activities.  Although they will not be able to earn a certificate, they will still be able to learn from the resources in a self-paced model.Porting their content outside of the Coursera platformCourse teams provided alternative access to their learning materials by duplicating their content on spaces outside of the Coursera platform. For example, all courses added their lecture videos to YouTube, which allowed learn-ers to both watch their videos without needing to create a Coursera account and to embed their videos in other spaces such as a learning management system or website.Encouragement of reuse through clear license statementTwo of the courses (Climate Literacy and Useful Genetics) added Creative Commons licenses to their materials, which explicitly allows for the reuse of those materials without the need for advanced permission as long as the terms of the licenses are met (such as non-commercial use only). 19CONCLUSIONCONCLUSIONUBC is internationally recognized for its pedagogical innovation and technology-enhanced learning initiatives.  By developing and deliver-ing the four non-credit courses through the Coursera platform, UBC had the opportunity to explore, learn about, and conduct experi-ments in the rapidly evolving on-line teaching and learning space occupied by MOOCs.  The University has many decades of experience with continuing education and distance education; however, MOOCs afford unique opportunities to examine new modes of scalable, open course delivery.The MOOC pilot helped inform new course development and delivery processes and strategies for learner-centred activities in the face-to-face classroom to support flexible learning options for learners who are registered in UBC courses. These strategies include new under-standings of different production models for creating media-enhanced learning such as flipped classrooms, agile approaches to course delivery methods, and instruction design strategies to better scaffold self-based, peer-based, and open learning efforts. Finally, the pilot provided UBC with an opportunity to gain experi-ence with MOOCs in the company of peer institutions, and in a fashion that allowed UBC to make a contribution to a growing inter-national movement towards open access to educational materials. Through this effort, UBC was able to expose a large number of Cana-dian and International learners to the quality of UBC instructors and course materials. 20APPENDIX A: SPECIFICCOURSE OVERVIEWS21 APPENDIX AUSEFUL GENETICSLengthCourse TitleInstructorCourse SummaryStart DateEstimated WorkloadSocial MediaImpactsRecommendedBackgroundCourse ElementsVideo StatisticsAssessmentStatisticsGrading PolicyStatements of AccomplishmentThresholdsUseful GeneticsRosemary RedfieldUseful Genetics aims to teach ordinary people the genetics they will actually use. The course focuses on gene function and inheritance and provides a solid understanding of genetic principles that helps students understand recent issues, but also future challenges.May 1, 201312 Weeks6-8 hours/weekTweets:  181Google+: 68Facebook Likes: 989No prerequisites Prior knowledge of basic biology helpfult7JEFPTMFDUVSFTXJUIJOWJEFPRVJ[[FTt0QFOBDDFTTSFBEJOHTt4FMGUFTUQSBDUJDFQSPCMFNTt.PEVMFRVJ[[FTt.JEUFSNRVJ[t'JOBM&YBNt0QUJPOBMPOMJOFMFBSOJOHBDUJWJUZCBTFEJO1FFS8JTFt5PUBMPG7JEFPTt5PUBMMFOHUIPGWJEFPTNJOTFDt-POHFTU7JEFPNJOTFDt"WFSBHFNJOTFDt5PUBMPG2VFTUJPOTt(SBEFEPG2VFTUJPOTt6OHSBEFEPG2VFTUJPOTt.PEVMFRVJ[[FTt.JEUFSNt'JOBM&YBNt1BTTJOH t%JTUJOHVJTIFELe thCourse TitleInstructorCourse SummaryStart DateEstimated WorkloadSocial MediaImpactsRecommendedBackgroundCourse ElementsVideo StatisticsAssessmentStatisticsGrading PolicyStatements of AccomplishmentThresholdsClimate Literacy: Navigating Climate Change Conversations4BSBI#VSDI4BSB)BSSJTClimate Literacy is an introduction to the basics of the climate system, models and predictions, human and natural impacts, mitigative and adaptive responses, and the evolution of climate policy. In the course students will learn the effect of human fossil emissions and land use DIBOHFTCFBCMFUPFYQMBJOUIFFWJEFODFBUUSJCVUJOHUPHMPCBMXBSNJOHCFDBVTFPGIVNBOTBOECFBCMFUPFYQSFTTJOGPSNFEPQJOJPOTPOUIFXPSLCFJOHEPOFUPNJUJHBUFBOEBEBQUUPclimate change.May 20, 201310 WeeksIPVSTXFFLTweets: 261Google+: 112Facebook Likes: 1.3kNonet7JEFPTMFDUVSFTXJUIJOWJEFPRVJ[[FTt0QFOBDDFTTSFBEJOHTBOEBDUJWJUJFTt2VJ[[FTt1FFS3FWJFXFEXSJUUFOBTTJHONFOUTt'JOBMFYBNt0QUJPOBMPOMJOFBDUJWJUZ%PDVNFOUJOHDMJNBUFDIBOHFJNQBDUTPO$MJNBUF-JUFSBDZ.BQt5PUBMPG7JEFPTt5PUBMMFOHUIPGWJEFPTNJOTFDt-POHFTU7JEFPNJOTFDt"WFSBHFNJOTFDt5PUBMPG2VFTUJPOTt(SBEFEPG2VFTUJPOT t6OHSBEFEPG2VFTUJPOTt1FFS"TTFTTNFOUTt2VJ[[FT	5PQPVUPGRVJ[[FTtQFFSBTTJHONFOUTt1BSUJDJQBUJPOJOUIFGPSVNt'JOBM&YBNt1BTTJOH t%JTUJOHVJTIFELengthSuggested Readings None22 APPENDIX ACLIMATE LITERACYGAME THEORY23 APPENDIX ACourse TitleInstructorCourse SummaryStart DateEstimated WorkloadSocial MediaImpactsRecommendedBackgroundCourse ElementsVideo StatisticsAssessmentStatisticsGrading PolicyStatements of AccomplishmentThresholdsGame TheoryGame theory is the strategic interactions among self-interested agents. The essential tool is to understand human interactions from auctions to international conflict. This course is an intro-EVDUJPOUPUIFCBTJDTPGHBNFUIFPSZTVDIBTSFQSFTFOUJOHHBNFTBOETUSBUFHJFTUIFFYUFOTJWFGPSNHBNFUSFFT#BZFTJBOHBNFTSFQFBUFEBOETUPDIBTUJDHBNFTBOENPSF$MBTTJDHBNFTand real-world applications are used to reinforce lessons.+BOVBSZ8FFLTIPVSTXFFL5XFFUT Google+: 1kFacebook Likes: 10kt7JEFPTMFDUVSFTXJUIJOWJEFPRVJ[[FTt.PEVMF2VJ[[FTt0OMJOF-BC&YFSDJTFTt1SPCMFN4FUTt'JOBM&YBNt4DSFFOTJEF$IBUTt5PUBMPG7JEFPTt5PUBMMFOHUIPGWJEFPTNJOTFDt-POHFTU7JEFPNJOTFDt"WFSBHFNJOTFDt5PUBMPG2VFTUJPOTt(SBEFEPG2VFTUJPOTt6OHSBEFEPG2VFTUJPOTt0OMJOFMBCFYFSDJTFTt.PEVMF2VJ[[FTt'JOBM&YBNt1BTTJOH t%JTUJOHVJTIFELength,FWJO-FZUPO#SPXO	6#$.BUUIFX0+BDLTPO	4UBOGPSE6OJWFSTJUZ:PBW4IPIBN	4UBOGPSE6OJWFSTJUZt$PNGPSUBCMFXJUINBUIFNBUJDBMUIJOLJOHBOESJHPSPVTBSHVNFOUTt-JUUMFTQFDJöDNBUIt-JHIUXFJHIUQSPCBCJMJUZUIFPSZt-JHIUXFJHIUDBMDVMVTCourse TitleInstructorCourse SummaryStart DateEstimated WorkloadSocial MediaImpactsRecommendedBackgroundCourse ElementsVideo StatisticsAssessmentStatisticsGrading PolicyStatements of AccomplishmentThresholdsIntroduction to Systematic Program Design 1Computers appear everywhere in our everyday lives, therefore the ability to design a program or at least communicate effectively with those that do is an important skill to have. This course focuses on a design method that teaches students how to systematically design a program.  %VSJOHUIFDPVSTFTUVEFOUTXJMMMFBSOIPXUPVTFQSPHSBNMBOHVBHFTBOEMJCSBSJFTBOEEFTJHODPNQMFYQSPHSBNUIBUDBOCFSFBECZOPUPOMZBDPNQVUFSCVUBMTPCFVOEFSTUPPECZQFPQMFJune 3, 20138 WeeksIPVSTXFFL5XFFUT (PPHMFFacebook Likes: 1.6kt7JEFPTt"TTJHOFEEFTJHOQSPCMFNTt4NBMMQSPKFDUTt'JOBMFYBNt5PUBMPG7JEFPTt5PUBMMFOHUIPGWJEFPTNJOTFDt-POHFTU7JEFPNJOTFDt"WFSBHFNJOTFDt5PUBMPG2VFTUJPOTt(SBEFEPG%FTJHO1SPCMFNCBTF2VFTUJPOTt6OHSBEFEPG2VFTUJPOTt1FFS"TTFTTFE1SPKFDUt)PNFXPSL	5PQPGt1FFS(SBEFE1SPKFDUt'JOBM&YBNt1BTTJOH t%JTUJOHVJTIFE(SFHPS,JD[BMFTNoneLengthSYSEMATIC PROGRAM DESIGN24 APPENDIX A/PUFEJSFDUDPTUTPOMZFTUJNBUFTEPOPUJODMVEFGBDVMUZUJNFBOEDPPSEJOBUJPOTVQQPSUAPPENDIX B: COURSEEXPENDITURE COMPARISONS25 APPENDIX BCost Category Useful Genetics Climate Literacy Systematic Program Design 1 Game Theory I & II Pilot Total Course Length 11 weeks 10 weeks 8 weeks  weeks /  weeks  Project Support      Content Licensing 10,000 10,000 10,000 10,000 0,000 Instructional 4upport, %evelopment Phase 8,8 2,10 ,0 ,8 2,210 Instructional 4upport, %elivery Phase 11,0 2,10 ,0 ,8 26,99 ProKect Manager/ Instructional %esigner 1,00 680 20 1080 22,680 Web Programmer 6,120 100 0 0 ,820 Graphic %esign 0 301 301 0 602       Media Production      Promotional 7ideo 1,00 1,00 1,00 1,00 6,000 %evelopment Meetings 10 0 0 0 10 Location Consulting 1,020 0 0 0 1,020 Course 7ideo Pre-Production 0 20,000 680 30 21,020 Course 7ideo Production  0 20,000 ,100 3,000 28,100 Course 7ideo Post Production 0 20,000 ,9 13,2 38,00       TOTAL $53,905 $84,321 $38,216 $41,215 $217,657        XFFLT XFFLT             *OTUSVDUJPOBM4VQQPSU%FWFMPQNFOU1IBTF    1SPKFDU.BOBHFS*OTUSVDUJPOBM FTJHOFS     1,080 22,680      (SBQIJD FTJHO FWFMPQNFOUeeti s 0 0 0 $PVSTF JEFPre-Production0 20,000 680  21,020$PVSTF7JEFPPost Production0 20,000         TOTAL , , , , ,APPENDIX C: POSITIVESTUDENT FEEDBACKThe following is a sampling of positive learner comments from the course discussion forums.  “ *U XBT POF PG UIFNPTU JOUFSFTUJOH JOGPSNBUJWF CFTU QSFTFOUFE BOE FOKPZBCMFDPVSTFT*hWFFWFSUBLFO	#4&&.#".ZDPNQMJNFOUTPOUIFPVUTUBOEJOHPSHBOJ-[BUJPOQSFTFOUBUJPOBOEUFDIOJDBMNBOBHFNFOUPGUIFDPVSTFNBUFSJBMTBOEUIFwebsite/forums.26 APPENDIX CThis class has been interesting, informative, stimulating, and a wonderful 10-week KPVSOFZUIBUIBTDIBOHFEFYQBOEFENZMFWFMPGUIJOLJOHJOTPNBOZXBZT*SFBMMZFOKPZFEUIFEFUBJMBOESJHPSPGUIJTDPVSTF5IJTXBTNZöSTUDPMMFHFMFWFMTDJFODFDPVSTFJOZFBSTTPNVDIPGUIFNBUFSJBMXBTDPNQMFUFMZOFXUPNF*GPVOE JU WFSZ FOHBHJOHMZ BOE DMFBSMZ QSFTFOUFE 5IF FYQFSJFODF PG UIF TUBòQBSUJDVMBSMZ%S3FEöFMESFBMMZTIPXFE*BMTPBQQSFDJBUFWFSZNVDIUIFXBZZPVengaged with us in the forums - fair and very reasonable. It seemed clear to me UIBUZPVXFSFBMMWFSZNVDIJOUFSFTUFEJOVTMFBSOJOHUIFNBUFSJBM"MPOHUIFXBZI found myself understanding more about how to think like a geneticist, and perhaps more importantly where the limits of my knowledge are. I want to make note that I believe the quality of the course and the knowledge imparted are on par or superior to the best undergraduate courses I have taken. "MSFBEZ*GFFMDPNQFUFOUUPVOEFSTUBOEJOGPSNBUJPOBCPVUHFOFUJDTBöFMEUIBUis increasingly in the public domain and important for the understanding of FYQFSJFODFTJONZEBZUPEBZMJGF“ *O UIJT DBTF * FOKPZFE MFBSOJOHCFDBVTFCPUI UIF JOTUSVDUPSTBSF SFBMMZHPPE JOtheir pedagogy techniques and also, they came down to a lay man level to teach UIBUJTTPNFUIJOHUSFNFOEPVTMZFMFNFOUBSZGPSUIFN"OEZFUJUXBTEPOFXJUIperfect ease and infinite patience with all our problems.27 APPENDIX CI would dare to say that this is by far the best course I've taken in coursera so far. It's really complete, I've seen great follow up from the teachers and great interac-tion in the discussion forums. I would be soooo happy if the teachers ever decided UPPòFSNPSFDPVSTFTUPCSPBEFOUIFUPQJDPSPOEJòFSFOUUPQJDT"GUFSUIJT *hWFFWFOCFHBOUPDPOTJEFSBNBTUFShTEFHSFFGSPN6#$*IBEOhUFWFOUUIPVHIUBCPVU6#$FSFWFO$BOBEBCFGPSFUIJT"TBOEZFBS UFBDIFSy *BNTPHMBE * UPPL UIF UJNF UP UBLF UIJT DMBTT *IBWFlearned so much and gained valuable resources which I will use this fall with my students. In addition, I teach a blended course... and am very impressed by the depth of your course as well as by the "online course specs". Love how the mod-VMFT IBWF CPUI UFYUVBM BOE WJEFP BTQFDUT XIJDI UJF UP QSFWJPVTknowledge/modules and prep for future ones. I also liked the variety of additional resources for each model: some videos, some websites, some documents, some TJNVMBUJPOT FUD /JDF WBSJFUZ UP LFFQ JU JOUFSFTUJOH 	QMBO PO TUFBMJOH TPNF PGUIFTFBTQFDUTUPBEEUPNZPOMJOFTJUFXIFO*HFUUIFUJNF*XBTBMJUUMFTLFQUJDBMBCPVU.00$TBOEUIFRVBMJUZPGUIFFEVDBUJPOQSPWJEFECVU*DPVMEOhUIBWFQSPWFENPSFXSPOH5IJTDPVSTFXBTWFSZXFMMPSHBOJ[FEUIFNBUFSJBMT	CPUIWJEFPBOESFBEJOHTEFUBJMFEBOEFOHBHJOHBOEUIFBNPVOUPGstudy time affordable ... I feel I've learned so much after only 10 weeks of studying UIJTDPNQMFUFMZOFXTVCKFDUUPNFBOEOPX*GFFMNPSFDPOöEFOUXIFOEJTDVTT-ing climate change and policy with others. I think I have all the necessary frame-work, and I'm sure it's going to be a perfect foundation for more in-depth SFTFBSDI BOE NBZCF XIP LOPXT FWFO DPOUJOVJOH TUVEZJOH UIJT TVCKFDU BUVOJWFSTJUZCLIMATE LITERACY“ Congrats to the professors. I wish all courses in Coursera would be like this one!28 APPENDIX CYour course has given me great insight into this world of math, calculus or what-FWFSZPVBSFVTJOHJOUIFTFFYQMBOBUJPOT"/%IBTTIPXONFUIBUBMJGFMPOHGFBS-factor called math- really can be interesting and informative if taught in an inter-esting wayGame Theory was my first course at Coursera and set my standards really high BCPVUUIFOFYUPOFT5IJTJTUIFöSTUPOMJOFDPVSTF*hWFUBLFOBOEJUhTCFFOBHSFBUFYQFSJFODF*UhTOPUBCPVUBTJNQMJöFEWFSTJPOPGBUPQJD	XFhWFHPUQMFOUZPGCMPHTXJLJQFEJBFOUSJFTBOEXFCDPNJDT GPS UIFQVCRVJ[BOTXFS UP hXIBU JTHBNF UIFPSZhCVUBCPVUapplying diligent academic rigour to learn something a bit deeper about a new topic of study.GAME THEORYThis is course is very well-done. The content is well structured and gives a broad UBTUFPGXIBUHBNFUIFPSZJTBCPVUMFDUVSFTBSFWFSZOJDFUIFFYFSDJTFTBMUIPVHIBCJUTJNQMFBSFOPUKVTUDPQZQBTUFPGUIFFYBNQMFTHJWFOJOUIFMFDUVSFTTPUIBUPOFDBOUIJOLBMJUUMFBOEUIFUJNJOHJTFYDFMMFOU“ This course has really helped me to understand how to "Think" like a programmer, and I am certain I will carry what I have learned through to many more courses.  The lectures are easy to follow and after finishing the practice problems every-thing makes perfect sense.  It is a very well designed course, even for beginner QSPHSBNNFST#ZGBSUIFCFTU.00$*IBWFUBLFO29 APPENDIX C*IBWFEPOFBGFX.00$TIFSFBOEPO6EBDJUZCVU*hMMIBWFUPBENJUUIJTJTPOFPGthe best in terms of course structure, design, teaching methodology, evaluation strategies, and a whole lot of other stuff.I have learned more in this class than I have done in any other class for a while - and I'm especially happy because my previous attempts to learn programming in BOZXBZIBWFOPUCFFOTVDDFTTGVM&WFOUIPVHI*hNQVUUJOHBMPUPGUJNFJOUPJU*often look forward to working on the problems and I'm gaining a lot of confi-dence. I already participated in most of programming course of Coursera and I surely can TBZUIBUUIJTDPVSTFJTKVTUUIFCFTU/PX*LOPXoOPUFYBDUMZOPUTPNFIPXDMPTFhow to think like a programmer and how to approach a problem in order to take it apart and make a solution pattern and lots and lots of more.SYSEMATIC PROGRAM DESIGN30ACKNOWLEDGEMENTSACKNOWLEDGEMENTSReport written by: 8JMM&OHMF4USBUFHJTU0QFO&EVDBUJPO*OJUJBUJWFT$FOUSFGPS5FBDIJOH-FBSOJOH5FDIOPMPHZWith support from:4JNPO#BUFT8FOEZ$IBO.JDIFMMF-BNCFSTPO+Fò.JMMFS*EP3PMMAcademic Teams:           Useful Genetics%S3PTFNBSZ3FEöFME"MBOB4DIJDL7JSHJOJB8PMPTIFO           Climate Literacy%S4BSBI#VSDI%S4BSB)BSSJT+VTUJO3JUDIJF,BUF-F4PVFG           4ZTUFNBUJD1SPHSBN%FTJHO*1SPG(SFHPS,JD[BMFT&OOBT"CEVTTBMBN%JBOB#VMGPOF-ZOTFZ)BZOFT5IPNBT-FVOH7JDUPSJB-PXFFO++/VOF[5IFB4JNQTPO.JDIBFM4VO           Game Theory%S,FWJO-FZUPO#SPXO%S.BUIFX0+BDLTPO%S:PBW4IPIBN&SJD)VBOH)PPZFPO-FF"ESJBO.BSQMF%BWJE5IPNQTPO+BNFT8SJHIUProject, Instructional, and Development Support:4JNPO#BUFT$ISJT$SPXMFZ4BFFE%ZBOBULBS8JMM&OHMF#SVDF.BSDIGFMEFS           Michelle Lamberson, Jeff Miller, William Kody, Jason Myers, Nicole Ronan, Chelsea Thompson&YDFQUXIFSFPUIFSXJTFOPUFEUIJTXPSLJTMJDFOTFEVOEFSB$SFBUJWF$PNNPOT"UUSJCVUJPO*OUFSOBUJPOBM-JDFOTF6#$MPHPTDPBUPGBSNTBOESFMBUFECSBOEJOHFMFNFOUTBSFDPQZSJHIUPG5IF6OJWFSTJUZPG#SJUJTI$PMVNCJBBOENBZOPUCFBMUFSFEJOBOZXBZPSSFVTFETFQBSBUFGSPNUIFXPSLXJUIPVUQFSNJTTJPO'PSHVJEFMJOFTPOUIFQFSNJUUFEVTFTPG6#$USBEFNBSLTQMFBTFSFGFSUP”


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items