A Tale of Two Students

Kieran Mathieson
October 27, 2011
kieran@coredogs.com

Education reform. Good stuff. How to do it? Here's one way.

Like it? Hate it? Ideas to improve? Want to do it yourself? Send me an email. Look me up on Google+.

November 5, 2012: The "Textbook Writer" referred to below is coming together - slowly - at Dolfinity.Com.

Mike and Eric

Mike and Eric are twins. Both are students at Public University. They want to learn how to make Web sites. They look at the course catalog, and see:

WEB 100. Students learn to make basic Web sites, using HTML, CSS, and JavaScript.

There are two sections of WEB 100. The section taught by Professor Trad Ishenul fits Mike's schedule, so he signs up for it. The section taught by Professor E. Fective fits Eric's schedule, so he signs up for that one.

Let's follow Mike and Eric through the semester. We'll get some help from Ann, a graduate student in educational psychology. She's doing her thesis on learning in skill-based classes, and is comparing the two sections of WEB 100.

Ishenul's course

Textbook

Prof. Ishenul assigns a traditional textbook from an academic publisher. The book covers all of the HTML, CSS, and JavaScript the professor thinks that students should know about. That's a lot of stuff, so the book is a heavy 1,043 pages. It costs $180 at the Public University book store. Mike shells out the cash, but isn't happy with the price, or the back-breaking weight of the book.

Book weighs as much as a cow

Yes, the book weighs as much as a cow. A small cow, but still.

Here's the book's table of contents:

Chapter 1. Introduction
Chapter 2. The history of the HTML, CSS, and JavaScript standards
Chapter 3. Basic HTML tags
Chapter 4. <img>, <a>, and <table>
Chapter 5. Floating page layouts
Chapter 6. Basic CSS rules
Chapter 7. Advanced CSS selectors
Chapter 8. JavaScript variables
Chapter 9. Conditional and looping statements
Chapter 10. The DOM
Chapter 11. Conclusion

The book is organized by technology: HTML, then CSS, and then JavaScript. The table of contents only makes sense to someone who already knows Web technology.

Course operation

It's a traditional lecture course. Class meets three times per week for an hour. Prof. Ishenul arrives early, and is available after class to answer questions. He also has office hours twice per week.

Prof. Ishenul is a diligent teacher. He makes PowerPoint slides for each class, being sure to summarize every topic in the book. When he lectures, he goes over examples. He always asks if there are any questions.

Assessment

Prof. Ishenul assigns two projects, and two exams. Students do the projects in teams of three. Prof. Ishenul asks students to rate each other's contribution to the team.

The exams are multiple choice. Prof Ishenul uses a question bank that came with the textbook.

Ann interviews Prof. Trad Ishenul

Ann: How did the course go this semester?

Ishenul: (Shrugs) OK. I put a lot of work into it, and make sure the students do, too. Web development is a big topic.

Ann: What do you do to help students learn?

Ishenul: I show many examples in class. I show code, then run through it in detail. I cover everything they need to know about the Web, at least in a first course. I always ask students if they have questions.

Ann: Any frustrations?

Ishenul: Yes! Most students just sit there when I ask if there are questions. One or two ask, but the rest are lazy. When I ask them questions, I can tell they haven't read the textbook. Oh, and hardly anyone comes to my office hours.

Ann: Do they know how to make Web sites?

Ishenul: Well, the projects are OK. The exams scores are all over the place.

Ann: Do the students remember what you taught?

Ishenul: (Sigh) No, they don't. When Prof. Moriarty sees them in the second Web development course, they've forgotten everything. They don't know how to make a simple page. It's like they never learned how.

Ann interviews Mike

Ann: How did the course go this semester?

Mike: Not good. The professor went so fast, it was hard to keep up. His examples were complicated. I didn't understand them.

Ann: Did you ask questions about the examples?

Mike: I was so confused, I didn't know what questions to ask.

Ann: Did you read the book before class?

Mike: Sometimes. But there was too much to read. Maybe 50 pages per class. Usually, I only had time to skim. When I did read, there was so much going on that I got confused.

Ann: How about the projects?

Mike: People goofed off. Jim, on my team, left all the work to the rest of us. I didn't want to grade him low on participation, because we're going to be in other classes together.

Ann: Did Jim learn much?

Mike: No, but he got a good grade. He pulled all-nighters before the exams. I don't think he remembers much now, though. Don't ask him to make a Web site for you!

Ann: Do you how to make a Web site?

Mike: I hate to say it, but no. I know some HTML tags and stuff, but I don't know how to use them to build a site. Maybe I'm just not cut out for this.

Fective's course

Let's look at the other section of WEB 100. Eric took the course from Prof. E. Fective. Was his experience different from his brother's?

Textbook

Prof. Fective uses CoreDogs, an online enhanced textbook. CoreDogs was written and published by Dr. Author, a professor at another university. Dr. Author used Textbook Writer to create CoreDogs. Textbook Writer is Web-based software that helps people write online textbook. Like this:

Overview

Textbook Writer is open source. It supports dozens of languages; authors can write in their own language, for their own culture. [Your foundation here] sponsored the development of Textbook Writer.

Dr. Author charges $19.95 to use CoreDogs. Although Textbook Writer is free, Author spent hundreds of hours creating the textbook, CoreDogs. She updates the textbook continuously, with new examples, links to current news stories, etc.

CoreDogs does not cover every topic in Web development. Instead, it covers just the core concepts that novice developers need to know about. Those topics it talks about in depth, explaining how to use them to create useful sites.

Here is the textbook's table of contents:

Table of contents

There are no technical terms in the table of contents, because CoreDogs is not organized by technology. Instead, CoreDogs is organized around tasks, like "Make a Web page with text."

Here are the topics in the "Web page with text" chapter. It includes a nontechnical lesson on "Writing for the Web":

A chapter

Here's what the lesson "Writing for the Web" covers:

Writing for the Web

"Writing for the Web" is not a lesson on technology. It's a lesson about the words that are on the page: using short sentences, simple words, active voice, lists, etc. Students should do these things if they are going to meet the goal: create good text pages.

A blended course

Prof. Fective's course is blended: part online, and part face-to-face. These are also called hybrid courses.

The online component

Students work through CoreDogs on their own. They complete about 60 exercises during the semester. The exercises are embedded in the textbook; there is no separation between reading and doing. Here's an example:

Exercise

Students submit their work through the textbook itself. Prof. Fective looks at each submission within a day or two. If a student's work isn't correct, she asks the student to fix it and resubmit. Students resubmit until they have it right.

The face-to-face component

Prof. Fective's class meets for only an hour per week. She doesn't lecture. Instead, she gives students problems to work on, in groups of two. She walks around the computer lab, helping as needed.

Assessment

Students get points for CoreDogs exercises. The CoreDogs exercises are worth 0.4% each, up to a maximum of 20%. Students also do three projects, and two exams.

Each student creates a personal branding Web site, called an eMe, to help them in their future job search. CoreDogs lessons help students gradually build their eMes.

The three projects are different versions of eMes. The first one is a few basic text pages. The second project adds formatting, images, and simple navigation. The final project extends the eMes to include work samples, slide shows, quizzes, and other features. By the time they are done, each student has a complete personal branding site.

The exams are held in a computer lab. Prof. Fective gives the students a task, and an hour to complete it. They can consult any non-living source during the exam. They can bring in books, notes, use Google, but they cannot call a friend.

Ann interviews Prof. E. Fective

Ann: How did the course go this semester?

Fective: Quite well. The eMes were very good. They'll help students in their job search.

Ann: Students mainly work alone. Aren't they isolated?

Fective: Not at all. Students work independently, but not alone. They're submitting about five exercises every week, and I give them individual feedback. This is a chance for a conversation about the work. In the weekly face-to-face session, I work with students one-on-one.

I get to know the students better than I used to when I taught a lecture class. When you're in a lecture, there's not much one-on-one time. But with this flipped approach, I get to spend more time with individuals, online and in person.

Ann: Flipped?

Fective: Yes, that's what this approach is called, where students work independently with a book or some videos, and then solve problems in class. You've heard of the Kahn Academy?

Ann: Yes, it's in the news.

Fective: Right. Mr. Kahn has people excited about flipped. But it doesn't have to be done with videos, the way Mr. Khan does it. In fact, I would say that CoreDogs does the independent study phase better than Khan Academy.

Ann: That's quite a claim.

Fective: Yes, but I think it's true. With Khan, students watch videos, passively. CoreDogs has all those exercises, built into the reading. CoreDogs is about doing, not watching.

Ann: Do students like it?

Fective: It varies. Some students are used to drifting through classes. But CoreDogs keeps them on-task. They have to work every week. They're surprised by the amount of work CoreDogs takes, and don't care for it.

The serious students like CoreDogs. Some tell me WEB 100 is the best course they've ever taken.

Ann: Could a student get a good grade by just cramming before the exam?

Fective: No. The exams are like the exercises in CoreDogs. Students have to build something. They're in a computer lab, with a fixed amount of time. They have to know what they're doing. That only comes with practice.

Ann: Speaking of all those exercises. You have 40 students, and 60 exercises. That's 2,400 things to grade. Do you do anything else?

Fective: (Chuckles) The grading takes time, that's for sure. But less than you might think. CoreDogs has a smooth interface for grading, designed so grading can be fast. Really fast. I can grade a basic solution in less than 30 seconds.

I find that if I spend about an hour every day, I can keep up.

Ann: Is all that grading worth it?

Fective: You bet! Web development is about building things, and that takes practice. CoreDogs makes sure students practice, practice, and then practice some more.

Ann: Couldn't the computer grade things automatically?

Fective: No. Well, it could grade multiple choice questions, but that's not a good measure of student actual skills. If you want students to learn skills, you have to bite the bullet. Do a lot of grading.

If you take learning outcomes seriously, there's no way around it. That's what you have to do.

Ann: Is all that grading productive?

Fective: Productivity is what CoreDogs is about. I could give lectures, but that doesn't use my time well. Students could just as easily read a book, or watch a video of a lecture. Instead, I spend my time giving feedback, and helping students in person. Things you need an expert for.

It's productive for students, too. No busy work, no puzzling through dense academic prose. Every hour students spend with CoreDogs is an hour spent learning.

Ann: I noticed that each exercise contributes a fraction of a percent to the final grade for the course. Why so little weight?

Fective: Cheating. The only way you can prevent cheating is to have students take an exam in a physical place you control. Students work on CoreDogs when they want, where they want. That's good. It gives them flexibility. But you can't stop friends emailing exercise solutions, and copy-and-pasting them in.

Ann: OK, I got it. The low weight makes cheating of little value. But some students will cheat anyway.

Fective: Yes, they will. I can't stop them, in any practical way. But I can show them what happens when they don't do the work. When they cheat, or just don't turn anything in.

CoreDogs keeps a lot of data about student performance. Here's a graph for a student. It shows he did many exercises:

High score

The light-colored lines in the background are quartiles for the class. Just ignore them if you don't know what that means.

There was an exam around day 45. This student did well. He got 20 out of 20. The only student to score that high.

Here's the graph for another student:

Low score

This student stopped doing exercises about two weeks into the course. His score on the exam: 4 out of 20.

So, the student who did many exercises got 20 out of 20. The student who did few exerercises got 4 out of 20.

Is it clear what happens to your grade when you don't do the exercises?

Ann: (Laughs) Yes, it couldn't be more clear. Is this real data?

Fective: Yes. It's from Fall, 2009. CoreDogs has been in use for a few years.

Ann: This data is from a CoreDogs report. Do the reports help in other ways?

Fective: Oh, yes. I expect students to start submitting exercise solutions in the first week of the semester. If I see a student hasn't submitted anything by week 3, I contact him or her, and check up. They quickly find out that I'm watching.

Ann: I have to ask you... In my graduate work, I've learned much about learning science research. The way you teach WEB 100, it matches best practices from learning science, better than any course I've seen. You must have spent years studying the learning science literature.

What made you do that? After all, you're a technology professor.

Fective: I'd like to take the credit, but I can't. This way of learning is built into CoreDogs. I didn't study learning science at all. I just use CoreDogs the way it was designed.

Ann: So Dr. Author, who wrote CoreDogs, did all the learning science work.

Fective: Well, no. I spoke to him about this. He wrote CoreDogs with software called Textbook Writer. Textbook Writer helps people write CoreDogs-like books. It has the feedback mechanism, reporting features, all that stuff built in.

Don't get me wrong. It's a lot of work to write a book like this. It took Dr. Author hundreds of hours. But Textbook Writer gave him an infrastructure. Because he used Textbook Writer in the right way, he ended up following the learning science principles you learned about.

To tell the truth, I don't even know what those principles are. But I don't need to. All I know is that when I use this approach, I'm happier, and the students learn more.

That's the bottom line, isn't it?

Ann interviews Eric

Ann: How was WEB 100?

Eric: Tough. More work than I expected. Stuff to turn in every week. You had to keep up.

I got behind twice. The prof emailed me both times, to check up. That's never happened before.

Ann: Never?

Eric: Other profs send emails about exam date changes, stuff like that. But Fective emailed just me, about my own work.

Ann: How did you feel about that?

Eric: Well, in most classes, you're just one in a crowd, and you can slack off sometimes. But in WEB 100, I knew that Prof. Fective was watching what I did. I didn't like that at first, but it made me work harder.

Oh, and Fective is picky, too. I'd turn in my work, and she'd tell me to fix something, and resubmit. I learned to make sure my work was right before I turned it in. If I didn't, Prof. Fective would bug me until it was right.

Ann: Ever get frustrated?

Eric: (Laughs) All the time. Making good Web pages isn't as easy as it looks. When you use the Web, you don't see all the work that goes into a good site. You have to think about goals. You have to learn how to find bugs and fix them. The HTML and stuff are only part of it.

Ann: What about the class itself? The textbook and such. Was that OK?

Eric: Umm, that was all OK. CoreDogs is easy to read. The class was mostly online, easy to fit in my schedule. Like I said, the thing that stressed me was that I had to do a lot of exercises.

It helped being able to meet Prof. Fective in person. We could sit down and go over something. Sometimes just me, sometimes a few of us.

Ann: Anything you particularly liked about the course?

Eric: Yes, I did like making my own site, the eMe. It's pretty good. I want to make the best impression I can, when I hunt for jobs. You can check it out. It's at http://erictawney.com.

Oh, and CoreDogs was the cheapest textbook I've ever had. Some profs make me spend ten times what I spent on CoreDogs, for books I hardly used.

Ann: Do you how to make a Web site?

Eric: Sure. I already made erictawney.com, and other things I did for CoreDogs. In fact, I'm making a site for the Student Investment Club.

Ann's thesis

Ann's thesis was titled "A Tale of Two Students: New Approaches to Teaching Skills." Here are excerpts.

(Ann resisted the temptation to start her thesis with: "It was the best of courses, it was the worst of courses." Sorry to disappoint.)

Prof I. followed a traditional model...

Research in the learning sciences suggests that the traditional approach has serious limitations...

Student M.'s frustrations are typical of the traditional model...

Prof F. used a different approach. There is no accepted name for this learning method, since it combines features of deep learning, outcome-based learning, and active learning. We will call it active skill learning (ASL). Some call it the CoreDogs Way...

The ASL approach is embodied in Web sites. CoreDogs is a site that uses ASL to help students learn about Web site development...

An ASL site has two parts:

1. Content - the body of the textbook: text, images, videos, etc. Content is about a topic, like Web development, statistics, or finance.

2. Software - tools that help authors create books, and help teachers and students use the books. The software is not specific to a content area. The same tools can be used to write textbooks on graphics design, algebra, and programming.

The software is Textbook Writer. It was sponsored by [your foundation here], and released as open source. Textbook Writer runs on Drupal, a popular open source content management system. Textbook Writer's technical structure is not discussed further...

Outcome-driven learning

ASL courses are designed around learning outcomes. The author identifies skills students should possess by the end of the course. The author then works backwards, listing component skills and facts. Only material that helps meet course outcomes is included in the course...

CoreDogs is a good example of outcome-based learning. Each chapter is about a task, like "Creating a Web page with text." The chapter shows the HTML tags and CSS rules that help with that task, and only those tags and rules...

CoreDogs covers about a third of the tags in the HTML 4.01 standard. These are the core tags, the ones that are most important in Web work...

Traditional publishers do not design textbooks around outcomes. Instead, they try to cover every topic that professors might think important. The result is that so many topics are included, none is covered in depth. The course is "a mile wide and one inch deep."

This is called the "coverage" approach. Students only have time to learn facts. They do not have time to learn how to use those facts. Yet it is the "how to" that is the essence of skill...

Deep learning

Learning science researchers compare shallow and deep learning.

Shallow learnersDeep learners
1. Know definitions of many facts.1. Know fewer definitions.
2. Cannot use facts to do tasks.2. Can use facts to complete tasks.
3. Forget facts after the exam.3. Remember facts and procedures.

ASL encourages deep learning:

  • Students do dozens of exercises.
  • Students get formative feedback about each exercise. More on this later.
  • Students watch other people do tasks. CoreDogs has two virtual students who follow the reader through the book. The reader watches them struggle, fail, find their mistakes, and recover...

Formative feedback

Researchers contrast summative and formative feedback. Summative feedback is separate from learning. The goal is to measure how much students have learned in the recent past. Student submit a project or take an exam, and get a grade.

In contrast, formative assessment is part of learning. Students submit work, and get feedback about what could be improved. Student are able to correct and resubmit.

Summative and formative feedback should be used together. Formative to help learning, and summative to measure student achievement. However, many professors only give summative feedback.

ASL makes heavy use of formative feedback. Students complete exercises every week, and submit them through the ASL software. Instructors assess the work, and ask for improvements. Students can change their solutions, and resubmit. The cycle continues until the grader is satisfied. The student then gets a completion badge for the exercise.

This approach to formative feedback truly sets ASL apart.

  • Students get far more practice than they do in a traditional course.
  • Every exercise is a point of contact between student and instructor, a chance to talk about the work. The instructor is always present in the course, even though students work independently.
  • "Early and often." Students start exercises early, and submit work throughout the semester. This lets instructors track the performance of each student continuously.
  • Students are kept on-task.

Metacognition

Metacognition happens when students think not about the content of a course, but about the way they learn the content, or their reaction to the content. The following are metacognitive statements:

  1. "I learned a lot this week."
  2. "I'm stuck. I don't know what's going on."
  3. "Argh! Too much information! I'm overwhelmed!"
  4. "Why am I learning this stuff? Will I ever use it?"

Metacognitive beliefs affect motivation and intention. For example:

  1. "I learned a lot this week. If I keep at it, I'll learn a lot next week, too."
  2. "I'm stuck. I don't know what's going on. I may as well give up."
  3. "Argh! Too much information! I'm overwhelmed! Does everyone else have this problem? Maybe I should ask the professor."
  4. "Why am I learning this stuff? Will I ever use it? Nah. Why bother."

CoreDogs uses virtual students to address these issues. For example:

Metacognition

The students (Renata and CC) complain about information overload. The book's author (a dog called Kieran) gives a solution: find "simple rules that work most of the time." He goes on to list some rules.

The conversation has two purposes. First, it gives a solution to a metacognitive problem (information overload). Second, the conversation shows how students should act. They should acknowledge problems, and seek help.

Situation-appropriate learning

Textbook Writer is the software that helps authors write ASL textbooks. Textbook Writer works in more than 40 languages.

This lets millions of educators around the world write textbooks for their own circumstances. Dozens of textbooks have been written in the two years since Textbook Writer was released. Hundreds more are in development.

For example:

  1. A teacher in Detroit wrote a math book specifically for students in that city. She uses locations and tasks familiar to Detroit students.
  2. Two community college professors in California wrote a personal finance book in Spanish. It is being used by thousands of Latino Americans. Exercise feedback is given by local volunteers, further strengthening their communities.
  3. A botany professor and an art professor in Cuba wrote a cartoon book about plants found on their university's campus. The book became an underground hit in Florida. There are even costume parties in Miami where people dress up as their favorite Cuban plants...

Conclusion

… Several factors contribute to Textbook Writer's extraordinary impact.

  1. TW is based on well-understood learning science principles, articulated in active skill learning (ASL). In fact, the initial design document is still available, at http://coredogs.com/article/designing-coredogs. It shows that TW was founded on learning, not technology.
  2. [Your foundation here] sponsored the development of the current version of TW, based on research prototypes. Professional programmers gave the project a solid technical base. The foundation's relatively modest investment will be generating returns for years to come.
  3. TW has attracted educators and computing professionals from around the world. They have donated thousands of hours, translating, creating open access textbooks, and adding new features to TW itself. The software has a life of its own. It does not depend on particular individuals, companies, or even countries.

The Open Textbook Partnership (OTP) will start operations soon, sponsored by Amazon, Apple, and Google. OTP will offer free TW hosting for open access books...

Critics question the value of educational technology investments. They are right to do so. Schools and universities have poured millions into technology, without clear results...

Many educational technology projects are about technology, not education. Learning is one of the most complex things people do. When technology is designed without deep study of learning, success is a matter of luck.

TW shows how educational technology should be developed. There was a blueprint based on learning science, before any code was written. Success was not left to chance. It was planned.


How to...

Lessons

User login

Log in problems? Try here


Dogs