Artificial intelligence is rapidly changing education, but one question remains: can AI actually improve student learning rather than simply provide answers? A recent research paper explores this question through Phosphor, an AI-powered interactive textbook platform designed to encourage active learning by embedding quizzes and AI-generated feedback directly into course materials.

The paper presents results from a pilot study conducted in an introductory statistics course at Dartmouth College and offers valuable insights into how AI can be used to improve engagement, reading completion, and academic performance.

What Is Phosphor?

Phosphor is an interactive digital textbook platform that integrates AI directly into the learning experience. Instead of relying on a standalone chatbot, students read course material while completing embedded assessments that receive immediate AI-generated feedback.

The platform includes:

  • Interactive online textbook lessons
  • Embedded quizzes after each lesson
  • AI grading for free-text (constructed response) answers
  • Traditional multiple-choice questions
  • Cumulative review quizzes
  • A retrieval-augmented (RAG) chatbot for course-related questions

Importantly, participation was completely optional and did not contribute to students' course grades.

The Study

The pilot was conducted in an introductory statistics course with 151 enrolled students, of whom 143 completed the course. Despite there being no grade incentive, researchers observed remarkably high levels of participation.

The study investigated three primary questions:

  • Can AI increase student engagement with textbook reading?
  • Does AI-assisted formative assessment improve learning outcomes?
  • Are open-ended questions more effective than multiple-choice quizzes?

High Student Engagement

Traditional textbook reading is often poor, with instructors estimating that only around 10–15% of students complete assigned readings.

The results with Phosphor were substantially different.

  • Over 90% of students used the platform.
  • Estimated reading completion increased dramatically.
  • Many students voluntarily completed most lessons despite receiving no course credit.

These findings suggest that integrating interactive assessments directly into reading materials can significantly increase student engagement.

Better Engagement Was Associated with Better Performance

Students who engaged most frequently with the platform generally performed better on the final examination.

The researchers found:

  • A strong positive relationship between lesson completion and exam scores.
  • Even after accounting for previous academic performance, students who completed more of the platform achieved substantially higher final exam results.

Because participation was voluntary, the study cannot prove that Phosphor alone caused the improvement. Motivated students may naturally be more likely both to use the platform and perform well academically. Nevertheless, the association remained significant even after statistical adjustments.

Open-Ended Questions Outperformed Multiple Choice

One of the most interesting findings came from an unexpected comparison between different quiz formats.

During the semester:

  • The first module included both multiple-choice and AI-graded free-response questions.
  • The second module relied almost entirely on multiple-choice questions.
  • The third module returned to free-response assessments.

Researchers observed that student performance improved much more during the modules containing constructed-response questions. During the multiple-choice-only section, additional quiz completion showed little measurable relationship with exam performance.

This finding supports educational research suggesting that actively generating answers promotes deeper learning than simply recognizing the correct option.

The Importance of Review Quizzes

Students who completed the cumulative review quizzes consistently outperformed those who skipped them.

The authors attribute this improvement to well-established learning principles, including:

  • Retrieval practice
  • Spaced repetition
  • Interleaving previously learned concepts

These techniques encourage long-term retention rather than short-term memorization.

Interestingly, Students Rarely Used the Chatbot

Although Phosphor included an AI chatbot capable of answering questions about the course material, students made relatively little use of it.

According to student feedback, several factors contributed:

  • General AI tools were often quicker to access.
  • The lesson content already answered most questions.
  • Students found the embedded quizzes more useful than conversational AI.

This suggests that AI may provide greater educational value when integrated into learning activities rather than functioning as a standalone assistant.

Key Design Principles

The paper argues that AI is most effective when it supports active learning rather than replacing it. Successful educational AI systems should:

  • Integrate directly into instructional materials.
  • Provide immediate formative feedback.
  • Assess open-ended responses instead of relying solely on multiple-choice questions.
  • Encourage retrieval practice.
  • Track student progress over time.

Limitations of the Study

While the findings are encouraging, the authors acknowledge several important limitations:

  • The study was observational rather than a randomized controlled trial.
  • Students chose whether to use the platform.
  • The research was conducted at a single university.
  • Some survey responses may be subject to bias.
  • Differences between course modules involved more than just question type, making causal conclusions difficult.

Conclusion

The Phosphor study suggests that artificial intelligence has considerable potential to improve higher education when it is integrated directly into the learning process. Rather than acting as a shortcut that simply provides answers, AI appears most valuable when it encourages students to think, retrieve information, and receive immediate feedback on their own responses.

Perhaps the most significant takeaway is that AI-powered formative assessment—not AI chatbots—may represent the most effective use of large language models in education.

Original Research Paper

You can read the complete paper here:

Balancing Efficacy and Engagement in Interactive Texts (PDF)