What an AI teaching assistant can tell you about your course

How Maize turns student questions into course insights, from the concepts students ask about to alerts when your materials miss a topic.

Short answer: every question a student asks an AI teaching assistant says something about your course: which concept is unclear, which problem is hard, which topic your materials never explain. Maize collects these signals in two places. The Analytics tab of each course shows what students ask about and where they get stuck. Coverage gap alerts email you when students ask about something your materials don’t explain. Neither shows student names, and nothing about what students asked appears until at least three students have enrolled.

Why student questions are worth reading

Office hours show you the students who come. An AI teaching assistant also hears from the ones who don’t: the student working through a problem set at 11 PM, or the one who would never raise a hand in a lecture hall. Their questions are the closest thing you have to a running record of where your course is hard.

Reading every transcript isn’t realistic, though. What helps is the pattern: which concepts keep coming up, which problems leave students stuck, and what students needed that your materials didn’t give them. In a small Carnegie Mellon study of middle-school math teachers using a classroom dashboard, teachers “paid a great amount of attention to student errors”, because errors “tend to be very actionable”. Teachers who asked for high-level summaries also said they would find them hard to trust without seeing the data behind them (Aleven et al.). Both findings shaped what Maize shows: where students go wrong, with the evidence one click away.

The Analytics tab

Each course has an Analytics tab with a date range: the last 7, 30 or 90 days, all time, or dates you pick. It has four parts.

What students ask about

Top topics groups questions by course concept, such as “Expected value and variance” or “Cournot competition”, rather than by the kind of help asked for. Many questions only name an assignment (“help with problem set 2, question 3”), so Maize reads each question together with the start of the assistant’s reply, which usually shows what the question was about.

One group is kept separate: Material the TA couldn’t find. A cluster there often means an assignment or reading hasn’t been uploaded yet.

Where students get stuck

Top challenges lists the assignments, lectures, readings, exams and problems students mention most, with each one’s share of all questions. Open a row to see a one- or two-sentence summary of what students struggled with. It is drawn from recent conversations as a whole, including the assistant’s replies and students’ follow-up questions, not only the opening question. A few recent questions are listed under it so you can check the summary against what students actually wrote. When the conversations don’t show what the difficulty was, the summary says so instead of guessing.

Questions your materials don’t cover

The topics students asked about that your materials don’t explain, how often each came up and when it was last asked. More on these below.

When students use it

A chart of student questions per day, the busiest day and hour, and how many of your enrolled students asked at least one question in the period. Days and hours follow your course’s time zone. Maize sets it from your institution where it can, and you can change it on the tab.

Coverage gap alerts

When a student asks about something your course materials don’t explain, the assistant says so and points the student to you or the assigned reading. It doesn’t fill the gap with an explanation of its own, which might not match how you teach the topic. The topic is recorded, and you can choose to hear about it.

Gaps come in two kinds:

  • Not in your materials. Nothing you uploaded covers the topic.
  • Named in your materials but not explained. A slide or reading mentions the term without explaining it. The alert says where it is listed.

You choose per course, on the Settings & billing tab, how often to hear about them:

  • Alert me every time. An email about 10 minutes after a topic comes up, so questions asked close together arrive in one email. The same topic isn’t sent again within a day; repeats are counted in a later email.
  • Alert me once a day (the default). One email listing the topics that came up, only on days when something did.
  • Never alert me. Topics are still recorded and listed on the Analytics tab.

You can add up to five more recipients, such as teaching fellows. Alerts list topics only, never students’ questions or names.

The fix is usually small: upload a handout, add a slide, or spend five minutes on it in the next lecture. Once new material is uploaded and indexed, the assistant can explain the topic from it.

Privacy

  • No names. Analytics never shows student names or email addresses.
  • A minimum class size. Topics, challenges and coverage gaps appear only once at least three students have enrolled. In a smaller course, a question could be traced back to the one student who asked it. Usage counts and the daily chart still show.
  • Your own testing doesn’t count. Questions you ask on the Test tab are left out of analytics and alerts.

What to keep in mind

  • Topic groups, challenge summaries and coverage gaps are identified by an AI model. They are usually right, but read the example questions before acting on a surprising one.
  • Top challenges depends on students naming the assignment, lecture or problem. Questions that don’t name one still count toward Top topics.
  • The Analytics tab is included on the Medium Course plan and above. Coverage gap alerts are on every plan.

A five-minute weekly routine

  1. Open the Analytics tab and pick the last 7 days.
  2. Start with the questions your materials don’t cover. Add material for anything that came up more than once.
  3. Open the top two challenges and read their summaries. If one names a misconception, address it in the next class.
  4. Skim Top topics for the shape of the week, and check whether anything landed under Material the TA couldn’t find.

Sources