Agreed, to (seemingly) not control or adjust for study duration seems quite the omission from the authors - time spent studying is one of the best correlates with grades achieved I believe (?).
Even as someone who has a (very, very) slight bias in favour of AI as a tool, if anything the fact there was seemingly a pretty non-significant improvement despite the control arm, as you say, probably just going home and playing football *and* considering that the students that *did* use the AI actually opted in and thus possessed some degree of intrinsic motivation is even more of a potential flag.
However, we should probably also (?) consider the fact that perhaps the kids that did opt-in were under-performers, with the kids opting-out not being so. I haven't been through the paper so perhaps they control for this, but it doesn't seem outside the realms of possibility if not.
Re what constitutes 'tutoring', I think I'd disagree slightly? Based on the description, it doesn't seem (to me) incorrect to deem this as tutoring.
I 100% agree that these things are absolutely not replacing teachers anytime soon (anyone who is building an education tool knows this or is lying), but it seems to me that your points for why it's not 'tutoring' (such as "the LLM didn't create the lesson" etc) *are* all within current LLMs ability (level-depending), so the fact the authors (for whatever reason) chose to not include this should be a mark against the authors potentially and not against the technology - in my opinion.
Let's say that the model used *were* to create some short lessons for the kids, ask questions, gauge their answers, evaluate, decide weak spots and outline the next lesson etc (very very loosely part of what shaeda.io will -hopefully- do) etc, this would be about as close to tutoring as tutoring can get, no?
I wrote an entire post on some potential issues/blindspots when it comes to using AI in classrooms, so I always enjoy reading similar blogs. The classic fire analogy (powerful if used right, disastrous if used wrong) is very fitting.
I think that's getting closer to tutoring, though I there's something still missing -- the thing that keeps kids from walking away from me in the middle of a tutoring session, but that would allow a kid to close a laptop or switch to another tab.
I mean, if the kid really wanted to walk away even in-person they will find a way. If anything I think there’s probably good scope here/there for an AI that’s had it’s personality somewhat tailored to however best the student would respond. If a football-loving kid gets tutored by a football-loving AI that will talk with them about football every 5 correct questions, this would seem a better option than the same kid being tutored by a teacher they don’t like.
That's not really how I see things. Motivation isn't mostly about keeping things interesting for the kid. It's almost a social or moral obligation -- you don't treat people like you treat a website.
The lack of RCT knowledge in this post is definitely concerning. Basic understanding of potential outcomes would be great for the author, plus maybe some tips on selection bias and internal validity. Go off! Give us nothing, King!
Agreed, to (seemingly) not control or adjust for study duration seems quite the omission from the authors - time spent studying is one of the best correlates with grades achieved I believe (?).
Even as someone who has a (very, very) slight bias in favour of AI as a tool, if anything the fact there was seemingly a pretty non-significant improvement despite the control arm, as you say, probably just going home and playing football *and* considering that the students that *did* use the AI actually opted in and thus possessed some degree of intrinsic motivation is even more of a potential flag.
However, we should probably also (?) consider the fact that perhaps the kids that did opt-in were under-performers, with the kids opting-out not being so. I haven't been through the paper so perhaps they control for this, but it doesn't seem outside the realms of possibility if not.
Re what constitutes 'tutoring', I think I'd disagree slightly? Based on the description, it doesn't seem (to me) incorrect to deem this as tutoring.
I 100% agree that these things are absolutely not replacing teachers anytime soon (anyone who is building an education tool knows this or is lying), but it seems to me that your points for why it's not 'tutoring' (such as "the LLM didn't create the lesson" etc) *are* all within current LLMs ability (level-depending), so the fact the authors (for whatever reason) chose to not include this should be a mark against the authors potentially and not against the technology - in my opinion.
Let's say that the model used *were* to create some short lessons for the kids, ask questions, gauge their answers, evaluate, decide weak spots and outline the next lesson etc (very very loosely part of what shaeda.io will -hopefully- do) etc, this would be about as close to tutoring as tutoring can get, no?
I wrote an entire post on some potential issues/blindspots when it comes to using AI in classrooms, so I always enjoy reading similar blogs. The classic fire analogy (powerful if used right, disastrous if used wrong) is very fitting.
I think that's getting closer to tutoring, though I there's something still missing -- the thing that keeps kids from walking away from me in the middle of a tutoring session, but that would allow a kid to close a laptop or switch to another tab.
I mean, if the kid really wanted to walk away even in-person they will find a way. If anything I think there’s probably good scope here/there for an AI that’s had it’s personality somewhat tailored to however best the student would respond. If a football-loving kid gets tutored by a football-loving AI that will talk with them about football every 5 correct questions, this would seem a better option than the same kid being tutored by a teacher they don’t like.
That's not really how I see things. Motivation isn't mostly about keeping things interesting for the kid. It's almost a social or moral obligation -- you don't treat people like you treat a website.
The lack of RCT knowledge in this post is definitely concerning. Basic understanding of potential outcomes would be great for the author, plus maybe some tips on selection bias and internal validity. Go off! Give us nothing, King!