Publicação
From prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental education
| Resumo: | Accurate text classification and placement remain challenges in U.S. higher education, with traditional automated systems like Accuplacer functioning as “black-box” models with limited assessment transparency. This study evaluates Large Language Models (LLMs) as complementary placement tools by comparing their classification performance against a human-rated gold standard and Accuplacer. A 450-essay corpus was classified using Claude, Gemini, GPT-3.5-turbo, and GPT-4o across four prompting strategies: Zero-shot, Few-shot, Enhanced, and Enhanced+ (definitions with examples). Two classification approaches were tested: (i) a 1-step, 3 class classification task, distinguishing DevEd Level 1, DevEd Level 2, and College-level texts in one single run; and (ii) a 2-step classification task, first separating College vs. Non-College texts before further classifying Non-College texts into DevEd sublevels. The results show that structured prompt refinement improves the precision of LLMs’ classification, with Claude Enhanced + achieving 62.22% precision (1 step) and Gemini Enhanced + reaching 69.33% (2 step), both surpassing Accuplacer (58.22%). Gemini and Claude also demonstrated strong correlation with human ratings, with Claude achieving the highest Pearson scores (ρ = 0.75; 1-step, ρ = 0.73; 2-step) vs. Accuplacer (ρ = 0.67). While LLMs show promise for DevEd placement, their precision remains a work in progress, highlighting the need for further refinement and safeguards to ensure ethical and equitable placement. |
|---|---|
| Autores principais: | Da Corte, Miguel |
| Outros Autores: | Baptista, Jorge |
| Assunto: | Large language models (LLMs) Developmental education (DevEd) Writing assessment Text classification English writing proficiency |
| Ano: | 2025 |
| País: | Portugal |
| Tipo de documento: | artigo |
| Tipo de acesso: | acesso aberto |
| Instituição associada: | Universidade do Algarve |
| Idioma: | inglês |
| Origem: | Sapientia - Universidade do Algarve |
| _version_ | 1877785680404283392 |
|---|---|
| author | Da Corte, Miguel |
| author2 | Baptista, Jorge |
| author2_role | author |
| author_facet | Da Corte, Miguel Baptista, Jorge |
| author_role | author |
| contributor_name_str_mv | Sapientia |
| country_str | PT |
| creators_json_txt | [{\"Person.name\":\"Da Corte, Miguel\",\"Person.identifier.orcid\":\"0000-0001-8782-8377\"},{\"Person.name\":\"Baptista, Jorge\",\"Person.identifier.cienciaID\":\"7010-5366-22C5\",\"Person.identifier.orcid\":\"0000-0003-4603-4364\",\"Person.identifier.researcherID\":\"H-7699-2013\",\"Person.identifier.scopusAuthorID\":\"14035269500\"}] |
| datacite.contributors.contributor.contributorName.fl_str_mv | Sapientia |
| datacite.creators.creator.creatorName.fl_str_mv | Da Corte, Miguel Baptista, Jorge |
| datacite.date.Accepted.fl_str_mv | 2025-01-01T00:00:00Z |
| datacite.date.available.fl_str_mv | 2026-04-09T09:40:00Z |
| datacite.date.embargoed.fl_str_mv | 2026-04-09T09:40:00Z |
| datacite.rights.fl_str_mv | http://purl.org/coar/access_right/c_abf2 |
| datacite.subjects.subject.fl_str_mv | Large language models (LLMs) Developmental education (DevEd) Writing assessment Text classification English writing proficiency |
| datacite.titles.title.fl_str_mv | From prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental education |
| dc.contributor.none.fl_str_mv | Sapientia |
| dc.creator.none.fl_str_mv | Da Corte, Miguel Baptista, Jorge |
| dc.date.Accepted.fl_str_mv | 2025-01-01T00:00:00Z |
| dc.date.available.fl_str_mv | 2026-04-09T09:40:00Z |
| dc.date.embargoed.fl_str_mv | 2026-04-09T09:40:00Z |
| dc.format.none.fl_str_mv | application/pdf |
| dc.identifier.none.fl_str_mv | http://hdl.handle.net/10400.1/28621 |
| dc.language.none.fl_str_mv | eng |
| dc.publisher.none.fl_str_mv | Schloss Dagstuhl – Leibniz-Zentrum für Informatik |
| dc.rights.cclincense.fl_str_mv | http://creativecommons.org/licenses/by/4.0/ |
| dc.rights.none.fl_str_mv | http://purl.org/coar/access_right/c_abf2 |
| dc.subject.none.fl_str_mv | Large language models (LLMs) Developmental education (DevEd) Writing assessment Text classification English writing proficiency |
| dc.title.fl_str_mv | From prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental education |
| dc.type.none.fl_str_mv | http://purl.org/coar/resource_type/c_6501 |
| description | Accurate text classification and placement remain challenges in U.S. higher education, with traditional automated systems like Accuplacer functioning as “black-box” models with limited assessment transparency. This study evaluates Large Language Models (LLMs) as complementary placement tools by comparing their classification performance against a human-rated gold standard and Accuplacer. A 450-essay corpus was classified using Claude, Gemini, GPT-3.5-turbo, and GPT-4o across four prompting strategies: Zero-shot, Few-shot, Enhanced, and Enhanced+ (definitions with examples). Two classification approaches were tested: (i) a 1-step, 3 class classification task, distinguishing DevEd Level 1, DevEd Level 2, and College-level texts in one single run; and (ii) a 2-step classification task, first separating College vs. Non-College texts before further classifying Non-College texts into DevEd sublevels. The results show that structured prompt refinement improves the precision of LLMs’ classification, with Claude Enhanced + achieving 62.22% precision (1 step) and Gemini Enhanced + reaching 69.33% (2 step), both surpassing Accuplacer (58.22%). Gemini and Claude also demonstrated strong correlation with human ratings, with Claude achieving the highest Pearson scores (ρ = 0.75; 1-step, ρ = 0.73; 2-step) vs. Accuplacer (ρ = 0.67). While LLMs show promise for DevEd placement, their precision remains a work in progress, highlighting the need for further refinement and safeguards to ensure ethical and equitable placement. |
| dirty | 0 |
| eu_rights_str_mv | openAccess |
| format | article |
| fulltext.url.fl_str_mv | https://sapientia.ualg.pt/bitstreams/875e67f9-aeb6-4a65-a895-586d86f84053/download |
| funder_facet_str_mv | FCT{{{_:::_}}}Fundação para a Ciência e a Tecnologia |
| funding.funder.alternateName_str_mv | FCT |
| funding.funder.identifier_str_mv | http://doi.org/10.13039/501100001871 |
| funding.funder.name_str_mv | Fundação para a Ciência e a Tecnologia |
| funding.identifier_str_mv | UIDB/50021/2020 |
| funding.name_str_mv | 6817 - DCRRNI ID |
| funding_str_mv | UIDB/50021/2020 info:eu-repo/grantAgreement/FCT/6817 - DCRRNI ID/UIDB%2F50021%2F2020/PT |
| id | sapientia_782a464ceeac20c41f0ebebe1b04ed2f |
| identifier.url.fl_str_mv | http://hdl.handle.net/10400.1/28621 |
| inst_facet_str | urn:organizationAcronym:ualg{{{_:::_}}}Universidade do Algarve |
| instacron_str | ualg |
| institution | Universidade do Algarve |
| instname_str | Universidade do Algarve |
| language | eng |
| network_acronym_str | sapientia |
| network_name_str | Sapientia - Universidade do Algarve |
| oai_identifier_str | oai:sapientia.ualg.pt:10400.1/28621 |
| organization_str_mv | urn:organizationAcronym:ualg |
| person_str_mv | Da Corte, Miguel Da Corte, Miguel http://orcid.org/0000-0001-8782-8377 0000-0001-8782-8377 Baptista, Jorge Baptista, Jorge https://www.ciencia-id.pt/7010-5366-22C5 7010-5366-22C5 http://orcid.org/0000-0003-4603-4364 0000-0003-4603-4364 |
| publishDate | 2025 |
| publisher.none.fl_str_mv | Schloss Dagstuhl – Leibniz-Zentrum für Informatik |
| repo_facet_str | urn:repositoryAcronym:sapientia{{{_:::_}}}Sapientia - Universidade do Algarve |
| reponame_str | Sapientia - Universidade do Algarve |
| repository_id_str | urn:repositoryAcronym:sapientia |
| service_str_mv | urn:repositoryAcronym:sapientia |
| spelling | engSchloss Dagstuhl – Leibniz-Zentrum für InformatikengAccurate text classification and placement remain challenges in U.S. higher education, with traditional automated systems like Accuplacer functioning as “black-box” models with limited assessment transparency. This study evaluates Large Language Models (LLMs) as complementary placement tools by comparing their classification performance against a human-rated gold standard and Accuplacer. A 450-essay corpus was classified using Claude, Gemini, GPT-3.5-turbo, and GPT-4o across four prompting strategies: Zero-shot, Few-shot, Enhanced, and Enhanced+ (definitions with examples). Two classification approaches were tested: (i) a 1-step, 3 class classification task, distinguishing DevEd Level 1, DevEd Level 2, and College-level texts in one single run; and (ii) a 2-step classification task, first separating College vs. Non-College texts before further classifying Non-College texts into DevEd sublevels. The results show that structured prompt refinement improves the precision of LLMs’ classification, with Claude Enhanced + achieving 62.22% precision (1 step) and Gemini Enhanced + reaching 69.33% (2 step), both surpassing Accuplacer (58.22%). Gemini and Claude also demonstrated strong correlation with human ratings, with Claude achieving the highest Pearson scores (ρ = 0.75; 1-step, ρ = 0.73; 2-step) vs. Accuplacer (ρ = 0.67). While LLMs show promise for DevEd placement, their precision remains a work in progress, highlighting the need for further refinement and safeguards to ensure ethical and equitable placement.application/pdfengFrom prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental educationPersonalDa Corte, MiguelDSpacehttp://dspace.org/items/4a524eae-b359-47fa-8978-028ac5ffb57eDSpacehttp://dspace.org/items/4a524eae-b359-47fa-8978-028ac5ffb57eDa CorteMiguelORCIDhttp://orcid.org0000-0001-8782-8377PersonalBaptista, JorgeDSpacehttp://dspace.org/items/e817fa28-a005-40e2-9ba4-03fdaedd7df3DSpacehttp://dspace.org/items/e817fa28-a005-40e2-9ba4-03fdaedd7df3BaptistaJorgeCiência IDhttps://www.ciencia-id.pt7010-5366-22C5ORCIDhttp://orcid.org0000-0003-4603-4364Researcher IDhttps://www.researcherid.comH-7699-2013Scopus Author IDhttps://www.scopus.com14035269500HostingInstitutionOrganizationalSapientiae-mailmailto:repositorio@ualg.ptrepositorio@ualg.ptDOIIsPartOf10.4230/OASIcs.SLATE.2025.12026-04-09T09:40:00Z20252025-01-01T00:00:00ZHandlehttp://hdl.handle.net/10400.1/28621http://purl.org/coar/access_right/c_abf2open accessLarge language models (LLMs)Developmental education (DevEd)Writing assessmentText classificationEnglish writing proficiency693557 bytesFundação para a Ciência e a TecnologiaInstituto de Engenharia de Sistemas e Computadores, Investigação e Desenvolvimento em Lisboainfo:eu-repo/grantAgreement/FCT/6817 - DCRRNI ID/UIDB%2F50021%2F2020/PTUIDB/50021/20206817 - DCRRNI IDCrossref Funder IDhttp://doi.org/10.13039/501100001871literaturehttp://purl.org/coar/resource_type/c_6501journal article2025http://creativecommons.org/licenses/by/4.0/http://purl.org/coar/access_right/c_abf2application/pdffulltexthttps://sapientia.ualg.pt/bitstreams/875e67f9-aeb6-4a65-a895-586d86f84053/download14th Symposium on Languages, Applications and Technologies (SLATE 2025)118 |
| spellingShingle | From prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental education Da Corte, Miguel Large language models (LLMs) Developmental education (DevEd) Writing assessment Text classification English writing proficiency |
| status | SINGLETON |
| subject.fl_str_mv | Large language models (LLMs) Developmental education (DevEd) Writing assessment Text classification English writing proficiency |
| title | From prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental education |
| title_full | From prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental education |
| title_fullStr | From prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental education |
| title_full_unstemmed | From prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental education |
| title_short | From prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental education |
| title_sort | From prediction to precision: leveraging LLMs for equitable and data-driven writing placement in developmental education |
| topic | Large language models (LLMs) Developmental education (DevEd) Writing assessment Text classification English writing proficiency |
| topic_facet | Large language models (LLMs) Developmental education (DevEd) Writing assessment Text classification English writing proficiency |
| url | http://hdl.handle.net/10400.1/28621 |
| visible | 1 |