{"id":237803,"date":"2025-11-14T05:00:00","date_gmt":"2025-11-14T05:00:00","guid":{"rendered":"https:\/\/schoolsweek.co.uk\/?p=237803"},"modified":"2025-11-13T22:32:58","modified_gmt":"2025-11-13T22:32:58","slug":"ofsted-slammed-over-consultation-analysis-method-and-still-keeps-it-secret","status":"publish","type":"post","link":"https:\/\/schoolsweek.co.uk\/ofsted-slammed-over-consultation-analysis-method-and-still-keeps-it-secret\/","title":{"rendered":"Ofsted slammed over consultation analysis method (and still keeps it secret)"},"content":{"rendered":"\n

Ofsted has again refused to release data showing how many people supported or opposed its report card inspection plans<\/a> \u2013 saying the \u201cvast majority\u201d of the 6,500 responses were not actually categorised by \u201csentiment\u201d.<\/p>\n\n\n\n

Legal experts on public consultations have said the admission means there is a \u201crisk\u201d decision-makers at Ofsted<\/a> were not \u201cprovided with the relevant information\u201d to inform their inspection overhaul plans.<\/p>\n\n\n\n

The watchdog\u2019s use of AI to code responses is also now facing scrutiny over accusations it could be \u201cunreliable\u201d, with potential wider implications for the use of such technology in government consultations.<\/p>\n\n\n\n

Ofsted refuses release (again)<\/h3>\n\n\n\n

Responding to the findings, Pepe Di\u2019Iasio, general secretary of leaders\u2019 union ASCL, said the process was \u201cvery flawed\u201d.<\/p>\n\n\n\n

\u201cIt is difficult to see how decision-makers could have had a granular understanding of the consultation responses,\u201d he added.<\/p>\n\n\n\n

Ofsted\u2019s inspection consultation ran from February to April. It consisted of 102 open-ended questions about proposed changes to inspections, which have started in schools this week. Three-quarters of respondents were education professionals.<\/p>\n\n\n\n

Ofsted only published a narrative summary of its consultation findings in September, despite saying it had \u201csentiment\u201d data on responses.<\/p>\n\n\n\n

Schools Week<\/em> requested this data under the Freedom of Information Act. We asked for overall responses based on Ofsted\u2019s sentiment categories of positive, negative, mixed and neutral. We also asked for this broken down per individual question.<\/p>\n\n\n\n

\u2018Most responses not coded\u2019<\/h3>\n\n\n\n

In its response, sent on Friday, Ofsted again refused to release the information. However, it provided new details on its analysis.<\/p>\n\n\n\n

Ofsted used a \u201chybrid approach\u201d to look at the \u201cvast amount of qualitative data\u201d. Human coders analysed a \u201csmall proportion\u201d of the early responses. They did code each individual response \u201cby theme and by sentiment\u201d.<\/p>\n\n\n\n

An AI model was then used to \u201canalyse all responses\u201d once the consultation closed. However, AI instead \u201cproduced several themes per question\u201d and then \u201cdescribed each theme, rather than individual response, by sentiment\u201d.<\/p>\n\n\n\n

In its FOI response, Ofsted said: \u201cThe vast majority of responses were not directly coded by sentiment at the individual level.\u201d It added that the hybrid approach \u201cprovided an overall understanding of sentiments and views\u201d.<\/p>\n\n\n\n

\"Daniel
Daniel Kebede<\/figcaption><\/figure>\n\n\n\n

Ofsted also refused to release the sentiment data collected by the human coders because it was \u201cnot representative\u201d.<\/p>\n\n\n\n

\u201cReleasing the sentiment data of early respondents in isolation would be misleading and potentially lead to incorrect conclusions being drawn by the public,\u201d added Verena Braehler, Ofsted\u2019s deputy director or research.<\/p>\n\n\n\n

National Education Union general secretary Daniel Kebede said the response \u201conly adds fuel to the fire\u201d.<\/p>\n\n\n\n

\u201cAccurate analysis of all responses was the very least we should have expected and we can’t even guarantee that was the case,\u201d he said. \u201cThe many teachers who took part should at least be able to have had faith in their responses being read.\u201d<\/p>\n\n\n\n

\u2018We weren\u2019t counting boxes ticked\u2019<\/h3>\n\n\n\n

However Ofsted said the theming method was \u201cjust as meaningful\u201d as coding individual responses by sentiment.<\/p>\n\n\n\n

The consultation \u201cwasn\u2019t a quantitative exercise and we weren\u2019t counting boxes ticked \u2013 this was free text and people expressed a range of views and sentiments within and across their individual responses, which were all captured in our analysis and reflected in our response to consultation\u201d.<\/p>\n\n\n\n

When shown details of Ofsted\u2019s FOI response, Sam Hart, an associate at law firm Sharpe Pritchard LLP, said: \u201cThere is a risk that the decision-makers within Ofsted [were not] provided with the relevant information.\u201d<\/p>\n\n\n\n

The \u2018Gunning Principles\u2019 are case law that lay out standards for how consultations should be conducted. The fourth says \u201cconscientious consideration\u201d must be given to responses before a decision is made.<\/p>\n\n\n\n

Hart said Ofsted\u2019s response must be \u201csufficiently robust to ensure that individual responses are being conscientiously considered\u201d. He added there is \u201ca risk that AI may overlook or deprioritise minority viewpoints even in circumstances\u201d.<\/p>\n\n\n\n

Andrea Squires, a partner at law firm Winckworth Sherwood, added there was a question over whether Ofsted \u201cwas acting reasonably\u201d in its use of AI.<\/p>\n\n\n\n

‘We use AI ethically’<\/h3>\n\n\n\n

Use of AI is becoming increasingly common in analysing government consultations. One tool used by government departments \u2013 Consult \u2013 states a recent evaluation found it accurately identified themes in 60 per cent of consultation responses.<\/p>\n\n\n\n

Schools Week <\/em>asked Ofsted for details of the accuracy of the AI model it used but it declined to provide this information.<\/p>\n\n\n\n

A spokesperson said they \u201cuse AI lawfully, ethically and responsibly. We ensure there is meaningful human control and validation of AI processes, and appropriate human oversight of the use of AI.\u201d<\/p>\n\n\n\n

Surveys, focus groups, test visits and stakeholder meetings also informed their inspection decisions.<\/p>\n\n\n\n

Leaders\u2019 union NAHT is considering whether to appeal a refusal by the high court to pursue a judicial review over Ofsted\u2019s new inspections.<\/p>\n\n\n\n

General secretary Paul Whiteman said: \u201cWe have repeatedly called on Ofsted to fully publish all its analysis of the consultation responses. The fact that they won\u2019t speaks volumes.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

Experts warn of ‘risk’ decision-makers at inspectorate were not ‘provided with the relevant information’<\/p>\n","protected":false},"author":314902,"featured_media":237791,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"om_disable_all_campaigns":false,"footnotes":""},"categories":[4],"tags":[25],"post-theme":[2378],"class_list":["post-237803","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news","tag-ofsted","post-theme-ofsted"],"acf":[],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t