From Tool Rollout to Operating Model

Topic

From Tool Rollout to Operating Model: Building sustainable, values-first AI implementation in schools

This professional development proposal reframes school AI adoption as an operating model problem rather than a tool rollout. The core message: AI implementation is not primarily about choosing a vendor, running a demo, or publishing a policy. It is a people-change, governance, curriculum, assessment, equity, evidence, and risk-management challenge.

The session helps school leaders and teacher leaders move from “What AI tool should we buy?” to “What decisions, routines, evidence, responsibilities, and safeguards will make AI use educationally valuable and aligned with our values?”

Audience / Use Case

Primary audience:

School and district leaders.
AI task forces or steering committees.
Instructional technology leaders.
Curriculum and assessment leaders.
Department chairs and teacher leaders involved in implementation.

Best use cases:

A leadership retreat or cabinet-level planning session.
A school AI committee kickoff.
A PD strand for instructional coaches and department chairs.
A post-pilot reflection session after initial AI experimentation.

This proposal can also support a two-part format: first, leaders diagnose their current AI operating model; second, teams design a local evaluation or pilot plan.

Framework / Core Claims

1. AI adoption is a people-change problem, not a tool rollout

The most important implementation question is not “Which product?” but “How will people change their practice responsibly?” The AI Edu Simplified piece on priorities and the people problem reports a survey of roughly 140 higher education respondents where priorities centered on culture, guidance, policy, and governance. Prosci data cited there names learning curve and proficiency as major barriers: 43% learning curve, 38% proficiency. The key line: “AI doesn’t change institutions; people do” (content/articles/aiedusimplified-substack-com-ai-priorities-and-the-peoples-problem.md).

For K–12, this implies that AI implementation must include:

Role clarity.
Professional learning.
Communication.
Policy interpretation.
Classroom routines.
Family/student guidance.
Feedback loops.
Evidence review.

2. Local tool evaluation beats generic benchmarks

Ethan Mollick’s “Giving your AI a job interview” argues that generic benchmarks are like hiring a VP by SAT score: they may be interesting, but they do not tell you whether the system can do the job you need. Schools should test AI tools on actual instructional, advising, assessment, administrative, and policy tasks (content/articles/oneusefulthing-org-giving-your-ai-a-job-interview.md).

A local AI “job interview” should ask:

Can the tool perform our real tasks with our constraints?
What does it get wrong in ways that matter educationally?
How does it handle student privacy, accessibility, bias, and source transparency?
What expertise is required to supervise it?
Where does it save time, and where does it create new work?

3. System altitude matters: different AI contexts require different evidence

Potkalitsky’s “Beyond the Hype” distinguishes high-, mid-, and low-altitude AI contexts that require different kinds of evidence (content/articles/nickpotkalitsky-substack-com-beyond-the-hype-why-your-schools.md). A classroom brainstorming routine, a schoolwide tutoring tool, and an AI scoring system do not carry the same risk profile.

A useful operating model asks teams to classify AI use by altitude:

Low altitude: Individual teacher or student use for brainstorming, drafting, feedback, planning, or exploration. Needs norms, examples, and reflection.
Mid altitude: Programmatic or department-level use affecting common assignments, interventions, advising, or curriculum. Needs shared criteria, pilot evidence, equity review, and training.
High altitude: Decisions affecting grades, placement, discipline, standardized scoring, compliance, or official records. Needs validation, bias testing, human oversight, documentation, and governance approval.

This distinction keeps schools from treating every AI use as equally dangerous or equally harmless.

4. Negative results are strategic evidence

A mature operating model values failed pilots. Kentz’s “negative results” piece argues that a failed tutoring/mentorship pilot revealed the need for formal structure and risk controls. Examples from UW, UCF, and Dayton show how negative data can create positive data; the key quote: “It was the negative data that created the positive data” (content/articles/mikekentz-substack-com-why-your-ai-strategy-needs-negative.md).

This is a leadership culture issue. If every AI pilot must be a success story, the organization will hide the most useful learning. Schools should design pilots that ask not only “Did it work?” but also:

For whom did it work?
Under what conditions?
What failed?
What risks emerged?
What support was missing?
What should we stop, scale, or redesign?

5. AI grading and scoring require special caution

AI scoring is not a single category. Potkalitsky’s article on AI grading risks distinguishes Ohio-style standardized scoring systems from ChatGPT-style generative grading and warns about prompt sensitivity, model drift, scoring persona, English learner bias, validation, bias testing, and human oversight (content/articles/nickpotkalitsky-substack-com-if-testing-companies-use-ai-to-grade.md).

The high-leverage question is: Which AI? A vendor’s validated scoring model, a teacher’s rubric assistant, and a general chatbot assigning grades are not equivalent. Any AI use tied to grades or official evaluation needs a much higher evidentiary bar.

6. Equity and access must be designed, not assumed

Equity cannot remain a slide at the end. AI access varies by device, paid tools, language, disability, home support, teacher comfort, and disciplinary background. Voice AI may expand access through voice reasoning, translation, transcription, accessibility support, advising, and real-time conversational help (content/articles/voice-ai-is-heading-to-the-classroom.md). But the opportunity only becomes equitable if schools decide who gets access, under what protections, with what training, and for which purposes.

Proposed Activities

Activity 1: Operating model map

Leadership teams map their current AI approach across six domains:

Governance and decision rights.
Curriculum and AI literacy.
Assessment and academic integrity.
Tool procurement and evaluation.
Equity, access, and accessibility.
Evidence, pilots, and feedback loops.

For each domain, teams mark:

What exists now?
Who owns it?
What is unclear?
What decision is needed next?

Deliverable: a one-page “current state / next decision” map.

Activity 2: Local AI tool job interview

Teams choose one real job for AI:

Generate feedback on a draft rubric.
Help a student brainstorm questions for a source set.
Translate family communication.
Summarize attendance patterns for an advisor.
Draft differentiated practice items.
Score sample writing with a rubric.

Then they test the tool using a structured protocol:

Task and success criteria.
Required context.
Red-line risks.
Sample inputs.
Expected expert review.
Failure modes.
Equity/access considerations.
Decision: reject, revise, pilot, or approve for limited use.

This converts enthusiasm into disciplined practice.

Activity 3: Negative-results pilot lab

Teams design a small pilot with explicit permission to learn from failure.

Pilot template:

Purpose: What problem are we trying to solve?
Users: Which teachers/students/staff?
AI use: What tool and task?
Guardrails: What is not allowed?
Evidence: What will count as success, mixed result, or failure?
Equity check: Who might benefit or be harmed?
Reflection: What negative data would be valuable?
Decision: stop, adapt, scale, or study further.

This activity is especially useful after early AI experiments have produced scattered anecdotes but no shared learning.

Activity 4: AI grading risk sort

Participants sort possible grading uses by risk:

Teacher uses AI to generate rubric language.
Student uses AI for feedback before submission.
Teacher uses AI to suggest comments but not grades.
Teacher uses AI to recommend grades for review.
AI automatically scores essays for official grades.
Vendor AI scores standardized assessments.

For each, teams identify evidence required before use. This makes the “which AI?” distinction practical and prevents blanket claims like “AI grading is fine” or “AI grading is always invalid.”

Activity 5: Equity/access scenario planning

Teams review scenarios:

Some students have paid AI access and others do not.
Voice AI helps multilingual families communicate with school.
A student with a disability uses transcription and conversational support.
AI feedback differs in quality by dialect, language proficiency, or background knowledge.
A ban pushes students to hidden use at home.

For each scenario, teams name a concrete policy, support, or infrastructure response.

Specific Evidence Supporting the Framework

People-change priority: AI Edu Simplified reports roughly 140 higher education survey respondents prioritizing culture, guidance, policy, and governance; Prosci barriers include 43% learning curve and 38% proficiency; “AI doesn’t change institutions; people do” (content/articles/aiedusimplified-substack-com-ai-priorities-and-the-peoples-problem.md).
Local evaluation: Mollick’s tool “job interview” frame rejects reliance on generic benchmarks and recommends testing AI on actual tasks (content/articles/oneusefulthing-org-giving-your-ai-a-job-interview.md).
System altitude: Potkalitsky argues schools need different evidence for high-, mid-, and low-altitude AI contexts (content/articles/nickpotkalitsky-substack-com-beyond-the-hype-why-your-schools.md).
Negative results: Kentz describes failed tutoring/mentorship work that revealed the need for formal structure and risk controls, connecting to UW, UCF, and Dayton examples; “It was the negative data that created the positive data” (content/articles/mikekentz-substack-com-why-your-ai-strategy-needs-negative.md).
AI grading risk: Potkalitsky distinguishes different forms of AI scoring and warns about prompt sensitivity, model drift, scoring persona, English learner bias, validation, bias testing, and human oversight (content/articles/nickpotkalitsky-substack-com-if-testing-companies-use-ai-to-grade.md).
Student behavior and policy pressure: Pew reports widespread teen chatbot use and widespread perception of cheating; students also ask for guidance rather than criminalization (content/articles/pewresearch-org-how-teens-use-and-view-ai.md; content/articles/fitzyhistory-substack-com-what-students-want-teachers-to-know.md).
Voice AI/accessibility: Voice AI creates opportunities for reasoning aloud, translation, transcription, accessibility, advising, and real-time contextual support (content/articles/voice-ai-is-heading-to-the-classroom.md).

Overlooked / High-Leverage Details

Implementation is people-change, not tool rollout. Procurement without practice change will disappoint.
Bans and restrictions need pedagogical infrastructure. Otherwise they may push use underground.
Failed pilots are strategic evidence. Schools should design pilots that can produce usable negative data.
Precise “which AI?” distinctions matter. Generic chatbot grading, validated scoring engines, and teacher-facing feedback assistants are different risk categories.
Equity must include access, language, disability, prior knowledge, teacher support, and paid/free tool differences.
AI misuse can reveal bad incentives. If students or staff use AI to avoid low-value work, the work design may need scrutiny.
System altitude protects nuance. Not every AI use requires the same governance, but high-impact uses need stronger evidence.
Local evidence beats vendor demos. Tools should be tested against local tasks and values.

Skeptic / Implementation Cautions

Avoid consultant-speak. Leaders and teachers need concrete routines, decision rights, and examples.
Do not treat AI strategy as a document. A policy without PD, classroom models, assessment redesign, and feedback loops is not an operating model.
Do not let pilots become theater. If the conclusion is predetermined, negative evidence will be hidden.
Do not overpromise efficiency. AI can save time in some tasks and create new work in others.
Equity cannot be vague. Name who gets access, what alternatives exist, and how students with different needs are supported.
AI grading is a high-risk area. Require validation, bias testing, human oversight, and clear appeal pathways before high-stakes use.
Beginner teachers need bounded classroom examples. Do not start faculty PD with “system altitude” jargon; use it with leaders as a planning tool.
Enthusiasts need workflow critique. Ask not only “Can AI do it?” but “Should this task be done this way, and what expert judgment remains necessary?”

Suggested Source / Wiki Anchors

content/articles/aiedusimplified-substack-com-ai-priorities-and-the-peoples-problem.md
content/articles/oneusefulthing-org-giving-your-ai-a-job-interview.md
content/articles/nickpotkalitsky-substack-com-beyond-the-hype-why-your-schools.md
content/articles/mikekentz-substack-com-why-your-ai-strategy-needs-negative.md
content/articles/nickpotkalitsky-substack-com-if-testing-companies-use-ai-to-grade.md
content/articles/voice-ai-is-heading-to-the-classroom.md
content/articles/pewresearch-org-how-teens-use-and-view-ai.md
content/articles/fitzyhistory-substack-com-what-students-want-teachers-to-know.md

Possible Session Title Options

“Beyond the Vendor Demo: Building an AI Operating Model for Schools”
“AI Strategy as Practice Change: Governance, Evidence, Equity, and Assessment”
“From Pilots to Practice: Values-First AI Implementation in Schools”

Clay's Wiki

Collections