An impressive classroom technology demonstration can answer almost every question except the one that matters most: will students learn more, understand more deeply, and become more independent because this product is here?
Imagine a school district reviewing a new adaptive mathematics platform. During the demonstration, students appear engaged. Teachers appreciate the dashboard, and the vendor highlights personalization and artificial intelligence. The purchasing committee is enthusiastic. Yet no one has established which mathematical misconception the platform should reduce, how learning will be measured, or what happens when teachers stop assigning the software.
That is not a technology problem. It is a decision-design problem.
Before buying EdTech, US schools should conduct a small, ethically managed pilot with explicit learning goals, baseline measures, teacher support, privacy and accessibility checks, and predefined go/no-go criteria. This article offers 15 practical questions school and district leaders can use to decide whether to buy, revise, extend, or decline an educational technology product.
The checklist is designed primarily for US K–12 school leaders, instructional leaders, curriculum directors, principals, purchasing teams, IT and student-data personnel, and teachers involved in pilot programs. It is not a substitute for legal, security, accessibility, or procurement review by the appropriate district specialists.
Why EdTech procurement should begin with learning, not the product
When a tool enters a classroom, it changes more than what appears on a screen. It may alter how teachers explain, how students practice, what feedback they receive, which information is collected, and what adults can observe about learning.
A product may be technically excellent yet educationally unnecessary. Another may offer useful feedback but demand more training time than a school can realistically provide. A third may improve short-term quiz performance without helping students transfer knowledge to independent work.
The US Department of Education's August 2026 guidance on responsible educational technology emphasizes instructional value, evidence, educator judgment, parent transparency, and meaningful student outcomes. It also urges schools to reconsider tools when the evidence does not demonstrate learning benefits.
The Digital Promise EdTech Procurement Framework similarly describes a cycle of analyzing needs, choosing a product, planning and implementing a pilot, evaluating results, and deciding whether to sustain or scale.
In other words, buying is not the first stage of responsible adoption. It should be the result of a credible evaluation.
The 15-question EdTech pilot checklist at a glance
| Decision area | Question to ask | Evidence to collect |
|---|---|---|
| Learning need | 1. What exact learning problem are we solving? | Baseline student work and teacher observations |
| Existing options | 2. Can an existing tool or non-digital intervention solve it? | Inventory and alternative comparison |
| Fit | 3. Which students and teachers is the product actually designed for? | Relevant population and use-case evidence |
| Learning design | 4. What will students do cognitively, beyond clicking? | Sample tasks and authentic student work |
| Evidence | 5. What independent evidence supports the claimed benefit? | Studies, methods, limitations |
| Pilot outcomes | 6. What results would count as success or failure? | Predetermined measures and thresholds |
| Comparison | 7. How will we tell product effects from other changes? | Appropriate comparison or credible baseline |
| Inclusion | 8. Can every intended learner use it meaningfully? | Accessibility and inclusion testing |
| Privacy | 9. What student information is collected, used, and retained? | Data map, agreements, deletion commitments |
| Security | 10. What happens if accounts, data, or systems fail? | Security and incident-response review |
| Integration | 11. Does it work with our existing school systems? | Tested roster, identity and LMS workflows |
| Teacher capacity | 12. What training and classroom support are required? | Implementation plan and time estimates |
| Student independence | 13. Does the tool develop capability or dependency? | Unassisted tasks and learner explanations |
| Cost | 14. What is the complete cost over the contract period? | Total-cost model and exit costs |
| Decision | 15. Who decides, using what evidence, and when? | Written decision rule and accountable owners |
The questions are sequenced deliberately. Questions 1–7 establish whether the tool is worth testing. Questions 8–12 determine whether it can be used responsibly and realistically. Questions 13–15 protect the educational mission and the long-term decision.
1. What exact learning problem are we trying to solve?
"Improve mathematics" is too broad to guide a purchase. "Help sixth-grade students distinguish proportional from non-proportional relationships" is a testable instructional need.
Start with a narrow statement that includes a learner group, an observable difficulty, and a target for improvement. Gather recent classroom work, assessment patterns, teacher descriptions of errors, and student explanations.
Ask teachers where a task breaks down. Do students lack prerequisite knowledge? Can they follow a demonstrated procedure but not choose it independently? Are students unable to understand written instructions? Different problems may need different forms of support.
Pilot decision test: Can the team complete the sentence, "We need this product because our students currently struggle to..." without using the vendor's marketing language?
If not, the school may be shopping before identifying its need.
2. Have we compared the product with what we already have?
A new license is not automatically a new capability. Before running a pilot, review existing subscriptions, built-in features of the learning management system, district-approved accessibility tools, available curriculum materials, and the teacher practices already in use.
Also compare non-digital alternatives. More carefully sequenced examples, a short daily retrieval routine, targeted tutoring, and better feedback may address the same problem without introducing additional software or student accounts.
This does not mean analog methods are always superior. It means a pilot should test whether the proposed product adds meaningful value beyond reasonable alternatives.
Record the comparison transparently. What does the proposed tool do that the current approach does not? What extra burden does it introduce? Could changes to an existing practice achieve similar outcomes?
Pilot decision test: The team can state the product's incremental benefit, not merely its feature list.
3. For which learners, subjects, and settings is the product suitable?
Evidence for one population does not automatically transfer to another. A tool tested with fluent English readers in upper elementary grades may not perform similarly with multilingual beginners or students requiring particular accommodations.
Request the age range, academic subject, required prior skills, device assumptions, language support, reading level, and intended implementation context. Ask which students were included and excluded from vendor-sponsored studies.
Have classroom teachers review real examples before selecting pilot classrooms. They can identify whether the content matches standards and whether the pacing and task design make sense for their students.
Avoid a vague "works for everyone" claim. A product can be useful for a defined group without being the right universal purchase.
Pilot decision test: You can name the intended participants, likely beneficiaries, important exclusions, and conditions that would require adaptation.
4. What are students actually doing while they use it?
Time on task, logins, completed levels, and animation-driven engagement do not by themselves establish learning.
Watch a student interact with the tool. Does the student have to retrieve information, explain a choice, compare alternatives, analyze a mistake, or solve an unfamiliar problem? Or is the main activity following prompts until the system displays a reward?
For some learning goals, guided practice and immediate feedback are valuable. For other goals, students need discussion, productive struggle, writing, manipulation of physical materials, or opportunities to create and defend a solution.
The 2024 National Educational Technology Plan distinguishes access to devices from the quality of how technology is designed and used for learning. A product should fit the instructional purpose, not replace it with screen activity.
Pilot decision test: Teachers can show an authentic example of student thinking produced during or after use—not just screenshots of the dashboard.
5. What evidence supports the vendor's learning claims?
Vendors may present testimonials, customer counts, usage growth, case studies, correlations, product evaluations, or controlled research. They are not interchangeable forms of evidence.
Ask what the research measured, who conducted it, whether the sample resembles your schools, what comparison was used, what outcomes were reported, and whether results persisted beyond the intervention.
An appealing success story can inspire investigation but cannot prove causation. Equally, a rigorous study from a different population may be informative without guaranteeing the same result locally.
The What Works Clearinghouse provides an approach to examining intervention evidence and study quality. Use it as a benchmark for asking whether a claimed effect rests on a credible design.
Pilot decision test: The committee can describe the strongest available evidence and at least one important limitation in ordinary language.
6. What would success look like before the pilot begins?
Define success while the team can still disagree openly, before enthusiastic teachers or a persuasive vendor presentation create pressure to buy.
Choose one or two primary learning outcomes. Examples include improvement on independently scored fraction explanations, better comprehension questions requiring inference, or more accurate application of a concept to new tasks.
Then establish secondary outcomes: educator workload, student experience, usability, attendance at training, cost, and access barriers. Set an evaluation date and a decision rule in writing.
Avoid inventing universal thresholds. A ten-point increase may be meaningful on one measure and trivial on another. What counts as educationally important depends on the measure, baseline, population, uncertainty, and district priorities.
Pilot decision test: The project charter includes a precise outcome, a measure, timing, analysis owner, and predefined criteria for continuation.
7. How will we distinguish improvement from a misleading comparison?
Students often improve during a term even without a new product. A simple before-and-after graph may reflect normal instruction, different assessment difficulty, extra tutoring, or changes in attendance.
Where practical and appropriate, compare similar groups receiving different supports, use the same or equivalent assessments, and document other changes during the pilot. When random assignment is inappropriate or infeasible, be explicit about what a less rigorous comparison can and cannot establish.
A pilot does not have to become a large research experiment. But it does need a credible logic for interpreting the evidence. A very short pilot may reveal usability problems yet be too brief to determine durable learning effects.
Protect teacher judgment from commercial pressure. If the original decision rule says the evidence is insufficient, a beautiful dashboard should not silently change the rule.
Pilot decision test: The evaluation plan explains what alternative explanations could account for any observed gains.
8. Can students with different needs participate without unnecessary barriers?
Accessibility is not a feature checkbox. Ask students and educators to test representative tasks using keyboard navigation, screen readers when applicable, captions, adjustable display settings, language supports, and the accommodations students already use.
Request the vendor's accessibility documentation and test what it describes. Do not assume that a vendor's accessibility statement proves that every classroom workflow is accessible.
The W3C Web Content Accessibility Guidelines provide useful technical criteria. The CAST Universal Design for Learning Guidelines 3.0 also help school teams consider barriers to engagement, representation, and action or expression.
Accessibility requirements and legal obligations vary with the setting and applicable laws. District accessibility specialists should determine which standards and compliance reviews apply.
Pilot decision test: The school has actually tested priority tasks with representative user needs and has documented barriers, workarounds, and remediation commitments.
9. What student data are collected, and under what authority?
Build a plain-language data map before students sign in. Identify the information the tool receives, whether it contains personally identifiable information, why it is collected, who can access it, whether subcontractors receive it, how long it is retained, and what happens at contract termination.
Ask whether student content, prompts, recordings, performance histories, or behavioral signals are used to train or improve AI models. Request contract terms and technical evidence rather than relying on a general reassurance.
FERPA guidance from the US Department of Education explains that disclosure of personally identifiable information from education records is restricted, subject to applicable exceptions and conditions. The FTC's COPPA guidance for schools separately addresses circumstances in which schools may consent to children's data collection for an educational purpose, not an unrelated commercial one.
Neither statute should be reduced to a single "compliant" badge. District counsel and privacy officers should review the specific use and the relevant state requirements.
Pilot decision test: Data flows, permitted uses, retention, deletion, and responsibilities are documented and approved before real student data are used.
10. How resilient and secure is the product in everyday school life?
A tool that fails during an assessment or exposes accounts through weak access controls can create instructional disruption even if its content is excellent.
Have IT and security staff review identity management, authentication, authorization, encryption practices, incident notification, backups, vulnerability management, service availability, and disaster recovery. Verify support response commitments and whether the product can operate in the district's actual bandwidth and device conditions.
Ask what staff should do when the service is offline. A practical fallback matters: a teacher should be able to continue teaching without rebuilding the entire lesson around a vendor system.
Security questionnaires should be proportionate to the risk and data involved. An app that never receives student information raises different issues from a platform storing years of student records.
Pilot decision test: There is a documented approval, a clear response plan, and a usable classroom fallback.
11. Will it integrate with our learning management system and records?
A product can look effortless in a demo and become difficult when teachers must manually recreate classes, reconcile grades, or reset accounts.
Test rostering, single sign-on, access permissions, learning management system integration, grade exchange, accessibility compatibility, and data export using the district's real technical environment.
Where relevant, ask vendors for precise claims about standards such as 1EdTech Learning Tools Interoperability or OneRoster. The 1EdTech TrustEd Apps directory helps teams verify specific certifications rather than accepting broad claims that "we support all standards."
Consider the exit process from the beginning. Can the school retrieve useful data in usable formats? What happens to classes and reports if it changes platforms?
Pilot decision test: The IT team has completed an end-to-end workflow test, including an export or exit scenario.
12. What must teachers learn to make the pilot work?
Licenses do not train teachers. A pilot often fails to produce useful evidence when teachers are given access without time, practical examples, technical support, or a clear teaching rationale.
Design a professional learning plan with participating teachers. Include setup, instructional uses, modeling, feedback routines, troubleshooting, and time to examine student work.
Some teachers may need technical onboarding; others may need help adapting a lesson so the technology supports productive thinking rather than replacing explanation or discussion. Ensure teachers can report problems without their comments being treated as resistance to innovation.
Digital Promise specifically recommends establishing a shared success statement, an implementation plan, measurement milestones, and clearly assigned support responsibilities before deployment through its design-and-plan guidance.
Pilot decision test: The district can name who supports teachers, how often, during which working hours, and how implementation will be monitored without turning the pilot into punitive evaluation.
13. Are students becoming more independent, or simply more dependent on the tool?
This is the question conventional product scorecards often overlook.
A tutoring app may guide a student through many correct answers. An AI assistant may produce a polished paragraph. But can the learner explain the reasoning, recognize an error, select a strategy, or solve a comparable task without assistance?
Include a small number of independent performance opportunities during the pilot. For example, after adaptive fraction practice, ask students to justify a solution on paper or solve a novel word problem without hints. After AI-supported writing, ask them to revise a paragraph and explain their choices unaided.
The goal is not to ban assistance. Well-designed scaffolding can help learners accomplish tasks that initially exceed their independent ability. The important question is whether support is eventually adjusted so the learner gains capability.
This is a central Learning Coach perspective: learning technology should amplify the teacher's ability to observe, question, coach, and transfer responsibility. It should not turn the teacher into a passive monitor of completed screens.
Pilot decision test: Student work shows growing independent understanding or performance, not merely higher platform activity.
14. What is the full cost of ownership?
The sticker price rarely captures the entire financial commitment.
Calculate license fees, setup and migration, devices, connectivity, professional learning, teacher preparation time, integration maintenance, additional accessibility work, renewal increases, support charges, and exit or data-retrieval costs.
Ask whether the contract scales with users, classrooms, campuses, or usage. Clarify unused-seat policies, renewal dates, cancellation conditions, and protections against material changes to the service.
Then connect cost to the learning need. A product is not cost-effective simply because its per-student license is inexpensive. A higher-cost intervention is not automatically unjustified if it addresses an important need with strong, credible evidence.
Define the decision in terms of the outcomes a school is actually trying to achieve, within its budget and constraints.
Pilot decision test: The finance team has a realistic multi-year cost estimate and an explicit comparison with at least one alternative.
15. Who decides whether to buy, and what happens if the answer is no?
A pilot should conclude with a written decision, not drift into permanent use because people have grown accustomed to the platform.
Name an accountable decision group from the beginning: instructional leadership, teaching representatives, IT, data privacy, accessibility, finance, and where appropriate families and students. Identify conflicts of interest and separate vendor input from the school's final judgment.
Establish a small set of outcomes: adopt, adopt with changes, extend the pilot for a specified evidence gap, or do not adopt. For each, record the evidence, uncertainty, costs, implementation issues, and next review date.
A disappointing result is still valuable. It can reveal that the original problem was misdiagnosed, the implementation support was insufficient, the product was mismatched to students, or the technology offered no meaningful incremental benefit.
The Digital Promise sustain-and-scale guidance recommends explaining pilot results to stakeholders and considering targeted or phased adoption instead of automatically scaling every tool districtwide.
Pilot decision test: The committee has the authority and willingness to decline the purchase, even after a successful sales demonstration.
A practical 30-day EdTech pilot planning schedule
The following schedule is an illustrative planning template, not a claim that 30 days is enough to establish academic impact. Short windows may reveal implementation and usability issues, while meaningful learning outcomes may require a longer evaluation.
| Period | School-team action | Deliverable |
|---|---|---|
| Days 1–5 | Define instructional need, student group, alternatives, and primary measure | Problem statement and baseline |
| Days 6–10 | Complete privacy, accessibility, security, and technical reviews | Approval or documented conditions |
| Days 11–15 | Train educators, test workflows, refine implementation and support | Launch readiness checklist |
| Days 16–25 | Run a limited pilot and collect student work, implementation and user feedback | Pilot evidence log |
| Days 26–30 | Review early evidence, unresolved risks, actual costs, and next steps | Continue, modify, or stop decision |
For a reading intervention, for example, a month might be suitable for checking whether students can log in, whether assignments fit classroom routines, and whether teachers can interpret feedback. It may be wholly inadequate for determining long-term gains in comprehension.
Separate implementation success from learning success. The first means the tool was deployed as intended. The second means it produced a meaningful educational improvement under reasonably credible evaluation conditions.
How to score the pilot without disguising uncertainty
Avoid reducing a complex decision to one star rating. Use a staged decision:
Mandatory gates: privacy, safety, accessibility, and required technical compatibility. A failure here may block adoption regardless of pedagogical enthusiasm.
Learning case: clearly identified need, sound instructional design, credible evidence, meaningful evaluation.
Operational readiness: professional learning, support, realistic teacher workflow, sustainable costs.
Student independence: observable explanation, transfer, and reduced reliance on unnecessary hints.
Decision quality: transparent limitations, alternatives, and a willingness to discontinue.
If a committee prefers numerical scoring, define the scale and weights before the pilot. Label scores as local judgments rather than research results. A three-point evidence scale cannot convert a weak study into a strong one by arithmetic.
Keep a short evidence ledger with four columns: claim, source, what was actually observed, and remaining uncertainty. That habit is especially important when a vendor's promotional material presents engagement, satisfaction, or adoption as though it were direct evidence of academic learning.
The role of school leaders: coach adoption rather than command usage
Buying a platform is a procurement decision. Making it educationally useful is a leadership and professional-learning challenge.
A principal can require logins, but cannot create excellent instructional practice through compliance alone. Teachers need clarity about the learning objective, time to plan, observation of student thinking, and constructive conversations about what works.
An instructional leader can organize a short weekly inquiry: What did students attempt? Where did the new tool help? Where did it hide confusion? What would we change next week? How would we know whether students could succeed independently?
This is distinct from an instructional coach, whose role commonly centers on helping teachers improve practice. In Learning Coach terminology, a pedagogical leader applying a coaching approach works at the level of schoolwide learning culture, coherence, teacher support, and student autonomy. These roles may collaborate but should not be conflated.
For a broader examination of that philosophy, Stop Teaching. Start Coaching. explores the movement from delivering explanations to observing learning and building learner independence. Its argument does not replace formal evaluation of any particular technology.
Educators and leaders interested in developing a coaching-oriented approach can explore the Learning Coach professional training pathways. Program availability, admission, and certification details should be checked on the official training page; this article makes no guarantee about a particular pathway.
Frequently asked questions
How long should an EdTech pilot last?
Long enough to observe the outcomes you claim to be evaluating. A few weeks may identify usability, accessibility, and onboarding problems; sustained learning and retention may require longer. Decide the window based on the learning goal and assessment schedule, not on the vendor's preferred sales timeline.
Should schools pilot every free app?
Not necessarily. An app may still expose student information, require accessibility review, consume teacher time, or disrupt instruction even if its license is free. District policy should define proportionate review paths according to risk and use.
Is a vendor study enough to justify buying?
Not by itself. Examine methodology, participant similarity, outcome measures, comparison groups, and limitations. Local pilot evidence can complement stronger external research, but neither should be exaggerated.
What if teachers love the tool but student outcomes are inconclusive?
Treat teacher experience as important implementation evidence, while being clear that it does not establish learning effects. Extend the evaluation only if a defined evidence gap can realistically be addressed, and if cost and risk are acceptable.
How should schools handle EdTech products with generative AI?
Apply the same learning, privacy, safety, accessibility, and evaluation gates, with additional attention to accuracy, fabricated output, age-appropriate use, student content and data processing, human review, and whether students can explain work independently.
The decision worth making
A school does not need the greatest number of digital products. It needs a manageable set of tools that solve authentic learning problems, work for its students and teachers, respect rights and constraints, and justify their costs with credible evidence.
The most valuable question is not "Can this platform do more?" It is "Can our students do more because we used it—and can we demonstrate that without confusing activity with learning?"
Ask that question before signing, keep asking it during the pilot, and be prepared to act on the answer.
