Artificial Intelligence in Schools: Privacy and Security Considerations

Privacy and security considerations for using Artificial Intelligence (AI) in K-12 education primarily revolve around safeguarding sensitive student data, appropriately minimizing data, and grappling with ethical considerations around data use and algorithmic decision-making.

The following is meant to serve as an initial guide for educators grappling with these issues. To aid educators who would like concrete examples of what has been done previously, our colleagues at New America’s Teaching, Learning, and Tech program have compiled a repository of AI guidelines and resources intended for use in K-12 education.

As with all emerging technologies, there are more questions than answers on what responsible and ethical AI use might look like. This means that school districts and educators need to be actively engaged in shaping how they want to utilize AI. The following questions are meant to ground initial conversations and decisions around AI use:

• What problem is this technology addressing and what are the concrete benefits of using this technology?
• Does this technology collect the minimum amount of data needed to solve the problem?
• Does this technology prioritize human decision-making or computer decision-making? Is the decision in question likely to have a significant impact on students?
• Who will be responsible for ensuring AI based decisions and predictions advance educational equity?
• What processes will be in place to ensure the predictions and decisions are accurate and appropriate?
• Are parents/guardians and students aware of how this technology will be used? If so, have they consented to their data being used in this way?
• What processes will be in place for parents/guardians and students to understand, contest, and seek redress for algorithmic decision-making?

Additionally, given the growing number of EdTech AI vendors and applications, caution is needed when deciding which applications should be used and when. Beyond ensuring vendors meet local, state, and federal student privacy requirements, it is important that schools and educators only work with third-party vendors who are responsible data stewards. The following guiding questions may be helpful in assessing vendors:

• What specific type of student data will you collect, store, or have access to?
• Who can access the data? Where will the data be stored?
• What processes do you have in place to ensure the data is protected? How do you ensure data in transit is secure? Where is the data stored?
• How do you ensure student privacy? Have you ever had any data breaches that involved student data?
• How do you work with schools to make sure we understand how a decision or prediction is made? What mechanisms are there to adjust the algorithm based on educator feedback?
• Do you plan to share student data with any third parties? If so, to what end?
• How and when will data shared by us be destroyed? What processes do you have in place for data correction?

There’s a need for more transparency around data collection and usage, especially when using AI, a technology that relies on large amounts of data to learn and make predictions. While school districts have long collected data to track student metrics and educational attainment, the growing use of EdTech by teachers and school administrators has led to an increase in both the type of information being collected and the number of entities that can access this data. While many EdTech tools offer significant promise, educators and school districts should carefully consider the efficacy of new tools. Because not all vendors prioritize student privacy and data security, school districts should select third-party vendors with care.

School districts should involve parents/guardians and students in deciding what information can and should be collected, shared, or used by AI models, even if they are using it for educational purposes. Technology policy guidelines should be easily accessible and understandable, making it clear to parents/guardians and students exactly what information will be collected and how it will be used. Teachers can help ensure digital literacy skills by talking about data collection and usage with their students in an age appropriate manner. Additionally, there should be clear protocols around student and guardian data access, correction, and deletion.

School districts should provide educators with professional development opportunities around algorithmic bias and ethical AI use. AI models make predictions based on a large amount of data, but it is important to remember that those models are not infallible and can amplify existing harms to different communities. Because of many AI models’ tendency to replicate errors in existing data and reinforce existing discriminatory assumptions or outcomes, strong caution is needed when using algorithmic decision making. Where feasible, school districts should require algorithmic transparency from the third-party apps they use.

Finally, considerations for AI involve protecting student data from unauthorized access and malicious attacks. Schools must implement measures such as encryption, access controls, breach protocol, and regular security audits to safeguard both the AI infrastructure and the sensitive data it processes. Unfortunately, data breaches are far too common and more data sharing means there are more opportunities for breaches. Again, this is why it is important to ensure educators are only working with trusted vendors. School districts should vet and set up data sharing agreements with any vendor that may receive student data whenever possible.