The information gap.

1The gap between records and understanding

Your district’s records may tell you what happened without giving you enough information to understand what is happening underneath it.

Consider a districtwide change introduced across several schools. The guidance is the same, but the work unfolds differently. One school adapts it easily. Another struggles to put it into practice. The implementation record may show that both received the same direction. It may not explain why the experience differs. The people doing the work may understand what is happening, but their knowledge isn’t necessarily recorded in a form leadership can examine and use.

In this paper, we use data to mean recorded values, such as transactions, timestamps, attendance, staffing records, or assessment responses. Information means those records organized and interpreted in relation to a question. The information gap is the distance between what a district can draw from its records and what leaders need to understand to decide well. Sometimes closing it requires new data. Sometimes it requires better use of what the district already has.

This concern has a long history. Russell Ackoff challenged the assumption that giving managers more information would necessarily improve their decisions.¹ Herbert Simon drew attention to a related constraint: information consumes the time and attention of the people receiving it.² Martha Feldman and James March examined how gathering and using information can serve symbolic purposes, including demonstrating competence, apart from its contribution to a decision.³

For us, these ideas lead to a practical question for district leadership: what do you need to understand that your current systems can’t tell you? A superintendent already has student, finance, HR, assessment, and operational systems. The challenge is often connecting what those systems record to what deserves attention, what the evidence can actually support, and what question should come next.

2Why the gap persists

Every information system reflects choices about what to record and how to organize it. A finance system may document a payment without capturing the full reasoning behind a tradeoff. A student information system may record enrollment and attendance without describing whether a school has the organizational capacity to absorb a change. An HR system may show staffing without explaining how responsibilities shifted after a vacancy or restructuring. Those systems can all be accurate and still leave leadership without a coherent organizational read.

Susan Leigh Star and Karen Ruhleder examined how information infrastructure is embedded in work, including the organizational and communication conditions that shape access and use.⁴ Geoffrey Bowker and Star showed how categories and standards shape what systems make visible.⁵ For the design problem considered here, the implication is that a record needs to be understood in relation to the choices that produced it. Similar entries may describe work carried out under very different conditions.

Research in healthcare and education makes this relationship concrete. Joan Ash, Marc Berg, and Enrico Coiera described unintended errors associated with patient-care information systems, including problems in communication and coordination.⁶ Cynthia Coburn and Erica Turner’s framework for data use in education emphasizes interpretation, organizational conditions, and power relations.⁷ These studies address different settings, but both direct attention to how information and practice interact.

We believe that relationship belongs in the design from the beginning. Before we ask an existing district system to answer a new question, we need to know what its records describe, what context is available, what structure sits behind the numbers, and what remains unknown.

3Five conditions, five responses

Before we decide what to build or measure, we need to understand why the question is difficult to answer.

We propose five conditions to help make that distinction. They are design considerations, not an exhaustive or validated classification. Several may apply to the same district question.

The first condition is that the information was never collected. There is no usable record of the practice, relationship, experience, or condition leadership needs to understand. The response is to design a way to collect it. Measurement requires us to define what we are studying, how we will observe it, and what the resulting evidence can support. It also creates obligations to the people providing the information.

The second condition is that the information is fragmented. Relevant records exist across district systems, but their identifiers, definitions, organizational groupings, or time periods don’t match. The response is to connect and structure what exists. That may require a shared definition of a school, function, role, or measurement period. New collection should fill a remaining gap, not duplicate information the district can already use.

The third condition is that the information is unsuitable. A record may be accurate without answering the question. Attendance at professional learning, for example, doesn’t by itself establish a change in practice. Donald Campbell’s work adds a related caution: consequential use of quantitative indicators can create pressure to distort the indicators and the processes they describe.⁸ The response is to clarify what the existing evidence supports and determine whether the original question requires additional measurement.

The fourth condition is that the information is uninterpretable for the intended use. Leadership may have a result without the baseline, comparison, coverage, or context needed to judge its meaning. The response is context: what the result describes, who it represents, what it can be compared with, and what a difference doesn’t establish. A single reading may describe current conditions. Examining change requires a defensible comparison.

The fifth condition is that the information is unavailable where it is needed. It may be held elsewhere, arrive too late, or be presented at a level that doesn’t support the decision. Donella Meadows identifies information flows as places where changes can affect a system’s behavior.⁹ The response is appropriate access: getting useful information to the people who need it, with the context and protections its use requires.

The distinction matters because these responses do different work. Connecting records can’t recover an observation that was never made. A new assessment won’t resolve conflicting identifiers. A finding delivered without context may invite a conclusion the evidence doesn’t support. Understanding the gap helps determine where the effort belongs.

4When information isn’t the problem

Better information doesn’t automatically produce better decisions. A superintendent may understand a problem more clearly and still face limits in authority, resources, policy, staffing, timing, or governance.

Paul Nutt’s research on organizational decisions draws attention to decision-making practices, including how alternatives are considered and implementation is pursued.¹⁰ Feldman and March also describe information use that serves purposes beyond choosing what to do.³ For this paper, the implication is a limit on the claim: providing better information and changing how a district makes decisions are related responsibilities, but they aren’t interchangeable.

Consider four possible obstacles. Authority: leadership understands the issue but can’t act on it alone. Resources: the response is clear, but the necessary time, staffing, or funding is missing. Incentives: people are rewarded for behavior that works against the intended change. Process: information reaches the right people, but there is no established way to review it, decide what follows, and return later to see what happened.

This is why we look beyond the information itself. A system may help reveal these conditions, preserve the evidence around them, or make a decision easier to revisit. It can’t resolve every constraint on its own. Leadership still has to decide what can change, what should be protected, and what needs human investigation.

5Choosing the right response

When better integration is sufficient

Integration may be sufficient when a district already has relevant observations, they are suitable for the question, and they can be connected and shared appropriately. Resolving definitions, identifiers, or access may answer the question without asking employees for more information. Integration still requires work. We begin with existing records to understand their value and limits before deciding what else is needed.

When new measurement is necessary

New measurement is warranted when existing records can’t adequately answer the question. That may concern practice, experience, relationships, or conditions that weren’t captured. It may also concern an unsuitable proxy or a missing baseline. The leadership question should determine the method, not our preference for a particular tool.

That work includes defining the construct, examining whether the items support the intended interpretation, and considering whether the scale can register meaningful differences or change. We also need to understand who responded, how coverage varies, and how the work differs across settings before making comparisons. A result shouldn’t carry a stronger claim than the method supports.

What measurement requires

Burden is part of the design. Every assessment asks employees for time and attention. Its length, frequency, and purpose should reflect the value of the information sought. We should be able to explain why we are asking and how the findings will be used.

Privacy concerns the use and movement of information, not only whether a name is attached. Helen Nissenbaum’s account of contextual integrity connects privacy to the norms governing information flows in a particular setting.¹¹ Applied here, that means making the purpose, audience, access, and limits clear before collection.

Disclosure risk requires its own attention. Group reporting doesn’t automatically protect individuals. Small groups and combinations of detail can make people identifiable. Statistical disclosure-limitation methods address these risks.¹² Reporting thresholds should be part of a broader protection design, not treated as a guarantee by themselves.

Interpretation requires clear boundaries. A self-report describes a person’s account of practice. It shouldn’t be presented as direct observation of everything that occurred. A difference between groups calls for examination, not an immediate explanation. Meadows’s work on feedback also reminds us to consider how returning information can change the setting being studied.⁹ What happens after a result is shared belongs in the context of the next reading.

6Designing information around the decision

Start with what leadership needs to understand. Then design how information will help them understand it.

We treat five design responsibilities as connected. Collection establishes what will be observed, from whom, and at what burden. Structure establishes the definitions, identifiers, organizational levels, and time periods that make the records usable. These choices need to be made with reporting in mind because they determine which comparisons will be possible.

Context establishes what accompanies a result: its scope, the comparison being made, and the limits of that comparison. Access establishes who receives which information, when, and under what protections. Interpretation establishes how a result will be examined and what conclusions it can support.

These responsibilities depend on one another. To report by school, we need to know how schools and other organizational scopes are defined. To protect small groups, we need to limit what can be displayed. To examine change, we need comparability across time. Resolving each choice separately can leave the overall design unable to answer the question leadership started with.

The system should keep evidence, interpretation, and decision distinct. It should also support returning to the question. Repeated measurement can help establish that conditions changed when methods, populations, and context are sufficiently comparable. A before-and-after difference doesn’t, by itself, establish what caused the change.

Artificial intelligence as an application

When a district introduces AI, the same discipline applies. Usage records and technical performance are relevant, but they don’t answer the whole question. Leadership may also need to understand what changed in workload, judgment, coordination, learning, capability, and the work people are responsible for doing.

RAND’s 2025 survey-panel report illustrates the importance of that context. Roughly half of students and of English language arts, mathematics, and science teachers reported using AI for school, while 45 percent of principals reported school or district policies or guidance on its use.¹³ These findings describe reported use and guidance. They don’t establish whether AI improved learning or organizational performance.

Our design position starts with a stronger question: what was true before adoption, what did leadership expect to improve, what changed after implementation, and what human capabilities or organizational conditions moved with it? Adoption isn’t evidence of improvement. Some questions require new measurement. Others require better use of existing records. In either case, the evidence has to fit the claim leadership intends to make.

8What this means for district leadership

For a superintendent and cabinet, this comes down to seven practical commitments.

1. Start with the question and the decision. Establish what leadership needs to understand before choosing a dashboard, integration, assessment, or new technology.
2. Understand the gap before choosing the response. Determine whether the information is missing, fragmented, unsuitable, difficult to interpret, or unavailable. More than one condition may apply.
3. Use existing information where it is fit for purpose. Measure where it isn’t. Neither integration nor new collection is the right answer in every setting.
4. Separate evidence, interpretation, and decision. Make clear what the district observed, what leadership thinks it may mean, and what leadership chose to do.
5. Make uncertainty and protection visible. Show where confidence stops, protect small groups, and don’t let a precise-looking number carry a stronger claim than the method supports.
6. Preserve the district’s institutional memory. Keep the questions, watches, interventions, decisions, assumptions, and evidence connected so the context can survive leadership transitions.
7. Return after action. Use comparable later evidence to see what changed. Treat movement as movement, not automatic proof of cause.

You don’t need a particular product to put these commitments into practice. You do need to treat what district leaders need to know as a design responsibility rather than leaving it to whatever records happen to be available.

Our position is straightforward: start with the decision. Determine what the district needs to understand. Then design the information system so leaders can see the evidence, know its limits, act with judgment, and return later to learn what happened.