A major goal of instructional design is to teach students more in less time. Another major goal is for students to be able to apply what they’ve learned to problems that were never directly taught. So far, so good, but neither goal can be achieved to its fullest extent without a careful analysis, and that work happens long before a single lesson is authored.
In this post I want to take up that neglected piece of instructional design, the content analysis. I will show what one looks like in four different subjects, and then make the case that building a course from a list of standards or released test items is a poor substitute for it.
A content analysis is the search for generative relations inside a content domain, and the arrangement of the content so those relations become the organizing structure of the course. A relation is generative to the degree that learning it lets a student respond correctly to items nobody taught them. In the DI literature, a content analysis is sometimes called a “sameness analysis,” which is a better name because it says what you’re actually looking for: what is the same across a universe of items that looks, on the surface, like a pile of unrelated things? If you find and teach that this is just like these, and that this strategy works for all these other examples too, then every untaught member of the universe comes along for free. So when the instructional designer spots a sameness in the content, the test is whether teaching it connects students across more of the course, in less time, than teaching the cases one by one.
Four examples make content analysis concrete.
Spelling. Most spelling instruction is designed without a proper content analysis. Words get grouped by theme, by holiday, by unit, by what students are interested in. Grouping words this way would only make sense if spelling contained no generative relations worth teaching, so you may as well organize by randomness or flavor of the week. But spelling has at least two powerful “samenesses” in it. The phonic strategy exploits the sameness of phoneme–grapheme correspondence, which means words can be grouped by the sounds they contain and the letters that spell those sounds. A student who has been taught how the /ā/ sound is usually spelled at the end of a word can then spell a whole class of words nobody put on a list for them, rather than memorizing each one on its own. In addition, the morphographic strategy exploits the fact that morphographs hold their spelling across words, so a limited set of combining rules generates enormous coverage. Spelling Through Morphographs teaches 750 morphographs and 14 rules, and students can derive the spelling of over 12,000 words. The payoff also extends to words that were never in the program. When a student meets an unknown word, they can break it into morphographs they already know and work out the spelling from there, rather than treating each unknown word as one more item to be memorized.
Math Facts. The default assumption in math fact teaching is that 100 addition facts and 100 subtraction facts are 200 paired associates to be memorized. But what if there was a better way that leveraged the power of generative relations, of sameness, between the facts, to reduce that number considerably? Like, take the fact that 2, 3, 5 is one fact family that produces four distinct facts. If I learn 2+3=5, I also learn to quickly “turn around” that fact to be 3+2=5, but the real magic is that if I can learn these facts as a family, my subtraction load is greatly reduced. If I know about the relations, I can quickly retrieve 5-2=3 and 5-3=2 when my focus is on the family of 2, 3, and 5, rather than having the facts treated all as separate and random. Additionally, a small set of rules can be taught that help even further to reduce the space. When I learn the rules for adding and subtracting zero and one, I now cover 72 facts on their own, and 36 families cover the remaining 128. Four rules and 36 families instead of 200 items. And the families keep paying out well past the fact table, because another invisible sameness in the content is that a word problem in the primary grades is just a family with one member missing. Two of the numbers are given, one is hidden, and the kid’s job is to work out which slot is empty. A student who learned facts as families doesn’t have to be talked into seeing that structure. It’s almost impossible for them not to see it, because that missing number is sitting in an organizing structure they’ve been constructing since first grade.
Earth Science. Look at most earth science courses and you’ll find geology, weather, and oceanography sitting in separate units, taught months apart, with nothing connecting them. A kid studies how hail forms in October and plate tectonics in April and has no reason to suspect the two have anything to do with each other. But they do. Fluids expand when they’re heated and become less dense. Less dense fluids rise. Rising fluids lose contact with their heat source and cool. Cooling fluids contract and become denser. Denser fluids sink. Once a student owns those five relations, they can explain a pot of water on the stove, the air circulating in a room, on-shore and off-shore winds, the updrafts that build hail inside a thundercloud, the trade winds, the deep ocean currents, and the movement of rock in the mantle. Convection is the big idea, and all these topics are its instances. Teaching with big ideas as organizing principles of the course helps the student to make sense of the universe of examples that are in it, as well as what happens when the kid goes home and boils pasta. Teach the course as discrete skills from a list, and each phenomenon becomes its own fact to remember, with whatever connects them left for the student to notice on their own.
U.S. History. The default organization of many history courses is chronology and a parade of facts. Each unit arrives with its own people, groups, dates, and causes, and the only thing linking the Stamp Act to the Louisiana Purchase is that one happened before the other. But Doug Carnine showed that history benefits from a content analysis too. People and groups in history are almost always acting on a problem, and those problems are mostly economic problems or problems about the rights of people. They have five solutions available to them: accommodate, dominate, move, invent, or tolerate. And there are three possible effects: the problem gets solved, the problem continues, or a new problem is created for somebody else. Problem, solution, effect. Layer on a few more samenesses and the frame gets sharper. The four factors of group success, which are capability, resources, leadership, and motivation, let a student work out which of two conflicting groups is likely to prevail and why. The four stages of cooperation, which run from getting together to discuss a problem, to occasional cooperation, to regular voluntary cooperation, to legally binding cooperation, give students a way to place the Albany Plan, the Continental Congress, the Articles of Confederation, and the Constitution on a single progression rather than treating them as four unrelated events. And at each step you can ask students how the geography, the climate, and the culture shaped what counted as a problem in the first place, which solutions were even available, and how the outcome was read afterward. We often talk in educational discourse about teaching kids to “think like a historian” without ever operationalizing what we mean. In Carnine’s analysis, students are taught to deploy these mental operations on the historical events in the course, but the beauty of it is that they can also make predictions and “think critically” about events nobody put in front of them. Carnine’s group tested exactly this in the studies on their program, Understanding U.S. History. In a year-long field test reported by Carnine et al. (1996), students using the conceptually organized textbook outperformed the comparison classes not only on tests drawn from their own book, but on tests drawn from the traditional textbook the comparison classes had been using all year. They could operate on the other book’s content better than the students who were taught from that book.
The common trap is to derive the course from the standards document, or from the released test items that go with it. A developer works through it entry by entry, treating each one as a unit of instruction to be authored. In current practice the authoring is often automated: each standard is handed to a language model, a lesson comes back, and the course is assembled from the results. The objection I am raising has nothing to do with AI authorship. The same outcome follows when this kind of shoddy work is done slowly by a committee of humans, as plenty of curriculum materials stamped with the “high-quality instructional materials” label demonstrate in living color.
Standards documents and item banks serve a separate purpose. They exist to sample and report on achievement, so they are organized by grade and cut up in a way that often does not respect the subject or the generative relations within it. Slocum and Rolf (2021) name the problem directly: instructional designs derived from skill assessments tend to omit the step of identifying the generative relations that would produce the repertoire the assessments sample. An assessment tells you which behaviors a student can perform. It does not tell you how to organize a course that is economical and intellectually rigorous.
Four consequences of skipping the content analysis follow, and they overlap.
The first should be obvious: skipping the content analysis increases the sheer volume of instruction. Content that shares a common structure is taught several times over in several places, and the big ideas that would have let one explanation serve many cases are never surfaced and made explicit. The cost is paid in instructional minutes, which are fixed, and in retention, because isolated procedures that connect to nothing are the most difficult material for students to remember. Dumber and dumber still, I’ve seen developers try to solve this with sheer repetition, running kids through more and more practice reps in the hope that enough reinforcement will fix the upstream problem. The practice load goes on accumulating in proportion to the number of skills rather than consolidating around a smaller set of relations.
The second consequence of skipping the content analysis is the proliferation of methods. Standards and released test items frequently describe the same underlying operation in several representational forms, and an author working item by item will render each form as a distinct procedure to be taught. Students are shown a succession of ways to approach what is structurally one problem, with multiple models, multiple representations, and multiple solution paths, and with no principled account of when each applies. This consumes practice time that would otherwise be spent bringing a single reliable strategy to fluency, and it obscures rather than reveals the sameness that makes generalizable, high-mileage strategies worth teaching.
The third is inconsistency of language. A good content analysis produces a vocabulary that is sequenced. Child-friendly words come first, so that students can talk about the content before they can name it formally, and the technical terms arrive later, once there is something for them to attach to. The progression is gentle, and it is deliberate. Without an analysis there is no progression at all. Lessons authored independently against separate standards do not share a controlled set of rules and vocabulary, so the same relation picks up several different names across the course, terms get used interchangeably when they do not in fact mean the same thing, and the strategy amounts to throwing as much language at kids as possible to see what sticks. A high performing kid may be able to sort through all that mumbo jumbo. The kids who need precision in instruction most will not.
The fourth follows from the first three. Once a course is defined as an arbitrarily organized list of separately assessed performances, every signal the developers get back during field testing is item-level, and design decisions start bending toward whatever increases performance on those items. That pull is hard to resist, because it comes with data attached. Item-specific behaviors are measured constantly during field testing, so gains in them are visible. A content analysis cannot be measured by a single item, if it can be measured at all, so its presence or absence shows up nowhere in the reporting. Student outcomes may well improve through this process, slowly and steadily, against the list the course set for itself, and everyone concludes the instructional design is working. But the course is still bloated with skills taught in isolation, still crowded with competing methods, and still inconsistent in the language it uses for the same ideas. Most of all, students come out without the generative repertoires that let them make sophisticated connections between areas of the subject and the world around them.
I am not saying standards have no place. Checking a finished course or app against a state’s standards is reasonable and often necessary work. It answers the question of whether anything required has been left out, and it is (sometimes) the right instrument for that question. The objection is to using standards in place of a content analysis rather than alongside one. A coverage check is a verification step, applied after the organization of the course has already been settled, and it can only detect omissions. It has nothing to say about whether the content has been arranged well, because a course in which every topic appears as its own isolated skill will pass a coverage check exactly as cleanly as a course built on a small set of generative relations. When the standards document is used as the design source rather than as the audit, the organization of the course simply inherits the organization of the reporting categories, and the analysis that would have produced a more effective, efficient, and enjoyable arrangement never takes place.
What most likely explains the persistence of the test-prep, standards-first approach to instructional design is that almost nobody can see what is wrong with it. A good content analysis is invisible, and so is the absence of one. Only a person with intimate knowledge of both the underlying design and the subject matter can tell the two apart. A content analysis lives in the sequence, in the composition of item sets, and in which things are treated as the same and which are treated as separate. A course built by starting with an arbitrary list of standards therefore ships looking entirely respectable: aligned to the standards, complete in coverage, and often professionally produced. We cannot keep setting the bar this low, where a course counts as good if someone fed a standards document to a large language model, churned out a lesson for every entry, and called the result a curriculum.
Start from the content analysis rather than from a list. In many domains, like phonics, these analyses already exist and do not need to be reinvented. But for courses for which a content analysis has not been conducted, the instructional designer’s task is to identify the generative relations that govern the domain and to build the course out of them, which is an entirely different activity from authoring a single lesson. And because neither instructional designers nor developers are necessarily experts in the subject, it is often necessary to bring in a subject-matter expert to advise on the accuracy of the content, on which parts of it are genuinely important, and on how far the explanations can be simplified while remaining true.
With the analysis settled, the course takes a particular shape, organized around big ideas, tracks, and terminal skills. Each track is a strand of instruction that develops gradually across many lessons, and the tracks are integrated so that they arrive together at the terminal skills. The terminal skills state what a learner will be able to do by the end of the level, and they define the universe the course is responsible for. Any single lesson is therefore a slice through several tracks, each at a different stage of development. Nothing is in that lesson because a standard called for it. Each item is there because a track needed it at that point, and because the analysis places it alongside the items around it.
Only then does the team turn to authoring the instructional sequences. This includes a great many nuances that I will be writing about on this Substack in the coming weeks: assembling an adequate range of examples for a given skill; interpolation, stipulation, and extrapolation from example sets; identifying minimum discriminations; teaching the general case before the exceptions; observing the three-day rule for new skills and the twenty percent rule for new material within a lesson; writing absolutely unambiguous wording; identifying everything a student response requires, which means reading the lesson as a naive learner would and avoiding assumicide; sequencing skills within a track and sequencing tracks against one another; fading scaffolding; and selecting vocabulary.
A content analysis is the instructional design move that uncovers the samenesses in a domain. Those relations then become the organizing structure of the course.
Teaching more in less time and getting transfer to untaught problems can only be achieved to the extent the content analysis allows.
Starting from standards or test items produces too much teaching of separate skills, a circus of competing methods, inconsistent wording across the course, and a design that optimizes for item-level performance rather than for strategies that generalize.
Standards are the right tool for checking coverage once a course is built. They are the wrong tool for deciding what the lessons are.
Once the content analysis is ironed out, begin reverse engineering from terminal skills, develop the content as tracks that are gradually integrated to arrive at them, and verify that the course accounts for everything in the universe it defines.
With thanks to Don Steely, whose correspondence and insights shaped this post throughout.
References
Carnine, D., Caros, J., Crawford, D., Hollenbeck, K., & Harniss, M. (1996). Designing effective U.S. history curricula for all students. Advances in Research on Teaching, 6, 207–256.
Slocum, T. A., & Rolf, K. R. (2021). Features of Direct Instruction: Content analysis. Behavior Analysis in Practice, 14(3), 775–784. https://doi.org/10.1007/s40617-021-00617-0
%26stroke%3Dnone%26strokeWidth%3D3.6){kind=small}
