LSAT Engine Strategy Blog

In 2020 the Law School Admissions Council was quick to respond to the Covid pandemic with the online LSAT FLEX, beginning in May of that year. Since then, there have been several successful online administrations of the LSAT. The initial FLEX format was 3 sections, one of each type, and without a break. This format was decided upon in part because the test-makers didn’t want to deal with the security challenge of allowing a 10 minute break during the first few online administrations.

One downside of the original FLEX format is that it didn’t include an experimental (unscored) section, because that would’ve created a 4 section test and required a break. Now that the online LSAT has been administered several times, the test-writers have more confidence in their format, and they’ve begun to include an experimental (unscored) section again as of August 2021, along with a 10 minute break between sections 2 and 3.

The reason an experimental section is so important is that it is a necessary part of developing new valid LSAT questions. Every real LSAT question must first appear in an experimental section so that it can be validated. For example, if a certain question has most high-scoring students miss it and most low-scoring students get it right, then it is likely invalid, and it can be said to not be “performing” properly. That question would have to be adjusted or removed before that section could be used on a real LSAT. In addition to validity, the test-makers are also gathering information about the difficulty of individual questions and whole sections.

With the LSAT now offered more frequently and with no new experimental questions, the test-writers are essentially running out of material in 2021. And they ended up having to re-use some LSATs more quickly than they probably would have preferred. Additionally, multi-day tests require more than one real section of each type in order to ensure fairness, so naturally they require more real LSAT questions than the traditional single-day LSAT administration.

Proctor-U (the system used to proctor test-takers) seems to have limits to the number of students that can be proctored simultaneously, hence the need for multi-day tests. Online LSATs have so far seen about 7500 students taking the test each day - likely close to the maximum number the system can handle. This is the reason why some online LSAT administrations have spanned 3 ,4, or 5 days in order to accommodate every test-taker.

You might be thinking “this could present a challenge for fairness. Can the people who take it on later days get information from students who took it earlier?” Thankfully not. The LSAC switches up the real sections throughout the multi-day administration, and LSAC tries to create confusion by having similar topics in multiple sections. For example: 2 real RC sections were shown to students on Saturday of the August exam, and both sections had a passage about Nigeria. But one was about an author, and the other was about the country itself (and its national language). The test-makers use a similar strategy on the LR sections, with multiple questions about speed limits and monkeys.

On Saturday morning of the August LSAT, most students saw 2 RC sections, and these sections were showing up as section 1 and section 4. But the order was different for different students, so that for some the section 1 was the experimental, and for others the section 4 was.

Real RC from Saturday morning (very hard)

  1. sentence reduction due to prosecutorial misconduct
  2. book by Lawrence Krauss quantum physics (comparative) (hardest)
  3. Ansel Adams photography
  4. Nigeria chooses a national language

Experimental RC from Saturday morning (medium)

  1. Supreme Court case water law for Native American lands
  2. Don Quixote
  3. GDP (comparative)
  4. expanding universe; galaxies have a red light shift

Then on Saturday afternoon, another real RC section started showing up instead of the earlier one. This one was a little easier, but also had a passage about Nigeria. This RC section first showed up as an experimental section in January of 2019. But more importantly, it is the real RC section from October 2020, which means that they have reused at least part of the October 2020 exam in August 2021, less than a year later. The test-writers know who took that previous LSAT, and the system ensures that they won’t see any sections from October 2020 a second time.

Real RC section from Saturday afternoon (hard) (October 2020 repeat)

  1. Nigerian author’s novel things fall apart
  2. EMF litigation property value loss
  3. privatization of public works projects (comparative) (hardest)
  4. fungi spores endophytes protecting trees

Meanwhile, most students had real games and LR on Saturday morning, since their RC was experimental. Here’s what those sections looked like:

Real Games from Saturday morning (easier)

  1. employees staff projects
  2. political parties
  3. film premieres and theaters
  4. fencing matches

Real LR from Saturday morning (medium)

  • A20 helps asthma
  • gutta percha
  • video games hours per day
  • monkeys learning language
  • perforated stamps

Saturday afternoon, a second real LR section started showing up:

Real LR from Saturday afternoon (medium)

  • billboards
  • star clusters
  • dinosaur tracks

And there was also an experimental LR that started showing up on Saturday afternoon:

Experimental LR from Saturday afternoon (medium)

  • roasted chicken
  • tyranny

Then on Sunday, students saw more of the October 2020 exam show up again, starting with the games, and followed by a section of LR:

Real Games from Sunday morning (medium) (October 2020)

  1. camp activities
  2. auditions musicians concert halls
  3. clay tablets
  4. fashion brand magazine boutique (hardest game)

Real LR from Sunday afternoon (medium) (October 2020)

  • dark feathered bird bones
  • neanderthals
  • shin splints
  • sand vipers
  • books and binding

There was also an experimental games section on Sunday morning, with an especially difficult game about security guards:

Experimental Games from Sunday morning (harder)

  1. bands on a bandstand
  2. house renovation projects
  3. reporters interviewing candidates
  4. security guards assigned to building shifts over 4 days (hardest of the weekend)

Finally there was also a new experimental RC section on Sunday:

Experimental RC from Sunday (medium)

  1. cowbirds
  2. inca beads
  3. mandatory disclosure
  4. restaurant ratings

Once each of these sections showed up, they continued to be used through Tuesday. In total, there were 4 LR sections (3 real, 1 experimental) , 3 Games sections (2 real, 1 experimental), and 4 RC sections (2 real, 2 experimental) used for the August 2021 exam.

Each of these sections has its own difficulty rating, and harder sections will be associated with easier, looser scoring curves. So even if you get the hardest games section and the hardest RC section offered, your test will still be fair because you can miss more questions to get a given score than someone who had an easier exam. In other words, each combination of sections comes with its own carefully calibrated scoring curve, to ensure fairness across different test forms.

Now that there are new questions being tested, the test-makers will have more options when composing future tests. And since the experimental section is a sign of what kinds of questions to expect in the future, it was nice to see some easier RC sections in the pipeline.

We expect the test-makers to release the October 2020 LSAT for public consumption this year, now that it has been fully used twice.

Posted: 9-15-2021