How Asian test-prep companies exposed SAT's loopholes

On the morning of Saturday, March 5, students gathered at test centres around the United States to take the SAT, the all-important college entrance exam.

The day was momentous – not simply for the test-takers but also for the College Board, the not-for-profit that owns the exam. The organization was debuting an entirely new version of the SAT whose redesign was years in the making.

In Asia, test-preparation companies were eager for information. Any details about what was on the new SAT might be invaluable to their clients. That’s especially true because for years, the College Board routinely has reused SAT tests overseas after first administering them in America.

East Asian cram schools have repeatedly exploited that practice to breach the SAT, and the College Board has come to see the test-prep industry as a daunting adversary. For the first offering of the redesigned SAT this month, the organization imposed an added security measure: It banned tutors and other non-students from taking the exam that day.

The battle to safeguard the new SAT was on. It was lost almost as soon as the test began.Test-prep companies had posted teachers outside US test centres, ready to grill exiting students about what was on the exam. Within hours, American test-takers headed online to discuss the new SAT in detail.

On the popular website College Confidential, students described portions of the reading section from exams given on March 5. There was an essay on plate tectonics. A letter by the 1960s labour activist Cesar Chavez. A scientific paper about baby fat. A passage from a Michael Chabon novel. And more. Test-prep companies in Asia picked up this chatter and reported back to clients.

Then, last week, Reuters was shown two documents that reveal far more substantial holes in the SAT’s defences.

Both documents contained entire sections from exams given on March 5. The College Board said it has a “long-standing policy” not to comment on what may be on an exam. Reuters verified the authenticity of the documents nonetheless with people familiar with the new SAT’s content - including students who took the test.

The first file, offered free by a Chinese online test-advice company called SAT Helper, reconstructs one version of that day’s exam booklet. It had a 52-question reading section with five text passages – including the Chavez letter and the plate tectonics essay.

The second document was shown to Reuters by a Chinese tipster who had warned the College Board last year about security breaches. It contained images of another version of the March 5 test. Among its reading passages? The Chavez letter, the baby-fat paper, the plate tectonics piece and the Chabon novel.

Authentic exam documents

Three high school students reviewed the documents and confirmed that the material came from the actual March 5 SATs they had taken. Reuters is not naming them because test-takers agree when they register for the exam not to disclose what's on it.

“That’s literally the one I took,” said one student, a high school junior in Maryland.

“It would have been better if I’d seen it before the test,” another high school student, a junior in Texas, said laughing.

“The questions started coming out as soon as I finished the test,” said a third student, a junior in Florida. “I thought this time the College Board had released them itself.”

The College Board says that test security and delivering valid scores are central to its mission. In addition to barring non-student test-takers from the March 5 exam - a practice that will continue on most test dates - it has taken other measures in recent years to thwart the Asian prep industry. SATs are now shipped to and from some test sites in lock boxes, and the College Board regularly sends out “take-down notices” if it sees test material online.

But as the brand-new bootleg test booklets show, the cram schools continue to find ways to subvert the defences of the College Board and its security contractor, Educational Testing Service of Princeton, New Jersey.

What has made such breaches especially damaging in recent years is the College Board’s routine reuse overseas of SAT test material previously given in America. The recycling of exam items has enabled test-prep operators to provide international students an advance look at reading passages, grammar problems and other material that may be on future tests. Sometimes, the cram schools even create answer keys for their clients.

The College Board confirmed to Reuters that it plans to continue recycling test material. The redesigned SAT will be administered for the first time overseas in May, and it’s unlikely that the first foreign tests will include material that was administered March 5 in America. Still, the fact that the new test booklets were so quickly circulated demonstrates that the redesigned SAT remains vulnerable.

“We’re working against cartel-like companies in China and other countries that will stop at nothing to enrich themselves,” said John McGrath, the College Board’s senior vice president for communications and marketing. “These bad actors will continue to lie, cheat and steal to the detriment of students who work hard and play by the rules.”

The recycling problem

USadmissions officers who were briefed on what Reuters found said the College Board ought to stop recycling exam material. “What they should do, step one, is consider ending the practice of reusing test content,” said Joy St. John, dean of admission at Wellesley College in Massachusetts. If applicants have seen exam material before taking the test, St. John said, “our ability to select students who are the best fits for Wellesley is really compromised.”

Even some cram-school operators agree, saying that the continued reuse of test material will make the new exam an easy mark.

The redesigned SAT “won’t resolve the fundamental problem, unless they have a continual flow of new questions, and use every test only once,” said Peng Wu, a general manager at Sanli, a Shanghai-based test-prep chain. Sanli created booklets of past test material to help students prepare for the old SAT. One former client, now a student at the University of California, Los Angeles, told Reuters a Sanli booklet helped him score a perfect 800 on the critical reading section of the SAT.

The likelihood that questions from the redesigned SAT will be recycled on future tests “is definitely a good thing for us,” Wu said.

In a letter to Reuters, College Board vice president Stacy Caldwell defended the organization’s handling of security. The College Board and its contractor, ETS, stand behind the validity of the test scores sent to US colleges and their actions “to protect the integrity of the exam,” she said. Ray Nicosia, who heads the Office of Testing Integrity at ETS, said the number of people who cheat on the SAT is “far less than 1 percent.”

With the new test, Caldwell pledged that the College Board “will continue to take bold actions to stop cheating and theft.”

How the College Board will succeed at that mission is unclear. Internal College Board documents show that test-preparation centres have been able to penetrate security for years, often by exploiting the routine reuse of test material overseas.

Over the past three years, interviews and documents show, the College Board has often tried but failed to plug the flow of leaks. At other times, Reuters found, the College Board decided to go ahead with exams even after being warned that test material it had previously administered was in wide circulation.

Some people in the testing and teaching professions say they were disheartened by how the College Board or ETS handled evidence of possible breaches.

The tipster in China who provided Reuters with a copy of the new SAT booklet said he sent the same material to an ETS investigator on March 18. He said he hasn’t received a response. A spokesman for ETS referred questions on the matter to the College Board, which said it does not comment “on specific test content.”

“Beautiful vision”

By 2012, the College Board had decided to redesign the SAT. Market pressures played a role in the decision to remake the test.

For years, the college entrance exam industry has been under strain. Several million American high school students take the SAT or the rival ACT exam each year. But a growing number of UScolleges and universities have stopped requiring the tests, and some educators question their usefulness in predicting a student’s success in college.

At the same time, the New York-based College Board has been losing market share in the United States. Its competitor, ACT of Iowa, attracted more test-takers than the SAT for the first time in 2012. The SAT remains No. 1 overseas.

Within months of becoming president of the College Board in the fall of 2012, David Coleman set out his “beautiful vision” for the re-engineered SAT. It “will appeal to students over ACT,” he wrote. It would be “more focused, transparent, and specific” than the ACT, he added. And unlike the ACT, it wouldn’t include a science test.

Coleman declined to comment for this story.

As College Board leaders began working on their new test, they encountered major problems protecting the existing SAT.

In May 2013, cram schools in South Korea, known as hagwons, succeeded at obtaining material from the exam the College Board intended to give that month. It isn’t clear how the material leaked. The security breach was discovered by South Korean law enforcement officials.

The College Board responded to that leak by cancelling the scheduled exam. “Because a number of test-takers have likely already been exposed to these test materials, we had no choice,” the organization said in a statement at the time. The College Board declined to say how many students saw the exam ahead of time, or whether it knows how the hagwons got the questions.

The decision was drastic. Never in the history of the SAT, first given in 1926, had the College Board cancelled an exam sitting across an entire nation for security reasons.

After the incident, officials assessed how many untainted versions of the exam remained. The College Board learned that half of the exams in its inventory had been compromised to some degree. To combat the breaches, the College Board reduced the number of test dates in South Korea and two Middle Eastern countries – to four times a year rather than six. Doing so would limit the potential for cheating, officials concluded.

But College Board officials told Reuters that they went forward with a full slate of exams in Greater China, where far more students take the test, because they found no evidence that test material had been stolen there.

“Celebrating” in the hall

Not long afterward, more breaches emerged in China.

In January 2014, a student from Concordia International School Shanghai took a scheduled break during an SAT sitting in one of China’s largest cities.

When the student stepped into the hall, he saw groups of kids “talking about the exam, sharing the answers, and ‘celebrating’ about the test,” according to an account the student wrote to the school’s principal. The school, which declined to name the student, provided the account to Reuters.

The test given in China that day included some of the same reading sections that had been used on an SAT given in December 2012, just 13 months earlier. A Shanghai test-prep centre, Veterans Education, had obtained an exam and given it to its students for practice.

To determine the extent of the problem, administrators at Concordia International School questioned all of their students who took the SAT that day. Nine students admitted they had studied at Veterans Education and had seen the entire test in advance, said Concordia principal Nicholas Kent. Concordia reported their names to ETS, which cancelled their scores, he said. ETS declined to comment on the episode.

It’s unclear how many other students saw test material in advance. Ben Yoon, the president of Veterans Education, confirmed that he shared the December 2012 test with students but said he had no idea it included actual questions from an upcoming exam.

“My institute gets all of its teaching materials through open sources,” including on the Internet, Yoon said in a written statement. “I was shocked and appalled by the very possibility that such test materials could be obtained so easily.”

By October 2014, more problems had surfaced. A few days before the October test was administered, a tutor in South Korea sent a full copy of what he believed was the exam to the College Board, the tutor told Reuters. The tutor, who declined to give his name to the College Board, said he warned the organization about other leaked tests - and about the brokers who were selling the exams. The exam was given as scheduled.

But later that month, the College Board withheld issuing test scores in parts of East Asia while it investigated possible cheating on the exam. It took the same step for the next three Asian test sittings as well. Withholding scores is a serious step: It means students applying to universities are in limbo while ETS determines whether their scores need to be cancelled. In the October exam, scores were withheld for every test-taker in China and South Korea, where a combined 55,000 tests were taken last school year. The College Board didn’t disclose the outcome of the investigation.

A tutor in Korea

That November, Linfeng Liu, now 20, took the SAT in Hong Kong. She said she was happy to recognise five questions on vocabulary and reading comprehension. Paraphrased versions of the same questions had been in a booklet provided by the test-prep centre she attended in China, she said.

“It helped,” said Liu, now a freshman at Dickinson College in Carlisle, Pennsylvania. “I did those questions more quickly, so I had more time for the reading comprehension section.”

The College Board’s Caldwell says that cram schools, not students like Liu, are to blame for such incidents.

The next month, December 2014, the College Board issued a statement about the problems in Asia. “Over the past three months, organizations and individuals have illegally obtained and shared test materials for their own profit,” the statement said.

The South Korean tutor - the person who had alerted the College Board to a possible leak ahead of the October 2014 exam - once again tried to warn officials. He showed Reuters an exchange he had with an ETS investigator in January 2015.

Attached to his email was what he believed to be the upcoming exam. Despite the tutor’s warning, the College Board went ahead with the exam on Jan. 24. Afterward, it again delayed releasing test scores.

“I was shocked and indignant that the College Board decided to proceed with the exams anyway,” the tutor said.

An ETS spokesman confirmed that one of its investigators “was in contact with this person. We have no additional comment other than the fact that ETS and the College Board take all tips seriously and investigate them thoroughly.”

“Completely leaked”

By September 2015, ETS was being warned by another tipster that the SAT given in March in the United States was circulating in Asia. This tipster, based in China, was one of the sources who sent Reuters images of the redesigned SAT earlier this month.

In his September 2015 email to ETS, the tipster said that the SAT given just six months earlier in the United States was circulating in Asia. The material had been “widely dispersed,” the tipster warned. As evidence, he attached a text file containing what he said were parts of the exam.

An investigator for ETS quickly replied. “The documents will be reviewed asap,” the investigator wrote, according to emails reviewed by Reuters.

Nine days later, the tipster corresponded with the investigator again and sent some screenshots of the exam. The investigator asked if the March 2015 test given in North America was “widely spread out.” The tipster responded that it “is completely leaked out” and said that he had a scan of the entire test. He also attached more evidence: photos of what he believed to be the test booklet itself. The tipster told Reuters he never heard back.

The tipster didn’t know it at the time, but the College Board was about to reuse the March 2015 USexam in China and elsewhere just a few days later, in its October sitting.

Despite the warnings, the test was given as planned. Caldwell, the College Board vice president in charge of the SAT, said ETS investigated the matter and didn’t think there had been “widespread student access” to exam material. She added that the “information lacked the specific facts to deem it credible.”

But two weeks before the exam took place overseas, parts of the test could be found on the Internet.

On Sept. 18, Zhan.com, a Chinese website with more than 500,000 estimated monthly visitors, posted several of the reading passages from the test the College Board was about to reuse.

The site is run by a Shanghai-based online test-prep company called Little Zhan Education. A Little Zhan teacher said the school only posts “mock papers,” but he acknowledged that “some of the questions might match those that actually appeared in the test.”

As the College Board went ahead with the SAT on Oct. 3, the tipster tried emailing admissions officers at 36 top American colleges.

“The reliability of the test has been severely compromised since last year,” he wrote that day, using the pseudonym “China anticheating.” “As you can see from the pictures I attached, these are the reading sections of the SAT test being conducted now.”

Some of the emails were sent to incorrect addresses. But officials at a few of the schools recalled receiving the message. Gregory Roberts, dean of admission at the University of Virginia, said his office contacted the College Board after reading the email.

“We spoke by phone,” Roberts said. “They were aware of these emails.” The College Board declined to comment.

On Jan. 20, days ahead of another exam overseas, the “China anticheating” tipster sent Reuters 47 photographs. The images showed what appeared to be parts of six sections of the SAT test booklet used in the United States the month before, as well as a list of answers. The tipster said he sent the same warning, with the photos, to a College Board official.

A PREDICTION

In his email to Reuters, he also made a prediction: the version of the test in his possession was “very likely” to be reused three days later, on Jan. 23, at the international sitting of the exam.

He was right: Parts of the test that was to be given that day were identical to the version given previously in America. He also was right about the exposure. Reuters found that dozens of questions on the test were shared and discussed on the popular website reddit.

The College Board tried to contain the damage. On Jan. 21, it disclosed that it was cancelling the scheduled SAT on Jan. 23 at every test centre in mainland China and Macau.

But elsewhere in Asia, the College Board gave the SAT as planned. The next month, the College Board notified American universities that it was delaying the scores of an undisclosed number of students because of “a security incident.” The delays affected test takers in Singapore and other Asian cities, according to a guidance counsellor in the region.

This month, Asian test-prep centres targeted the redesigned SAT.

For the first sitting of the test, on March 5, the College Board barred people who aren’t applying to college from taking the SAT. The step prevented cram school teachers who’d registered for the test from getting a direct look at the exam. Test-prep operators found ways around the measure.

Sanli, the Chinese test-prep chain, says it sent 11 teachers to the United States to collect information on the redesigned exam. They debriefed 40 Sanli students studying at UShigh schools who took the new SAT as they exited test centres, according to Wu, the general manager. Sanli presented its findings at a seminar at a Shanghai hotel.

Other Asian operators harvested material from the new exam simply by going online. After the test ended, the website College Confidential was full of talk about the exam. The site said last week that it received and complied with one take-down request from the College Board after the new test.

And within hours of the test, a Chinese SAT coach who calls himself “Roy” had pieced together items that were on the reading and optional essay sections. Soon he was sharing his take on the test in a video posted on social media.

“Have you read College Confidential?” Roy said in an interview. “All the detailed material is there.”