By David A. Fahrenthold
The Washington Post
BOYERS, Pa. — The trucks full of paperwork come every day, turning off a country road north of Pittsburgh and descending through a gateway into the earth. Underground, they stop at a metal door decorated with an American flag.
Behind the door, a room opens up as big as a supermarket, full of five-drawer file cabinets and people in business casual. About 230 feet below the surface, there is easy-listening music playing at somebody's desk.
This is one of the weirdest workplaces in the U.S. government — both for where it is and for what it does.
Here, inside the caverns of an old Pennsylvania limestone mine, there are 600 employees of the Office of Personnel Management. Their task is nothing top secret. It is to process the retirement papers of the government's own workers.
But that system has a spectacular flaw. It still must be done entirely by hand, and almost entirely on paper.
The employees here pass thousands of case files from cavern to cavern and then key in retirees' personal data, one line at a time. They work underground not for secrecy, but for space. The old mine's tunnels have room for more than 28,000 file cabinets of paper records.
This odd place is an example of how hard it is to get a time-wasting bug out of a big bureaucratic system.
Held up by all that paper, work in the mine runs as slowly now as it did in 1977.
"The need for automation was clear — in 1981," said James W. Morrison Jr., who oversaw the retirement-processing system under President Ronald Reagan. In a telephone interview this year, Morrison recalled his horror upon learning that the system was all run on paper: "After a year, I thought, 'God, my reputation will be ruined if we don't fix this.' "
Morrison was told the system still relies on paper files.
"Wow," he said.
The existence of a mine full of federal paperwork is not well known: Even within the federal workforce, it is often treated as an urban legend, mythic and half-believed. "That crazy cave," said Aneesh Chopra, who served as President Obama's chief technology officer.
But the mine is real, and the process inside it belongs to a stubborn class of government problem: old breaking points, built-in mistakes that require vital bureaucracies to waste money and busy workers to waste time.
In some cases, the breaking point is caused by a vague or overcomplicated law.
In New Jersey, for instance, one researcher found that the approval process for a bridge project dragged on for years in part because officials were required to do a historic survey of all buildings within two miles and to seek comment from Indian tribes as far away as Oklahoma.
In other places, what breaks is the government's technology.
The rollout of HealthCare.gov, of course, was ruined by glitches in the website, but there are other examples: The Census Bureau had a failed experiment with hand-held computers, then reverted to paper, which cost up to $3 billion extra. The Department of Veterans Affairs had trouble with an online records system and, while they struggled with it, accumulated so much paperwork in one office that auditors feared the floor might collapse.
Obama took office with the hope that these hang-ups could be separated from Washington's endless wars over the size of government. In theory, these are problems everybody wants to fix.
"The question we ask today is not whether our government is too big or too small, but whether it works," Obama said in his first inaugural address.
In many places, however, these federal systems still don't work well. Some of the explanation can be found here, in this baroque underground bureaucracy.
Over the past 30 years, administrations have spent more than $100 million trying to automate the old-fashioned process in the mine and make it run at the speed of computers.
So now the mine continues to run at the speed of human fingers and feet. That failure imposes costs on federal retirees, who have to wait months for their full benefit checks. And it has imposed costs on the taxpayer: The Obama administration has now made the mine run faster, but mainly by paying for more fingers and feet.
The staff working in the mine has increased by at least 200 people in the past five years. And the cost of processing each claim has increased from $82 to $108, as total spending on the retirement system reached $55.8 million.
In a statement issued Saturday, OPM Director Katherine Archuleta said: "I do not believe that the current level of service is acceptable." She added that modernizing the system is a priority for her.
In an interview inside the mine this month, another federal official called the operation "very successful."
But that official balked when asked if it was modern. "What does 'modern' mean?" the official said. The OPM allowed a reporter in the mine on the condition that interviews with some officials there would not be conducted on the record.
This is how the mine works:
Step 1 begins when a federal employee submits retirement paperwork to his or her own agency. That happens at least 100,000 times a year. Within a few days, the government starts sending "interim payments" to the retirees — checks worth about 80 percent of their full pensions. This is meant to tide them over while the mine works on the case.
Then, the paper begins to move. The retiree's agency assembles a paper file of personnel records and ships it off at rush speed.
Most agencies send these files using FedEx, and their packages arrive the next day. The Postal Service, however, ships its own retirees' paperwork by U.S. mail.
Its packages arrive in two days, officials in the mine said.
Nearly all of those packages come here — over the winding roads, into the tunnel and through the door with the American flag.
"You don't forget that it's a cave," said Ashley Weber, a former temp who worked on the mine's incoming files. "But they try to make it look as not-cave-like as you can."
But why is it in a cave at all?
The answer to that question is that, back in 1958, the U.S. government was in the market for storage space. It needed 30,000 square feet to hold personnel files that were being relocated from a building in Washington. Officials looked at buildings in Richmond, Va., and Syracuse, N.Y., before choosing this place, an underground complex where 1,000 workers had once cut limestone to feed the steel mills.
A private company had turned the place into an enormous safe-deposit box: safe from the weather and the Soviets, kept naturally cool as a cave. Today, the complex is owned by the company Iron Mountain, which leases out other caverns to store old Hollywood movie reels and photo archives.
The government moved its old records here in 1960. At first, it was just a file room. Records were shipped to Washington for processing. But over time, the government began to hire more people to work in the mine itself.
They worked hard. And since there were few other office jobs available in this rural area, they tended to stay.
"Nobody up there goes on to another job. You can work Monday through Friday, 8 to 5. . . . There's mostly overtime, if you want it. They're really flexible about using leave," said Patty DeCaria, 57, who retired last year after 38 years in the mine. DeCaria said she also enjoyed the sense that she was helping people who deserved it. "People don't leave Boyers," she said.
Still, at best, it's a good job in a bad place.
In the winter, employees enter the mine in the dark and leave in the dark. Food must be brought in from outside, because you can't have an open flame in a mine. So there is a pizza guy, with a security clearance, who arrives every day at 11:30 a.m. Another vendor, Randy Armagost, trucks in hot lunches and an assortment of at least four deep-fried items every day.
"People are crabby. They're miserable. I mean, you can't blame them. They never see any sunlight," Armagost said. "I'm only down there for 2 1/2 hours a day, and I can't stand it."
For workers inside the mine, Step 2 in the paperwork process is to take the retiree's newly arrived file and match it up with any records already stored in the mine.
In about 15 percent of the cases, that means a long walk into the mine's eight massive file caverns. Inside, they are empty enough to be spooky.
"I heard rumors of ghosts, out in the files," DeCaria said. "They just pull drawers open."
In most cases, however, Step 2 can be completed without a walk. The retiree's files have been scanned into a digital archive and can be looked up on a computer.
But there's a problem: All the information must go in the retiree's manila folder.
And you can't put a computer file in a manila folder.
"We do print them out, right now. But we won't in the future," said Doug Berger, who supervises this operation. The printed-out documents are put in the folder, and it continues.
Now, Step 3: The file moves around the corner to an adjacent cavern. The workers there have a vital but frustrating job. They must call, email, fax, badger and harass workers in other federal agencies to find paperwork that has been left out of the file.
"I used to chase people for months — literally — for one signature on one piece of paper. You want to talk about an egregious waste of taxpayer money?" recalled said one worker who left the mine recently and declined to be named because of fears of retribution.
Step 3 usually takes a few days to a few weeks. But if anybody's file is misplaced along the way, it slows everybody's work down.
"On a daily basis, we would get from five to 50 emails, asking everybody to take time out of their day to search their desks for case files," the former worker said. That worker said the experience of hunting down lost paperwork and lost files inside an underground cavern had been bad enough to force a career change.
The worker's new job: setting off explosives.
"I'm handling live ordnance on a daily basis, just to get out of there," said the worker, whose company blasts holes in the ground for oil and gas wells. "One of the five worst jobs in the world was a great alternative to being down there."
Finally, when all the file's missing papers are found, the file moves on to a new set of workers in a new set of caverns.
This is Step 4.
Now that all the retiree's digital data have been turned into paperwork, these workers turn that paperwork into digital data again. They type all the pertinent information into a computer, by hand.
"You can do a case in as little as an hour," said Bonnie McCandless, the president of the center's local labor union, whose job is entering this data. "Or you can do a case as long as eight hours, or two days."
The task takes so much time in part because Congress has made the federal retirement rules extremely complex. The center's workers must verify and key in information that answers a huge range of questions: What were the retiree's three years of highest salary? Was the retiree a firefighter? A military veteran? A cafeteria worker at the U.S. Capitol? What about part-time service?
All those answers can change the final pension payment. "One hundred years of bad laws," McCandless said.
The nightmare cases are the "re-employed annuitants." A government worker retires. Then un-retires. Then gets another job with the government. Then retires again.
The law allows that. But it is a heck of a mess to deal with.
"I'm working on one, and it's going on three weeks," said an employee sitting near McCandless.
When all the data are entered into the computer, it is onto Step 5. Another employee reviews the case to be sure the data were entered correctly. Then, at last, the case is "triggered." The retiree gets the full check.
That process now takes, on average, at least 61 days. That's the same amount of time it took in 1977, according to a federal audit from that time. Many state retirement systems, which also handle large loads of employees, do it much faster. Florida takes 47 days. The California teachers' retirement system takes 23. Texas takes two.
Those three process their files digitally, not on paper. Since the 1980s, the U.S. government has been trying — and failing — to do the same thing here.
The first time, work began in 1987. Years passed. About $25 million was spent, according to the Government Accountability Office. But within the government, officials started to worry that it wasn't working.
"The reports [from the contractor] just asserted that they had written X lines of code. . . . For an executive, that's just invisible; you don't know what it means," said Curtis Smith, who oversaw retirement processing from 1989 to 1994. He was a longtime federal employee with a doctorate in English literature, supervising a massive technology project.
"I had no idea [if] they were making progress from month to month. And I just sort of took it on faith that they could make it work," Smith said. "And they never did."
In 1996, two years after Smith left the government, officials finally pulled the plug on that project. Then, in 1997, the government tried again.
First it tried revamping the system in-house. Then it scrapped that plan and hired contractors. After years of work, the system the contractors built was supposed to be ready by early 2008.
But by 2007, there were concrete warnings that it again wasn't going to work.
"Every time we would do what I would call a stress test, we would come up with abysmal numbers — like an 18 percent success rate," said Robert Danbeck, who was overseeing the project. The root of the problem, he said, was that the system had trouble synthesizing information from so many sources and calculations based on so many laws. "We would go back and look at what caused it, and it was always just so many pieces, trying to tie things together."
Danbeck quit. In early 2008, the system went live.
Then it broke and was eventually scrapped, after more than $106 million had been spent. In the mine, the files continued to move on paper.
Contained in all those failures, experts say, is a very brief history of the federal government's recent troubles with information technology.
A recent study by the Standish Group, a firm in Boston that researches failures, found that only 5 percent of large federal IT projects in the last decade fully succeeded.
Of the rest, 41 percent were failures, canceled before they were turned on. The reasons often echoed the problems in the mine: Federal officials either tried to buy a technology they didn't fully understand because they lacked the technical skill, or they didn't test what they were getting until it was too late.
At the low point, in the first years of Obama's presidency, the processing time dragged out to 156 days. In response, officials did not try to eliminate the glitch. Instead, they hired more people to wrestle with it and rearranged the old process so that the paperwork moved more quickly.
Jonathan May, a recent retiree from the Department of Justice, was pleasantly surprised that his case only took three months to process. He'd expected far worse.
"I was actually bracing for it. I had saved up all my annual leave . . . went out with about 440 hours [of stored-up leave], just in case I had to live off of that for a while," May, who lives on Long Island, said in a telephone interview. "I was just amazed at how smoothly everything went."
Inside the mine, officials said they were gradually increasing the number of records that are stored digitally. Eventually, they said, the entire operation would run on computers. They had faith in the government's ability to eliminate this breaking point.
"There's a rover on Mars, mister," one OPM official said.
In the meantime, the workers who make this old-fashioned system work get a special — if unofficial — benefit.
When they retire, they don't have to wait on it.
"OPM employees get special treatment," said DeCaria, the recently retired mine worker. The agency said this is not its official policy. But when DeCaria retired, a colleague spotted the file and moved it to the front of the line.
- - -
Washington Post researcher Alice Crites contributed to this report.