Transparency troubles at Recovery.gov
When the American Recovery and Reconstruction Act passed back in February, its backers promised an unparalleled level of disclosure about where the money is going. While I applaud the sentiment, the release of stimulus data in the last month has been anything but smooth.
It’s a classic case of over-promising and under-delivering.
First, it was the job numbers. Now comes the phantom Congressional districts.
Since much of the direct stimulus aid goes to state capitals, any attempt to analyze stimulus spending by Congressional district alone would skew the figures. In the chart below, about half of the stimulus awards in Oklahoma — $1.3 billion — is going to the 5th Congressional District, which includes Oklahoma City.
Source: Recovery.gov
After our story came out today, one reader e-mailed and said recipients could have entered their state House or Senate districts by mistake instead of the Congressional districts. For me, more troubling than the so-called “fake” Congressional districts is the fact that a good chunk of the data don’t include any Congressional district at all (the “Null” field above).
Still, in the wake of all the negative press about the Congressional districts, the officials behind Recovery.gov said they have updated the data on the site that erroneously placed stimulus awards in nonexistent districts.
If a recipient reported an incorrect or invalid congressional district, the code “ZZ” appears in the “Congressional District” field as a placeholder. The recipient will change the report with the correct congressional district during the next reporting period, beginning January 1, 2010.
Also, the federal watchdogs in charge of stimulus spending were grilled on Capitol Hill this morning. In a related report, the Government Accountability Office had some interesting things to say about how stimulus transparency has fared so far. What I found interesting is its investigators started doing error analysis on the same day the data was released to the general public.
GAO performed an initial set of basic analyses on the final recipient report data that first became available at www.recovery.gov on October 30, 2009; reviewed documents; interviewed relevant state and federal officials; and conducted fieldwork in selected states, focusing on a sample of highway and education projects.
While much of the blame for erroneous or missing data is being placed at the recipients who filled out the Web forms on FederalReporting.gov, many are asking if there shouldn’t be some type of validation for the data before it’s released to the public.
The guidance released back in the summer by the Office of Management and Budget lays out who is responsible for data quality in stimulus reporting.
Data quality is an important responsibility of key stakeholders identified in the Recovery Act. Prime recipients, as owners of the data submitted, have the principal responsibility for the quality of the information submitted. Sub-recipients delegated to report on behalf of prime recipients share in this responsibility. Agencies funding Recovery Act projects and activities provide a layer of oversight that augments recipient data quality. Oversight authorities including the OMB, the Recovery Board, and Federal agency Inspectors General also have roles to play in data quality. The general public and non-governmental entities interested in “good government” can help with data quality, as well, by highlighting problems for correction.
To be fair, more than 100,000 reports on stimulus spending have been filed so far. Some data entry errors or miscategorizations were bound to happen when you have that many people filling in Web forms. Here’s what the GAO said it found:
Our review also identified a number of cases in which other anomalies suggest a need for review: discrepancies between award amounts and the amounts reported as received, implausible amounts, or misidentification of awarding agencies. While these occurred in a relatively small number of cases, they indicate the need for further data quality efforts.
Officials hope that future stimulus reports will contain fewer errors as recipients become more comfortable filling out the forms and the requirements are refined. In the meantime, Recovery.gov could make an easy fix by allowing users of the site to flag data that plainly looks wrong. It could function much like the “star ratings” systems on other sites, where users could “grade” their view of the data accuracy in a report.
For more on Recovery.gov data quality, check out the Sunlight Foundation and ProPublica. The White House also published a lengthy retort to critics earlier this week on its blog.
–Paul
UPDATE, 6:10 p.m., 11/19/09: From the prepared remarks of the Earl Devaney, chairman of the Recovery Accountability and Transparency Board:
These mistakes do not surprise me, however, and in a serendipitious way, they are not unequivocally bad. In reality, this data should serve in the long run as evidence of what transparency can achieve.
In the past, this data would have been scrubbed from top to bottom before its release, and the agencies would never have released the information until it was perfect. You — and the American public — are now seeing what agencies have seen, internally, in the past. And what we are all seeing, at least following this first reporting period, is not particularly pretty.
This raw-form, unsanitized data may cause embarrassment for some agencies and recipients, but my expectation is that any embarrassment suffered will encourage self-correcting behavior and lead to more accurate reporting in the future.
Did Cash for Clunkers help?
There’s some healthy debate out there as to whether the government’s Cash for Clunkers program really helped spur auto sales and boosted the economy last summer.
My story about the Oklahoma rebates is here, but some data I requested from the Oklahoma Tax Commission came in too late for my deadline. I wanted to know whether the Cash for Clunkers rebates had much effect on the number of new vehicles titled by the commission. Here’s the month-by-month breakdown for January to October for 2008 and 2009:
Source: Oklahoma Tax Commission
I’m wary of drawing too many conclusions, but it looks like the number of new vehicles titled in Oklahoma did get a boost in the late summer months this year. It’s clear that new vehicle sales were pretty slow in the early part of this year. Overall, more than 8,700 vehicles in Oklahoma qualified for Cash for Clunkers rebates. Those sales were spread over the three months of the program, which ended in late August and spilled over into early September.
Paul Taylor, the chief economist for the National Automobile Dealers Association, said the Cash for Clunkers program probably helped the economy in the third quarter as states received extra tax revenue and showrooms stayed busy. It may also have some spillover effects into the fourth quarter as auto manufacturers, which had slashed production in the wake of sluggish sales, ramped up production on assembly lines to replace inventory.
The Clunkers program lit up the market. Auto showrooms went from almost empty to overflowing. It’s hard to imagine how anyone who takes an objective look at the Cash for Clunkers program can reach any conclusion other than it gave a dramatic boost to retail sales and manufacturing output,” Taylor said.
The results on the environmental front are a little more mixed. Sure, some true clunkers and gas guzzlers were taken off the road and crushed in salvage yards. But many of the new vehicles bought using the rebates were trucks, so it’s done little to change consumer habits. And gas prices are down from their record highs in 2008, so that tiny economical car doesn’t look as attractive as it once did when gasoline was topping $4/gallon.
–Paul
–UPDATE: If you want to download the Oklahoma data yourself, just go here. It’s a pretty large Excel file of about 5MB.
–UPDATE: I just stumbled across this Daily Show clip from last night. Apparently, the demolition-derby constituency isn’t very happy with the whole Cash for Clunkers program.
| The Daily Show With Jon Stewart | Mon – Thurs 11p / 10c | |||
| Crash for Clunkers | ||||
|
||||
About those stimulus job numbers so far
New figures were released this morning on Recovery.org about the estimated jobs saved and/or created from stimulus spending so far. The numbers are coming from the first-round of information reported by contractors earlier this month.
In Oklahoma, the results are underwhelming to say the least. According to the site, Oklahoma companies have signed 120 stimulus-related contracts so far for $92.3 million. And the jobs created or saved? Just 202.
Nationwide, about 30,300 jobs have been created or saved so far, according to data collected so far. That’s not much considering the economy needs to be creating about 100,000 jobs each month just to keep up with population growth.
One White House economist, Jared Bernstein, said it’s still too early to say whether the stimulus is working as intended. But he pointed to “private estimates” as proof that many more jobs are being created.
“It is too soon to draw any global conclusions from this partial and preliminary data, as it reports on just $16 billion of the $339 billion in Recovery Act efforts before September 30th, but the early indications are quite positive. The direct count by Recovery Act recipients of jobs created or saved from this small percentage of the Recovery Act exceeds our projections. All signs — from private estimates to this fragmentary data — point to the conclusion that the Recovery Act did indeed create or save about 1 million jobs in its first seven months, a much needed lift in a very difficult period for our economy. We look forward to the much larger, comprehensive report due on October 30th.”
Just last month, the president’s Council of Economic Advisers put out its estimates of stimulus-related job creation in the first-quarter. Here’s the relevant table:
Source: The White House
Buried deep in the report, the council says it used three methodologies to estimate job impacts by state.
None of these three approaches does a perfect job of measuring the geographic distribution of employment effects, and each has advantages and disadvantages relative to the others. Thus, to obtain a reasonable estimate of state-level job impacts, we use a simple average of the three approaches.
Of course, simply because their populations are larger, we estimate that larger states have seen larger jobs impacts. Similarly, because their employment is more cyclically sensitive, industrial states are estimated to have had larger employment effects relative to their populations. Finally, both because of their industrial composition and because state fiscal relief and aid to those directly impacted have been larger in states hit harder by the recession, we estimate that states with higher unemployment rates at the time of passage have seen larger employment effects of the ARRA relative to their populations.
The Washington Post has a good wrap-up of the expectations created, and the reality of reporting job figures, here:
… Others say the reports being released this month will underscore the challenge of trying to quantify the jobs being created. Initial recipients of the stimulus money, and any government or company that they pass it on to, must report how they use the funds and how many jobs they create. But the reporting requirements do not apply to additional levels of contractors who receive the money.
My advice is to treat those early job numbers as estimates and best-guesses, at least until we get more information later this month and in the months to come.
To find out who’s getting stimulus contracts so far, just check out Recovery.gov. Here’s a list of the Oklahoma contracts signed as of earlier this month, either by Oklahoma companies or for work to be done in Oklahoma. (We also have a link to the state government’s stimulus site on our Right to Know page, which includes other databases of local interest.)
–Paul
Oklahoma stimulus funding update
The latest stimulus figures were released this morning by the state. Here’s a quick glance at which agency has gotten funding through Aug. 31.
Source: Oklahoma Office of State Finance
–Paul
OK ranks low in stimulus Web site survey
I’ve already pointed out a few issues with the federal recovery.gov site and our state’s stimulus tracking site in an earlier post, but now a national group has come out with a report ranking every state’s stimulus Web site.
The results are not encouraging. Oklahoma’s main stimulus site manages a score of just 20 out of 100 possible points, according to the rankings by Good Jobs First. The Oklahoma Department of Transportation’s stimulus site fares a little better, at 33 out of 100.
“Given the Recovery Act’s high profile, we expected better results, but most state ARRA [American Recovery and Reinvestment Act] sites simply do not measure up,” said Philip Mattera, research director of Good Jobs First and principal author of the report. “The challenge is not insurmountable. States such as Maryland, Colorado and Washington are doing a very good job in conveying vital information about stimulus spending and are leading the way in establishing best practices for state ARRA disclosure.”
Good Jobs First does say Oklahoma’s site includes good information on the broad allocation of stimulus funds. But it faulted Oklahoma for not including information about jobs saved and/or created and for failing to provide stimulus funds by geography.
If there is a silver lining in the report, it’s that most other states scored close to Oklahoma. The average score in the Good Jobs First report was 28. Just six states scored 50 or better for their main stimulus site: Maryland, Colorado, Washington, West Virginia, New York and Pennsylvania.
Good Jobs First had several recommendations for state stimulus Web sites:
1. Put a summary of key information about ARRA spending at the top of the home page of the site. A clear bar graph, pie chart or table showing the main spending flows goes a long way in helping the user begin to see what the Recovery Act is all about. There should be clear links to pages with more details about the various programs.
2. Provide a map or a table showing how overall ARRA spending and the amounts in key categories are being distributed geographically around the state.
3. Along with information on spending streams, report on individual projects being funded by those programs. Where possible, display the location of the projects on maps. Interactive displays that allow one to drill down for more details are better than static PDF maps. [emphasis mine]
4. For projects carried out by private contractors, be open about the contract award process and the identity of the companies that win bidding competitions. Post the bids and the details, including the full text of the contract awarded to the winner.
5. While the federal government’s Council of Economic Advisers is responsible for estimating the overall employment impacts of ARRA and the Recovery.gov website will report jobs data on some (but not all) individual projects, state ARRA sites should also make an effort to include employment data in their project reporting.
6. ARRA sites should provide readily accessible information about the ways that individuals, organizations and businesses can apply for stimulus grants and contracts.
I’m sympathetic to a point about some of the Oklahoma Web site’s shortcomings. After all, the stimulus money continues to trickle out of Washington to the states. And we’re all new at finding the quickest and most effective ways of keeping track of it.
The folks at OK.gov, who administer the state’s stimulus site on behalf of the ARRA Coordinating Council, put me in touch with the Webmaster for the stimulus site. I’ve also got a call into the governor’s office. I’ll update when I hear back from them.
UPDATE: Behind the scenes, budget officials, agency coordinators and Web programmers are working to get additional information on the state’s stimulus site by the federal deadline in October. Among the possibilities are map mashups and raw data feeds and downloads.
Meanwhile, Paul Sund, Gov. Brad Henry’s spokesman, said the state ARRA Coordinating Council will meet again, but no definitive date has been set. The council last met in March.
–Paul
State aid and stimulus money to schools
The state Department of Education has released its initial projections of how much money each school district can expect from state coffers in the upcoming school year. You can read my colleague Dawn Marks’ story here.
We’ve compiled the projections into a searchable database on our Right to Know page. You can search for your school district by either county or district name, or both. You can also download the spreadsheet and do your own analysis.
Included in the state aid this year is about $167 million in federal stimulus money that lawmakers added to the state Education Department budget to avoid cuts. Districts can expect more stimulus money from the state later in the year.
Those figures don’t include other stimulus money each district is eligible for in special education funding and what they call Title I help for math and reading programs in districts with higher proportions of low-income students. (For more on that chunk of stimulus money, read Dawn’s earlier story here.)
Looking at the figures, aid to most schools is down this fiscal year as compared to the final amounts they received in FY 2009. And financial officials in the districts expect this year’s amounts to decline as the state revenue picture becomes clearer:
Because revenue collections for the state have been lower than expected, allocations could change, said James White, assistant state superintendent for finance. “It may get worse. We may have to reduce those later,” White said. “Right now we’re telling school districts not to do anything drastic but to plan for cuts.”
Without stimulus money, the picture could have been bleaker, state officials said. It’s also important to remember that the state aid allocation is just one part of the funding for public schools. Other money comes from local property taxes and regular, non-stimulus, federal funding.
Here’s a quick look at the top 20 districts and their FY 2010 projected state aid amounts compared to last year:
Source: Oklahoma State Department of Education
–Paul
The perils of dirty data and “overpriced” ham
So, we paid a government contractor $1.1 million for 2 pounds of sliced ham? That seemed to be the story as the Drudge Report started linking items from the federal government’s Recovery.gov site this morning.
Not so fast, said the Agriculture Department, as it swatted down Drudge’s reports with a rare rebuttal.
Now, all this back-and-forth might seem a bit excessive for a few pounds of sliced ham, but it illustrates one of the perils of transparency without context. Our government spends billions each year to collect, maintain and analyze all kinds of data. But it’s collected by humans, who make mistakes.
Take the ham fiasco. Everything appears above board in the original description on Recovery.gov. But because of the “Description of Work/Service Performed” field, it looks like we paid a bunch of money for some pork. That’s not necessarily an error, but it’s definitely not clear to most readers. (I probably would have drawn the same conclusion, although I would check it out first before writing a story.)
The people who deal with government data on a regular basis know all too well the problems associated with collecting and disseminating data. In the field of computer-assisted reporting, we call it “dirty data,” and we’re on guard for it all the time. (A chunk of my time as Database Editor is spent cleaning up data we get from various local, state and federal sources.)
Here’s how the folks at the Institute for Analytic Journalism put it in 2006:
An uncountable number of public agency databases have been created in the past 30 years. More and more, public and private decision-makers draw on this collected, digital data to make decisions about everything from disciplining doctors to zoning decisions to law enforcement to deciding who gets to vote. The often-unquestioned assumption is that the data, as found, analyzed and presented by a government or quasi-government agency, is valid. Increasingly, anecdotal evidence indicates that data is riddled with serious errors. Often, if initial investigations indicate the data is too suspect — and the cost to clean the data by hand or automatically too high — then good and important analysis and investigations are put aside.
The Government Accountability Office recently put out its own report on the subject of government data. The report is mainly a guide for government auditors, but they recognized the problems of all these disparate sources of data, and the public’s appetite to put it all online.
While this guide focuses only on the reliability of data in terms of completeness and accuracy, other data quality considerations are just as important. In particular, consider validity. Validity (as used here) refers to whether the data actually represent what you think is being measured. For example, if we are interested in analyzing job performance and a field in the database is labeled “annual evaluation score,” we need to know whether that field seems like a reasonable way to gain information on a person’s job performance or whether it represents another kind of evaluation score.
In journalism, we try to follow the age-old advice of, “If your Mother says she loves you, check it out.” Maybe Drudge should do the same thing?
–Paul
Stimulus: Recovery.gov adds a few maps
Tracking federal stimulus spending can get to be a full-time job in itself, so I always like it when officials make government data easy and accessible to use.
Take the latest maps from the federal recovery.gov site:
Looks pretty nice, right? It is, until you start drilling down into some of the data. Unfortunately, bureaucrat-ese is still alive and well. Here’s the description that popped up when I selected a stimulus project underway in Kingfisher:
1 Project
Recipient Name :URS GROUP, INC.
Project Description :COTTONWOOD 15 DECOMMISSIONING DESIGN – THIS TASK ORDER IS ISSUED UNDER AN EXISTING IDIQ (COMPETITIVE). THE ORIGINAL IDIQ DID NOT OBLIGATED FUNDS AND WAS NOT ENTERED INTO FPDS BY THE ORIGINAL CONTRACTING OFFICER.
Available Funds :$138,905
Project Location : KINGFISHER, OK
I have to admit, I have no idea what any of that means, other than who got the contract and for how much.
It took two more clicks to finally get a clue about what that project entailed. Apparently, the money is going to a Denver-based firm for some type of soil conservation engineering project in Kingfisher.
This is all fairly normal, at least according to the stimulus site explanation:
The orange dots indicate the location of a project funded under the Recovery Act. Click on a dot and see the company that received Recovery money, the project, the amount of money allocated for the project, and its location. In most cases, the company and the project are in the same state, but they may not be in the same city. There are instances when a company based in one state has a project in another state. For example, a company with headquarters in the South may have a project in the Midwest or Northeast. As data is submitted, we’ll update the map.
Meanwhile, over at Oklahoma’s official recovery site, here’s the latest table of disbursements, from early June. (Click to see larger version):
In fact, much of the Oklahoma-specific data remains trapped in PDF files here, with very little in the way of organization. And according to the site, the last meeting the coordinating council had about the stimulus was back in March. Maybe our auditor, the governor and others should take a look at what’s going on at other state stimulus sites, like Maryland.
–Paul
Oklahoma launches stimulus Web site
Oklahoma is the latest state to have a dedicated Web site for federal stimulus oversight and transparency. The site went live on Friday with very little fanfare from state officials.
In all, the Oklahoma is expected to get about $2.6 billion from the massive $787 billion federal stimulus bill signed into law last month. Here’s a quick breakdown of the money from the state’s Web site:
With the launch, Oklahoma joins more than 20 other states with official stimulus/recovery Web sites.
From the site’s introduction:
In February of 2009, President Obama and the U.S. Congress approved the American Recovery and Reinvestment Act to help kick-start the nation’s economy. The landmark measure includes targeted tax relief and significant investments in such vital areas as education, transportation, healthcare, science and technology and energy-efficiency.
In an effort to ensure full transparency and accountability, Gov. Brad Henry ordered the creation of a state Web site to help track the use of Oklahoma stimulus funds. This site will provide the citizens of Oklahoma access to clear and concise information about the federal stimulus initiative.
The Web site fulfills the transparency side of the equation. Henry last week appointed state Auditor and Inspector Steve Burrage to oversee stimulus spending in the state.
As part of that oversight, Burrage will make sure agencies are following the stimulus’ rules for responsible spending. From the FAQs:
* Strict time limits for obligating funds;
* Public access to contract and grant information;
* Requirements for competitive contracting;
* Certifications by the Governor or local officials that infrastructure expenditures have been reviewed and are an appropriate use of tax dollars; and
* Reports on how the state distributes funds it receives, the estimated number of new or saved jobs, tax increases averted and other information.
You can also sign up for alerts when new information on stimulus funds is released by state agencies.
–Paul
Gov. Henry appoints auditor as stimulus watchdog
Gov. Brad Henry has appointed state Auditor and Inspector Steve Burrage as the cop who will keep track of federal stimulus money in Oklahoma.
“In light of the size and scope of the stimulus package, complete transparency is essential to ensure public confidence,” said the governor. “Oklahoma taxpayers need and deserve to know how every dollar will be spent in our state. By posting all information on the web and having the auditor oversee the process, we can bolster public trust and ensure an informed citizenry.”
Apparently, the state will use its existing Open Books site to add a section for stimulus transparency.
–Paul
