COMMENTARY: New Mexico law guarantees your right to review government data. But when it comes to giving you meaningful access to the numbers that shape and reflect public policy, the legal guarantee ends.
While state law requires public bodies to give you most types of public data upon request, it leaves “how” that access will occur largely up to the agency from which you request it. The regulations that control how the law is applied create a series of uneven and unpredictable roadblocks that complicate — or even prevent — meaningful analysis of public data. Searchlight New Mexico and other media organizations repeatedly run into these when we try to open public records to the public.
Courtesy photo
John R. Roby
This is about numbers, but don’t turn away because of that. Your government’s numbers determine your taxes and influence your children’s education and your quality of life. This is about what your elected officials pay themselves, how much they spend on contracts and to which vendors, and who’s giving money to elect them. On a more basic level, this is about bus schedules and arrest rates and traffic lights and which books are most popular at your local library.
All these details exist in public records as numbers. Because they’re numbers and because they’re public, they should be easy for people to use for analysis, visualization and product creation. When you can do those things, the numbers are called “open data.”
Advocates of open data and government transparency tell us open data is valuable, even if it is not financial data. Open data can be put in terms of the money that is saved by solutions to big problems and smaller quandaries. For every public problem, large or small, there is a pool of amateurs and professionals eager to put open data to work. But without a commitment to open data at all levels of government — one that’s backed by both political will and funding support — New Mexico is wasting money and leaving available talent on the table.
“The people’s data is worthless sitting in a hard drive, unused,” said Peter St. Cyr, executive director of the New Mexico Foundation for Open Government, a nonprofit that advises the public and private sectors on state open records issues. “When they allow the journalists and scientists and engineers to come along and have access to that data, to develop apps with that data, we start applying value to data, and that is beneficial to the community.”
About open data
Journalists aren’t the only ones who chase open data — data that are free to access, free from restrictions on use, and reside in a machine-readable, non-proprietary electronic format. Throughout the world, open data has allowed reporters, entrepreneurs and citizen-coders to:
- Plot job maps for the cities of Los Angeles and Boston
- Flag oversights in the City of Buffalo’s lead testing program
- Visualize ridership data for New York City’s bike-sharing program
- Develop real-time arrival apps for transit systems in Seattle, Atlanta and Tampa
Open data promotes accountability, according to Stephen Larrick, Open Cities director for the Washington, D.C-based Sunlight Foundation.
“Let’s say a government is using data to analyze where it should put a new cycling lane,” he said. “That data could take into account information that people who bicycle can provide — maybe most of the cycling community prefers the next road over for certain reasons. That kind of context and ground-truthing is helpful and is only possible when the data is open.”
Open data does not just happen, but requires planning, policymaking and public input. The Sunlight Foundation has developed a scalable approach to building open data policies that three dozen cities have adopted.
“We think of data as just a proxy for, are we making decisions based on fact? But you want to make sure your facts are fact-checked,” Larrick said.
When municipalities begin to open data — to treat government records, particularly numerical data, as open by default and made publicly available as a matter of course — they begin to find efficiencies and cost-savings.
In fact, said Tom Johnson, coordinator of the Santa Fe-based volunteer organization It’s The People’s Data and a longtime open data advocate, governments themselves tend to benefit from an open data philosophy long before the general public does. As different city departments — that previously had to acquire record through a laborious interagency process — can directly access these records, efficiencies abound. When more information is public, Johnson noted, people tend to feel like they’re getting a deal on their government.
But the direct economic benefits are even greater.
“All business decisions begin with taking data in and putting information out, and if you can speed that process and get greater utility from public data, it bodes well for economic growth,” Johnson said.
It is vital that in an open data system, numbers exist in a format that can be read and operated upon by a computer. That’s what allows public data to be turned into apps, or built into charts to visualize patterns, or mathematically modeled across time and space to evaluate a government program or policy.
Data that can be read by computers is called, reasonably enough, “machine-readable data.” And it’s in machine-readable data that New Mexico, by and large, has room to improve.
The tyranny of the PDF
Why machine-readable data matters is best shown by example.
Imagine you have a table of numbers, say, the population of three counties in the 2011 and 2016 census estimates. It has three columns, showing the county name, the population and the census year, and six rows, one for each county in each year. Like this:
You might want to calculate how each county’s population grew or shrank in that intervening decade.
If the numbers that go into the table are saved in a file in a machine-readable data format, you can simply open the file in a spreadsheet program like Excel or read it into software like RStudio, and perform your analysis via mouse clicks (using Excel) or by writing a small program (using the R language). You can then make a plot of the population changes, post it online and write about what it all means. In the true spirit of open data, you might also post the original data file — machine-readable, of course — and invite others to analyze it themselves.
But if the numbers behind that table have been saved as a PDF, you can’t do any of that. You can’t edit the text if a county’s name is misspelled, you can’t copy the population numbers into another part of the table, you can’t add new rows when the next census estimate comes out, and you certainly can’t perform any calculations on the data. A PDF is literally just a picture, and its one job is to lock data into place so nobody can ever do anything but view the same thing over and over and over again.
Best practices for open data, according to Larrick, require governments to get input from the people who would use it. That includes groups inside government itself, as well as the private sector and interested citizens and media. That means understanding where people are in terms of what data and what formats, they can use and prefer. According to Larrick, there is a basic truth that PDFs are hard to search and hard to use.
“In an ideal world everything would be open-format and have an API and be viewable and downloadable and with tools that allow for interaction,” Larrick said. “Those kinds of things should continue to be the North Star for any open data application. But as you make those incremental steps toward open data you can start with the things most important in the here-and-now, and that requires knowing what’s important to the community.”
In New Mexico, journalists, entrepreneurs or the regular public find that gathering public numerical data often means the path is often buried under a landslide of PDFs.
Uneven data landscape
The state’s counties and municipalities routinely responded to Searchlight New Mexico’s recent requests for payroll data with PDFs, despite our requests for data in an open, machine-readable format.
The Sunlight Foundation has given New Mexico’s legislature a C grade on open data, taking off points in particular because data were available only in PDF.
The state’s open records law states government agencies must provide electronic records “in the file format in which it exists at the time of the request.” Data cannot easily be entered into a PDF document. So when an agency tells you its data exist only in PDF, at some point that agency made a conscious decision to lock its previously machine-readable data away into a format that you cannot enlist for analysis, visualization or development.
Some government official decided to deliberately deny the public meaningful access to data to which they are entitled.
It doesn’t have to be that way, and it isn’t always that way.
The City of Rio Rancho, for example, maintains a data portal where it regularly posts contracts, salary and other data, some of it in both display and machine-readable formats. Albuquerque and Bernalillo County also have made steps toward open data, with the latter enlisting the OpenGov company to build an app that approximates how property tax dollars are divided among various government programs.
Advocates of open data like St. Cyr suggest that when Bernalillo County moved toward open data, the greater transparency provides an antidote to claims of government waste. The app showing how property taxes are apportioned, he said, illustrates the connection between spending and long-term development goals.
“Once the community realizes this sort of thing is available, they’ll be more trustful of government’s financial reporting,” St. Cyr said. “If people can see how money that’s coming in is applied to a community program, then it makes more sense and the spending doesn’t seem out of control.”
Yet some other government agency responses to Searchlight New Mexico’s ongoing open records data requests have trended toward noncompliance. Part of the problem, advocates for government transparency say, lies in New Mexico’s open records law itself.
‘An essential function’
The state’s Inspection of Public Records Act (IPRA) lays out your right to inspect and obtain copies of public records. You can read the statute here, and you can also download the state attorney general’s compliance guide for public officials and citizens. You should do both.
IPRA puts your right to public records in broad terms: “all persons are entitled to the greatest possible information regarding the affairs of government and the official acts of public officers and employees.”
What counts as a “public record?” Again, it’s broad. The statute defines public records as all materials of a government body or an agency created by a government body, “regardless of physical form or characteristics, that are used, created, received, maintained or held by or on behalf of any public body and relate to public business.”
Naturally, what started broad must in some instances be narrowed. IPRA defines nine instances in which government may restrict your right to access public records, such as when the records contain trade secrets, detail law enforcement tactical response plans or privacy matters such as letters of reference concerning employment of public officials.
IPRA also lays out timetables for how public agencies must respond to an open records request, and includes a remarkable statement declaring it the public policy of New Mexico “that to provide persons with [public] information is an essential function of a representative government and an integral part of the routine duties of public officers and employees” (emphasis added).
So, while the state’s open records law guarantees access to public records and embeds that access within the daily work responsibilities of public employees, IPRA is regrettably silent on the importance of open data. By not dictating the appropriate recordkeeping standards, IPRA fails to encourage governments to reach out to the public on questions of use and access.
In this failure, IPRA neglects the well-tested principal that an enthusiastic embrace of open data saves public money, builds confidence in government and attracts new investment. Even more, advocates say, a properly focused IPRA would work to bolster civic engagement.
“Providing public records is as essential a public service as police response and fire aid,” St. Cyr said. “In a self-governing democracy it’s an essential function, not a burden.”
John R. Roby is database editor for Searchlight New Mexico. Agree with his opinion? Disagree? We welcome your views. Learn about submitting your own commentary here.