{"id":437089,"date":"2017-10-02T09:17:34","date_gmt":"2017-10-02T15:17:34","guid":{"rendered":"http:\/\/nmpolitics.net\/index\/?p=437089"},"modified":"2017-10-02T09:17:34","modified_gmt":"2017-10-02T15:17:34","slug":"its-time-for-new-mexico-to-get-serious-about-open-data","status":"publish","type":"post","link":"https:\/\/nmpolitics.net\/index\/2017\/10\/its-time-for-new-mexico-to-get-serious-about-open-data\/","title":{"rendered":"It\u2019s time for New Mexico to get serious about open data"},"content":{"rendered":"<p><strong>COMMENTARY:<\/strong> New Mexico law guarantees your right to review government data. But when it comes to giving you meaningful access to the numbers that shape and reflect public policy, the legal guarantee ends.<\/p>\n<p>While state law requires public bodies to give you most types of public data upon request, it leaves \u201chow\u201d that access will occur largely up to the agency from which you request it. The regulations that control how the law is applied create a series of uneven and unpredictable roadblocks that complicate \u2014 or even prevent \u2014 meaningful analysis of public data. Searchlight New Mexico and other media organizations repeatedly run into these when we try to open public records to the public.<\/p>\n<div id=\"attachment_437096\"  class=\"wp-caption module image alignleft\" style=\"max-width: 336px;\"><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-437096\" src=\"http:\/\/nmpolitics.net\/index\/wp-content\/uploads\/2017\/10\/Roby-John-R-336x276.jpg\" alt=\"John R. Roby\" width=\"336\" height=\"276\" srcset=\"https:\/\/nmpolitics.net\/index\/wp-content\/uploads\/2017\/10\/Roby-John-R-336x276.jpg 336w, https:\/\/nmpolitics.net\/index\/wp-content\/uploads\/2017\/10\/Roby-John-R.jpg 400w\" sizes=\"auto, (max-width: 336px) 100vw, 336px\" \/><p class=\"wp-media-credit\">Courtesy photo<\/p><p class=\"wp-caption-text\">John R. Roby<\/p><\/div>\n<p>This is about numbers, but don\u2019t turn away because of that. Your government\u2019s numbers determine your taxes and influence your children\u2019s education and your quality of life. This is about what your elected officials pay themselves, how much they spend on contracts and to which vendors, and who\u2019s giving money to elect them. On a more basic level, this is about bus schedules and arrest rates and traffic lights and which books are most popular at your local library.<\/p>\n<aside class=\"scaip scaip-1\"><\/aside>\n<p>All these details exist in public records as numbers. Because they\u2019re numbers and because they\u2019re public, they should be easy for people to use for analysis, visualization and product creation. When you can do those things, the numbers are called \u201copen data.\u201d<\/p>\n<p>Advocates of open data and government transparency tell us open data is valuable, even if it is not financial data. Open data can be put in terms of the money that is saved by solutions to big problems and smaller quandaries. For every public problem, large or small, there is a pool of amateurs and professionals eager to put open data to work. But without a commitment to open data at all levels of government \u2014 one that\u2019s backed by both political will and funding support \u2014 New Mexico is wasting money and leaving available talent on the table.<\/p>\n<p>\u201cThe people\u2019s data is worthless sitting in a hard drive, unused,\u201d said Peter St. Cyr, executive director of the New Mexico Foundation for Open Government, a nonprofit that advises the public and private sectors on state open records issues. \u201cWhen they allow the journalists and scientists and engineers to come along and have access to that data, to develop apps with that data, we start applying value to data, and that is beneficial to the community.\u201d<\/p>\n<aside class=\"scaip scaip-2\"><\/aside>\n<h3>About open data<\/h3>\n<p>Journalists aren\u2019t the only ones who\u00a0<a href=\"http:\/\/searchlightnm.com\/2017\/09\/29\/new-mexico-has-a-data-problem-were-fixing-it\/\" target=\"_blank\" rel=\"noopener noreferrer\">chase open data<\/a>\u00a0\u2014 data that are free to access, free from restrictions on use, and reside in a machine-readable, non-proprietary electronic format. Throughout the world, open data has allowed reporters, entrepreneurs and citizen-coders to:<\/p>\n<ul>\n<li><a href=\"http:\/\/coaxs.mit.edu\/job-map\/\" target=\"_blank\" rel=\"noopener noreferrer\">Plot job maps<\/a>\u00a0for the cities of Los Angeles and Boston<\/li>\n<li>Flag oversights in the City of\u00a0<a href=\"http:\/\/interactive.investigativepost.org\/lead-buffalo-2016\/\" target=\"_blank\" rel=\"noopener noreferrer\">Buffalo\u2019s lead testing program<\/a><\/li>\n<li>Visualize ridership data for\u00a0<a href=\"https:\/\/cloud.google.com\/blog\/big-data\/2016\/12\/analyzing-nyc-biking-data-with-google-bigquery\" target=\"_blank\" rel=\"noopener noreferrer\">New York City\u2019s bike-sharing program<\/a><\/li>\n<li>Develop real-time\u00a0<a href=\"https:\/\/www.microsoft.com\/en-us\/store\/p\/onebusaway\/9nblggh0cbd9\" target=\"_blank\" rel=\"noopener noreferrer\">arrival apps for transit systems<\/a>\u00a0in Seattle, Atlanta and Tampa<\/li>\n<\/ul>\n<p>Open data promotes accountability, according to Stephen Larrick, Open Cities director for the Washington, D.C-based Sunlight Foundation.<\/p>\n<p>\u201cLet\u2019s say a government is using data to analyze where it should put a new cycling lane,&#8221; he said. &#8220;That data could take into account information that people who bicycle can provide \u2014 maybe most of the cycling community prefers the next road over for certain reasons. That kind of context and ground-truthing is helpful and is only possible when the data is open.\u201d<\/p>\n<p>Open data does not just happen, but requires planning, policymaking and public input. The Sunlight Foundation has developed a scalable approach to\u00a0<a href=\"http:\/\/whatworkscities.sunlightfoundation.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">building open data policies<\/a>\u00a0that three dozen cities have adopted.<\/p>\n<p>\u201cWe think of data as just a proxy for, are we making decisions based on fact? But you want to make sure your facts are fact-checked,\u201d Larrick said.<\/p>\n<aside class=\"module align-left half type-aside\">\n<h3>About this article<\/h3>\n<p>This article\u00a0comes from\u00a0<a href=\"http:\/\/searchlightnm.com\/2017\/09\/29\/time-nm-serious-open-data\/\" target=\"_blank\" rel=\"noopener noreferrer\">Searchlight New Mexico<\/a>, a nonprofit, non-partisan news organization dedicated to investigative journalism in the interest of New Mexico residents. Sign up for\u00a0<a href=\"http:\/\/searchlightnm.com\/sign-up\/\" target=\"_blank\" rel=\"noopener noreferrer\">their newsletter<\/a>.<\/p>\n<\/aside>\n<p>When municipalities begin to open data \u2014 to treat government records, particularly numerical data, as open by default and made publicly available as a matter of course\u00a0\u2014 they begin to find efficiencies and cost-savings.<\/p>\n<p>In fact, said Tom Johnson, coordinator of the Santa Fe-based volunteer organization It\u2019s The People\u2019s Data and a longtime open data advocate, governments themselves tend to benefit from an open data philosophy long before the general public does. As different city departments \u2014 that previously had to acquire record through a laborious interagency process \u2014 can directly access these records, efficiencies abound. When more information is public, Johnson noted, people tend to feel like they&#8217;re getting a deal on their government.<\/p>\n<p>But the direct economic benefits are even greater.<\/p>\n<p>\u201cAll business decisions begin with taking data in and putting information out, and if you can speed that process and get greater utility from public data, it bodes well for economic growth,\u201d Johnson said.<\/p>\n<p>It is vital that in an open data system, numbers exist in a format that can be read and operated upon by a computer. That\u2019s what allows public data to be turned into apps, or built into charts to visualize patterns, or mathematically modeled across time and space to evaluate a government program or policy.<\/p>\n<p>Data that can be read by computers is called, reasonably enough, \u201cmachine-readable data.\u201d And it\u2019s in machine-readable data that New Mexico, by and large, has room to improve.<\/p>\n<h3>The tyranny of the PDF<\/h3>\n<p>Why machine-readable data matters is best shown by example.<\/p>\n<p>Imagine you have a table of numbers, say, the population of three counties in the 2011 and 2016 census estimates. It has three columns, showing the county name, the population and the census year, and six rows, one for each county in each year. Like this:<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-437094\" src=\"http:\/\/nmpolitics.net\/index\/wp-content\/uploads\/2017\/10\/SearchlightChart-771x293.jpg\" alt=\"\" width=\"771\" height=\"293\" srcset=\"https:\/\/nmpolitics.net\/index\/wp-content\/uploads\/2017\/10\/SearchlightChart-771x293.jpg 771w, https:\/\/nmpolitics.net\/index\/wp-content\/uploads\/2017\/10\/SearchlightChart-336x128.jpg 336w, https:\/\/nmpolitics.net\/index\/wp-content\/uploads\/2017\/10\/SearchlightChart-768x292.jpg 768w, https:\/\/nmpolitics.net\/index\/wp-content\/uploads\/2017\/10\/SearchlightChart-780x297.jpg 780w, https:\/\/nmpolitics.net\/index\/wp-content\/uploads\/2017\/10\/SearchlightChart.jpg 781w\" sizes=\"auto, (max-width: 771px) 100vw, 771px\" \/><\/p>\n<p>You might want to calculate how each county\u2019s population grew or shrank in that intervening decade.<\/p>\n<p>If the numbers that go into the table are saved in a file in a machine-readable data format, you can simply open the file in a spreadsheet program like Excel or read it into software like\u00a0<a href=\"https:\/\/www.rstudio.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">RStudio<\/a>, and perform your analysis via mouse clicks (using Excel) or by writing a small program (using\u00a0<a href=\"https:\/\/www.r-project.org\/\" target=\"_blank\" rel=\"noopener noreferrer\">the R language<\/a>). You can then make a plot of the population changes, post it online and write about what it all means. In the true spirit of open data, you might also post the original data file \u2014 machine-readable, of course \u2014 and invite others to analyze it themselves.<\/p>\n<p>But if the numbers behind that table have been saved as a PDF, you can\u2019t do any of that. You can\u2019t edit the text if a county\u2019s name is misspelled, you can\u2019t copy the population numbers into another part of the table, you can\u2019t add new rows when the next census estimate comes out, and you certainly can\u2019t perform any calculations on the data. A PDF is literally just a picture, and its one job is to lock data into place so nobody can ever do anything but view the same thing over and over and over again.<\/p>\n<p>Best practices for open data, according to Larrick, require governments to get input from the people who would use it. That includes groups inside government itself, as well as the private sector and interested citizens and media. That means understanding where people are in terms of what data and what formats, they can use and prefer. According to Larrick, there is a basic truth that PDFs are hard to search and hard to use.<\/p>\n<p>&#8220;In an ideal world everything would be open-format and have an API and be viewable and downloadable and with tools that allow for interaction,&#8221; Larrick said. &#8220;Those kinds of things should continue to be the North Star for any open data application. But as you make those incremental steps toward open data you can start with the things most important in the here-and-now, and that requires knowing what\u2019s important to the community.\u201d<\/p>\n<p>In New Mexico, journalists, entrepreneurs or the regular public find that gathering public numerical data often means the path is often buried under a landslide of PDFs.<\/p>\n<h3>Uneven data landscape<\/h3>\n<p>The state\u2019s counties and municipalities routinely responded to Searchlight New Mexico\u2019s recent requests for payroll data with PDFs, despite our requests for data in\u00a0<a href=\"http:\/\/searchlightnm.com\/2017\/09\/29\/holes-in-open-records-law-block-your-right-to-know\/\" target=\"_blank\" rel=\"noopener noreferrer\">an open, machine-readable format<\/a>.<\/p>\n<p>The Sunlight Foundation has given New Mexico\u2019s legislature\u00a0<a href=\"https:\/\/openstates.org\/reportcard\/\" target=\"_blank\" rel=\"noopener noreferrer\">a C grade on open data<\/a>, taking off points in particular because data were available only in PDF.<!--more--><\/p>\n<p>The state&#8217;s open records law states government agencies must provide electronic records &#8220;in the file format in which it exists at the time of the request.&#8221;\u00a0Data cannot easily be entered into a PDF document. So when an agency tells you its data exist only in PDF, at some point that agency made a conscious decision to lock its previously machine-readable data away into a format that you cannot enlist for analysis, visualization or development.<\/p>\n<p>Some government official decided to deliberately deny the public meaningful access to data to which they are entitled.<\/p>\n<p>It doesn\u2019t have to be that way, and it isn\u2019t always that way.<\/p>\n<p>The City of Rio Rancho, for example, maintains a data portal where it regularly posts\u00a0<a href=\"https:\/\/rrnm.gov\/1642\/RR-360-Government-Transparency\" target=\"_blank\" rel=\"noopener noreferrer\">contracts, salary and other data<\/a>, some of it in both display and machine-readable formats. Albuquerque and Bernalillo County also have made steps toward open data, with the latter enlisting\u00a0the OpenGov company to build an app that approximates how property tax dollars are divided among\u00a0<a href=\"http:\/\/www.bernco.gov\/finance\/tax-dollars-at-work.aspx\" target=\"_blank\" rel=\"noopener noreferrer\">various government programs<\/a>.<\/p>\n<p>Advocates of open data like St. Cyr suggest that when Bernalillo County moved toward open data, the greater transparency provides an antidote to claims of government waste. The app showing how property taxes are apportioned, he said, illustrates the connection between spending and long-term development goals.<\/p>\n<p>\u201cOnce the community realizes this sort of thing is available, they\u2019ll be more trustful of government\u2019s financial reporting,\u201d St. Cyr said. \u201cIf people can see how money that\u2019s coming in is applied to a community program, then it makes more sense and the spending doesn\u2019t seem out of control.\u201d<\/p>\n<p>Yet some\u00a0<a href=\"http:\/\/searchlightnm.com\/2017\/09\/29\/holes-in-open-records-law-block-your-right-to-know\/\" target=\"_blank\" rel=\"noopener noreferrer\">other government agency<\/a>\u00a0responses to\u00a0<a href=\"http:\/\/searchlightnm.com\/2017\/09\/29\/new-mexico-has-a-data-problem-were-fixing-it\/\" target=\"_blank\" rel=\"noopener noreferrer\">Searchlight New Mexico&#8217;s ongoing open records data requests<\/a>\u00a0have trended toward noncompliance.\u00a0Part of the problem, advocates for government transparency say, lies in New Mexico\u2019s open records law itself.<\/p>\n<h3>&#8216;An essential function&#8217;<\/h3>\n<p>The state\u2019s Inspection of Public Records Act (IPRA) lays out your right to inspect and obtain copies of public records. You can\u00a0<a href=\"http:\/\/nmfog.org\/read-states-sunshine-laws\/\" target=\"_blank\" rel=\"noopener noreferrer\">read the statute here<\/a>, and you can also download the state\u00a0<a href=\"http:\/\/www.nmag.gov\/uploads\/files\/Publications\/ComplianceGuides\/Inspection%20of%20Public%20Records%20Compliance%20Guide%202015.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">attorney general\u2019s compliance guide<\/a>\u00a0for public officials and citizens. You should do both.<\/p>\n<p>IPRA puts your right to public records in broad terms: \u201call persons are entitled to the greatest possible information regarding the affairs of government and the official acts of public officers and employees.\u201d<\/p>\n<p>What counts as a \u201cpublic record?\u201d Again, it\u2019s broad. The statute defines public records as all materials of a government body or an agency created by a government body, \u201cregardless of physical form or characteristics, that are used, created, received, maintained or held by or on behalf of any public body and relate to public business.\u201d<\/p>\n<p>Naturally, what started broad must in some instances be narrowed. IPRA defines nine instances in which government may restrict your right to access public records, such as when the records contain trade secrets, detail law enforcement tactical response plans or privacy matters such as letters of reference concerning employment of public officials.<\/p>\n<p>IPRA also lays out timetables for how public agencies must respond to an open records request, and includes a remarkable statement declaring it the public policy of New Mexico \u201cthat to provide persons with [public] information is an essential function of a representative government and\u00a0<i>an integral part of the routine duties of public officers and employees<\/i>\u201d (emphasis added).<\/p>\n<p>So, while the state&#8217;s open records law guarantees access to public records and embeds that access within the daily work responsibilities of public employees, IPRA is regrettably silent on the importance of open data. By not dictating the appropriate recordkeeping standards, IPRA fails to encourage governments to reach out to the public on questions of use and access.<\/p>\n<p>In this failure, IPRA neglects the well-tested principal that an enthusiastic embrace of open data saves public money, builds confidence in government and attracts new investment. Even more, advocates say, a properly focused IPRA would work to bolster civic engagement.<\/p>\n<p>\u201cProviding public records is as essential a public service as police response and fire aid,\u201d St. Cyr said. \u201cIn a self-governing democracy it\u2019s an essential function, not a burden.\u201d<\/p>\n<p><em>John R. Roby is database editor for <a href=\"http:\/\/searchlightnm.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">Searchlight New Mexico<\/a>.\u00a0Agree with his opinion? Disagree? We welcome your views. Learn about submitting your own commentary\u00a0<a href=\"http:\/\/www.nmpolitics.net\/index\/commentary-submissions\/\" target=\"_blank\" rel=\"noopener noreferrer\">here<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>New Mexico law guarantees your right to review government data. But when it comes to giving you meaningful access to the numbers that shape and reflect public policy, the legal guarantee ends.<\/p>\n","protected":false},"author":1,"featured_media":437096,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1192,16],"tags":[107,706],"class_list":["post-437089","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-commentary","category-guest-columns","tag-roundhouse","tag-transparency"],"_links":{"self":[{"href":"https:\/\/nmpolitics.net\/index\/wp-json\/wp\/v2\/posts\/437089","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nmpolitics.net\/index\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nmpolitics.net\/index\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nmpolitics.net\/index\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nmpolitics.net\/index\/wp-json\/wp\/v2\/comments?post=437089"}],"version-history":[{"count":0,"href":"https:\/\/nmpolitics.net\/index\/wp-json\/wp\/v2\/posts\/437089\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nmpolitics.net\/index\/wp-json\/wp\/v2\/media\/437096"}],"wp:attachment":[{"href":"https:\/\/nmpolitics.net\/index\/wp-json\/wp\/v2\/media?parent=437089"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nmpolitics.net\/index\/wp-json\/wp\/v2\/categories?post=437089"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nmpolitics.net\/index\/wp-json\/wp\/v2\/tags?post=437089"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}