Warning: United States Municipal IT Infrastructure at Risk

There is a problem that threatens to harm us all. If ignored, or not addressed properly, this problem will deeply wound us as Americans and possibly continue harming our country long into the future. The crisis is very real and the damage that could be caused is pervasive and exceedingly destructive.

What comes to mind when we think of Infrastructure?

The word infrastructure encompasses the world around us. Much of it is simply taken for granted as it goes on functioning in plain sight or quietly unseen. As a country, we made enormous investments in the initial development of our infrastructure. We are all familiar with the aboveground infrastructures that we encounter during our daily lives, such as roads, bridges, telecommunication lines, and electrical distribution lines, but less obvious are the underground systems such as water, gas and oil pipelines, and drainage systems that run beneath our feet. Not to mention things that are just far from sight like dams or power generation plants.

Most Americans are aware that we need to repair our roads and bridges. This has been in the news a lot the last few years and debated by various politicians. These parts of our infrastructure continue to age as we watch, and some have completely failed due to time and neglect. There are many examples of this, such as the 109 year old Park Avenue Bridge failure in Rhode Island last year, or when the Interstate 35W Bridge in Minnesota collapsed into the Mississippi River in 2007, killing 13 people and injuring 145. And then there are the dams, as documented by the Association of State Dam Safety, more than 70 US dams have failed since 2010. Our aging roads and bridges are often mentioned in the news, likely because we see them every day, but what about the infrastructure that we cannot see?

In addition to bridge and dam failures, there have been major gas and oil pipeline failures that have been highly visible and incalculably damaging. There have also been many water pipeline failures that, for the most part, have not gained as much notoriety. In the vein of pipeline infrastructure problems, there are many that do not just fail; these pipelines are still performing, but are deteriorated and leaking. Some have been moderate leaks of water, gas or oil, while others are more severe. While these pipelines still function, the consequences of not replacing them can be tragically fatal, as illustrated by the Flint Water Crisis.

Some of the more severe pipeline leaks have recently been in the news, such as the Natural gas leaks that have been found in the North Texas’ Barnett Shale region, where thousands of tons of methane are emitted every hour. A similar leak in Southern California gained national attention when it was first reported on Oct. 23 2015 and over 1,700 homes had to be evacuated. Millions of pounds of methane leaked into the atmosphere until the leak was finally sealed 112 days later on February 18^th 2016. And the risks for more incidents continue. According to NOAA and as mapped by ArcGIS, there are over 25,000 miles of active underwater oil and gas pipelines in the Gulf of Mexico. Some of these pipelines are decades old.

I find it doubtful that anyone would argue that we are on top of these all of these problems and that they are all under control and being addressed. This is simply not the case. These are big, physical, and easily understood problems that are widespread, overwhelming, and expensive to fix.

The unseen Information Technology Infrastructure:

Just as the major physical parts of our Nation’s infrastructure need to be repaired or replaced, this is also true for our technological infrastructure.

For many, our Nation’s Municipal Information Technology (IT) Infrastructure has been an afterthought, if considered at all. County, city, and town municipalities tend to be carefully run and are notoriously slow adopters of technology. Nonetheless, as IT solutions have proved successful and stable, they have grown to become an integral part in the operation of every state and municipality.

IT solutions help municipalities be more efficient. These solutions have allowed for more services and capabilities to be provided to citizens, and have helped municipalities scale and keep up with the ever-growing demands we place upon them. Regardless of whether or not we recognize it, we all depend on these solutions and systems today.

Municipalities, primarily counties, are responsible for a whole host of things. We may not always be aware these responsibilities or appreciate their importance and consider the consequential nature of some of these tasks. Counties handle various major things that affect us all in one way or another; one example is property taxes. They often assess the value of residential and commercial property as well as assess and collect the property taxes

Municipalities handle everything from mundane things like your local traffic tickets, animal control, and trash collection to more serious areas such as Municipal Courts and their associated rulings and case details. These same organizations are often responsible for managing voter registration, voter ballots, voter polling locations and the vote tallies/ counts. This is an especially sensitive topic, in light of the upcoming election.

Beyond all of the above, some municipalities also provide one or all of the local utilities, such as Electrical services, Water services, and Gas services. IT solutions are used throughout these services to provide for and track supply levels, customer demand, and distribution. IT is used in monitoring for problems, documenting the underground and wire-line networks, and ensuring services are not accidentally disrupted due to construction, as well as supporting the general operation of each service. And lastly, the time-consuming back office work such as customer management and billing is made simpler and more efficient thanks to the implementation of IT solutions.

For so many parts of our countries infrastructure we have enjoyed what I call security through obscurity. Things have remained secure because most people are unaware of those things and the value they may hold, represent or protect.

Municipal Information Technology.

The brief description I have provided for ‘The unseen Information Technology Infrastructure” really does not do the overall effort justice, but I think it helps provide some context and understanding of the enormity of the task. This is a huge job for any municipality and its IT department. While states have similar problems, they are better equipped to address them with their larger budgets and internal IT departments. But with only 50 states, these IT issues are easier to manage and track relative to the over 3,100 municipalities or equivalents in the US.

It should give us pause as we consider this. There are over 3,100 municipalities in the US. Over 3,100! Each with such a huge and wide-ranging set of responsibilities, each with likely multiple IT departments or divisions; All with critical responsibilities.

Municipalities are formed to serve the public interest and are politically driven by elected officials. The Chief Executive equivalent is a politician and likely not best suited to run a technology-based organization. The municipal IT staff is often composed of hardworking, conscientious people. The IT departments tend to be understaffed and underfunded to accomplish the jobs before them. Nonetheless, they succeed in spite of these facts and come up with inventive solutions to the problems at hand. They deliver the services their communities demand day-in and day-out. It’s important to understand that these staff members often choose this career track in part because they themselves have similarities to the organizations they join. They are risk-averse and seek out these more predictable job circumstances, clear career goals, steady pay, and eventual retirement. They are not the ‘startup techies’ we might imagine from the movies, they are the more predictable and pragmatic, 9-5, salt-of-the-earth type of people.

The equipment provided to the municipal IT staff may have been current when it was first installed, but life cycles for IT equipment are short – typically 3-5 years maximum. These are not like the traditional infrastructures we are accustomed to like roads or buildings. IT equipment is fragile and needs frequent care. In some cases a maintenance plan is purchased along with the equipment, these support agreements are for specific periods of time but can be renewed until the equipment reaches it’s end-of-life (as defined by the equipment provider) and is no longer supported. But the IT staff has to deal with the reality of things, and in many cases they are not funded in such a way that they can replace and re-engineer their systems. They end up running the systems well beyond their expected useful lifespan, and in some cases end up custom rigging solutions to keep systems functioning. In the end, they only replace the things they can or must over time.

Some municipalities may have these issues completely solved, with enough funding, staff, training and technological preparation needed for the responsibilities and services they have been tasked with. But even in this best-case scenario, IT security is ever evolving. Even if all of an organization’s IT equipment is configured to what is understood to be today’s standards there will likely still be flaws found by hackers. Finding and responding to these quickly will always remain a critical component to long-term IT operations.

The Breach

Recently, two breaches of voter registration databases were reported. These were at the State level; one in Illinois and the other in Arizona. The degree of damages is not clear, nor is the full extent of what may have been hacked. All that is clear is that these were both indeed breaches.

Beyond the flashy news releases of these incidents, let’s consider this further. Voter registration begins at the local level and is based on the county you live in.

The rules may be set at the state level, but the local voter registration information and vote counts roll up to the state level. They are initiated and gathered at the local level… the municipal level.

States are heavily involved but voting begins and ends at the local municipality, and the final tallies are reported to the State. The IT staff at those 3,100+ municipalities are responsible for the voting machines, the printouts used at the polling centers, and the collection and counting of the local votes.

And herein lies the unstated problem. I’d venture to say that most of those 3,100 municipalities are vulnerable to being breached. The severity may vary, but the likelihood is high that breaches have already occurred. I have some significant experience building and securing IT infrastructures from the growing and ongoing range of attacks and attempted attacks, as well as firsthand experience scanning for holes in private IT infrastructure here in the US. The frequency and breadth of these attacks has grown remarkably in just the last few years. Attempts to access and breach private networks has gone from being relatively rare to occurring several times a day and in some cases several time an hour.

This is the hidden infrastructure problem of Our Age. Just considering the depth and breadth of responsibilities our municipalities have, it is easy to quickly imagine several possible disruptive and disastrous results from a breach.

Consider the Federal breach reported last year:

July 9^th, 2015: OPM government data breach impacted 21.5 million.

Government investigators now report that the data theft from the Office of Personnel Management compromised sensitive personal information, including Social Security numbers, of roughly 21.5 million people from both inside and outside the government.

I was one of those individuals and received a letter from the OPM informing me of the breach. In the letter they suggest, I do things like a credit check to determine if others were using my information.

In an article from Dark Reading earlier this year, they reported some telling figures about Federal Agencies and Data Breaches.

Dark Reading, March 28^th 2016: Most Federal Agencies Have Suffered A Data Breach

• More than 60% of Federal agencies have suffered a data breach, and nearly 20% have experienced a breach in the last year.
• 90% of Federal government organizations feel vulnerable to data threats, and many are planning to increase security spending this year.

And then there are denial-of-service (DOS) attacks and distributed denial-of-service (DDOS) attacks. They can be used to take down servers or services as well as be used to break-in.

These types of attacks have been around from the beginning of the Internet but the breath and harm caused has grown enormously. Just this October 21^st, 2016 there was an unprecedented distributed denial-of-service attack on the US and many popular websites were temporally taking down; websites including Twitter, Netflix Github, Vox, Spotify, Airbnb, and Reddit. As reported by many news organizations including The Wall Street Journal “Cyberattack Knocks Out Access to Websites”.

Now consider this on the municipal level, and think past your ID and identity theft.

What if someone wanted to cause some harm or confusion? They could potentially do things such as make slight alterations to the property tax databases. Perhaps they could do something more disruptive, such as altering the owners’ names or street names of a small percentage (5-10%) of the properties. That would be a minor disaster for any municipality. One could decrease the tax amount for similar numbers of properties, thus reducing the taxes collected that year. Consider the consequences such an event would cause. What if a small percentage (8%) of court records were deleted, slowly, only a few at a time over several weeks? It would be nearly impossible to detect in time to stop and peoples’ lives would hang in the balance.

Now, let’s considering voting. The problem here has been hinted at in the press, but it has not been explored much or clearly stated. There are many ways to disrupt voting and though some of the voting machines may have vulnerabilities, hacking significant numbers of them is simply impractical.

Instead, let’s consider attacks on the underlying systems.

Examples: 1) What if a varying degree of voters with known party affiliation were selected from the voter databases? Imagine the names of 7-12% of voters in various districts were mixed at random. When they arrived at the polls to vote, their name would not match what was shown at the polling place. It would cause delays and frustration, and the lines would back up as they and the poll workers tried to figure the issue out. In the end, they typically would be asked to fill out a provisional ballot. This would function as a very effective voter suppression technique. Eliminating such a percentage of either party’s voters would work to ensure the other party wins. 2) Or perhaps the vote tallies could be attacked, making very slight and random changes in favor of one party over the other as the votes came into the databases. The change would not have to be large at all and could go unnoticed. Similar to what occasionally happens in developing countries, the exit polls would be slightly off from the actual count. Some would argue, but the evidence is circumstantial and any case would be difficult to prove.

These examples consider the general case and the fact that the hack went unnoticed. But as I have said, there are over 3,100 municipalities. Though they could be individually hacked, it would be difficult to orchestrate such a hack on a scale that could affect them all in such a short period of time. They could do this over a period of weeks or even months but the likelihood of being noticed increases with time. Rather than attempt this, the hackers could focus on a few battleground locations and specific counties where their efforts would make the most impact.

What happens if we discover the breach either afterward or while it was occurring? I have been on systems while an outside hacker was also on and attempting to do harm. We could see one another and each would fight for control and attempt to lock the other out. These sorts of battles are happening every day now, and few people hear about them or care to understand the circumstances involved. These are real newsworthy fights, but few private organizations would care to disclose to the World that such a battle happened on their servers, and even fewer municipalities would want to report this if it occurred to them.

So in this instance, let’s say we discovered the breach. We would likely know or be able to figure out what the attack profile looked like, and would try to figure out what other sites were also breached. But we may never really know the full extent of the breach for sure. And regardless of what was understood on the technological level, the public trust may be lost. There could be some counties that were not part of the hack that would question their voter results. It would be a difficult mess to clean up, and depending on how things were handled it could be the cause for lasting, damaging effects to our nation.

Incidents like these have been cause for Civil War in many Countries. If this happened in the US, it may not lead to an internal war or civil unrest, though some are clearly seeking that, but it could be the cause of instability within our Country. Such instability would work to remove the US from the World stage and leave an open field for others. Depending on the severity, it could cause titanic negative effects to the stock markets worldwide. People and other countries invest in the US because of it’s stability; take that stability away and global financial chaos would result. Our Nation leads the World. In many countries they say that if the US gets a cold, they get pneumonia. If such instability happened within the US, the rest of the World would get much more than pneumonia.

What can we do about this?

Our government is not completely unaware of these vulnerabilities, but technical and complex problems are unpopular topics that rarely get addressed by politicians. So predictability, as with most issues these days, dealing these IT vulnerabilities has become politicized.

In an effort to alleviate these issues, Congress has passed two Acts. The Election Integrity Act of 2015 that defines and limits the electronic voting equipment states acquire. And the Election Infrastructure and Security Promotion Act of 2016 that defines voting technology as a critical infrastructure similar to power or water.

A good article that discusses this can be found in Wired Magazine, September 21^st, 2016. Titled: Officials Are Scrambling to Protect the Election From Hackers This is a Federal effort lead by the Department of Homeland Security. Along with shoring up obvious holes perhaps they may deploy more advanced network monitoring tools such as one they refer to as Einstein.

Unfortunately, this effort has become derailed by politics with some state officials declaring that it is not help, but rather a Federal power grab. As discussed by Eric Geller in his Politico article on August 28th, 2016 titled: Elections security: Federal help or power grab?

While the two new Acts are a start they do little to solve the problem. The fact remains that all of these efforts begin at the state level as they likely should, but in reality the state is far removed from where the problems actually exist. As a result, the issues continue widely unaddressed.

It’s pretty clear that this, largely unacknowledged, IT Infrastructure Risk is one of the most volatile issues facing our Nation and its continued stability. I say this due to the very low level of effort required to cause a very widespread high degree of harm. The ability to continue to enjoy security through obscurity is quickly evaporating. As the interest and abilities in technology hacking continues to increase, this issue is only exasperated. Until something is done to solve these problems and a new approach is taken, our future will remain at risk. The longer we delay, the more we guarantee highly damaging breaches.

My suggestion would run along the lines of creating National standards for the services municipalities deliver. And provide strong financial and technical support to help ensure they meet the standards. There should also be clear and enforceable repercussions should they fail to meet standards; these need to ensure, regardless of the circumstances, that the work gets done properly and the citizens are not left at risk.

Perhaps this could be something similar to the Internet Engineering Task Force (IETF) but for municipal IT combined with how we build and maintain freeways using federal funding as part of the financial resources to do the work. This Municipal Internet Task Force could be organized and lead by municipalities and help all municipalities Nationally share their experiences, knowledge base and establish best practices. Municipal IT should remain locally run and outsourcing its functions would be a big mistake. The local IT personnel are committed to their individual communities and are best suited to serve them.

Summary

I felt called to write this paper and alert people to this problem. Given my unique knowledge and understanding of the multifaceted elements involved, I see this as simply my civic responsibility. I tried to put things in terms most Americans will understand and worked to leave out all of the relevant but confusing technical jargon. In doing so I am also reminded of the quote “All tyranny needs to gain a foothold is for people of good conscience to remain silent. – Edmund Burke”.

This paper is meant to alert the public and not to suggest a precise course of action. We just need to get behind our municipalities and help ensure their success. Let’s get a conversation started. The best thing that you can do is to reach out to your local municipal leaders and express your concerns. They work for you, they provide the services you depend upon and pay for. It is your money they are working with and our country hangs in the balance.