How Operational Intelligence Is Saving One County Time and Money on Troubleshooting
An IT operations use case has spawned a security use case as well in Hennepin County, and they’re not alone.
Hennepin County, Minnesota, which includes the city of Minneapolis, requires sheriff’s deputies and social workers providing direct services to be retrained and recertified annually. When large video training files started going missing, it was taking days to recreate them.
The county’s Human Services Department alone has 3,500 employees, and deputies involved in drug seizure cases need to follow proper procedures. Needless to say, the deleted content was mission critical.
After the third instance, a large meeting was called including management, tech staff and Linux staff—with app administrators assuming system administrators were to blame. The app admins were adamant they hadn’t touched anything, but using Splunk operational intelligence software, IT Technical Services Supervisor Jeff Greene was able to show amid their denials that two of them had, 3 minutes apart, run commands coinciding with the times the files went missing.
In pointing their script, intended to clean the development environment, at production, app admins were unintentionally wiping out the files—necessitating a fix.
“They really thought they were doing the right thing cleaning up after themselves,” Greene told Route Fifty by phone. “But, really, they were shooting themselves in the foot.”
Between the three deletions and the emergency meeting, Greene estimated the debacle had cost the county $45,000. By comparison, San Francisco-based Splunk’s machine data analysis platform costed the county $25,000 to purchase.
Greene’s background is in software engineering, so he’s used to pouring over log files to troubleshoot problems. But correlating data between 220-250 Linux-based servers, trying to find information on any one of them, is a “huge challenge,” he said.
An initial, small-volume Splunk purchase quickly provided Hennepin County with a “complete picture” of who was on what server environment, what commands they were running and when using log files, Greene said.
The operations use case of looking at how servers are employed has since matured into a security use case.
“It’s a new paradigm,” said Kevin Davis, Splunk vice president of the U.S. public sector, in a sit-down interview. “The key is to start with a single use case, find success quickly and go from there.”
A total of 43 out of 50 states use Splunk in some capacity, including 50 percent of the 25 largest cities and 65 percent of the 25 largest counties.
Security and IT operations, specifically log management, are the two biggest use cases, but the Internet of Things is driving more jurisdictions to analyze the computer-generated data from sensors.
Hennepin County now has automated awareness of virus, ransomware and other malware activity—flagging large numbers of account lockouts. IT personnel also monitor the health and performance of exchange environments like email and briefs the county board on its security posture, as well as where attacks are coming from, analyzing risk and preventing recurrence.
Within five minutes, an infected machine can be removed from the county’s network and sent off for sanitation.
“The fact we haven’t been in the news media, the fact county administrators are not asking questions … it shows value,” Greene said.
The city of Los Angeles, another Splunk customer, uses the platform for 24/7 network monitoring for anomalies.
At a time when cities and counties are clamoring for more cybersecurity personnel, the software allows highly skilled employees to focus on their day jobs.
“It allows them, rather than being reactive, to ensure proactive network safeguards are in place,” Davis said.
Next up for Hennepin County is the use of predictive analytics of facilities data to forecast how many buildings are needed to house employees, how to distribute them, home many network jacks are needed and what occupancy might look like at different times of day. The county hopes to save money avoiding building to much facility or too many retrofits
“I’m not sure how well that will work,” Greene said. “But it’s an intriguing effort.”
Dave Nyczepir is a News Editor at Government Executive’s Route Fifty and is based in Washington, D.C.
NEXT STORY: Coalition of Strange Bedfellows Opposes N.Y. Constitutional Convention Proposal