The city of Lewiston, in north central Idaho, has a population of around 32,000 and an information systems budget of around $800,000 a year.
But it wasn't too small for attackers. For example, the city council meetings, streamed online, were being watched by people in Russia.
"Why are they watching this?" said Danny Santiago, the city's information systems administrator.
Then there were the phishing attempts.
"We are negotiating a $2 million contract for road work, and we had spearphishing attacks," he said. "Luckily it's a small town, and everyone knows everyone, so people called us."
The city needed a security information and event management (SIEM) system, but the price was a major obstacle.
"Most of the software that we looked at was six figures," he said.
Plus, the city would have to hire at least one new employee, which would have been a $70,000-a-year position not including benefits. Santiago and his team didn't have the time to become full-time security admins, he said.
The city began looking at options last year, and finally went with the AlienVault USM platform, which has SIEM built-in.
Before signing the contract, AlienVault conducted a proof of concept for the city where they installed a virtual machine and let it run for two weeks to collect data.
"Two weeks later, they go back on and they do a demo with real live data, and they said, you have a live attack going on right now," said Santiago. "I said, 'Are you kidding me?' 'No, brute force, here it is.'"
The attacker was in town, trying to log in with old access credentials.
"It turned out, after we got the police involved, that it was a former sysadmin who left under very bad terms," Santiago said.
The city decided to move ahead with the service, and installed the AlienVault appliance in February.
The first month was tough, as Santiago spent around six to eight hours a day fixing outstanding issues, on top of his regular responsibilities.
"I had a dashboard with the things you need to worry about, this machine may be vulnerable, this machine has a problem, and I worked my way down to the bottom, through the environment variables, and the patches," he said. "After the first month, I would spend about an hour or two a day. Now it's about 30 minutes a day."
As an unexpected side benefit, network speeds improved dramatically when orphaned software was removed. In one case, for example, an obsolete VPN client on a user machine had been trying to connect to servers that were replaced several years ago.
AlienVault also caught an attack from China, where a hacker tried to brute force his way in, with 20 login attempts per second for nine months.
[ RELATED: SIEM: 14 questions to ask before you buy ]
"AlienVault support took me through it, showed me how to find it, and how to stop it," Santiago said.
AlienVault's pricing starts at $5,050 a month, and goes up based on the number of assets monitored.
"It's incredibly cost-effective compared to everything else, even for tight budgets," he said. "And it was very easy to prove its value."
Smaller firms still need big security
Today, more than half of all companies have SIEM systems, according to 451 Research.
And last year, SIEMs were the fastest-growing segment of the security market, according to Gartner.
"It's growing at 15 to 20 percent a year," said Gartner analyst Oliver Rochford.
However, traditional SIEMs are expensive, difficult to setup, and hard to manage. Many companies also struggle with having enough trained staff to run the SIEMs.
That has historically put SIEMs out of reach of many small and midsized organizations.
That's a problem, said Vijay Basani, CEO at EiQ Networks, a security-as-a-service vendor that just released its own SIEM platform built from the ground up to work in the cloud.
"Small to medium-sized enterprises very much need a SIEM solution or some kind of security monitoring solution in place," he said. "The majority of attacks are taking place in the mid-market segment right now. As much as people love talking about reading about large companies, the majority of the action is actually taking place in the mid market."
So it's no surprise that the growth in the SIEM market is coming from the smaller vendors.
According to Gartner, the software revenue market share of the top five suppliers fell by 3 percent last year, to 38 percent of the market, after falling the previous year, as well.
The managed security services category is also becoming very important, said Gartner's Rochford.
"When we talk to companies about SIEM, we're talking about managed security service providers about 40 percent of the time," he said.
"Some of the biggest SIEM vendors are actually offering services themselves," he added. "IBM would be an example. Others are partnering with MSSPs."
Machine learning and advanced analytics help the providers become more efficient and support more customers, lowering prices overall, he said.
"We also have automation around incident response, containment and remediation," he said. "And the majority of clients have something in the cloud -- it makes things easier for MSSPs."
SIEMs moving off-premises
Some companies still prefer to run their own SIEMs, on their premises.
"No one knows your business better than you," said Joseph Blankenship, an analyst at Forrester Research. "Security requires a certain amount of business context, especially for monitoring, so you understand whether a behavior you're seeing is normal business activity or abnormal to the business."
For external vendors, however, the economies of scale kick in when many clients are using the same systems and services.
Extensive personalized support hurts that business model. Plus, the outside vendor might not have the best understanding of how individual companies work.
"They might not know what certain systems are utilized for, and the roles that certain users are playing in the organization," he said.
When setting up an on-premises SIEM, however, the up-front price of the technology is only part of the total cost. Staffing is a significant burden, especially for smaller companies.
According to the latest 451 Research survey, 44 percent of respondents said that lack of expertise was limiting their ability to make full use of their SIEMs, and 28 percent cited inadequate staffing.
There's a lot of specialized knowledge that goes into setting up and managing a SIEM, said Blackenship. And if a staffer leaves the company, that creates an immediate gap that might be difficult to fill.
Plus, there's the whole issue of having people watching for problems around the clock.
Midsized companies that buy their own SIEM systems can use service providers to help cover systems during off-hours, or to fill in for staffers who are on vacation, away for training, or while positions are unfilled, said Blackenship.
"And you can rely on the outside subject matter expertise," he added. "This is what they do for a living, they're invested in training their people, in keeping their platform up to date, and providing other services like threat intelligence and incident response."
But more and more, outside service providers are providing both the SIEM system and the associated management, monitoring and forensic services.
Of course, that still leaves the issue of what to do when the MSSP finds a problem.
A company can easily go from being overwhelmed with SIEM management issues, to being overwhelmed with alerts from their MSSP.
This is a problem that some vendors are looking to address as well.
At Masergy Communications, for example, when the SIEM generates an alert, engineers first look at logs, scans, API data, and packet data to weed out false positives. The customers are informed via an email or phone call, depending on the severity of the alert, and Masergy can also step in and interact with peripheral devices to stop attacks.
"We work that out with the customer on how we'll go ahead in certain cases and block certain types of activity," said Craig D’Abreo, vice president of security operations at Masergy Communications.
Otherwise, all remediation takes place on the customer side, with Masergy just providing the forensic information about the attack.
But in its communications, Masergy not only provides the details of the problem, but instructions for how it should be addressed as well, and sends that to either the customer's help desk or IT department.
Another company that offers SIEM services to midsized companies is Arctic Wolf Networks.
One of its customers, sunglasses manufacturer and distributor Costa Del Mar, was too small for a traditional SIEM deployment, but big enough that it needed that level of security.
"We have a large enough user base that we definitely need to have those services active in our systems," said Glenn Shapanka, information technology director at Daytona Beach, Fla.-based Costa Del Mar. "But to hire someone and have 24-7 monitoring -- it just doesn't make any sense for us."
With an IT department of seven people, counting Shapanka himself, there aren't enough people to spare. And since the company didn't want to purchase and own the SIEM system, outsourcing the whole thing was the obvious choice.
"They mailed the devices to us, told us where to plug them in, and configured them remotely," he said. "They collect the logs, analyze them, and are very responsive. Not only in detecting unusual activities and alerting us, but also giving us recommendations for the fixes."
As with Lewiston, the engagement started out with a long list of problems that needed to be fixed.
After that, Costa Del Mar received monthly reports about new vulnerabilities.
"We closed things up, patched holes, and reduced risks," Shapanka said. "Then the reports got shorter, so we reduced them to once per quarter. We get calls if there's an emergency, but we don't get a lot of them - they're smart enough to know what is a real issue and what is just noise. Now remediation is pretty painless."