1. The man behind the city’s cyberfortress
By KATHERINE CONNOR, The Daily Transcript
Friday, February 20, 2015
How does a city with some 10,000 employees spread across 40 departments, all constantly
looking to onboard the latest and greatest tech products, software and applications, ensure that
hackers seeking valuable infrastructure or financial information are kept out of the cyberdomain?
It takes a layered approach, continuous monitoring, close communication with other public and
private entities, and one key ingredient: Gary Haslip.
Haslip is the city of San Diego’s chief information security officer who served for 30 years in the
Navy in an array of network defense and operations roles before joining the city staff nearly two
years ago.
He is a well-known face in both the cyber and startup communities — which are frequently one
and the same, especially in San Diego — and often draws on local startups to work with the city
in a sort of symbiotic beta trial.
He said San Diego’s networks are secure, but still sees room for improvement, particularly in
securing cloud-based technologies and Internet of Things devices, the two areas he said are the
most challenging — and downright scary — from a cybersecurity perspective.
But what does success mean in an environment where just one successful breach of the 300,000
attacks waged on the city each week is all it takes to do serious damage?
“I look at my infection rate-to-user ratio,” Haslip said, on measuring success. “So if you have
10,000 users, I would want to go ahead and keep my infection rate to less than 1 percent, which
is 100 machines, per month. Right now we’re averaging about 70 machines, so we’re averaging
about 0.7 percent, and to me, even that’s too much — I’d like to cut that in half.”
2. Gary Hayslip, the chief information security officer for the City of San Diego.
Reaching zero infections is impossible, largely due to human nature — someone will click on a
bad link no matter how many technical backstops are in place.
“No matter how much you train your users, I’ve got to get all 10,000 of them to be correct every
time — they’ve got to get them all to be wrong once,” Hayslip said.
“So the odds are in the bad guys’ favor. And I understand that. I really work hard, talk with the
mayor and staff to get them to understand that this is a fantasy that we will never have a breach
or have anything bad happen.”
Who are these “bad guys” launching 300,000 attacks a week on the city’s systems with
increasingly targeted phishing attempts down to the individual level?
Haslip said the vast majority are from Asia, and he said the traffic from a shellshock attack on all
the city’s systems in October — which didn’t breach the neutral zone separating the city network
from the public domain — came from China.
To counter these attacks and minimize damage, Hayslip and the city’s 100-person IT team — 10
of which work specifically on security — coupled with contractors from the city’s three prime
contractors, Xerox, Atos and CGI, have implemented a traditional layered approach starting
from the perimeter and working in, with continuous monitoring solutions, including one that has
been funded by the mayor for next fiscal year.
An area needing a face-lift from the supplier side is in report read-outs. Hayslip said reports from
such partners as Palo Alto Networks, which the city uses for some of its firewalls, can be
hundreds of pages long and data-dense.
3. “People at the executive level, they don’t really care about threat vectors; it’s all about business,”
Hayslip said. “I find as a CISO, you straddle the business side and the security side. You have to
be able to take all the security stuff you’re used to talking up with your engineers, and put it in
business talk when you talk with your executive staff and your C-suite.
"It really helps if whatever product you're using is doing an awesome job and can do executive
reports so you can really show a visual picture of 'This is the value of this product, this is why I
asked for $100,000, because it’s doing this.' ”
The other area of concern is the cloud. Hayslip said many people have the perception that
anything can be thrown into the cloud without consequence, which is not the case.
In addition to government compliance issues for citizens’ data, the cloud poses its own security
risks, and the city is moving toward a hybrid solution to manage and house data — some kept
within the city’s networks, and some on a private cloud or possibly a Microsoft Azure portal that
would allow for a public-facing side to facilitate apps for water bill payment or pothole
notification, for example.
“There are a lot of different things that the mayor’s talking about, and the IT department here,
our job is to go ahead and take a look at that and figure out the technology, the plumbing behind
it,” Hayslip said. “What kind of infrastructure do we need to put in place, or do we need to
upgrade what we have?”
One of the first places Hayslip looks for new infrastructure or upgrade solutions is his own
backyard, the local startup community.
In fact, two San Diego startups have been, or will soon be, on-boarded by the city at no cost —
the city gets to use the most innovative products and potentially discover new vulnerabilities, and
the companies get to test their products on this very large scale, and make the case for purchase
down the road.
Cyberflow Analytics, which is a machine-learning analytics software that listens to the network
and learns what is normal to then find abnormal and dangerous activity with a unique
visualization component, was brought on in December, and PacketSled, a real-time breach
detection forensics platform, plans to integrate with the city in the coming weeks.
Both Tom Caldwell, president of CyberFlow, and Matt Harrigan, president of PacketSled, met
Hayslip through CyberTech, a local cyber startup incubator and resource provider, and other
startup and community groups.
“Guys like Gary are key to the success of the city,” Harrigan said. “He’s an incredibly engaging
individual, and I think a lot of his community outreach, involvement with the local vendors, local
security experts — that’s what makes him even better at his job than he already is. He involves
people who can help the city to address what are really serious issues on an ongoing basis.”
4. Hayslip said this level of involvement makes San Diego stand out as one of three national
cyberhubs, along with the Beltway around Washington, D.C., and San Antonio.
Palo Alto Networks recently hosted a CISO dinner for officers from local public municipalities
as well as large private players like Qualcomm Inc. (Nasdaq: QCOM) and Sempra Energy
(NYSE: SRE) to get to know each other, which Hayslip found funny since they all already meet,
communicate and share information on a regular basis — something that doesn’t happen in many
other major cities.
This collaboration and openness are going to be necessary to deal with the Internet of Things, the
one area Hayslip singled out as keeping him up at night.
“What really concerns me, and a lot of the research and stuff I’m reading on it, is that security is
not included in a lot of these devices — it’s not built in from the beginning,” he said.
“And you’ve got to remember a lot of the personal data that’s now collected is location data … A
lot of people don’t think of that as being privacy data, but it is. And these type of devices, they
give a lot of that away,” particularly smart devices in the home, which Hayslip said is
constitutionally considered a safe and private space.
“These things are inside your home, and the kind of data they’re giving out and everything — I
think in a lot of ways society is going to shift. There’s going to be a lot of changes with how we
view privacy.”
Luckily, Hayslip, who has pursued all of the qualifications of his own volition simply for the
love of security, will be there pushing for, and working on, cybersecurity until the very end.
“I will probably do it all the way up until I pass away, I guess,” he said. “I’ll probably be buried
with my laptop — it’s just something I love to do.”