In our NOC, we maintain situational awareness of all physical security zones (reactive video feeds), some basic information about the physical characteristics of the data centers, the weather and a national news feed. Are there other things you recommend a NOC monitor, or is this considered good enough.
5 Answers
This is a pretty broad question, but I'm assuming we're avoiding service- or logical-level monitoring (ie, SSH listening, web sites responding properly, disk space, CPU usage, etc.).
Your NOC should be doing both constant hands-off monitoring as well as periodic hands-on/eyes-on monitoring.
Constantly monitor:
- Ambient temperature/humidity from multiple sensors in datacenter
- Power draw for all circuits in datacenter
- AC unit's self-reported load/health
- Video feeds of datacenter interior, all datacenter entrances, and entrance to NOC area
- UPS battery status
- Log all entrance/exits from datacenter (whether by swipe card or manual sign-in)
- Switch port/router interface status events (up/down/change in speed)
Periodic walkthroughs:
- Racks closed and locked
- Any audible or visual hardware alarms (lights, tones, status LCDs)
- Floor tiles in place and in good condition
- Datacenter entrances closed and locked
- Cameras undisturbed
- No unexpected visitors in or around datacenter
- Failed lights, broken windows, damaged doors, anything that makes physical security easier to breach
Also, why bother with a national news feed? Unless you have datacenters distributed throughout the country, it makes much more sense to tune into local news stations. The value of this as a whole is questionable, though, since it's going to be a very high noise:signal ratio. If anything, just subscribe to an RSS feed from your relevant news outlets.
Finally, this may not be what you're looking for, but I've found testing outside my network to be invaluable. Testing for high latency or packet loss to several well known networks (Google, Yahoo, Microsoft, etc.) with small and large packets, checking common websites for proper return codes, and measuring round trip latency for email to/from several popular free mail sites (Yahoo, Hotmail, Gmail) has given me the jump on several subtle problems before my users started calling.
-
Some excellent points. We don't do walkthroughs but rather rely on video monitoring and local security droids to ensure physical issues are covered. We do have a DC on east/west coasts. We do some service availability monitoring, but it's limited. Jun 18, 2009 at 22:43
-
If you're providing services out of these datacenters, it is absolutely critical that you monitor everything you've got as thoroughly as you can. In fact, you should probably drop everything you're doing until that's done. If you just provide some basic infrastructure for clients to store their gear in, you probably want to monitor a couple high-profile ones so you can respond quickly if someting you do breaks them.– sh-betaJun 18, 2009 at 22:47
-
outside hosts are a must. I have shell accounts on 2 reliable servers on both of the major, local consumer ISPs and another account across the country. I think its absolutely essential to be able to test your network from the outside.– neoiceOct 17, 2009 at 13:13
The Radar available from Securitywizardry.com is always something I've wanted to put up on a big plasma in a NOC/SOC for the cool factor: http://www.securitywizardry.com/radar.htm
It includes security news, "health" levels, emerging threats, and information on tool updates.
-
2That's some great infoporn. For network security issues, we have a separate security staff that has feeds from log aggregation boxes and, I think, the kaspersky feed: kaspersky.com/viruswatch3 Jun 18, 2009 at 22:45
My thought is the walk though physical side of things.
Walk though often, if you are onsite and a 24/7 office then have 2 people walk though seperately at the start of their shift. If they walk though together they will just chat and not see what they should be seeing.
- If not on site try and setup a regular walk though by either an employee or hire a local contractor to do it.
- I would say weekly min, but daily is better.
- Make this more frequent if you have any liquid pipes in there (HVAC chiller lines, domestic water lines, sewage from a bathroom above you, etc)
- Look for the out of place
- Dripping pipes can be caught by a human eye before an electronic sensor
- Check under the floor
- Feel the HVAC vents to make sure they are on
- LISTEN to the UPS's. If they are humming louder then normal you might to check it
- Same goes for any transformers you have
- SMELL, there are a lot of electronics to start frying in there
- Keep it clean
If you have a seperate generator/utility room, check that too
- Make sure the fuel lines are not leaking
- Keep it clean
- Noise and smell
- If the generator is on an auto test, check it after the test, make sure nothing sprung a leak on it.
Its not a bad idea to have your electrical panels thermo-scanned a couple of times a year. This will find gear that is nearing failure since it will appear hotter.
Every failure starts small, if you catch it when small, you can fix it on your schedule
How about:
NOC staff RSS feeds/Tweets
How do each of you at the NOC share information? Yammer (https://www.yammer.com/) could be a good way to share enterprise-wide information/status (via email/IM/RSS).
RSS feeds of internal or external events might be useful. We use an internal blog for change controls, and pipe the output to a Mac with the RSS reader screensaver.
answered Jun 18, 2009 at 22:21
