What is a NOC?
A network operations center (NOC) is the centralized location (in-office or virtual) for an organization’s network team. This team monitors networks, servers, application infrastructure, cloud usage, and more for events that could degrade or disrupt service for customers and users, which creates risk and cost for the business.
NOC duties often include:
- Providing 24/7 coverage, typically across two or three shifts
- Watching for anomalies, service disruptions, and outages
- Managing a queue of tickets from people reporting issues
- Working through runbooks to triage or resolve issues
- Notifying the team responsible for a given service when an outage occurs and the NOC cannot solve it
NOCs are often staffed with several levels of expertise. This keeps escalations to subject matter experts (SMEs) to a minimum, which controls cost while still protecting customers from incident impact.
Common NOC roles are:
- L0: “Role” is a misnomer for L0. This level should be fully automated and act as the first line of defense before a human ever sees an issue. This is possible with event-driven automation, which cuts redundant incidents from the start and auto-remediates well-understood issues.
- L1: The first human to see an event. L1 engineers work through attached, predetermined runbooks to resolve routine, documented issues.
- L2: If an L1 can’t resolve a problem with a runbook, the L2 handles further triage and troubleshooting using deeper expertise. They escalate to an SME only if they can’t resolve the issue themselves. L2s may also call a major incident if the issue meets the criteria.
- Director, VP, or CIO: NOC leadership. The director makes sure the NOC is staffed appropriately, uses resources wisely, and meets its goals and KPIs. The VP or CIO has a broader scope, but the NOC is a prime opportunity to modernize for better customer experiences, lower business risk, and lower operating costs.
Here’s why this team matters so much to a modern enterprise.
Importance of a NOC
NOCs are under pressure to monitor and maintain a growing number of business-critical services as the pace of innovation accelerates. With this added complexity and data volume, traditional NOC practices like the “command and control” approach and sequential, manual workflows no longer keep up. Under a mandate to do more with less, many organizations are evolving their NOCs to centralize and standardize incident management across tech and teams. The benefits include fewer SLA penalties, less risk and lost revenue, fewer interruptions for SMEs, and a stronger brand reputation.
To get there, legacy NOCs must address technical, people, and process challenges.
Top NOC challenges
Here are the top three challenges NOCs face:
Increase in events/data: As applications and services grow more complex, the volume of data coming into the NOC has grown significantly. It’s no longer possible to maintain eyes-on-glass methods with engineers watching screens all day to pinpoint issues. There are too many data points to synthesize, so problems slip under the radar. Customers end up finding out about disruptions before technical teams do, which costs money and hurts the brand.
Manual processes: From running diagnostics, to finding the right runbook, to routing incidents via catch-and-dispatch, the NOC is full of processes that automation can handle. Without automation, NOC engineers repeat the same routine steps for each issue, which delays the fix. Time spent on toil also makes it harder to get proactive about incident response, and more issues trickle down to SMEs as the NOC struggles with the volume of work.
Expertise: NOC personnel contact SMEs or service owners when a specific application or service has an incident. They usually aren’t deep-domain experts in the affected systems because they didn’t build them. So when an incident arises, it can be challenging and costly to manually navigate complex escalation paths to find the right SME. This manual escalation extends the time it takes to diagnose and resolve issues that are actively affecting customers.
These challenges are solvable. The following practices help modernize the NOC for real-time operations.
NOC best practices
To modernize the NOC, teams need to shift their mindset and find ways to use automation as their first line of defense and throughout the incident management process. These tactics apply at each stage of incident management:
|
Stage |
What to do |
|
Detect |
Route all your monitoring systems into a central system. Without one, the full picture of an issue’s severity is fragmented, and responders can’t tell how to categorize it. With monitoring flowing through a single source of truth, NOCs can immediately see what needs priority across the organization, no matter who owns the service. Worried about extra noise? Use automation to suppress and deduplicate redundant alerts across different tools for a clear signal. |
|
Mobilize |
The right response depends on the issue, whether that’s auto-remediation, L1/L2 NOC response, or escalation to SMEs or major incident management (MIM) teams. Create automation that categorizes issues and routes them to the right person immediately. Clear routing rules and escalation policies keep issues from falling through the cracks. |
|
Mitigate |
Distinguish between high- and low-priority issues. Services that aren’t customer-facing or have few dependencies can wait, since they carry less risk and business impact. Use auto-remediation to prevent customer-facing problems for well-understood issues. Create runbooks for routine issues that do need NOC response so fixes are as streamlined as possible. Make sure these runbooks are automatically populated and cover MIM criteria and practices (or better yet, automate those criteria) so major incident management kicks off immediately when needed. |
|
Resolve |
Let machines handle what they can, and loop humans in only when automation can’t resolve the problem. Use runbooks to keep escalations low, with set criteria for when escalation is required. Give responders the right information immediately by using automation to pull and update diagnostics, and AI to surface relevant history such as related or past incidents and how they were resolved. |
|
Document |
Integrate with the rest of the organization’s technical systems, including systems of record like Jira or ServiceNow. NOCs rely on these tools to pull data from or send data to. Connect them for every service the NOC is responsible for, and build automation to update them so no one has to do manual data entry. |
|
Learn |
Conduct postmortems to recognize what went well during response and what could have gone better. Use those lessons to create new automation that helps responders act faster and reduces toil. This feedback cycle directly affects customers and the business: fewer repeat incidents mean less risk, lower costs, and better morale. |
These practices won’t happen overnight. They improve the NOC’s ability to react, troubleshoot, and resolve incidents, but they work best when adopted deliberately and formalized in documentation.
PagerDuty helps NOCs put these practices into action, with AIOps to reduce alert noise and Runbook Automation to automate diagnostics and remediation. Start a free trial to see how it fits your NOC.