For years, we measured success by how fast we could fix broken systems. Then we had an outage that changed everything. We realized we weren't measuring the right thing. The real win wasn't speed-it was prevention. By unifying our observability data, we can now stop problems forming before they hit our users. This is what the future of IT operations looks like. 

For a long time, Mean Time to Resolution (MTTR) has been 'the' metric that drives IT teams, often used as the ultimate measure of success for digital resilience.  

But what if I told you that in Cisco IT, MTTR is no longer the sole focus? In the past 18 months, we've dramatically reduced incidents - not only by responding faster, but also preventing problems before they happen.

While MTTR remains a critical measure, we are maturing to a more impactful standard: proactive incident prevention and avoidance. We're shifting the question from 'how fast can we fix it?' to 'how can we prevent it from happening at all?'  This moves us beyond reactive fixes and toward predictive and preventative observability. 

Watch the video and keep reading to discover more.

Real-world consequences: Fragmented data and missed correlations 

In 2024, we experienced a database outage that drove us to rethink how we use observability to improve our digital resilience.  

Although alerts were generated across multiple related devices, our data was fragmented, preventing us from correlating those signals. This correlation gap delayed our ability to identify and remediate the issue.  

After the outage, we realized that we could have prevented 30-40% of our major issues had we been able to make key correlations, or at least provided enough warning (more than 15 minutes) to take proactive action.  

This correlation challenge is especially critical for my team, responsible for the observability across applications, infrastructure, services, cloud, and data centers - as these environments are highly interconnected and dynamic.  

Like other large enterprises, our applications rely on a complex interconnection of underlying infrastructure and services, often spanning on-premises data centers and multiple cloud providers. A single issue in one area can result in trickling effects in others, making it difficult to pinpoint root causes without unified, correlated data. 

The high volume and diversity of telemetry generated across these domains adds to the complexity. Traditional monitoring tools and dashboards weren't providing the real-time, end-to-end visibility we needed to make correlations. Without centralizing and correlating data from all our sources, we wererisking missing early warning signs, responding too slowly, or misdiagnosing issues altogether. 

To address these complexities, we transformed our observability approach to centralize data and insights across our entire IT landscape.  

Cisco's advantage: The technology behind our observability transformation 

Our network intelligence (ThousandEyes, Catalyst, Meraki) gives us visibility early in the incident chain. We see the pattern in network behavior before it cascades to applications.  

We built a central nervous system for IT: one platform to see metrics, logs, and dependencies from every device, app, and service. The result: correlation. When network behavior changes, we see which apps are affected. When an app slows, we see why.  

To enable this, we rely on a strategic combination of integrated Cisco solutions, data, and AI-driven workflows: 

• Splunk Cloud Platform: Aggregates logs generated from any devices, applications, Cisco controllers, 3rd party controllers, or network devices. Its scalability allows us to use AI tools to quickly identify anomalies by prompting questions such as "Are there any anomalies in my X logs?" 
• Splunk Observability Cloud: We are centralizing metrics from ThousandEyes, AppDynamics, applications, databases, and various domain-specific controllers (ie. Meraki and Catalyst Center) into Splunk Observability Cloud. This integration enables us to correlate performance data across our infrastructure. Through Splunk Log Observer Connect, we can easily query logs from Splunk Cloud Platform, enabling us to troubleshoot application and infrastructure behavior using high-context logs.
• ThousandEyes: Provides end-to-end network monitoring and synthetic application testing, helping to ensure our applications are available and performing up to par. We capture critical metrics of our user endpoints (i.e. laptops) and feed all ThousandEyes logs and metrics into our centralized observability platforms - enabling us to correlate metrics across our environment to find root cause of end user performance issues.  
• AppDynamics: Provides real-time visibility into how application, transaction, and end-user data impact our business metrics.  AppDynamics logs and metrics are also fed into our centralized observability platforms, further enhancing end-to-end visibility, 
• Topology:  Topology is a solution which pulls the physical and logical relationships across the IT stack from our Configuration Management Database and merges that data with other critical operational data (changes, incidents, problems) into a high-throughput, low-latency data store for real-time analytics and streaming data by our observability solutions.
• AI Operations: As we centralize our telemetry, we are deploying AI-driven solutions to power new observability experiences and workflows, with Splunk at the core. For example, we have deployed a custom, AI powered Observability Agent for apps and infrastructure providing health insights, AI summaries, resolution tips, and topology-based visualization along with alerts, incidents, and changes. We're also deploying AI capabilities on top of Splunk Observability Cloud, enabling natural language queries like, "Is there a performance issue with my application?" These AI capabilities are only made possible because our observability data is centralized in our Splunk platform.   

The outcome: prevention. Most enterprises discover problems when customers complain. For example, in Cisco IT Networking, through automation and agentic workflows we address 99.998% of alerts, preventing them from escalating into a problem or incident.

The competitive edge: Enterprises that prevent incidents instead of reacting to them will win. 

The ability to centralize and correlate data is truly game-changing. It not only enables faster MTTR but moves us to stronger predictive capabilities and closer to preventative systems.  

In the past 18 months we've seen a clear correlation between our increased use of observability and a decrease in the number of incidents, along with a reduction in the time it takes to resolve them. 

• 25% fewer major incidents 
• 45% faster Mean Time to Detect and Resolve, 54 minutes sooner per incident. 

For Cisco IT, observability is an ongoing transformation, and we look forward to continuing to share updates on our advancements and innovations. 

Resources

• Discover Cisco IT's full observability approach - read the case study and blog 
• Explore more case studies on how Cisco is strengthening digital resilience