1. Avoid Relying Solely on Email for Alerts

Email is not a reliable alerting mechanism. Alerts can be delayed, filtered as spam, or ignored due to cluttered inboxes. From a user experience perspective, email-based alerts are often messy and ineffective.

Recommendation: Use a dedicated alerting app or platform that supports webhook integrations. Azure and most modern monitoring tools offer this functionality. These solutions provide more reliable and timely alerting mechanisms.
2. Be Selective and Strategic with Monitors

Not all systems or services require immediate alerts. It’s important to evaluate the criticality of each component before creating alerts.

Example: If a non-business-critical website goes offline at 2:00 AM, there’s usually no need for an immediate response—as long as it’s operational by 8:00 AM.

Recommendation: Establish alert priorities based on business impact and define appropriate response times.
3. Assign Clear Ownership for Every Alert

Every alert must have a single, clearly responsible person. While teamwork is encouraged, having a designated owner ensures accountability.

Important Note: If the responsible person’s shift ends, they must ensure a proper handover to the next team member. Avoid the “shared responsibility” trap where everyone assumes someone else will act.
4. Root Cause Analysis and Resolution

When an alert is triggered, don’t just silence it—identify and resolve the root cause.

Example: A system with insufficient memory caused batch jobs to crash the website. The long-term solution was to double the memory. If the client refuses this fix, acknowledge that the issue will persist and adjust alerting priorities accordingly.
5. Shift Monitoring Responsibility to the Right Teams

In a modern DevOps environment, the team that builds a service should also monitor and maintain it. This ownership drives quality and accountability.

Tip: If external support teams are used for first-line alert response, provide them with clear, documented “first aid” guides. If they report issues they couldn’t resolve, update the documentation accordingly.
6. Use the Right Tools for High-Value Systems

Business-critical environments demand robust monitoring solutions.

Recommendation: Tools like OpsGenie are worth the investment for their reliability and advanced features. While Azure has solid alerting capabilities, it lacks some of OpsGenie’s incident management features—review both to understand the gaps.
7. Set Proper Alert Thresholds

Avoid excessive noise by tuning alert thresholds carefully. Poorly configured alerts can lead to fatigue and missed critical issues.

Note: Nobody wants to be woken up at night for a non-critical alert—thresholds should reflect operational realities.
8. Understand Performance Metrics Beyond Averages

Averages can mask performance problems. For example, if 9 requests take 1 second and 1 takes 10 seconds, the average of 2 seconds appears acceptable, but the user experience for that one request is not.

Recommendation: Use percentile-based metrics (like P95, P99) instead of averages for performance analysis. Azure’s Kusto Query Language (KQL) is particularly powerful for detailed and insightful queries. Learning KQL benefits not just monitoring, but also resource management and analysis.