Maybe, one of these days, my SRE team can define SLOs that actually make sense. This “alert on whatever the customer wants” stance really really sucks.
Maybe, one of these days, my SRE team can define SLOs that actually make sense. This “alert on whatever the customer wants” stance really really sucks.
Definitely not that lucky. We have customers who seem to watch dashboards and create a Sev1 anytime latency degrades by 10%. They explain to their account manager that they need to have perfect performance all the time. The AM then comes to us demanding that we increase the sensitivity of the alert. Management agrees. And then, voila, just like that we have an alert that flaps all day and all night that we aren’t “allowed” to remove until someone can show that the noise is literally stopping us from catching other stuff.
It’s insanity.
EDIT: I only stay because new leadership seems like they want to fix it earnestly. And things are headed in the right direction, but it takes a long time to turn a ship.