[19:32:44] What’s going on with all the restbase alerts? [19:36:59] Looks to be recovering, the restbase and proton links to wikitech don’t exist though [19:37:23] https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [19:37:54] https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [19:38:25] https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [19:45:05] there was a large latency spike on the apiservers (less so on the main appservers), which ofc is called by all those services [19:45:10] looks to be s8 load that's the problem [19:45:21] i think there is a new bot making queries that are expensive for s8 [19:46:29] cdanis: thanks for looking, should a task be filed about changing the links on the icinga alerts? [19:47:28] imo those icinga alerts should be removed entirely, and replaced with ones that are aggregated (so there isn't a "request failed" alert for literally every host, there's a single "requests failed on too many hosts" alert) [19:47:53] you mean you dont like to be pinged every 5 seconds cdanis /s [19:48:02] fortunately they aren't pages, just IRC spam [19:48:21] cdanis: i could imagine if they were pages, considering don't you guys get SMS messages from icinga? [19:48:26] indeed [19:48:58] cdanis: anyways have a great rest of the weekend... don't work too hard [19:49:05] thanks, appreciate it :) [19:49:13] np [19:50:27] Have a good weekend cdanis, Shall I go ahead and create a task to sort that tonight or you want to? [19:51:27] if you don't mind RhinosF1, please create a task to aggregate the restbase, mobileapps, and proton alerts, and to add proper documentation links :) [19:51:43] cdanis: np [19:57:17] cdanis: {{done}} https://phabricator.wikimedia.org/T250017 [19:57:26] thanks! [19:57:40] Happy to help!