[03:41:16] PROBLEM - Check the last execution of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [04:34:38] RECOVERY - Check the last execution of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [04:47:12] 10Analytics, 10Operations, 10Traffic: Artificial spike in offset of unique devices from November to February 6th on wikidata - https://phabricator.wikimedia.org/T165560 (10Ladsgroup) 05Open→03Declined Three years have passed from this incident and as result, there's no data left from that time to examin... [07:20:02] PROBLEM - Throughput of EventLogging EventError events on alert1001 is CRITICAL: 39.72 ge 30 https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Administration https://grafana.wikimedia.org/dashboard/db/eventlogging?panelId=13&fullscreen&orgId=1 [07:56:56] RECOVERY - Throughput of EventLogging EventError events on alert1001 is OK: (C)30 ge (W)20 ge 19.32 https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Administration https://grafana.wikimedia.org/dashboard/db/eventlogging?panelId=13&fullscreen&orgId=1 [10:10:46] PROBLEM - Throughput of EventLogging EventError events on alert1001 is CRITICAL: 41.28 ge 30 https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Administration https://grafana.wikimedia.org/dashboard/db/eventlogging?panelId=13&fullscreen&orgId=1 [10:27:50] RECOVERY - Throughput of EventLogging EventError events on alert1001 is OK: (C)30 ge (W)20 ge 14.66 https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Administration https://grafana.wikimedia.org/dashboard/db/eventlogging?panelId=13&fullscreen&orgId=1 [12:18:32] PROBLEM - Check the last execution of performance-asoranking on stat1007 is CRITICAL: CRITICAL: Status of the systemd unit performance-asoranking https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [20:53:21] 10Analytics-Kanban, 10Trash, 10Upstream: --- DISCUSSED BELOW --- - https://phabricator.wikimedia.org/T114124 (10Aklapper) [21:16:01] 10Analytics, 10Analytics-EventLogging, 10Research-Backlog: 20K events by a single user in the span of 20 mins - https://phabricator.wikimedia.org/T202539 (10Aklapper) 05Stalled→03Open The previous comments don't explain who or what (task?) exactly this task is stalled on (["If a report is waiting for fur... [21:23:34] 10Analytics, 10Analytics-Wikistats: Stats.wikimedia.org - https://phabricator.wikimedia.org/T266974 (10Jnovikov) [21:28:40] 10Analytics, 10Analytics-Wikistats: Stats.wikimedia.org - https://phabricator.wikimedia.org/T266974 (10Aklapper) Hi @Jnovikov, thanks for taking the time to report this and welcome to Wikimedia Phabricator! For future reference, please follow https://www.mediawiki.org/wiki/How_to_report_a_bug - thanks. > Lots... [21:50:36] 10Analytics, 10Analytics-Wikistats: Stats.wikimedia.org - https://phabricator.wikimedia.org/T266974 (10Jnovikov) Isn't it enough? [23:57:07] 10Analytics, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Reedy)