[03:37:55] Analytics-Cluster, Analytics-Kanban, Datasets-Archiving, Datasets-Webstatscollector: Mediacounts missing top1000 files after 2016-01-01: rsync fails - https://phabricator.wikimedia.org/T122864#1939452 (Hydriz) Resolved>Open It doesn't seem to be fixed yet. There are still no top1000 files af... [07:41:58] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [08:23:58] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 26.67% of data above the critical threshold [30.0] [08:26:54] !log restarting eventlogging to see if it'll help burrow reported kafka consumer lag [08:26:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [08:51:47] PROBLEM - Overall insertion rate from MySQL consumer on graphite1001 is CRITICAL: CRITICAL: 20.00% of data under the critical threshold [10.0] [08:53:48] RECOVERY - Overall insertion rate from MySQL consumer on graphite1001 is OK: OK: Less than 20.00% under the threshold [100.0] [08:57:38] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [09:00:09] PROBLEM - Overall insertion rate from MySQL consumer on graphite1001 is CRITICAL: CRITICAL: 20.00% of data under the critical threshold [10.0] [10:00:35] Analytics-General-or-Unknown, Community-Advocacy, Wikimedia-Extension-setup, Wikipedia-iOS-App-Product-Backlog: enable Piwik on ru.wikimedia.org - https://phabricator.wikimedia.org/T91963#1939598 (Multichill) >>! In T91963#1939084, @Milimetric wrote: > To keep the archives happy: > > We've deploy... [14:03:08] Analytics-Cluster, Analytics-Kanban, Datasets-Archiving, Datasets-Webstatscollector: Mediacounts missing top1000 files after 2016-01-01: rsync fails - https://phabricator.wikimedia.org/T122864#1939748 (ezachte) Open>Resolved There was lingering test code. Files are up to date. [15:07:38] RECOVERY - Overall insertion rate from MySQL consumer on graphite1001 is OK: OK: Less than 20.00% under the threshold [100.0] [16:00:53] Analytics-EventLogging, MediaWiki-API, Easy, Google-Code-In-2015, Patch-For-Review: ApiJsonSchema implements ApiBase::getCustomPrinter for no good reason - https://phabricator.wikimedia.org/T91454#1939833 (Florian) Adding @greg and @20after4 because of the comment from @Anomie on the change: htt... [18:40:19] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [20:10:09] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [21:41:29] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [22:09:17] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [23:06:18] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [23:46:18] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 26.67% of data above the critical threshold [30.0] [23:54:47] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0]