[02:39:14] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [02:42:55] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [09:15:28] (Abandoned) Hashar: Jenkins job validation (DO NOT SUBMIT) [analytics/kraken] - https://gerrit.wikimedia.org/r/114531 (owner: Hashar) [09:27:28] Analytics-Cluster, Ops-Access-Requests, operations: Sudo permissions for hdfs user madhuvishy on analytics-hadoop - https://phabricator.wikimedia.org/T104020#1464736 (fgiunchedi) [09:28:42] Analytics-Cluster, Ops-Access-Requests, operations: Sudo permissions for hdfs user madhuvishy on analytics-hadoop - https://phabricator.wikimedia.org/T104020#1464742 (fgiunchedi) p:Triage>Normal [10:33:04] Analytics-Cluster, operations, Patch-For-Review: Can't download large datasets from datasets.wikimedia.org - https://phabricator.wikimedia.org/T104004#1464806 (fgiunchedi) confirmed this is still a problem, I think what's happening is that we're no longer caching in varnish but it will still try to fet... [10:33:25] Analytics-Cluster, operations, Patch-For-Review: Can't download large datasets from datasets.wikimedia.org - https://phabricator.wikimedia.org/T104004#1464807 (fgiunchedi) p:Triage>Normal [10:59:24] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [11:01:25] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [14:31:29] Analytics-Cluster, operations: Can't download large datasets from datasets.wikimedia.org - https://phabricator.wikimedia.org/T104004#1465137 (fgiunchedi) [14:41:09] Analytics-Cluster, operations, ops-eqiad: rack new hadoop worker nodes - https://phabricator.wikimedia.org/T104463#1465156 (Cmjohnson) analytics1042-1045 are racked and ready for install in row D2. Racktables has been updated. analytics1045.mgmt.eqiad.wmnet has address 10.65.4.17 analytics1044.mgmt.e... [15:29:05] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [15:33:14] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [15:35:05] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [16:12:05] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [16:17:55] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [16:54:58] when's otto back? [18:01:36] Analytics-Features: Request feature - build and populate database using a LocalSettings.php file - https://phabricator.wikimedia.org/T106340#1465546 (CipherWizard) NEW [18:45:42] Quarry, Easy: String "Your query is currently executing" should be "This query..." - https://phabricator.wikimedia.org/T103275#1465629 (matej_suchanek) [22:31:24] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [22:39:05] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0]