[01:15:04] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:50:06] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [02:15:30] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:55:31] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [03:42:31] PROBLEM - Host deployment-parsoid05 is DOWN: CRITICAL - Host Unreachable (10.68.16.120) [03:58:47] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,contintLabsSlave && UbuntuTrusty build #138: 04FAILURE in 2 min 46 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/138/ [04:17:44] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #138: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/138/ [04:26:41] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [04:29:03] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:01:36] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [05:09:07] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:20:15] Project beta-update-databases-eqiad build #11261: 04FAILURE in 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11261/ [06:27:37] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:59:29] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:02:37] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:15] Project beta-update-databases-eqiad build #11262: 04STILL FAILING in 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11262/ [07:34:29] RECOVERY - Puppet run on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:20:18] Project beta-update-databases-eqiad build #11263: 04STILL FAILING in 17 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11263/ [09:21:02] Yippee, build fixed! [09:21:03] Project beta-update-databases-eqiad build #11264: 09FIXED in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11264/ [12:07:27] (03PS1) 10Glaisher: Enable composer-test on mediawiki/extensions/GlobalBlocking [integration/config] - 10https://gerrit.wikimedia.org/r/309829 [12:31:51] (03CR) 10Paladox: [C: 031] Enable composer-test on mediawiki/extensions/GlobalBlocking [integration/config] - 10https://gerrit.wikimedia.org/r/309829 (owner: 10Glaisher) [12:53:58] 10Gerrit, 13Patch-For-Review, 07Upstream: Free-form tagging in gerrit - https://phabricator.wikimedia.org/T37534#2626670 (10Paladox) @Nikerabbit also hashtag requires us to use NoteDB, I'm not sure if we can use both MySQL and NoteDB. [13:27:10] 10Gerrit, 13Patch-For-Review, 07Upstream: Free-form tagging in gerrit - https://phabricator.wikimedia.org/T37534#2626693 (10Paladox) I filled https://bugs.chromium.org/p/gerrit/issues/detail?id=4542 for the hashtag thing. [13:33:19] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:51:10] 10Gerrit, 13Patch-For-Review, 07Upstream: Free-form tagging in gerrit - https://phabricator.wikimedia.org/T37534#2626710 (10Paladox) I just tested this and it seems this is supported in gerrit 2.13. So this is blocked until we update to gerrit 2.13. [14:13:19] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:31:45] 06Release-Engineering-Team, 10DBA, 10MediaWiki-Maintenance-scripts, 06Operations, and 2 others: Add section for long-running tasks on the Deployment page (specially for database maintenance) - https://phabricator.wikimedia.org/T144661#2626722 (10jcrespo) I would like some opinion from other ops or develope... [14:40:49] 10Gerrit, 05Goal: Migrate subversion to git - https://phabricator.wikimedia.org/T24596#2626728 (10Aklapper) [14:40:52] 10Gerrit, 13Patch-For-Review, 07Upstream: Free-form tagging in gerrit - https://phabricator.wikimedia.org/T37534#2626727 (10Aklapper) [14:47:28] 10Continuous-Integration-Config: BotPassword file for FLOSSbot - https://phabricator.wikimedia.org/T145331#2626737 (10dachary) [15:02:56] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10DBA, 10Datasets-General-or-Unknown, and 3 others: Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#2626763 (10jcrespo) [15:04:05] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10DBA, 10Datasets-General-or-Unknown, and 3 others: Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#1417808 (10jcrespo) [15:46:05] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:21:06] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [16:46:27] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:03:06] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:21:28] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:40] 06Release-Engineering-Team, 10Dumps-Generation: getMWVersion, used by dumps, was removed. Please restore. - https://phabricator.wikimedia.org/T145336#2626948 (10ArielGlenn) [18:44:09] (03CR) 10Legoktm: [C: 032] Enable composer-test on mediawiki/extensions/GlobalBlocking [integration/config] - 10https://gerrit.wikimedia.org/r/309829 (owner: 10Glaisher) [18:44:46] (03Merged) 10jenkins-bot: Enable composer-test on mediawiki/extensions/GlobalBlocking [integration/config] - 10https://gerrit.wikimedia.org/r/309829 (owner: 10Glaisher) [18:45:10] !log deploying https://gerrit.wikimedia.org/r/309829 [18:45:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:12:32] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:42:15] 10Gerrit, 06Developer-Relations: Add a welcome bot to Gerrit for first time contributors - https://phabricator.wikimedia.org/T73357#2627012 (10Aklapper) @demon: Any technical input? [19:51:28] 10Gerrit, 06Developer-Relations: Add a welcome bot to Gerrit for first time contributors - https://phabricator.wikimedia.org/T73357#745463 (10Legoktm) >>! In T73357#2585966, @Aklapper wrote: > Can someone judge whether the python module dependencies could be fulfilled in our setup and if there would be other t... [19:52:28] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:09:21] PROBLEM - Puppet run on deployment-ms-be02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:29:20] RECOVERY - Puppet run on deployment-ms-be02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:45:42] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:51:12] Sep 11 20:40:24 deployment-mathoid puppet-agent[19129]: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type sysctl::parameters at /etc/puppet/modules/base/manifests/sysctl.pp:35 on node deployment-mathoid.deployment-prep.eqiad.wmflabs [20:58:31] RECOVERY - Puppet staleness on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [3600.0] [21:16:04] PROBLEM - Puppet run on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:20:43] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [21:21:05] @Acer: what is your "reasonable, valid, correct, ethic reasons to do so." Making your purpose less clear make not more reasonable, valid, correct an ethic ? [21:21:15] wrong window [21:23:20] Dereckson is that for Acer the actual laptop maker? [21:51:57] 10Beta-Cluster-Infrastructure: deployment-pdf01 low free space warning - https://phabricator.wikimedia.org/T145343#2627173 (10AlexMonk-WMF) [21:53:05] we now have open tickets for all shinken alerts in deployment-prep [22:16:44] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:38:53] Krenair: well done [22:51:42] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0]