[00:13:32] 10Continuous-Integration-Infrastructure: Upgrade PHPUnit to 4.0+ - https://phabricator.wikimedia.org/T99982#1303101 (10Ejegg) 3NEW [00:14:47] 6Release-Engineering, 10Continuous-Integration-Config: MobileFrontend should run Gather tests as well - https://phabricator.wikimedia.org/T99877#1303108 (10hashar) In short, we need a qunit job that clone and his triggered by a bunch of extensions like the mediawiki-testextensions jobs. [00:23:33] 10Continuous-Integration-Infrastructure, 6Scrum-of-Scrums, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Jenkins is using php-luasandbox 1.9-1 for zend unit tests; precise should be upgraded to 2.0-8 or equivalent - https://phabricator.wikimedia.org/T88798#1303125 (10hashar) Apparently all the... [00:27:17] PROBLEM - Puppet failure on integration-slave-precise-1014 is CRITICAL 60.00% of data above the critical threshold [0.0] [00:47:38] (03PS1) 10Awight: Can't run the new tests on the fundraising/REL1_25 branch, yet [integration/config] - 10https://gerrit.wikimedia.org/r/212739 (https://phabricator.wikimedia.org/T94528) [00:47:41] (03CR) 10jenkins-bot: [V: 04-1] Can't run the new tests on the fundraising/REL1_25 branch, yet [integration/config] - 10https://gerrit.wikimedia.org/r/212739 (https://phabricator.wikimedia.org/T94528) (owner: 10Awight) [00:49:09] (03PS2) 10Awight: Can't run the new tests on the fundraising/REL1_25 branch, yet [integration/config] - 10https://gerrit.wikimedia.org/r/212739 (https://phabricator.wikimedia.org/T94528) [00:52:18] RECOVERY - Puppet failure on integration-slave-precise-1014 is OK Less than 1.00% above the threshold [0.0] [02:07:47] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<50.00%) [02:11:16] matanya: I believe you can un-archive projects [02:12:01] "activate project" on the project details page [05:59:52] matanya: yeah, you can [06:09:22] 6Release-Engineering, 10Wikidata: make-wmf-branch should check if it is up to date - https://phabricator.wikimedia.org/T99998#1303327 (10JanZerebecki) 3NEW [06:17:57] 10Continuous-Integration-Infrastructure, 6Scrum-of-Scrums, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Jenkins is using php-luasandbox 1.9-1 for zend unit tests; precise should be upgraded to 2.0-8 or equivalent - https://phabricator.wikimedia.org/T88798#1303337 (10Anomie) The other failures... [06:20:53] 6Release-Engineering: No /static/1.26wmf7 tree -- mediawiki.org missing "Powered by MediaWiki" logo and ResourceLoader ?debug=1 mode broken - https://phabricator.wikimedia.org/T99886#1303339 (10mmodell) https://gerrit.wikimedia.org/r/#/c/212753/ adds a unit test to verify that the requisite branch symlinks exist. [06:39:53] 6Release-Engineering, 10Wikidata: make-wmf-branch should check if it is up to date - https://phabricator.wikimedia.org/T99998#1303344 (10mmodell) a:3mmodell I'll implement this [06:40:29] 6Release-Engineering, 10Wikidata: make-wmf-branch should check if it is up to date - https://phabricator.wikimedia.org/T99998#1303346 (10mmodell) p:5Triage>3High [06:57:45] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [07:05:01] (03PS1) 1020after4: make-wmf-branch should check that it's running the latest origin/master [tools/release] - 10https://gerrit.wikimedia.org/r/212757 (https://phabricator.wikimedia.org/T99998) [08:28:07] RECOVERY - Puppet staleness on deployment-restbase02 is OK Less than 1.00% above the threshold [3600.0] [08:28:17] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL 44.44% of data above the critical threshold [0.0] [08:28:21] RECOVERY - Puppet staleness on deployment-restbase01 is OK Less than 1.00% above the threshold [3600.0] [08:38:22] RECOVERY - Puppet failure on deployment-restbase02 is OK Less than 1.00% above the threshold [0.0] [08:56:42] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: Connection refused [09:03:17] PROBLEM - Puppet failure on deployment-sca02 is CRITICAL 100.00% of data above the critical threshold [0.0] [09:07:49] thanks greg-g and twentyafterfour [09:22:26] PROBLEM - Puppet failure on deployment-mx is CRITICAL 100.00% of data above the critical threshold [0.0] [10:36:46] PROBLEM - Puppet failure on deployment-salt is CRITICAL 30.00% of data above the critical threshold [0.0] [10:47:37] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL 20.00% of data above the critical threshold [0.0] [10:48:37] PROBLEM - Puppet failure on deployment-test is CRITICAL 40.00% of data above the critical threshold [0.0] [10:48:41] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL 30.00% of data above the critical threshold [0.0] [10:49:43] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL 50.00% of data above the critical threshold [0.0] [10:50:06] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL 55.56% of data above the critical threshold [0.0] [10:50:22] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 33.33% of data above the critical threshold [0.0] [10:51:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL 60.00% of data above the critical threshold [0.0] [10:51:28] PROBLEM - Puppet failure on deployment-bastion is CRITICAL 33.33% of data above the critical threshold [0.0] [10:52:40] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL 50.00% of data above the critical threshold [0.0] [10:52:54] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL 50.00% of data above the critical threshold [0.0] [10:53:00] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL 30.00% of data above the critical threshold [0.0] [10:55:08] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL 62.50% of data above the critical threshold [0.0] [10:55:10] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL 66.67% of data above the critical threshold [0.0] [11:06:45] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [11:15:05] RECOVERY - Puppet failure on deployment-mediawiki01 is OK Less than 1.00% above the threshold [0.0] [11:16:27] RECOVERY - Puppet failure on deployment-memc02 is OK Less than 1.00% above the threshold [0.0] [11:17:37] RECOVERY - Puppet failure on deployment-fluoride is OK Less than 1.00% above the threshold [0.0] [11:18:39] RECOVERY - Puppet failure on deployment-sca01 is OK Less than 1.00% above the threshold [0.0] [11:18:39] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [11:19:37] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [11:20:10] RECOVERY - Puppet failure on deployment-mediawiki02 is OK Less than 1.00% above the threshold [0.0] [11:20:10] RECOVERY - Puppet failure on deployment-apertium01 is OK Less than 1.00% above the threshold [0.0] [11:20:23] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [11:21:26] RECOVERY - Puppet failure on deployment-bastion is OK Less than 1.00% above the threshold [0.0] [11:22:44] RECOVERY - Puppet failure on deployment-pdf02 is OK Less than 1.00% above the threshold [0.0] [11:22:46] PROBLEM - Puppet failure on deployment-salt is CRITICAL 60.00% of data above the critical threshold [0.0] [11:22:54] RECOVERY - Puppet failure on deployment-sentry2 is OK Less than 1.00% above the threshold [0.0] [11:23:00] RECOVERY - Puppet failure on deployment-memc04 is OK Less than 1.00% above the threshold [0.0] [11:25:10] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL 55.56% of data above the critical threshold [0.0] [11:42:48] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [11:50:10] RECOVERY - Puppet failure on deployment-elastic06 is OK Less than 1.00% above the threshold [0.0] [12:03:47] PROBLEM - Puppet failure on deployment-salt is CRITICAL 30.00% of data above the critical threshold [0.0] [12:15:32] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL 60.00% of data above the critical threshold [0.0] [12:17:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL 50.00% of data above the critical threshold [0.0] [12:18:54] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL 30.00% of data above the critical threshold [0.0] [12:19:00] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL 20.00% of data above the critical threshold [0.0] [12:23:47] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [12:31:09] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL 22.22% of data above the critical threshold [0.0] [12:34:02] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL 22.22% of data above the critical threshold [0.0] [12:35:06] PROBLEM - Puppet failure on deployment-upload is CRITICAL 22.22% of data above the critical threshold [0.0] [12:35:37] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL 20.00% of data above the critical threshold [0.0] [12:36:03] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL 22.22% of data above the critical threshold [0.0] [12:36:21] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL 50.00% of data above the critical threshold [0.0] [12:37:39] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 50.00% of data above the critical threshold [0.0] [12:38:36] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL 60.00% of data above the critical threshold [0.0] [12:38:36] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL 50.00% of data above the critical threshold [0.0] [12:39:44] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL 50.00% of data above the critical threshold [0.0] [12:39:44] PROBLEM - Puppet failure on deployment-test is CRITICAL 60.00% of data above the critical threshold [0.0] [12:41:08] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL 33.33% of data above the critical threshold [0.0] [12:41:10] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL 22.22% of data above the critical threshold [0.0] [12:41:22] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 55.56% of data above the critical threshold [0.0] [12:42:28] PROBLEM - Puppet failure on deployment-bastion is CRITICAL 44.44% of data above the critical threshold [0.0] [12:43:38] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL 70.00% of data above the critical threshold [0.0] [12:44:48] PROBLEM - Puppet failure on deployment-salt is CRITICAL 10.00% of data above the critical threshold [0.0] [12:45:02] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL 55.56% of data above the critical threshold [0.0] [12:45:18] PROBLEM - Puppet failure on deployment-db1 is CRITICAL 30.00% of data above the critical threshold [0.0] [12:45:23] PROBLEM - Puppet failure on deployment-stream is CRITICAL 50.00% of data above the critical threshold [0.0] [12:45:37] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL 20.00% of data above the critical threshold [0.0] [12:45:59] PROBLEM - Puppet failure on deployment-db2 is CRITICAL 20.00% of data above the critical threshold [0.0] [12:46:39] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 30.00% of data above the critical threshold [0.0] [12:47:05] PROBLEM - Puppet failure on deployment-fluorine is CRITICAL 33.33% of data above the critical threshold [0.0] [12:47:25] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL 60.00% of data above the critical threshold [0.0] [12:49:03] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL 40.00% of data above the critical threshold [0.0] [12:49:51] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [12:51:27] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL 66.67% of data above the critical threshold [0.0] [13:01:11] RECOVERY - Puppet failure on deployment-elastic06 is OK Less than 1.00% above the threshold [0.0] [13:02:19] RECOVERY - Puppet failure on deployment-elastic05 is OK Less than 1.00% above the threshold [0.0] [13:05:17] RECOVERY - Puppet failure on deployment-db1 is OK Less than 1.00% above the threshold [0.0] [13:05:37] RECOVERY - Puppet failure on deployment-logstash1 is OK Less than 1.00% above the threshold [0.0] [13:06:02] RECOVERY - Puppet failure on deployment-db2 is OK Less than 1.00% above the threshold [0.0] [13:06:30] RECOVERY - Puppet failure on deployment-cxserver03 is OK Less than 1.00% above the threshold [0.0] [13:06:40] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [13:07:06] RECOVERY - Puppet failure on deployment-fluorine is OK Less than 1.00% above the threshold [0.0] [13:09:02] RECOVERY - Puppet failure on deployment-mediawiki03 is OK Less than 1.00% above the threshold [0.0] [13:13:36] RECOVERY - Puppet failure on deployment-redis01 is OK Less than 1.00% above the threshold [0.0] [13:14:02] RECOVERY - Puppet failure on deployment-mathoid is OK Less than 1.00% above the threshold [0.0] [13:14:38] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [13:15:33] RECOVERY - Puppet failure on deployment-memc03 is OK Less than 1.00% above the threshold [0.0] [13:16:05] RECOVERY - Puppet failure on deployment-mediawiki01 is OK Less than 1.00% above the threshold [0.0] [13:16:23] RECOVERY - Puppet failure on deployment-elastic08 is OK Less than 1.00% above the threshold [0.0] [13:16:23] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [13:17:27] RECOVERY - Puppet failure on deployment-memc02 is OK Less than 1.00% above the threshold [0.0] [13:17:39] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [13:18:35] RECOVERY - Puppet failure on deployment-fluoride is OK Less than 1.00% above the threshold [0.0] [13:18:43] RECOVERY - Puppet failure on deployment-pdf02 is OK Less than 1.00% above the threshold [0.0] [13:18:55] RECOVERY - Puppet failure on deployment-sentry2 is OK Less than 1.00% above the threshold [0.0] [13:19:41] RECOVERY - Puppet failure on deployment-sca01 is OK Less than 1.00% above the threshold [0.0] [13:20:09] RECOVERY - Puppet failure on deployment-upload is OK Less than 1.00% above the threshold [0.0] [13:20:38] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [13:21:07] RECOVERY - Puppet failure on deployment-mediawiki02 is OK Less than 1.00% above the threshold [0.0] [13:21:09] RECOVERY - Puppet failure on deployment-apertium01 is OK Less than 1.00% above the threshold [0.0] [13:22:30] RECOVERY - Puppet failure on deployment-bastion is OK Less than 1.00% above the threshold [0.0] [13:24:02] RECOVERY - Puppet failure on deployment-memc04 is OK Less than 1.00% above the threshold [0.0] [13:25:04] RECOVERY - Puppet failure on deployment-parsoid05 is OK Less than 1.00% above the threshold [0.0] [13:25:24] RECOVERY - Puppet failure on deployment-stream is OK Less than 1.00% above the threshold [0.0] [14:13:17] RECOVERY - Puppet failure on deployment-sca02 is OK Less than 1.00% above the threshold [0.0] [14:16:41] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.019 second response time [14:34:05] RECOVERY - Puppet failure on deployment-zotero01 is OK Less than 1.00% above the threshold [0.0] [14:36:42] RECOVERY - Puppet staleness on deployment-zotero01 is OK Less than 1.00% above the threshold [3600.0] [14:51:42] 10Continuous-Integration-Infrastructure, 6Labs: Designate should support split horizon resolution to yield private IP of instances behind a public DNS entry - https://phabricator.wikimedia.org/T95288#1303702 (10yuvipanda) [15:08:45] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<50.00%) [17:52:32] 6Release-Engineering, 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Grant access for aklapper to phab-admins - https://phabricator.wikimedia.org/T97642#1303913 (10RobH) 5Open>3stalled @Aklapper, Since you are likely busy with hackathon and the like, I've placed this task to stalled. No one i... [18:21:31] PROBLEM - Puppet failure on integration-zuul-packaged is CRITICAL 100.00% of data above the critical threshold [0.0] [19:11:49] 10Deployment-Systems, 6operations, 5Patch-For-Review: need package 'trebuchet-trigger' for trusty - https://phabricator.wikimedia.org/T99919#1304005 (10Dzahn) a:3Dzahn [20:29:09] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL 33.33% of data above the critical threshold [43200.0] [20:34:22] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 33.33% of data above the critical threshold [43200.0] [21:17:26] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL 100.00% of data above the critical threshold [43200.0] [22:43:42] 10Deployment-Systems, 6operations, 5Patch-For-Review: need package 'trebuchet-trigger' for trusty - https://phabricator.wikimedia.org/T99919#1304168 (10Dzahn) adjusted python-git dependency, built 0.5.6 for trusty, added to repo, puppet installed it on mira ``` Notice: /Stage[main]/Deployment::Deployment_... [22:47:12] 10Deployment-Systems, 6operations, 5Patch-For-Review: need package 'trebuchet-trigger' for trusty - https://phabricator.wikimedia.org/T99919#1304169 (10Dzahn) @cscott for completeness: the "import to our apt repo"-step was to scp all the files to host carbon, into /srv/wikimedia/incoming, then become root... [22:47:39] 10Deployment-Systems, 6operations, 5Patch-For-Review: need package 'trebuchet-trigger' for trusty - https://phabricator.wikimedia.org/T99919#1304170 (10Dzahn) 5Open>3Resolved