[00:03:19] [00:16:34] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T191051#4230452 (10matmarex) [00:25:37] (03PS4) 10Legoktm: Remove dockerfiles/build.py [integration/config] - 10https://gerrit.wikimedia.org/r/434000 (owner: 10Chad) [00:28:31] (03CR) 10jerkins-bot: [V: 04-1] Remove dockerfiles/build.py [integration/config] - 10https://gerrit.wikimedia.org/r/434000 (owner: 10Chad) [00:33:53] (03PS5) 10Legoktm: Remove dockerfiles/build.py [integration/config] - 10https://gerrit.wikimedia.org/r/434000 (owner: 10Chad) [00:45:24] (03PS1) 10Chad: makerelease.py: Clean up package dir management, actually return extensions and skins [tools/release] - 10https://gerrit.wikimedia.org/r/435069 [00:51:09] (03PS1) 10Chad: makerelease: Clone with a depth of 1 [tools/release] - 10https://gerrit.wikimedia.org/r/435072 [00:56:23] (03CR) 10Legoktm: [C: 032] Remove dockerfiles/build.py [integration/config] - 10https://gerrit.wikimedia.org/r/434000 (owner: 10Chad) [00:57:36] (03Merged) 10jenkins-bot: Remove dockerfiles/build.py [integration/config] - 10https://gerrit.wikimedia.org/r/434000 (owner: 10Chad) [00:58:29] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736#4230486 (10Krenair) Yes, this instance's presence will not be optional in future, it will be needed for things like T194962 (and als... [00:59:01] ^ I would very much like an explanation for that. [00:59:47] 10Beta-Cluster-Infrastructure, 10Puppet: deployment-secureredirexperiment puppet error - https://phabricator.wikimedia.org/T191663#4230498 (10Krenair) 05Open>03Resolved a:03Krenair killed it [01:04:03] 10Beta-Cluster-Infrastructure: Beta: acme-setup failing in beta deployment-cache-upload04 - https://phabricator.wikimedia.org/T178404#4230514 (10Krenair) 05Open>03Resolved I don't know what this was but it's not happening now [01:07:34] 10Beta-Cluster-Infrastructure, 10cloud-services-team, 10Puppet: labs-puppetmaster/Labs Puppetmaster HTTPS is UNKNOWN since [...] - https://phabricator.wikimedia.org/T191553#4109867 (10Krenair) It's supposed to have a certificate signed like that, clients of that puppetmaster will trust it as it's added to th... [01:15:01] 10Beta-Cluster-Infrastructure, 10Shinken, 10cloud-services-team, 10Puppet: labs-puppetmaster/Labs Puppetmaster HTTPS is UNKNOWN since [...] - https://phabricator.wikimedia.org/T191553#4230518 (10Krenair) ```root@shinken-01:~# openssl s_client -connect labs-puppetmaster.wikimedia.org:8140 CONNECTED(00000003... [01:26:41] 10Beta-Cluster-Infrastructure, 10Shinken, 10cloud-services-team, 10Puppet: labs-puppetmaster/Labs Puppetmaster HTTPS is UNKNOWN since [...] - https://phabricator.wikimedia.org/T191553#4230537 (10Krenair) Looks like this was broken by @herron in https://gerrit.wikimedia.org/r/#/c/392423/ - check_https_port_... [01:34:07] 10Beta-Cluster-Infrastructure, 10Shinken, 10cloud-services-team, 10Patch-For-Review, 10Puppet: labs-puppetmaster/Labs Puppetmaster HTTPS is UNKNOWN since [...] - https://phabricator.wikimedia.org/T191553#4230559 (10Krenair) a:03Krenair [02:11:41] 10Beta-Cluster-Infrastructure: deployment-deploy1001 has puppet error - https://phabricator.wikimedia.org/T194927#4213221 (10Krenair) why on earth is that host named 1001? it doesn't make sense to use that convention in labs which is eqiad-only [02:14:44] 10Beta-Cluster-Infrastructure: puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts - https://phabricator.wikimedia.org/T194926#4230618 (10Krenair) a:03EddieGP [02:16:21] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Services (next), 10User-Joe: puppet dependency loop on deployment-sca hosts - https://phabricator.wikimedia.org/T171173#3456426 (10Krenair) How easy would it be to move beta to using discovery like prod does? [02:17:48] 10Beta-Cluster-Infrastructure, 10Puppet: deployment-eventlog05 puppet error about missing mysql heartbeat.heartbeat table - https://phabricator.wikimedia.org/T191109#4230621 (10Krenair) [02:18:05] 10Beta-Cluster-Infrastructure: deployment-deploy1001 has puppet error - https://phabricator.wikimedia.org/T194927#4230622 (10Krenair) anyway this has gotta be some jessie vs. stretch thing [05:32:28] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568#4230710 (10MoritzMuehlenhoff) We could also avoid downtime by temporarily reusing mw1298 (former image scaler) and reinstalling it as phab10... [05:59:38] (03CR) 10Physikerwelt: [C: 031] Do not run mwgate-rake-docker job for Math [integration/config] - 10https://gerrit.wikimedia.org/r/434950 (https://phabricator.wikimedia.org/T162455) (owner: 10Zfilipin) [06:50:38] PROBLEM - Free space - all mounts on deployment-mx is CRITICAL: CRITICAL: deployment-prep.deployment-mx.diskspace._var_log.byte_percentfree (<100.00%) [07:02:46] RECOVERY - Free space - all mounts on deployment-mx is OK: OK: All targets OK [07:09:43] (03CR) 10Hashar: [C: 032] "Fabric should work with either python2 or python3 from your system!" [integration/config] - 10https://gerrit.wikimedia.org/r/434941 (owner: 10Zfilipin) [07:11:34] (03Merged) 10jenkins-bot: Ignore .python-version file needed by pyenv-virtualenv [integration/config] - 10https://gerrit.wikimedia.org/r/434941 (owner: 10Zfilipin) [07:21:11] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Fundraising/REL1_27 tests broken - https://phabricator.wikimedia.org/T195516#4230802 (10hashar) >>! In T195516#4229632, @Reedy wrote: > It tries to checkout fundraising/REL1_27 branches of various extensions, but as the... [08:08:25] PROBLEM - Puppet errors on integration-quibble-demo is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:27:50] PROBLEM - Host integration-quibble-demo is DOWN: CRITICAL - Host Unreachable (10.68.23.5) [08:43:43] ^^ I have deleted that instance [08:50:46] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4230857 (10hashar) [08:50:49] (03PS1) 10Hashar: Migrate Widgets to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435133 (https://phabricator.wikimedia.org/T183512) [08:51:47] (03CR) 10Hashar: [C: 032] Migrate Widgets to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435133 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [08:53:15] (03Merged) 10jenkins-bot: Migrate Widgets to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435133 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [09:19:28] (03PS1) 10Hashar: Migrate ActiveAbstract to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435138 (https://phabricator.wikimedia.org/T183512) [09:19:58] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4230906 (10hashar) [09:20:02] (03CR) 10Hashar: [C: 032] Migrate ActiveAbstract to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435138 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [09:21:40] (03Merged) 10jenkins-bot: Migrate ActiveAbstract to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435138 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [09:27:04] 10Release-Engineering-Team (Kanban), 10Wikidata, 10Wikidata-Ministry-Of-Magic-Tech-Debt, 10Wikidata-Turtles-Tech-Debt, and 4 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#4230917 (10zeljkofilipin) I am not sure what happened, [[ https://integration.wikimedi... [09:40:42] 10Project-Admins: Create a new phabricator project for Linua Libre - https://phabricator.wikimedia.org/T195565#4230965 (100x010C) [09:40:45] (03PS1) 10Hashar: Migrate MinervaNeue to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435141 (https://phabricator.wikimedia.org/T183512) [09:41:16] (03CR) 10Hashar: [C: 032] Migrate MinervaNeue to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435141 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [09:41:17] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4230976 (10hashar) [09:43:18] (03Merged) 10jenkins-bot: Migrate MinervaNeue to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435141 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:05:39] zeljkof: hi! just fyi, due to the fun we had with wikidata last night, I will probably not get to look today at all the stuff you did with our browser tests [10:06:10] zeljkof: thanks a lot, I saw many things moving! Will hopefully get back to all the open questions, weird behaviour etc on Monday [10:06:40] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Fundraising/REL1_27 tests broken - https://phabricator.wikimedia.org/T195516#4231058 (10Reedy) FR doesn't need to test all the various extensions though? [10:06:44] zeljkof: oh, and your use of emojis on phabricator is pretty impressive :) [10:07:41] leszek_wmde: my emoji-fu is black belt! 💪 [10:33:47] 10Beta-Cluster-Infrastructure, 10Cloud-Services, 10Wikimedia-General-or-Unknown: rename -labs.php to -beta.php - https://phabricator.wikimedia.org/T150268#4231118 (10Reedy) 05stalled>03declined [10:48:48] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T191051#4231130 (10Deskana) [10:51:46] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T191051#4231133 (10Deskana) [12:00:20] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4231233 (10Trizek) [12:02:57] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4231249 (10Trizek) a:05Aklapper>03None De-assigning Andre, while they are pleinty of Phab admins who can do that. :) @0x010c, Andre told me that there is a bug tracking system on the Rid repo you... [12:22:47] (03PS1) 10Hashar: Migrate FlaggedRevs to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435158 (https://phabricator.wikimedia.org/T183512) [12:23:21] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4231294 (10hashar) [12:23:23] (03CR) 10Hashar: [C: 032] Migrate FlaggedRevs to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435158 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [12:24:43] (03Merged) 10jenkins-bot: Migrate FlaggedRevs to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435158 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [12:26:45] 10Continuous-Integration-Infrastructure, 10AntiSpoof, 10Patch-For-Review: AntiSpoof extension does not pass quibble-vendor-mysql-php70-docker - https://phabricator.wikimedia.org/T195020#4231297 (10hashar) [12:26:48] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4231296 (10hashar) [12:38:40] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:42:58] 10Continuous-Integration-Infrastructure, 10AntiSpoof, 10Patch-For-Review: AntiSpoof extension does not pass quibble-vendor-mysql-php70-docker - https://phabricator.wikimedia.org/T195020#4231338 (10hashar) The quibble* jobs runs with MariaDB with the collation `latin1_swedish_ci`. Maybe that `ci` stands for... [12:44:05] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4231350 (100x010C) That's it. Wikimedians are more used nowadays to report bugs / ask features on phabricator, and the board feature here better fits our needs. Furthermore, in a later point in time wh... [12:54:27] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:08:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4231391 (10hashar) [13:36:35] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:55:17] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Other: TimezoneSelector isLoggedIn() on null ($wgUser) - https://phabricator.wikimedia.org/T195585#4231483 (10hashar) p:05Triage>03Normal [13:55:58] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Other: TimezoneSelector isLoggedIn() on null ($wgUser) - https://phabricator.wikimedia.org/T195585#4231500 (10hashar) [13:56:25] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Other: TimezoneSelector isLoggedIn() on null ($wgUser) - https://phabricator.wikimedia.org/T195585#4231483 (10hashar) a:05hashar>03None [13:56:45] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4231505 (10hashar) [13:57:36] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Other: TimezoneSelector isLoggedIn() on null ($wgUser) - https://phabricator.wikimedia.org/T195585#4231509 (10Jayprakash12345) [14:02:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MediaWiki-Vagrant, 10User-zeljkofilipin: Continuous integration for mediawiki-vagrant - https://phabricator.wikimedia.org/T183456#3853760 (10Tgr) This problem was later fixed in {T187978}, right? I think this would be somewhat... [14:04:37] (03PS1) 10Hashar: tests: hint at integration-config-qa job [integration/config] - 10https://gerrit.wikimedia.org/r/435173 [14:05:32] (03CR) 10Hashar: [C: 032] tests: hint at integration-config-qa job [integration/config] - 10https://gerrit.wikimedia.org/r/435173 (owner: 10Hashar) [14:07:16] (03Merged) 10jenkins-bot: tests: hint at integration-config-qa job [integration/config] - 10https://gerrit.wikimedia.org/r/435173 (owner: 10Hashar) [14:07:17] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4231557 (10hashar) [14:23:48] are there deployments on Monday, May 28th [14:23:50] ? [14:24:13] specifically, swatty ploys? [14:28:18] AndyRussG: https://wikitech.wikimedia.org/wiki/Deployments [14:29:16] thedj: yeah :) but it's a US holiday, and a lot of staff outside the US also might not be around, thought that maybe someone forgot to cancel those.... [14:30:35] it's a us holiday ? for what ? [14:32:54] thedj: https://en.wikipedia.org/wiki/Memorial_Day [14:33:31] ah [14:36:39] 10Beta-Cluster-Infrastructure: deployment-deploy1001 has puppet error - https://phabricator.wikimedia.org/T194927#4231653 (10EddieGP) >>! In T194927#4230615, @Krenair wrote: > why on earth is that host named 1001? it doesn't make sense to use that convention in labs which is eqiad-only True. But previously it w... [14:37:23] (03PS1) 10Hashar: Run mwext-mw-selenium-jessie on PHP 7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/435178 (https://phabricator.wikimedia.org/T195499) [14:37:26] (03PS1) 10Hashar: Migrate MultimediaViewer to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435179 (https://phabricator.wikimedia.org/T183512) [14:38:44] (03CR) 10Hashar: [C: 032] "The composer variants have been switched to PHP 7.0 already." [integration/config] - 10https://gerrit.wikimedia.org/r/435178 (https://phabricator.wikimedia.org/T195499) (owner: 10Hashar) [14:38:46] (03CR) 10Hashar: [C: 032] Migrate MultimediaViewer to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435179 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [14:40:02] ^ greg-g ? [14:40:13] (03Merged) 10jenkins-bot: Run mwext-mw-selenium-jessie on PHP 7.0 [integration/config] - 10https://gerrit.wikimedia.org/r/435178 (https://phabricator.wikimedia.org/T195499) (owner: 10Hashar) [14:40:17] (03Merged) 10jenkins-bot: Migrate MultimediaViewer to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/435179 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [14:48:56] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736#4231684 (10EddieGP) >>! In T187736#4230486, @Krenair wrote: > (and also it should've been being used for things like SSH host key ga... [15:13:27] 10Beta-Cluster-Infrastructure, 10Puppet: deployment-secureredirexperiment puppet error - https://phabricator.wikimedia.org/T191663#4231753 (10EddieGP) DNS entries instance-deployment-secureredirexperiment.deployment-prep.wmflabs.org. and *.secureredirtest.wmflabs.org. can probably go away as well? [15:27:43] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:06] 10MediaWiki-Codesniffer: PHPCS Internal error when using SPDX license expressions like OR - https://phabricator.wikimedia.org/T195429#4231818 (10Umherirrender) `SpdxLicenses::isDeprecatedByIdentifier` does no array index check [15:33:42] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:41:51] (03Abandoned) 10Chad: makerelease: Clone with a depth of 1 [tools/release] - 10https://gerrit.wikimedia.org/r/435072 (owner: 10Chad) [15:42:07] (03CR) 10Chad: [C: 032] makerelease.py: Clean up package dir management, actually return extensions and skins [tools/release] - 10https://gerrit.wikimedia.org/r/435069 (owner: 10Chad) [15:42:40] (03Merged) 10jenkins-bot: makerelease.py: Clean up package dir management, actually return extensions and skins [tools/release] - 10https://gerrit.wikimedia.org/r/435069 (owner: 10Chad) [15:43:10] no_justification hi, wondering when you have time could you push 2.15 so i can begin testing it in a prod like enviroment please? :) [15:44:18] I made a stable-2.15 branch, I guess I gotta build and upload said artifacts :) [15:47:37] no_justification yep :) [15:49:20] (03PS1) 10Chad: makerelease: Kill export() function, clean up patch mgmt a tad [tools/release] - 10https://gerrit.wikimedia.org/r/435186 [15:49:22] (03CR) 10Chad: [C: 032] makerelease: Kill export() function, clean up patch mgmt a tad [tools/release] - 10https://gerrit.wikimedia.org/r/435186 (owner: 10Chad) [15:50:11] (03Merged) 10jenkins-bot: makerelease: Kill export() function, clean up patch mgmt a tad [tools/release] - 10https://gerrit.wikimedia.org/r/435186 (owner: 10Chad) [15:50:15] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [15:53:18] 10Beta-Cluster-Infrastructure: puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts - https://phabricator.wikimedia.org/T194926#4231910 (10EddieGP) 05Open>03Resolved All the hosts that are failing puppet in deployment-prep (as seen on shinken.wmflabs.org) look familiar and seem to... [15:55:53] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review, 10Prometheus-metrics-monitoring, 10User-fgiunchedi: Move deployment-prep redis instances to stretch - https://phabricator.wikimedia.org/T179371#4231914 (10EddieGP) These are two out of four remaining trusty instances in deployment-prep, and... [16:01:16] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<22.22%) [16:30:58] 10Gerrit, 10MediaWiki-Vagrant: git-review fails with "The requested URL /changes/ was not found on this server." - https://phabricator.wikimedia.org/T163242#4232042 (10Tgr) We have since started using polygerrit and this wasn't fixed. Maybe it is some nonstandard configuration used by the WMF gerrit instance? [16:31:40] 10Gerrit, 10MediaWiki-Vagrant: git-review fails with "The requested URL /changes/ was not found on this server." - https://phabricator.wikimedia.org/T163242#3191054 (10Paladox) Hi, which version of git-review are you using? this was fixed in git-review 1.6. [16:32:18] 10Gerrit, 10MediaWiki-Vagrant: git-review fails with "The requested URL /changes/ was not found on this server." - https://phabricator.wikimedia.org/T163242#4232050 (10Tgr) As you can see from the comments, this started happening in 1.25. [16:34:13] 10Gerrit, 10MediaWiki-Vagrant: git-review fails with "The requested URL /changes/ was not found on this server." - https://phabricator.wikimedia.org/T163242#4232067 (10Paladox) Ok, this has been the problem with 1.5 for a while. This is fixed in 1.6 :). There's a few tasks already filed but closed as resolved... [16:39:22] AndyRussG: thanks for the ping. I removed the SF/US centric deploy windows on Monday (left the EU swat window) [16:40:15] greg-g: ah ok thx much! [17:01:40] PROBLEM - Host deployment-secureredirproto is DOWN: CRITICAL - Host Unreachable (10.68.22.127) [17:07:53] (03CR) 10Umherirrender: [C: 031] Revert "Archive PushToWatch" because is extension now active. [integration/config] - 10https://gerrit.wikimedia.org/r/433197 (https://phabricator.wikimedia.org/T194785) (owner: 10Zoranzoki21) [17:09:56] anyone about? [17:11:34] can someone run this on deploy1001.eqiad.wmnet and paste the result, for https://phabricator.wikimedia.org/T194927 ? apt-cache policy npm [17:13:19] gjg@deploy1001:~$ apt-cache policy npm [17:13:22] N: Unable to locate package npm [17:13:27] huh [17:13:32] stretch evidently doesn't have it. I outlined it in the broken stuff on: https://phabricator.wikimedia.org/T192561#4183530 [17:13:33] yeah [17:13:56] ah [17:13:59] npm is very old in debian [17:14:05] I think we should merge T194927 into T192561 [17:14:05] T194927: deployment-deploy1001 has puppet error - https://phabricator.wikimedia.org/T194927 [17:14:06] T192561: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561 [17:14:08] nodejs should include a updated version. [17:15:58] 10Beta-Cluster-Infrastructure: deployment-deploy1001 has puppet error - https://phabricator.wikimedia.org/T194927#4232204 (10Krenair) I suggest we merge this into T192561 which shows the host is still mid-setup and this is a known issue [17:17:44] 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline: Update Debian package of Blubber (0.4.0-1) - https://phabricator.wikimedia.org/T195609#4232208 (10dduvall) [17:26:57] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568#4232221 (10Dzahn) Yes, fully agree. We had already made a very similar plan on IRC, just wasn't sure which specific server to pick. I'll go... [17:31:16] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [17:37:17] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<22.22%) [17:42:09] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4232240 (10Dzahn) >>! In T175288#4227190, @Dzahn wrote: > We'll need the database grants above added or that might block the swi... [17:56:56] no_justification https://gerrit-review.googlesource.com/c/gerrit/+/180850 :) [18:07:16] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [18:12:32] 10Beta-Cluster-Infrastructure, 10Analytics, 10Puppet: deployment-eventlog05 puppet error about missing mysql heartbeat.heartbeat table - https://phabricator.wikimedia.org/T191109#4232297 (10Krenair) [18:13:17] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<11.11%) [18:14:01] PROBLEM - Puppet errors on deployment-webperf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:14:13] 10Project-Admins, 10User-Urbanecm: Create project for WMCZ Tracker - https://phabricator.wikimedia.org/T195610#4232302 (10Urbanecm) [18:17:00] PROBLEM - Puppet errors on deployment-certcentral-testclient is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:17:15] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736#4232319 (10Krenair) According to the audit log, @joe shut it down 10 Jan 2018, 2:51 p.m. [18:18:00] 10Project-Admins, 10User-Luke081515, 10User-Urbanecm: Create project for WMCZ Tracker - https://phabricator.wikimedia.org/T195610#4232321 (10Luke081515) a:03Luke081515 [18:20:41] 10Project-Admins, 10User-Luke081515, 10User-Urbanecm: Create project for WMCZ Tracker - https://phabricator.wikimedia.org/T195610#4232336 (10Luke081515) 05Open>03Resolved I've created #WMCZ-Tracker, please add a good description when you have some time. [18:21:12] 10Project-Admins, 10User-Luke081515, 10User-Urbanecm: Create project for WMCZ Tracker - https://phabricator.wikimedia.org/T195610#4232338 (10Luke081515) p:05Triage>03Normal [18:38:18] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:46:15] 10Deployments, 10Release-Engineering-Team, 10User-greg, 10User-notice: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#4232400 (10Petar.petkovic) [18:50:02] Reedy hi, wondering if you could review https://gerrit.wikimedia.org/r/#/c/435205/ please? :) [18:50:05] (php7.2 fix) [18:50:12] http://php.net/manual/en/migration72.incompatible.php [18:56:03] 10Continuous-Integration-Config, 10ContentTranslation, 10ContentTranslation-Deployments, 10Wikidata, and 3 others: ContentTranslation tests for mwext-testextension-zend tests fails in wmf19 and wmf20 - https://phabricator.wikimedia.org/T110518#4232445 (10Petar.petkovic) [19:10:29] 10Beta-Cluster-Infrastructure: deployment-deploy1001 has puppet error - https://phabricator.wikimedia.org/T194927#4232523 (10EddieGP) [19:10:33] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561#4232526 (10EddieGP) [19:13:56] 10Beta-Cluster-Infrastructure, 10Shinken, 10cloud-services-team, 10Patch-For-Review, 10Puppet: labs-puppetmaster/Labs Puppetmaster HTTPS is UNKNOWN since [...] - https://phabricator.wikimedia.org/T191553#4232546 (10EddieGP) 05Open>03Resolved Per the link in the task description `labs-puppetmaster/Lab... [19:15:29] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736#4232550 (10EddieGP) Apparently the reason was just "unused": https://tools.wmflabs.org/sal/log/AWDgixgYwg13V6286cnS [19:53:41] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568#4232666 (10mmodell) +1 this sounds like a good plan. [20:18:29] (03Draft3) 10Cicalese: Add a dependency on VisualEditor for VEForAll. [integration/config] - 10https://gerrit.wikimedia.org/r/435221 [20:25:20] (03PS1) 10SamanthaNguyen: Archive NewUsersList, RandomFeaturedUser, and RandomUsersWithAvatars [integration/config] - 10https://gerrit.wikimedia.org/r/435278 (https://phabricator.wikimedia.org/T195612) [20:56:34] 10Gerrit, 10Release-Engineering-Team (Someday): Update gerrit to 2.15.2 - https://phabricator.wikimedia.org/T177201#4232828 (10Paladox) [21:41:38] !log cleaned up disk space on deployment-tin . Reenabled jenkins slave. [21:41:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:47:07] 10Phabricator (Upstream), 10Upstream: Phabricator does not let me to move task on multiple workboards at once - https://phabricator.wikimedia.org/T195279#4233010 (10Aklapper) [21:47:09] 10Phabricator (Upstream), 10Upstream, 10User-Matthewrbowker: Ability to move task on multiple workboards on same action - https://phabricator.wikimedia.org/T156953#4233013 (10Aklapper) [21:50:01] 10MediaWiki-Codesniffer: MediaWiki.Classes.UnusedUseStatement.UnusedUse reporting against the wrong file - https://phabricator.wikimedia.org/T195632#4233015 (10Reedy) [22:24:00] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T191051#4233086 (10Jdforrester-WMF) [22:25:20] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Quibble: quibble: with hhvm, could not load extension asset - https://phabricator.wikimedia.org/T195634#4233087 (10hashar) [22:45:41] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Determine a standard way of installing MediaWiki lib/extension dependencies within containers - https://phabricator.wikimedia.org/T193824#4180847 (10Legoktm) I don't have a full picture of what the deployment container stuff is going to look like, so ple... [22:51:04] 10MediaWiki-Codesniffer: MediaWiki.Classes.UnusedUseStatement.UnusedUse reporting against the wrong file - https://phabricator.wikimedia.org/T195632#4233144 (10Legoktm) 05Open>03Invalid PHPCS runs against the merged commit of your patch and master, and not your patch exactly. And the use statement was actual... [22:58:11] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: debian-glue errors out with: hooks/B90lintian: line 21: realpath: command not found - https://phabricator.wikimedia.org/T193906#4233152 (10Dzahn) [22:58:15] 10Continuous-Integration-Infrastructure, 10Operations: Build and upload jenkins-debian-glue_0.18.4-wmf3 for jessie - https://phabricator.wikimedia.org/T193910#4233150 (10Dzahn) 05Open>03Resolved [install1002:~/jenkins-debian-glue] $ export REPREPRO_BASE_DIR=/srv/wikimedia [install1002:~/jenkins-debian-glue... [23:02:39] (03CR) 10Legoktm: [C: 032] Archive NewUsersList, RandomFeaturedUser, and RandomUsersWithAvatars [integration/config] - 10https://gerrit.wikimedia.org/r/435278 (https://phabricator.wikimedia.org/T195612) (owner: 10SamanthaNguyen) [23:03:05] (03CR) 10Legoktm: [C: 032] Revert "Archive PushToWatch" because is extension now active. [integration/config] - 10https://gerrit.wikimedia.org/r/433197 (https://phabricator.wikimedia.org/T194785) (owner: 10Zoranzoki21) [23:03:12] (03CR) 10Legoktm: [C: 032] Do not run mwgate-rake-docker job for Math [integration/config] - 10https://gerrit.wikimedia.org/r/434950 (https://phabricator.wikimedia.org/T162455) (owner: 10Zfilipin) [23:03:30] (03CR) 10Legoktm: [C: 032] Add Edlira to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/434727 (owner: 10Gergő Tisza) [23:03:44] (03CR) 10Legoktm: [C: 032] Add a dependency on VisualEditor for VEForAll. [integration/config] - 10https://gerrit.wikimedia.org/r/435221 (owner: 10Cicalese) [23:04:18] (03Merged) 10jenkins-bot: Archive NewUsersList, RandomFeaturedUser, and RandomUsersWithAvatars [integration/config] - 10https://gerrit.wikimedia.org/r/435278 (https://phabricator.wikimedia.org/T195612) (owner: 10SamanthaNguyen) [23:05:09] (03Merged) 10jenkins-bot: Revert "Archive PushToWatch" because is extension now active. [integration/config] - 10https://gerrit.wikimedia.org/r/433197 (https://phabricator.wikimedia.org/T194785) (owner: 10Zoranzoki21) [23:05:10] (03Merged) 10jenkins-bot: Do not run mwgate-rake-docker job for Math [integration/config] - 10https://gerrit.wikimedia.org/r/434950 (https://phabricator.wikimedia.org/T162455) (owner: 10Zfilipin) [23:05:12] (03Merged) 10jenkins-bot: Add Edlira to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/434727 (owner: 10Gergő Tisza) [23:05:14] (03Merged) 10jenkins-bot: Add a dependency on VisualEditor for VEForAll. [integration/config] - 10https://gerrit.wikimedia.org/r/435221 (owner: 10Cicalese) [23:05:15] 10Phabricator (Upstream), 10Upstream: Phabricator does not let me to add tag to task and move it on tag's workboard at once - https://phabricator.wikimedia.org/T195638#4233168 (10Dvorapa) [23:08:08] !log deployed https://gerrit.wikimedia.org/r/435278 https://gerrit.wikimedia.org/r/433197 https://gerrit.wikimedia.org/r/434950 https://gerrit.wikimedia.org/r/434727 https://gerrit.wikimedia.org/r/435221 [23:08:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:20:12] 10Phabricator (Upstream), 10Upstream: Phabricator does not let me to add tag to task and move it on tag's workboard at once - https://phabricator.wikimedia.org/T195638#4233197 (10Peachey88) [23:20:16] 10Phabricator (Upstream), 10Upstream, 10User-Matthewrbowker: Ability to move task on multiple workboards on same action - https://phabricator.wikimedia.org/T156953#4233199 (10Peachey88) [23:21:47] 10Phabricator (Upstream), 10Upstream: Phabricator does not let me to add tag to task and move it on tag's workboard at once - https://phabricator.wikimedia.org/T195638#4233200 (10Dvorapa) 05duplicate>03Open These two seems different