[00:20:03] Project beta-update-databases-eqiad build #39611: 04STILL FAILING in 3.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39611/ [01:20:06] Project beta-update-databases-eqiad build #39612: 04STILL FAILING in 5.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39612/ [01:45:07] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10Legoktm) [01:46:30] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10Legoktm) [02:05:27] 10MediaWiki-Releasing, 10Reference Previews, 10MW-1.35-release: Bundle Popups extension with MediaWiki - https://phabricator.wikimedia.org/T231797 (10Legoktm) Popups is a pain for redistributors because it distributes minified, compiled JavaScript without the accompanying tools to do so. [02:20:03] Project beta-update-databases-eqiad build #39613: 04STILL FAILING in 3.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39613/ [03:20:03] Project beta-update-databases-eqiad build #39614: 04STILL FAILING in 3.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39614/ [04:20:04] Project beta-update-databases-eqiad build #39615: 04STILL FAILING in 3.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39615/ [05:20:07] Project beta-update-databases-eqiad build #39616: 04STILL FAILING in 6.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39616/ [06:20:03] Project beta-update-databases-eqiad build #39617: 04STILL FAILING in 3.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39617/ [06:29:47] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: decommission phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T238957 (10Marostegui) I have removed the users from the database [07:20:03] Project beta-update-databases-eqiad build #39618: 04STILL FAILING in 3.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39618/ [08:14:22] PROBLEM - Free space - all mounts on deployment-logstash03 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash03.diskspace.root.byte_percentfree (<100.00%) [08:20:03] Project beta-update-databases-eqiad build #39619: 04STILL FAILING in 3.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39619/ [08:53:07] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Testing, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-CentralAuth, and 2 others: Login to en.wikipedia.beta.wmflabs.org sometimes fails with `There seems to be a problem with your login sesion` - https://phabricator.wikimedia.org/T243123 (10zeljko... [09:20:07] Project beta-update-databases-eqiad build #39620: 04STILL FAILING in 6.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39620/ [10:20:03] Project beta-update-databases-eqiad build #39621: 04STILL FAILING in 3.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39621/ [11:20:03] Project beta-update-databases-eqiad build #39622: 04STILL FAILING in 3.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39622/ [11:23:53] 10Continuous-Integration-Config, 10Language-Team (Language-2020-January-March), 10Patch-For-Review, 10Unplanned-Sprint-Work, 10affects-translatewiki.net: translatewiki-rake-docker failing: Gem::InstallError: gettext requires Ruby version >= 2.5.0. - https://phabricator.wikimedia.org/T242339 (10Nikerabbit) [12:11:37] 10Continuous-Integration-Config: Upgrade bundler to 2.x in rake-docker jobs - https://phabricator.wikimedia.org/T243280 (10Nikerabbit) [12:20:03] Project beta-update-databases-eqiad build #39623: 04STILL FAILING in 3.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39623/ [13:20:03] Project beta-update-databases-eqiad build #39624: 04STILL FAILING in 3.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39624/ [13:29:37] 10Gerrit, 10Operations: setup/install gerrit1001 - https://phabricator.wikimedia.org/T231046 (10Marostegui) [14:20:07] Project beta-update-databases-eqiad build #39625: 04STILL FAILING in 6.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39625/ [14:34:30] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10MoritzMuehlenhoff) p:05Triage→03Normal a:03jbond [14:58:38] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10jbond) >>! In T243226#5817998, @Krenair wrote: > am guessing this is just us needing to get a new puppetmaster with buster instead of stretch I added the puppet-master version 5 packa... [15:04:45] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Security Concept Review, 10Security-Team, 10Needs Discussion: Security Concept Review For new CI - https://phabricator.wikimedia.org/T240943 (10chasemp) [15:33:24] Project beta-update-databases-eqiad build #39626: 15ABORTED in 13 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39626/ [15:54:47] Project beta-update-databases-eqiad build #39627: 15ABORTED in 21 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39627/ [16:00:46] ukwiki in beta cluster has interesting issues [16:01:01] the system just hangs when it reaches "...echo_event table does not contain event_page_namespace field." there [16:03:34] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: logspam.pl: Shorten paths and include fatals - https://phabricator.wikimedia.org/T242252 (10brennen) 05Open→03Resolved [16:06:40] Amir1: Run update.php manually against it? [16:06:51] yeah [16:07:17] Project beta-update-databases-eqiad build #39628: 15ABORTED in 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39628/ [16:09:42] 10Phabricator, 10Security-Team, 10PM: Create 'secteam-discussion' tag - https://phabricator.wikimedia.org/T242765 (10chasemp) [16:09:55] Yippee, build fixed! [16:09:55] Project beta-update-databases-eqiad build #39629: 09FIXED in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/39629/ [16:10:40] 10Phabricator, 10Security-Team, 10PM: Create 'secteam-discussion' tag - https://phabricator.wikimedia.org/T242765 (10chasemp) thinking about the concerns presented here and keeping the scope of this contained I decided to rename to #secteam-discussion. I think this will contain the proliferation but also le... [16:11:34] 10Beta-Cluster-Infrastructure, 10Wikimedia-General-or-Unknown: session_name(): Cannot change session name when headers already sent in CommonSettings.php on line 510 - https://phabricator.wikimedia.org/T243219 (10Reedy) Yeah, they're still there now it's passing https://integration.wikimedia.org/ci/job/beta-u... [16:12:18] 10Phabricator, 10Security-Team, 10PM: Create 'secteam-discussion' tag - https://phabricator.wikimedia.org/T242765 (10chasemp) Thanks for discussing @aklapper and the history @Peachey88 [16:13:18] yup, fixed now. Thanks Reedy for the hint [16:13:25] cool [16:13:31] it seems to happen occasionally :) [16:39:51] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Google-Code-in-2019, and 3 others: Stop using jsonlint and instead use eslint-plugin-json for the linting - https://phabricator.wikimedia.org/T220036 (10Jdforrester-WMF) [16:40:54] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Google-Code-in-2019, and 3 others: Stop using jsonlint and instead use eslint-plugin-json for the linting - https://phabricator.wikimedia.org/T220036 (10Jdforrester-WMF) [16:47:42] 10MediaWiki-Codesniffer, 10Wikidata: Release a new version of WikibaseCodeSniffer - https://phabricator.wikimedia.org/T243296 (10MaxSem) [16:47:55] 10MediaWiki-Codesniffer, 10Wikidata: Release a new version of WikibaseCodeSniffer - https://phabricator.wikimedia.org/T243296 (10MaxSem) [17:00:53] 10Beta-Cluster-Infrastructure, 10Wikimedia-General-or-Unknown: session_name(): Cannot change session name when headers already sent in CommonSettings.php on line 510 - https://phabricator.wikimedia.org/T243219 (10Daimona) Hmmmm interesting. I can think of two main reasons: - Some other PHP error/warning/notic... [17:03:01] 10Beta-Cluster-Infrastructure, 10Wikimedia-General-or-Unknown: session_name(): Cannot change session name when headers already sent in CommonSettings.php on line 510 - https://phabricator.wikimedia.org/T243219 (10Reedy) p:05Triage→03Low [17:05:20] 10MediaWiki-Codesniffer, 10Wikidata: Release a new version of WikibaseCodeSniffer - https://phabricator.wikimedia.org/T243296 (10Daimona) CC @Ladsgroup as releaser of the last two versions and (AFAICS) most active maintainer. [17:24:24] (03CR) 10Jforrester: "Thank you." [integration/config] - 10https://gerrit.wikimedia.org/r/566001 (https://phabricator.wikimedia.org/T211784) (owner: 10Addshore) [17:25:06] (03PS3) 10Jforrester: Rename labmon1001.eqiad.wmnet to cloudmetrics1001.eqiad.wmnet [integration/config] - 10https://gerrit.wikimedia.org/r/565766 (owner: 10Paladox) [17:26:31] (03PS4) 10Jforrester: jjb: Rename labmon1001.eqiad.wmnet to cloudmetrics1001.eqiad.wmnet [integration/config] - 10https://gerrit.wikimedia.org/r/565766 (owner: 10Paladox) [18:16:46] 10Project-Admins: Proposal for global tag: "Needs Discussion" - https://phabricator.wikimedia.org/T226671 (10Aklapper) 05duplicate→03Declined [18:19:32] it certainly needed discussion :P [18:22:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Remove record-node-stats publisher - https://phabricator.wikimedia.org/T243305 (10dduvall) [18:23:18] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Remove record-node-stats publisher - https://phabricator.wikimedia.org/T243305 (10dduvall) p:05Triage→03Normal [18:27:03] (03PS1) 10Dduvall: Revert "Publish job duration for labeled nodes to labmon1001" [integration/config] - 10https://gerrit.wikimedia.org/r/566332 (https://phabricator.wikimedia.org/T243305) [18:28:15] (03CR) 10Dduvall: [C: 04-1] "The propose fix works for me, but I would prefer to remove the publisher entirely. See I76d6f155c5d07ba660c86f186e7c400a393e36e7" [integration/config] - 10https://gerrit.wikimedia.org/r/565766 (owner: 10Paladox) [18:33:56] 10Phabricator, 10User-LokalProfil, 10WMSE (IT): Move subproject into a user project - https://phabricator.wikimedia.org/T240710 (10Lokal_Profil) >>! In T240710#5801901, @mmodell wrote: > The script to move projects doesn't deal well with trees of projects. It seems that the subprojects under #magnus_sälgö ar... [18:36:50] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: decommission phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T238957 (10Dzahn) Thanks Manuel !:) Production IPs removed from DNS. [18:40:24] 10Release-Engineering-Team, 10serviceops: decommission phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T238957 (10Dzahn) a:05Dzahn→03Jclark-ctr [18:42:05] 10Release-Engineering-Team, 10serviceops: decommission phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T238957 (10Dzahn) This server has been temporarily assigned in T215335 and used in T221389. Giving it back to the pool of spares after it has served its purpose.. It has been originally purchased in... [18:42:15] 10Phabricator, 10User-LokalProfil, 10WMSE (IT): Move subproject into a user project - https://phabricator.wikimedia.org/T240710 (10mmodell) @Lokal_Profil I'll attempt to move the project today if I can. Will report back after it's done. [18:48:15] 10MediaWiki-Codesniffer, 10Wikidata: Release a new version of WikibaseCodeSniffer - https://phabricator.wikimedia.org/T243296 (10thiemowmde) Note: I was the driving force when we introduced the WikibaseCodeSniffer rule set. The main motivations back then: 1. Being able to experiment with additional, custom sni... [18:55:20] (03PS1) 10Thcipriani: make-deploy-notes: account for gerrit json pagination [tools/release] - 10https://gerrit.wikimedia.org/r/566337 (https://phabricator.wikimedia.org/T231956) [18:55:44] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Release-Tools, 10Patch-For-Review: makedeploynotes.py omits some changes for no apparent reason - https://phabricator.wikimedia.org/T231956 (10thcipriani) p:05Triage→03Normal a:03thcipriani [18:59:45] 10Beta-Cluster-Infrastructure, 10MediaWiki-General: PHP Warning: socket_sendto(): Host lookup failed [-10001]: Unknown host in /srv/mediawiki/php-master/vendor/liuggio/statsd-php-client/src/Liuggio/StatsdClient/Sender/SocketSender.php on line 47 - https://phabricator.wikimedia.org/T243322 (10MarcoAurelio) [19:01:59] hauskatze: That's kind of a dupe [19:02:04] We have a few tasks for very similar stuff [19:02:10] But no direct task to dupe it against [19:02:20] 10Beta-Cluster-Infrastructure, 10MediaWiki-General: PHP Warning: socket_sendto(): Host lookup failed [-10001]: Unknown host in /srv/mediawiki/php-master/vendor/liuggio/statsd-php-client/src/Liuggio/StatsdClient/Sender/SocketSender.php on line 47 - https://phabricator.wikimedia.org/T243322 (10MarcoAurelio) [19:03:10] Reedy: hmm, sorry then. I did a brief search before [19:03:33] There's various of these vague "Host lookup failed" ones. Some linking to captcha stuff [19:03:45] 10Beta-Cluster-Infrastructure, 10MediaWiki-General: PHP Warning: socket_sendto(): Host lookup failed [-10001]: Unknown host [...] SocketSender.php - https://phabricator.wikimedia.org/T243322 (10MarcoAurelio) [19:04:33] (03CR) 10Paladox: [C: 03+1] Revert "Publish job duration for labeled nodes to labmon1001" [integration/config] - 10https://gerrit.wikimedia.org/r/566332 (https://phabricator.wikimedia.org/T243305) (owner: 10Dduvall) [19:04:37] (03Abandoned) 10Paladox: jjb: Rename labmon1001.eqiad.wmnet to cloudmetrics1001.eqiad.wmnet [integration/config] - 10https://gerrit.wikimedia.org/r/565766 (owner: 10Paladox) [19:04:56] there's also "[_field_stats] endpoint is deprecated! Use [_field_caps] instead or run a min/max aggregations on the desired fields." [19:05:15] dcausse: hi, I can't remember, you worked on ElasticSearch? [19:06:55] I have T241485 for that [19:06:56] T241485: [_field_stats] endpoint is deprecated! Use [_field_caps] instead or run a min/max aggregations on the desired fields. - https://phabricator.wikimedia.org/T241485 [19:09:57] hauskatze: I believe this is the elasticsearch for logstash itself which I don't maintain [19:10:33] ah, okay dcausse , thanks :) [19:11:02] perhaps bumping kibana to a newer version will help? [19:12:15] (03CR) 10Jforrester: [C: 03+1] Revert "Publish job duration for labeled nodes to labmon1001" [integration/config] - 10https://gerrit.wikimedia.org/r/566332 (https://phabricator.wikimedia.org/T243305) (owner: 10Dduvall) [19:12:47] I guess that requires working on in the deployment-logstash03 instance [19:13:02] I do not feel confident doing that [19:13:14] *and I do not [...] [19:15:09] hauskatze: thanks for the ping but sorry I can't help much on this one, sorry :( [19:15:30] dcausse: no problem :) and thanks anyway [19:16:31] !log updating 105 jobs affected by https://gerrit.wikimedia.org/r/c/integration/config/+/566332 [19:16:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:16:59] marxarelli: Oh, hah. Will not push it for you, then. :-) [19:17:24] James_F: oh. whoops. did you already update the jobs? [19:17:26] (03PS2) 10Jforrester: Revert "Publish job duration for labeled nodes to labmon1001" [integration/config] - 10https://gerrit.wikimedia.org/r/566332 (https://phabricator.wikimedia.org/T243305) (owner: 10Dduvall) [19:17:36] marxarelli: Yeah. [19:17:47] marxarelli: Well, the wikibase and quibble ones. [19:18:19] Didn't update mediawiki-quibble-api-testing-vendor-mysql-php72-docker, mediawiki-selenium-integration-docker, mwext-phpunit-coverage-patch-docker, or parsoidsvc-parsertests-docker yet. [19:18:47] alright. i'll go ahead and update all the affected ones just in case [19:19:09] (03PS1) 10Thiemo Kreuz (WMDE): Improve performance of InArrayUsage sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/566343 [19:21:52] (03CR) 10Dduvall: [C: 03+2] "105 affected jobs updated" [integration/config] - 10https://gerrit.wikimedia.org/r/566332 (https://phabricator.wikimedia.org/T243305) (owner: 10Dduvall) [19:22:48] (03Merged) 10jenkins-bot: Revert "Publish job duration for labeled nodes to labmon1001" [integration/config] - 10https://gerrit.wikimedia.org/r/566332 (https://phabricator.wikimedia.org/T243305) (owner: 10Dduvall) [19:36:01] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: train-deploy-notes Jenkins job fails in conjunction with branch.py - https://phabricator.wikimedia.org/T243330 (10brennen) [20:18:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Remove record-node-stats publisher - https://phabricator.wikimedia.org/T243305 (10Jdforrester-WMF) 05Open→03Resolved [20:20:17] thcipriani: gerrit1002 is now available with role(gerrit), no puppet errors, service name: gerrit-test server name: gerrit1002 https://gerrit-test.wikimedia.org/r/ [20:20:24] pheew :p [20:20:27] \o/ [20:20:45] mutante: awesome [20:22:53] :) https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit1002.wikimedia.org [20:25:34] 10Gerrit, 10Operations, 10vm-requests, 10Patch-For-Review: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) https://gerrit.wikimedia.org/r/q/topic:%22gerrit-test%22+(status:open%20OR%20status:merged) https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit1002... [20:29:56] 10Gerrit, 10Operations, 10vm-requests, 10Patch-For-Review: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) The VM is now usable. It has the `role(gerrit)` on it and no more puppet errors. It uses its own service name/IP: https://gerrit-test.wikimedia.org Shell acces... [20:30:02] (03CR) 10Jforrester: [C: 03+2] "Fun." [tools/release] - 10https://gerrit.wikimedia.org/r/566337 (https://phabricator.wikimedia.org/T231956) (owner: 10Thcipriani) [20:30:43] (03Merged) 10jenkins-bot: make-deploy-notes: account for gerrit json pagination [tools/release] - 10https://gerrit.wikimedia.org/r/566337 (https://phabricator.wikimedia.org/T231956) (owner: 10Thcipriani) [20:31:23] 10Gerrit, 10Operations, 10vm-requests, 10Patch-For-Review: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) The mysql user has also been made configurable (along with backups / monitoring) and it is using: ` 104 hostname = m2-master.eqiad.wmnet 105 database... [20:33:38] 10Gerrit, 10Operations, 10vm-requests, 10Patch-For-Review: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) ` 94 heapLimit = 5g 95 slave = false 116 canonicalWebUrl = https://gerrit-test.wikimedia.org/r 218 [sshd] 219 listenAddress = gerrit-test.wi... [20:34:12] 10Gerrit, 10Operations, 10vm-requests, 10Patch-For-Review: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) 05Open→03Resolved a:03Dzahn [20:34:16] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review: Upgrade to Gerrit 2.16.13 - https://phabricator.wikimedia.org/T200739 (10Dzahn) [20:41:54] 10Gerrit, 10Operations, 10vm-requests, 10Patch-For-Review: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) The gerrit acmechief TLS cert has been updated to contain "gerrit-test" in addition to gerrit and gerrit-replica. The "gerrit-new" name has been removed from it.... [20:47:31] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Release-Tools, 10Patch-For-Review: makedeploynotes.py omits some changes if the repo has more than 100 changes in a given branch (failing to follow pagination) - https://phabricator.wikimedia.org/T231956 (10Jdforrester-WMF) [20:47:37] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Release-Tools, 10Patch-For-Review: makedeploynotes.py omits some changes if the repo has more than 100 changes in a given branch (failing to follow pagination) - https://phabricator.wikimedia.org/T231956 (10Jdforrester-WMF) Resolved? [20:54:29] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Security Concept Review, 10Security-Team, 10SecTeam Discussion: Security Concept Review For new CI - https://phabricator.wikimedia.org/T240943 (10chasemp) The scope of a successful conceptual assessment with associated early risk reasoning is dep... [20:54:58] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Security Concept Review, 10Security-Team, 10SecTeam Discussion: Security Concept Review For new CI - https://phabricator.wikimedia.org/T240943 (10chasemp) 05Open→03Stalled [20:55:01] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Create a threat model for new CI - https://phabricator.wikimedia.org/T240679 (10chasemp) [20:57:19] <3 mutante [20:57:26] https://gerrit-test.wikimedia.org/r/ works \o/ [20:58:06] paladox: indeed. acmechief fixed [20:58:17] paladox: this meant the prod gerrit cert was affected as well..btw [20:58:29] gerrit-test was added to it and gerrit-new was removed [20:58:36] and it applied to ALL gerrit servers at the same time [20:58:37] yup, at least it renewed the cert early :P [20:58:44] heh, yes [20:58:54] at least nothing went wrong with renewing it in prod [20:58:56] yea [20:59:05] yup [20:59:06] now we just need to rsync the data from gerrit1001 -> gerrit1002 [20:59:41] cc thcipriani ^ [20:59:50] 10Phabricator, 10Security-Team, 10Documentation, 10Security: Set Security Issue Task Type as default for Security reporting - https://phabricator.wikimedia.org/T240605 (10chasemp) [21:00:26] ok, i am checking the rsync part [21:00:40] :) [21:00:56] 10Phabricator, 10Security-Team, 10Documentation, 10Security: Set Security Issue Task Type as default for Security reporting - https://phabricator.wikimedia.org/T240605 (10chasemp) 05Open→03Resolved related to {T242018} but not dependent on. So far no more stragglers so I'm resolving. [21:03:05] paladox: not going to work yet. gerrit1001 only allows... gerrit1001 right now [21:03:22] ok, so we only need to change some params, right? [21:03:54] i think we already did the work to un-bundle rsync stuff from "active_server" ? [21:04:04] or was that phab.. heh [21:04:34] i remember we did some work for this. [21:04:42] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Release-Tools, 10Patch-For-Review: makedeploynotes.py omits some changes if the repo has more than 100 changes in a given branch (failing to follow pagination) - https://phabricator.wikimedia.org/T231956 (10thcipriani) 05Open→03Resolve... [21:04:43] yea..same [21:05:10] https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/gerrit/migration.pp [21:06:35] that class is made for a different purpose [21:06:36] 10Phabricator, 10Security-Team: Enforce PermanentlyPrivate tag as on Security Issue type and #security tagged issues - https://phabricator.wikimedia.org/T242638 (10chasemp) a:05chasemp→03mmodell tossing to @mmodell bc I think all that's left to do is deploy. Thanks again sir! [21:06:45] it's when you move gerrit to a new host [21:06:51] and apply this first [21:09:11] paladox: i think that stuff we did was phabricator :) this still does "if $::fqdn == $dst_host" [21:09:44] also we confuse ourselves like every time because this installs rsync server if fqdn == dst_host [21:10:14] normally i expect the rsyncd on the source and pull from it [21:12:21] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Quibble: Quibble should inject php-invoker into MediaWiki core's composer dependencies at run time - https://phabricator.wikimedia.org/T243324 (10Jdforrester-WMF) [21:13:59] oh [21:14:30] switching this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/566362/1/hieradata/role/common/gerrit.yaml [21:14:39] means rsyncd setup gets _removed_ on 2001 [21:14:42] until you switch it back [21:14:58] because this still does the "smart" trick of comparing to FQDN [21:15:21] which we removed in other places when we wanted to be able to have a temp. 3rd server and not just exactly 2 [21:15:36] https://puppet-compiler.wmflabs.org/compiler1003/20496/gerrit2001.wikimedia.org/ [21:15:54] i mean.. it also installs it again if you switch back.. but it's not perfect [21:16:19] yup [21:16:53] gerrit::server::rsync_dst_host is used by the migration host. [21:17:01] *file [21:17:41] yea.. except i don't want it to remove stuff on 2001.. ideally [21:18:36] paladox: also isn't it confusing that it's "if $::fqdn == $dst_host {" [21:18:55] followed by class { '::rsync::server': } [21:19:34] yea.. look: [21:19:35] https://puppet-compiler.wmflabs.org/compiler1003/20496/gerrit1002.wikimedia.org/ [21:19:35] yeh, but "remove stuff on 2001", you mean the rsync config, or? [21:19:40] this installs rsyncd on 1002 [21:19:47] ah [21:19:48] who said we need that [21:19:50] we don't [21:19:56] we want 1001 to allow 1002 [21:19:57] that's all [21:20:08] yup [21:20:14] so this whole thing is backwards [21:20:28] but i also thought that last time we touched it for actual gerrit migration [21:20:35] and yet here we are [21:21:19] so we DID push _from_ the prod server [21:21:19] yeh [21:21:21] to others [21:21:24] and not pull from it [21:21:41] unlike in other cases where we do the exact opposite [21:21:45] gerrit1001 to gerrit2001 i think? [21:21:54] not sure if we did it for the cobalt migration though [21:25:26] either "dst_host" needs to become "dst_hosts" and an array ...and compare if fqdn is in it... [21:25:35] or disable puppet on 2001 temp [21:28:31] I remember thinking the whole thing was backwards during the cobalt migration, but it makes sense once you squint at it. No real risk to prod data from rsyncs run amok this way. [21:34:49] yea. i'll turn "dst_host" into "dst_hosts" array [21:41:11] https://gerrit.wikimedia.org/r/c/operations/puppet/+/566367 workshttps://puppet-compiler.wmflabs.org/compiler1001/20498/ [21:44:04] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T233864 (10brennen) [21:45:23] gerrit1002 is getting the rsyncd now [21:46:09] cc thcipriani ^ [21:46:13] mutante thank you! [21:48:46] !log gerrit - rsyncing git data from gerrit1001 to gerrit1002 (T239151) [21:48:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:48:49] T239151: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 [22:00:32] paladox: git and plugin data synced but not with --delete [22:00:56] ok, i guess we *should* try --delete [22:01:18] and then there is review_site [22:02:49] review_site i think we can just run init, unless we want to rsync over it. [22:03:17] but the data can be regenerated (all that mattered was that the git data + other non regenerating data be synced over) [22:03:29] (e.g index/cache) [22:04:22] in our docs from the migration we said to rsync it too [22:04:47] ok [22:09:48] !log gerrit - rsyncing 'git' and 'plugin' data dirs and /var/lib/gerrit2/review_site/ from gerrit1001 to gerrit1002 WITH --delete T239151 [22:09:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:09:50] T239151: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 [22:10:51] !log deployment-prep added jbond as admin, T243226 [22:10:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:10:53] T243226: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 [22:10:59] bloody hexchat keeps randomly leaving channels [22:11:39] heh [22:11:47] paladox: all done [22:11:49] Textual should come to windows :P [22:11:53] mutante thank you! [22:12:15] thcipriani ^ (guess we're ready to test the 2.16 upgrade now :)) [22:15:07] \o/ [22:20:29] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T233864 (10brennen) [22:27:15] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Quibble: Quibble should inject php-invoker into MediaWiki core's composer dependencies at run time - https://phabricator.wikimedia.org/T243324 (10Krinkle) Does the "You should really fix these slo... [23:02:42] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Krenair) >>! In T243226#5819512, @jbond wrote: > however the bigger problem is the puppetdb server. this will need to be rebuilt on buster as it is not a simple task to backport the... [23:02:51] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Krenair) a:05jbond→03Krenair [23:22:47] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Tag a new release of scap - https://phabricator.wikimedia.org/T243358 (10LarsWirzenius) [23:27:31] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap: Make scap skip restarting php-fpm when using --force - https://phabricator.wikimedia.org/T243009 (10LarsWirzenius) a:03LarsWirzenius [23:37:50] 10MediaWiki-Codesniffer, 10Wikidata: Release a new version of WikibaseCodeSniffer - https://phabricator.wikimedia.org/T243296 (10Legoktm) If there's still value in having separate Wikibase configuration (I don't think so, but that's just me), then I think we could ship it inside the MediaWiki-Codesniffer repo... [23:48:45] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Krenair) Well, my new puppetdb instance seems to not be working very well yet: ` root@deployment-puppetmaster03:/var/lib/git/operations/puppet(production u+14)# pu... [23:52:47] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Krenair) rCLIPec5f9c1645d2eadc8db259755bf163c69e0409d6 Reloaded apache2 on deployment-puppetmaster03 [23:59:50] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Update parsoid image and helm charts for php - https://phabricator.wikimedia.org/T242259 (10jeena) @Dzahn Thanks for the response. So parsoid is using a standalone daemon, right?