[02:25:32] Project beta-scap-eqiad build #318337: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318337/ [02:31:41] Project beta-scap-eqiad build #318338: 04STILL FAILING in 5 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318338/ [02:37:31] Project beta-scap-eqiad build #318339: 04STILL FAILING in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318339/ [02:47:32] Project beta-scap-eqiad build #318340: 04STILL FAILING in 3 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318340/ [02:57:29] Project beta-scap-eqiad build #318341: 04STILL FAILING in 3 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318341/ [03:07:31] Project beta-scap-eqiad build #318342: 04STILL FAILING in 3 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318342/ [03:17:33] Project beta-scap-eqiad build #318343: 04STILL FAILING in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318343/ [03:27:21] Project beta-scap-eqiad build #318344: 04STILL FAILING in 3 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318344/ [03:37:24] Project beta-scap-eqiad build #318345: 04STILL FAILING in 3 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318345/ [03:47:27] Project beta-scap-eqiad build #318346: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318346/ [03:57:23] Project beta-scap-eqiad build #318347: 04STILL FAILING in 3 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318347/ [04:07:32] Project beta-scap-eqiad build #318348: 04STILL FAILING in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318348/ [04:17:32] Project beta-scap-eqiad build #318349: 04STILL FAILING in 3 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318349/ [04:27:34] Project beta-scap-eqiad build #318350: 04STILL FAILING in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318350/ [04:37:36] Project beta-scap-eqiad build #318351: 04STILL FAILING in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318351/ [04:47:25] Project beta-scap-eqiad build #318352: 04STILL FAILING in 3 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318352/ [04:57:26] Project beta-scap-eqiad build #318353: 04STILL FAILING in 3 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318353/ [05:07:31] Project beta-scap-eqiad build #318354: 04STILL FAILING in 3 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318354/ [05:17:15] Project beta-scap-eqiad build #318355: 04STILL FAILING in 3 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318355/ [05:27:13] Project beta-scap-eqiad build #318356: 04STILL FAILING in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318356/ [05:37:16] Project beta-scap-eqiad build #318357: 04STILL FAILING in 3 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318357/ [05:47:13] Project beta-scap-eqiad build #318358: 04STILL FAILING in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318358/ [05:57:14] Project beta-scap-eqiad build #318359: 04STILL FAILING in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318359/ [06:01:24] Project beta-scap-eqiad build #318360: 04STILL FAILING in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318360/ [06:04:39] 10Release-Engineering-Team, 10MediaWiki-Docker, 10Developer Productivity, 10User-brennen, 10User-kostajh: mediawiki-docker test feature - https://phabricator.wikimedia.org/T262369 (10jeena) We could do task T262370 and update the instructions, while keeping @Addshore 's idea in mind for after we accompli... [06:09:16] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10MediaWiki-Docker, 10Developer Productivity, and 3 others: Command-line wrapper for interacting with core's docker-compose stack - https://phabricator.wikimedia.org/T246111 (10jeena) [06:09:18] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10MediaWiki-Docker, 10Developer Productivity, and 2 others: Creation of docker-compose.override.yml for Linux users - https://phabricator.wikimedia.org/T262363 (10jeena) 05Open→03Invalid My mistake, I did not r... [06:14:04] Project beta-scap-eqiad build #318361: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318361/ [06:19:36] Project beta-scap-eqiad build #318362: 04STILL FAILING in 4 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318362/ [06:27:28] Project beta-scap-eqiad build #318363: 04STILL FAILING in 3 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318363/ [06:29:00] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10greg) And the spam of beta scap failures again tonight: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ [06:39:25] Project beta-scap-eqiad build #318364: 04STILL FAILING in 5 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318364/ [06:47:42] Project beta-scap-eqiad build #318365: 04STILL FAILING in 3 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318365/ [06:50:18] 10LibUp, 10git-protocol-v2: Have libup use git procotol v2 - https://phabricator.wikimedia.org/T263595 (10hashar) Awesome, thank you ;) [06:57:38] Project beta-scap-eqiad build #318366: 04STILL FAILING in 3 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318366/ [07:13:40] Yippee, build fixed! [07:13:40] Project beta-scap-eqiad build #318367: 09FIXED in 9 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318367/ [07:26:00] 10Gerrit: Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" - https://phabricator.wikimedia.org/T263293 (10hashar) Looking at @xSavitar recent accesses: sshd_log.2020-09-15.gz:[2020-09-15T22:01:13.725+0000] 68480692 [SshCommandDestroy-0] d3r1ck01 a/2842 git-upload-pack./m... [07:28:30] 10Gerrit: Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" - https://phabricator.wikimedia.org/T263293 (10hashar) I thought it might be the same as T249410, but that one had a meaningful error message: ` java.net.SocketTimeoutException: waitForCondition(Window[server/remo... [07:52:10] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10dom_walden) [07:57:38] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10dom_walden) [08:06:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Fundraising-Backlog: JJB / zuul support for upgrading payments-wiki to php7.3 and MW 1.35 - https://phabricator.wikimedia.org/T261777 (10hashar) Eventually... [08:19:53] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10zeljkofilipin) p:05Triage→03High [08:22:45] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10zeljkofilipin) Logging in fails even via the API. From [[ https://integration.wikimedia.org/ci/view/Selenium/job/sele... [08:25:57] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10zeljkofilipin) There was a similar problem early in 2020: {T243123}. [08:36:39] (03PS1) 10Hashar: Had dedicated pipelines for fundraising/REL1_35 [integration/config] - 10https://gerrit.wikimedia.org/r/629320 (https://phabricator.wikimedia.org/T261777) [08:44:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Fundraising-Backlog, 10Patch-For-Review: JJB / zuul support for upgrading payments-wiki to php7.3 and MW 1.35 - https://phabricator.wikimedia.org/T261777 (... [08:54:05] (03PS1) 10Hashar: jjb: support DonationInterface fundraising/REL1_35 branch [integration/config] - 10https://gerrit.wikimedia.org/r/629322 (https://phabricator.wikimedia.org/T261777) [09:01:45] (03CR) 10Hashar: [C: 03+2] "Deployed and the job now properly fetches fundraising/REL1_35 for DonationInterface." [integration/config] - 10https://gerrit.wikimedia.org/r/629322 (https://phabricator.wikimedia.org/T261777) (owner: 10Hashar) [09:02:46] (03Merged) 10jenkins-bot: jjb: support DonationInterface fundraising/REL1_35 branch [integration/config] - 10https://gerrit.wikimedia.org/r/629322 (https://phabricator.wikimedia.org/T261777) (owner: 10Hashar) [09:16:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Fundraising-Backlog, 10Patch-For-Review: JJB / zuul support for upgrading payments-wiki to php7.3 and MW 1.35 - https://phabricator.wikimedia.org/T261777 (... [10:07:00] (03PS1) 10Phedenskog: layout: Add support for CI test for performance/mobile-synthetic-monitoring-tests [integration/config] - 10https://gerrit.wikimedia.org/r/629340 [10:07:59] (03CR) 10jerkins-bot: [V: 04-1] layout: Add support for CI test for performance/mobile-synthetic-monitoring-tests [integration/config] - 10https://gerrit.wikimedia.org/r/629340 (owner: 10Phedenskog) [10:23:52] (03PS2) 10Phedenskog: layout: Add support for CI test for mobile phone performance tests. [integration/config] - 10https://gerrit.wikimedia.org/r/629340 [10:25:28] Project beta-scap-eqiad build #318388: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318388/ [10:28:39] (03PS2) 10Jforrester: Add dedicated pipelines for fundraising/REL1_35 [integration/config] - 10https://gerrit.wikimedia.org/r/629320 (https://phabricator.wikimedia.org/T261777) (owner: 10Hashar) [10:28:41] (03PS1) 10Filippo Giunchedi: log: add @cee: token for structured logging [tools/scap] - 10https://gerrit.wikimedia.org/r/629342 (https://phabricator.wikimedia.org/T86969) [10:31:12] Project beta-scap-eqiad build #318389: 04STILL FAILING in 4 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318389/ [10:31:23] 10LibUp, 10MediaWiki-Codesniffer, 10Patch-For-Review: Drop "php5,inc" from MediaWiki-CodeSniffer ruleset of extensions - https://phabricator.wikimedia.org/T200956 (10Jdforrester-WMF) Hurrah. [10:31:29] 10LibUp, 10MediaWiki-Codesniffer, 10Patch-For-Review, 10Technical-Debt: Drop "php5,inc" from MediaWiki-CodeSniffer ruleset of extensions - https://phabricator.wikimedia.org/T200956 (10Jdforrester-WMF) [10:37:31] Project beta-scap-eqiad build #318390: 04STILL FAILING in 3 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318390/ [10:47:36] Project beta-scap-eqiad build #318391: 04STILL FAILING in 3 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318391/ [10:56:13] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10ArielGlenn) I assume there's no easy way to move around the paritions live, so the only solution is to create a new instance with a bigger root partition? [10:57:36] Project beta-scap-eqiad build #318392: 04STILL FAILING in 3 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318392/ [11:02:50] 10LibUp, 10MediaWiki-Codesniffer, 10Patch-For-Review, 10Technical-Debt: Drop "php5,inc" from MediaWiki-CodeSniffer ruleset of extensions - https://phabricator.wikimedia.org/T200956 (10Jdforrester-WMF) a:03Jdforrester-WMF [11:03:01] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10Reedy) >>! In T245402#6486931, @ArielGlenn wrote: > I assume there's no easy way to move around the paritions live, so the only solution is to create a new... [11:07:39] Project beta-scap-eqiad build #318393: 04STILL FAILING in 3 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318393/ [11:10:27] (03CR) 10Kosta Harlan: "I spent some more time on this, but kept running into various issues (using --user=1000:1000 results in MySQL being unable to install; not" [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [11:13:10] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [11:15:48] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [11:17:36] Project beta-scap-eqiad build #318394: 04STILL FAILING in 3 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318394/ [11:27:35] Project beta-scap-eqiad build #318395: 04STILL FAILING in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318395/ [11:31:11] Project beta-scap-eqiad build #318396: 04STILL FAILING in 3 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318396/ [11:34:48] Project beta-scap-eqiad build #318397: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318397/ [11:39:28] Project beta-scap-eqiad build #318398: 04STILL FAILING in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318398/ [11:41:30] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10dom_walden) I have just been able to successfully login to https://en.wikipedia.beta.wmflabs.org. [12:20:34] hi folks, who should I add to review https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/629342 ? [12:37:27] 10Release-Engineering-Team (Logspam), 10GlobalUsage, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10mmodell) [12:37:31] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T257978 (10mmodell) [12:38:25] 10Release-Engineering-Team (Logspam), 10GlobalUsage, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10mmodell) added as blocker even though this isn't new. It's cu... [13:00:29] 10Release-Engineering-Team (Logspam), 10GlobalUsage, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10mmodell) and yet the code in question hasn't been touched sinc... [13:14:15] godog: I added Lars, who's the scap dev lead. [13:14:39] James_F: TYVM sir [13:20:14] !log Restarted Releases Jenkins for T263289 [13:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:20:49] !log Restarted CI Jenkins for T263289 [13:20:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:26:27] (03CR) 10Lars Wirzenius: "Is there documentation of @cee ?" [tools/scap] - 10https://gerrit.wikimedia.org/r/629342 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi) [13:33:44] !log Kicked the Beta Cluster node on Jenkins to make it come back again. [13:33:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:37:55] Project beta-scap-eqiad build #318399: 04STILL FAILING in 3 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318399/ [13:47:30] Project beta-scap-eqiad build #318400: 04STILL FAILING in 3 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318400/ [13:49:59] Beta update still unhappy ^^ :-( [13:55:31] (03CR) 10Filippo Giunchedi: "> Patch Set 1:" [tools/scap] - 10https://gerrit.wikimedia.org/r/629342 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi) [13:57:30] Project beta-scap-eqiad build #318401: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318401/ [14:04:24] “rsync: write failed on "/srv/mediawiki/wikiversions-labs.php": No space left on device (28)” [14:04:34] Yeah. [14:04:40] Various out-of-space errors. [14:07:46] Project beta-scap-eqiad build #318402: 04STILL FAILING in 3 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318402/ [14:17:24] (03CR) 10Lars Wirzenius: "> The original documentation for cee is offline sadly, it was here: https://cee.mitre.org" [tools/scap] - 10https://gerrit.wikimedia.org/r/629342 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi) [14:17:30] Project beta-scap-eqiad build #318403: 04STILL FAILING in 3 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318403/ [14:26:40] (03CR) 10Filippo Giunchedi: "> Patch Set 1:" [tools/scap] - 10https://gerrit.wikimedia.org/r/629342 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi) [14:27:30] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T257978 (10Jdforrester-WMF) [14:27:35] Project beta-scap-eqiad build #318404: 04STILL FAILING in 3 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318404/ [14:28:45] (03CR) 10Lars Wirzenius: [C: 03+2] log: add @cee: token for structured logging [tools/scap] - 10https://gerrit.wikimedia.org/r/629342 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi) [14:30:59] (03Merged) 10jenkins-bot: log: add @cee: token for structured logging [tools/scap] - 10https://gerrit.wikimedia.org/r/629342 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi) [14:32:13] Project beta-scap-eqiad build #318405: 04STILL FAILING in 3 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318405/ [14:37:34] Project beta-scap-eqiad build #318406: 04STILL FAILING in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318406/ [14:47:32] Project beta-scap-eqiad build #318407: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318407/ [14:57:37] Project beta-scap-eqiad build #318408: 04STILL FAILING in 3 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318408/ [14:59:43] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10MW-on-K8s: Sketch MediaWiki production image composition and workflows - https://phabricator.wikimedia.org/T259817 (10Joe) To further clarify, I think the main question would be transforming a good porti... [15:07:38] Project beta-scap-eqiad build #318409: 04STILL FAILING in 3 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318409/ [15:14:49] 10Project-Admins, 10Unofficial-Apps-Wikipedia-Windows-8: Archive unofficial-apps-wikipedia-windows-8 tag ? - https://phabricator.wikimedia.org/T259930 (10Aklapper) @brion: Could you please answer the last comment? Thanks in advance! [15:17:33] Project beta-scap-eqiad build #318410: 04STILL FAILING in 3 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318410/ [15:22:19] 10Release-Engineering-Team, 10Wikidata, 10Documentation: Update links to doc.wikimedia.org - https://phabricator.wikimedia.org/T251259 (10Aklapper) @Lea_Lacroix_WMDE: Given the two last comments, can this be resolved? If not, where to look? Thanks! [15:27:17] 10Beta-Cluster-Infrastructure, 10Maps, 10Product-Infrastructure-Team-Backlog: beta-scap-eqiad FAILED - deployment-maps06.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T262101 (10MSantos) 05Open→03Resolved a:03MSantos It resolved by itsel... [15:27:43] Project beta-scap-eqiad build #318411: 04STILL FAILING in 3 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318411/ [15:31:41] 10Release-Engineering-Team (Logspam), 10GlobalUsage, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10thcipriani) p:05Low→03Unbreak! train blockers are UBNs [15:32:49] 10Release-Engineering-Team (Logspam), 10GlobalUsage, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10mmodell) So the code in question appears to be ok, I think som... [15:37:34] Project beta-scap-eqiad build #318412: 04STILL FAILING in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318412/ [15:38:12] 10Release-Engineering-Team (Logspam), 10GlobalUsage, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10thcipriani) Requests are generating a lot of noise here. For e... [15:47:32] Project beta-scap-eqiad build #318413: 04STILL FAILING in 3 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318413/ [15:56:28] 10Gerrit, 10Abstract Wikipedia: mediawiki/extensions/WikiLambda repository has a leading space in the name - https://phabricator.wikimedia.org/T257775 (10Aklapper) We might once again be blocked here on {T239693}? Sigh [15:57:30] Project beta-scap-eqiad build #318414: 04STILL FAILING in 3 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318414/ [15:59:01] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T257978 (10Urbanecm) [16:07:31] Project beta-scap-eqiad build #318415: 04STILL FAILING in 3 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318415/ [16:17:33] Project beta-scap-eqiad build #318416: 04STILL FAILING in 3 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318416/ [16:22:40] 10Continuous-Integration-Infrastructure, 10Wikidata: Authenticate github API requests in CI - https://phabricator.wikimedia.org/T254319 (10Pablo-WMDE) 05Open→03Resolved a:03Pablo-WMDE This is working for us - the variable from which this is sourced may not be ideal(?) but it's not hurting operation. Tha... [16:27:36] Project beta-scap-eqiad build #318417: 04STILL FAILING in 3 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318417/ [16:33:34] 10Beta-Cluster-Infrastructure: deployment-snapshot01.deployment-prep.eqiad.wmflabs keeps running out of space - https://phabricator.wikimedia.org/T245402 (10dancy) The easiest thing to do is to enlarge the virtual disk or add an additional vdisk to house /srv. How do we move forward with one of those options? [16:37:30] Project beta-scap-eqiad build #318418: 04STILL FAILING in 3 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318418/ [16:37:59] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10User-Addshore: Remove DENY rules for https://integration.wikimedia.org/ci/computer/XXXX/builds - https://phabricator.wikimedia.org/T178458 (10hashar) >>! In T178458#6483469, @CDanis wrote: > > At this point wouldn't it be better to try removing... [16:40:47] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10User-brennen: logspam-watch: add seconds to datestamp / refresh display each iteration - https://phabricator.wikimedia.org/T260826 (10brennen) 05Open→03Resolved [16:46:23] 10Scap: Investigate deployment concurrency limitations for ORES - https://phabricator.wikimedia.org/T188281 (10calbon) [16:47:31] Project beta-scap-eqiad build #318419: 04STILL FAILING in 3 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318419/ [16:57:25] Project beta-scap-eqiad build #318420: 04STILL FAILING in 3 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318420/ [17:07:15] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Abstract Wikipedia: mediawiki/extensions/WikiLambda repository has a leading space in the name - https://phabricator.wikimedia.org/T257775 (10hashar) Indeed there is an extra repo... [17:07:46] Project beta-scap-eqiad build #318421: 04STILL FAILING in 3 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318421/ [17:08:52] !log Gerrit: removed "/srv/gerrit/git/ mediawiki/extensions/WikiLambda" (note the extra space in front of 'mediawiki') # T257775 [17:08:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:08:55] T257775: mediawiki/extensions/WikiLambda repository has a leading space in the name - https://phabricator.wikimedia.org/T257775 [17:09:21] heh [17:11:39] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Abstract Wikipedia: mediawiki/extensions/WikiLambda repository has a leading space in the name - https://phabricator.wikimedia.org/T257775 (10hashar) 05Open→03Resolved a:03ha... [17:14:27] Lucas_WMDE: that was a ghost WikiLambda :] [17:14:49] 👻 😱 [17:15:04] (we’re allowed to chat in emoji now, wikitech-l said so ;) ) [17:17:29] Project beta-scap-eqiad build #318422: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318422/ [17:22:17] (03CR) 10Dduvall: [C: 03+2] pipeline config: promote to blubberoid chart [blubber] - 10https://gerrit.wikimedia.org/r/629211 (owner: 10Jeena Huneidi) [17:25:28] (03Merged) 10jenkins-bot: pipeline config: promote to blubberoid chart [blubber] - 10https://gerrit.wikimedia.org/r/629211 (owner: 10Jeena Huneidi) [17:27:29] Project beta-scap-eqiad build #318423: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318423/ [17:29:18] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10GitLab-Test, 10User-brennen: Document GitLab code review workflows - https://phabricator.wikimedia.org/T263668 (10brennen) [17:31:30] 👻👻👻👻👻👻👻👻👻👻👻👻👻 [17:34:41] o [17:35:03] (03PS1) 10Hashar: changelog: fix rst links [integration/quibble] - 10https://gerrit.wikimedia.org/r/629438 [17:35:51] (03CR) 10Hashar: "That comes from Quibble 0.0.40 CHANGELog.rst" [integration/quibble] - 10https://gerrit.wikimedia.org/r/629438 (owner: 10Hashar) [17:36:23] hey, apparently beta cluster updates haven't worked recently, but they are now fixed. is that true? i'm trying to explain why some changes were not visible on beta [17:36:38] i see that https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL says that James_F "kicked" something but it lacks detail [17:37:27] Project beta-scap-eqiad build #318424: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318424/ [17:38:04] oh, is it "STILL FAILING" in bold, red and uppercase? [17:47:27] Project beta-scap-eqiad build #318425: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318425/ [17:57:28] Project beta-scap-eqiad build #318426: 04STILL FAILING in 3 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318426/ [17:59:28] MatmaRex: yeah, the -snapshots host is out of space due to the l10n files, it's on the releng team's radar (was discussed just now in the team meeting) [17:59:41] 10Release-Engineering-Team, 10Analytics-Radar, 10observability, 10serviceops, 10User-jijiki: Should we create a separate 'mwdebug' cluster? - https://phabricator.wikimedia.org/T262202 (10jijiki) @Milimetric that would be great, if it is not too much work, I would appreciate it. I will work on the varnish... [18:00:07] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10MW-on-K8s: Sketch MediaWiki production image composition and workflows - https://phabricator.wikimedia.org/T259817 (10dduvall) >>! In T259817#6487642, @Joe wrote: > To further clarify, I think the main q... [18:01:22] 10Release-Engineering-Team, 10Analytics-Radar, 10observability, 10serviceops, 10User-jijiki: Create a separate 'mwdebug' cluster - https://phabricator.wikimedia.org/T262202 (10jijiki) [18:07:16] thanks greg-g [18:07:35] Project beta-scap-eqiad build #318427: 04STILL FAILING in 3 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318427/ [18:07:51] ap dang [18:11:05] Project beta-scap-eqiad build #318428: 04STILL FAILING in 3 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318428/ [18:17:23] Project beta-scap-eqiad build #318429: 04STILL FAILING in 3 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318429/ [18:18:23] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T257978 (10thcipriani) [18:20:55] thcipriani: i cant make train log triage today but doesn't look like there are any new client side errors to talk about today. Will let you know if I see anything after the group 1 deploy [18:24:44] Jdlrobson: ack, thank you [18:26:33] Project beta-scap-eqiad build #318430: 04STILL FAILING in 3 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318430/ [18:31:10] Project beta-scap-eqiad build #318431: 04STILL FAILING in 3 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318431/ [18:37:28] Project beta-scap-eqiad build #318432: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318432/ [18:47:21] Project beta-scap-eqiad build #318433: 04STILL FAILING in 3 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318433/ [18:57:10] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)): gerrit yields Error 400 (Bad Request): Error in operator parentproject:integration - https://phabricator.wikimedia.org/T263682 (10hashar) [18:57:25] Project beta-scap-eqiad build #318434: 04STILL FAILING in 3 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318434/ [18:59:00] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)): gerrit yields Error 400 (Bad Request): Error in operator parentproject:integration - https://phabricator.wikimedia.org/T263682 (10hashar) [19:07:42] Project beta-scap-eqiad build #318435: 04STILL FAILING in 3 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318435/ [19:11:35] (03PS1) 10Jeena Huneidi: PipelineStage: Add node label for promote step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/629451 [19:11:36] !log gerrit: flushing "project_list" cache after deletion of " mediawiki/extensions/WikiLambda": gerrit flush-caches --cache project_list # T263682 T257775 [19:11:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:11:41] T263682: gerrit yields Error 400 (Bad Request): Error in operator parentproject:integration - https://phabricator.wikimedia.org/T263682 [19:11:41] T257775: mediawiki/extensions/WikiLambda repository has a leading space in the name - https://phabricator.wikimedia.org/T257775 [19:13:17] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)): gerrit yields Error 400 (Bad Request): Error in operator parentproject:integration - https://phabricator.wikimedia.org/T263682 (10hashar) 05Open→03Resolved a:03hashar I deleted... [19:16:11] (03PS2) 10Jeena Huneidi: PipelineStage: Add node label for promote step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/629451 [19:16:52] 10Gerrit, 10Release-Engineering-Team (Development services): Support renaming repositories in Gerrit - https://phabricator.wikimedia.org/T239693 (10hashar) [19:17:30] Project beta-scap-eqiad build #318436: 04STILL FAILING in 3 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318436/ [19:19:34] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)): gerrit yields Error 400 (Bad Request): Error in operator parentproject:integration - https://phabricator.wikimedia.org/T263682 (10thcipriani) >>! In T263682#6489090, @hashar wrote: >... [19:22:28] !log operations/software/gerrit changing HEAD from deploy/wmf/stable-2.15 to deploy/wmf/stable-3.2 # T254158 [19:22:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:22:31] T254158: Gerrit 3.2 upgrade - https://phabricator.wikimedia.org/T254158 [19:33:17] Yippee, build fixed! [19:33:17] Project beta-scap-eqiad build #318437: 09FIXED in 9 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/318437/ [19:37:33] (03CR) 10Jeena Huneidi: "Added the labels in Jenkins already" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/629451 (owner: 10Jeena Huneidi) [19:51:51] 10Gerrit, 10Release-Engineering-Team (Development services): Install rename-project plugin - https://phabricator.wikimedia.org/T201953 (10hashar) [20:00:46] 10Release-Engineering-Team, 10Move-Files-To-Commons, 10ci-test-error: Tests failing for the REL1_34 branch of ext:FileImporter within quibble-composer-mysql-php72-noselenium-docker - https://phabricator.wikimedia.org/T263690 (10sbassett) [20:13:31] 10Release-Engineering-Team, 10Move-Files-To-Commons, 10MW-1.34-release, 10ci-test-error: Tests failing for the REL1_34 branch of ext:FileImporter within quibble-composer-mysql-php72-noselenium-docker - https://phabricator.wikimedia.org/T263690 (10Reedy) [20:14:39] 10Gerrit: Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" - https://phabricator.wikimedia.org/T263293 (10matmarex) I got this issue again today trying to pull operations/mediawiki-config. It failed twice, then worked on the third try. ` C:\mediawiki\operations\mediawik... [20:43:34] 10Release-Engineering-Team (Logspam), 10GlobalUsage, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10DannyS712) [21:14:04] 10MediaWiki-Releasing, 10Security: Write and send pre-release announcements for MediaWiki 1.31.9/1.34.3/1.35.0 - https://phabricator.wikimedia.org/T256336 (10Reedy) 05Open→03Resolved a:03Reedy [21:14:06] 10MediaWiki-Releasing, 10Security: Write and send pre-release announcements for MediaWiki 1.31.9/1.34.3/1.35.0 - https://phabricator.wikimedia.org/T256336 (10Reedy) https://lists.wikimedia.org/pipermail/mediawiki-announce/2020-September/000259.html [21:14:09] 10MediaWiki-Releasing, 10Security: Write and send pre-release announcements for MediaWiki 1.31.9/1.34.3/1.35.0 - https://phabricator.wikimedia.org/T256336 (10Reedy) [21:54:23] (03CR) 10Dduvall: [C: 04-1] PipelineStage: Add node label for promote step (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/629451 (owner: 10Jeena Huneidi) [22:50:59] 10Release-Engineering-Team (Logspam), 10GeoData, 10Discovery-Search (Current work), 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: Elastica\Exception\ResponseException from line 56 of includes/Searcher.php: - https://phabricator.wikimedia.org/T258641 (10Reedy) [23:01:26] 10MediaWiki-Releasing, 10MW-1.35-notes, 10MW-1.35-release, 10Platform Team Workboards (Initiatives): Release MW 1.35.0 - https://phabricator.wikimedia.org/T256375 (10Reedy) [23:02:04] 10MediaWiki-Releasing, 10Documentation, 10MW-1.35-notes, 10MW-1.35-release, and 2 others: Finish up RELEASE-NOTES for 1.35.0 - https://phabricator.wikimedia.org/T259349 (10Reedy) 05Open→03Resolved a:03Reedy [23:09:36] 10MediaWiki-Releasing, 10MW-1.35-notes, 10MW-1.35-release, 10Platform Team Workboards (Initiatives): Release MW 1.35.0 - https://phabricator.wikimedia.org/T256375 (10Reedy) [23:27:25] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T257978 (10dancy) [23:30:14] 10Release-Engineering-Team, 10Move-Files-To-Commons, 10MW-1.34-release, 10ci-test-error: Tests failing for the REL1_34 branch of ext:FileImporter within quibble-composer-mysql-php72-noselenium-docker - https://phabricator.wikimedia.org/T263690 (10Reedy) 05Open→03Resolved a:03Ammarpad [23:30:25] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T257978 (10dancy) [23:30:43] 10Release-Engineering-Team (Logspam), 10GlobalUsage, 10MW-1.36-notes (1.36.0-wmf.11; 2020-09-29), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10dancy) [23:47:15] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T257978 (10DannyS712) @dancy can I ask why you removed the blocker subtasks instead of resolving them?... [23:51:33] (03CR) 10Jeena Huneidi: PipelineStage: Add node label for promote step (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/629451 (owner: 10Jeena Huneidi) [23:52:38] (03PS3) 10Jeena Huneidi: PipelineStage: Add node label for promote step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/629451