[00:24:30] Project beta-scap-eqiad build #271185: 04FAILURE in 4.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/271185/ [00:36:23] Yippee, build fixed! [00:36:23] Project beta-scap-eqiad build #271186: 09FIXED in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/271186/ [03:55:35] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: (Service Check Timed Out) [03:59:16] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10wikimedia.biterg.io, 10Upstream: Gerrit replica does not support the 'query' command - https://phabricator.wikimedia.org/T235251 (10Aklapper) [04:05:25] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 49240 bytes in 0.527 second response time [06:27:44] 10Project-Admins: Create projects for WLM Italy - https://phabricator.wikimedia.org/T235379 (10Aklapper) @Ferdi2005: We host code repositories in Git/Gerrit. Please see https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests for that - thanks! :) [08:39:57] (03PS1) 10Hashar: ooui job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542868 [08:42:06] (03PS1) 10Hashar: Fundraising jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542869 [08:47:18] (03PS1) 10Hashar: Migrate Phan jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542872 [08:54:23] (03CR) 10Hashar: [C: 03+2] "Passed on a dummy change: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/DonationInterface/+/324066/" [integration/config] - 10https://gerrit.wikimedia.org/r/542869 (owner: 10Hashar) [09:03:30] (03PS2) 10Hashar: ooui job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542868 [09:03:32] (03PS2) 10Hashar: Fundraising jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542869 [09:03:34] (03PS2) 10Hashar: Migrate Phan jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542872 [09:03:36] (03PS1) 10Hashar: Fix ooui-npm-special-node10-plus-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/542877 [09:06:39] (03CR) 10Hashar: [C: 03+2] "I am not sure what this job is supposed to do but it is still broken after that :\" [integration/config] - 10https://gerrit.wikimedia.org/r/542877 (owner: 10Hashar) [09:06:42] (03CR) 10Hashar: [C: 03+2] ooui job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542868 (owner: 10Hashar) [09:06:50] (03CR) 10Hashar: [C: 03+2] Migrate Phan jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542872 (owner: 10Hashar) [09:09:26] (03Merged) 10jenkins-bot: Fix ooui-npm-special-node10-plus-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/542877 (owner: 10Hashar) [09:09:29] (03Merged) 10jenkins-bot: ooui job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542868 (owner: 10Hashar) [09:09:33] (03Merged) 10jenkins-bot: Fundraising jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542869 (owner: 10Hashar) [09:09:36] (03Merged) 10jenkins-bot: Migrate Phan jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542872 (owner: 10Hashar) [09:14:01] (03PS1) 10Hashar: Upgrade Fresnel job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542878 [09:15:39] (03PS1) 10Hashar: Upgrade codehealth jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542879 [09:16:48] (03PS1) 10Hashar: Upgrade mwselenium integration job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542880 [09:16:55] (03CR) 10Hashar: [C: 03+2] Upgrade codehealth jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542879 (owner: 10Hashar) [09:16:59] (03CR) 10Hashar: [C: 03+2] Upgrade Fresnel job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542878 (owner: 10Hashar) [09:17:05] (03CR) 10Hashar: [C: 03+2] Upgrade mwselenium integration job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542880 (owner: 10Hashar) [09:19:46] (03Merged) 10jenkins-bot: Upgrade Fresnel job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542878 (owner: 10Hashar) [09:19:49] (03Merged) 10jenkins-bot: Upgrade codehealth jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542879 (owner: 10Hashar) [09:19:51] (03Merged) 10jenkins-bot: Upgrade mwselenium integration job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542880 (owner: 10Hashar) [09:20:59] (03PS1) 10Hashar: Upgrade coverage jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542881 [09:24:01] (03CR) 10Hashar: [C: 03+2] Upgrade coverage jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542881 (owner: 10Hashar) [09:24:51] (03PS1) 10Hashar: Upgrade mwselenium-quibble-docker to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542883 [09:24:53] (03PS1) 10Hashar: Upgrade Wikibase jos to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542884 [09:26:20] (03Merged) 10jenkins-bot: Upgrade coverage jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542881 (owner: 10Hashar) [09:26:32] (03PS1) 10Hashar: Upgrade wmf-quibble* jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542885 [09:27:01] (03CR) 10Hashar: [C: 03+2] Upgrade wmf-quibble* jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542885 (owner: 10Hashar) [09:27:06] (03CR) 10Hashar: [C: 03+2] Upgrade Wikibase jos to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542884 (owner: 10Hashar) [09:27:12] (03CR) 10Hashar: [C: 03+2] Upgrade mwselenium-quibble-docker to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542883 (owner: 10Hashar) [09:29:12] (03PS1) 10Hashar: Upgrade parsoid parser tests job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542886 [09:29:30] (03Merged) 10jenkins-bot: Upgrade mwselenium-quibble-docker to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542883 (owner: 10Hashar) [09:29:57] (03Merged) 10jenkins-bot: Upgrade Wikibase jos to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542884 (owner: 10Hashar) [09:30:00] (03Merged) 10jenkins-bot: Upgrade wmf-quibble* jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542885 (owner: 10Hashar) [09:30:16] (03CR) 10Hashar: [C: 03+2] Upgrade parsoid parser tests job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542886 (owner: 10Hashar) [09:31:07] (03PS1) 10Hashar: Upgrade MediaWiki jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542887 [09:31:15] (03CR) 10Hashar: [C: 03+2] Upgrade MediaWiki jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542887 (owner: 10Hashar) [09:32:40] (03Merged) 10jenkins-bot: Upgrade parsoid parser tests job to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542886 (owner: 10Hashar) [09:32:52] !log Deleting old release-quibble* jobs from Jenkins. They are no more used. [09:32:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:33:36] (03Merged) 10jenkins-bot: Upgrade MediaWiki jobs to Quibble 0.0.38 [integration/config] - 10https://gerrit.wikimedia.org/r/542887 (owner: 10Hashar) [09:34:37] !log Upgraded CI jobs to Quibble 0.0.38 [09:34:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:40:00] !log contint1001: deleting old Quibble containers [11:40:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:44:08] 10Deployments, 10MediaWiki-SWAT-deployments: Figure out what to do with `fatalmonitor` script - https://phabricator.wikimedia.org/T234345 (10Urbanecm) As @Joe said, we can use https://logstash.wikimedia.org/app/kibana#/dashboard/Fatal-Monitor instead of the script, until it is fixed. [11:45:04] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10International-Developer-Events, 10Wikimedia-Technical-Conference-2019, and 2 others: Wikimedia Technical Conference 2019 Session: System level testing: patterns an... - https://phabricator.wikimedia.org/T234635 [11:45:47] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10International-Developer-Events, 10Wikimedia-Technical-Conference-2019, and 2 others: Wikimedia Technical Conference 2019 Session: System level testing: patterns an... - https://phabricator.wikimedia.org/T234635 [11:49:41] 10Deployments, 10MediaWiki-SWAT-deployments: Figure out what to do with `fatalmonitor` script - https://phabricator.wikimedia.org/T234345 (10Lucas_Werkmeister_WMDE) And this version of the link only shows the past 5 minutes and auto-updates itself every 5 seconds: https://logstash.wikimedia.org/app/kibana#/da... [11:50:27] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907): Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10jcrespo) This happens to me, sometimes multiple times a day. [11:58:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910): Bump CI php72 images given we've moved production to 7.2.22 - https://phabricator.wikimedia.org/T232165 (10hashar) The Quibble images are at 7.2.22 since I have rebuild them... [12:06:55] (03PS1) 10Hashar: docker: doxygen image depends on php72 [integration/config] - 10https://gerrit.wikimedia.org/r/542907 [12:16:53] (03PS1) 10Hashar: docker: rebuild for php7.2.22 [integration/config] - 10https://gerrit.wikimedia.org/r/542910 (https://phabricator.wikimedia.org/T232165) [12:17:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Patch-For-Review: Bump CI php72 images given we've moved production to 7.2.22 - https://phabricator.wikimedia.org/T232165 (10hashar) a:03hashar [12:17:22] (03CR) 10Hashar: [C: 03+2] docker: doxygen image depends on php72 [integration/config] - 10https://gerrit.wikimedia.org/r/542907 (owner: 10Hashar) [12:19:03] (03Merged) 10jenkins-bot: docker: doxygen image depends on php72 [integration/config] - 10https://gerrit.wikimedia.org/r/542907 (owner: 10Hashar) [13:25:11] Project beta-scap-eqiad build #271271: 04FAILURE in 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/271271/ [13:36:35] Yippee, build fixed! [13:36:35] Project beta-scap-eqiad build #271272: 09FIXED in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/271272/ [13:45:07] 00:00:43.036 Fatal error: Uncaught ConfigException: Failed to load configuration from etcd: (curl error: 28) Timeout was reached in /srv/mediawiki-staging/php-master/includes/config/EtcdConfig.php:201 [13:45:11] so I guess it is transient [13:45:30] no other builds have failed in the same way today [14:35:45] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10TechCom: Expand Gerrit Manager permissions - https://phabricator.wikimedia.org/T234474 (10jijiki) p:05Triage→03Normal [14:45:48] 10Phabricator, 10Operations, 10Traffic: Access Forbidden to Phabricator at WikiArabia 2019 (Morocco) - https://phabricator.wikimedia.org/T234598 (10jijiki) p:05Triage→03High [15:48:32] 10Release-Engineering-Team, 10Scap, 10Operations, 10Wikimedia-General-or-Unknown, 10serviceops: "Currently active MediaWiki versions:" broken on noc/conf - https://phabricator.wikimedia.org/T235338 (10jijiki) p:05Triage→03Normal a:03jijiki [15:49:01] 10Release-Engineering-Team, 10Scap, 10Operations, 10Wikimedia-General-or-Unknown, and 2 others: "Currently active MediaWiki versions:" broken on noc/conf - https://phabricator.wikimedia.org/T235338 (10jijiki) [17:47:52] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [17:47:52] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [18:32:21] (03CR) 10Hashar: [C: 03+2] "The job have been deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/540872 (https://phabricator.wikimedia.org/T199113) (owner: 10Zfilipin) [18:35:02] (03Merged) 10jenkins-bot: WIP All repositories with Selenium tests should use wdio-mediawiki [integration/config] - 10https://gerrit.wikimedia.org/r/540872 (https://phabricator.wikimedia.org/T199113) (owner: 10Zfilipin) [18:51:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 3 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10hashar) Zuul does emit timing per pipelin... [18:55:07] 10Continuous-Integration-Config, 10Release-Engineering-Team: quibble-selenium jobs re-downloading npm packages (castor not loading) - https://phabricator.wikimedia.org/T234738 (10hashar) The node10 container does run `npm ci`. Quibble does `npm install` and uses `--prefer-offline` since July 25th (70447e23289... [19:07:10] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10wikimedia.biterg.io, 10Upstream: Gerrit replica does not support the 'query' command - https://phabricator.wikimedia.org/T235251 (10hashar) [19:07:39] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10wikimedia.biterg.io, 10Upstream: Gerrit replica does not support the 'query' command - https://phabricator.wikimedia.org/T235251 (10hashar) Reported #upstream: https://bugs.chromium.org/p/gerrit/issue... [19:07:48] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10wikimedia.biterg.io, 10Upstream: Gerrit replica does not support the 'query' command - https://phabricator.wikimedia.org/T235251 (10hashar) p:05Triage→03Normal [19:12:22] * paladox looks at hashar [19:12:28] you reported it as a polygerrit issue :D [19:12:41] paladox: yeah by mistake :-\\\ [19:12:47] paladox: and I could not find how to edit the component [19:12:58] hashar yeh you need permission to do that [19:13:01] * paladox removes it [19:13:10] I guess someone will triage it properly [19:13:19] I guess I just missed the components part [19:13:33] done [19:15:13] paladox: the actual fix is most probably straightforward code wise. The devil is figuring out whether "gerrit query" should be allowed on slave/replica hosts [19:15:19] and that part, I have no clue ;] [19:15:46] yup [19:16:04] as long as the query cannot write, then i would think it would be fine [19:22:07] probably yes :) paladox [19:24:20] hashar https://gerrit-review.googlesource.com/c/gerrit/+/240873 [19:24:47] ;-] [19:33:59] I am off *wave* [19:39:38] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Code-Health: Quibble + PHPUnit: Run databaseless and database tests in parallel - https://phabricator.wikimedia.org/T235449 (10kostajh) [19:44:45] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Browser-Tests, and 2 others: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002 (10kostajh) > Have folks considered using child_process.exec() to r... [20:04:45] 10Continuous-Integration-Config, 10Release-Engineering-Team: quibble-selenium jobs re-downloading npm packages (castor not loading) - https://phabricator.wikimedia.org/T234738 (10Krinkle) Sounds good to me! [20:40:28] 10Continuous-Integration-Config, 10Release-Engineering-Team: quibble-selenium jobs re-downloading npm packages (castor not loading) - https://phabricator.wikimedia.org/T234738 (10hashar) There is also the incredibly slows: ` > core-js@3.2.1 postinstall xxx > node scripts/postinstall || echo "ignore" ` I haven... [20:51:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Code-Health: Quibble + PHPUnit: Run databaseless and database tests in parallel - https://phabricator.wikimedia.org/T235449 (10Krinkle) This following is entirely optional, but would it be... [23:12:36] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/BlueSpiceDistributionConnector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/543017 [23:12:38] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/BlueSpiceDistributionConnector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/543017 (owner: 10QChris) [23:12:47] (03PS1) 10QChris: Import done. Revoke import grants [extensions/BlueSpiceDistributionConnector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/543018 [23:12:49] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/BlueSpiceDistributionConnector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/543018 (owner: 10QChris)