[00:05:14] (03CR) 10Jeena Huneidi: "Haven't been able to get this to run in minikube after following the directions, but maybe I've made a mistake somewhere during the build " [releng/dev-images] - 10https://gerrit.wikimedia.org/r/525842 (https://phabricator.wikimedia.org/T222494) (owner: 10Brennen Bearnes) [00:25:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10Tgr) >>! In T232796#5507156, @mmodell wrote: > @tgr: where did you see these logs spe... [00:53:09] (03CR) 10Jforrester: [C: 03+2] [PropertySuggester] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/538090 (https://phabricator.wikimedia.org/T224774) (owner: 10Umherirrender) [00:54:44] (03Merged) 10jenkins-bot: [PropertySuggester] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/538090 (https://phabricator.wikimedia.org/T224774) (owner: 10Umherirrender) [01:00:43] !log Zuul: [PropertySuggester] Add phan dependency T224774 [01:00:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:00:46] T224774: Add phan to PropertySuggester extension - https://phabricator.wikimedia.org/T224774 [01:10:27] (03PS1) 10Jforrester: layout: [PropertySuggster] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/538132 (https://phabricator.wikimedia.org/T224774) [01:11:05] (03PS2) 10Jforrester: layout: [PropertySuggester] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/538132 (https://phabricator.wikimedia.org/T224774) [01:11:07] (03CR) 10Jforrester: [C: 03+2] layout: [PropertySuggester] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/538132 (https://phabricator.wikimedia.org/T224774) (owner: 10Jforrester) [01:12:21] (03Merged) 10jenkins-bot: layout: [PropertySuggester] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/538132 (https://phabricator.wikimedia.org/T224774) (owner: 10Jforrester) [01:12:39] !log Zuul: [PropertySuggester] Add phan [01:12:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:26:33] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T220748 (10mmodell) 05Open→03Resolved [03:16:19] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:16:34] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:21:16] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48597 bytes in 6.250 second response time [03:21:32] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 48070 bytes in 6.215 second response time [04:27:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:33] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 48060 bytes in 6.438 second response time [04:53:08] 10Release-Engineering-Team-TODO, 10Operations, 10Traffic, 10Patch-For-Review: Blubberoid endpoint intermittently routing to MediaWiki backend - https://phabricator.wikimedia.org/T233369 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez before: `vgutierrez@cp1075:~$ curl -H 'Host: blubberoid.wikimedia.o... [08:00:03] Project mwcore-phpunit-coverage-master build #185: 04FAILURE in 5 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/185/ [08:12:42] ^^ slow agent :-\ [08:23:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-merger fails to fetch from Gerrit - https://phabricator.wikimedia.org/T233390 (10hashar) [08:23:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-merger fails to fetch from Gerrit - https://phabricator.wikimedia.org/T233390 (10hashar) p:05Triage→03Unbreak! [08:24:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-merger fails to fetch from Gerrit - https://phabricator.wikimedia.org/T233390 (10hashar) On contint1001 ` $ sudo su - zuul $ cd /srv/zuul/git/operations/puppet... [08:26:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-merger fails to fetch from Gerrit - https://phabricator.wikimedia.org/T233390 (10hashar) Gerrit has two connections from each contint servers for a total of fo... [08:30:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-merger fails to fetch from Gerrit - https://phabricator.wikimedia.org/T233390 (10hashar) 05Open→03Resolved I have upgraded zuul on contint2001 (T203846) wh... [08:31:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-merger fails to fetch from Gerrit - https://phabricator.wikimedia.org/T233390 (10Joe) @Hashar I guess the CI servers should have more relaxed thresholds? Is it... [08:32:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-merger fails to fetch from Gerrit - https://phabricator.wikimedia.org/T233390 (10Joe) and yes, if you need to have a service not start when the package is inst... [08:35:22] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-server should not start on spare server when the Debian package is upgraded - https://phabricator.wikimedia.org/T233391 (10hashar) [08:40:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-merger fails to fetch from Gerrit - https://phabricator.wikimedia.org/T233390 (10hashar) >>! In T233390#5508670, @Joe wrote: > @Hashar I guess the CI servers s... [08:48:05] PROBLEM - Host deployment-alex-test is DOWN: CRITICAL - Host Unreachable (172.16.6.31) [08:55:08] (03CR) 10Awight: [C: 03+2] "Looks like a big improvement. We might go ahead and kill all the workers immediately, but that can be done in a later patch." (033 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/538020 (https://phabricator.wikimedia.org/T233143) (owner: 10Hashar) [08:55:49] (03Merged) 10jenkins-bot: Exit on cloning failure [integration/quibble] - 10https://gerrit.wikimedia.org/r/538020 (https://phabricator.wikimedia.org/T233143) (owner: 10Hashar) [08:56:21] (03CR) 10jenkins-bot: Exit on cloning failure [integration/quibble] - 10https://gerrit.wikimedia.org/r/538020 (https://phabricator.wikimedia.org/T233143) (owner: 10Hashar) [09:01:32] <_joe_> hashar: how do I promote a job from being experimental to being run on all changes? [09:01:52] _joe_: in zuul/layout.yaml instead of: [09:01:55] experimental: [09:01:57] - yourjob [09:01:58] <_joe_> ack! [09:02:04] add definitions for test and gate-and-submit: [09:02:05] test: [09:02:08] - yourjob [09:02:11] gate-and-submit: [09:02:13] - yourjob [09:02:22] though we might want to use test-prio: [09:02:35] <_joe_> this is the repository for deploys [09:02:38] <_joe_> so yeah [09:02:42] test / test-prio reacts when a new patchset is proposed [09:02:47] with test-prio having high precedence [09:03:00] gate-and-submit reacts on code-review +2 events and would eventually submit the change on success [09:03:05] so yeah test-prio: [09:03:06] ;] [09:03:09] <_joe_> sorry you told me all this multiple times already, it all came back [09:03:31] <_joe_> but it ends up in the cold cache because I don't need to do it that often [09:04:31] _joe_: yeah don't worry, I am happy to repeat :] [09:07:14] 10Continuous-Integration-Config, 10Operations, 10Release Pipeline, 10local-charts, and 3 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10Joe) [09:07:34] (03PS1) 10Giuseppe Lavagetto: zuul: use helm-lint job to test deployment-charts [integration/config] - 10https://gerrit.wikimedia.org/r/538167 (https://phabricator.wikimedia.org/T233291) [09:09:01] <_joe_> hashar: whenever suits you, ^^ [09:09:19] <_joe_> I am going to be bold and add the rakefile to local-charts too [09:09:21] (03CR) 10Hashar: [C: 03+2] zuul: use helm-lint job to test deployment-charts [integration/config] - 10https://gerrit.wikimedia.org/r/538167 (https://phabricator.wikimedia.org/T233291) (owner: 10Giuseppe Lavagetto) [09:09:29] * hashar awards 1 CI point to _joe_ [09:10:05] and yeah that is a lot of manual actions (create the container, build it, define the ci config etc) [09:10:18] and we want to get rid of that for the future CI [09:10:27] namely, stop releng/CI maintainers from being the bottleneck [09:10:43] and let developers to define whatever they need. Similar to Travis [09:10:44] <_joe_> yeah that was the reasoning behind my "quick win" proposal [09:10:54] (03Merged) 10jenkins-bot: zuul: use helm-lint job to test deployment-charts [integration/config] - 10https://gerrit.wikimedia.org/r/538167 (https://phabricator.wikimedia.org/T233291) (owner: 10Giuseppe Lavagetto) [09:11:01] the pipeline stuff for mediawiki services is more or less that [09:11:16] though it still has to be added to the CI zuul/layout.yaml config, that is a limitation of the Zuul version we use [09:11:25] <_joe_> yes the pipeline is a bit specialized though [09:11:33] yup [09:11:35] baby steps :-] [09:11:50] <_joe_> sure I am a fan of incremental improvements [09:11:54] change deployed [09:11:59] <_joe_> <3 [09:11:59] so you can "recheck" or just +2 [09:12:02] and that should merge maybe [09:12:09] though maybe jenkins lacks rights to submit a change on that repo [09:12:12] hmm [09:12:24] <_joe_> oh, I don't think so [09:12:31] <_joe_> devs have +2 rights there [09:12:40] <_joe_> but yeah let's see [09:12:43] https://gerrit.wikimedia.org/r/#/admin/projects/operations/deployment-charts,access [09:12:46] it has submit rights :] [09:12:47] all set [09:12:53] so zuul should be able to gerrit review --submit [09:12:54] :] [09:13:49] <_joe_> yeah doing the dance right now [09:14:04] <_joe_> Change has been successfully merged by jenkins-bot [09:14:07] <_joe_> :) [10:21:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul, 10Patch-For-Review: zuul-server should not start on spare server when the Debian package is upgraded - https://phabricator.wikimedia.org/T233391 (10hashar) p:05T... [11:14:09] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201908), 10Code-Stewardship-Reviews, 10Graphoid, and 3 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10dr0ptp4kt) Hello all. We're going to turn this into a client-side feature and divest of... [12:58:58] Hmm, was this operations/debs/pkg-php/php-ast repo deleted? [12:59:02] I see Skipping change 404284 because the corresponding repository was not found [13:02:53] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review: Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 (10hash... [13:03:28] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review: Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 (10hash... [13:13:55] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review: Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 (10hash... [13:14:12] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review: Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 (10hash... [13:14:31] !log Pooled integration-agent-docker-1001 based on Stretch # T226233 [13:14:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:14:36] T226233: Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 [13:14:41] 10Release-Engineering-Team (Unit & Int & System Tooling), 10MinervaNeue, 10MW-1.34-notes (1.34.0-wmf.15; 2019-07-23), 10Readers-Web-Backlog (Tracking), and 2 others: Flaky wdio test: MinervaNeue/history_steps "Page diff Added and removed content" - https://phabricator.wikimedia.org/T229031 (10zeljkofilipin) [13:18:41] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Selenium tests should be easier to run - https://phabricator.wikimedia.org/T182691 (10zeljkofilipin) [13:19:00] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10Documentation, and 2 others: Blog posts about new Selenium framework features - https://phabricator.wikimedia.org/T191982 (10zeljkofilipin) [13:19:17] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Update Selenium/Node.js/Target Custom MediaWiki - https://phabricator.wikimedia.org/T210518 (10zeljkofilipin) [13:19:28] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Update files required by Selenium in core, extensions and skins - https://phabricator.wikimedia.org/T210726 (10zeljkofilipin) [13:19:39] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10Browser-Tests, 10User-zeljkofilipin: Usage instructions in tests/selenium/README.md are confusing - https://phabricator.wikimedia.org/T214708 (10zeljkofilipin) [13:19:58] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Patch-For-Review, 10User-zeljkofilipin: Cypress testing framework evaluation - https://phabricator.wikimedia.org/T230729 (10zeljkofilipin) [13:20:22] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10User-zeljkofilipin: Cypress testing framework evaluation - https://phabricator.wikimedia.org/T230729 (10zeljkofilipin) [13:22:16] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-beta-REPOSITORY Jenkins jobs broken since 2019-09-08 - https://phabricator.wikimedia.org/T232357 (10zeljkofilipin) [13:22:33] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin) [13:22:41] (03PS1) 10Hashar: Remove r-lang / ortiz job [integration/config] - 10https://gerrit.wikimedia.org/r/538265 (https://phabricator.wikimedia.org/T153856) [13:23:00] !log Deleting Jenkins agent integration-r-lang-01 (unused) # T153856 [13:23:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:23:03] T153856: Add lint/CI to all wikimedia/discovery analytics repositories - https://phabricator.wikimedia.org/T153856 [13:23:15] (03CR) 10Hashar: [C: 03+2] Remove r-lang / ortiz job [integration/config] - 10https://gerrit.wikimedia.org/r/538265 (https://phabricator.wikimedia.org/T153856) (owner: 10Hashar) [13:25:18] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Discovery, and 3 others: Add lint/CI to all wikimedia/discovery analytics repositories - https://phabricator.wikimedia.org/T153856 (10hashar) 05Open→03Declined Re declining, we did a... [13:25:28] tech debt cleanup! [13:25:39] (03Merged) 10jenkins-bot: Remove r-lang / ortiz job [integration/config] - 10https://gerrit.wikimedia.org/r/538265 (https://phabricator.wikimedia.org/T153856) (owner: 10Hashar) [13:27:56] PROBLEM - Host integration-r-lang-01 is DOWN: CRITICAL - Host Unreachable (172.16.3.27) [13:29:02] ^^ expected (I deleted it) [13:30:03] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Make Scrum of Scrums meeting notes easier to read - https://phabricator.wikimedia.org/T227176 (10zeljkofilipin) [13:31:01] 10Release-Engineering-Team (Other / Uncategorized), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Make Scrum of Scrums meeting notes easier to read - https://phabricator.wikimedia.org/T227176 (10zeljkofilipin) [13:31:51] 10Release-Engineering-Team (Other / Uncategorized), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Make Scrum of Scrums meeting notes easier to read - https://phabricator.wikimedia.org/T227176 (10zeljkofilipin) [13:45:18] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Epic, and 2 others: Enable mediawiki/mediawiki-phan-config on all Wikimedia-deployed repositories - https://phabricator.wikimedia.org/T224783 (10Umherirrender) [13:46:26] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Migrate contint* hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224591 (10hashar) From a quick chat with @MoritzMuehlenhoff : python2... [13:46:39] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [13:47:09] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [13:47:43] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MinervaNeue, 10Readers-Web-Backlog, 10User-zeljkofilipin: selenium-daily-beta-Minerva and MobileFrontend has failed for 6 days - https://phabricator.wikimedia.org/T232708 (10zeljkofilipin) [13:48:02] 10Release-Engineering-Team-TODO (201909), 10MobileFrontend, 10Readers-Web-Backlog, 10User-zeljkofilipin: MobileFrontend and Minerva Selenium tests failing with "Promise timeout" - https://phabricator.wikimedia.org/T233193 (10zeljkofilipin) [13:49:13] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [13:56:59] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin) [14:03:58] PROBLEM - Host integration-slave-docker-1041 is DOWN: CRITICAL - Host Unreachable (172.16.1.36) [14:07:12] PROBLEM - Host integration-slave-docker-1040 is DOWN: CRITICAL - Host Unreachable (172.16.3.86) [14:46:49] !sal [14:46:49] https://tools.wmflabs.org/sal/releng [14:47:27] !log Pooled integration-agent-docker-1002 integration-agent-docker-1003 based on Stretch # T226233 [14:47:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:47:31] T226233: Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 [15:01:34] PROBLEM - Host integration-slave-docker-1043 is DOWN: CRITICAL - Host Unreachable (172.16.6.184) [15:04:25] 10Gerrit, 10Accessibility, 10Patch-For-Review: Gerrit red-green diff make the code hard to read (impossible for color-blind users) - https://phabricator.wikimedia.org/T232893 (10Krinkle) >>! In T232893#5502664, @kostajh wrote: > Hmm, the blue in the elegent theme (after) screenshot looks like user-selected t... [15:04:36] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin) [15:14:26] !log Pooled integration-agent-docker-1004 based on Stretch # T226233 [15:14:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:14:29] T226233: Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 [15:16:20] !log Pooled integration-agent-docker-1005 based on Stretch # T226233 [15:16:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:18:23] 10Gerrit, 10Accessibility, 10Patch-For-Review: Gerrit red-green diff make the code hard to read (impossible for color-blind users) - https://phabricator.wikimedia.org/T232893 (10Krinkle) [15:55:50] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin) [16:04:03] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10QuickSurveys, 10Browser-Tests, 10User-zeljkofilipin: selenium-QuickSurveys browser test went from 20 seconds to 1+ minute - https://phabricator.wikimedia.org/T233430 (10zeljkofilipin) [16:05:59] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10QuickSurveys, 10Browser-Tests, 10User-zeljkofilipin: selenium-QuickSurveys browser test went from 20 seconds to 1+ minute - https://phabricator.wikimedia.org/T233430 (10zeljkofilipin) {F30408311} [16:06:17] 10Release-Engineering-Team (Unit & Int & System Tooling), 10QuickSurveys, 10Browser-Tests, 10Ruby, 10User-zeljkofilipin: selenium-QuickSurveys browser test went from 20 seconds to 1+ minute - https://phabricator.wikimedia.org/T233430 (10zeljkofilipin) [16:06:50] 10Release-Engineering-Team, 10QuickSurveys, 10Browser-Tests, 10Ruby, 10User-zeljkofilipin: selenium-QuickSurveys browser test went from 20 seconds to 1+ minute - https://phabricator.wikimedia.org/T233430 (10zeljkofilipin) [16:07:05] 10Release-Engineering-Team, 10QuickSurveys, 10Browser-Tests, 10Ruby: selenium-QuickSurveys browser test went from 20 seconds to 1+ minute - https://phabricator.wikimedia.org/T233430 (10zeljkofilipin) [16:07:40] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10QuickSurveys, 10Browser-Tests, 10Ruby: selenium-QuickSurveys browser test went from 20 seconds to 1+ minute - https://phabricator.wikimedia.org/T233430 (10zeljkofilipin) [16:08:44] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10QuickSurveys, 10Browser-Tests, 10Ruby: selenium-QuickSurveys browser test went from 20 seconds to 1+ minute - https://phabricator.wikimedia.org/T233430 (10zeljkofilipin) Since this a long deprecated #ruby framewo... [16:08:55] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10QuickSurveys, 10Browser-Tests, 10Ruby: selenium-QuickSurveys browser test went from 20 seconds to 1+ minute - https://phabricator.wikimedia.org/T233430 (10zeljkofilipin) p:05Triage→03Low [16:33:23] the gerrit config allows us to list more than one LDAP server.. but docs do not say anything whether this means round-robin or primary/fallback [16:33:48] depending which it is we should set 2 servers or just the one in the local DC [16:34:04] longma: hey! sorry, but i didn't have time this week at all to review your patch, i'll try to get to it early next week [16:39:16] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201909): Upgrade Gerrit to 2.15.15 - https://phabricator.wikimedia.org/T229110 (10hashar) Upstream has released a 2.15.17 > Gerrit version 2.15.17 is now available. This release includes a few minor fixes since 2.1... [16:42:16] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10QuickSurveys, 10Browser-Tests, 10Ruby: selenium-QuickSurveys browser test went from 20 seconds to 1+ minute - https://phabricator.wikimedia.org/T233430 (10hashar) I filled that task solely because the time went u... [16:47:06] mobrovac: okay, thank you! [16:57:26] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201909): Upgrade Gerrit to 2.15.17 - https://phabricator.wikimedia.org/T229110 (10Paladox) [17:06:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Cloud-VPS (Quota-requests): Raise quota for integration project - https://phabricator.wikimedia.org/T233440 (10hashar) [17:25:14] (03PS1) 10Umherirrender: [JsonConfig] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/538305 (https://phabricator.wikimedia.org/T224768) [17:26:15] (03PS1) 1020after4: make branch.py work for wmf branches [tools/release] - 10https://gerrit.wikimedia.org/r/538306 [17:29:13] (03PS2) 1020after4: make branch.py work for wmf branches [tools/release] - 10https://gerrit.wikimedia.org/r/538306 [17:30:19] (03CR) 10Thcipriani: [C: 03+2] make branch.py work for wmf branches [tools/release] - 10https://gerrit.wikimedia.org/r/538306 (owner: 1020after4) [17:31:15] (03Merged) 10jenkins-bot: make branch.py work for wmf branches [tools/release] - 10https://gerrit.wikimedia.org/r/538306 (owner: 1020after4) [17:42:19] (03CR) 10Jforrester: "> I can't find an example now, but pretty sure I've seen some persistent zuul cloning errors for a nonexistent repo "npm-test", which will" [integration/quibble] - 10https://gerrit.wikimedia.org/r/538020 (https://phabricator.wikimedia.org/T233143) (owner: 10Hashar) [18:11:42] (03PS2) 10Umherirrender: [JsonConfig] Add phan dependency and phan [integration/config] - 10https://gerrit.wikimedia.org/r/538305 (https://phabricator.wikimedia.org/T224768) [18:17:33] twentyafterfour: Notice: /Stage[main]/Packages::Python_pygerrit2/Package[python-pygerrit2]/ensure: created [18:18:00] mutante: awesome, thanks! will use that next tuesday to do the branch cut [18:18:25] yw [18:48:49] Yippee, build fixed! [18:48:49] Project mwcore-phpunit-coverage-master build #186: 09FIXED in 3 hr 48 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/186/ [19:06:18] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations: Reimage cobalt and gerrit2001 as buster - https://phabricator.wikimedia.org/T176774 (10Dzahn) p:05Lowest→03Normal [19:06:29] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations: Reimage cobalt and gerrit2001 as buster - https://phabricator.wikimedia.org/T176774 (10Dzahn) a:03Dzahn [19:10:44] 10Gerrit, 10Release-Engineering-Team, 10Accessibility, 10Patch-For-Review: Gerrit red-green diff make the code hard to read (impossible for color-blind users) - https://phabricator.wikimedia.org/T232893 (10Krinkle) a:05Krinkle→03thcipriani [19:11:14] 10Gerrit, 10Release-Engineering-Team, 10Accessibility, 10Patch-For-Review: Gerrit red-green diff make the code hard to read (impossible for color-blind users) - https://phabricator.wikimedia.org/T232893 (10Krinkle) p:05Normal→03Low Handing the stick over for deployment. [19:38:44] (03CR) 10Jforrester: [C: 03+2] [JsonConfig] Add phan dependency and phan [integration/config] - 10https://gerrit.wikimedia.org/r/538305 (https://phabricator.wikimedia.org/T224768) (owner: 10Umherirrender) [19:40:15] (03Merged) 10jenkins-bot: [JsonConfig] Add phan dependency and phan [integration/config] - 10https://gerrit.wikimedia.org/r/538305 (https://phabricator.wikimedia.org/T224768) (owner: 10Umherirrender) [19:41:28] !log Zuul: [JsonConfig] Add phan dependency and phan [19:41:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:41:43] !log Zuul: Drop ortiz-rlang (merged but not deployed) [19:41:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:44:04] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Epic, and 2 others: Enable mediawiki/mediawiki-phan-config on all Wikimedia-deployed repositories - https://phabricator.wikimedia.org/T224783 (10Jdforrester-WMF) [19:57:10] 10Project-Admins, 10MediaWiki-Page-Diffs, 10MediaWiki-Page-History: Split #MediaWiki-History-or-Diffs into #MediaWiki-Page-History and #MediaWiki-Page-Diffs - https://phabricator.wikimedia.org/T209651 (10Aklapper) 05Open→03Resolved This is done. Thanks everyone! * https://phabricator.wikimedia.org/tag/me... [20:07:02] 10Project-Admins, 10MediaWiki-Page-Diffs, 10MediaWiki-Page-History, 10Developer-Advocacy (Jul-Sep 2019): Split #MediaWiki-History-or-Diffs into #MediaWiki-Page-History and #MediaWiki-Page-Diffs - https://phabricator.wikimedia.org/T209651 (10Aklapper) [20:41:53] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909): CI mediawiki/core run times have increased since July 26th - https://phabricator.wikimedia.org/T232626 (10Jdforrester-WMF) Quibble 0.0.35 (with selenium de-duping) has been released and is being used since 2019-09-17, but the p95 hasn't... [20:50:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Cloud-VPS (Quota-requests): Raise quota for integration project - https://phabricator.wikimedia.org/T233440 (10thcipriani) To add some more data/justification for this ta... [20:51:55] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909), 10CirrusSearch, 10Discovery-Search (Current work), 10Patch-For-Review: Unit tests in mediawiki-config repo don't pass under PHP72, only HHVM - https://phabricator.wikimedia.org/T232691 (10Jdforrester-WMF) 05Open→03Resolved [20:55:12] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909): CI mediawiki/core run times have increased since July 26th - https://phabricator.wikimedia.org/T232626 (10thcipriani) Latest graph, FYI: {F30410393} [20:56:31] hashar: BTW, how are the jenkins jobs running on the new docker agents? I thought they are pinned to "DebianJessieDocker"? [21:04:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul, 10Patch-For-Review: zuul-server should not start on spare server when the Debian package is upgraded - https://phabricator.wikimedia.org/T233391 (10hashar) 05Ope... [21:04:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-merger fails to fetch from Gerrit - https://phabricator.wikimedia.org/T233390 (10hashar) [21:05:26] James_F: oh [21:05:33] James_F: annoying, the label is not updated [21:05:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: zuul-server should not start on spare server when the Debian package is upgraded - https://phabricator.wikimedia.org/T233391 (10Jdforrester-WMF) [21:05:50] the OS name in the label is irrelevant nowada [21:05:53] Ha. [21:05:59] So they're claiming to be Jessie but aren't? [21:06:01] I guess we could just apply a "docker" label on every node [21:06:13] then change the jobs to switch from DebianJessieDocker to that new Docker label [21:06:13] Yeah, unless we worry about future docker changes e.g. buster? [21:06:19] WFM. [21:06:21] unlikely [21:06:29] How do we set the label? In puppet? [21:06:42] on each of the nodes [21:06:57] Oh, manually in jenkins? [21:07:01] https://integration.wikimedia.org/ci/label/DebianJessieDocker/ [21:07:07] edit eeach of them [21:07:10] add a "Docker" label [21:07:27] in JJB replace DebianJessieDocker with Docker [21:07:41] edit each of the computers again and remove the, now unused, DebianJessieDocker label [21:07:43] OK, will do. [21:07:59] I think I also filled a task that lists all the labels [21:08:04] to figure out which one we still need [21:16:33] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Upgrade to Gerrit 2.16.12 - https://phabricator.wikimedia.org/T200739 (10Paladox) [21:17:58] hashar: We still have contintLabsSlave labels? [21:20:01] (03PS1) 10Jforrester: jjb: Re-label 'DebianJessieDocker' to just 'Docker' [integration/config] - 10https://gerrit.wikimedia.org/r/538335 [21:21:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Cloud-VPS (Quota-requests): Raise quota for integration project - https://phabricator.wikimedia.org/T233440 (10bd808) Current quota/utilization: `22 / 37 instances. 125 /... [21:26:34] (03CR) 10Jforrester: [C: 03+2] jjb: Re-label 'DebianJessieDocker' to just 'Docker' [integration/config] - 10https://gerrit.wikimedia.org/r/538335 (owner: 10Jforrester) [21:28:07] hashar: Ha, there's a bunch of jobs at https://integration.wikimedia.org/ci/label/DebianJessieDocker/ still. I guess this means they're all dead jobs as they're not longer defined in jjb and I should delete them? [21:28:25] hmm [21:28:25] Oh, but mediawiki-tarball-builder ran an hour ago? [21:28:27] not sure [21:28:37] (03Merged) 10jenkins-bot: jjb: Re-label 'DebianJessieDocker' to just 'Docker' [integration/config] - 10https://gerrit.wikimedia.org/r/538335 (owner: 10Jforrester) [21:28:48] differential-docker-test was for scap / differential . twentyafterfour would know [21:29:00] mwgate-npm-node-6-docker that got migrated by James to node10 [21:29:13] mediawiki-tarball-builder, I have no clue. I guess it just poll continuously mediawiki/core [21:29:23] but if you go to the job https://integration.wikimedia.org/ci/job/mediawiki-tarball-builder/ [21:29:33] on the side bar there is a Job Config History link [21:29:41] which might trace back to whoever created the job [21:29:49] Pushed by legoktm on 1 November last year. [21:29:54] Never changed since. [21:30:07] maybe that is a pending change in integration/config [21:30:12] not sure whether it is still needed [21:30:18] php-compile-hhvm-jessie-docker can be deleted [21:30:21] we no more have jessie [21:30:27] Yeah. [21:30:35] !log jjb: Dropping Schedule a Build [21:30:36] puppet-zookeeper-rake-docker forgot to delete it I guess [21:30:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:30:38] etc [21:30:44] oh [21:30:55] Bah. [21:31:01] Hello [21:31:04] the wmf-sonar-scanner* they were for the codeheath thingie [21:31:06] I was summoned [21:31:08] !log jjb: Dropping php-compile-hhvm-jessie-docker [21:31:09] they are probably no more needed [21:31:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:31:20] legoktm: yeah about https://integration.wikimedia.org/ci/job/mediawiki-tarball-builder/ :) [21:31:22] legoktm: Do you remember what the mediawiki-tarball-builder jjb job was for? [21:31:50] !log jjb: Dropping mwgate-npm-node-6-docker [21:31:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:32:45] !log jjb: Dropping puppet-zookeeper-rake-docker [21:32:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:33:18] !log jjb: Dropping wikimedia-fundraising-crm-composer-php56-docker [21:33:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:33:50] All the sonar stuff is now done in the codehealth jobs. [21:34:00] !log jjb: Dropping wmf-sonar-* [21:34:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:35:00] !log jjb: Dropping differential-docker-test [21:35:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:35:23] :] [21:35:34] OK, just legoktm's one left. [21:36:19] oh [21:36:24] and before the label disappear [21:36:26] https://integration.wikimedia.org/ci/label/DebianJessieDocker/load-statistics :D [21:36:45] we have gained 2 instances by rebuilding from 32G instances to 24G instances [21:36:46] Oh, yeah, we'll lose that. :-( [21:36:50] yeah [21:36:55] so possibly that label can be renamed [21:36:58] then jobs changed [21:36:59] but really [21:37:02] it does not matter [21:37:23] Label's can't be renamed, FWICT. [21:37:41] :] [21:38:01] ah indeed [21:38:05] the input box was misleading [21:38:12] Yeah. [21:38:17] Hmmm [21:38:37] anyway I am off and heading to bed, happy afternoon [21:38:46] See you, hashar. [21:44:23] legoktm: Surely you can instantly remember. I mean, it's only from 11 months ago! ;-) [21:45:15] !log jjb: All executors moved from DebianJessieDocker to Docker [21:45:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:45:18] legoktm: James_F https://gerrit.wikimedia.org/r/#/c/integration/config/+/471036/ [21:45:31] Yeah. [21:45:39] annyway sleep time for real [21:45:46] thx for the label fix :] [21:45:50] Did it ever work? If yes, should we merge it? [21:46:01] no clue, that is for legoktm ;] [21:47:05] (03PS6) 10Jforrester: [WIP] mediawiki-tarball-builder job [integration/config] - 10https://gerrit.wikimedia.org/r/471036 (https://phabricator.wikimedia.org/T208527) (owner: 10Legoktm) [21:47:35] !log jjb: Deleting mediawiki-tarball-builder; follow-up can happen in I52d40ae453 [21:47:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:50:02] (03CR) 10Jforrester: "PS6: Rebase. Note that I deleted the job from CI for now, but it can be re-pushed." [integration/config] - 10https://gerrit.wikimedia.org/r/471036 (https://phabricator.wikimedia.org/T208527) (owner: 10Legoktm) [21:52:58] Yeah ill take a look later [21:53:02] Sorry about that [21:59:47] No worries at all. :-) [22:20:41] 10Project-Admins: Create GLOW Project board - https://phabricator.wikimedia.org/T233461 (10Moushira) [22:41:05] 10Continuous-Integration-Infrastructure, 10MediaWiki-General, 10Performance-Team (Radar): Quibble should not rely on dynamically detecting the value of $wgServer - https://phabricator.wikimedia.org/T233140 (10Krinkle) [22:41:12] 10Continuous-Integration-Infrastructure, 10MediaWiki-General: Quibble should not rely on dynamically detecting the value of $wgServer - https://phabricator.wikimedia.org/T233140 (10Krinkle) [22:59:52] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909), 10Operations, 10Release Pipeline, and 3 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10Jdforrester-WMF) Done? [23:00:58] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10serviceops, 10Kubernetes, 10User-fsero: add CI job into operations/deployments-charts repo that helm lint packages and perform the helm index after merge. - https://phabricator.wikimedia.org/T216049 (10Jdforrester-WMF) 05Open→... [23:01:00] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909), 10Operations, 10Release Pipeline, and 3 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10Jdforrester-WMF) [23:58:20] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds