[00:21:44] 10Phabricator, 10RelEng-Archive-FY201718-Q1, 10Wikimedia-Incident: Disallow blocked users on mediawiki to create accounts on phabricator - https://phabricator.wikimedia.org/T162996 (10bd808) [00:29:55] 10Release-Engineering-Team (Watching / External), 10Operations, 10puppet-compiler, 10Puppet: Integrate the puppet compiler in the puppet CI pipeline - https://phabricator.wikimedia.org/T166066 (10ayounsi) That's useful, thank you. It didn't work for https://gerrit.wikimedia.org/r/c/operations/puppet/+/397... [00:37:58] 10Project-Admins: Cleaning up #Mediawiki-extensions-other - https://phabricator.wikimedia.org/T213880 (10Isarra) Well, I just totally failed to find at least one LogoFunctions task I was specifically looking for in there because all I found was a different one, and filed a duplicate... I'd say if they've got mo... [00:38:47] 10Project-Admins: Cleaning up #Mediawiki-extensions-other - https://phabricator.wikimedia.org/T213880 (10Isarra) Anyway, that was annoying, so now we've made a project for that one, at any rate. >.> [01:33:49] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Patch-For-Review: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927 (10Krenair) I think we should try to get these commits merged and then call it done: * https://gerrit.w... [02:22:38] (03PS1) 10Kosta Harlan: (wip) report back quality gate result using polling method [integration/config] - 10https://gerrit.wikimedia.org/r/501465 [03:49:11] https://opendev.org/ [06:33:46] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [06:41:15] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations, 10Patch-For-Review: Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562 (10Dzahn) [06:41:21] 10Gerrit, 10Release-Engineering-Team (Next), 10DBA, 10Operations: Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working - https://phabricator.wikimedia.org/T176532 (10Dzahn) 05Resolved→03Open let's only resolve stuff that is actually resolved, not what will be resolved i... [06:43:48] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:14:57] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10Legoktm) >>! In T219086#5082664, @greg wrote: > The right answer here is to add Kunal to the Gerrit-Mangers group,... [07:37:26] Project beta-code-update-eqiad build #241537: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/241537/ [07:38:47] Yippee, build fixed! [07:38:48] Project beta-code-update-eqiad build #241538: 09FIXED in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/241538/ [07:44:05] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [08:03:26] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10Legoktm) a:05Legoktm→03None [08:05:11] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:05:28] (03PS1) 10DCausse: Add a gerrit plugin to CI [integration/config] - 10https://gerrit.wikimedia.org/r/501507 [08:15:47] (03PS2) 10Legoktm: Add tox job for labs/tools/ldap [integration/config] - 10https://gerrit.wikimedia.org/r/501416 (https://phabricator.wikimedia.org/T220151) (owner: 10Hashar) [08:16:19] (03CR) 10Legoktm: [C: 03+2] Add tox job for labs/tools/ldap [integration/config] - 10https://gerrit.wikimedia.org/r/501416 (https://phabricator.wikimedia.org/T220151) (owner: 10Hashar) [08:18:38] (03CR) 10jerkins-bot: [V: 04-1] Add tox job for labs/tools/ldap [integration/config] - 10https://gerrit.wikimedia.org/r/501416 (https://phabricator.wikimedia.org/T220151) (owner: 10Hashar) [08:19:43] eh [08:21:25] (03Merged) 10jenkins-bot: Add tox job for labs/tools/ldap [integration/config] - 10https://gerrit.wikimedia.org/r/501416 (https://phabricator.wikimedia.org/T220151) (owner: 10Hashar) [08:21:54] !log deploying https://gerrit.wikimedia.org/r/501416 [08:21:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:24:32] 10Continuous-Integration-Config, 10Tool-ldap, 10Patch-For-Review: Add CI checks to tool-ldap - https://phabricator.wikimedia.org/T220151 (10Legoktm) 05Open→03Resolved a:03hashar Thanks! [08:49:29] dcausse: good to see you jumped into gerrit java code ] [08:49:44] :) [08:50:00] (03CR) 10Hashar: "Someone the git-daemon on contint1001 considered the path to not be a repository. I have no idea why though :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/501416 (https://phabricator.wikimedia.org/T220151) (owner: 10Hashar) [08:50:25] dcausse: the mystery is how do we get it compiled against the gerrit version we use. But maybe it does not matter [08:50:48] the version is set in the pom [08:51:09] it's compiled against the gerrit-api artifact which I hope is left untouched by us [08:51:35] otherwize that'd we've made big customization to gerrit [08:51:50] otherwize that'd *mean* we've made big customizations to gerrit [08:51:58] (03CR) 10Hashar: [C: 03+2] "Job deployed. Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/501507 (owner: 10DCausse) [08:52:20] dcausse: so we get some gerrit from maven central don't we? [08:52:32] gerrit-api jar yes [08:52:32] I guess if you target gerrit 2.15 yes that should be fine [08:52:54] I targeted 2.15.12 but I've seen we'll be rolling back to .11 [08:52:54] I am super happy to see a java developer to step in gerrit [08:53:09] cause really, java is a big mystery for us releng :-\ [08:53:50] honestly java is easy compared to what you guys are doing... :) [08:54:02] * hashar blushes [08:54:33] shamelessly, I had the opportunity to have a first job for Sun in France for some java/staroffice thing at the time [08:54:48] but really. That sounded dated and all I really wanted to do is work for an ISP / Telco / internet [08:54:51] so dismissed it [08:55:01] else, I would probavbly know java very well by now [08:55:14] it's never too late :) [08:55:22] (or I will be a sales account manager for Oracle selling licenses per CPU) [08:55:25] ;D [08:55:49] (03Merged) 10jenkins-bot: Add a gerrit plugin to CI [integration/config] - 10https://gerrit.wikimedia.org/r/501507 (owner: 10DCausse) [08:56:31] !log Reloaded Zuul for operations/software/gerrit/plugins/barricade https://gerrit.wikimedia.org/r/#/c/integration/config/+/501507/ [08:56:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:57:29] (google amazes me, I wrongly search for "^pha" and I end up with an infobox about Polyhydroxyalcanoate [08:57:49] dcausse: I will give some feedback / use cases on the task :] [08:59:43] I mean. Just reading the test suite is already too many informations :D [09:45:54] (03Abandoned) 10Hashar: Fix support for pbr 1.x (change version to 2.5.1-1.wmf1) [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/412884 (https://phabricator.wikimedia.org/T162787) (owner: 10Paladox) [09:46:26] (03Abandoned) 10Hashar: Add tests for Extension:Blackout [integration/config] - 10https://gerrit.wikimedia.org/r/456667 (owner: 10Daimona Eaytoy) [09:47:51] 10Release-Engineering-Team (Watching / External), 10Operations, 10puppet-compiler, 10Puppet: Integrate the puppet compiler in the puppet CI pipeline - https://phabricator.wikimedia.org/T166066 (10jbond) > In addition, Jenkins doesn't seem to like having more than Change-id and Bug in the footer: Seems to... [09:48:10] (03Abandoned) 10Hashar: operations-dnslint: Update docker image [integration/config] - 10https://gerrit.wikimedia.org/r/480258 (owner: 10Thcipriani) [09:58:10] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Facilitate Scrum of scrums - https://phabricator.wikimedia.org/T220130 (10zeljkofilipin) [11:20:46] 10Phabricator, 10Documentation: Re-evaluate installed Wikimedia Phabricator applications - https://phabricator.wikimedia.org/T109186 (10Aklapper) [11:20:49] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Uninstall Conpherence - https://phabricator.wikimedia.org/T127640 (10Aklapper) 05Declined→03Open p:05Lowest→03Low a:03Aklapper Three years later, I'd like to re-evaluate our current use of Conpherence by going through the comments brought up in this... [11:21:03] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10Aklapper) [12:10:01] (03Abandoned) 10Hashar: Restore wmf-quibble-* jobs for REL1_XX branches [integration/config] - 10https://gerrit.wikimedia.org/r/473311 (https://phabricator.wikimedia.org/T189560) (owner: 10Legoktm) [12:13:59] (03Abandoned) 10Hashar: Rebuild java8 images for apt security update [integration/config] - 10https://gerrit.wikimedia.org/r/485959 (owner: 10Legoktm) [12:14:02] (03Abandoned) 10Hashar: jjb: Update java8 images for apt security update [integration/config] - 10https://gerrit.wikimedia.org/r/485960 (owner: 10Legoktm) [12:15:11] (03Abandoned) 10Hashar: (WIP) clone wmf branches for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/320792 (https://phabricator.wikimedia.org/T115713) (owner: 10Hashar) [12:15:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10Technical-Debt: Clone mediawiki into mediawiki-config when running test's via jenkins - https://phabricator.wikimedia.org/T115713 (10hashar) Sorry that will need to be revisited eventual... [12:16:49] (03Abandoned) 10Hashar: (WIP) Quibble job for release tarballs [integration/config] - 10https://gerrit.wikimedia.org/r/443578 (https://phabricator.wikimedia.org/T197469) (owner: 10Hashar) [12:19:54] (03CR) 10Sbisson: [C: 03+1] Add dependencies for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/495689 (owner: 10Daimona Eaytoy) [12:20:50] (03PS3) 10Hashar: utils: zuul layout generator [integration/config] - 10https://gerrit.wikimedia.org/r/493180 [12:21:19] (03CR) 10Hashar: [C: 03+2] utils: zuul layout generator [integration/config] - 10https://gerrit.wikimedia.org/r/493180 (owner: 10Hashar) [12:23:18] (03Merged) 10jenkins-bot: utils: zuul layout generator [integration/config] - 10https://gerrit.wikimedia.org/r/493180 (owner: 10Hashar) [12:25:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Zuul, 10Patch-For-Review, 10User-Addshore: Upgrade pbr for zuul - https://phabricator.wikimedia.org/T162787 (10hashar) [12:26:09] 10Continuous-Integration-Infrastructure, 10Operations-Software-Development, 10Patch-For-Review, 10cloud-services-team (Kanban): puppet broken on integration WMCS instances due to openstack Debian packages - https://phabricator.wikimedia.org/T218559 (10hashar) [12:30:01] (03CR) 10Hashar: [C: 03+2] [CleanChanges] Add seccheck [integration/config] - 10https://gerrit.wikimedia.org/r/496020 (owner: 10Umherirrender) [12:32:02] (03Merged) 10jenkins-bot: [CleanChanges] Add seccheck [integration/config] - 10https://gerrit.wikimedia.org/r/496020 (owner: 10Umherirrender) [12:34:39] (03CR) 10Hashar: [C: 04-1] [WikimediaMessages] Add dependencies for used classes (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/500116 (owner: 10Umherirrender) [12:35:02] (03CR) 10Hashar: [C: 03+2] [NavigationTiming] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/500124 (owner: 10Umherirrender) [12:37:01] (03Merged) 10jenkins-bot: [NavigationTiming] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/500124 (owner: 10Umherirrender) [12:46:55] (03CR) 10Hashar: "I have a feeling it should be the other way around and probably Echo should not be tied to OpenStackManager or Wikibase." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/495689 (owner: 10Daimona Eaytoy) [12:52:36] (03PS6) 10Hashar: Better arg handling [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 (https://phabricator.wikimedia.org/T218357) [13:00:05] (03CR) 10Sbisson: Add dependencies for Echo (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/495689 (owner: 10Daimona Eaytoy) [13:02:20] 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Quibble space separated options shallow arguments - https://phabricator.wikimedia.org/T218357 (10hashar) [13:22:47] (03PS1) 10Hashar: Emit a warning for EXT_DEPENDENCIES and SKIN_DEPENDENCIES [integration/quibble] - 10https://gerrit.wikimedia.org/r/501566 (https://phabricator.wikimedia.org/T220199) [13:23:48] (03CR) 10jerkins-bot: [V: 04-1] Emit a warning for EXT_DEPENDENCIES and SKIN_DEPENDENCIES [integration/quibble] - 10https://gerrit.wikimedia.org/r/501566 (https://phabricator.wikimedia.org/T220199) (owner: 10Hashar) [13:50:13] (03PS2) 10Umherirrender: [WikimediaMessages] Add dependencies for used classes [integration/config] - 10https://gerrit.wikimedia.org/r/500116 [13:50:24] (03CR) 10Umherirrender: [WikimediaMessages] Add dependencies for used classes (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/500116 (owner: 10Umherirrender) [14:14:43] 10Phabricator: Reports: Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T220207 (10GTirloni) [14:20:19] 10Phabricator: Reports: Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T220207 (10Aklapper) [14:20:22] 10Phabricator, 10Performance: /maniphest/report/project/ : Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T125357 (10Aklapper) [14:25:40] (03PS2) 10Kosta Harlan: wmf-sonar-scanner: Report back quality gate result using polling [integration/config] - 10https://gerrit.wikimedia.org/r/501465 (https://phabricator.wikimedia.org/T217008) [14:27:36] (03CR) 10Kosta Harlan: "Example output:" [integration/config] - 10https://gerrit.wikimedia.org/r/501465 (https://phabricator.wikimedia.org/T217008) (owner: 10Kosta Harlan) [15:31:11] (03PS1) 10Umherirrender: [TEI] Run seccheck [integration/config] - 10https://gerrit.wikimedia.org/r/501603 [15:31:12] (03CR) 10Gehel: [C: 03+1] "LGTM and is a nice step forward!" [integration/config] - 10https://gerrit.wikimedia.org/r/501465 (https://phabricator.wikimedia.org/T217008) (owner: 10Kosta Harlan) [15:35:19] 10Release-Engineering-Team (Kanban), 10Browser-Tests, 10MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), 10Patch-For-Review, and 4 others: CI failing: Steps that require login are failing - https://phabricator.wikimedia.org/T219920 (10zeljkofilipin) a:05zeljkofilipin→03None [15:50:38] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) [15:50:46] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) p:05Triage→03Normal [15:51:10] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) Blocked on {T199116}. [15:51:44] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) [15:52:27] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) 05Open→03Stalled [15:53:27] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) Ruby framework is still working, but failing sometimes, see {T219920}. Nobody has the time to debug failures. [15:55:21] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-ResourceLoader, 10Performance-Team: Run `maintenance/resources/manageForeignResources.php verify` as a test on MediaWiki core - https://phabricator.wikimedia.org/T203694 (10Krinkle) [15:57:49] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) To completely deprecate Ruby Selenium framework, I guess only a few Jenkins jobs should be deleted: - all... [15:57:52] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-ResourceLoader, 10Performance-Team: Run ForeignResourceManager verification on MediaWiki core commits - https://phabricator.wikimedia.org/T203694 (10Krinkle) [15:58:24] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10MediaWiki-Core-Testing, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) [16:36:19] 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, 10Traffic, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10Krenair) Ran into this in {... [16:44:31] (03CR) 10Hashar: [C: 03+2] [TEI] Run seccheck [integration/config] - 10https://gerrit.wikimedia.org/r/501603 (owner: 10Umherirrender) [16:47:26] (03Merged) 10jenkins-bot: [TEI] Run seccheck [integration/config] - 10https://gerrit.wikimedia.org/r/501603 (owner: 10Umherirrender) [16:49:48] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10JavaScript, 10MW-1.32-notes (WMF-deploy-2018-04-24 (1.32.0-wmf.1)), 10User-zeljkofilipin: Run Selenium Cucumber tests in CI - https://phabricator.wikimedia.org/T179190 (10Jdlrobson) [16:49:51] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Quibble, 10User-zeljkofilipin: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10Jdlrobson) 05Stalled→03Open I'd like to unstall thi... [16:51:03] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/501603 (owner: 10Umherirrender) [17:18:04] 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline, 10Core Platform Team Kanban (Done with CPT), 10Services (done): Track and install additional npm packages for all service container images - https://phabricator.wikimedia.org/T205911 (10mobrovac) FYI, [service-runner v2.6... [17:26:02] gerrit still not happy with the upgrade? ugh [17:26:15] I hope 2.16 goes more smoothly when it comes [17:27:16] apergos: There'll just be different bugs. [17:27:25] I'm sure there will [17:27:55] but there's degrees of severity... [17:35:41] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [17:58:59] (03PS1) 10Krinkle: zuul: Make the filter apply as-you-type [integration/docroot] - 10https://gerrit.wikimedia.org/r/501652 [17:59:07] (03CR) 10Krinkle: [C: 03+2] zuul: Make the filter apply as-you-type [integration/docroot] - 10https://gerrit.wikimedia.org/r/501652 (owner: 10Krinkle) [17:59:32] (03Merged) 10jenkins-bot: zuul: Make the filter apply as-you-type [integration/docroot] - 10https://gerrit.wikimedia.org/r/501652 (owner: 10Krinkle) [17:59:39] (03CR) 10jenkins-bot: zuul: Make the filter apply as-you-type [integration/docroot] - 10https://gerrit.wikimedia.org/r/501652 (owner: 10Krinkle) [18:01:25] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [18:02:29] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10greg) >>! In T219086#5087558, @Legoktm wrote: >>>! In T219086#5082664, @greg wrote: >> The right answer here is to... [18:05:14] 10Release-Engineering-Team (Next), 10Citoid, 10Release Pipeline, 10serviceops, and 2 others: Migrate citoid to kubernetes - https://phabricator.wikimedia.org/T213194 (10mobrovac) \o/ Thank you @akosiaris and @jijiki :) [18:53:10] PROBLEM - Host deployment-db04 is DOWN: CRITICAL - Host Unreachable (172.16.5.5) [19:07:38] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10AbuseFilter, 10CX-deployments, and 2 others: mediawiki/vendor REL1_* no more ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560 (10hashar) Sorry I have a long backlog. Yes the... [19:08:30] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10AbuseFilter, 10CX-deployments, and 2 others: mediawiki/vendor REL1_* no longer ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560 (10Reedy) [19:13:26] 10Continuous-Integration-Infrastructure, 10ContentTranslation: Merge Blocker: Test failures in WikibaseLexeme - https://phabricator.wikimedia.org/T220229 (10KartikMistry) [19:13:49] 10Continuous-Integration-Infrastructure, 10ContentTranslation: Merge Blocker: Test failures in WikibaseLexeme - https://phabricator.wikimedia.org/T220229 (10KartikMistry) p:05Triage→03Unbreak! [19:15:46] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Jade, 10Scoring-platform-team: Clone gerrit repo mediawiki/extensions/JADE to mediawiki/extensions/Jade - https://phabricator.wikimedia.org/T212180 (10mmodell) Should I lock and archive the old extension? [19:25:44] (03PS3) 10Kosta Harlan: wmf-sonar-scanner: Report back quality gate result using polling [integration/config] - 10https://gerrit.wikimedia.org/r/501465 (https://phabricator.wikimedia.org/T217008) [19:29:11] (03PS4) 10Kosta Harlan: wmf-sonar-scanner: Report back quality gate result using polling [integration/config] - 10https://gerrit.wikimedia.org/r/501465 (https://phabricator.wikimedia.org/T217008) [19:30:37] 10Continuous-Integration-Infrastructure, 10ContentTranslation, 10Lexicographical data, 10Wikidata: Merge Blocker: Test failures in WikibaseLexeme: Special:RecentChanges shows lemmas in title links to lexemes - https://phabricator.wikimedia.org/T220229 (10Nikerabbit) [19:41:47] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Jade, 10Scoring-platform-team: Clone gerrit repo mediawiki/extensions/JADE to mediawiki/extensions/Jade - https://phabricator.wikimedia.org/T212180 (10hashar) Yes it should be archived (and not deleted since we would lost changes and reviews made to it). #c... [19:47:04] (03PS5) 10Kosta Harlan: wmf-sonar-scanner: Report back quality gate result using polling [integration/config] - 10https://gerrit.wikimedia.org/r/501465 (https://phabricator.wikimedia.org/T217008) [19:52:48] (03PS1) 10Kosta Harlan: (wip) Invoke wmf-sonar-scanner after code coverage is generated [integration/config] - 10https://gerrit.wikimedia.org/r/501685 (https://phabricator.wikimedia.org/T218598) [19:54:43] (03PS5) 10Jforrester: [MoblileApps] Move from npm-docker to node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/478762 (https://phabricator.wikimedia.org/T211785) [19:54:45] (03PS1) 10Jforrester: docker: Provide a -deploy-node10-docker job template [integration/config] - 10https://gerrit.wikimedia.org/r/501687 [19:54:47] (03PS1) 10Jforrester: [MobileApps/Deploy] Add deploy-node10-docker job as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/501688 [19:54:49] (03PS1) 10Jforrester: [MobileApps/Deploy] Use node10-docker job, drop node6 ones [integration/config] - 10https://gerrit.wikimedia.org/r/501689 (https://phabricator.wikimedia.org/T211785) [19:54:51] (03CR) 10jerkins-bot: [V: 04-1] (wip) Invoke wmf-sonar-scanner after code coverage is generated [integration/config] - 10https://gerrit.wikimedia.org/r/501685 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [19:56:08] (03CR) 10jerkins-bot: [V: 04-1] [MoblileApps] Move from npm-docker to node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/478762 (https://phabricator.wikimedia.org/T211785) (owner: 10Jforrester) [19:56:13] (03CR) 10jerkins-bot: [V: 04-1] docker: Provide a -deploy-node10-docker job template [integration/config] - 10https://gerrit.wikimedia.org/r/501687 (owner: 10Jforrester) [19:56:15] (03CR) 10jerkins-bot: [V: 04-1] [MobileApps/Deploy] Add deploy-node10-docker job as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/501688 (owner: 10Jforrester) [19:56:29] (03CR) 10jerkins-bot: [V: 04-1] [MobileApps/Deploy] Use node10-docker job, drop node6 ones [integration/config] - 10https://gerrit.wikimedia.org/r/501689 (https://phabricator.wikimedia.org/T211785) (owner: 10Jforrester) [20:02:09] (03PS2) 10Kosta Harlan: Invoke wmf-sonar-scanner after PHP code coverage is generated [integration/config] - 10https://gerrit.wikimedia.org/r/501685 (https://phabricator.wikimedia.org/T218598) [20:03:26] (03PS1) 10Brennen Bearnes: Add Release Engineering ownership to dev-images [releng/dev-images] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/501693 [20:03:31] (03CR) 10jerkins-bot: [V: 04-1] Invoke wmf-sonar-scanner after PHP code coverage is generated [integration/config] - 10https://gerrit.wikimedia.org/r/501685 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [20:04:41] (03CR) 10Thcipriani: [C: 03+2] wmf-sonar-scanner: Report back quality gate result using polling [integration/config] - 10https://gerrit.wikimedia.org/r/501465 (https://phabricator.wikimedia.org/T217008) (owner: 10Kosta Harlan) [20:05:17] 10Beta-Cluster-Infrastructure: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10KartikMistry) [20:06:37] (03Merged) 10jenkins-bot: wmf-sonar-scanner: Report back quality gate result using polling [integration/config] - 10https://gerrit.wikimedia.org/r/501465 (https://phabricator.wikimedia.org/T217008) (owner: 10Kosta Harlan) [20:06:40] (03CR) 10Jforrester: "Halp." [integration/config] - 10https://gerrit.wikimedia.org/r/501687 (owner: 10Jforrester) [20:07:37] (03PS2) 10Thcipriani: Add Release Engineering ownership to releng/dev-images [releng/dev-images] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/501693 (owner: 10Brennen Bearnes) [20:07:46] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] Add Release Engineering ownership to releng/dev-images [releng/dev-images] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/501693 (owner: 10Brennen Bearnes) [20:09:39] (03PS1) 10Umherirrender: [GoogleNewsSitemap] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501707 [20:11:27] !log updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/501465 [20:11:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:12:52] (03PS1) 10Umherirrender: [FundraiserLandingPage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501709 [20:21:20] 10Continuous-Integration-Infrastructure: quibble-vendor-mysql-php73-docker fails on experimental run with filesystem error - https://phabricator.wikimedia.org/T220237 (10Umherirrender) [20:25:26] (03PS1) 10Umherirrender: [MapSources] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501714 [20:28:46] (03PS1) 10Umherirrender: [NewUserMessage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501782 [20:32:22] (03PS1) 10Umherirrender: [Quiz] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501784 [20:40:38] (03PS1) 10Umherirrender: [wikihiero] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501786 [20:45:46] hashar: legoktm: phan memory. please advice. go or no go on disabling it. [20:45:49] it's been too many days. [20:47:02] I replied on the task [20:47:14] either disable the progress bar (I could not find why it is enabled) [20:47:38] or set the env variable COLUMN which shortcircuit the logic which does a proc_open() and thus a fork() [20:48:11] Krinkle: https://phabricator.wikimedia.org/T219114#5084302 :) [20:49:18] hashar: I think you should make a decision, I think that's okay :) [20:49:32] It's been over a week with broken merges. Can always improve later. [20:54:16] (03PS1) 10Jforrester: [VisualEditor-MW] Move from custom job to extension-javascript-documention [integration/config] - 10https://gerrit.wikimedia.org/r/501789 [20:54:23] (03PS2) 10Jforrester: [VisualEditor-MW] Move from custom job to extension-javascript-documention [integration/config] - 10https://gerrit.wikimedia.org/r/501789 [20:55:18] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10Wikimedia-production-error (Shared Build Failure): Jenkins build for MediaWiki should fail when "PHP Warning" is emitted - https://phabricator.wikimedia.org/T214552 (10Krinkle) @Legoktm Aha, right. I forgot that we don't actually capture the PHP st... [20:55:23] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10Wikimedia-production-error (Shared Build Failure): Jenkins build for MediaWiki should fail when "PHP Warning" is emitted - https://phabricator.wikimedia.org/T214552 (10Krinkle) p:05Triage→03Normal [20:55:31] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10Quibble, 10Wikimedia-production-error (Shared Build Failure): Jenkins build for MediaWiki should fail when "PHP Warning" is emitted - https://phabricator.wikimedia.org/T214552 (10Krinkle) [20:55:36] Has zuul fallen over? No jobs seemingly being consumed. [20:56:02] Oh, hmm, I vaguely recall being told off and that it'd be gearman that's broken, and zuul is wonderful. Or something. :-) [20:56:47] Zuul is up and has nothing in its queue waiting to run. That's odd. [20:56:55] So Gearman issue, or Gerrit isn't sending anything. [20:57:04] Could be gerrit maintenance [20:57:14] 10Phabricator: Unhandled exception error on Phabricator Calendar - https://phabricator.wikimedia.org/T220241 (10RhinosF1) [20:59:27] 10Phabricator: Unhandled exception error on Phabricator Calendar - https://phabricator.wikimedia.org/T220241 (10RhinosF1) Device: iPhone 8 (MQ6H2B/A or A1905) OS Version: iOS 12.2 Chrome Version:73.0.3683.68 [21:00:09] I don't think there is maintenance happening now [21:00:48] there was a restart of it a few hours ago for a downgrade [21:02:14] (03CR) 10Kosta Harlan: "This doesn't work yet per the test errors, but wanted to get a quick review from Legoktm before proceeding any further on this." [integration/config] - 10https://gerrit.wikimedia.org/r/501685 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [21:03:35] Yeah, and it was running an hour ago, after the downgrade. [21:07:11] 10Phabricator: Unhandled exception error on Phabricator Calendar - https://phabricator.wikimedia.org/T220241 (10RhinosF1) [21:08:55] should we be asking thcipriani? [21:08:58] to look, I mean [21:09:44] (03CR) 10Kosta Harlan: "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/501685 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [21:11:23] (03PS1) 10Krinkle: zuul: Convert pgsql/sqlite branch filter to skip-if format [integration/config] - 10https://gerrit.wikimedia.org/r/501793 [21:15:42] * thcipriani does look [21:16:35] (03CR) 10Krinkle: "Potential differences:" [integration/config] - 10https://gerrit.wikimedia.org/r/501789 (owner: 10Jforrester) [21:16:41] (03CR) 10Krinkle: [C: 03+2] [VisualEditor-MW] Move from custom job to extension-javascript-documention [integration/config] - 10https://gerrit.wikimedia.org/r/501789 (owner: 10Jforrester) [21:18:03] bah [21:18:24] that sounds like a bah from someone who knows something [21:18:28] well, I can say stream-events are happening, and zuul doesn't see them, but there is a connection from zuul to gerrit [21:18:46] (03PS3) 10Krinkle: [VisualEditor-MW] Move from custom job to extension-javascript-documention [integration/config] - 10https://gerrit.wikimedia.org/r/501789 (owner: 10Jforrester) [21:18:56] (03CR) 10Krinkle: "removing dead zuul jobs override" [integration/config] - 10https://gerrit.wikimedia.org/r/501789 (owner: 10Jforrester) [21:18:59] (03CR) 10Krinkle: [C: 03+2] [VisualEditor-MW] Move from custom job to extension-javascript-documention [integration/config] - 10https://gerrit.wikimedia.org/r/501789 (owner: 10Jforrester) [21:19:20] I swear I havent upgraded it [21:19:21] no CI = my cue to take off early on a Friday [21:21:01] Krinkle: I will implement the fixes hashar suggested on the task, give it another day, and then revert the phan upgrade if it's still not fixed while we figure out how to get bigger CI instances [21:21:28] legoktm: we dont need more ram [21:21:51] legoktm: the other fix is to instruct the kernel to still honor the memory allocation by setting vm.overcommit = 1 (or 2 I cant remember) :D [21:22:06] is that safe to set on all CI instances? [21:22:11] but that is in the kernel and thus affects everything. So I dont really know how bad it is [21:22:35] but eventually, something else will fork() as well [21:22:43] so we would have to set that vm.overcommit anyway maybe [21:23:04] thcipriani: I am afraid it might be gerrit :( [21:23:39] https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=activeThreads [21:23:45] shows a higher thread count [21:23:48] and groups are stalling for me [21:24:06] which dont really explain why I can receive the events [21:24:08] while zuul does not [21:25:22] it does, at least this happened a few weeks ago (rest api side not ssh) [21:25:40] (03PS1) 10Legoktm: Hopefully reduce memory consumption by phan on MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/501794 (https://phabricator.wikimedia.org/T219114) [21:26:05] fwiw if stream events was not working, then wikibugs would also not be reporting things [21:26:14] yeah [21:28:04] sudo kill -SIGUSR2 24045 [21:28:09] that dumps a stacktrace in zuul [21:29:17] (03PS1) 10Legoktm: Bump MediaWiki core to mediawiki-phan:0.1.14 [integration/config] - 10https://gerrit.wikimedia.org/r/501796 [21:29:28] ... right, I should not be waiting for the zuul-layout test to come back, given that's the thing you guys are talking about isn't working. [21:29:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI / Zuul is no more processing events - https://phabricator.wikimedia.org/T220243 (10hashar) [21:30:05] https://phabricator.wikimedia.org/T220243 [21:30:11] oh uh [21:31:07] Could this be "accounts" locking again? [21:31:13] by that i mean the cache [21:31:26] 10Beta-Cluster-Infrastructure, 10serviceops, 10Core Platform Team Backlog (Later), 10Services (later): Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10mobrovac) p:05Triage→03Normal [21:31:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI / Zuul is no more processing events - https://phabricator.wikimedia.org/T220243 (10hashar) [21:32:29] Krinkle: related to the skip-if and branch filters. I wrote a very dumb test to check the filters https://gerrit.wikimedia.org/r/#/c/integration/config/+/500065/ might help to verify your changes [21:32:31] I don't see any thread in a blocked state, although I do see most threads are waiting on a lock [21:32:34] Krinkle: the patch is bad though [21:33:00] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Later), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10mobrovac) I wonder if we can have an automatic CI job that pushes the new code upon merge. Or,... [21:33:04] and https://phabricator.wikimedia.org/P8357 is the zuul trace [21:33:07] hashar: I verify it by looking at zuul-layout diff. [21:33:11] https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=activeThreads&period=jour [21:33:14] which I'll do later. [21:33:16] Krinkle: clever :] [21:33:42] I'm currently writing a python test to find unused job overrides and macros etc. [21:33:53] Krinkle: also have a look at: utils/zuul-layout-tester.sh zuul/layout.yaml [21:34:00] thanks [21:34:12] there might be a differ in that utils directory as well [21:34:24] anywy, I should look at that zuul issue [21:34:48] thcipriani this is all i see https://phabricator.wikimedia.org/F28585509 [21:34:57] it's okay, I'm patient. This is what I do when I'm blocked, or need to wait. So whenever I have 30 seconds, I look at the result in Gerrit/Jenkins from the day before, enough time for the job to run showing me the layout diff there. [21:35:01] for loading a simple group with not alot of members [21:35:24] it's been stucked for like 4+ mins [21:35:50] I've also noticed degraded perf on gerrit. E.g. autocompletion for adding reviewers timed out several times which is unusual. [21:36:09] was the spike after 05:00 or so the issue that caused today's rollback to an earlier gerrit? [21:36:20] yes [21:36:32] * thcipriani takes thread dump, restarts gerrit [21:38:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI / Zuul is no more processing events - https://phabricator.wikimedia.org/T220243 (10hashar) So Zuul has two connection established: ` contint1001$ sudo netstat -tp|grep gerrit tcp6 0 0 contint1001.wikim:53720 gerrit.wiki... [21:38:06] thcipriani: hold on please [21:38:14] hashar: too late [21:38:17] :D [21:38:31] https://phabricator.wikimedia.org/T220243#5089612 [21:39:04] seems zuul is blocked on reading a query it has made for some change [21:40:05] gerrit query --format json --all-approvals --comments --commit-message --current-patch-set --dependencies --files --patch-sets --submit-records 501765 [21:40:25] was what I saw in the queue from jenkins-bot [21:41:56] zuul's working now [21:41:57] https://integration.wikimedia.org/zuul/ [21:44:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI / Zuul is no more processing events - https://phabricator.wikimedia.org/T220243 (10hashar) [2019-04-05 17:28:48,600 +0000] 2765248a jenkins-bot a/75 LOGIN FROM 2620:0:861:1:208:80:154:17 [2019-04-05 20:33:32,056 +0000] a746f4a4 j... [21:44:54] do we need to "recheck" stuff? [21:45:02] ohh [21:45:14] why do I always look at debug.log instead of error.log [21:46:50] (03PS1) 10Legoktm: Use custom npm-php job for wikipeg [integration/config] - 10https://gerrit.wikimedia.org/r/501807 [21:47:42] (03CR) 10Legoktm: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/501794 (https://phabricator.wikimedia.org/T219114) (owner: 10Legoktm) [21:47:54] (03CR) 10Legoktm: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/501796 (owner: 10Legoktm) [21:48:21] hmm [21:48:29] 2019-04-05 21:38:34,745 ERROR zuul.GerritEventConnector: Exception moving Gerrit event: [21:48:33] error: [Errno 111] Connection refused [21:48:55] (03CR) 10jerkins-bot: [V: 04-1] Use custom npm-php job for wikipeg [integration/config] - 10https://gerrit.wikimedia.org/r/501807 (owner: 10Legoktm) [21:50:13] (03PS2) 10Legoktm: Use custom npm-php job for wikipeg [integration/config] - 10https://gerrit.wikimedia.org/r/501807 [21:51:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI / Zuul is no more processing events - https://phabricator.wikimedia.org/T220243 (10hashar) Then looking in the Zuul error.log: ` 2019-04-05 21:38:34,742 ERROR zuul.GerritEventConnector: Exception moving Gerrit event: Traceback (mo... [21:51:27] I assume now is not a good time to deploy zuul layout changes? [21:54:08] (03CR) 10Legoktm: [C: 03+2] "Thanks. Long-term (probably in a year or two) once we have significantly expanded phan coverage and it has autofix capability, I think we " [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/487033 (owner: 10Thiemo Kreuz (WMDE)) [21:54:11] on a friday? :D [21:54:32] I normally deploy CI changes on weekends :) [21:54:47] I just meant if you're still looking at the logs and debugging I don't want to get in the way [21:55:27] (03Merged) 10jenkins-bot: Streamline PHPDoc comment parsing in UnusedUseStatement sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/487033 (owner: 10Thiemo Kreuz (WMDE)) [21:55:54] (03CR) 10jenkins-bot: Streamline PHPDoc comment parsing in UnusedUseStatement sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/487033 (owner: 10Thiemo Kreuz (WMDE)) [21:56:56] (03PS1) 10Legoktm: Release 25.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/501809 [21:57:13] James_F|Away: whenever you're around ^ [21:57:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI / Zuul is no more processing events - https://phabricator.wikimedia.org/T220243 (10hashar) 05Open→03Resolved Solved by restarting #gerrit due to an issue which I think is already tracked somewhere else. [21:58:19] (03CR) 10Hashar: [C: 03+1] Hopefully reduce memory consumption by phan on MediaWiki core (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/501794 (https://phabricator.wikimedia.org/T219114) (owner: 10Legoktm) [21:58:41] legoktm: I completely missed the container ran phan -p and thus enabled the progress bar there :/ [21:58:58] it is too late for me to rebuild the image, but feel free to do so [21:59:14] will do [21:59:24] the main reason it has -p is for people (aka me) running the container locally [21:59:30] but that's not really a good enough [21:59:37] reason to keep it [21:59:40] (03CR) 10Legoktm: [C: 03+2] Hopefully reduce memory consumption by phan on MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/501794 (https://phabricator.wikimedia.org/T219114) (owner: 10Legoktm) [22:01:47] (03Merged) 10jenkins-bot: Hopefully reduce memory consumption by phan on MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/501794 (https://phabricator.wikimedia.org/T219114) (owner: 10Legoktm) [22:02:25] !log rebuilding mediawiki-phan docker image https://gerrit.wikimedia.org/r/501794 [22:02:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:02:45] [contint1001.wikimedia.org] out: latest: Pulling from wikimedia-stretch [22:02:46] $<3>$<3> [22:02:48] that's new [22:02:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI / Zuul is no more processing events - https://phabricator.wikimedia.org/T220243 (10hashar) And for the record Gerrit had a thread for Zuul doing: SSH gerrit query --format json --all-approvals --comments --commit-message --cur... [22:04:13] legoktm: Yay. But can you actually run libraryupgrader right now? No gerrit HTTP Auth… [22:04:23] (03CR) 10Jforrester: [C: 03+2] Release 25.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/501809 (owner: 10Legoktm) [22:05:00] no, I'll just bump MediaWiki core manually for now to unblock the rest of the phan patches [22:05:02] (03Merged) 10jenkins-bot: Release 25.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/501809 (owner: 10Legoktm) [22:05:29] (03CR) 10jenkins-bot: Release 25.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/501809 (owner: 10Legoktm) [22:05:34] Kk. [22:05:51] I think 26.0.0 shouldn't be too far off, there's a bunch of tickets from parsoid folks that I/others need to address [22:07:44] I imagine we'll be working on 7.3 soon enough. [22:08:06] Which will have its own set of new MWCS tickets, probably. [22:08:08] 7.3 seems to have some segfault with redis [22:08:16] when using the mw import dump script [22:08:24] OK, maybe not so soon. :-) [22:08:30] it never does end, does it [22:08:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI / Zuul is no more processing events - https://phabricator.wikimedia.org/T220243 (10hashar) Or it is because the cache were being flushed and its a redhearing SSH gerrit flush-caches --cache accounts (thcipriani) sun.misc.Unsafe.p... [22:09:18] This was the dump i took of it James_F https://phabricator.wikimedia.org/P8359 [22:10:29] legoktm: Eventually the heat death of the Universe will come as blessèd relief from fixes. [22:10:35] (03CR) 10Legoktm: "Looks fine for the postmerge -publish jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/501685 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [22:10:51] kostajh: ^ hopefully that's helpful [22:12:37] (03CR) 10Legoktm: "> also, while reviewing this, I'm interested to hear your thoughts on the best way to incorporate JavaScript code coverage." [integration/config] - 10https://gerrit.wikimedia.org/r/501685 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [22:14:25] (03CR) 10Krinkle: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/501793 (owner: 10Krinkle) [22:14:28] (03CR) 10Krinkle: [C: 03+2] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/501789 (owner: 10Jforrester) [22:15:22] legoktm: yeah the fab task to rebuild the docker images now always pull the images provided by SRE. to make sure they are up to date [22:15:33] legoktm: I should be better at announcing changes sorry :/ [22:15:47] awesome [22:15:50] no worries :) [22:15:51] that is because docker-pkg does not pull images it does not control [22:16:32] (03Merged) 10jenkins-bot: [VisualEditor-MW] Move from custom job to extension-javascript-documention [integration/config] - 10https://gerrit.wikimedia.org/r/501789 (owner: 10Jforrester) [22:23:34] Krinkle: If you deploy the VE-MW one I'll verify it's working. [22:24:00] huh, recheck also retook the =2 [22:24:02] ok [22:24:05] wasn't ready for that yet [22:24:08] rolling out now [22:24:15] now that we disabled the phan progress bar it no longer outputs the memory usage... [22:24:32] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501789 [22:24:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:24:36] James_F: done [22:24:39] Krinkle: Any comment on a C+2'ed task by someone that voted C+2 will always try to re-run gate-and-submit. [22:25:05] Yeah, Gerrit is always transmitting the current state now, not just the part you added/changed. [22:25:09] Since 2.15 I think [22:25:12] Still have to get used to it. [22:25:20] Yup. [22:25:27] Quite often I'll see it say Krinkle: C: +2, this; Krinkle: C: +2, that. When neither time I pressed +2 [22:25:33] (03CR) 10Legoktm: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 1" [integration/config] - 10https://gerrit.wikimedia.org/r/501796 (owner: 10Legoktm) [22:28:15] (03Merged) 10jenkins-bot: Bump MediaWiki core to mediawiki-phan:0.1.14 [integration/config] - 10https://gerrit.wikimedia.org/r/501796 (owner: 10Legoktm) [22:34:21] (03CR) 10Jforrester: docker: Provide a -deploy-node10-docker job template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/501687 (owner: 10Jforrester) [22:37:33] (03PS1) 10Krinkle: utils: Add zuul-layout shaker [integration/config] - 10https://gerrit.wikimedia.org/r/501812 [22:37:48] (03CR) 10Krinkle: "Output:" [integration/config] - 10https://gerrit.wikimedia.org/r/501812 (owner: 10Krinkle) [22:38:45] (03PS1) 10Krinkle: zuul: Remove unused "^(mwgate-|)composer-validate$" override [integration/config] - 10https://gerrit.wikimedia.org/r/501813 [22:38:57] (03CR) 10jerkins-bot: [V: 04-1] utils: Add zuul-layout shaker [integration/config] - 10https://gerrit.wikimedia.org/r/501812 (owner: 10Krinkle) [22:39:40] Krinkle: the event for your comment (eg: "recheck") also carries the votes you might have set previously [22:40:17] hashar: Yeah, that wasn't the case in older versions of Gerrit. It's fine though, it means we get "re+2" for free basically. [22:40:23] instead of removing +2, and re-adding after zuul issue [22:40:37] oh [22:40:57] i think someone from openstack did that change. [22:42:18] https://gerrit.wikimedia.org/r/#/c/integration/config/+/501789/ [22:42:22] so on that change [22:43:00] I guess the event is handled twice so :] [22:43:10] once by test since it is a recheck [22:43:21] and once by gate-and-submit since the change already had +2 [22:44:09] yep [22:44:57] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 9 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10Jdlrobson) [22:50:37] (03CR) 10Jforrester: "Confirmed working." [integration/config] - 10https://gerrit.wikimedia.org/r/501789 (owner: 10Jforrester) [22:52:34] (03PS2) 10Krinkle: zuul: Convert pgsql/sqlite branch filter to skip-if format [integration/config] - 10https://gerrit.wikimedia.org/r/501793 [22:55:36] Krinkle: Oops, hadn't spotted that that hadn't merged. [23:10:47] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10Krinkle) [23:16:55] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 9 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10Jdlrobson) I was surprised to see this ta... [23:20:02] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10Legoktm) [23:28:38] legoktm: I'm disappointed BTW that you didn't say you'd "whitelisted more Phannotations". ;-) [23:28:48] LOL [23:29:06] maybe next time we have more to whitelist :p [23:32:33] 10Phabricator: Unhandled exception error on Phabricator Calendar - https://phabricator.wikimedia.org/T220241 (10JJMC89) I have replicated the exception while logged in and logged out using the below browsers on Windows 10. - Google Chrome Version 73.0.3683.86 (64-bit) - Firefox Quantum 66.0.2 (64-bit) [23:41:18] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 9 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10JJMC89) [23:51:14] 10Phabricator, 10Release-Engineering-Team (Kanban): Unhandled exception error on Phabricator Calendar - https://phabricator.wikimedia.org/T220241 (10mmodell) a:03mmodell [23:53:47] !log Beta cluster puppetmaster is stalled behind origin/production as of 24 hours ago (57 patches behind) due to a local merge conflict [23:53:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:54:10] twentyafterfour: not sure what the usual process for this is, do you know? [23:54:43] Krinkle: for fixing beta puppetmaster? [23:54:56] yeah, is there an alert and runbook for it, who/when [23:55:22] I don't know of any. I think what is normally done is to just reset the local conflict [23:55:37] if there is a commit then contact the author [23:55:54] Krinkle: I can try to take care of it if you'd like [23:56:12] I don't know which one is conflicting. Noticed it while cherry-picking something. [23:56:17] I'm logged off from the puppetmaster now. [23:56:28] there is https://phabricator.wikimedia.org/T135427