[00:01:16] I imagine if we don't do something, we're going to end up with numerous bug reports if MF is broken [00:33:38] 10Project-Admins: New WMF user can't see Herald - https://phabricator.wikimedia.org/T227050 (10MBinder_WMF) [01:50:03] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [05:41:33] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MW-1.33-release, 10Patch-For-Review: Release MW 1.33 - https://phabricator.wikimedia.org/T220652 (10RazeSoldier) There is a small problem with the release notes. I already push a patch. (03PS1) 10Awight: [WIP] playing with step generators [integration/quibble] - 10https://gerrit.wikimedia.org/r/520171 [05:49:15] (03CR) 10jerkins-bot: [V: 04-1] [WIP] playing with step generators [integration/quibble] - 10https://gerrit.wikimedia.org/r/520171 (owner: 10Awight) [06:55:03] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:08:39] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Operations, and 4 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff [07:25:11] 10Project-Admins: New WMF user can't see Herald - https://phabricator.wikimedia.org/T227050 (10Aklapper) 05Open→03Invalid See https://www.mediawiki.org/wiki/Phabricator/Help/Herald_Rules [08:24:44] hashar: Morning :) I would love to nudge this forwards, it's a substantive change that probably needs discussion: https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/510709/ [08:34:36] awight: hi. Yeah I am hoping to look into it this week :-] [08:36:26] Rad. It's only a minor priority for us, there are some browser tests blocked on a customization of LocalSettings.php, which can only happen when our browser tests are run in isolation, you know the deal... [08:36:52] Actually, the way I'm editing LocalSettings.php is irresponsible, since it can cause interference with other tests. [08:37:54] 10Continuous-Integration-Config: ext-gmp missing from docker-registry.wikimedia.org/releng/quibble-stretch-php72:0.0.32-2 - https://phabricator.wikimedia.org/T227043 (10hashar) [08:46:02] 10Continuous-Integration-Config: ext-gmp missing from docker-registry.wikimedia.org/releng/quibble-stretch-php72:0.0.32-2 - https://phabricator.wikimedia.org/T227043 (10hashar) https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WebAuthn/+/508788/ https://integration.wikimedia.org/ci/job/mwext-php72-phan-doc... [08:52:41] (03PS1) 10Hashar: Add ext-gmp for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/520192 (https://phabricator.wikimedia.org/T227043) [08:53:17] (03CR) 10Hashar: [C: 03+2] Add ext-gmp for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/520192 (https://phabricator.wikimedia.org/T227043) (owner: 10Hashar) [08:53:47] 10Continuous-Integration-Config, 10Patch-For-Review: ext-gmp missing from docker-registry.wikimedia.org/releng/quibble-stretch-php72:0.0.32-2 - https://phabricator.wikimedia.org/T227043 (10hashar) Should be good. ` INFO:jenkins_jobs.builder:Number of jobs generated: 10 INFO:jenkins_jobs.builder:Reconfiguring... [08:54:52] 10Continuous-Integration-Config, 10Patch-For-Review: ext-gmp missing for Phan jobs based on php 7.2 - https://phabricator.wikimedia.org/T227043 (10hashar) [08:55:26] (03Merged) 10jenkins-bot: Add ext-gmp for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/520192 (https://phabricator.wikimedia.org/T227043) (owner: 10Hashar) [08:57:39] 10Continuous-Integration-Config, 10MediaWiki-extensions-OATHAuth, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Contractor - Doing), 10Patch-For-Review: Enable Phan and seccheck on WebAuthn extension - https://phabricator.wikimedia.org/T226737... [08:57:41] 10Continuous-Integration-Config: mwext-php72-phan-docker running composer under PHP 7.0 - https://phabricator.wikimedia.org/T226752 (10hashar) 05Resolved→03Open There is a problem with that change. The jjb macro `docker-setup-mwext-for-phan` is used by both php7.0 and php7.2 based jobs. So when editing it, i... [09:06:36] (03PS1) 10Hashar: jjb: drop REL1_27 from quibble-integration [integration/config] - 10https://gerrit.wikimedia.org/r/520194 [09:09:46] (03CR) 10Hashar: [C: 03+2] jjb: drop REL1_27 from quibble-integration [integration/config] - 10https://gerrit.wikimedia.org/r/520194 (owner: 10Hashar) [09:12:03] (03Merged) 10jenkins-bot: jjb: drop REL1_27 from quibble-integration [integration/config] - 10https://gerrit.wikimedia.org/r/520194 (owner: 10Hashar) [09:32:36] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 4 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10hashar) [09:49:00] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:51:08] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:52:29] 10Gerrit, 10Security: Cannot assign user name "XXX" to account ####; name already in use. - https://phabricator.wikimedia.org/T216605 (10akosiaris) ΟΚ, LGTM then. [10:24:55] 10Continuous-Integration-Infrastructure, 10Browser-Tests: Run browser tests in parallel - https://phabricator.wikimedia.org/T226869 (10zeljkofilipin) > Is it reasonable to run multiple chromedrivers on one machine? How much more memory will this require? I didn't test this, but I'm pretty sure we could reuse... [11:06:25] 10Phabricator, 10Cleanup, 10Repository-Admins: Do not remove Wikidata items when archiving extensions and skins - https://phabricator.wikimedia.org/T221771 (10Aklapper) I've updated https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/33/ by replacing T221771#5186613 with T221771#518... [11:11:55] 10Project-Admins: Replace tracking T40638 by new project tag for "Interface messages" - https://phabricator.wikimedia.org/T140077 (10Aklapper) a:05Danny_B→03None Unassigning to avoid cookie-licking. [11:12:24] 10Project-Admins: Replace tracking bug T5646 by new project tag "Feeds" - https://phabricator.wikimedia.org/T102495 (10Aklapper) a:05Danny_B→03None Unassigning to avoid cookie-licking. [11:12:28] 10Project-Admins: Convert T28213 Strip marker issues (tracking) to a "Strip-Marker" tag - https://phabricator.wikimedia.org/T127350 (10Aklapper) a:05Danny_B→03None Unassigning to avoid cookie-licking. [11:13:02] 10Phabricator, 10Project-Admins: Use Conpherence for project-admins requests - https://phabricator.wikimedia.org/T128528 (10Aklapper) 05Open→03Declined Declining as per T127640 [11:19:58] (03CR) 10Awight: [C: 03+2] "Everything looks good to me. Default behavior will be unchanged, new behavior is covered by tests." [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [11:20:43] (03Merged) 10jenkins-bot: Clone requirements from ext dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [11:21:15] (03CR) 10jenkins-bot: Clone requirements from ext dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [11:32:15] (03CR) 10Hashar: Clone requirements from ext dependencies (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [11:32:44] (03CR) 10Hashar: "Thank you. I have a few other things I need to roll before being to roll a deployment of that feature :-]" [integration/quibble] - 10https://gerrit.wikimedia.org/r/502286 (https://phabricator.wikimedia.org/T193824) (owner: 10Hashar) [11:33:22] hashar: ^ please lmk if it's helpful that I occasionally merge things, or if I should leave that to the adults ;-) [11:35:21] awight: yes it is definitely helpful :] [11:35:39] but I don't quite remember what was my migration plan for that feature :-\ [11:35:55] haha I think you left the dastardly plan in a gerrit comment [11:37:16] > The idea is to add a new set of jobs that have the feature flag --resolve-requires and do not have dependencies injected by CI (the EXT_DEPENDENCIES/SKIN_DEPENDENCIES). [11:37:37] > If some dependency is missing, surely the install will fail (it does not right now due to T225512) or some test will fail down the line. [11:37:37] T225512: install.php --with-extensions silently ignores extensions whose dependencies are not satisfied - https://phabricator.wikimedia.org/T225512 [11:37:47] > --fail-on-extra-requires is for a narrow specific use case which is the wmf-quibble jobs. They have a harcoded set of extensions, if one add an extra requirement in extension.json, the job would need to be updated as well. So that flag is merely to fail when a gap is detected between the hardcoded list of repo and the declared dependencies. Eg T204252 //Have dependencies of gated extensions in [11:37:50] T204252: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 [11:37:53] the gate//. [11:37:56] ^ your notes on PS3 [12:04:19] awight: I should write some documentation really :-@ [12:05:35] 10Continuous-Integration-Config: Setup CI for operations/debs/coredns.git - https://phabricator.wikimedia.org/T226700 (10hashar) p:05Triage→03Normal [12:06:44] (03PS1) 10Hashar: Add non voting debian-glue to operations/debs/coredns [integration/config] - 10https://gerrit.wikimedia.org/r/520219 (https://phabricator.wikimedia.org/T226700) [12:07:41] (03CR) 10Hashar: [C: 03+2] "It has a git buildpackage configuration file at least ( https://gerrit.wikimedia.org/r/#/c/operations/debs/coredns/+/519966/ )" [integration/config] - 10https://gerrit.wikimedia.org/r/520219 (https://phabricator.wikimedia.org/T226700) (owner: 10Hashar) [12:09:10] (03Merged) 10jenkins-bot: Add non voting debian-glue to operations/debs/coredns [integration/config] - 10https://gerrit.wikimedia.org/r/520219 (https://phabricator.wikimedia.org/T226700) (owner: 10Hashar) [12:11:52] (03Restored) 10Hashar: Update skin-tests template [integration/config] - 10https://gerrit.wikimedia.org/r/323245 (owner: 10Paladox) [12:12:11] (03PS5) 10Hashar: Update skin-tests template [integration/config] - 10https://gerrit.wikimedia.org/r/323245 (owner: 10Paladox) [12:12:21] (03Abandoned) 10Hashar: Update skin-tests template [integration/config] - 10https://gerrit.wikimedia.org/r/323245 (owner: 10Paladox) [12:16:52] 10Continuous-Integration-Config: Setup CI for operations/debs/coredns.git - https://phabricator.wikimedia.org/T226700 (10hashar) I have configured the debian glue job (non voting) and gave it a try on a dummy change https://gerrit.wikimedia.org/r/#/c/520221/ . The build fail though. There are a bunch of ` dpkg-... [12:23:35] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10fgiunchedi) >>! In T222166#5296941, @MoritzMuehlenhoff wrote: > @fgiunchedi Shall we close this task? the current jessie package is rolled... [12:42:40] 10Scap, 10Patch-For-Review: Send scap log directly to logstash via syslog input - https://phabricator.wikimedia.org/T86969 (10fgiunchedi) [13:34:10] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MW-1.33-release: Release MW 1.33 - https://phabricator.wikimedia.org/T220652 (10Reedy) [13:34:49] zeljkof: hi! I have a couple Gerrit changes for Scrum of Scrum. Where could I add them please? that is for fundraising team [13:36:12] hashar: either our team etherpdad, sos section https://etherpad.wikimedia.org/p/RelEngTeam-Weekly or sos etherpad, releng section https://etherpad.wikimedia.org/p/Scrum-of-Scrums [13:39:04] zeljkof: good! I have added them to https://etherpad.wikimedia.org/p/RelEngTeam-Weekly lines 150-153 [13:39:05] :) [13:39:46] hashar: thanks! :) [13:42:43] where are we on the branch cut? noticed a change that will mildly break dumps, and I'd like it held back somehow if possible (it's already merged to master) [13:43:11] apergos: 4th of july so there is no train this week :) [13:43:20] aaaaaahhhhhhhhhhhhh [13:43:25] * apergos heaves a sigh of relief [13:43:25] and the best way to hold a feature back would be a feature flag I guess :-]]] [13:43:30] but yeah [13:43:36] you get a free extra week! [13:43:46] no it's fine, almost certainly y next week there will be the tiniest of patches to fix this [13:44:20] I don't usually have warm fuzzy feelings about u.s. independence day but this year I'll make an exception :-D [13:45:40] (03PS3) 10Awight: [WIP] Introduce recursive planning [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 [13:45:42] (03PS1) 10Awight: Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 [13:45:44] (03PS1) 10Awight: Run clone steps regardless of --dry-run [integration/quibble] - 10https://gerrit.wikimedia.org/r/520241 [13:46:00] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Introduce recursive planning [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 (owner: 10Awight) [13:46:03] (03CR) 10jerkins-bot: [V: 04-1] Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (owner: 10Awight) [13:46:05] (03CR) 10jerkins-bot: [V: 04-1] Run clone steps regardless of --dry-run [integration/quibble] - 10https://gerrit.wikimedia.org/r/520241 (owner: 10Awight) [13:54:38] (03PS2) 10Awight: Reduce side-effects [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 [13:56:44] (03CR) 10Awight: "This is just a clean-up, should have no effect on the logic." [integration/quibble] - 10https://gerrit.wikimedia.org/r/520240 (owner: 10Awight) [13:59:40] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 4 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) [14:01:19] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MW-1.33-release: Release MW 1.33 - https://phabricator.wikimedia.org/T220652 (10Reedy) a:03Reedy [14:03:15] 10Continuous-Integration-Config: Setup CI for operations/debs/coredns.git - https://phabricator.wikimedia.org/T226700 (10hashar) 05Open→03Resolved a:03hashar [14:06:37] 10Continuous-Integration-Config, 10Patch-For-Review: ext-gmp missing for Phan jobs based on php 7.2 - https://phabricator.wikimedia.org/T227043 (10Reedy) Looks like we need ext-bcmath too... :/ https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/1261/console ` 09:55:10 Problem 1 09:55:10 -... [14:06:54] 10Continuous-Integration-Config: ext-gmp/ext-bcmath missing for Phan jobs based on php 7.2 - https://phabricator.wikimedia.org/T227043 (10Reedy) [14:23:42] 10Phabricator, 10Cleanup, 10Repository-Admins: Do not remove Wikidata items when archiving extensions and skins - https://phabricator.wikimedia.org/T221771 (10Kghbln) 05Open→03Resolved a:03Aklapper > Anything else left to do here? Not from my side. Thanks a ton for doing this! Closing as resolved. :) [14:37:42] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MW-1.33-release: Release MW 1.33 - https://phabricator.wikimedia.org/T220652 (10Reedy) 05Open→03Resolved [14:44:59] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 in CI containers - https://phabricator.wikimedia.org/T218719 (10Daimona) @Jdforrester-WMF Thanks, but was there something wrong with the PHP71+ jobs? [14:54:58] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 46.67% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:06:29] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Operations, 10SRE-Access-Requests: Request access to deployment cluster for Alaa Sarhan - https://phabricator.wikimedia.org/T223698 (10greg) >>! In T223698#5281575, @jbond wrote: > @greg are you able to approve this request? Approved. (sorry f... [15:19:24] 10Release-Engineering-Team-TODO (201907), 10Release, 10Train Deployments: 1.34.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T220738 (10greg) a:03jeena [15:28:30] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:30:54] 10Release-Engineering-Team (Local Dev), 10Release Pipeline, 10serviceops-radar, 10Patch-For-Review: Make helm chart template in deployment-charts support local development - https://phabricator.wikimedia.org/T226660 (10thcipriani) p:05Triage→03Normal a:03jeena Assigning to @jeena since she already ha... [15:59:26] (03PS2) 10Jforrester: [jjb] Codehealth: Skip MW install, generate code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/519641 (owner: 10Kosta Harlan) [16:12:35] 10Phabricator (Upstream), 10WMSE-Bug-Reporting-and-Translation-2019, 10Upstream: "Move Tasks to Column..." opens dialogue for moving to project - https://phabricator.wikimedia.org/T220585 (10epriestley) I think this will be substantially improved by the upstream change in (03CR) 10Jforrester: "\o/" [integration/config] - 10https://gerrit.wikimedia.org/r/520194 (owner: 10Hashar) [16:21:11] is zuul having problems? https://gerrit.wikimedia.org/r/519996 isn’t getting another gate-and-submit [16:23:24] 10Project-Admins: Reading teams would like a tag to identify spikes - https://phabricator.wikimedia.org/T138380 (10MBinder_WMF) Reanimating this zombie thread. Now that there are subtypes (Bug, Task, Deadline, etc), and boards can be filtered on said subtypes, I wonder if this tag should be deprecated to mitigat... [16:24:51] hm, https://gerrit.wikimedia.org/r/520205 isn’t getting gate-and-submit either, for no reason that I can discern… cc hashar James_F [16:25:05] 10Project-Admins: New WMF user can't see Herald - https://phabricator.wikimedia.org/T227050 (10MBinder_WMF) Gotcha, that makes sense (most people I know who have interest in Herald rules have batch access, too, I just never made the connection). > (Please use https://www.mediawiki.org/wiki/Talk:Phabricator/He... [16:25:50] Lucas_WMDE: oops :-\ [16:26:08] also https://gerrit.wikimedia.org/r/520267 I think? [16:26:32] hashar: do you know what’s wrong or should I create a task? [16:26:36] it got overloaded somehow [16:26:56] Lucas_WMDE: yeah a task never hurts :] [16:26:59] ok ^^ [16:27:39] 10Project-Admins: Reading teams would like a tag to identify spikes - https://phabricator.wikimedia.org/T138380 (10MBinder_WMF) One use case currently being employed are ticket templates/links. Subtypes would have to be a field that can be manipulated. I think this is via the form number, which may mean that the... [16:28:38] 10Zuul: Zuul is no longer adding +2ed changes to the gate-and-submit pipeline - https://phabricator.wikimedia.org/T227111 (10Lucas_Werkmeister_WMDE) [16:28:47] Lucas_WMDE: thx: [16:29:06] np [16:29:22] 10Release-Engineering-Team, 10Zuul: Zuul is no longer adding +2ed changes to the gate-and-submit pipeline - https://phabricator.wikimedia.org/T227111 (10Lucas_Werkmeister_WMDE) p:05Triage→03High [16:29:25] 10Release-Engineering-Team, 10Zuul: Zuul is no longer adding +2ed changes to the gate-and-submit pipeline - https://phabricator.wikimedia.org/T227111 (10Jdforrester-WMF) p:05High→03Unbreak! [16:29:47] No code can merge -> UBN!. :-) [16:30:02] fair enough [16:30:17] +1 [16:32:01] in fact, all the other pipelines look suspiciously empty as well [16:32:16] I think it’s not submitting starting jobs, period [16:32:33] s/submitting starting/starting any/ [16:32:47] You have to admit, it's much more efficient this way. ;-) [16:32:58] it’s definitely not going to be overloaded anymore [16:33:03] 10Release-Engineering-Team, 10Zuul: Zuul is no longer adding +2ed changes to the gate-and-submit pipeline - https://phabricator.wikimedia.org/T227111 (10jbond) at least one more https://gerrit.wikimedia.org/r/c/operations/puppet/+/519227 [16:34:00] 10Release-Engineering-Team, 10Zuul: Zuul is no longer adding +2ed changes to the gate-and-submit pipeline - https://phabricator.wikimedia.org/T227111 (10hashar) I am looking into it but I don't know yet what is happening. At least some jobs are running in Jenkins and Zuul does process some events. So it is not... [16:34:22] 10Release-Engineering-Team, 10Zuul: Zuul is no longer adding +2ed changes to the gate-and-submit pipeline - https://phabricator.wikimedia.org/T227111 (10Lucas_Werkmeister_WMDE) Other pipelines are also not getting filled – this probably isn’t specific to gate-and-submit, that’s just where it was more obvious.... [16:35:47] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding +2ed changes to the gate-and-submit pipeline - https://phabricator.wikimedia.org/T227111 (10Lucas_Werkmeister_WMDE) Let’s add “shared build failure” since that’s where people might look for “why... [16:39:28] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding +2ed changes to the gate-and-submit pipeline - https://phabricator.wikimedia.org/T227111 (10Paladox) I am seeing this: ` com.google.inject.ProvisionException: Unable to provision, see the follow... [16:39:38] hashar ^^ [16:39:43] thcipriani ^^ [16:42:27] huh [16:42:31] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding +2ed changes to the gate-and-submit pipeline - https://phabricator.wikimedia.org/T227111 (10hashar) `gerrit stream-events` works locally. The zuul scheduler and the couple zuul merger have some... [16:42:37] I guess I will not look for someone to +2 my change just this minute... [16:43:21] apergos: you could look for an extra courageous person to force-merge your change ;) [16:43:23] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding jobs to any jenkins pipelines - https://phabricator.wikimedia.org/T227111 (10Jdforrester-WMF) [16:43:29] nope [16:43:37] a regular merge is just fine thanks :-D [16:44:46] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10MobileFrontend, 10JavaScript, 10Readers-Web-Backlog (Tracking): Update MobileFrontend-npm-run-lint-modules-docker to run node10 - https://phabricator.wikimedia.org/T224997 (10Jdlrobson) [16:46:16] hashar: any known issues as to zuul not working atm? [16:47:13] I just got a large number of e-mails from gerrit all at once. [16:47:19] Possibly a deadlock got resolved? [16:47:23] hauskatze see https://phabricator.wikimedia.org/T227111 [16:47:26] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding jobs to any jenkins pipelines - https://phabricator.wikimedia.org/T227111 (10hashar) Only thing I can imagine is that hmm Zuul lost its connection to Gerrit some how :-\ [16:47:42] I've just +2ed a patch but it is not in any queue yet [16:47:51] and only 3 jobs on zuul that I can see [16:48:03] hauskatze: https://phabricator.wikimedia.org/T227111 [16:48:03] paladox: ah, that's it [16:48:11] et hashar [16:49:47] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding jobs to any jenkins pipelines - https://phabricator.wikimedia.org/T227111 (10hashar) Thread dump {P8701} [16:52:04] I am just going to restart zuul [16:52:58] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding jobs to any jenkins pipelines - https://phabricator.wikimedia.org/T227111 (10thcipriani) >>! In T227111#5300903, @hashar wrote: > Only thing I can imagine is that hmm Zuul lost its connection to... [16:57:45] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding jobs to any jenkins pipelines - https://phabricator.wikimedia.org/T227111 (10greg) `lang=irc 09:47:13 I just got a large number of e-mails from gerrit all at once. 09:47:19 Po... [16:58:03] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding jobs to any jenkins pipelines - https://phabricator.wikimedia.org/T227111 (10Lucas_Werkmeister_WMDE) p:05Unbreak!→03High Seems to be working again on https://gerrit.wikimedia.org/r/520252. [16:59:33] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding jobs to any jenkins pipelines - https://phabricator.wikimedia.org/T227111 (10hashar) So something somehow send a huge dependency loop to TimedMediaHandler and Zuul got confused. I noticed a spik... [17:00:02] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding jobs to any jenkins pipelines - https://phabricator.wikimedia.org/T227111 (10thcipriani) >>! In T227111#5300974, @greg wrote: > `lang=irc > 09:47:13 I just got a large number of e-mails... [17:13:17] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding jobs to any jenkins pipelines - https://phabricator.wikimedia.org/T227111 (10thcipriani) >>! In T227111#5300995, @thcipriani wrote: >>>! In T227111#5300974, @greg wrote: >> `lang=irc >> 09:47:13... [17:15:20] 10Release-Engineering-Team, 10Zuul, 10Wikimedia-production-error (Shared Build Failure): Zuul is no longer adding jobs to any jenkins pipelines - https://phabricator.wikimedia.org/T227111 (10hashar) 05Open→03Resolved a:03hashar The root cause is some changes for AbuseFilter. There is a serie of patche... [17:35:18] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 35.71% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:38:12] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:53:56] Hello everyone, I have an issue for proton but don't know if that's the best channel to ask [17:54:13] I've spotted a weird problem with node 6.11.x and proton. It's failing for code that is already in production https://integration.wikimedia.org/ci/job/chromium-render-npm-browser-node-6-docker/241/console [17:54:20] but prod is fine! [17:54:29] it's weird that it fails locally and for CI only with node 6 [17:56:13] mateusbs17: That's odd. You should probably file a task. Does that mean you can't push new docker containers? [17:58:16] Not sure I understand your question, but this means that I'm tied to node 8+ to commit any fix to proton and now it seems that the CI is too [17:59:01] Did the tests start failing without any changes in the repo? [17:59:09] Yes [17:59:48] Do you know when? [18:01:38] I don't know, but it looks like it has been around for a while [18:01:40] https://gerrit.wikimedia.org/r/c/mediawiki/services/chromium-render/+/516710 [18:01:51] 10Project-Admins: Reading teams would like a tag to identify spikes - https://phabricator.wikimedia.org/T138380 (10MBinder_WMF) As part of the ongoing conversation with myself here, I noticed that there is a form for Spikes, specifically, and it includes a deadline section. :) [18:01:54] ^ this patch doesn't change code and still breaks [18:03:12] Oh, yes, I never investigated that. [18:03:26] I assumed that you would know what was broken about the repo and I could rebase later. [18:04:08] I'll fill a task to investigate it. Thanks! [18:06:47] 10Gerrit, 10Security: Cannot assign user name "XXX" to account ####; name already in use. - https://phabricator.wikimedia.org/T216605 (10thcipriani) 05Open→03Resolved a:03thcipriani Applied patch and reindexed accounts 2019-07-02 17:53 UTC. @Mess your problem should be resolved. [18:08:00] someone lemme know when things have settled back down a bit [18:12:14] 10Release-Engineering-Team, 10Core Platform Team, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Services: [Bug] Proton fails in CI and locally for node 6.x - https://phabricator.wikimedia.org/T227124 (10MSantos) [18:16:27] 10Release-Engineering-Team, 10Core Platform Team, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Services: [Bug] Proton fails in CI and locally for node 6.x - https://phabricator.wikimedia.org/T227124 (10Jdforrester-WMF) There might be a breaking change upstream in one of the items npm pulls in? Beca... [18:16:42] apergos: Things are settled. [18:17:24] fishing for someone to give a thumbs up on https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/516787/ [18:17:30] (03PS1) 10Umherirrender: Add quibble for empty repostiory [integration/config] - 10https://gerrit.wikimedia.org/r/520289 [18:17:44] it's not that there's a rush but if I don't find a taker it will just sit there forever [18:19:25] (03PS2) 10Umherirrender: Add quibble for empty repository [integration/config] - 10https://gerrit.wikimedia.org/r/520289 [18:20:42] apergos: C+2'ed. [18:20:50] <3 [18:23:33] (03PS2) 10Kosta Harlan: Skip MediaWiki install for composer-test job, run unit tests after [integration/config] - 10https://gerrit.wikimedia.org/r/519644 [18:27:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Regularly run mwext-{name}-testextension-* jobs to make sure they are still passing after core or dependency changes - https://phabricator.wikimedia.org/T93414 (10Tgr) So is this some... [18:41:11] 10Release-Engineering-Team, 10Core Platform Team, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Core Platform Team Kanban (Team 2): [Bug] Proton fails in CI and locally for node 6.x - https://phabricator.wikimedia.org/T227124 (10WDoranWMF) p:05Triage→03Normal [18:45:00] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 4 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) [18:47:42] 10Gerrit, 10Security: Cannot assign user name "XXX" to account ####; name already in use. - https://phabricator.wikimedia.org/T216605 (10Mess) >>! In T216605#5301271, @thcipriani wrote: > Applied patch and reindexed accounts 2019-07-02 17:53 UTC. > > @Mess your problem should be resolved. @thcipriani I confi... [18:48:22] 10Project-Admins: Reading teams would like a tag to identify spikes - https://phabricator.wikimedia.org/T138380 (10JMinor) Yeah, I think with the "Spike" ticket type we can remove the tag. My only question is would it apply retroactively (ie. would closed tickets with the Spike tag "lose" that tag?). That could... [18:48:35] at last my change has made it to the part of the zuul queue being executed... [18:50:54] 10Release-Engineering-Team, 10Core Platform Team, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Core Platform Team Kanban (Team 2): [Bug] Proton fails in CI and locally for node 6.x - https://phabricator.wikimedia.org/T227124 (10Jdforrester-WMF) OK, the last successful build of [[https://integration... [18:53:49] 10Release-Engineering-Team, 10Core Platform Team, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Core Platform Team Kanban (Team 2): [Bug] Proton fails in CI and locally for node 6.x - https://phabricator.wikimedia.org/T227124 (10Mholloway) Thanks for investigating, @Jdforrester-WMF. Looks like it's... [18:54:33] 10Release-Engineering-Team, 10Core Platform Team, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Core Platform Team Kanban (Team 2): [Bug] Proton fails in CI and locally for node 6.x - https://phabricator.wikimedia.org/T227124 (10Jdforrester-WMF) That'd be lovely! One fewer node6 image for me to have... [18:55:09] 10Release-Engineering-Team, 10Analytics: issues with artifact cache in an-coord1001 - https://phabricator.wikimedia.org/T227132 (10Nuria) [18:55:35] 10Release-Engineering-Team, 10Analytics: issues with artifact cache in an-coord1001 - https://phabricator.wikimedia.org/T227132 (10Ottomata) This also often affects other hosts with relatively small /srv partitions, like notebook* hosts. [18:55:54] 10Release-Engineering-Team, 10Analytics: issues with artifact cache in an-coord1001 - https://phabricator.wikimedia.org/T227132 (10Nuria) Can the release engineering chime as to whether scap config settings should also delete artifacts from the target of the deploy? [18:56:30] 10Release-Engineering-Team, 10Analytics: issues with artifact cache in an-coord1001 - https://phabricator.wikimedia.org/T227132 (10Nuria) ping @greg [19:01:46] (03CR) 10Kosta Harlan: [C: 04-1] "Going to rework this." [integration/config] - 10https://gerrit.wikimedia.org/r/519644 (owner: 10Kosta Harlan) [19:02:04] merged \o/ [19:02:10] (I'm easily pleased) [19:03:10] im glad polygerrit now has a footer extension point, allowed me to do https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/520295/6/modules/gerrit/files/homedir/review_site/static/gerrit-theme.html :) [19:04:21] 10Project-Admins: Reading teams would like a tag to identify spikes - https://phabricator.wikimedia.org/T138380 (10MBinder_WMF) > My only question is would it apply retroactively (ie. would closed tickets with the Spike tag "lose" that tag?). That could be a problem for record keeping/finding old tickets. I be... [20:07:13] 10Release-Engineering-Team, 10Core Platform Team, 10Proton, 10Core Platform Team Kanban (Team 2), and 2 others: [Bug] Proton fails in CI and locally for node 6.x - https://phabricator.wikimedia.org/T227124 (10MSantos) [20:14:31] 10Release-Engineering-Team, 10Proton, 10Reading-Infrastructure-Team-Backlog (Kanban): [Bug] Proton fails in CI and locally for node 6.x - https://phabricator.wikimedia.org/T227124 (10WDoranWMF) [20:21:02] (03PS3) 10Jforrester: layout: Add quibble for PopupAlerts, UnblockMe and Scribe [integration/config] - 10https://gerrit.wikimedia.org/r/520289 (owner: 10Umherirrender) [20:31:02] (03CR) 10Jforrester: [C: 03+2] layout: Add quibble for PopupAlerts, UnblockMe and Scribe [integration/config] - 10https://gerrit.wikimedia.org/r/520289 (owner: 10Umherirrender) [20:33:02] (03Merged) 10jenkins-bot: layout: Add quibble for PopupAlerts, UnblockMe and Scribe [integration/config] - 10https://gerrit.wikimedia.org/r/520289 (owner: 10Umherirrender) [20:34:05] !log Zuul: Adding quibble to the PopupAlerts, UnblockMe and Scribe extensions [20:34:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:36:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10Dzahn) > RAID1 over the new disks /dev/sdc and /dev/sdd apt-get install parted parted... [20:45:16] (03PS1) 10Jforrester: layout: [chromium-render] Enable pipeline tests [integration/config] - 10https://gerrit.wikimedia.org/r/520317 (https://phabricator.wikimedia.org/T225680) [21:18:52] 10Project-Admins: Create tags for all Wikimedia projects - https://phabricator.wikimedia.org/T154549 (10Aklapper) 05Stalled→03Declined Declining for the time being - see T154549#2930629. If //specific// tags are wanted, please file specific tasks and explaining how these tags will be used and by whom. Thanks! [21:22:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10Dzahn) >>! In T207707#5271473, @hashar wrote: > * a LVM volume group pvcreate /dev/md3... [21:22:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10Dzahn) ` root@contint1001:/mnt/docker# df -h Filesystem Size... [21:26:10] thcipriani, hashar: on contint1001 the 2 new disks are now a RAID1 with LVM on top, 931G volume group size.. and a 250G logical volume with ext4 mounted for now on /mnt/docker [21:27:05] mutante: awesome! thank you! [21:27:13] https://phabricator.wikimedia.org/T207707#5301898 so maybe from there you can take a look again .. the config part i mean [21:27:26] cool, yw [21:28:23] i dunno, it felt too confusing to mount the new volume on /srv/docker when the existing one is /srv/ so i used /mnt ..dunno [21:29:16] contint1001--vg-data (old) contint1001--data-docker (new) [21:30:25] I wonder why hash wanted a 250G lv... [21:31:04] > we might need an extra partition soonish as well. So maybe for the Docker images we can go with [21:31:06] ah [21:31:29] cool, thanks again, /mnt seems sane to me. [21:31:40] i think he reduced it from 500G to 250G in later comment [21:31:52] ok, great [21:34:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10Dzahn) a:05Dzahn→03hashar [21:36:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10Dzahn) Hi Hashar, at this point i think it makes sense to assign back to to you to chec... [22:04:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10Operations, 10serviceops: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10thcipriani) >>! In T207707#5302044, @Dzahn wrote: > Hi Hashar, at this point i think it... [22:15:52] (03CR) 10Jforrester: [C: 03+2] "Oops, I pushed this before but didn't merge." [integration/config] - 10https://gerrit.wikimedia.org/r/519641 (owner: 10Kosta Harlan) [22:18:11] (03Merged) 10jenkins-bot: [jjb] Codehealth: Skip MW install, generate code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/519641 (owner: 10Kosta Harlan) [22:34:00] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MW-1.31-release, 10MW-1.32-release, 10phan: Decide on future of running Phan tests on release branches - https://phabricator.wikimedia.org/T226945 (10Jdforrester-WMF)