[02:52:03] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [06:57:04] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:01:59] (03PS1) 10Hashar: Make debian glue voting on python-git-archive-all [integration/config] - 10https://gerrit.wikimedia.org/r/523099 [08:02:39] (03PS2) 10Hashar: Make debian glue voting on python-git-archive-all [integration/config] - 10https://gerrit.wikimedia.org/r/523099 [08:02:50] (03CR) 10Hashar: [C: 03+2] Make debian glue voting on python-git-archive-all [integration/config] - 10https://gerrit.wikimedia.org/r/523099 (owner: 10Hashar) [08:03:45] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [tools/scap] - 10https://gerrit.wikimedia.org/r/522370 (owner: 10Hashar) [08:04:04] (03CR) 10jerkins-bot: [V: 04-1] Make debian glue voting on python-git-archive-all [integration/config] - 10https://gerrit.wikimedia.org/r/523099 (owner: 10Hashar) [08:04:16] (03CR) 10jerkins-bot: [V: 04-1] Make debian glue voting on python-git-archive-all [integration/config] - 10https://gerrit.wikimedia.org/r/523099 (owner: 10Hashar) [08:05:13] (03PS3) 10Hashar: Make debian glue voting on python-git-archive-all [integration/config] - 10https://gerrit.wikimedia.org/r/523099 [08:05:27] (03CR) 10Hashar: [C: 03+2] Make debian glue voting on python-git-archive-all [integration/config] - 10https://gerrit.wikimedia.org/r/523099 (owner: 10Hashar) [08:06:59] (03Merged) 10jenkins-bot: Make debian glue voting on python-git-archive-all [integration/config] - 10https://gerrit.wikimedia.org/r/523099 (owner: 10Hashar) [08:10:16] (03Abandoned) 10Hashar: layout: [WebAuthn] Skip HHVM in extension-quibble [integration/config] - 10https://gerrit.wikimedia.org/r/509529 (owner: 10Jforrester) [08:19:45] (03Abandoned) 10Hashar: (DO NOT SUBMIT) result of zuul-migrate [integration/config] - 10https://gerrit.wikimedia.org/r/517547 (owner: 10Hashar) [08:24:04] (03Abandoned) 10Hashar: quibble: Upgrade to phpunit-patch-coverage 0.0.10 [integration/config] - 10https://gerrit.wikimedia.org/r/495641 (owner: 10Legoktm) [08:29:12] (03PS6) 10Hashar: Remove gate-and-submit-swat from composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/510507 (owner: 10Jforrester) [08:29:33] (03CR) 10Hashar: [C: 03+2] "Rebased and I have dropped the gate-and-submit-swat pipeline entirely." [integration/config] - 10https://gerrit.wikimedia.org/r/510507 (owner: 10Jforrester) [08:31:08] (03Merged) 10jenkins-bot: Remove gate-and-submit-swat from composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/510507 (owner: 10Jforrester) [08:52:16] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10Services: Debian package for operations/software/service-checker FTBS due to missing tag upstream/0.1.5 - https://phabricator.wikimedia.org/T227859 (10hashar) Seems we would want to tag 0068b08e120d43b09a41115aa08b1043deb58f4b ` com... [08:52:27] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10Services: Debian package for operations/software/service-checker FTBS due to missing tag upstream/0.1.5 - https://phabricator.wikimedia.org/T227859 (10hashar) p:05Triage→03Normal [08:54:13] How does one access beta's logstash? [08:54:15] I followed instructions on https://www.mediawiki.org/wiki/Beta_Cluster#Testing_changes_on_Beta_Cluster [08:54:18] Times out when I ssh deployment-deploy01.deployment-prep.eqiad.wmflabs from login.tools.wmflabs.org [08:55:14] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10Quibble, 10Patch-For-Review: Quibble may need to rebuild localization cache before running tests - https://phabricator.wikimedia.org/T196347 (10hashar) 05Open→03Resolved I have made Quibble to populate the localization cache,... [08:55:58] dwalden: hi [08:56:05] dwalden: it is not on tool labs ;] [08:56:43] Ah, fair enough [08:56:55] dwalden: you would need your ssh configuration to pass through bastion.wmflabs.org [09:04:59] I don't appear to have access to bastion [09:11:10] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201907), 10Code-Stewardship-Reviews, 10Graphoid, and 3 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10akosiaris) > At the moment, engineering resources at the Foundation are committed to ot... [09:11:28] dwalden: Hi, I'm not sure but maybe this will help? https://wikitech.wikimedia.org/wiki/Help:Create_a_Wikimedia_developer_account#VPS_and_General_Users [09:12:38] It sounds like you already have a toolforge account though, and I'm unclear on whether the process is the same as for new users, or if you need to make an access request to change your existing account. #wikimedia-cloud might be a good place to ask. [09:13:27] Thanks! I'll ask [09:26:00] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201907), 10Services: Debian package for operations/software/service-checker FTBS due to missing tag upstream/0.1.5 - https://phabricator.wikimedia.org/T227859 (10akosiaris) 05Open→03Resolved a:03akosiaris https://gerrit.wikimedia.org/g/o... [09:34:10] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request access to deployment-prep - https://phabricator.wikimedia.org/T228021 (10dom_walden) [10:32:27] 10Release-Engineering-Team-TODO (201907), 10Wikidata, 10I18n, 10Performance-Team (Radar), and 2 others: [Regression wmf.13] Wikidata localisation is broken - https://phabricator.wikimedia.org/T227814 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved >>! In T227814#5329364, @thcipriani wrote: > Looking at ht... [10:32:32] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T220738 (10Lucas_Werkmeister_WMDE) [10:36:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Core-Testing, and 2 others: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) a:05zeljkofi... [10:37:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907), 10MediaWiki-Core-Testing, and 2 others: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10zeljkofilipin) 05Open→03S... [10:40:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201907): Selenium tests failing for Video repo...except nobody ever set them up in the first place so they shouldn't be running - https://phabricator.wikimedia.org/T219409 (10zel... [10:45:05] (03CR) 10Hashar: [C: 03+2] "Updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/522424 (https://phabricator.wikimedia.org/T227870) (owner: 10Daimona Eaytoy) [10:47:19] (03Merged) 10jenkins-bot: jjb: Use core mode for core-seccheck-docker [integration/config] - 10https://gerrit.wikimedia.org/r/522424 (https://phabricator.wikimedia.org/T227870) (owner: 10Daimona Eaytoy) [11:14:43] (03CR) 10Daimona Eaytoy: "> Updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/522424 (https://phabricator.wikimedia.org/T227870) (owner: 10Daimona Eaytoy) [11:15:56] 10Continuous-Integration-Config, 10phan-taint-check-plugin: mediawiki-core-php72-phan-seccheck-docker runs in extension mode - https://phabricator.wikimedia.org/T227870 (10Daimona) 05Open→03Resolved Another thing I noticed is that with check experimental, CI also runs a separate job for extensions, mwext-p... [12:44:24] 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Quibble, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 2 others: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10awight) Friends, I see... [12:53:52] (03PS1) 10Hashar: puppet compiler: bump threads 2 -> 4 [integration/config] - 10https://gerrit.wikimedia.org/r/523158 (https://phabricator.wikimedia.org/T221969) [12:55:35] !log Creating compiler1003.puppet-diffs.eqiad.wmflabs [172.16.2.46] m1.large / 4vCPUs # T221969 [12:55:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:55:40] T221969: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 [13:04:41] 10Continuous-Integration-Infrastructure, 10puppet-compiler: puppet compiler fails on releases1001.eqiad.wmnet - https://phabricator.wikimedia.org/T228047 (10hashar) [13:10:09] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:14:06] 10Continuous-Integration-Infrastructure, 10puppet-compiler, 10Patch-For-Review: puppet compiler fails on releases1001.eqiad.wmnet - https://phabricator.wikimedia.org/T228047 (10hashar) On releases1001.eqiad.wmnet in /var/lib/puppet/state/state.yaml there is a `Service[bacula-fd]` but no `Service[bacula]`. T... [13:14:29] 10Continuous-Integration-Infrastructure, 10Operations, 10puppet-compiler, 10Patch-For-Review: puppet compiler fails on releases1001.eqiad.wmnet due to lack of Service[bacula-director] - https://phabricator.wikimedia.org/T228047 (10hashar) [13:25:49] [ 1821.925513] rc.local[435]: [ 1822.516128] serial8250: too much work for irq4 [13:25:49] bah [13:27:35] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:30:35] !log Deleting compiler1003.puppet-diffs.eqiad.wmflabs .. insanely slow for some reason (was on cloudvirt1002) # T221969 ) [13:30:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:30:37] T221969: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 [13:31:34] (03PS1) 10Kosta Harlan: Zuul: Add MobileFrontend dependency for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/523164 [13:32:18] (03CR) 10Hashar: [C: 03+2] Zuul: Add MobileFrontend dependency for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/523164 (owner: 10Kosta Harlan) [13:32:43] that was fast [13:32:51] thx hashar [13:33:06] sometime :] [13:34:22] (03Merged) 10jenkins-bot: Zuul: Add MobileFrontend dependency for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/523164 (owner: 10Kosta Harlan) [13:39:27] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10SRE-Access-Requests: Request access to deployment cluster for Alaa Sarhan - https://phabricator.wikimedia.org/T223698 (10akosiaris) @alaa_wmde. Gentle reminder about generating and posting a separate SSH key... [13:46:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10puppet-compiler, 10Patch-For-Review: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar) [13:46:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10puppet-compiler, 10Patch-For-Review: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar) 05Open→03Stalled I tried provisioning a new instance but puppet takes... [13:53:14] 10Release-Engineering-Team (Deployment services), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Request access to deployment cluster for Jakob_WMDE - https://phabricator.wikimedia.org/T227193 (10akosiaris) 05Open→03Resolved a:03akosiaris Task has been opened for the required amount of days... [13:54:03] 10Release-Engineering-Team, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: issues with artifact cache in an-coord1001 - https://phabricator.wikimedia.org/T227132 (10Ottomata) Docs updated: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Deploy/Refinery#Deploying_to_notebook*_hosts [13:58:36] hashar: if https://gerrit.wikimedia.org/r/c/integration/quibble/+/522506 looks ok to you, any chance it could get deployed later this week? We want to announce the migration to MediaWikiUnitTestCase but currently none of the unit tests are run via the plain phpunit entrypoint in CI, and it would be nice to have [13:59:44] 10Continuous-Integration-Infrastructure, 10Operations, 10puppet-compiler, 10Patch-For-Review: puppet compiler fails on releases1001.eqiad.wmnet due to lack of Service[bacula-director] - https://phabricator.wikimedia.org/T228047 (10akosiaris) 05Open→03Invalid ` Warning: Could not find resource 'Service[... [14:22:37] kostajh: sorry was busy on something else [14:22:57] I guess master works, but is that the case for other branches? [14:23:04] 10Release-Engineering-Team, 10Analytics, 10Analytics-Kanban: issues with artifact cache in an-coord1001 - https://phabricator.wikimedia.org/T227132 (10Ottomata) I've manually removed a bunch of old refinery artifact jar versions from the refinery deploy on notebook hosts to free up space. [14:24:26] * hashar tries [14:24:27] hashar: yeah, if you pass a non-existent testsuite to `vendor/bin/phpunit` it will just exit 0 [14:24:35] ;] [14:24:41] unlike `composer {somescript}` which exits 1 if somescript doesn't exist [14:24:54] ah [14:24:56] and doesn't seem like composer has an `--if-exists` flag like npm, sadly [14:25:01] and CI is still at 0.0.32 [14:25:10] ie it does not make use of the recent phpunit unit magic [14:25:40] I guess I can just +2 and tag a new release [14:26:54] Time: 5.68 seconds, Memory: 64.00MB [14:26:59] OK (3576 tests, 20863 assertions) [14:27:00] lovely [14:27:27] * hashar tries ZUUL_BRANCH=REL1_31 quibble --git-cache /home/hashar/projects --db sqlite --run phpunit-unit [14:35:07] * awight leans forward. sorry to merge that one without considering release branches. [14:36:42] hashar: thx. It shouldn't need the `--db` flag, if you use `--skip-install` [14:38:37] kostajh: it fails on rel1_31 :\ [14:38:51] hmm, how? [14:39:10] https://phabricator.wikimedia.org/P8747 [14:39:39] not much informations :-\ [14:39:52] 10Phabricator, 10Documentation: Document workboard column triggers - https://phabricator.wikimedia.org/T227300 (10epriestley) Triggers currently //only// activate when a task card is dropped into a column with triggers (see [[ https://secure.phabricator.com/T5474#245062 | this comment ]] for details). They don... [14:40:31] * kostajh looks [14:41:32] I can reproduce it locally, so checking to see what's up [14:42:32] vendor/bin/phpunit --testsuite=fooaez [14:42:33] fails as well [14:42:44] so yeah it just exists when it does not find the testsuite ? :\ [14:43:28] hrm [14:43:31] (03CR) 10Hashar: "That fails on REL1_31 https://phabricator.wikimedia.org/P8747" [integration/quibble] - 10https://gerrit.wikimedia.org/r/522506 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [14:43:47] not sure why that works in master though [14:44:24] alright, well the alternative (probably preferable anyway) is this one https://gerrit.wikimedia.org/r/c/integration/quibble/+/521515 [14:45:55] the only thing (I think) missing there is mocking composer.json for the test [14:46:55] kostajh: or we skip that step unless there is a /phpunit.xml.dist file ? [14:47:49] kostajh: If you end up needing to mock composer.json in tests, here's an example: https://github.com/wikimedia/quibble/blob/master/tests/test_commands.py#L119 [14:47:54] yeah, that could work too [14:49:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10puppet-compiler, 10Patch-For-Review: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar) On compiler1003 I have applied `role::puppet_compiler` and the hiera conf... [14:49:31] awight: thanks, I looked at that but it's a bit beyond my python abilities + available time right now [14:49:51] kostajh: Feel free to ping me in CR, if I can help fill in any gaps. [14:50:30] awight: will do! [14:50:47] This unit test change is a nice upgrade to our CI, thanks for the push. [14:50:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10Technical-Debt: Investigate whether GD is still needed on appservers - https://phabricator.wikimedia.org/T227734 (10akosiaris) p:05Triage→03Normal [14:51:33] before we work on the test, do you all have a preference between the `composer phpunit:unit` patch (https://gerrit.wikimedia.org/r/c/integration/quibble/+/521515) or the `vendor/bin/phpunit --testsuite=...` patch (https://gerrit.wikimedia.org/r/c/integration/quibble/+/522506)? [14:51:57] the `composer phpunit:unit` one is nice in that we can change the definition of what gets executed without needing to modify quibble [14:52:34] OTOH, I don't anticipate needing to make any more changes, so `vendor/bin/phpunit --testsuite=...` is probably safe to go with. [14:53:20] One thing I like about the composer script is that it seems easier for casual developers to run. [14:53:53] it's also what the README and docs will point people to [14:53:59] (03Restored) 10Kosta Harlan: Don't run phpunit-unit stage if the composer script doesn't exist [integration/quibble] - 10https://gerrit.wikimedia.org/r/521515 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [14:54:17] (03CR) 10Kosta Harlan: "Adam, if you have time to help me with the unit test I'd very much appreciate it. thanks!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/521515 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [14:57:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10puppet-compiler, 10Patch-For-Review: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar) In hiera config: ` lang=diff - profile::puppetdb::master: compiler1002.pu... [14:57:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10puppet-compiler, 10Patch-For-Review: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar) [15:02:08] (03PS2) 10Awight: Don't run phpunit-unit stage if the composer script doesn't exist [integration/quibble] - 10https://gerrit.wikimedia.org/r/521515 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [15:02:11] kostajh: ^ :-) [15:02:57] awight: cheers! [15:03:33] (03Abandoned) 10Kosta Harlan: Use vendor/bin/phpunit instead of composer phpunit:unit [integration/quibble] - 10https://gerrit.wikimedia.org/r/522506 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [15:10:34] 10Phabricator: No results for specific string in Global Phab Search, while Advanced Search lists expected results - https://phabricator.wikimedia.org/T224082 (10epriestley) I believe this install is currently configured to use ElasticSearch for global search. Search in other interfaces (including Maniphest advan... [15:14:00] (03PS2) 10Jforrester: layout: Add seccheck for CentralAuth [integration/config] - 10https://gerrit.wikimedia.org/r/522911 (https://phabricator.wikimedia.org/T189227) (owner: 10Daimona Eaytoy) [15:14:05] (03CR) 10Jforrester: [C: 03+2] layout: Add seccheck for CentralAuth [integration/config] - 10https://gerrit.wikimedia.org/r/522911 (https://phabricator.wikimedia.org/T189227) (owner: 10Daimona Eaytoy) [15:14:23] 10Phabricator, 10Regression: Phabricator login page does not show Welcome text / "Click button below" text anymore - https://phabricator.wikimedia.org/T223926 (10epriestley) I think {nav Auth > Customize Messages > Login Screen Instructions} should now let you configure this. I expect this approach should prov... [15:15:20] 10Phabricator (Upstream), 10Upstream: Exception when accessing to a Phabricator mock: "Attempting to add more metadata after metadata has been locked." - https://phabricator.wikimedia.org/T218450 (10epriestley) [15:16:11] (03Merged) 10jenkins-bot: layout: Add seccheck for CentralAuth [integration/config] - 10https://gerrit.wikimedia.org/r/522911 (https://phabricator.wikimedia.org/T189227) (owner: 10Daimona Eaytoy) [15:16:36] 10Phabricator (Upstream), 10Upstream: Exception when accessing to a Phabricator mock: "Attempting to add more metadata after metadata has been locked." - https://phabricator.wikimedia.org/T218450 (10epriestley) This is linked elsewhere in connected tasks already, but I believe the upstream change in (03CR) 10Hashar: "Seems fine but I am not sure when we should skip it or where should the skip logic be." (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/521515 (https://phabricator.wikimedia.org/T87781) (owner: 10Kosta Harlan) [15:19:38] (03CR) 10Jforrester: [C: 03+2] layout: Seccheck back to voting for SecurePoll [integration/config] - 10https://gerrit.wikimedia.org/r/522938 (https://phabricator.wikimedia.org/T205972) (owner: 10Daimona Eaytoy) [15:20:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201907), 10puppet-compiler, 10Patch-For-Review: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar) Puppet fails due to /etc/puppetdb being a directory when it tries to make... [15:21:01] (03CR) 10Thcipriani: [C: 03+2] "Deployed:" [integration/config] - 10https://gerrit.wikimedia.org/r/519083 (https://phabricator.wikimedia.org/T207702) (owner: 10Thcipriani) [15:23:19] (03PS4) 10Mholloway: Python: Use --system flag when installing pip packages [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) [15:24:01] (03Merged) 10jenkins-bot: Maintenance: cleanup contint1001, too [integration/config] - 10https://gerrit.wikimedia.org/r/519083 (https://phabricator.wikimedia.org/T207702) (owner: 10Thcipriani) [15:24:30] (03PS5) 10Mholloway: Python: Add support for use-system-flag directive [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) [15:25:56] (03PS6) 10Mholloway: Python: Add support for use-system-flag directive [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) [15:36:01] (03PS2) 10Jforrester: layout: Seccheck back to voting for SecurePoll [integration/config] - 10https://gerrit.wikimedia.org/r/522938 (https://phabricator.wikimedia.org/T205972) (owner: 10Daimona Eaytoy) [15:36:06] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/522938 (https://phabricator.wikimedia.org/T205972) (owner: 10Daimona Eaytoy) [15:37:04] !log Zuul: [CentralAuth] Add seccheck T189227 [15:37:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:37:07] T189227: taint-checks for CentralAuth failing - https://phabricator.wikimedia.org/T189227 [15:37:50] (03Merged) 10jenkins-bot: layout: Seccheck back to voting for SecurePoll [integration/config] - 10https://gerrit.wikimedia.org/r/522938 (https://phabricator.wikimedia.org/T205972) (owner: 10Daimona Eaytoy) [15:38:40] !log Zuul: [SecurePoll] Make seccheck voting T205972 [15:38:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:38:42] T205972: Fixup taint-check-plugin errors in SecurePoll - https://phabricator.wikimedia.org/T205972 [15:51:36] (03CR) 10Jforrester: [C: 03+2] Add phan dependencies for MinervaNeue [integration/config] - 10https://gerrit.wikimedia.org/r/522958 (https://phabricator.wikimedia.org/T133664) (owner: 10Daimona Eaytoy) [15:51:45] 10Phabricator: Permission to edit Forms No. 31, 32 and 33 for MarcoAurelio - https://phabricator.wikimedia.org/T227267 (10mmodell) So it seems that it's not possible to override the policy on form editing. @epriestley: shouldn't this use the application "can edit / can configure application" policy for the `tran... [15:55:37] James_F: could you please reload zuul so that https://gerrit.wikimedia.org/r/c/integration/config/+/523164 can take effect? [15:55:59] Or maybe it is already in effect with the changes you made recently? [15:56:05] kostajh: Already done, yes. :-) [15:56:14] James_F: thx! [15:56:59] (03PS1) 10Jforrester: layout: [skins/MinervaNeue] Add phan in non-voting mode [integration/config] - 10https://gerrit.wikimedia.org/r/523210 (https://phabricator.wikimedia.org/T133664) [15:57:01] (03PS1) 10Jforrester: layout: [skins/MinervaNeue] Make phan voting [integration/config] - 10https://gerrit.wikimedia.org/r/523211 (https://phabricator.wikimedia.org/T133664) [15:58:00] 10Phabricator: Permission to edit Forms No. 31, 32 and 33 for MarcoAurelio - https://phabricator.wikimedia.org/T227267 (10epriestley) My expectation is that you need "Can Configure Application" on the Maniphest Application to edit Maniphest forms, and so on: https://phabricator.wikimedia.org/applications/view/P... [15:58:48] (03Merged) 10jenkins-bot: Add phan dependencies for MinervaNeue [integration/config] - 10https://gerrit.wikimedia.org/r/522958 (https://phabricator.wikimedia.org/T133664) (owner: 10Daimona Eaytoy) [15:58:56] (03CR) 10Jforrester: [C: 04-1] "Not ready yet." [integration/config] - 10https://gerrit.wikimedia.org/r/523211 (https://phabricator.wikimedia.org/T133664) (owner: 10Jforrester) [15:59:30] !log Zuul: Adding Echo and MobileFrontend dependencies to skins/MinervaNeue T133664 [15:59:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:59:34] T133664: Statically analyse MinervaNeue codebases with Phan - https://phabricator.wikimedia.org/T133664 [15:59:35] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 64.29% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:59:39] James_F: want to try https://gerrit.wikimedia.org/r/c/integration/config/+/521190 and https://gerrit.wikimedia.org/r/c/integration/config/+/521192 sometime this week? [16:04:08] !log Run urbanecm@deployment-deploy01:~$ mwscript migrateUserGroup.php --wiki=commonswiki Image-reviewer image-reviewer (T216406) [16:04:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:04:10] T216406: Rename `Image-reviewer` to `image-reviewer`, then migrate all its members - https://phabricator.wikimedia.org/T216406 [16:06:21] 10Project-Admins: Create #ci-test-error tag for tracking Gerrit repos failing tests - https://phabricator.wikimedia.org/T227992 (10Legoktm) >>! In T227992#5331418, @Krinkle wrote: > A few process questions: Should it cover only if they fail in WMF CI, or also if it fails for someone locally? What if the test sub... [16:07:37] (03CR) 10jerkins-bot: [V: 04-1] layout: [skins/MinervaNeue] Add phan in non-voting mode [integration/config] - 10https://gerrit.wikimedia.org/r/523210 (https://phabricator.wikimedia.org/T133664) (owner: 10Jforrester) [16:08:10] (03PS2) 10Jforrester: layout: [skins/MinervaNeue] Make phan voting [integration/config] - 10https://gerrit.wikimedia.org/r/523211 (https://phabricator.wikimedia.org/T133664) [16:11:33] 10Project-Admins: Create #ci-test-error tag for tracking Gerrit repos failing tests - https://phabricator.wikimedia.org/T227992 (10hashar) I would guess that the new tag would be kind of a super set of #jenkins-failure isn't it? Regardless yes please be bold :-] [16:11:46] oof, Zuul is *super* busy right now :( [16:12:08] and there’s not even a Phabricator task for all these changes dropping deprecated bits? [16:13:08] 10Phabricator, 10Release-Engineering-Team-TODO (201907), 10Wikimedia-Logstash: Improve error reporting / integration between Kibana and Phabricator - https://phabricator.wikimedia.org/T185155 (10mmodell) [16:13:15] 10Release-Engineering-Team-TODO (201907), 10Developer Productivity: Gather and Analyze Information Around Developer Tooling Woes - https://phabricator.wikimedia.org/T212454 (10jeena) 05Open→03Resolved [16:13:17] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Developer Productivity, 10Epic: FY201819 TEC12 Program – Developer Productivity - https://phabricator.wikimedia.org/T212449 (10jeena) [16:17:41] Lucas_WMDE: sorry, context? [16:18:28] there’s hundreds of jobs on https://integration.wikimedia.org/zuul/ [16:18:49] there's 5 important ones [16:18:50] most of them apparently due to a huge chain ending in https://gerrit.wikimedia.org/r/522569 dropping various bits for compatibility [16:19:12] coverage/patch-performance/codehealth/experimental all run in delayed lower priority queues [16:19:38] ah [16:19:47] (03PS1) 10Hashar: Build pytest [tools/scap] - 10https://gerrit.wikimedia.org/r/523223 [16:19:58] no, a phab task isn't a requirement in the deprecation policy [16:24:15] (03CR) 10Hashar: "And of course master FTBS:" [tools/scap] - 10https://gerrit.wikimedia.org/r/523223 (owner: 10Hashar) [16:25:12] kostajh: I will reviews patches for Quibble for sure and definitely get a new version cut and deployed to support the unit tests split [16:26:04] thanks [16:26:46] maybe I've missed something, but I don't see MobileFrontend cloned for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/521357#message-3705579f3d44877c1d41018aacbddbdb9eeb2f97, and https://gerrit.wikimedia.org/r/c/integration/config/+/523164 was merged + zuul reloaded [16:29:56] (03CR) 10Daimona Eaytoy: [C: 03+1] layout: [skins/MinervaNeue] Add phan in non-voting mode [integration/config] - 10https://gerrit.wikimedia.org/r/523210 (https://phabricator.wikimedia.org/T133664) (owner: 10Jforrester) [16:36:34] 10Continuous-Integration-Infrastructure, 10Operations, 10puppet-compiler, 10Patch-For-Review: puppet compiler fails on releases1001.eqiad.wmnet due to lack of Service[bacula-director] - https://phabricator.wikimedia.org/T228047 (10hashar) *facepalm* I was debugging using the `production` branch catalog in... [16:44:41] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:45:12] (03PS1) 10Jforrester: jjb: Define mwskin-php72-phan-docker-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/523227 [16:45:31] (03CR) 10Jforrester: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/523227 (owner: 10Jforrester) [16:45:36] (03CR) 10Jforrester: [C: 03+2] layout: [skins/MinervaNeue] Add phan in non-voting mode [integration/config] - 10https://gerrit.wikimedia.org/r/523210 (https://phabricator.wikimedia.org/T133664) (owner: 10Jforrester) [16:48:10] (03Merged) 10jenkins-bot: jjb: Define mwskin-php72-phan-docker-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/523227 (owner: 10Jforrester) [16:48:13] (03Merged) 10jenkins-bot: layout: [skins/MinervaNeue] Add phan in non-voting mode [integration/config] - 10https://gerrit.wikimedia.org/r/523210 (https://phabricator.wikimedia.org/T133664) (owner: 10Jforrester) [16:49:10] !log Zuul: [MinervaNeue] Add phan as non-voting T133664 [16:49:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:49:12] T133664: Statically analyse MinervaNeue codebases with Phan - https://phabricator.wikimedia.org/T133664 [16:56:32] (03PS1) 10Kosta Harlan: Add MobileFrontend to phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/523230 [16:57:28] (03CR) 10Daimona Eaytoy: [C: 03+1] Add MobileFrontend to phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/523230 (owner: 10Kosta Harlan) [16:58:22] (03PS2) 10Jforrester: phan dependencies: [GrowthExperiments] Add MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/523230 (owner: 10Kosta Harlan) [16:58:26] (03CR) 10Jforrester: [C: 03+2] phan dependencies: [GrowthExperiments] Add MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/523230 (owner: 10Kosta Harlan) [17:00:00] (03Merged) 10jenkins-bot: phan dependencies: [GrowthExperiments] Add MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/523230 (owner: 10Kosta Harlan) [17:00:56] !log Zuul: [GrowthExperiments] Add MobileFrontend to phan dependencies [17:00:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:03:53] (03CR) 10Thcipriani: [C: 04-1] "Looks good!" (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) (owner: 10Mholloway) [17:30:11] (03CR) 10Thcipriani: "This seems to have produced a working .deb. Maybe we can get rid of the custom scap deb job?" [tools/scap] - 10https://gerrit.wikimedia.org/r/522370 (owner: 10Hashar) [17:40:43] (03PS7) 10Mholloway: Python: Add support for use-system-flag directive [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) [17:40:57] (03CR) 10Mholloway: Python: Add support for use-system-flag directive (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) (owner: 10Mholloway) [17:54:11] (03CR) 10Thcipriani: [C: 03+2] Python: Add support for use-system-flag directive [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) (owner: 10Mholloway) [17:56:41] (03Merged) 10jenkins-bot: Python: Add support for use-system-flag directive [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) (owner: 10Mholloway) [17:58:44] (03PS3) 10Jforrester: layout: [skins/MinervaNeue] Make phan voting [integration/config] - 10https://gerrit.wikimedia.org/r/523211 (https://phabricator.wikimedia.org/T133664) [17:58:45] 10Gerrit: Support posting screenshots in Gerrit - https://phabricator.wikimedia.org/T228084 (10kostajh) [17:58:55] (03CR) 10PipelineBot: "pipeline-dashboard: blubber-pipeline-publish" [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) (owner: 10Mholloway) [17:58:57] (03CR) 10PipelineBot: "pipeline-dashboard: blubber-pipeline-publish" [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) (owner: 10Mholloway) [17:58:59] (03CR) 10jenkins-bot: Python: Add support for use-system-flag directive [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) (owner: 10Mholloway) [17:59:01] (03CR) 10Jforrester: [C: 03+2] layout: [skins/MinervaNeue] Make phan voting [integration/config] - 10https://gerrit.wikimedia.org/r/523211 (https://phabricator.wikimedia.org/T133664) (owner: 10Jforrester) [17:59:51] 10Gerrit: Support posting screenshots in Gerrit - https://phabricator.wikimedia.org/T228084 (10kostajh) [18:00:35] (03Merged) 10jenkins-bot: layout: [skins/MinervaNeue] Make phan voting [integration/config] - 10https://gerrit.wikimedia.org/r/523211 (https://phabricator.wikimedia.org/T133664) (owner: 10Jforrester) [18:05:04] !log Zuul: [MinervaNeue] Make phan voting T133664 [18:05:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:05:07] T133664: Statically analyse MinervaNeue codebases with Phan - https://phabricator.wikimedia.org/T133664 [18:07:31] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Epic, 10phan: Enable mediawiki/mediawiki-phan-config on all Wikimedia-deployed repositories - https://phabricator.wikimedia.org/T224783 (10Jdforrester-WMF) [18:11:44] "Copying from deploy2001.codfw.wmnet to mwdebug1001.eqiad.wmnet" [18:11:49] "Copying from deploy1001.eqiad.wmnet to mwdebug1002.eqiad.wmnet" [18:11:57] why does mwdebug1001 pull from codfw? [18:12:09] is that working correctly? Like, it proxies by there? [18:12:22] thcipriani: ^ [18:13:00] hrm, that looks wrong [18:13:05] * thcipriani looks [18:13:37] (although they should have the same code) [18:14:34] Krinkle: where did you see that? running scap pull manually? or in the logs from a sync from deploy1001? [18:15:50] 10Continuous-Integration-Config, 10Wikimedia-General-or-Unknown, 10phan-taint-check-plugin, 10Patch-For-Review: Enable phan-taint-check-plugin on all Wikimedia-deployed repositories where it is currently passing - https://phabricator.wikimedia.org/T201219 (10Jdforrester-WMF) [18:16:05] thcipriani: I noticed it in mwdebug log stash in scap channel for those two servers [18:16:10] during a normal scap-sync file [18:16:11] I think [18:16:25] > Running rsync command: `sudo -u mwdeploy -n -- /usr/bin/rsync --archive --delete-delay --delay-updates --compress --delete --exclude=**/cache/l10n/*.cdb --exclude=*.swp --no-perms --exclude=**/.git --include=/wmf-config --include=/wmf-config/InitialiseSettings.php --exclude=* deploy1001.eqiad.wmnet::common /srv/mediawiki` [18:16:29] > Running rsync command: `sudo -u mwdeploy -n -- /usr/bin/rsync --archive --delete-delay --delay-updates --compress --delete --exclude=**/cache/l10n/*.cdb --exclude=*.swp --no-perms --exclude=**/.git --include=/wmf-config --include=/wmf-config/InitialiseSettings.php --exclude=* deploy2001.codfw.wmnet::common /srv/mediawiki` [18:17:40] seems like that might cause it to have outdated code unless it synced to codfw first which I don't know if it does. and certainly for manual testing if 'scap pull' does it wrong a swel that would mean we can't debug on mwdebug1001 from eqiad but I haven't confirmed that [18:18:13] scap pull shouldn't do that unless you tell it to manually. [18:18:27] during a sync-file, deploy2001 does have the same code, so it's not an error [18:18:31] but it is weird [18:18:44] we have some code that picks the "closest" server based on ttl [18:19:01] closest proxy with the code you need, that is [18:20:24] this doesn't seem like something that would cause inconsistent deploys (and looking at mwdbug1001 it's configured correctly), but this does seem strange [18:27:22] mwdebug1002 heard back from deploy2001 faster than deploy1001 using this function: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/scap/utils.py#182 [19:04:39] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] Gerrit v2.15.14 [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/522133 (owner: 10Paladox) [19:22:34] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-General-or-Unknown, 10TechCom-RFC, and 2 others: Drop PHP 7.0 support from MediaWiki - https://phabricator.wikimedia.org/T216165 (10Catrope) We shouldn't need to do an RFC for every single PHP version drop. Let... [19:32:24] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-General-or-Unknown, 10TechCom-RFC, and 2 others: Drop PHP 7.0 support from MediaWiki - https://phabricator.wikimedia.org/T216165 (10Legoktm) >>! In T216165#5284682, @Krinkle wrote: > I'd like to try to swing th... [19:35:43] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-General-or-Unknown, 10TechCom-RFC, and 2 others: Drop PHP 7.0 support from MediaWiki - https://phabricator.wikimedia.org/T216165 (10Legoktm) >>! In T216165#5334530, @Catrope wrote: > Meanwhile, we should drop 7... [19:39:30] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-General-or-Unknown, 10TechCom-RFC, and 2 others: Drop PHP 7.0 support from MediaWiki - https://phabricator.wikimedia.org/T216165 (10Jdforrester-WMF) [Again, PLEASE have this conversation on a generic task, not... [20:14:01] 10Gerrit: Support posting screenshots in Gerrit - https://phabricator.wikimedia.org/T228084 (10Paladox) That extension does not appear to support PolyGerrit. [20:18:18] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:52:00] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [services/kartotherian] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/523304 [20:52:02] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [services/kartotherian] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/523304 (owner: 10QChris) [20:52:46] (03PS1) 10QChris: Import done. Revoke import grants [services/kartotherian] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/523307 [20:52:48] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [services/kartotherian] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/523307 (owner: 10QChris) [21:00:15] thcipriani: So sync-file syncs to codfw before it goes to canaries? [21:00:46] Krinkle: syncs to the masters in both datacenters before canaries, yes [21:01:17] thcipriani: and scap pull doesn't use this logic, so that when testing something not for deployment one can indeed reliably pull from eqiad/deply1001 on any mwdebug currently? [21:01:29] s/can/does/ [21:01:58] yep, scap pull uses the master_rsync defined in /etc/scap.cfg [21:02:04] ok no worries the [21:02:06] then* [21:02:17] I do see it use codfw quite often in the logs, but I'll ignore that then [21:02:21] e.g. https://logstash.wikimedia.org/app/kibana#/dashboard/mwdebug1002 [21:11:05] folks, wikidata entity dumps are (again) broken, by a mediawiki core commit, fix is incoming once AaronSchulz and hoo settle one issue in CR... we'll likely ask for an immediate backport [21:11:20] (this is week 3 out of 4 with failures from mw deploys) [21:12:29] AaronSchulz: if you're here we'd like to resolve https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/521920/ one way or another [21:15:51] apergos: Fun. :-( [21:16:07] yeah, I'm hating my life righ tnow and especially timezones [21:16:32] anyone have aaron in eyesight and can wave at him to look at irc? [21:16:36] heh [21:16:44] ah you are there, bless you [21:16:47] He doesn't work from the office, so no. :-) [21:17:03] well a virtual glance mus thave done it :-) [21:17:22] I'll look in a few min (the current patch can go straight to wmf-x if needed) [21:17:31] thanks much! [21:18:36] yes, getting an immediate fix out to the current branch would be excellent, we could restart these and get them out with only two days delay [21:20:40] Krinkle: I thought I saw something about logstash being broken? is that ok now, are deploys ok to happen? [21:22:49] apergos: yes, as of a few minutes ago it seems to be working again. [21:22:57] ah thank goodness [21:23:22] I've been trying to get up to speed on the opcache/interned strings issue [21:23:34] okay [21:23:41] there's no shortage of reading material on the two tasks about that [21:27:06] James_F: telework? [21:27:13] (cf. not working from office) [21:28:36] I've been looking a little at the code to get a better sense of what actually goes on (not that this has shed any light on where other bugs might be coming from) [21:29:41] I did find this joyous little issue though: https://github.com/php/php-src/pull/4161 [21:30:55] 10Release-Engineering-Team-TODO (201907), 10Developer Productivity, 10dev-images, 10local-charts: Move dev-images PHP + Apache image from mod_php to php-fpm - https://phabricator.wikimedia.org/T222494 (10brennen) [21:31:59] 10Release-Engineering-Team-TODO (201907), 10Developer Productivity, 10dev-images, 10local-charts: Move dev-images PHP image from php -S to Apache + php-fpm - https://phabricator.wikimedia.org/T222494 (10brennen) [21:33:35] how in the world is there already another npm security issue that affects all of our repos [21:33:58] some weeks it doesn't pay to get out of bed [21:33:58] "npm". [21:34:09] https://gerrit.wikimedia.org/r/q/hashtag:%22cve-2019-10744%22+(status:open%20OR%20status:merged) [21:34:59] lockfiles seem like a great idea, but if we're having to update them everyday, that's going to get annoying real fast [21:34:59] 10Project-Admins: Create #ci-test-error tag for tracking Gerrit repos failing tests - https://phabricator.wikimedia.org/T227992 (10Aklapper) Looks like consensus to remove the alternative hashtag `#jenkins-failure` from https://phabricator.wikimedia.org/project/profile/3298/ but it's unclear to me which relation... [21:37:15] legoktm: "ugh, npm." [21:37:40] hehe [21:37:57] looks like you nailed it choosing that motto [21:38:29] I was thinking of having it automatically switch to "yay, npm!" if there were no security issues, but I don't know if that day will ever come [21:39:02] lol [21:39:21] like "wall of sadness" --> wall of superpowers [21:40:02] to their credit, the npm team has done a really nice job with the "npm audit" command, it's super trivial to get exactly what I need out of it [21:41:20] yup, I run it locally when patching stuff just in case [21:52:00] 10Project-Admins: Create #ci-test-error tag for tracking Gerrit repos failing tests - https://phabricator.wikimedia.org/T227992 (10Krinkle) I don't think it should have an explicit relation, I think top-level generic/yellow tags work best as a flat system. [22:13:33] 10Project-Admins: Create #ci-test-error tag for tracking Gerrit repos failing tests - https://phabricator.wikimedia.org/T227992 (10Jdforrester-WMF) Yeah, let's just create it. [22:20:00] (03PS1) 10Jforrester: layout: [CentralAuth] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/523332 (https://phabricator.wikimedia.org/T189371) [22:24:07] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:24:31] hey AaronSchulz just a heads up that I can probably stick it out until 2 am (another half hour) but not much beyond that, pretty beat [22:26:52] aye [22:30:46] (03CR) 10Mholloway: "I think there's a bug here that the new test didn't catch because it concerns YAML marshalling. I get an unmarshal error when I actually " [blubber] - 10https://gerrit.wikimedia.org/r/522553 (https://phabricator.wikimedia.org/T227919) (owner: 10Mholloway) [22:30:53] (03CR) 10Jforrester: [C: 03+2] layout: [CentralAuth] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/523332 (https://phabricator.wikimedia.org/T189371) (owner: 10Jforrester) [22:32:58] (03Merged) 10jenkins-bot: layout: [CentralAuth] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/523332 (https://phabricator.wikimedia.org/T189371) (owner: 10Jforrester) [22:34:21] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:34:51] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10LibUp, 10Patch-For-Review: LibraryUpgrader CI normalisation tasks, June/July 2019 - https://phabricator.wikimedia.org/T225325 (10Legoktm) > `.eslintrc` should be renamed to `.eslintrc.... [22:37:59] 10Deployments, 10Release-Engineering-Team (Long-Lived-Branches), 10Release-Engineering-Team-TODO (201907): Static asset time on disk - https://phabricator.wikimedia.org/T140921 (10Jdforrester-WMF) [22:39:38] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10LibUp, 10Patch-For-Review: LibraryUpgrader CI normalisation tasks, June/July 2019 - https://phabricator.wikimedia.org/T225325 (10Jdforrester-WMF) >>! In T225325#5335215, @Legoktm wrote:... [22:40:55] !log Zuul: [CentralAuth] Add phan T189371 [22:40:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:40:58] T189371: phan and taint/sec for CentralAuth - https://phabricator.wikimedia.org/T189371 [22:42:10] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: Static asset time on disk - https://phabricator.wikimedia.org/T140921 (10greg) [22:42:14] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: Take heat off day before the weekly branch-cut? - https://phabricator.wikimedia.org/T118212 (10greg) [22:43:15] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907): Static asset time on disk - https://phabricator.wikimedia.org/T140921 (10greg) (oops) [22:55:27] https://github.com/Kronuz/esprima-python ^.^ [23:13:51] crashing out, will look tomorrow, hopefully the fix will have gone around [23:14:01] night, folks [23:38:50] * Krinkle misread apergos message, thought they were "cashing out" and wondering where the money would came from [23:40:21] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907), 10Performance-Team (Radar): Static asset time on disk - https://phabricator.wikimedia.org/T140921 (10Krinkle) [23:41:40] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: Take heat off day before the weekly branch-cut? - https://phabricator.wikimedia.org/T118212 (10Krinkle) [23:52:19] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO: Take heat off day before the weekly branch-cut? - https://phabricator.wikimedia.org/T118212 (10Krinkle) Another idea for the short-to-mid term could be to have Beta Cluster only run `master` Tuesday-Thursday, bu... [23:53:32] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Performance-Team (Radar): Take heat off day before the weekly branch-cut? - https://phabricator.wikimedia.org/T118212 (10Krinkle)