[00:10:20] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:11:54] PROBLEM - Free space - all mounts on deployment-imagescaler01 is CRITICAL: (Service Check Timed Out) [00:16:46] RECOVERY - Free space - all mounts on deployment-imagescaler01 is OK: OK: All targets OK [00:19:28] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:35:11] 10Project-Admins, 10User-DannyS712: Create component #MediaWiki-Core-Hooks - https://phabricator.wikimedia.org/T249170 (10Krinkle) The new hooks system is being created by Core Platform Team, I assume they are its code stewards as well. I didn't notice until now that this task was created by @DannyS712 and not... [02:21:01] 10Release-Engineering-Team-TODO, 10Performance-Team, 10Core Platform Team Workboards (Clinic Duty Team), 10PHP 7.2 support, 10Wikimedia-Incident: Performance regression from Apcu/ExtensionRegistry::loadFromQueue on PHP7 - https://phabricator.wikimedia.org/T187154 (10Krinkle) [05:07:25] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: (Service Check Timed Out) [05:17:15] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [08:14:00] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [08:24:00] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [09:10:18] 10Release-Engineering-Team-TODO, 10Scap, 10observability, 10Regression: Scap no longer sets 'level' in Logstash messages - https://phabricator.wikimedia.org/T246793 (10fgiunchedi) >>! In T246793#6019942, @thcipriani wrote: >>>! In T246793#6019857, @mmodell wrote: >> this looks like it's likely the culprit:... [09:19:59] 10Phabricator, 10Patch-For-Review, 10User-Zoranzoki21: Phabricator does not show videos in webm format as embedded - https://phabricator.wikimedia.org/T244162 (10mmodell) I think this is a duplicate of {T215360} [09:20:31] 10Phabricator, 10Patch-For-Review, 10User-Zoranzoki21: Phabricator does not show videos in webm format as embedded - https://phabricator.wikimedia.org/T244162 (10mmodell) I'd rather not encourage videos uploaded to phabricator due to the storage limitations we currently have for phab file uploads. [09:25:07] 10Scap, 10Patch-For-Review, 10User-fgiunchedi: Send scap log directly to logstash via syslog input - https://phabricator.wikimedia.org/T86969 (10fgiunchedi) [09:25:42] 10Phabricator, 10Patch-For-Review, 10User-Zoranzoki21: Phabricator does not show videos in webm format as embedded - https://phabricator.wikimedia.org/T244162 (10Zoranzoki21) >>! In T244162#6021778, @mmodell wrote: > I think this is a duplicate of {T215360} Hmm yes, you're right. I'm going to close this tas... [09:26:43] 10Phabricator, 10User-Zoranzoki21: Phabricator does not show videos in webm format as embedded - https://phabricator.wikimedia.org/T244162 (10Zoranzoki21) [09:27:16] 10Phabricator, 10User-Zoranzoki21: Phabricator does not show videos in webm format as embedded - https://phabricator.wikimedia.org/T244162 (10Zoranzoki21) [09:27:19] 10Phabricator, 10Patch-For-Review: Allow video embeds in formats other than OGV (e.g. WEBM) - https://phabricator.wikimedia.org/T215360 (10Zoranzoki21) [09:45:59] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations, 10serviceops: Build and publish a python2 based container to build wheels - https://phabricator.wikimedia.org/T249110 (10Joe) 05Open→03Resolved a:03Joe Successfully published i... [09:46:03] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Zuul, 10Patch-For-Review: Convert zuul to use scap - https://phabricator.wikimedia.org/T215458 (10Joe) [09:51:28] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations, 10serviceops: Build and publish a python2 based container to build wheels - https://phabricator.wikimedia.org/T249110 (10hashar) `counterexample $ docker pull docker-registry.wikimed... [10:15:07] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations, 10serviceops: Build and publish a python2 based container to build wheels - https://phabricator.wikimedia.org/T249110 (10hashar) There is some issue with the registry, but eventually... [10:16:52] (03CR) 10Hashar: "The Docker image docker-registry.wikimedia.org/python2-build-buster is available." [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/577846 (https://phabricator.wikimedia.org/T215458) (owner: 1020after4) [10:19:25] 10Gerrit, 10Phabricator, 10Release-Engineering-Team: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10jcrespo) [10:25:09] 10Gerrit, 10Phabricator, 10Release-Engineering-Team: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10jcrespo) And I know about Gitiles ( https://gerrit.wikimedia.org/g/operations/debs/wmf-pt-kill ), but phabricator or Github... [10:36:06] 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-GitHub: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10Peachey88) I'm betting the GitHub mirror was never created. [10:41:03] awight: congratulations on updating to webdriverio v5 :) [10:41:17] a few days ago v6 was released :D [10:41:32] upgrade to it should be much more simpler than to v5 [10:42:52] see T247254 [10:42:52] T247254: Upgrade WebdriverIO to version 6 for all repositories - https://phabricator.wikimedia.org/T247254 [10:43:08] 10Phabricator, 10Operations, 10Security-Team, 10Patch-For-Review, 10Security: Adjust onboarding/offboarding logic to accommodate changes to #security (now acl*security) - https://phabricator.wikimedia.org/T245771 (10jbond) The script has been updated please feel free to test it further on mwmaint1002 and... [10:43:29] 10Phabricator, 10Operations, 10Security-Team, 10Patch-For-Review, 10Security: Adjust onboarding/offboarding logic to accommodate changes to #security (now acl*security) - https://phabricator.wikimedia.org/T245771 (10jbond) a:05jbond→03None [10:46:58] awight: and there are updates to drag-and-drop https://webdriver.io/blog/2020/03/26/webdriverio-v6-released.html [11:24:39] 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-GitHub: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10jcrespo) [12:24:27] 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-GitHub: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10Reedy) ` $ ssh -p 29418 gerrit.wikimedia.org replication start --wait operations/debs/wmf-pt-kill Rep... [12:33:17] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10Dzahn) Icinga alert is in state OK and also in downtime. And it's testing-only. [12:35:36] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10Dzahn) 05Open→03Resolved /dev/vda1 63G 41G 20G 68% / [12:35:39] 10Gerrit, 10Operations, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) [12:57:16] 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-GitHub: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10jcrespo) So is this something that has to be done manually, and not done for every repo? What was the... [13:01:41] 10Diffusion, 10Release-Engineering-Team, 10Projects-Cleanup, 10Wikimedia-GitHub: Delete/Archive Github and Diffusion mirrors of Gerrit repos related wikiba.se - https://phabricator.wikimedia.org/T239429 (10Reedy) [13:04:59] 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-GitHub: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10Reedy) >>! In T249205#6022365, @jcrespo wrote: > So is this something that has to be done manually, an... [13:25:04] 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-GitHub: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10jcrespo) > I thought we had (or were going to have) a plugin to do that repo creation automatically at... [13:38:40] 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-GitHub: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10Reedy) >>! In T249205#6022442, @jcrespo wrote: >> I thought we had (or were going to have) a plugin to... [13:48:54] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10akosia... [13:49:41] !log Add Jenkins bot wmf-insecte to #wikimedia-analytics # T210271 [13:49:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:49:43] T210271: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 [14:00:12] (03PS1) 10Hashar: jjb: irc notification for analytics jobs [integration/config] - 10https://gerrit.wikimedia.org/r/585497 (https://phabricator.wikimedia.org/T210271) [14:33:43] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap: beta-publish-deb fails to find a .deb artifact from parent job - https://phabricator.wikimedia.org/T249225 (10hashar) [14:34:39] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap: beta-publish-deb fails to find a .deb artifact from parent job - https://phabricator.wikimedia.org/T249225 (10hashar) I am pretty sure it is a race condition, the sequence would be: | scap-... [14:36:07] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10jbond) 05Resolved→03Open @Dzahn see my [[ https://phabricator.wikimedia.org/T243808#6018042 | comment above ]] I think we should investigate fi... [14:36:10] 10Gerrit, 10Operations, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10jbond) [14:41:25] (03PS1) 10Hashar: jjb: fix race in beta-publish-deb [integration/config] - 10https://gerrit.wikimedia.org/r/585506 (https://phabricator.wikimedia.org/T249225) [14:41:44] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10Patch-For-Review: beta-publish-deb fails to find a .deb artifact from parent job - https://phabricator.wikimedia.org/T249225 (10hashar) a:03hashar [14:41:57] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10Patch-For-Review: beta-publish-deb fails to find a .deb artifact from parent job - https://phabricator.wikimedia.org/T249225 (10hashar) [14:49:38] 10Gerrit, 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-GitHub: Cannot find repo operations/debs/wmf-pt-kill on diffusion or Github Wikimedia mirror - https://phabricator.wikimedia.org/T249205 (10jcrespo) Yep, the request wasn't directed to you, Reedy, but to Phab maintainer CCed already 0:-) Than... [14:50:33] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10Patch-For-Review: beta-publish-deb fails to find a .deb artifact from parent job - https://phabricator.wikimedia.org/T249225 (10LarsWirzenius) I find it... [14:50:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services): Delete integration-agent-jessie-docker-1001 - https://phabricator.wikimedia.org/T249226 (10hashar) [14:57:24] (03CR) 10Lars Wirzenius: "How would you test this, without merging it?" [integration/config] - 10https://gerrit.wikimedia.org/r/585506 (https://phabricator.wikimedia.org/T249225) (owner: 10Hashar) [14:58:25] (03CR) 10Hashar: "Usually: deploy the job to Jenkins, run it and verify. If all goes well, +2 here :]" [integration/config] - 10https://gerrit.wikimedia.org/r/585506 (https://phabricator.wikimedia.org/T249225) (owner: 10Hashar) [14:58:54] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10JAllem... [15:06:57] (03CR) 10Lars Wirzenius: "In that case, since I'm not capable of doing the testing myself, and I don't understand JJB well enough to be confident the change is vali" [integration/config] - 10https://gerrit.wikimedia.org/r/585506 (https://phabricator.wikimedia.org/T249225) (owner: 10Hashar) [15:20:14] !log Updated https://integration.wikimedia.org/ci/job/scap-beta-deb # T249225 [15:20:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:20:16] T249225: beta-publish-deb fails to find a .deb artifact from parent job - https://phabricator.wikimedia.org/T249225 [15:20:17] liw: ^ [15:20:29] rebuilding last [15:23:21] https://integration.wikimedia.org/ci/job/scap-beta-deb/60/console failed [15:23:33] because of the sub job failing: https://integration.wikimedia.org/ci/job/beta-publish-deb/470/console [15:23:44] which leads to ERROR: can't open database: open /srv/packages/db/LOG: permission denied [15:25:39] (03CR) 10Hashar: [C: 03+2] "No worries :] I gave it a try on https://integration.wikimedia.org/ci/job/scap-beta-deb/60/console" [integration/config] - 10https://gerrit.wikimedia.org/r/585506 (https://phabricator.wikimedia.org/T249225) (owner: 10Hashar) [15:25:50] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10Patch-For-Review: beta-publish-deb fails to find a .deb artifact from parent job - https://phabricator.wikimedia.org/T249225 (10hashar) 05Open→03Resol... [15:26:38] (03Merged) 10jenkins-bot: jjb: fix race in beta-publish-deb [integration/config] - 10https://gerrit.wikimedia.org/r/585506 (https://phabricator.wikimedia.org/T249225) (owner: 10Hashar) [15:28:14] liw: I am assuming it is all fixed [15:28:20] the aptly one still has to be figured out though [15:52:16] !log deployment-deploy01: sudo chown jenkins-deploy:wikidev /srv/packages/db/* [15:52:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:52:29] https://integration.wikimedia.org/ci/job/beta-publish-deb/471/console [15:52:33] liw: seems to work now [15:53:34] hashar, thanks [15:53:44] I would gues something broke on March 23 ~ 16:15 [15:53:54] that the last modified time of the files owned by root:root [15:54:28] which I would blame thcipriani / liw for :] [15:54:40] both accounts were connected at that time [15:55:18] * thcipriani looks suspicious [15:55:26] we did a scap test then, and the permissions problem was there already, we just worked around it by running aptly as root [15:56:15] I think puppet probably changed perms back around that time. [15:57:17] possibly [15:57:25] * hashar runs puppet [16:01:59] hashar, can you ping me when you're done, I want to test Scap on beta in prep for a release [16:02:07] puppet ran yeah [16:02:12] some files are owned by root [16:02:24] then I dont know anything about aptly [16:02:25] :/ [16:02:38] maybe the latest is available now [16:05:04] ack, thanks, I'll see what happens [16:08:26] 10Continuous-Integration-Config, 10Wikidata: Remove Math extension from Wikibase CI jobs again - https://phabricator.wikimedia.org/T249235 (10Lucas_Werkmeister_WMDE) [16:10:09] (03PS1) 10Lucas Werkmeister (WMDE): Revert "Run Math tests for every change in Wikibase" [integration/config] - 10https://gerrit.wikimedia.org/r/585533 (https://phabricator.wikimedia.org/T249235) [16:12:24] (03CR) 10Jforrester: [C: 03+2] Revert "Run Math tests for every change in Wikibase" [integration/config] - 10https://gerrit.wikimedia.org/r/585533 (https://phabricator.wikimedia.org/T249235) (owner: 10Lucas Werkmeister (WMDE)) [16:13:33] (03Merged) 10jenkins-bot: Revert "Run Math tests for every change in Wikibase" [integration/config] - 10https://gerrit.wikimedia.org/r/585533 (https://phabricator.wikimedia.org/T249235) (owner: 10Lucas Werkmeister (WMDE)) [16:15:09] !log Zuul: Stop running Math tests with every change in Wikibase T249235 [16:15:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:15:11] T249235: Remove Math extension from Wikibase CI jobs again - https://phabricator.wikimedia.org/T249235 [16:15:14] Lucas_WMDE: Done. [16:18:29] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review: Remove Math extension from Wikibase CI jobs again - https://phabricator.wikimedia.org/T249235 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved a:03Lucas_Werkmeister_WMDE That was quick, thanks! [16:20:48] James_F: thanks! [16:41:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Delete integration-agent-jessie-docker-1001 - https://phabricator.wikimedia.org/T249226 (10Jdforrester-WMF) a:03Jdforrester-WMF OK. I removed the tags over... [16:42:59] !log Shut off integration-agent-jessie-docker-1001 ahead of removal T249226 [16:43:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:43:01] T249226: Delete integration-agent-jessie-docker-1001 - https://phabricator.wikimedia.org/T249226 [16:43:44] !log Deleted integration-agent-jessie-docker-1001 from jenkins T249226 [16:43:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:44:23] !log Deleted integration-agent-jessie-docker-1001 from WMCS T249226 [16:44:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:45:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Delete integration-agent-jessie-docker-1001 - https://phabricator.wikimedia.org/T249226 (10Jdforrester-WMF) [16:45:36] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (... [16:45:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Delete integration-agent-jessie-docker-1001 - https://phabricator.wikimedia.org/T249226 (10Jdforrester-WMF) 05Open→03Resolved [16:45:41] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (... [16:46:39] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (... [16:49:03] PROBLEM - Host integration-agent-jessie-docker-1001 is DOWN: CRITICAL - Host Unreachable (172.16.5.149) [17:06:57] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: (Service Check Timed Out) [17:07:24] !log testing upcoming scap on beta [17:07:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:16:49] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [17:21:32] 10Release-Engineering-Team-TODO, 10Scap, 10observability, 10Developer Productivity, 10Regression: Scap no longer sets 'level' in Logstash messages - https://phabricator.wikimedia.org/T246793 (10Krinkle) [17:41:55] 10Release-Engineering-Team-TODO, 10Scap, 10serviceops: Deploy Scap version 3.14.0-1 - https://phabricator.wikimedia.org/T249250 (10LarsWirzenius) [17:49:10] (03PS1) 10Lars Wirzenius: Update RELEASE.md based on notes from release just made [tools/scap] - 10https://gerrit.wikimedia.org/r/585553 [17:49:50] (03CR) 10Lars Wirzenius: "I've just made a release and updated RELEASE.md while doing that. These are the changes." [tools/scap] - 10https://gerrit.wikimedia.org/r/585553 (owner: 10Lars Wirzenius) [17:51:17] (03CR) 1020after4: [C: 03+1] "> Patch Set 11:" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/577846 (https://phabricator.wikimedia.org/T215458) (owner: 1020after4) [18:01:35] (03CR) 10Jforrester: [C: 03+1] "Let's ship it?" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/577846 (https://phabricator.wikimedia.org/T215458) (owner: 1020after4) [18:04:34] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments: 1.35.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T247773 (10Jdforrester-WMF) [18:12:41] (03CR) 1020after4: [C: 03+2] Scap3 deploy repo for zuul [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/577846 (https://phabricator.wikimedia.org/T215458) (owner: 1020after4) [18:32:09] 10Release-Engineering-Team-TODO, 10Phatality, 10Developer Productivity: Phatality: Keep error message and trace closer together - https://phabricator.wikimedia.org/T237682 (10Krinkle) OK. Is this blocked on {T237706}? [18:45:55] (03CR) 10Jforrester: [V: 03+2 C: 03+1] "No CI." [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/577846 (https://phabricator.wikimedia.org/T215458) (owner: 1020after4) [19:00:31] 10Release-Engineering-Team-TODO, 10Phatality, 10Developer Productivity: Phatality: Keep error message and trace closer together - https://phabricator.wikimedia.org/T237682 (10mmodell) @krinkle yeah I need to at least have an sre help me deploy it due to {T237706} [19:17:00] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [19:21:52] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10Dzahn) Alright. This is a one-time installation though to test the Gerrit upgrade to 2.16 and then remove it again. But that doesn't mean there ca... [19:27:01] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [19:33:42] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [19:38:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Zuul: Abort a Zuul pipeline when one job completed with failures. - https://phabricator.wikimedia.org/T248531 (10Krinkle) From {T225730}, T225730#5490156 and T225730#6001349: >>! In T225730#6001349, @kostajh wrote: >> I cant tell wh... [19:39:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Zuul: Abort a Zuul pipeline when one job completed with failures. - https://phabricator.wikimedia.org/T248531 (10Krinkle) [19:39:19] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Zuul, 10Code-Health, 10Developer Productivity: Change zuul scheduler's failure check from areAllJobsComplete to didAnyJobFail to abort CI failures earlier - https://phabricator.wikimedia.org/T248596 (10Krinkle) [19:39:37] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) @Dzahn For the data migrations we will... [19:39:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Zuul, 10Developer Productivity: Abort a Zuul pipeline when one job completed with failures. - https://phabricator.wikimedia.org/T248531 (10Krinkle) [19:40:27] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments: 1.35.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T247774 (10Jdforrester-WMF) [19:42:04] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [19:43:46] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [19:44:19] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10Dzahn) a:03Dzahn [19:44:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 (10Dzahn) [19:44:31] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10Dzahn) 05Stalled→03Open [19:46:02] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 (10Dzahn) [19:54:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Zuul, 10Developer Productivity: Abort a Zuul pipeline when one job completed with failures (change zuul scheduler's failure check... - https://phabricator.wikimedia.org/T248531 [19:57:06] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T247773 (10Jdforrester-WMF) 05Open→03Resolved [20:06:17] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) Looking at the Test Result from... [20:11:36] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Daimona) >>! In T225730#6024365, @Krinkle... [20:16:02] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Jdforrester-WMF) >>! In T225730#6024375,... [20:21:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Reduce size of artifacts stored on the CI Jenkins master - https://phabricator.wikimedia.org/T249268 (10hashar) [20:24:49] (03PS1) 10Hashar: jjb: disable SonarQube debug output [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) [20:25:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: Reduce size of artifacts stored on the CI Jenkins master - https://phabricator.wikimedia.org/T249268 (10hashar) a:03hashar [20:26:18] (03PS2) 10Hashar: jjb: disable SonarQube debug output [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) [20:27:26] (03CR) 10Hashar: "Looking at the console such as https://integration.wikimedia.org/ci/job/mwext-codehealth-patch/lastSuccessfulBuild/console it is mostly d" [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [20:27:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: Reduce size of artifacts stored on the CI Jenkins master - https://phabricator.wikimedia.org/T249268 (10hashar) p:05Triage→03Medium [20:28:06] (03CR) 10Jforrester: "Wow, that's why it was so huge?" [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [20:29:18] (03CR) 10Hashar: "Yes, that -X turns debug logging:" [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [20:29:40] (03CR) 10Kosta Harlan: jjb: disable SonarQube debug output (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [20:29:43] James_F: and mediawiki-fresnel-patch-docker is 60GBytes ;D [20:30:02] Woah. [20:30:35] (03CR) 10Kosta Harlan: "it's fine with me -- seems like we should remove it from both the core and extension jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [20:30:46] hashar: Also, you claimed to be ending for the day *hours* ago. :-P [20:31:05] oh [20:31:22] yeah well I really mean: I need to have dinner [20:31:27] Shoo. [20:32:21] (03PS3) 10Hashar: jjb: disable SonarQube debug output [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) [20:33:17] (03CR) 10Hashar: "Amended for the core jobs as well (thank you Kosta!)" [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [20:33:19] (03CR) 10Jforrester: [C: 03+1] jjb: disable SonarQube debug output (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [20:33:26] but for those trace files [20:33:32] I think I can get them gzipped on the master [20:34:17] then find a way for apache to serve them as is with some headers to mention its gzipped and it is actually a json type [20:35:01] so that theorically Apache will stream the file as is from Jenkins, the browser will uncompress the stream and render it directly since it is a json type [20:35:04] maybe that will work [20:35:29] * James_F nods. [20:36:59] (03CR) 10Hashar: "@Kosta if that works for you will update those jobs :]" [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [21:06:52] (03CR) 10Kosta Harlan: [C: 03+1] jjb: disable SonarQube debug output [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [21:07:57] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/GroupWhitelist] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/585606 [21:07:59] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/GroupWhitelist] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/585606 (owner: 10QChris) [21:08:01] (03PS1) 10QChris: Import done. Revoke import grants [extensions/GroupWhitelist] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/585607 [21:08:03] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/GroupWhitelist] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/585607 (owner: 10QChris) [21:18:48] 10Continuous-Integration-Infrastructure (Slipway), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Release Pipeline, 10Wikimedia-Portals: Migrate wikimedia-portals-build to Docker container - https://phabricator.wikimedia.org/T213806 (10Jdforrester-WMF) [21:30:13] (03CR) 10Hashar: [C: 03+2] "Jobs updated! Thank you Kosta :)" [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [21:30:41] hashar: Ha, I was about to do it for you. :-) [21:30:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: Reduce size of artifacts stored on the CI Jenkins master - https://phabricator.wikimedia.org/T249268 (10hashar) [21:31:05] (03Merged) 10jenkins-bot: jjb: disable SonarQube debug output [integration/config] - 10https://gerrit.wikimedia.org/r/585600 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar) [21:31:05] James_F: thanks !! :] [21:31:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: Reduce size of artifacts stored on the CI Jenkins master - https://phabricator.wikimedia.org/T249268 (10hashar) [21:33:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: Reduce size of artifacts stored on the CI Jenkins master - https://phabricator.wikimedia.org/T249268 (10hashar) For Fresnel we might be... [21:36:20] (03PS3) 10Jforrester: dockerfiles: [composer-php70] Inject GitHub OAuth token into composer runtime [integration/config] - 10https://gerrit.wikimedia.org/r/583483 (https://phabricator.wikimedia.org/T106452) (owner: 10BryanDavis) [21:37:07] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: [composer-php70] Inject GitHub OAuth token into composer runtime [integration/config] - 10https://gerrit.wikimedia.org/r/583483 (https://phabricator.wikimedia.org/T106452) (owner: 10BryanDavis) [21:44:14] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (<11.11%) [21:46:52] !log deployment-deploy02: apt-get autoremove --purge [21:46:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:47:25] unlikely to fix it , but it is a low hanging fruit [21:47:28] * hashar sleeps [21:48:06] (03PS4) 10Jforrester: dockerfiles: [composer-php70] Inject GitHub OAuth token into composer runtime [integration/config] - 10https://gerrit.wikimedia.org/r/583483 (https://phabricator.wikimedia.org/T106452) (owner: 10BryanDavis) [21:48:08] (03PS1) 10Jforrester: jjb: Update composer docker images and inject GitHub OAuth token [integration/config] - 10https://gerrit.wikimedia.org/r/585609 (https://phabricator.wikimedia.org/T106452) [21:48:52] (03CR) 10Brennen Bearnes: [C: 03+1] "> I guess this is something worth discussing a bit further. Where possible, I think it makes sense to keep development settings universal " (031 comment) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/582498 (https://phabricator.wikimedia.org/T247072) (owner: 10Kosta Harlan) [21:49:06] (03CR) 10jerkins-bot: [V: 04-1] jjb: Update composer docker images and inject GitHub OAuth token [integration/config] - 10https://gerrit.wikimedia.org/r/585609 (https://phabricator.wikimedia.org/T106452) (owner: 10Jforrester) [21:51:32] (03PS2) 10Jforrester: jjb: Update composer docker images and inject GitHub OAuth token [integration/config] - 10https://gerrit.wikimedia.org/r/585609 (https://phabricator.wikimedia.org/T106452) [21:54:15] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [21:54:37] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [composer-php70] Inject GitHub OAuth token into composer runtime [integration/config] - 10https://gerrit.wikimedia.org/r/583483 (https://phabricator.wikimedia.org/T106452) (owner: 10BryanDavis) [21:55:35] (03Merged) 10jenkins-bot: dockerfiles: [composer-php70] Inject GitHub OAuth token into composer runtime [integration/config] - 10https://gerrit.wikimedia.org/r/583483 (https://phabricator.wikimedia.org/T106452) (owner: 10BryanDavis) [21:55:43] James_F: is that binding stuff not needed to get the env var set? [21:58:12] bd808: It is, but it's a JJB change, so we can't deploy it until we've merged and deployed and built the docker changes. [21:58:22] Also, you applied it to the wrong jobs. [21:58:38] We want `mediawiki-core-php72-phan-docker` etc. to have the credentials. [21:59:02] But the php.yaml file only defines the very basic composer ones, which quibble's inheritance tree doesn't use. [21:59:09] * James_F is thinking about how to do it. [21:59:59] !log Docker: [composer-php70] Inject GitHub OAuth token into composer runtime T106452 T248387 [22:00:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:00:02] T248387: Could not authenticate against github.com - https://phabricator.wikimedia.org/T248387 [22:00:03] T106452: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452 [22:00:43] (03PS3) 10Jforrester: jjb: Update composer docker images and inject GitHub OAuth token [integration/config] - 10https://gerrit.wikimedia.org/r/585609 (https://phabricator.wikimedia.org/T106452) [22:00:56] (03CR) 10Jforrester: [C: 04-1] "We want this to go everywhere, not just to those few jobs. :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/585609 (https://phabricator.wikimedia.org/T106452) (owner: 10Jforrester) [22:02:09] Actually, I'll split out the wrapper changes even further. [22:02:15] Just to make it clearer. [22:03:52] James_F: cool. thanks for the explanation. I was 100% flailing in the dark with the jjb changes :) [22:05:26] (03PS4) 10Jforrester: jjb: Inject GitHub OAuth token [integration/config] - 10https://gerrit.wikimedia.org/r/585609 (https://phabricator.wikimedia.org/T106452) [22:05:28] (03PS1) 10Jforrester: jjb: Update composer docker images [integration/config] - 10https://gerrit.wikimedia.org/r/585610 [22:05:36] bd808: All of us do. It's a nightmare of a system. [22:05:51] * James_F is very much hoping that the new CI system will get greenlit. [22:06:30] James_F: well, speaking from experience of managing 100s of Jenkins jobs by direct manipulation of XML on the filesystem, JJB is better than that :) [22:09:01] Sure. [22:09:34] * James_F has to wait for the server-side docker build before he can locally test the jjb default changes. [22:19:40] Only three left! [22:21:34] OK, all built, now publishes. [22:41:10] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T247773 (10Pchelolo) [22:41:35] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T247773 (10Pchelolo) [22:43:43] (03PS5) 10Jforrester: jjb: Inject GitHub OAuth token as a default for all jobs [integration/config] - 10https://gerrit.wikimedia.org/r/585609 (https://phabricator.wikimedia.org/T106452) [22:45:22] Okie-dokie. [22:45:46] I don't think I've ever done a JJB patch to 385 jobs before. [22:45:53] 😱 [22:50:59] (03CR) 10Jforrester: [C: 03+2] jjb: Update composer docker images [integration/config] - 10https://gerrit.wikimedia.org/r/585610 (owner: 10Jforrester) [22:52:03] (03Merged) 10jenkins-bot: jjb: Update composer docker images [integration/config] - 10https://gerrit.wikimedia.org/r/585610 (owner: 10Jforrester) [22:53:16] !log Updated ooui-npm-special-node10-plus-composer-docker with COMPOSER_GITHUB_OAUTHTOKEN injected T248387 [22:53:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:53:18] T248387: Could not authenticate against github.com - https://phabricator.wikimedia.org/T248387 [23:12:44] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review: Could not authenticate against github.com - https://phabricator.wikimedia.org/T248387 (10Jdforrester-WMF) 05Open→03Resolv... [23:14:07] (03CR) 10Jforrester: [C: 03+2] "Deployed. Works." [integration/config] - 10https://gerrit.wikimedia.org/r/585609 (https://phabricator.wikimedia.org/T106452) (owner: 10Jforrester) [23:15:02] (03Merged) 10jenkins-bot: jjb: Inject GitHub OAuth token as a default for all jobs [integration/config] - 10https://gerrit.wikimedia.org/r/585609 (https://phabricator.wikimedia.org/T106452) (owner: 10Jforrester) [23:16:03] bd808: Nice work. :-) [23:16:21] yay! Glad that it is working [23:16:41] and thank you James_F for carrying this across the deployment line [23:16:46] Sadly it's not a fix for all WMCS things, but it'll segment off a lot of the traffic. [23:16:49] Ha. Happy to help. [23:17:04] meh. other folks can use their own tokens if needed [23:17:33] I think this will separate the CI traffic from the sahred IP counter upstream, but that's untested at this point [23:17:38] I doubt all WMCS users together other than us will often reach the threshold. [23:17:44] Yeah, fair, I'm assuming. [23:17:56] Let's wait and see if it's still an issue. [23:18:07] * James_F wanders off to get tea for the next meeting. [23:18:07] +1