[00:28:54] 10MediaWiki-Releasing, 10AntiSpoof, 10MW-1.36-release: Bundle AntiSpoof extension with MediaWiki - https://phabricator.wikimedia.org/T191736 (10Legoktm) [00:30:02] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.36-release: Expand the set of bundled extensions and skins in MediaWiki 1.36 - https://phabricator.wikimedia.org/T246381 (10Legoktm) [00:30:05] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10Legoktm) [00:30:18] 10MediaWiki-Releasing, 10AntiSpoof, 10MW-1.36-release: Bundle AntiSpoof extension with MediaWiki - https://phabricator.wikimedia.org/T191736 (10Legoktm) [00:30:19] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.36-release: Expand the set of bundled extensions and skins in MediaWiki 1.36 - https://phabricator.wikimedia.org/T246381 (10Legoktm) [00:30:22] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10Legoktm) [00:32:42] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.35-release: Expand the set of bundled extensions and skins in MediaWiki 1.35 - https://phabricator.wikimedia.org/T232854 (10Legoktm) 05Open→03Resolved a:03Jdforrester-WMF Newly bundled in 1.35: SecureLinkFi... [00:32:44] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic: Expand the set of bundled extensions and skins to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349 (10Legoktm) [00:36:19] 10MediaWiki-Releasing, 10MediaWiki-extensions-OATHAuth, 10MW-1.36-release: Bundle WebAuthn extension with MediaWiki - https://phabricator.wikimedia.org/T258007 (10Legoktm) [01:18:41] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Epic, 10Technical-Debt: Figure out which production extensions have conflicting tests and fix them - https://phabricator.wikimedia.org/T238492 (10Legoktm) (#ci-test-error because there's no actual CI failure...yet?) [01:21:36] (03PS1) 10Jeena Huneidi: Support env vars and credentials in docker run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/612692 (https://phabricator.wikimedia.org/T257526) [01:32:40] (03PS2) 10Jeena Huneidi: Support env vars and credentials in docker run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/612692 (https://phabricator.wikimedia.org/T257526) [01:49:54] (03PS3) 10Jeena Huneidi: Support env vars and credentials in docker run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/612692 (https://phabricator.wikimedia.org/T257526) [01:55:03] (03PS4) 10Jeena Huneidi: Support env vars and credentials in docker run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/612692 (https://phabricator.wikimedia.org/T257526) [02:05:27] (03PS5) 10Jeena Huneidi: Support env vars and credentials in docker run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/612692 (https://phabricator.wikimedia.org/T257526) [02:46:16] 10MediaWiki-Releasing, 10Core Platform Team, 10MediaWiki-Vendor, 10MW-1.35-notes, 10MW-1.35-release: Prune /vendor for REL1_35 - https://phabricator.wikimedia.org/T256379 (10Legoktm) 05Open→03Resolved [02:46:18] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.35-notes, 10MW-1.35-release: Release MW 1.35 - https://phabricator.wikimedia.org/T256375 (10Legoktm) [04:35:19] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.36-release: Enhance installer to support dependencies between skins and extensions - https://phabricator.wikimedia.org/T196670 (10Legoktm) [05:38:32] 10Beta-Cluster-Infrastructure: Certificate for *.beta.wmflabs.org has expired - https://phabricator.wikimedia.org/T257968 (10Vgutierrez) I think we have two bugs here: 1. The API service must be restarted as well after an acme-chief upgrade. 2. The API service shouldn't list non allowed files, I'd suspect that d... [06:06:27] (03CR) 10Jforrester: [C: 03+1] Validate .phpcs.xml files using phpcs.xsd [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424970 (owner: 10Legoktm) [08:24:44] 10Project-Admins: Project creation for tool commonsapi.toolforge.org - https://phabricator.wikimedia.org/T257937 (10Aklapper) [10:46:55] 10Phabricator (Upstream), 10Upstream: Feature request: New notification option for when somebody @mentions you in a Task - https://phabricator.wikimedia.org/T257633 (10MarcoAurelio) Thank you @epriestley - I'll try the Mail Stamps feature. Thanks for your detailed reply and for your hard work Upstream which is... [11:45:48] 10Beta-Cluster-Infrastructure, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Enable CheckUser for beta cluster - https://phabricator.wikimedia.org/T214820 (10Aklapper) 05Stalled→03Open [11:46:18] 10Beta-Cluster-Infrastructure, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Enable CheckUser for beta cluster - https://phabricator.wikimedia.org/T214820 (10Aklapper) @Rxy: Could you answer the last question please? Thanks! [11:52:53] (03PS1) 10Lokal Profil: Add Zuul template for WikispeechSpeechDataCollector [integration/config] - 10https://gerrit.wikimedia.org/r/612855 (https://phabricator.wikimedia.org/T258050) [12:06:17] 10Gerrit: Gerrit 3.2 UI does not allow to remove reviewers with long names - https://phabricator.wikimedia.org/T257105 (10Aklapper) [12:06:25] 10Gerrit: Gerrit 3.2 UI does not allow to remove reviewers with long names - https://phabricator.wikimedia.org/T257105 (10Aklapper) [12:06:28] 10Gerrit, 10Patch-For-Review: Gerrit 3.2 upgrade - https://phabricator.wikimedia.org/T254158 (10Aklapper) [12:07:04] 10commit-message-validator: commit-message-validator prints a weird error message "Expected at least 3 lines" - https://phabricator.wikimedia.org/T257107 (10Aklapper) [12:35:30] dancy: thanks for fixing T239928! [12:35:31] T239928: wikibugs2 should not post on IRC for Gerrit changes marked 'WIP' - https://phabricator.wikimedia.org/T239928 [13:08:20] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T256669 (10Jdforrester-WMF) Group1 looks reasonable; one logspam issue, {T258056}, but m... [13:31:39] 10Phabricator: Phabricator: "500 Internal Server Error" when using VPN - https://phabricator.wikimedia.org/T258059 (10NicoV) [14:34:03] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T256669 (10Lucas_Werkmeister_WMDE) [14:34:18] ^ we’re already fixing that one, no need to roll back the train [14:34:30] but I thought it should be linked to the train blockers [14:38:27] 10Release-Engineering-Team, 10ApiFeatureUsage, 10Commons, 10Pywikibot, 10Upstream: Persistant error 500 getting category members - https://phabricator.wikimedia.org/T255981 (10matej_suchanek) When looking into T258036, I could reproduce this. This is the request sent to server: `lang=json "action": "que... [15:02:14] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T256669 (10Majavah) [15:24:40] (03CR) 10C. Scott Ananian: [C: 03+1] "LGTM, I didn't realize that wasn't included in any of the extension templates by default." [integration/config] - 10https://gerrit.wikimedia.org/r/612673 (https://phabricator.wikimedia.org/T233736) (owner: 10Arlolra) [15:41:07] (03PS1) 10Ahmon Dancy: Fix the example executions of blubber in README.md [blubber] - 10https://gerrit.wikimedia.org/r/612885 [15:43:40] (03CR) 10Reedy: [C: 03+2] Turn on API integration tests for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/612673 (https://phabricator.wikimedia.org/T233736) (owner: 10Arlolra) [15:44:44] (03Merged) 10jenkins-bot: Turn on API integration tests for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/612673 (https://phabricator.wikimedia.org/T233736) (owner: 10Arlolra) [15:54:17] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/612673 [15:54:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:16:05] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10Isaac) Ok, let's go ahead and enable it then to see! [16:54:06] (03PS1) 10Art.tsymbar: add tsymbar, remove baltai [integration/config] - 10https://gerrit.wikimedia.org/r/612901 [17:01:20] (03CR) 10Art.tsymbar: "Please, have a look." [integration/config] - 10https://gerrit.wikimedia.org/r/612901 (owner: 10Art.tsymbar) [17:08:35] (03CR) 10Ppchelko: [C: 03+1] add tsymbar, remove baltai [integration/config] - 10https://gerrit.wikimedia.org/r/612901 (owner: 10Art.tsymbar) [17:29:02] 10Release-Engineering-Team (Onboarding), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline (Blubber): Blubber: Allow more than one layer of copying - https://phabricator.wikimedia.org/T254629 (10thcipriani) [17:29:43] 10Release-Engineering-Team (Onboarding), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline (Blubber): Blubber: Allow more than one layer of copying - https://phabricator.wikimedia.org/T254629 (10thcipriani) p:05Triage→03Medium a:03dancy [17:31:58] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review: Pipelinelib: support passing environment variables to docker images - https://phabricator.wikimedia.org/T257526 (10thcipriani) p:05Triage→03Medium [17:32:21] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline: Make pipelinelib able to update deployment chart image tags - https://phabricator.wikimedia.org/T255835 (10thcipriani) p:05Triage→03Medium [17:39:12] 10Gerrit, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): check gerrit-replica healthcheck alerts - https://phabricator.wikimedia.org/T247186 (10thcipriani) p:05High→03Medium [17:43:46] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 715 bytes in 0.090 second response time [17:49:15] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.88 ms [17:49:22] 10Phabricator, 10Editing-Team-Request, 10Editing-team (Tracking): Custom task form for #Editing-team-request - https://phabricator.wikimedia.org/T256123 (10mmodell) @aklapper: I'll take care of form visibility but for posterity: To make a form visible to specific group of people: 1. "Mark as create form" 2... [17:51:36] 10Phabricator, 10Editing-Team-Request, 10Editing-team (Tracking): Custom task form for #Editing-team-request - https://phabricator.wikimedia.org/T256123 (10mmodell) 05Open→03Resolved I guess this is resolved? Thanks for your help @aklapper! [17:54:14] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [17:55:04] 10Diffusion, 10Phabricator, 10Operations, 10Puppet: Diffusion (Phabricator) operations-puppet repo synchronization error - https://phabricator.wikimedia.org/T257895 (10mmodell) Some of the files in the OPUP.git/objects directory were owned by root. I fixed that and it should fix the replication error. [17:58:35] 10LibUp, 10WikibaseMediaInfo, 10Patch-For-Review: libup run of repo WikibaseMediaInfo fails on npm ci with failing wikibase dependencies - https://phabricator.wikimedia.org/T243787 (10matthiasmullie) The latest is actually intentional in this case, because it's only used in tests and we want to ensure that t... [18:07:52] 10Diffusion, 10Phabricator, 10Operations, 10Puppet: Diffusion (Phabricator) operations-puppet repo synchronization error - https://phabricator.wikimedia.org/T257895 (10mmodell) 05Open→03Resolved a:03mmodell I just checked and there are no other repositories with objects owned by root. I also see that... [18:10:09] !log preparing to upgrade jenkins on contint1001, contint1002 [18:10:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:10:14] 10Release-Engineering-Team (Logspam), 10css-sanitizer, 10Wikimedia-production-error: Wikimedia\CSS\Objects\ComponentValueList may not contain tokens of type "[". - https://phabricator.wikimedia.org/T258093 (10jeena) [18:41:40] !log finished jenkins upgrade & restart on all hosts [18:41:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:41:46] PROBLEM - Free space - all mounts on integration-agent-docker-1012 is CRITICAL: CRITICAL: integration.integration-agent-docker-1012.diskspace._srv.byte_percentfree (<10.00%) [18:42:54] maintenance-disconnect-full-disks build 194434 integration-agent-docker-1012 (/srv: 98%): OFFLINE due to disk space [18:45:20] maintenance-disconnect-full-disks build 194435 integration-agent-docker-1012: OFFLINE due to disk space [18:50:21] maintenance-disconnect-full-disks build 194436 integration-agent-docker-1009 (/srv: 99%): OFFLINE due to disk space [18:52:21] PROBLEM - Free space - all mounts on integration-agent-docker-1009 is CRITICAL: CRITICAL: integration.integration-agent-docker-1009.diskspace._srv.byte_percentfree (<33.33%) [18:54:39] PROBLEM - Free space - all mounts on integration-agent-docker-1010 is CRITICAL: CRITICAL: integration.integration-agent-docker-1010.diskspace._srv.byte_percentfree (<20.00%) [18:55:20] maintenance-disconnect-full-disks build 194437 integration-agent-docker-1010 (/srv: 98%): OFFLINE due to disk space [19:01:44] RECOVERY - Free space - all mounts on integration-agent-docker-1012 is OK: OK: All targets OK [19:05:58] Hey folks. I'm trying to have chrisalbon push some changes to a gerrit repo directly (this is a work-around for broken mirroring). We are getting "authentication failed" and he's using his creds from Wikitech. Is that right? [19:06:10] Relevant group that *should* give him push rights: https://gerrit.wikimedia.org/r/admin/groups/cddbf2315647ba438f5741826fffaeedfdcdfe8a,members [19:06:29] But give the "Authentication failed" error, I'm guessing it's not rights but rather we're using the wrong creds. [19:07:03] creds from wikitech? [19:07:12] Is he trying to push over HTTPS? [19:07:31] yes [19:08:40] it's generally easier to just use SSH [19:08:41] but.. [19:08:49] IIRC there's a separate password used for HTTPS pushing [19:08:54] you generate it in gerrit settings? [19:08:55] yeah, I was just trying to get a link to it [19:09:00] this could be horifically outdated info though :) [19:09:05] https://gerrit.wikimedia.org/r/settings/#HTTPCredentials [19:09:26] https://gerrit.wikimedia.org/r/settings/#HTTPCredentials [19:09:27] darn [19:09:41] RECOVERY - Free space - all mounts on integration-agent-docker-1010 is OK: OK: All targets OK [19:09:45] halfak, ^ [19:09:50] Aha! [19:09:59] I wonder if I did that at some point. [19:10:09] awesome thanks all [19:10:18] I am definitely switching to ssh later today [19:10:18] maintenance-disconnect-full-disks build 194440 integration-agent-docker-1009: OFFLINE due to disk space [19:10:19] maintenance-disconnect-full-disks build 194440 integration-agent-docker-1010: OFFLINE due to disk space [19:10:19] maintenance-disconnect-full-disks build 194440 integration-agent-docker-1012: OFFLINE due to disk space [19:10:22] Thanks for the quick help! [19:15:53] 10Phabricator: Phabricator: "500 Internal Server Error" when using VPN - https://phabricator.wikimedia.org/T258059 (10Aklapper) @NicoV: Likely depends on your IP and provider (we block some ranges due to vandalism), like in {T229575}. Not sure you'd like to share such private information in this public ticket. A... [19:17:25] RECOVERY - Free space - all mounts on integration-agent-docker-1009 is OK: OK: All targets OK [19:35:19] maintenance-disconnect-full-disks build 194445 integration-agent-docker-1010: OFFLINE due to disk space [19:35:19] maintenance-disconnect-full-disks build 194445 integration-agent-docker-1012: OFFLINE due to disk space [19:37:58] (03PS2) 10Reedy: Run node10-docker on research/landing-page [integration/config] - 10https://gerrit.wikimedia.org/r/612284 (https://phabricator.wikimedia.org/T257843) [19:38:07] (03CR) 10Reedy: [C: 03+2] Run node10-docker on research/landing-page [integration/config] - 10https://gerrit.wikimedia.org/r/612284 (https://phabricator.wikimedia.org/T257843) (owner: 10Reedy) [19:39:08] (03Merged) 10jenkins-bot: Run node10-docker on research/landing-page [integration/config] - 10https://gerrit.wikimedia.org/r/612284 (https://phabricator.wikimedia.org/T257843) (owner: 10Reedy) [19:39:40] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/612284 [19:39:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:42:33] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T256669 (10Urbanecm) Heads-up: Not sure if T258097 is relevant to wmf.41 deployment, but... [19:43:17] !log bringing integration-agent-docker-1010 back online in jenkins [19:43:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:45:50] Urbanecm: Meh. Probably just a “don’t do that then”. [19:45:57] (03PS1) 10C. Scott Ananian: Load Parsoid as an extension when it is cloned into services/parsoid [integration/quibble] - 10https://gerrit.wikimedia.org/r/612937 [19:46:20] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10Reedy) Ok first run against a noop change https://gerrit.wikimedia.org/r/c/research/landing-page/+/612907 Nothing causing a failure, but looks like the pack... [19:46:26] (03PS2) 10C. Scott Ananian: Load Parsoid as an extension when it is cloned into services/parsoid [integration/quibble] - 10https://gerrit.wikimedia.org/r/612937 (https://phabricator.wikimedia.org/T227352) [19:46:32] James_F: I guess it used to work. As I said, I'm not sure that's relevant only to babel, or if something more is going on - it's just a headsup :) [19:46:50] Yeah, thanks. [19:47:18] !log brennen@integration-agent-docker-1012:/srv/jenkins/workspace/workspace$ sudo rm -rf * [19:47:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:47:27] It might well have been on the limit before, and recently passed it due to work somewhere like refactorings. [19:47:36] 10Gerrit: Gerrit is giving tips on how to write comments, make sure it does not send our comments to some third party - https://phabricator.wikimedia.org/T258102 (10QChris) [19:47:58] maybe :) [19:48:19] !log bringing integration-docker-1012 back online in jenkins [19:48:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:48:36] 10Gerrit: Gerrit is giving tips on how to write comments, make sure it does not send our comments to some third party - https://phabricator.wikimedia.org/T258102 (10QChris) 05Open→03Resolved a:03QChris As explained in the bug description above, our comments/reviews/... are not getting sent to a third party... [19:48:57] 10Gerrit: Gerrit is giving tips on how to write comments, make sure it does not send our comments to some third party - https://phabricator.wikimedia.org/T258102 (10QChris) [19:49:00] 10Gerrit, 10Patch-For-Review: Gerrit 3.2 upgrade - https://phabricator.wikimedia.org/T254158 (10QChris) [19:50:12] (03Abandoned) 10C. Scott Ananian: (wip) Add configuration for Parsoid/PHP [integration/quibble] - 10https://gerrit.wikimedia.org/r/553333 (https://phabricator.wikimedia.org/T218534) (owner: 10Kosta Harlan) [19:51:33] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10Reedy) Oh, and maybe a bit of a love more generally ;) https://github.com/wikimedia/research-landing-page/blob/master/package.json#L11 ``lang=json "url... [19:57:12] (03CR) 10C. Scott Ananian: "Even though we have Parsoid checked out and using the standard extension CI tests, if you look at (eg) https://integration.wikimedia.org/c" [integration/quibble] - 10https://gerrit.wikimedia.org/r/612937 (https://phabricator.wikimedia.org/T227352) (owner: 10C. Scott Ananian) [20:14:17] 10Phabricator, 10itWikiCon-2020: Create a countdown for itWikiCon 2020 call for papers - https://phabricator.wikimedia.org/T255668 (10Aklapper) >>! In T255668#6295957, @valerio.bozzolan wrote: > Thank you @Aklapper. I also wonder if we could just [[ https://en.wikipedia.org/wiki/Wikipedia:Be_bold | be bold ]]... [20:16:13] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10Isaac) > The URL is seemingly to an old (or, probably more accurately out of date) version of the repo Ooof...thanks for catching that. Easy to fix. I'll sta... [20:17:04] !log deploying test version of maintenance-disconnect-full-disks (which might be wrong) [20:17:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:21:18] (03CR) 10Kosta Harlan: "cool! Just flagging that as noted in https://gerrit.wikimedia.org/r/c/integration/quibble/+/553333/1#message-65486b186dd4f4ebc00122bf0b1a0" [integration/quibble] - 10https://gerrit.wikimedia.org/r/612937 (https://phabricator.wikimedia.org/T227352) (owner: 10C. Scott Ananian) [20:28:50] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10Reedy) There's probably someone in #release-engineering-team or similar that can probably advise (better than I can) about updating the gruntfile to be gener... [20:29:14] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.37 ms [20:32:46] PROBLEM - Free space - all mounts on integration-agent-docker-1012 is CRITICAL: CRITICAL: integration.integration-agent-docker-1012.diskspace._var_lib_docker.byte_percentfree (<10.00%) [20:33:58] !log taking integration-agent-docker-1012 offline to experiment with maintenance-disconnect-full-disks [20:34:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:34:14] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [20:36:16] Can we backport the addition of UserFactory::newAnonymous and make User::__construct not stable for calling in 1.35? [20:42:43] RECOVERY - Free space - all mounts on integration-agent-docker-1012 is OK: OK: All targets OK [20:43:26] 10Phabricator: Set visibility policy of the Phabricator's Countdown application to "public" - https://phabricator.wikimedia.org/T258106 (10valerio.bozzolan) [20:48:03] 10Phabricator: Set visibility policy of the Phabricator's Countdown application to "public" - https://phabricator.wikimedia.org/T258106 (10valerio.bozzolan) [20:48:56] 10Phabricator: Set visibility policy of the Phabricator's Countdown application to "public" - https://phabricator.wikimedia.org/T258106 (10valerio.bozzolan) [20:49:26] 10Phabricator, 10itWikiCon-2020: Create a countdown for itWikiCon 2020 call for papers - https://phabricator.wikimedia.org/T255668 (10valerio.bozzolan) [20:49:31] 10Phabricator: Set visibility policy of the Phabricator's Countdown application to "public" - https://phabricator.wikimedia.org/T258106 (10valerio.bozzolan) [20:59:34] 10Phabricator, 10Operations, 10Security-Team: HTTP 500 error trying to access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Aklapper) [21:09:56] 10Phabricator: Set visibility policy of the Phabricator's Countdown application to "public" - https://phabricator.wikimedia.org/T258106 (10Aklapper) For the records, I'm neutral on this request [21:15:54] 10Phabricator: Set visibility policy of the Phabricator's Countdown application to "public" - https://phabricator.wikimedia.org/T258106 (10valerio.bozzolan) [21:19:18] 10Beta-Cluster-Infrastructure, 10User-bd808, 10cloud-services-team (Kanban): Rename labswiki to deploymentwiki - https://phabricator.wikimedia.org/T72108 (10bd808) [21:19:50] 10Release-Engineering-Team, 10Cloud-Services, 10User-bd808, 10cloud-services-team (Kanban): Get labtestwikitech/labtestweb2001 on the deployment train - https://phabricator.wikimedia.org/T163694 (10bd808) [21:30:31] (03PS1) 10Brennen Bearnes: maintenance-disconnect-full-disks: image cleanup logic [integration/config] - 10https://gerrit.wikimedia.org/r/612957 [21:36:02] (03CR) 10Ahmon Dancy: [C: 03+1] maintenance-disconnect-full-disks: image cleanup logic [integration/config] - 10https://gerrit.wikimedia.org/r/612957 (owner: 10Brennen Bearnes) [21:47:11] !log bringing integration-agent-docker-1012 back online [21:47:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:49:57] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap: beta-publish-deb failure do to "/srv/packages/db/LOG: permission denied" - https://phabricator.wikimedia.org/T250157 (10dancy) 05Open→03Resolved [22:10:43] (03PS6) 10Jeena Huneidi: Support env vars and credentials in docker run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/612692 (https://phabricator.wikimedia.org/T257526) [22:11:51] (03CR) 10jerkins-bot: [V: 04-1] Support env vars and credentials in docker run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/612692 (https://phabricator.wikimedia.org/T257526) (owner: 10Jeena Huneidi) [22:42:00] 10Gerrit, 10Toolforge, 10Documentation, 10cloud-services-team (Kanban): Document and update Gerrit groups for repositories related to Toolforge - https://phabricator.wikimedia.org/T159051 (10bd808) p:05Medium→03Triage [22:59:14] 10Phabricator, 10Operations, 10vm-requests, 10Patch-For-Review: eqiad: 1 VM request for aphlict - https://phabricator.wikimedia.org/T257617 (10Dzahn) a:03Dzahn [23:06:17] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Security-Team, 10Security, 10user-sbassett: Wikimedia deployers audit - https://phabricator.wikimedia.org/T237696 (10Dzahn) >>! In T237696#5934288, @sbassett wrote: > @chasemp - LGTM, thanks for... [23:15:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 716 bytes in 0.120 second response time [23:15:36] PROBLEM - Host deployment-maps05 is DOWN: CRITICAL - Host Unreachable (172.16.4.168)