[00:17:23] RECOVERY - Free space - all mounts on deployment-puppetmaster04 is OK: OK: All targets OK [00:24:02] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Android-app-Bugs, 10CirrusSearch, 10Discovery-Search, and 5 others: RESTBase 500 spike of all /page/related/ hits following 1.35.0-wmf.19 all-wiki deployment - https://phabricator.wikimedia.org/T245202 (10Jdforrester-WMF) Provisional plan is to m... [00:28:34] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Android-app-Bugs, 10CirrusSearch, 10Discovery-Search, and 5 others: RESTBase 500 spike of all /page/related/ hits following 1.35.0-wmf.19 all-wiki deployment - https://phabricator.wikimedia.org/T245202 (10EBernhardson) As an alternate solution, w... [00:29:40] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Android-app-Bugs, 10CirrusSearch, 10Discovery-Search, and 5 others: RESTBase 500 spike of all /page/related/ hits following 1.35.0-wmf.19 all-wiki deployment - https://phabricator.wikimedia.org/T245202 (10Jdforrester-WMF) Sounds workable. Let's c... [01:13:21] PROBLEM - Free space - all mounts on deployment-puppetmaster04 is CRITICAL: CRITICAL: deployment-prep.deployment-puppetmaster04.diskspace.root.byte_percentfree (<50.00%) [05:48:49] 10Phabricator: Phab stalling on "exporting data" of bulk job - https://phabricator.wikimedia.org/T245236 (10TerraCodes) [06:54:37] 10Phabricator, 10Security-Team, 10Security, 10Security Related: Update protect as security issue to use acl*security - https://phabricator.wikimedia.org/T245201 (10Bugreporter) As the policy of space is absolute, other users can not be invited to the task. [10:44:02] 10Phabricator: Phab stalling on "exporting data" of bulk job - https://phabricator.wikimedia.org/T245236 (10Aklapper) 05Open→03Declined p:05Triage→03Lowest > and I have more than just like 200 or etc) No, you have 6536. Given such vague search criteria I can imagine some code in the software stack to si... [10:44:18] 10Gerrit: Rename my Phabricator (done, thx) and Gerrit account - https://phabricator.wikimedia.org/T244537 (10Aklapper) a:05Aklapper→03None [11:11:15] (03Abandoned) 10Lokal Profil: Add Karl Wettin (WMSE) [integration/config] - 10https://gerrit.wikimedia.org/r/571478 (https://phabricator.wikimedia.org/T244839) (owner: 10Lokal Profil) [11:11:25] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10WMSE-Wikispeech-Speech-Data-Collector-2019, 10Patch-For-Review, 10User-LokalProfil: Add Kalle to trusted reviewers - https://phabricator.wikimedia.org/T244839 (10Lokal_Profil) [11:14:39] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10WMSE-Wikispeech-Speech-Data-Collector-2019, 10Patch-For-Review, 10User-LokalProfil: Add Kalle to trusted reviewers - https://phabricator.wikimedia.org/T244839 (10Lokal_Profil) a:05Jdforrester-WMF→03None >>! In T244839#5872791, @Jdforrester-WM... [11:22:28] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10WMSE-Wikispeech-Speech-Data-Collector-2019, 10Patch-For-Review, 10User-LokalProfil: Add Kalle to trusted reviewers - https://phabricator.wikimedia.org/T244839 (10Lokal_Profil) @kalle Your git clone errors sound like https://www.mediawiki.org/wiki... [11:22:31] 10Phabricator, 10Release-Engineering-Team, 10User-Addshore: "Production Error" tasks do not have a Story Points field - https://phabricator.wikimedia.org/T240343 (10Aklapper) I made https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/68/ show the "Estimate Story Points" field on 202... [11:38:10] (03PS1) 10Kosta Harlan: (wip) Create image for EventLogging service [releng/dev-images] - 10https://gerrit.wikimedia.org/r/572212 (https://phabricator.wikimedia.org/T223643) [12:44:29] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: Read for EngProd bookclub IV: What nobody tells you about documentation by Daniele Procida - https://phabricator.wikimedia.org/T245044 (10zeljkofilipin) p:05Triage→03Medium [12:44:43] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: Read for EngProd bookclub IV: What nobody tells you about documentation by Daniele Procida - https://phabricator.wikimedia.org/T245044 (10zeljkofilipin) [12:45:04] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Quality-and-Test-Engineering-Team (QTE), 10User-brennen, 10User-zeljkofilipin: Read for EngProd bookclub IV: What nobody tells you about documentation by Daniele Procida - https://phabricator.wikimedia.org/T245044 (10zeljkofilipin) [12:49:28] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:40:34] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:09:35] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Testing, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-CentralAuth, and 3 others: Login to at least en.wikipedia.beta.wmflabs.org and commons.wikimedia.beta.wmflabs.org sometimes fails with `There seems to... - https://phabricator.wikimedia.org/T243123 [16:28:43] marxarelli: expecting to try the train again today? [16:35:25] eh.. hold on. Friday deploys and Monday is a holiday. Deployment calendar is empty [16:40:48] ok, i was told before leaving last night that it might be rolling [16:41:40] if unblocked on friday the train will roll on monday (next business day), as the email notice said [16:41:58] if we want to change that policy then that requires a cross-team conversation (at least) [16:46:12] ebernhardson: i feel fairly comfortable about deploying but yeah, apergos, to your point i'll check in with other folks [16:46:37] (03PS3) 10Jforrester: build: Remove `| sort` from gen-changelog.sh [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/488227 (owner: 10Legoktm) [16:47:30] fwiw, i was about to call the deploy {{done}} yesterday based on the quiet mediawiki logs [16:47:36] uh so what I meant was: the email sent around yesterday about the train being blocked specifically said: [16:47:37] Once these issues are resolved train can resume. If these issues are resolved on a Friday the train will resume Monday. [16:47:49] so that may be boilerplate language but it's there because that's our expectations [16:48:01] but then noticed the varnish 500s spike and had to rollback and investigate further [16:48:17] if we want to change that generally, we need to hash that out with at least releng and sre [16:48:33] i don't think we want to change it generally [16:48:39] (03CR) 10Jforrester: [C: 03+2] build: Remove `| sort` from gen-changelog.sh [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/488227 (owner: 10Legoktm) [16:48:45] changing it now for this friday, unless it's a UBN, would be a strong no from me [16:49:11] (03Merged) 10jenkins-bot: build: Remove `| sort` from gen-changelog.sh [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/488227 (owner: 10Legoktm) [16:52:24] * marxarelli nods [16:54:28] heh, that is boilerplate language, but it does seem like good boilerplate language. [16:54:40] 10Phabricator, 10Security-Team, 10Security, 10Security Related: Update protect as security issue to use acl*security - https://phabricator.wikimedia.org/T245201 (10mmodell) @chasemp: The best way I can think of to make the transition easy (and ease future maintainability of the code) is to make the project... [16:54:52] i have a meeting in 5 but i can check in with folks following that [17:00:29] (03PS2) 10Jforrester: Detect missing & for ParamNameNoMatch error code and autofix [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/571023 (owner: 10Umherirrender) [17:00:38] (03CR) 10Jforrester: [C: 03+1] Detect missing & for ParamNameNoMatch error code and autofix [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/571023 (owner: 10Umherirrender) [17:08:18] RECOVERY - Free space - all mounts on deployment-puppetmaster04 is OK: OK: All targets OK [17:16:08] (03CR) 10Thcipriani: [C: 03+2] "Nice :)" (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/571849 (https://phabricator.wikimedia.org/T243330) (owner: 10Dduvall) [17:16:43] is this the right place to ask questions about Blubber? [17:17:32] (03Merged) 10jenkins-bot: train-deploy-notes: Run when wgVersion is changed from -alpha [integration/config] - 10https://gerrit.wikimedia.org/r/571849 (https://phabricator.wikimedia.org/T243330) (owner: 10Dduvall) [17:17:51] hnowlan yes [17:19:19] (03CR) 1020after4: [C: 03+1] "This is going to break sooner or later but +1" [integration/config] - 10https://gerrit.wikimedia.org/r/571849 (https://phabricator.wikimedia.org/T243330) (owner: 10Dduvall) [17:19:52] cool - I have a service that builds a container fine, but when the container comes up the included service files are owned by `somebody:somebody` but the container's entrypoint is run as `runuser`. When the service starts, the runuser user can't read the service files and so it errors out. Am I missing something basic that's causing this? [17:20:18] !log updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/571849 [17:20:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:20:32] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep (Puppet agent broken in Beta Cluster) - https://phabricator.wikimedia.org/T243226 (10herron) Turning off debug logging on puppetmaster04 (I set `logdest = /dev/null` in `/etc/puppet/puppet.conf`) has helped with the disk usage, and... [17:22:47] hnowlan: the different users are intentional, but "somebody" should just not be able to write files, not not be able to read files. You can use "runs: {insecurely: true}" to override the users thing (https://wikitech.wikimedia.org/wiki/Blubber/User_Guide#Common_Config) but this jsounds like a bug for which I would refer you to marxarelli [17:23:19] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/571849 [17:23:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:30:54] (03PS1) 10Thcipriani: train-deploy-notes: bump docker image used for job [integration/config] - 10https://gerrit.wikimedia.org/r/572283 [17:30:56] (03CR) 1020after4: [C: 03+2] train-deploy-notes: bump docker image used for job [integration/config] - 10https://gerrit.wikimedia.org/r/572283 (owner: 10Thcipriani) [17:32:15] (03Merged) 10jenkins-bot: train-deploy-notes: bump docker image used for job [integration/config] - 10https://gerrit.wikimedia.org/r/572283 (owner: 10Thcipriani) [17:34:16] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/571849 [17:34:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:37:38] thcipriani: thanks! [17:40:12] (03Abandoned) 10Brennen Bearnes: config: Add basic validation steps [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/518333 (https://phabricator.wikimedia.org/T225335) (owner: 10Brennen Bearnes) [17:45:04] Is there anyone here who understands about puppetdb in deployment-prep? [17:47:24] andrewbogott: Kren.air might be the only one who knows the current state of that, honestly. I haven't been able to follow that work :( [17:47:43] (03PS1) 1020after4: Symlink the mwrelease dir to unbreak train-deploy-notes job [tools/release] - 10https://gerrit.wikimedia.org/r/572288 [17:47:53] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep (Puppet agent broken in Beta Cluster) - https://phabricator.wikimedia.org/T243226 (10Andrew) That last issue (the resolution failure) was a side-effect of work I was doing for T229441. That issue is resolved, but now the failure us... [17:48:05] thcipriani: ok. Looks like Keith has been working on that a bit too [17:48:48] (03PS2) 1020after4: Symlink the mwrelease dir to unbreak train-deploy-notes job [tools/release] - 10https://gerrit.wikimedia.org/r/572288 [17:49:57] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] "Tested with replication off, switched it to on and updated deploy." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/571621 (owner: 10Jeena Huneidi) [17:52:10] (03CR) 10Thcipriani: [C: 03+2] Symlink the mwrelease dir to unbreak train-deploy-notes job [tools/release] - 10https://gerrit.wikimedia.org/r/572288 (owner: 1020after4) [17:52:40] (03Merged) 10jenkins-bot: Symlink the mwrelease dir to unbreak train-deploy-notes job [tools/release] - 10https://gerrit.wikimedia.org/r/572288 (owner: 1020after4) [17:53:38] (03CR) 10Jforrester: "Are these hard symlinks rather than soft ones? Why does gerrit show them as A = added rather than W = [linked but with a W]?" [tools/release] - 10https://gerrit.wikimedia.org/r/572288 (owner: 1020after4) [18:00:50] (03CR) 10Brennen Bearnes: "> This doesn't work due to various random errors in the install process." [releng/dev-images] - 10https://gerrit.wikimedia.org/r/572212 (https://phabricator.wikimedia.org/T223643) (owner: 10Kosta Harlan) [18:15:13] 10MediaWiki-Codesniffer: Create sniff to help prevent logstash_formatter_key_conflict in mediawiki logs - https://phabricator.wikimedia.org/T245289 (10Reedy) [18:23:28] (03CR) 10Kosta Harlan: "This needs to be published still, I think." [releng/dev-images] - 10https://gerrit.wikimedia.org/r/571828 (owner: 10Kosta Harlan) [18:28:53] (03PS1) 10Thcipriani: Partially Revert "train-deploy-notes: Run when wgVersion is changed from -alpha" [integration/config] - 10https://gerrit.wikimedia.org/r/572298 [18:29:31] (03CR) 10Brennen Bearnes: [C: 03+2] "Acknowledging slightly maddening debugging session." [integration/config] - 10https://gerrit.wikimedia.org/r/572298 (owner: 10Thcipriani) [18:30:37] (03Merged) 10jenkins-bot: Partially Revert "train-deploy-notes: Run when wgVersion is changed from -alpha" [integration/config] - 10https://gerrit.wikimedia.org/r/572298 (owner: 10Thcipriani) [18:55:50] PROBLEM - Free space - all mounts on deployment-mwmaint01 is CRITICAL: CRITICAL: deployment-prep.deployment-mwmaint01.diskspace.root.byte_percentfree (<30.00%) [18:56:02] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<30.00%) [18:56:22] Project beta-scap-eqiad build #288175: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288175/ [19:03:55] Project beta-scap-eqiad build #288176: 04STILL FAILING in 5 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288176/ [19:04:35] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep (Puppet agent broken in Beta Cluster) - https://phabricator.wikimedia.org/T243226 (10Krenair) a:03Krenair [19:06:48] (03CR) 1020after4: "> Patch Set 2:" [tools/release] - 10https://gerrit.wikimedia.org/r/572288 (owner: 1020after4) [19:10:20] Project beta-scap-eqiad build #288177: 04STILL FAILING in 4 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288177/ [19:15:59] (03CR) 10Jforrester: "Odd. 🤷🏽‍♂️" [tools/release] - 10https://gerrit.wikimedia.org/r/572288 (owner: 1020after4) [19:19:13] Project beta-scap-eqiad build #288178: 04STILL FAILING in 4 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288178/ [19:21:07] (03CR) 1020after4: "see also: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/release/+/master/make-deploy-notes/" [tools/release] - 10https://gerrit.wikimedia.org/r/572288 (owner: 1020after4) [19:21:17] (03PS2) 10Brennen Bearnes: (wip) Create image for EventLogging service [releng/dev-images] - 10https://gerrit.wikimedia.org/r/572212 (https://phabricator.wikimedia.org/T223643) (owner: 10Kosta Harlan) [19:27:15] > rsync: write failed on "/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-mk.cdb.json": No space left on device (28) [19:27:21] well, that'd do it, I guess [19:29:12] Project beta-scap-eqiad build #288179: 04STILL FAILING in 4 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288179/ [19:29:23] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments: 1.35.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T233868 (10Jdforrester-WMF) [19:30:38] (03CR) 10Jforrester: "The mysteries of gerrit." [tools/release] - 10https://gerrit.wikimedia.org/r/572288 (owner: 1020after4) [19:31:14] !log deployment-snapshot01:sudo apt-get clean cleared 1.2G of space [19:31:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:42:42] Project beta-scap-eqiad build #288180: 04STILL FAILING in 8 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288180/ [19:44:33] James_F: I don't understand https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-docker/46414/testReport/junit/(root)/ExtensionJsonValidationTest/testPassesValidation_with_data_set__1/ [19:44:41] can you help? [19:46:06] * James_F Hmm. [19:47:16] validateRegistrationFile.php indeed fails [19:47:22] but I'm not seeing why [19:47:36] description is right and the license is an spdx-approved tag [19:48:03] This is for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/UploadBlacklist/+/571068 – it's thinking that the `license` key isn't set? [19:48:27] It's a schema validation error; it's valid JSON. [19:48:45] but it is [19:48:56] the licence is there [19:49:03] Project beta-scap-eqiad build #288181: 04STILL FAILING in 4 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288181/ [19:50:07] What happens if you use the more normal format for author? [19:50:37] same, locally [19:50:44] * James_F pushes. [19:50:46] let me remove the url and resubmit [19:50:59] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [19:51:04] If https://gerrit.wikimedia.org/r/c/mediawiki/extensions/UploadBlacklist/+/571068/6..7 fixes it I'll be sad. [19:51:07] Anyway, back in a little. [19:51:16] lol James_F we commit-conflicted [19:51:24] now I'll re-amend the commit-message [19:51:26] :) [19:55:36] https://pastebin.com/hz2eENJT [19:59:13] Project beta-scap-eqiad build #288182: 04STILL FAILING in 4 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288182/ [20:05:40] !log deployment-mwmaint01:rm lingering .~tmp~ directories from failed rsync's taking up space [20:05:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:09:06] annnd again [20:09:14] Project beta-scap-eqiad build #288183: 04STILL FAILING in 4 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288183/ [20:09:34] hauskatze: Ooooh. `license-name` not `license`? Hah. :-) [20:10:35] James_F: yeah, silly me [20:10:50] Well, I didn't spot it either. [20:10:54] Okay I changed the config section and now jenkins is happy James_F [20:11:03] I guess review/testing is now needed [20:11:16] Yeah. [20:11:25] I have some doubts [20:11:48] for example, I didn't included the `public static function log()` in extension.json anywhere [20:12:02] but log is self::log ut supra [20:12:14] so I guess it's still calling the functions? [20:12:17] Also, eurgh, very old code. [20:13:15] Yeah [20:13:40] If the extension is broken maybe archiving it would be best. IIRC WikiApiary didn't show much customers [20:14:45] Project beta-scap-eqiad build #288184: 15ABORTED in 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288184/ [20:14:58] ^ me [20:18:33] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-Incident: Document emergency deploy procedures - https://phabricator.wikimedia.org/T245046 (10Jdforrester-WMF) p:05Triage→03Medium [20:20:50] RECOVERY - Free space - all mounts on deployment-mwmaint01 is OK: OK: All targets OK [20:25:01] Yippee, build fixed! [20:25:02] Project beta-scap-eqiad build #288185: 09FIXED in 4 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/288185/ [20:25:50] :D [20:25:51] diner [20:25:54] see you [20:33:10] !log Updating dev-images docker-pkg files on contint1001 to attempt a rebuild for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/571828 [20:33:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:43:40] 10Phabricator, 10Security-Team, 10Security, 10Security Related: Update protect as security issue to use acl*security - https://phabricator.wikimedia.org/T245201 (10chasemp) >>! In T245201#5885238, @mmodell wrote: > @chasemp: The best way I can think of to make the transition easy (and ease future maintaina... [21:02:52] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10LGoto) [21:10:05] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10Peachey88) You can't change which column is default, You need to make a new one, move contents out then rename (at least last time I researched) [21:10:52] 10MediaWiki-Codesniffer: Release mediawiki-codesniffer 30.0 - https://phabricator.wikimedia.org/T245309 (10Reedy) [21:10:57] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10RhinosF1) The default column can not be changed as far as I can tell. You’d need to rename ‘backlog’ to ‘Needs Triage’ and move tasks. [21:14:23] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10Reedy) >However, this problem here is the large number of tasks which would create a ton of notifications. Generally, don't worry about that. If people don't care about column... [21:21:48] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10greg) Or we can do the mass move for you silently. When doing a bulk job there's a interstitial that says something like "to run this job without notification, run this command... [21:30:19] 10MediaWiki-Codesniffer: Release mediawiki-codesniffer 30.0 - https://phabricator.wikimedia.org/T245309 (10Reedy) Are we waiting on any other tickets/open patchsets? [21:31:48] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10LGoto) @reedy thanks, good to know! The column move would probably be fine, but maybe it's safer to do the silent move. @greg Sorry if this is an obvious question, but how do... [21:33:17] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10Peachey88) You need to be in the triage group and use bulk edit, but I just checked and there isn't any workboard move options. [21:35:38] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10Reedy) >>! In T245308#5886156, @LGoto wrote: > @greg Sorry if this is an obvious question, but how do I figure out the job number? It doesn't look like this is possible for a... [21:39:34] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10greg) >>! In T245308#5886156, @LGoto wrote: > @greg Sorry if this is an obvious question, but how do I figure out the job number? You find the job number when doing a bulk edi... [21:41:28] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10Reedy) >>! In T245308#5886169, @greg wrote: > I'm sorry I promised something we didn't have; this use case always bites me. Googling looks like this solution was the implement... [21:48:12] 10MediaWiki-Codesniffer: Release mediawiki-codesniffer 30.0 - https://phabricator.wikimedia.org/T245309 (10Daimona) >>! In T245309#5886152, @Reedy wrote: > Are we waiting on any other tickets/open patchsets? AFAIAA, no. Also, I think it makes sense to tag 30.0.0 now, given the number of changes in 29..master. [21:51:02] 10MediaWiki-Codesniffer: Release mediawiki-codesniffer 30.0 - https://phabricator.wikimedia.org/T245309 (10Jdforrester-WMF) Possibly https://gerrit.wikimedia.org/r/c/mediawiki/tools/codesniffer/+/571023 ? [21:54:22] PROBLEM - Host deployment-puppetmaster03 is DOWN: CRITICAL - Host Unreachable (172.16.4.91) [21:54:34] PROBLEM - Host deployment-puppetdb02 is DOWN: CRITICAL - Host Unreachable (172.16.4.104) [21:55:11] RECOVERY - Host deployment-puppetmaster03 is UP: PING OK - Packet loss = 0%, RTA = 0.98 ms [21:55:25] RECOVERY - Host deployment-puppetdb02 is UP: PING OK - Packet loss = 0%, RTA = 0.57 ms [22:25:27] 10Phabricator, 10Release-Engineering-Team: change "default" column on Parsoid workboard - https://phabricator.wikimedia.org/T245308 (10LGoto) No worries, thanks everyone for looking into this. I'll give the team a heads up and then move the tasks via the workboard. [22:40:15] PROBLEM - Host deployment-puppetmaster04 is DOWN: CRITICAL - Host Unreachable (172.16.1.3) [22:43:00] RECOVERY - Host deployment-puppetmaster04 is UP: PING OK - Packet loss = 0%, RTA = 2.22 ms [22:45:59] (03CR) 10Brennen Bearnes: "See https://phabricator.wikimedia.org/T245304" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/571828 (owner: 10Kosta Harlan) [22:57:52] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Core Platform Team: Audit all existing code to ensure that any extension currently or previously adding blobs to ExternalStore has been registering a reference in the text table (and fix up if wrong) - https://phabricator.wikimedia.org/T106388 (10... [23:07:00] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [23:15:15] PROBLEM - Host deployment-puppetmaster04 is DOWN: CRITICAL - Host Unreachable (172.16.1.3) [23:15:41] (03CR) 10Jeena Huneidi: "> Patch Set 1: Verified+2 Code-Review+2" [releng/local-charts] - 10https://gerrit.wikimedia.org/r/571621 (owner: 10Jeena Huneidi) [23:16:59] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [23:17:59] RECOVERY - Host deployment-puppetmaster04 is UP: PING OK - Packet loss = 0%, RTA = 2.18 ms [23:22:49] (03PS2) 10Jeena Huneidi: Disable MariaDB replication [releng/local-charts] - 10https://gerrit.wikimedia.org/r/571621 [23:30:17] PROBLEM - Host deployment-puppetmaster04 is DOWN: CRITICAL - Host Unreachable (172.16.1.3) [23:33:01] RECOVERY - Host deployment-puppetmaster04 is UP: PING OK - Packet loss = 0%, RTA = 1.73 ms