[00:30:40] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.38.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T300198 (10Krinkle)
[02:20:14] <wikibugs>	 10Phabricator: Ask before leaving a Create Task form - https://phabricator.wikimedia.org/T301751 (10diegodlh)
[02:53:28] <wikibugs>	 (03PS1) 10Jeena Huneidi: Upgrade to helm 3 [tools/train-dev] - 10https://gerrit.wikimedia.org/r/762572 (https://phabricator.wikimedia.org/T301266)
[06:31:43] <wikibugs>	 10Phabricator, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui)
[06:34:36] <wikibugs>	 10Phabricator, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui)
[06:35:22] <wikibugs>	 10Phabricator, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui)
[06:35:57] <wikibugs>	 10Phabricator, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui) To clean up orchestrator: ` delete from heartbeat where server_id=171966678 `
[06:36:14] <wikibugs>	 10Phabricator, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui)
[06:37:50] <wikibugs>	 10Phabricator, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui)
[07:01:27] <wikibugs>	 10Phabricator, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui)
[07:07:16] <wikibugs>	 10Phabricator, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui)
[07:07:27] <wikibugs>	 10Phabricator, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui) All pre-failover steps are done
[07:47:39] <wikibugs>	 10Phabricator, 10DBA, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui)
[08:01:19] <wikibugs>	 10Phabricator, 10DBA, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui) test after switch
[08:01:48] <wikibugs>	 10Phabricator, 10DBA, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui) p:05Triage→03Medium
[08:01:54] <wikibugs>	 10Phabricator, 10DBA, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui) p:05Medium→03Triage testing priority change
[08:03:20] <wikibugs>	 10Phabricator, 10DBA, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui)
[08:04:08] <wikibugs>	 10Phabricator, 10DBA, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui)
[08:04:38] <wikibugs>	 10Phabricator, 10DBA, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui) 05Open→03Resolved All done - it took around 50 seconds read-only time.
[09:23:09] <wikibugs>	 (03PS1) 10Lucas Werkmeister (WMDE): Remove myself from UTC late backport window [tools/release] - 10https://gerrit.wikimedia.org/r/762771
[09:26:17] <wikibugs>	 (03PS2) 10Lucas Werkmeister (WMDE): Remove myself from UTC late backport window [tools/release] - 10https://gerrit.wikimedia.org/r/762771
[09:32:10] <wikibugs>	 10Phabricator (Upstream): Ask before leaving a Create Task form - https://phabricator.wikimedia.org/T301751 (10Aklapper)
[09:35:02] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "Hi Lucas can you repurpose this change to solely remove yourself from the late window and I will happily +2 it." [tools/release] - 10https://gerrit.wikimedia.org/r/762771 (owner: 10Lucas Werkmeister (WMDE))
[09:35:44] <hashar>	 Lucas_WMDE: hi for the deployment calendar we intentionally got rid of the terms morning / afternoon / evening : https://phabricator.wikimedia.org/T290859  :]
[09:36:00] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): Remove myself from UTC late backport window (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/762771 (owner: 10Lucas Werkmeister (WMDE))
[09:36:03] <Lucas_WMDE>	 replied
[09:39:54] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): Remove myself from UTC late backport window (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/762771 (owner: 10Lucas Werkmeister (WMDE))
[09:47:06] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): Remove myself from UTC late backport window (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/762771 (owner: 10Lucas Werkmeister (WMDE))
[10:40:43] <wikibugs>	 (03PS3) 10Jaime Nuche: help output: align command descriptions [tools/scap] - 10https://gerrit.wikimedia.org/r/761980
[10:41:17] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] help output: align command descriptions [tools/scap] - 10https://gerrit.wikimedia.org/r/761980 (owner: 10Jaime Nuche)
[11:02:28] <hashar>	 Lucas_WMDE: looks like there are a few mistakes indeed :]  I will revisit it this afternoon
[11:02:42] <Lucas_WMDE>	 ok :)
[12:32:55] <wikibugs>	 (03PS4) 10Jaime Nuche: help output: align command descriptions [tools/scap] - 10https://gerrit.wikimedia.org/r/761980
[12:38:53] <wikibugs>	 (03CR) 10Jaime Nuche: help output: align command descriptions (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/761980 (owner: 10Jaime Nuche)
[13:39:16] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] Remove myself from UTC late backport window (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/762771 (owner: 10Lucas Werkmeister (WMDE))
[13:40:03] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Remove myself from UTC late backport window [tools/release] - 10https://gerrit.wikimedia.org/r/762771 (owner: 10Lucas Werkmeister (WMDE))
[13:40:12] <hashar>	 Lucas_WMDE: +2ed !
[13:40:15] <Lucas_WMDE>	 \o/
[13:40:28] <hashar>	 I have done a sniper review this morning
[13:40:40] <wikibugs>	 (03Merged) 10jenkins-bot: Remove myself from UTC late backport window [tools/release] - 10https://gerrit.wikimedia.org/r/762771 (owner: 10Lucas Werkmeister (WMDE))
[13:40:42] <Lucas_WMDE>	 maybe I could’ve explained it better in the commit message, it’s confusing that there was one / two windows
[13:40:44] <hashar>	 and I guess your patch triggered a bit of confusion
[13:40:48] <hashar>	 yeah
[13:40:57] <hashar>	 that is why I usually like small tiny commits :]
[13:41:04] <Lucas_WMDE>	 hehe
[13:41:13] <Lucas_WMDE>	 I could’ve first split it and then removed myself, but not the other way aronud ^^
[13:41:15] <hashar>	 like one to rename, another one to add the late window and a 3rd one to drop you from the late window
[13:41:19] <hashar>	 which is a lot of paper work ;]
[13:41:50] <hashar>	 of course having two windows at different time having the same name caused a bit of confusion but that is resolved now!
[13:42:17] <hashar>	 as for applying those changes, well hmm that will be for next week when Tyler run the script 
[13:42:40] <Lucas_WMDE>	 ok
[13:42:59] <hashar>	 you can always manually edit the [[Deployments]] page :]
[13:43:12] <Lucas_WMDE>	 meh, I’ll just let jouncebot ping me when I’m not there :p
[13:43:28] <hashar>	 indeed :D
[13:43:47] <hashar>	 also this week I am running the train / promote at 10am CET  instead of 3pm
[13:44:01] <hashar>	 so wmf.22 will land on commons / wikidata around 10am
[13:44:09] <Lucas_WMDE>	 good to know
[14:19:25] <wikibugs>	 (03PS5) 10Jaime Nuche: help output: align command descriptions [tools/scap] - 10https://gerrit.wikimedia.org/r/761980 (https://phabricator.wikimedia.org/T243659)
[14:20:36] <wmf-insecte>	 maintenance-disconnect-full-disks build 360615 integration-agent-qemu-1003 (/: 96%, /srv: 25%, /var/lib/docker: 1%): OFFLINE due to disk space
[14:25:32] <wmf-insecte>	 maintenance-disconnect-full-disks build 360616 integration-agent-qemu-1003 (/: 88%, /srv: 25%, /var/lib/docker: 1%): RECOVERY disk space OK
[14:39:52] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto)
[14:46:53] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) >>! In T297411#7669397, @Dzahn wrote: > []  installed gitlab-ce package post-installation script subprocess returned error exit status 1 > [] nginx initial...
[14:49:01] <hashar>	 will check qemu-1003
[14:57:05] <hashar>	  /home/dancy has a bunch of virtual machine images
[14:57:09] <hashar>	 so not a concern really :)
[15:09:09] <hashar>	 I have dropped images from my $HOME so that is fixed
[15:17:33] <wikibugs>	 (03PS1) 10Nikki Nikkhoui: jjb, Zuul: [mediawiki/services/service-scaffold-node] add test pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/762833
[15:23:51] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb, Zuul: [mediawiki/services/service-scaffold-node] add test pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/762833 (owner: 10Nikki Nikkhoui)
[15:23:57] <hashar>	 nikkinikk: deplyoing ^ :)
[15:25:33] <wikibugs>	 (03Merged) 10jenkins-bot: jjb, Zuul: [mediawiki/services/service-scaffold-node] add test pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/762833 (owner: 10Nikki Nikkhoui)
[15:26:22] <hashar>	 !log Reloading Zuul for If80b4b4cfa5c1a869ceb220f5b11c272b384a721
[15:26:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:35:38] <hashar>	 nikkinikk: looks like the repo requires a `.pipeline/config.yaml` now ;]
[15:37:42] <dancy>	 hashar: I just deleted my big files.
[15:37:50] <hashar>	 success
[15:38:11] <dancy>	 Sorry about that
[15:38:16] <hashar>	 not a big deal :)
[15:38:25] <hashar>	 I had a few big images myself which I have deleted
[15:38:28] <hashar>	 dancy: I haven't followed on your review from yesterday regarding using virt-builder and the upstream debian image they provide
[15:38:47] <hashar>	 but it is definitely on my list of next action. I got distracted by train, meeting and other errands ;D
[15:56:26] <wikibugs>	 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10valerio.bozzolan)
[15:57:59] <wikibugs>	 10GitLab (Infrastructure), 10GitLab-Test, 10Cloud-VPS: Add wmcloud.org DNS zone to WMCS project devtools - https://phabricator.wikimedia.org/T301793 (10Jelto)
[16:03:15] <wikibugs>	 10GitLab (Infrastructure), 10GitLab-Test, 10Cloud-VPS: Add wmcloud.org DNS zone to WMCS project devtools - https://phabricator.wikimedia.org/T301793 (10Majavah) a:03Majavah
[16:04:29] <Krinkle>	 hashar: it seems qemu jobs are failing, stuck waiting for VM and time out after 30min
[16:04:34] <Krinkle>	 seen it 3x so far
[16:04:45] <hashar>	 Krinkle: checking
[16:04:48] <Krinkle>	 https://integration.wikimedia.org/ci/job/fresh-test/232/console
[16:04:52] <Krinkle>	 (on-going)
[16:05:12] <hashar>	 hmm I guess I forgot to revert the changes I have made yesterdat afternoon
[16:06:05] <hashar>	 !log Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
[16:06:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:06:12] <hashar>	 Krinkle: done, gotta cancel the job and recheck
[16:06:31] <hashar>	 I tested changes live yesterday by pooling qemu-1003 and unpooling qemu-1001
[16:06:53] <hashar>	 then I have rolled that back but forgot to restore the job to the previous state :(
[16:07:08] <hashar>	 should be good now
[16:08:58] <wikibugs>	 10GitLab (Infrastructure), 10GitLab-Test, 10Cloud-VPS: Add wmcloud.org DNS zone to WMCS project devtools - https://phabricator.wikimedia.org/T301793 (10Majavah) 05Open→03Resolved Done! You should be able to manage the new zone via Horizon just like the old one, please let us know if you have any issues w...
[16:18:08] <Krinkle>	 hashar: thx
[16:20:08] <wikibugs>	 10GitLab (Infrastructure), 10GitLab-Test, 10Cloud-VPS: Add wmcloud.org DNS zone to WMCS project devtools - https://phabricator.wikimedia.org/T301793 (10Jelto) thanks for the quick help 🚀  I created a new record in the zone successfully.
[16:20:42] <wikibugs>	 10GitLab (Infrastructure), 10GitLab-Test, 10Cloud-VPS: Add wmcloud.org DNS zone to WMCS project devtools - https://phabricator.wikimedia.org/T301793 (10Jelto)
[16:20:46] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto)
[16:21:09] <Krinkle>	 hashar: hm.. it seems I now get the opposite error
[16:21:11] <Krinkle>	 > 16:19:57 cp: cannot stat '/srv/vm-images/qemu-debian10buster-2020_05_04c.img': No such file or directory
[16:21:19] <Krinkle>	 > Building remotely on integration-agent-qemu-1003 
[16:21:56] <hashar>	 !log Upgrading Jenkins plugins on releases Jenkins
[16:21:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:22:23] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto)
[16:22:39] <hashar>	 Krinkle: ARGhh I swear I thought qemu-1003 was offline! I have put it offline now
[16:22:41] <hashar>	 :-\
[16:24:02] * Krinkle rolls the dice again
[16:24:52] <hashar>	 !log Restarting CI Jenkins for plugins updates
[16:24:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:25:08] <hashar>	 Krinkle: I am hoping to complete the migration by end of this week
[16:25:14] <hashar>	 there are a few details that need to be polished
[16:25:36] <hashar>	 notably docker pull taking ages cause uncompressing some 250+MB layer ends up taking a while
[16:55:56] <Amir1>	 btw I have caused a couple of fatals in mwdebug1001, don't worry when you see them
[17:06:22] <hashar>	 Amir1: thanks :)
[17:06:36] <hashar>	 I am off for the rest of the evening. See you all tomorrow for the rest of the train
[17:08:29] <dancy>	 Layer Antoine
[17:08:36] <dancy>	 *Later
[17:30:00] <wmf-insecte>	 Yippee, build fixed!
[17:30:00] <wmf-insecte>	 Project beta-update-databases-eqiad build #56677: 09FIXED in 9 min 59 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/56677/
[17:51:02] <Krinkle>	 James_F: I wonder if CongressLookup makes sense to keep prod branched and deployed... T205049
[17:51:03] <stashbot>	 T205049: Disable extension-CongressLookup - https://phabricator.wikimedia.org/T205049
[17:51:11] <Krinkle>	 Noticed it stand out in mw:Maintainers
[17:54:39] <wikibugs>	 (03CR) 10Jeena Huneidi: [C: 03+1] python: ban setuptools==60.9.0 from installing [blubber] - 10https://gerrit.wikimedia.org/r/762552 (https://phabricator.wikimedia.org/T301690) (owner: 10BryanDavis)
[17:58:10] <wikibugs>	 (03CR) 10Ahmon Dancy: help output: align command descriptions (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/761980 (https://phabricator.wikimedia.org/T243659) (owner: 10Jaime Nuche)
[17:59:11] <apergos>	 Krinkle: the EARNIT act is coming up, so maybe... >_<
[18:07:03] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 04-1] "A couple of tweaks requested.  Otherwise it seems to work fine." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/762572 (https://phabricator.wikimedia.org/T301266) (owner: 10Jeena Huneidi)
[18:15:29] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 03+2] Don't update wmf-config/ExtensionMessages* if unchanged [tools/scap] - 10https://gerrit.wikimedia.org/r/762506 (owner: 10Ahmon Dancy)
[18:16:00] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 03+2] Optionally build mw container image during scap sync-* [tools/scap] - 10https://gerrit.wikimedia.org/r/762546 (https://phabricator.wikimedia.org/T297673) (owner: 10Ahmon Dancy)
[18:23:11] <wikibugs>	 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T300198 (10Jdlrobson) ^ Train blocker: T301810. Please don't run the train further until this has been resolved.
[18:25:27] <wikibugs>	 (03Merged) 10jenkins-bot: Don't update wmf-config/ExtensionMessages* if unchanged [tools/scap] - 10https://gerrit.wikimedia.org/r/762506 (owner: 10Ahmon Dancy)
[18:25:29] <wikibugs>	 (03Merged) 10jenkins-bot: Optionally build mw container image during scap sync-* [tools/scap] - 10https://gerrit.wikimedia.org/r/762546 (https://phabricator.wikimedia.org/T297673) (owner: 10Ahmon Dancy)
[18:31:02] <wikibugs>	 (03PS2) 10Jeena Huneidi: Upgrade to helm 3 [tools/train-dev] - 10https://gerrit.wikimedia.org/r/762572 (https://phabricator.wikimedia.org/T301266)
[18:31:21] <wikibugs>	 (03CR) 10Jeena Huneidi: Upgrade to helm 3 (032 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/762572 (https://phabricator.wikimedia.org/T301266) (owner: 10Jeena Huneidi)
[18:40:28] <wikibugs>	 10Release-Engineering-Team (Seen), 10serviceops: contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10Papaul)
[18:40:31] <wikibugs>	 10Gerrit, 10SRE, 10serviceops: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Papaul)
[19:05:20] <wikibugs>	 10Beta-Cluster-Infrastructure: Upgrade deployment-prep Kafka clusters to Debian Buster or later - https://phabricator.wikimedia.org/T301819 (10Majavah)
[19:06:21] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Majavah)
[19:35:12] <wikibugs>	 (03PS3) 10Jeena Huneidi: Upgrade to helm 3 [tools/train-dev] - 10https://gerrit.wikimedia.org/r/762572 (https://phabricator.wikimedia.org/T301266)
[19:48:16] <James_F>	 Krinkle: Yeah, could do. Given there's active efforts by our Public Policy colleagues in this area again, however, maybe it might still be needed?
[19:57:28] <wikibugs>	 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T300198 (10Jdforrester-WMF)
[20:06:44] <wikibugs>	 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T300198 (10Jdlrobson)
[20:47:47] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 03+2] "Tested in train-dev." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/762572 (https://phabricator.wikimedia.org/T301266) (owner: 10Jeena Huneidi)
[20:48:12] <wikibugs>	 (03Merged) 10jenkins-bot: Upgrade to helm 3 [tools/train-dev] - 10https://gerrit.wikimedia.org/r/762572 (https://phabricator.wikimedia.org/T301266) (owner: 10Jeena Huneidi)
[20:59:15] <dancy>	 !log Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
[20:59:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:01:27] <mutante>	 dancy: I think the entire beta is down right now
[21:01:50] <dancy>	 I see a bunch of spinning php-fpm7.2 processes on deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud
[21:01:54] <dancy>	 I'm going to restart them
[21:02:05] <mutante>	 cool, thanks
[21:03:00] <dancy>	 hmm.
[21:05:40] <mutante>	 dancy: does it have /usr/local/sbin/check-and-restart-php ?
[21:06:27] <dancy>	 It doesn't, but I can sudo to root and use systemctl restart there, which I did, and they did restart.. but it didn't help.
[21:06:37] <dancy>	 Trying to collect some error message 
[21:06:37] <mutante>	 hmm. then I would reboot the entire VM
[21:06:45] <dancy>	 not ready for that yet
[21:06:49] <mutante>	 ok
[21:11:50] <dancy>	 lots of php-fpm7.2 segfaults logged in dmegs.
[21:11:52] <dancy>	 *dmesg.
[21:11:55] <dancy>	 I'm going to reboot.
[21:12:10] <mutante>	 ACK
[21:12:20] <dancy>	 !log rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
[21:12:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:14:11] <mutante>	 dancy: something works... slow.. but more than before
[21:14:28] <mutante>	 the original user report was also "just slow yesterday but NOW down"
[21:15:05] <dancy>	 yeah.. still not good
[21:17:44] <mutante>	 we do have  /usr/local/sbin/restart-php7.4-fpm    I could try that
[21:18:01] <mutante>	 nevermind, that is literally just a wrapper for systemctl
[21:18:14] <dancy>	 It's the php-fpm7.2 processes that are spinning
[21:18:50] <mutante>	 both 7.4 and 7.2 are installed
[21:19:23] <dancy>	 Nod.  The 7.4 processes appear to be idle.
[21:19:27] * mutante checks APT history log
[21:19:38] <mutante>	 because those hosts do unattended upgrades.. I think
[21:20:21] <mutante>	 sees python3-wmflib and scap .. samba (lol, samba)
[21:20:40] <dancy>	 haha
[21:20:45] <mutante>	 smbclient..
[21:21:13] <mutante>	 yes, unattended upgrades but nothing PHP related
[21:21:36] <dancy>	 I think a bot is hammering it.
[21:21:43] <dancy>	 something related to dewiki.
[21:22:17] <mutante>	 Seekport Crawler; http://seekport.com/
[21:22:22] <mutante>	 this guy?
[21:22:24] <dancy>	 yeah
[21:22:29] <mutante>	 ACK, I see that too
[21:23:16] <mutante>	 crawl1-214.oi.tb.007ac9.net
[21:24:06] <wikibugs>	 (03CR) 10Jeena Huneidi: [C: 03+2] python: ban setuptools==60.9.0 from installing [blubber] - 10https://gerrit.wikimedia.org/r/762552 (https://phabricator.wikimedia.org/T301690) (owner: 10BryanDavis)
[21:26:44] <wikibugs>	 (03Merged) 10jenkins-bot: python: ban setuptools==60.9.0 from installing [blubber] - 10https://gerrit.wikimedia.org/r/762552 (https://phabricator.wikimedia.org/T301690) (owner: 10BryanDavis)
[21:27:51] <bd808>	 jeena: thank you :)
[21:28:28] <jeena>	 :)
[21:31:01] <wikibugs>	 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security: Create initial proof of concept application security pipeline repository - https://phabricator.wikimedia.org/T289293 (10sbassett)
[21:31:32] <wikibugs>	 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security: Create initial proof of concept application security pipeline repository - https://phabricator.wikimedia.org/T289293 (10sbassett)
[21:33:05] <wmf-insecte>	 Project beta-mediawiki-config-update-eqiad build #163: 04FAILURE in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/163/
[21:35:02] <wmf-insecte>	 Project beta-code-update-eqiad build #380268: 04FAILURE in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/380268/
[21:41:45] <wmf-insecte>	 Yippee, build fixed!
[21:41:45] <wmf-insecte>	 Project beta-mediawiki-config-update-eqiad build #164: 09FIXED in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/164/
[21:44:36] <wmf-insecte>	 Yippee, build fixed!
[21:44:36] <wmf-insecte>	 Project beta-code-update-eqiad build #380269: 09FIXED in 1 min 35 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/380269/
[22:04:54] <wikibugs>	 10Release-Engineering-Team, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, 10Security: Create nancy ci yaml template - https://phabricator.wikimedia.org/T301831 (10sbassett) Hey #release-engineering-team - so we hit a snag here trying to use the available golang images under docker-registr...
[22:25:21] <wikibugs>	 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security: Create initial proof of concept application security pipeline repository - https://phabricator.wikimedia.org/T289293 (10sbassett)
[22:42:50] <wikibugs>	 10Continuous-Integration-Config, 10Codex, 10Design-Systems-team (Design Systems Team FY2021-22 Kanban Board), 10Patch-For-Review: Deploy current main branch's docs site to doc.wikimedia.org - https://phabricator.wikimedia.org/T293704 (10egardner) 05Open→03Resolved
[22:53:52] <wikibugs>	 10Continuous-Integration-Config, 10Codex, 10Design-Systems-team (Design Systems Team FY2021-22 Kanban Board), 10Patch-For-Review: Deploy current main branch's docs site to doc.wikimedia.org - https://phabricator.wikimedia.org/T293704 (10Dzahn) I would just like to add that nowdays the options are not limit...