[00:43:17] <wikibugs>	 (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/Wikibase] Merge experimental sections [integration/config] - 10https://gerrit.wikimedia.org/r/682782
[00:45:22] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/Wikibase] Merge experimental sections [integration/config] - 10https://gerrit.wikimedia.org/r/682782 (owner: 10Jforrester)
[00:46:48] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Wikibase] Merge experimental sections [integration/config] - 10https://gerrit.wikimedia.org/r/682782 (owner: 10Jforrester)
[00:53:38] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10Beta-Cluster-reproducible: Unable to edit on beta cluster, unable to log in or out - https://phabricator.wikimedia.org/T281198 (10Etonkovidova)
[04:20:52] <wikibugs>	 10Phabricator, 10DBA: Upgrade mysql on db1132 (phabricator db master) - https://phabricator.wikimedia.org/T279625 (10Marostegui) test
[04:22:48] <wikibugs>	 10Phabricator, 10DBA: Upgrade mysql on db1132 (phabricator db master) - https://phabricator.wikimedia.org/T279625 (10Marostegui) 05Open→03Resolved a:03Marostegui This was done RO starts: 04:19:51 RO stops: 04:20:18
[05:29:03] <Majavah>	 !log restart cassandra on deployment-sessionstore03 refs T281198
[05:29:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[05:29:07] <stashbot>	 T281198: Unable to edit on beta cluster, unable to log in or out - https://phabricator.wikimedia.org/T281198
[05:31:19] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10Beta-Cluster-reproducible: Unable to edit on beta cluster, unable to log in or out - https://phabricator.wikimedia.org/T281198 (10Majavah) This is likely {T263617} again, but Cassandra (the database storing all the sessions) is cu...
[05:38:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10Beta-Cluster-reproducible: Unable to edit on beta cluster, unable to log in or out - https://phabricator.wikimedia.org/T281198 (10Majavah)
[05:39:18] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10User-Majavah, 10User-zeljkofilipin: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10Majavah)
[05:43:06] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10User-Majavah, 10User-zeljkofilipin: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10Majavah) So this is happening again, and looks like Cassandra is running ou...
[05:49:50] <Majavah>	 hey all, would like a second opinion on https://phabricator.wikimedia.org/T263617#7036797, should I try to play with Cassandra settings on that box or just make a new VM with more RAM allocated and hope that fixes the issue?
[06:34:08] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10User-Majavah, 10User-zeljkofilipin: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10Majavah) Given that's a Stretch server that will need replacing soon anyway...
[06:40:34] <Majavah>	 !log installing deployment-sessionstore04 T263617
[06:40:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[06:40:37] <stashbot>	 T263617: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617
[07:14:23] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C: 04-1] "There are 140 of these in the core test run alone. I honestly don't think this is helpful. I can see two ways forward:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/682713 (https://phabricator.wikimedia.org/T281190) (owner: 10DannyS712)
[07:18:00] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Thanks!" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/682272 (https://phabricator.wikimedia.org/T274016) (owner: 10Umherirrender)
[08:33:53] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T278347 (10LarsWirzenius) Filed {T281223} but not as a blocker.
[08:34:19] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10Patch-For-Review, 10User-Majavah, 10User-zeljkofilipin: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10dom_walden) @Majavah This seems to be happening again...
[08:35:18] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10Patch-For-Review, 10User-Majavah, 10User-zeljkofilipin: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10Majavah) >>! In T263617#7037136, @dom_walden wrote: >...
[08:35:46] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T278347 (10LarsWirzenius) Filed {T281224} but not as a blocker.
[08:45:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10Patch-For-Review, 10User-Majavah, 10User-zeljkofilipin: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617 (10dom_walden) >>! In T263617#7037139, @Majavah wrote: >...
[08:58:47] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T278347 (10LarsWirzenius) Filed {T281226}, making it a blocker.
[08:58:51] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T278347 (10LarsWirzenius)
[09:17:58] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Readers-Web-Backlog: MinervaNeue legacy ruby browser tests need to be updated to work on buster/ruby2.5 - https://phabricator.wikimedia.org/T280901 (10ovasileva)
[09:18:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Readers-Web-Backlog: MinervaNeue legacy ruby browser tests need to be updated to work on buster/ruby2.5 - https://phabricator.wikimedia.org/T280901 (10ovasileva) p:05Triage→03Medium
[09:27:48] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T278347 (10LarsWirzenius) Filed {T281229} but not yet as a blocker
[09:28:41] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T278347 (10Urbanecm_WMF) >>! In T278347#7037339, @LarsWirzenius wrote: > Filed {T281229} but not yet as a blocker  You're so quick, was just going to fill this :).
[09:37:51] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[09:39:55] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[09:44:26] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T278347 (10Urbanecm_WMF) >>! In T278347#7037339, @LarsWirzenius wrote: > Filed {T281229} but not yet as a blocker  Note: I deleted `MediaWiki:GrowthExperimentsConfig.json`...
[10:17:09] <wikibugs>	 (03PS4) 10Thiemo Kreuz (WMDE): Minor clean up to LowerCamelFunctionsNameSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/681374 (owner: 10DannyS712)
[10:20:25] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "I rearranged the code a tiny little bit to have the most expensive piece of the condition executed as late as possible. This way we are ne" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/681374 (owner: 10DannyS712)
[10:21:05] <wikibugs>	 (03PS5) 10Thiemo Kreuz (WMDE): Minor clean up to LowerCamelFunctionsNameSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/681374 (owner: 10DannyS712)
[10:21:10] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Minor clean up to LowerCamelFunctionsNameSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/681374 (owner: 10DannyS712)
[11:19:12] <wikibugs>	 10Beta-Cluster-Infrastructure: Global developer for Zabe on beta - https://phabricator.wikimedia.org/T281239 (10Zabe)
[11:28:57] <wikibugs>	 10Beta-Cluster-Infrastructure: Global developer for Zabe on beta - https://phabricator.wikimedia.org/T281239 (10Majavah) I don't see any issues, but given the sensitive permissions assigned to that group I'd like a second opinion before granting.
[12:07:00] <wikibugs>	 (03CR) 10Daimona Eaytoy: "> * I believe we need to either fix most code first (at least the most critical codebases) before making this a strict rule." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/682713 (https://phabricator.wikimedia.org/T281190) (owner: 10DannyS712)
[12:11:06] <hashar>	 !log Upgrading Jenkins plugins on the releases jenkins
[12:11:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:13:51] <wikibugs>	 10Beta-Cluster-Infrastructure: Global developer for Zabe on beta - https://phabricator.wikimedia.org/T281239 (10Leaderboard) I don't see a problem personally; this user is clearly trusted.
[12:21:50] <Leaderboard>	 Hi, I've created an account on the Beta-Cluster but the main page is gibberish but I can't change. 
[12:22:06] <Leaderboard>	 This is for Wikibooks by the way: https://en.wikibooks.beta.wmflabs.org/wiki/Main_Page
[12:22:25] <Leaderboard>	 If I should be asking elsewhere, do let me know; the Beta Cluster sent me here
[12:23:43] <hashar>	 Leaderboard: it got protected and can only be edited by administrators https://en.wikibooks.beta.wmflabs.org/w/index.php?title=Main_Page&action=history
[12:24:22] <Leaderboard>	 hashar: any case I can get sysop/crat access on the beta cluster? I'm a sysop on the production version of the wiki, if it matters
[12:24:27] <hashar>	 !log Upgraded releases Jenkins from 2.263.3 to 2.277.2  (with ldap plugin 1.26)
[12:24:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:25:06] <Majavah>	 Leaderboard: sure, done
[12:28:02] <Leaderboard>	 Majavah: can you give it to me on meta as well? I can't actually edit Wikibooks' main page because it appears to be caught by a Meta global abuse filter
[12:28:04] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Doing): Upgrade Jenkins to 2.277.x - https://phabricator.wikimedia.org/T279033 (10hashar) The note about the LDAP plugin is that we must have 1.26 installed which is the latest still compatible with the 2.263.3 Jenkins we are run...
[12:29:03] <Leaderboard>	 And weirdly, it says "Actions taken: none" but the error on Wikibooks tells otherwise. Apparently local abuse filters don't exist, and https://en.wikibooks.beta.wmflabs.org/wiki/Special:AbuseLog is blocked
[12:29:10] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Doing): Upgrade Jenkins to 2.277.x - https://phabricator.wikimedia.org/T279033 (10hashar) p:05Triage→03High a:03hashar
[12:30:05] <Majavah>	 Leaderboard: it's a local filter https://en.wikibooks.beta.wmflabs.org/wiki/Special:AbuseFilter/29 that disallows you from editing, and you can't see them since you're not (auto)confirmed
[12:30:38] <Leaderboard>	 Majavah: got it, I was confused. I think I need confirmed then, but can't do it myself since sysops can't add confirmed
[12:30:45] <Majavah>	 done
[12:40:03] <Leaderboard>	 Majavah: sorry to bother you again, but I'm trying to see a private abuse filter on Meta. Is it possible to give me either global AFH (which I have on production) or meta admin (which I don't have)? Sorry for the repetitive questions; just trying to do this I normally would do otherwise
[12:40:35] <Leaderboard>	 *do things
[12:40:52] <Majavah>	 happy to help, don't worry about flooding me with questions
[12:43:04] <Majavah>	 Leaderboard: {{done}}
[12:44:04] <hashar>	 !log Restarted CI Jenkins for plugins upgrade
[12:44:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:46:00] <Leaderboard>	 I ended up being blocked Majavah, can you unblock? 
[12:46:07] <Leaderboard>	 (sorry I wish I could do this myself!)
[12:46:27] <Leaderboard>	 And no, I didn't add a spam link
[12:47:04] <Majavah>	 I thought you could unblock yourself if you were blocked by the abuse filter
[12:47:17] <Majavah>	 ah, on meta
[12:48:46] <Majavah>	 sorry about that, they filters there are tuned quite aggressively due to the amount of spambots plus that we trust its users to be able to report and get unblocked/added to a group/etc 
[12:50:02] <Leaderboard>	 I would put that to disallow rather than block personally, unless it's really a problem
[13:01:38] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T278347 (10LarsWirzenius) There's a train blocker, but I'm promoting train to group0 to see how bad things get. I can roll back if need be.
[13:14:22] <Leaderboard>	 Majavah: could I have 'crat access on wikibooks? I have interfaceadmin on prod. If that's unreasonable that's fine
[13:23:50] <Majavah>	 Leaderboard: I trust you have an actual use case for that, done now
[13:25:43] <Leaderboard>	 Majavah: mainly to allow interfaceadmin
[13:25:49] <Leaderboard>	 as I have that on production
[13:29:31] <gehel>	 hashar: if you have some time to have a look at https://gerrit.wikimedia.org/r/c/integration/config/+/681988 (please and thank you)
[13:29:38] * gehel needs to run to another meeting
[13:37:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Doing): Upgrade Jenkins to 2.277.x - https://phabricator.wikimedia.org/T279033 (10hashar) The CI Jenkins upgrade apparently went fine \o/  @MoritzMuehlenhoff  I guess we can now import Jenkins 2.277.3 to `buster-wikimedia/thirdpa...
[13:51:06] <_joe_>	 hi, do we have any stats handy on the number of VMs we use for CI right now?
[13:55:57] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Doing): Upgrade Jenkins to 2.277.x - https://phabricator.wikimedia.org/T279033 (10MoritzMuehlenhoff) >>! In T279033#7038156, @hashar wrote: > The CI Jenkins upgrade apparently went fine \o/ >  > @MoritzMuehlenhoff  I guess we can...
[14:02:06] <wikibugs>	 (03Abandoned) 10Hashar: jjb: git topology debug for Maven Sonar analysis [integration/config] - 10https://gerrit.wikimedia.org/r/674277 (https://phabricator.wikimedia.org/T264873) (owner: 10Hashar)
[14:03:31] <Leaderboard>	 Majavah: I can't import from enwiki - do I have to file a phab task for that?
[14:18:25] <hashar>	 !log Updating most jenkins jobs to change cleanup commands from stretch to buster | https://gerrit.wikimedia.org/r/c/integration/config/+/680476
[14:18:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:19:06] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Nice. I have run a full update of jjb which should update all the affected jobs :]" [integration/config] - 10https://gerrit.wikimedia.org/r/680476 (owner: 10Jforrester)
[14:21:03] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Switch clean-up tasks from stretch to buster [integration/config] - 10https://gerrit.wikimedia.org/r/680476 (owner: 10Jforrester)
[14:56:51] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10SRE, 10Patch-For-Review, 10Technical-Debt: debian-glue jobs ignored error messages about libeatmydata.so in LD_PRELOAD - https://phabricator.wikimedia.org/T240430 (10hashar)
[15:07:14] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.36-notes, 10MW-1.36-release: Release 1.36.0-rc.0 - https://phabricator.wikimedia.org/T279462 (10Jdforrester-WMF)
[15:13:22] <James_F>	 hashar: Woah, a full jjb update?! :-)
[15:13:33] <addshore>	 :D
[15:24:05] <hashar>	 bah all nodes are down :/
[15:25:53] <hashar>	 addshore: I have absolutely NO IDEA  what those pages compaction are about :-\
[15:26:48] <hashar>	 so yeah loop in cloud folks
[15:27:05] <hashar>	 there is a system to compress memory, but that is the first time I hear about it being used on our system
[15:28:18] <hashar>	 or it is a linux feature I have never heard about (most probably
[15:33:21] <hashar>	 addshore: maybe that is a memory pressure issue?
[15:34:11] <hashar>	 good to see https://grafana-labs.wikimedia.org/d/000000590/instance-details  has been dramatically improved
[16:16:05] <wikibugs>	 (03PS1) 10Jforrester: dockerfiles: [composer-scratch] Upgrade to 1.10.22 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/682987
[16:20:54] <wikibugs>	 (03PS2) 10Jforrester: dockerfiles: [composer-scratch] Upgrade to 1.10.22 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/682987 (https://phabricator.wikimedia.org/T281283)
[16:21:10] <wikibugs>	 (03CR) 10Reedy: [C: 03+1] dockerfiles: [composer-scratch] Upgrade to 1.10.22 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/682987 (https://phabricator.wikimedia.org/T281283) (owner: 10Jforrester)
[16:27:43] <wikibugs>	 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 3.17.1-1 - https://phabricator.wikimedia.org/T279695 (10jijiki) @dancy thank you very much, everything worked well!  I have installed scap 3.17.1-1 on our api canaries, I will proceed with a full rollout this week.
[16:28:26] <wikibugs>	 (03PS3) 10Jforrester: dockerfiles: [composer-scratch] Upgrade to 1.10.22 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/682987 (https://phabricator.wikimedia.org/T281283)
[16:32:19] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] dockerfiles: [composer-scratch] Upgrade to 1.10.22 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/682987 (https://phabricator.wikimedia.org/T281283) (owner: 10Jforrester)
[16:34:10] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: [composer-scratch] Upgrade to 1.10.22 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/682987 (https://phabricator.wikimedia.org/T281283) (owner: 10Jforrester)
[16:35:18] <James_F>	 !log Docker: Publishing composer-scratch 1.10.22 and its cascade for T281283
[16:35:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:58:45] <wikibugs>	 (03PS1) 10Jforrester: jjb: Switch use of composer* images to ones with composer 1.10.22 [integration/config] - 10https://gerrit.wikimedia.org/r/682994 (https://phabricator.wikimedia.org/T281283)
[16:58:47] <wikibugs>	 (03PS1) 10Jforrester: jjb: Switch use of quibble* and phan* images to ones with composer 1.10.22 [integration/config] - 10https://gerrit.wikimedia.org/r/682995 (https://phabricator.wikimedia.org/T281283)
[17:18:19] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Disable Gerrit user Mdhollo - https://phabricator.wikimedia.org/T281291 (10Mholloway)
[17:24:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10SRE, 10Patch-For-Review, 10Technical-Debt: debian-glue jobs ignored error messages about libeatmydata.so in LD_PRELOAD - https://phabricator.wikimedia.org/T240430 (10hashar) 05Open→03Resolved I have confirmed that eatmydat...
[17:42:51] <James_F>	 It turns out that re-build almost all our CI docker images takes time. Who'd'a thunk it?
[17:47:04] <wikibugs>	 (03CR) 10Umherirrender: "If T254936 is fixed, it is up to the class to annotate if its support DI or not." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/682713 (https://phabricator.wikimedia.org/T281190) (owner: 10DannyS712)
[17:50:39] <wikibugs>	 10Continuous-Integration-Infrastructure: Drop CI for REL1_31 branch once it's EOL - https://phabricator.wikimedia.org/T281294 (10Jdforrester-WMF)
[17:50:53] <wikibugs>	 10Continuous-Integration-Infrastructure: Drop CI for REL1_31 branch once it's EOL - https://phabricator.wikimedia.org/T281294 (10Jdforrester-WMF)
[17:50:55] <wikibugs>	 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Formally EOL REL1_31 - https://phabricator.wikimedia.org/T279858 (10Jdforrester-WMF)
[17:51:29] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MW-1.31-release: Drop CI for REL1_31 branch once it's EOL - https://phabricator.wikimedia.org/T281294 (10Jdforrester-WMF)
[17:52:03] <Majavah>	 !log delete deployment-sessionstore03 T263617 T278641
[17:52:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:52:08] <stashbot>	 T263617: Cannot login to beta cluster: "There seems to be a problem with your login session..." - https://phabricator.wikimedia.org/T263617
[17:52:08] <stashbot>	 T278641: Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641
[17:52:34] <wikibugs>	 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Majavah)
[17:55:41] <wikibugs>	 10Phabricator, 10DBA, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10mmodell)
[17:56:24] <wikibugs>	 10Phabricator, 10DBA, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10mmodell) I've marked this as blocked by {T171498} because that sounds like the righ...
[17:56:49] <wikibugs>	 10Phabricator, 10DBA, 10serviceops, 10User-brennen: Phabricator intermittently slow; db connection failures to m3-master.eqiad.wmnet with "Temporary failure in name resolution" - https://phabricator.wikimedia.org/T279013 (10mmodell) p:05Triage→03Low
[18:03:48] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-Release-Tools: Disable hhvm/php5.x (composer-hhvm-docker) tests for release-tools - https://phabricator.wikimedia.org/T229370 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Did this ages ago, didn't know about this task.
[18:04:00] <wikibugs>	 (03PS1) 10Jforrester: jjb: Switch use of phpmetrics* images to ones with composer 1.10.22 [integration/config] - 10https://gerrit.wikimedia.org/r/683028 (https://phabricator.wikimedia.org/T281283)
[18:04:02] <wikibugs>	 (03PS1) 10Jforrester: tests: Switch from php70 to php74 in examples [integration/config] - 10https://gerrit.wikimedia.org/r/683029
[18:04:06] <wikibugs>	 (03PS1) 10Jforrester: [DNM] dockerfiles: [composer-scratch] Upgrade to 2.0.13 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/683030 (https://phabricator.wikimedia.org/T279857)
[18:04:08] <wikibugs>	 (03PS1) 10Jforrester: [DNM] Zuul: Drop CI support for REL1_31 branch [integration/config] - 10https://gerrit.wikimedia.org/r/683031 (https://phabricator.wikimedia.org/T281294)
[18:04:10] <wikibugs>	 (03PS1) 10Jforrester: jjb: [quibble] Drop PHP70 and PHP71 testing [integration/config] - 10https://gerrit.wikimedia.org/r/683032
[18:04:12] <wikibugs>	 (03PS1) 10Jforrester: Zuul: Update T226945 comment now we've dropped PHP70/71 support [integration/config] - 10https://gerrit.wikimedia.org/r/683033
[18:07:39] <wikibugs>	 10Release-Engineering-Team (Radar), 10MediaWiki-Vagrant, 10Parsoid (Tracking): Decide the fate of Parsoid/JS Puppet in Mediawiki-Vagrant - https://phabricator.wikimedia.org/T259988 (10Izno)
[18:13:23] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] jjb: Switch use of composer* images to ones with composer 1.10.22 [integration/config] - 10https://gerrit.wikimedia.org/r/682994 (https://phabricator.wikimedia.org/T281283) (owner: 10Jforrester)
[18:15:03] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Switch use of composer* images to ones with composer 1.10.22 [integration/config] - 10https://gerrit.wikimedia.org/r/682994 (https://phabricator.wikimedia.org/T281283) (owner: 10Jforrester)
[18:42:04] <wikibugs>	 (03PS9) 10Jgleeson: Added image for civiproxy [releng/dev-images] - 10https://gerrit.wikimedia.org/r/664919 (https://phabricator.wikimedia.org/T268683)
[18:49:32] <James_F>	 Gah. hasharAway updated the sury GPG key but didn't cascade, and that's what broke the updates.
[18:51:41] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Update sury-php images for updated gpg key - https://phabricator.wikimedia.org/T277742 (10Jdforrester-WMF) >>! In T277742#6925080, @hashar wrote: > The sury base image has been updated via https://gerrit.wikimedia.org/r/c/integration/config/+...
[19:05:33] <hashar>	 James_F: my bad
[19:06:15] <hashar>	 I am tired of those images inheritance it is soo cumbersome to maintain
[19:06:23] <James_F>	 hashar: They are, yeah. :-(
[19:06:34] <James_F>	 Am locally re-building to test, but it turns out that takes a while. :-)
[19:06:41] <hashar>	 then if the parent image got the proper gpg sury key
[19:06:54] <wikibugs>	 10Release-Engineering-Team (Logspam), 10Growth-Team, 10PageCuration, 10JavaScript, and 2 others: TypeError: format.replace is not a function in randomToken function used by SearchSatisfaction schema - https://phabricator.wikimedia.org/T272904 (10Jdlrobson) 05Resolved→03Open p:05Triage→03Unbreak! I...
[19:06:59] <James_F>	 hashar: On the plus side, in a couple of months we can delete PHP70 and PHP71 entirely!
[19:07:01] <hashar>	 I would expect a later rebuild of the child images to have th enew key?
[19:07:22] <James_F>	 hashar: Direct children would, if we re-built them. But indirects won't.
[19:07:26] <hashar>	 as for the slow rebuild
[19:07:27] <wikibugs>	 10Release-Engineering-Team (Logspam), 10Growth-Team, 10PageCuration, 10JavaScript, and 2 others: PageTriage extension causes TypeError: format.replace is not a function in randomToken function im SearchSatisfaction schema - https://phabricator.wikimedia.org/T272904 (10Jdlrobson)
[19:07:43] <hashar>	 ah yeah bummer
[19:07:53] <hashar>	 the rebiuld is slow cause we have HDD on the machines
[19:08:05] <hashar>	 and iirc contint1001 has faster disks than contint2001
[19:08:11] <James_F>	 So in the chain sury-php > php80 > composer-php80, a rebuild of composer-php80 won't pick it up.
[19:08:20] <James_F>	 Well, it's slow locally and I have an SDD.
[19:08:24] <James_F>	 Err. SSD.
[19:08:29] <hashar>	 tss tss
[19:08:34] <hashar>	 tmpfs man! tmpfs!
[19:08:41] <James_F>	 "SSD lifetime left: 20%"
[19:08:50] <James_F>	 That's a good sign for my new-ish work laptop. Oops.
[19:10:01] <hashar>	 ;D
[19:10:02] <hashar>	 oh
[19:10:12] <hashar>	 and congratulations on pushing the stretch > buster things
[19:10:20] <James_F>	 Well, we're not finished yet.
[19:10:22] <James_F>	 But thank you. :-)\
[19:10:29] <hashar>	 super happy that wikibase folks found out the source of the test breakage
[19:10:37] <James_F>	 Oh yeah, that was great of them.
[19:10:47] <hashar>	 for ruby 2.5  , that needs a bunch of updates here and there
[19:10:49] <James_F>	 The Wikibase team are all brilliant people.
[19:11:05] <hashar>	 then mediawiki_selenium has been deprecated for years (since 2017?) :-\
[19:11:10] <James_F>	 We're down to only two repos with Ruby CI at this point.
[19:11:19] <James_F>	 (Wikibase and MobileFrontend.)
[19:11:30] <hashar>	 greaat
[19:11:45] <hashar>	 and we might be able to cut a new release of mediawiki selenium if that is really needed
[19:11:50] <hashar>	 it might be trivial enough
[19:11:59] <James_F>	 Sure.
[19:12:11] <James_F>	 But we might also just dump the code into the Wikibase repo.
[19:12:27] <James_F>	 If MF switches to Node (which seems likely).
[19:12:27] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] tests: Switch from php70 to php74 in examples [integration/config] - 10https://gerrit.wikimedia.org/r/683029 (owner: 10Jforrester)
[19:13:17] <wikibugs>	 (03PS1) 10Jforrester: dockerfiles: Follow-up 49281f5fd and actually cascade these updates [integration/config] - 10https://gerrit.wikimedia.org/r/683040 (https://phabricator.wikimedia.org/T277742)
[19:13:51] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] "Oy veh." [integration/config] - 10https://gerrit.wikimedia.org/r/683040 (https://phabricator.wikimedia.org/T277742) (owner: 10Jforrester)
[19:16:04] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: Follow-up 49281f5fd and actually cascade these updates [integration/config] - 10https://gerrit.wikimedia.org/r/683040 (https://phabricator.wikimedia.org/T277742) (owner: 10Jforrester)
[19:16:35] <James_F>	 !log Docker: Rebuilding all Sury-php derivatives for T277742.
[19:16:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:16:38] <stashbot>	 T277742: Update sury-php images for updated gpg key - https://phabricator.wikimedia.org/T277742
[19:18:49] <wikibugs>	 (03CR) 10Jforrester: [C: 04-1] jjb: [quibble] Drop PHP70 and PHP71 testing [integration/config] - 10https://gerrit.wikimedia.org/r/683032 (owner: 10Jforrester)
[19:28:17] <wikibugs>	 (03Abandoned) 10Jforrester: jjb: Switch use of phpmetrics* images to ones with composer 1.10.22 [integration/config] - 10https://gerrit.wikimedia.org/r/683028 (https://phabricator.wikimedia.org/T281283) (owner: 10Jforrester)
[19:35:02] <wikibugs>	 (03PS1) 10Hashar: jjb: compress fresnel browser performance reports [2] [integration/config] - 10https://gerrit.wikimedia.org/r/683045 (https://phabricator.wikimedia.org/T249268)
[19:38:51] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: compress fresnel browser performance reports [2] [integration/config] - 10https://gerrit.wikimedia.org/r/683045 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar)
[19:40:14] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: compress fresnel browser performance reports [2] [integration/config] - 10https://gerrit.wikimedia.org/r/683045 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar)
[19:50:21] <wikibugs>	 (03PS2) 10Jforrester: tests: Switch from php70 to php74 in examples [integration/config] - 10https://gerrit.wikimedia.org/r/683029
[19:50:28] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] tests: Switch from php70 to php74 in examples [integration/config] - 10https://gerrit.wikimedia.org/r/683029 (owner: 10Jforrester)
[19:51:29] <wikibugs>	 (03Merged) 10jenkins-bot: tests: Switch from php70 to php74 in examples [integration/config] - 10https://gerrit.wikimedia.org/r/683029 (owner: 10Jforrester)
[19:53:01] <wikibugs>	 (03PS1) 10Hashar: jjb: properly delete rawSeleniumVideoGrabs dir [integration/config] - 10https://gerrit.wikimedia.org/r/683051 (https://phabricator.wikimedia.org/T249268)
[19:55:51] <wikibugs>	 (03CR) 10Hashar: "Have to test it" [integration/config] - 10https://gerrit.wikimedia.org/r/683051 (https://phabricator.wikimedia.org/T249268) (owner: 10Hashar)
[19:59:09] <James_F>	 hashar: Have we thought about adding a /test.sh script to the docker images to assert that they work? This'd stop the docker-pkg build if e.g. the PHP upgrade meant that `php --version` didn't return or whatever.
[20:01:15] <hashar>	 James_F: is that a new feature?
[20:01:41] <James_F>	 Maybe?
[20:01:53] <James_F>	 2021-04-27 19:58:52,058 [docker-pkg-build] INFO - Successfully tagged docker-registry.discovery.wmnet/releng/mediawiki-phan:0.8.1-s5 (image.py:210)
[20:01:53] <James_F>	 2021-04-27 19:58:52,144 [docker-pkg-build] INFO - Could not find path {path}/test.sh {image}, skipping verification of docker-registry.discovery.wmnet/releng/mediawiki-phan (image.py:428)
[20:06:33] <hashar>	 guess it is documented somewhere
[20:07:41] <hashar>	 there is https://doc.wikimedia.org/docker-pkg/_modules/docker_pkg/image.html#DockerImage.verify
[20:08:12] <James_F>	 Should I file a task?
[20:08:29] <hashar>	 ah https://doc.wikimedia.org/docker-pkg/index.html?highlight=verify#configuration
[20:08:53] <hashar>	 which seems to be to run some testsuite against image names
[20:09:04] <hashar>	 no idea really. Guess some more doc or a tutorial might help
[20:09:16] <James_F>	 Oh, it's for the whole repo rather than per-image?
[20:09:37] <hashar>	 have no idea
[20:09:56] <wikibugs>	 10Continuous-Integration-Infrastructure: Consider using the 'verify' config with docker-pkg to verify CI images when they're built - https://phabricator.wikimedia.org/T281306 (10Jdforrester-WMF)
[20:10:03] <hashar>	 it might be something different
[20:10:11] * James_F nods.
[20:11:07] <hashar>	 https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/673845/2/images/mcrouter/test.sh  ;D
[20:11:29] <James_F>	 Ha.
[20:11:34] <James_F>	 But that's interesting.
[20:11:48] <hashar>	 https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/672768
[20:12:08] <hashar>	 and the docker-pkg patch https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-pkg/+/661138
[20:12:14] <James_F>	 Neat.
[20:12:22] <hashar>	 which finally leads me to the spec https://phabricator.wikimedia.org/T273427
[20:12:48] <hashar>	 to which I have replied
[20:12:53] <hashar>	 and apparently completely forgot about
[20:13:58] <James_F>	 :-D
[20:15:00] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Yak Shaving 🐃🪒): Yak:Train stats - https://phabricator.wikimedia.org/T281307 (10thcipriani)
[20:15:11] <James_F>	 I'd dearly love for docker-pkg to give some sort of indicator of progress. Ah well.
[20:19:26] <hashar>	 James_F: --info maybe?
[20:21:06] <James_F>	 No, I mean like context and progress.
[20:22:05] <James_F>	 "Building image node10-test-browser:0.6.3-s2, 3 of 10 | Step 5 of 12"
[20:25:42] <James_F>	 Finally!
[20:26:00] <James_F>	 So that re-build took ~75 minutes.
[20:27:24] <wikibugs>	 (03PS2) 10Jforrester: jjb: Switch use of PHP images to ones with composer 1.10.22 [integration/config] - 10https://gerrit.wikimedia.org/r/682995 (https://phabricator.wikimedia.org/T281283)
[20:28:37] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T278347 (10jeena)
[20:31:36] <legoktm>	 you can tail -f /tmp/docker-pkg-build.log I guess, idk why that doesn't automatically go to stdout
[20:32:07] <wikibugs>	 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for **xSavitar** - https://phabricator.wikimedia.org/T281309 (10xSavitar)
[20:39:52] <hashar>	 James_F: so yeah docker-pkg --info    would give a bit more details
[20:40:19] <hashar>	 it might be possible to have a 3 of 10 steps progress somehow. Pretty sure the docker api exposes that
[20:40:45] <James_F>	 legoktm: With --info it does, but it's more like --debug. Vast amounts of information, nothing top-level.
[20:41:52] <James_F>	 So you get "2021-04-27 13:40:37,976 [docker-pkg-build] INFO - Step 8/14 : RUN apt-get update …" (good) alongside "2021-04-27 13:40:34,623 [docker-pkg-build] INFO - Processing triggers for libc-bin…" (pointless for most docker work).
[20:44:29] <legoktm>	 James_F: makes sense, file a bug? :)
[20:44:49] <James_F>	 legoktm: Will do,.
[20:52:12] <James_F>	 legoktm: https://phabricator.wikimedia.org/T281312
[21:02:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10serviceops, 10Regression: Apache on doc1001 does not see updated PHP files for hours/days after deployment - https://phabricator.wikimedia.org/T275468 (10hashar)
[21:19:50] <wikibugs>	 (03PS1) 10Ahmon Dancy: mirror-repos.sh: Add --archive option [tools/train-dev] - 10https://gerrit.wikimedia.org/r/683087
[21:20:18] <wikibugs>	 (03CR) 10Ori.livneh: "This change is ready for review." [tools/release] - 10https://gerrit.wikimedia.org/r/682028 (https://phabricator.wikimedia.org/T244075) (owner: 10Ori.livneh)
[21:25:13] <wikibugs>	 (03CR) 10Jforrester: [C: 04-1] jjb: Switch use of PHP images to ones with composer 1.10.22 [integration/config] - 10https://gerrit.wikimedia.org/r/682995 (https://phabricator.wikimedia.org/T281283) (owner: 10Jforrester)
[21:54:11] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+1] "medium risk per security team (T244076#7038729), accepted by WMF manager (T244076#7038729). This means beta deployment can happen, and thi" [tools/release] - 10https://gerrit.wikimedia.org/r/682028 (https://phabricator.wikimedia.org/T244075) (owner: 10Ori.livneh)
[21:59:06] <wikibugs>	 10MediaWiki-Releasing, 10Documentation, 10MW-1.31-release: Announce upcoming formal EOL of REL1_31 - https://phabricator.wikimedia.org/T279859 (10Reedy)
[22:04:49] <wikibugs>	 (03PS1) 10Ahmon Dancy: Add quotes around a string to make my syntax highligher happier [tools/train-dev] - 10https://gerrit.wikimedia.org/r/683096
[22:05:47] <wikibugs>	 (03PS2) 10Ahmon Dancy: Add quotes around a string to make my syntax highligher happier [tools/train-dev] - 10https://gerrit.wikimedia.org/r/683096
[22:05:49] <wikibugs>	 (03PS1) 10Ahmon Dancy: Rename setup to initialize-mediawiki-staging [tools/train-dev] - 10https://gerrit.wikimedia.org/r/683097
[22:05:51] <wikibugs>	 (03PS1) 10Ahmon Dancy: Rename prep to scap-prep-mw-branches [tools/train-dev] - 10https://gerrit.wikimedia.org/r/683098
[22:06:30] <wikibugs>	 (03PS3) 10Ahmon Dancy: Add quotes around a string to make my syntax highlighter happier [tools/train-dev] - 10https://gerrit.wikimedia.org/r/683096
[22:06:32] <wikibugs>	 (03PS2) 10Ahmon Dancy: Rename setup to initialize-mediawiki-staging [tools/train-dev] - 10https://gerrit.wikimedia.org/r/683097
[22:06:34] <wikibugs>	 (03PS2) 10Ahmon Dancy: Rename prep to scap-prep-mw-branches [tools/train-dev] - 10https://gerrit.wikimedia.org/r/683098
[22:36:38] <wikibugs>	 10Release-Engineering-Team (Logspam), 10Growth-Team, 10PageCuration, 10JavaScript, and 2 others: PageTriage extension causes TypeError: format.replace is not a function in randomToken function im SearchSatisfaction schema - https://phabricator.wikimedia.org/T272904 (10Umherirrender) For me it is https://te...
[22:45:40] <wikibugs>	 10Release-Engineering-Team (Logspam), 10Growth-Team, 10PageCuration, 10JavaScript, and 3 others: PageTriage extension causes TypeError: format.replace is not a function in randomToken function im SearchSatisfaction schema - https://phabricator.wikimedia.org/T272904 (10Jdlrobson) I would recommend PageTriag...