[06:17:25] <legoktm>	 Majavah: ping when you're around?
[06:18:16] <wikibugs>	 10Continuous-Integration-Config, 10Gerrit: Unarchive analytics/udplog repository - https://phabricator.wikimedia.org/T276422 (10Legoktm) 05Stalled→03Open This should be ready now, @Majavah fixed the remaining lintian error.
[06:28:22] <Majavah>	 legoktm: will be in ~15 minutes
[06:43:49] <Majavah>	 legoktm: hi, now I am properly here
[06:46:08] <legoktm>	 Majavah: if you wanna `apt install udplog` it'll upgrade it 
[06:46:52] <legoktm>	 And if the log events continue to flow properly I'll do the upgrade in prod too
[06:48:17] <Majavah>	 legoktm: few warnings, but logs are still flowing https://phabricator.wikimedia.org/P14639
[06:49:29] <legoktm>	 Hmm
[06:51:28] <Majavah>	 you left a comment about /etc/udp2log being a directory, but I thought you already fixed that
[06:51:54] <legoktm>	 I did
[06:52:05] <legoktm>	 But
[06:52:07] <legoktm>	 https://gerrit.wikimedia.org/g/operations/puppet/+/dcfaf91a471f13f8148880d0b022d7813a36107c/modules/udp2log/manifests/init.pp#37
[06:55:22] <legoktm>	 https://gerrit.wikimedia.org/g/operations/puppet/+/dcfaf91a471f13f8148880d0b022d7813a36107c/modules/udp2log/manifests/instance.pp#57
[06:55:35] <legoktm>	 So it's supposed to be a directory?
[06:56:47] <Majavah>	 looks like it, yes
[07:22:17] <legoktm>	 well, certainly isn't a new issue
[07:24:46] <wikibugs>	 10Beta-Cluster-Infrastructure: Beta SWIFT seems to be broken - https://phabricator.wikimedia.org/T276179 (10Majavah) Beta PrivateSettings refers to a nonexistent server `deployment-ms-fe02`:  `lang=php $wmfSwiftConfig = array( 'eqiad' => array(         'authUrl'            => 'http://deployment-ms-fe02.deploymen...
[08:07:38] <Majavah>	 uhhh why did deployment-logstash03 stop receiving events when I ran puppet on it to try to investigate its puppet failures
[08:08:02] <wikibugs>	 10Gerrit: Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" - https://phabricator.wikimedia.org/T263293 (10hashar) From T276500 ` GIT_TRACE=1` doesn't give much more details. We need the ssh client side verbose mode: `GIT_SSH_COMMAND='ssh -v' git clone ssh://.....`
[08:09:48] <Majavah>	 UDP listener died {:exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>,
[08:12:56] <Majavah>	 https://phabricator.wikimedia.org/T241481, port conflict with rsyslogd
[08:13:49] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-logstash03 puppet errors - https://phabricator.wikimedia.org/T276521 (10Majavah)
[08:17:21] <wikibugs>	 10Beta-Cluster-Infrastructure, 10SRE: deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd - https://phabricator.wikimedia.org/T241481 (10Majavah)
[08:24:19] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Jessie Deprecation): Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch) - https://phabricator.wikimedia.org/T238707 (10Majavah)
[08:24:21] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-logstash03 puppet errors - https://phabricator.wikimedia.org/T276521 (10Majavah)
[08:32:26] <Majavah>	 !log deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair T241481, directory is different on deployment-logstash2
[08:32:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:32:29] <stashbot>	 T241481: deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd - https://phabricator.wikimedia.org/T241481
[08:48:43] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10Majavah) This happened again, it broke when I ran puppet on logstash03 when investigating {T276521...
[09:09:11] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10Majavah) Found this error message in `/var/log/logstash/logstash-plain-2021-03-05.log`:  `   Plugi...
[09:52:23] <hashar>	 qchris: if you are around, any reason to have created   operations.git repo ? ;)
[09:54:33] <qchris>	 For most parts of the repo hierarchy, we have permission inheritance from a parent like `foo/bar` inherits from `foo`. `operations` was missing, so I created it, when today creating `operations/alerts`.
[09:54:42] <qchris>	 Is the `operations` repo in the way somehow?
[09:55:14] <hashar>	 na wondering. So that is supposed to be a permission only one
[09:55:17] <hashar>	 not to host code
[09:55:23] <qchris>	 Right.
[09:55:34] <hashar>	 I am not sure what are the parents for the various operations/*/* repos now ;]
[09:56:27] <qchris>	 If you say `operations` should go away, let's delete it again. I just followed what we have elsewhere. Like all extensions (hopefully) inherit from `mediawiki/extensions` and so on.
[09:57:04] <hashar>	 no it is fine to have it
[09:57:22] <hashar>	 I just got some annoying notification from github that automatically made me to subscribe to all change made to github.com/wikimedia/operations
[09:57:35] <hashar>	 and I guess I will just convert it to a permission only repo
[09:57:43] <hashar>	 just have to change HEAD to point to refs/meta/config
[09:57:44] <hashar>	 iir
[09:57:45] <hashar>	 c
[09:58:36] <hashar>	 done and I will delete the github repo
[09:58:43] <hashar>	 qchris: thanks!
[09:59:16] <qchris>	 Ok. Cool. Thanks!
[09:59:35] <hashar>	 that whole hierarchy of repositories and groups is really the messy part in our gerrit instance
[09:59:46] <hashar>	 it is super hard to figure out which rights end up actually applied :]
[10:00:23] <qchris>	 I try to converge against proper parents when I create repos. But there are quite a number of outliers :-/
[10:00:40] <qchris>	 But I guess the days of that problem are counted :-)
[10:01:05] <hashar>	 hmm ...  rand(2-4) * 365.25
[10:01:19] <qchris>	 :-D
[10:01:38] <hashar>	 gerrit ls-projects --tree   somehow gives some clue
[10:05:53] <hashar>	 qchris: I have reparented bunch of operations/* repo to use operations.git instead of All-Projects.git ;)
[10:06:03] <qchris>	 Nice. \o/
[10:06:38] <qchris>	 That's something I would not dare :-)
[10:10:18] <wikibugs>	 10Project-Admins: New project: Wiki-Techstorm-2021 - https://phabricator.wikimedia.org/T268223 (10Aklapper) 05Stalled→03Resolved a:03Aklapper I decided to be bold: https://phabricator.wikimedia.org/project/view/5240/
[10:12:47] <wikibugs>	 10Project-Admins, 10Maps: Create geoshapes component under the Maps project - https://phabricator.wikimedia.org/T276479 (10Aklapper) I suppose this will also be a [milestone](https://www.mediawiki.org/wiki/Phabricator/Project_management#Parent_Projects,_Subprojects_and_Milestones), looking at https://phabricat...
[10:16:23] <wikibugs>	 10Project-Admins, 10WMIT-Infrastructure: Create a #WMIT-LimeSurvey Tag - https://phabricator.wikimedia.org/T276343 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #wmit-limesurvey has been created: https://phabricator.wikimedia.org/project/view/5241/  Interested people are welcome to...
[11:39:25] <wikibugs>	 (03CR) 10Kosta Harlan: "> Patch Set 1:" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/668549 (https://phabricator.wikimedia.org/T276499) (owner: 10Kosta Harlan)
[11:48:27] <Majavah>	 !log live hack beta puppetmaster to fix hopefully trust store location; T276521 and possibly others
[11:48:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[11:48:33] <stashbot>	 T276521: deployment-logstash03 puppet errors - https://phabricator.wikimedia.org/T276521
[11:57:41] <wikibugs>	 10MediaWiki-Codesniffer, 10Wikidata, 10wdwb-tech-focus: Consider adding bash scripts to mediawiki-codesniffer for only running on touched files - https://phabricator.wikimedia.org/T251533 (10Lucas_Werkmeister_WMDE)
[13:04:28] <wikibugs>	 10Beta-Cluster-Infrastructure: Beta SWIFT seems to be broken - https://phabricator.wikimedia.org/T276179 (10Tgr) An attempt to upload a file fails with `Could not create directory "mwstore://local-multiwrite/local-public/c/c7".`  Manually, the current Swift config in PrivateSettings seems to work: ` tgr@deployme...
[13:09:16] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-logstash03 puppet errors - https://phabricator.wikimedia.org/T276521 (10Majavah) `modules/profile/manifests/logstash/collector.pp` had a path to Kafka trust store that didn't exist. I applied a beta-only hack to change it to see if it works. The machine now runs puppet p...
[13:13:48] <Majavah>	 !log move profile::etcd::cluster_name hiera key from deployment-etcd prefix to deployment-etcd-01 vm specific
[13:13:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:14:23] <tgr_>	 what's up with deployment-fluorine02? logs say it has been shut down yesterday, but in the docs it is still mentioned as the logging host
[13:14:44] <RhinosF1>	 Majavah: ^
[13:14:53] <Majavah>	 tgr_: is it still mentioned in docs? it's jessie and I made deployment-mwlog01 as buster yesterday
[13:15:05] <RhinosF1>	 tgr_: I'd guess the docs need changing
[13:17:35] <tgr_>	 done
[13:17:43] <Majavah>	 thanks!
[13:18:07] <tgr_>	 a SAL entry might help in similar cases
[13:18:19] <Majavah>	 I thought I did that
[13:18:33] <Majavah>	 https://sal.toolforge.org/log/O4Mt_XcB1jz_IcWugoRj at least
[13:18:47] <Majavah>	 and https://sal.toolforge.org/log/GJ5j_XcBa_6PSCT9h9Az
[13:20:13] <tgr_>	 sorry, must have made a mistake when searching, then
[13:21:51] <Majavah>	 please shout if you see anything weird with deployment-mwlog01, otherwise I'm planning on deleting fluorine02 after some time
[13:40:20] <Majavah>	 !log create deployment-etcd02 and sign its puppet certificate T276462
[13:40:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:40:24] <stashbot>	 T276462: Replace deployment-etcd-01 with a Buster host - https://phabricator.wikimedia.org/T276462
[14:00:51] <wikibugs>	 (03CR) 10Ottomata: "> what do you think, @ottomata?" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/626739 (owner: 10Mholloway)
[14:33:33] <wikibugs>	 (03CR) 10Mholloway: "> Patch Set 3:" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/626739 (owner: 10Mholloway)
[14:41:48] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10User-Addshore, and 3 others: Run browser tests as part of "npm test" for wikidata/query/gui in CI - https://phabricator.wikimedia.org/T222200 (10Addshore) 05Open→03Resolved
[14:43:47] <wikibugs>	 (03CR) 10Ottomata: "Ok, let's do it!" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/626739 (owner: 10Mholloway)
[15:17:22] <wikibugs>	 (03PS1) 10Zoranzoki21: Add Zabe in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/668712
[15:35:46] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Add Zabe in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/668712 (owner: 10Zoranzoki21)
[15:38:39] <wikibugs>	 (03Merged) 10jenkins-bot: Add Zabe in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/668712 (owner: 10Zoranzoki21)
[16:26:13] <wikibugs>	 (03PS5) 10Hashar: jjb: switch striker jobs to one based off buster [integration/config] - 10https://gerrit.wikimedia.org/r/651297 (owner: 10Jforrester)
[16:26:57] <wikibugs>	 (03CR) 10Hashar: "Rebased to increment the images versions.  Will look at deploying that eventually, trigger the jobs and report the stracktraces to develop" [integration/config] - 10https://gerrit.wikimedia.org/r/651297 (owner: 10Jforrester)
[16:34:38] <wikibugs>	 (03PS1) 10Ahmon Dancy: create-databases: use a more complex admin password [tools/train-dev] - 10https://gerrit.wikimedia.org/r/668728
[16:35:17] <wikibugs>	 (03PS1) 10Ahmon Dancy: mirror-repos.sh: add --update mode [tools/train-dev] - 10https://gerrit.wikimedia.org/r/668729
[16:35:34] <wikibugs>	 (03PS1) 10Ahmon Dancy: nginx: Add X-Forwarded-Proto: https header [tools/train-dev] - 10https://gerrit.wikimedia.org/r/668730
[16:35:52] <wikibugs>	 (03PS1) 10Ahmon Dancy: Terminate ControlMaster ssh process after playbooks [tools/train-dev] - 10https://gerrit.wikimedia.org/r/668731
[16:36:07] <wikibugs>	 (03PS1) 10Ahmon Dancy: mirror-repos.sh: Add additional repos [tools/train-dev] - 10https://gerrit.wikimedia.org/r/668732
[16:36:19] <wikibugs>	 (03PS1) 10Ahmon Dancy: Move to 1.36.0-wmf.30 and 1.36.0-wmf.31 [tools/train-dev] - 10https://gerrit.wikimedia.org/r/668733
[17:31:03] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "https://gerrit.wikimedia.org/r/c/labs/tools/ldap/+/668739  passed" [integration/config] - 10https://gerrit.wikimedia.org/r/651297 (owner: 10Jforrester)
[17:32:31] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: switch striker jobs to one based off buster [integration/config] - 10https://gerrit.wikimedia.org/r/651297 (owner: 10Jforrester)
[17:33:01] <wikibugs>	 (03CR) 10Hashar: "And there is:" [integration/config] - 10https://gerrit.wikimedia.org/r/651297 (owner: 10Jforrester)
[17:50:13] <Majavah>	 !log switch deployment-prep hiera key etcd_host to use deployment-etcd02 ref T276462
[17:50:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:50:18] <stashbot>	 T276462: Replace deployment-etcd-01 with a Buster host - https://phabricator.wikimedia.org/T276462
[17:58:14] <wikibugs>	 (03CR) 10Hashar: "Striker is not ready for Buster. Its mysql library does not recognizes libmariadbclient.so.3 shipped by Buster: T276605.  Will revert and " [integration/config] - 10https://gerrit.wikimedia.org/r/651297 (owner: 10Jforrester)
[18:00:10] <wikibugs>	 (03PS1) 10Hashar: jjb: revert labs/tools/striker job to Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/668752 (https://phabricator.wikimedia.org/T276605)
[18:00:37] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/668752 (https://phabricator.wikimedia.org/T276605) (owner: 10Hashar)
[18:02:06] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: revert labs/tools/striker job to Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/668752 (https://phabricator.wikimedia.org/T276605) (owner: 10Hashar)
[18:25:31] <wikibugs>	 (03CR) 10Mholloway: "> Patch Set 3:" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/626739 (owner: 10Mholloway)
[18:28:10] <James_F>	 legoktm: codesearch2phab sounds cool, but like it could be massively abused…
[18:28:58] <James_F>	 Tie it together with OAUTH and you too can create a mass-whine link for all people on the Village Pump to complain by creating duplicate tickets! ;-(
[18:29:41] <legoktm>	 haha
[18:29:57] <legoktm>	 I think if it was limited to #Trusted-Contributors or something it might be managable
[19:04:29] <James_F>	 Perhaps.
[19:14:59] <Majavah>	 !log beta cluster etcd was switched from deployment-etcd-01 to deployment-etcd02 ref T276462
[19:15:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:15:05] <stashbot>	 T276462: Replace deployment-etcd-01 with a Buster host - https://phabricator.wikimedia.org/T276462
[19:22:42] <greg-g>	 Majavah: +1, ty
[19:23:23] <Majavah>	 greg-g: this is the second Jessie I've migrated, too many to go
[19:25:18] * greg-g nods
[19:25:45] <legoktm>	 2 less jessie hosts makes the world a happier place :)
[19:26:39] <Majavah>	 do we have any plans for mw-on-k8s-on-deployment-prep? someone needs to implement it and I'm not sure who that someone is
[19:26:59] <James_F>	 I asked yesterday.
[19:27:02] <James_F>	 "No".
[19:27:27] <Majavah>	 ouch
[19:27:44] <James_F>	 Indeed.
[19:27:55] <legoktm>	 Personally I think there's a definite need for a beta cluster-like environment but I don't think anyone has actually come up with any plans
[19:28:26] <Majavah>	 then there is the case of deployment-sca[01-02] which are Jessie and have services to be migrated to somewhere, the puppet classes creating said services are long gone
[19:28:41] <James_F>	 Yeah, that's my particular interest.
[19:29:01] <James_F>	 Trialling new production services in Beta Cluster is, umm, undefined behaviour.
[19:29:22] <Majavah>	 If there is a sensible solution I'm happy to help implement it, but I'm not setting up a k8s cluster by myself :/
[19:29:35] <James_F>	 Just a little one! ;-(
[19:29:43] <legoktm>	 I think that is the sensible solution fwiw, setting up a k8s cluster
[19:29:51] <James_F>	 Theoretically it should be relatively simpler than the faux scb mirroring.
[19:30:00] <Majavah>	 Setting up the cluster is the easy part, getting services there isn't
[19:30:10] <Majavah>	 I think we're running some services in docker there
[19:30:30] <Majavah>	 !log shutdown deployment-etcd-01 to see if anything breaks, will delete if nothing has broken during next week T276462
[19:30:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:30:33] <stashbot>	 T276462: Replace deployment-etcd-01 with a Buster host - https://phabricator.wikimedia.org/T276462
[19:30:40] <legoktm>	 yeah, all the charts have to be compatible and then figuring out some auto-deployment mechanism
[19:34:50] <Amir1>	 Does anyone know where is this file from? https://gerrit.wikimedia.org/r/c/operations/puppet/+/666979/4/modules/phabricator/manifests/tools.pp
[19:34:58] <Amir1>	 https://github.com/search?q=org%3Awikimedia+public_task_dump&type=code
[19:35:03] <Amir1>	 brings nothing
[19:35:19] <Amir1>	 neither codesearch
[19:35:28] <mutante>	 I think it must be in the phabricator deploy repo
[19:35:32] <mutante>	 twentyafterfour: ?
[19:36:12] <Majavah>	 !log release deployment-prep floating ip 185.15.56.7, was used for mailman upgrade which is now on its own project
[19:36:13] <mutante>	 Amir1: Diffusion?
[19:36:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:36:17] <mutante>	 https://phabricator.wikimedia.org/rOPUP2195a4062819d6904fdc4c9a0b20db6abe772ce4
[19:36:19] <James_F>	 https://phabricator.wikimedia.org/rPHTOde38d14f7f2529160d53009b93d6902426ce687c#change-roNf9viR1OHj
[19:36:22] <James_F>	 Ha.
[19:36:46] <James_F>	 https://phabricator.wikimedia.org/diffusion/PHTO/browse/master/public_task_dump.py
[19:38:17] <Amir1>	 Thanks
[19:38:26] <Amir1>	 why all files in this repo are 777?
[19:38:37] <Amir1>	 most
[19:38:55] <James_F>	 Phabricator tools are dark and full of shadows.
[19:39:35] <Amir1>	 :((((
[19:39:48] <Majavah>	 legoktm: as an SRE, do you happen to know if/when T224579 is going to happen? I need the ircd packaged for buster to get rid of deployment-ircd
[19:39:48] <stashbot>	 T224579: Migrate irc.wikimedia.org/kraz to Buster - https://phabricator.wikimedia.org/T224579
[19:42:46] <Majavah>	 thinking about it, is irc rc really necessary on beta?
[19:43:56] <legoktm>	 so...there's this plan to move away from our custom patched ircd to this new Python-kafka thing, see https://phabricator.wikimedia.org/T234234
[19:44:12] <legoktm>	 I have no idea about the status of it, ask on the task? 
[19:45:09] <legoktm>	 no idea if it's "necessary", maybe people use it for monitoring? or maybe the goal was just mirroring it from prod into beta
[19:45:10] <James_F>	 Should that really block this work, though?
[19:45:27] <James_F>	 But yeah, I'd just dump IRC RC from Beta Cluster for now.
[19:45:34] <legoktm>	 one of the reasons for the switch is that we don't want to keep patching the ircd
[19:45:42] * James_F nods.
[19:45:45] <James_F>	 Definitely long-term.
[19:45:53] <legoktm>	 someone will have to rebase the patch against the new version for creating a buster package
[19:46:00] <James_F>	 But "OMG jessie!" shouldn't be blocked by deciding to re-platform.
[19:46:14] <James_F>	 (Easy for me to say, I don't have the rights to do any of the work.)
[19:47:47] <Majavah>	 hmh
[19:47:56] <ottomata>	 status of T234234 has always been 'needs an owner / prioritized'  
[19:47:56] <stashbot>	 T234234: Port architecture of irc-recentchanges to Kafka - https://phabricator.wikimedia.org/T234234
[19:48:16] <ottomata>	 we def want to do it, but it hasn't really been resourced or prioritized
[19:48:16] <James_F>	 :-(
[19:48:23] <Majavah>	 that definitely won't finish before Jessie needs to be gone :/
[19:48:29] <James_F>	 No.
[19:48:38] <James_F>	 Kill it with fire, along with the Sentry boxes.
[19:48:51] <James_F>	 If someone complains, good news, you've found the owner.
[19:49:21] <Majavah>	 I think sentry boxes were already deleted when someone realized deployment-prep was running out of quota
[19:49:39] <Majavah>	 in theory it should be as simple as overriding $wmgUseRC2UDP to false
[19:49:53] <legoktm>	 Majavah: I'd ask on the first migration to buster task if it's going to happen, and then we can see about uploading a new ircd-ratbox for buster
[19:50:20] <Majavah>	 on T224579?
[19:50:21] <stashbot>	 T224579: Migrate irc.wikimedia.org/kraz to Buster - https://phabricator.wikimedia.org/T224579
[19:50:40] <legoktm>	 yeah, if we are going to upgrade kraz, we probably will want to test it elsewhere first
[19:51:56] <legoktm>	 ircd-ratbox was removed from Debian, so we probably won't upgrade as long as it still works (on Buster)
[19:53:31] <Majavah>	 left a comment on there
[19:58:51] <James_F>	 The sca ones can presumably just be deleted?
[19:58:57] <James_F>	 Oh, wait, no.
[20:12:31] <Majavah>	 no, they still have some things running
[20:14:14] <mutante>	 Majavah: I have one thing to say about "sentry": https://gerrit.wikimedia.org/r/c/operations/puppet/+/654912
[20:15:12] <James_F>	 Majavah: Have updated T218729 to hopefully make it simpler (and not have outdated entries).
[20:15:12] <stashbot>	 T218729: Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster - https://phabricator.wikimedia.org/T218729
[20:16:15] <legoktm>	 mcs01 sounds like mobile content service
[20:17:55] <James_F>	 Ooh. A non-k8s legacy deploy of a single service?
[20:18:01] <James_F>	 Meh. But makes sense.
[20:21:52] <James_F>	 I'm going to shut off memc06 and 07 per the determination on T250585 that they're unused.
[20:21:53] <stashbot>	 T250585: Determine purpose of deployment-memc0[67] - https://phabricator.wikimedia.org/T250585
[20:23:38] <James_F>	 !log Disabling deployment-memc07 on the grounds that it's an unreferenced Jessie box we don't want any more T250585
[20:23:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:25:00] <James_F>	 !log Disabling deployment-memc06 on the grounds that it's an unreferenced Jessie box we don't want any more T250585
[20:25:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:11:18] <wikibugs>	 10Continuous-Integration-Config, 10Striker: Striker CI fails when run under Buster due to libmariadbclient - https://phabricator.wikimedia.org/T276605 (10bd808)
[21:21:37] <Lucas_WMDE>	 FYI, beta cluster currently has no JS (my fault, https://phabricator.wikimedia.org/T276619) – a fix is up at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/668781 if anyone would like to review it :)
[21:28:34] <legoktm>	 +2'd
[21:30:01] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikibase Service Migration, 10Wikidata, 10wdwb-tech-focus, 10Patch-For-Review: No JavaScript on Beta due to PHP error in load.php: ArgumentCountError: array_merge() expects at least 1 parameter, 0 given - https://phabricator.wikimedia.org/T276619 (10Lucas_Werkmeister_WMDE...
[21:30:09] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10Lucas_Werkmeister_WMDE)
[21:30:11] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikibase Service Migration, 10Wikidata, 10wdwb-tech-focus, 10Patch-For-Review: No JavaScript on Beta due to PHP error in load.php: ArgumentCountError: array_merge() expects at least 1 parameter, 0 given - https://phabricator.wikimedia.org/T276619 (10Lucas_Werkmeister_WMDE)
[21:30:19] <Lucas_WMDE>	 ah, thanks!
[21:31:41] <wikibugs>	 10Beta-Cluster-Infrastructure, 10cloud-services-team (Kanban): Delegate beta.wmcloud.org and svc.deployment-prep.eqiad1.wikimedia.cloud NDS zones to deployment-prep - https://phabricator.wikimedia.org/T276624 (10Majavah)
[21:40:47] <wikibugs>	 10Beta-Cluster-Infrastructure, 10cloud-services-team (Kanban): Delegate beta.wmcloud.org and svc.deployment-prep.eqiad1.wikimedia.cloud DNS zones to deployment-prep - https://phabricator.wikimedia.org/T276624 (10Majavah)
[21:42:05] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: forward external traffic to gitlab VMs (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) I think this ticket now just turned into "open p...
[21:42:46] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn)
[21:43:25] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) 05Open→03Stalled
[21:43:28] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10Dzahn)
[21:43:30] <wikibugs>	 10Release-Engineering-Team-TODO, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10Dzahn)
[21:55:12] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Jessie Deprecation): Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster - https://phabricator.wikimedia.org/T218729 (10Mholloway) deployment-mcs01 is obsolete, superseded by deployment-docker-mobileapps01.  I'll shut it down momentarily.
[21:59:51] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Jessie Deprecation): Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster - https://phabricator.wikimedia.org/T218729 (10Mholloway)
[22:02:40] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikibase Service Migration, 10Wikidata, 10wdwb-tech-focus, and 2 others: No JavaScript on Beta due to PHP error in load.php: ArgumentCountError: array_merge() expects at least 1 parameter, 0 given - https://phabricator.wikimedia.org/T276619 (10Lucas_Werkmeister_WMDE) 05Ope...
[22:02:42] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10Lucas_Werkmeister_WMDE)
[22:06:12] <mutante>	 releases1002 - crit disk space alert since 9 days, probably nobody is getting notified.  letting it ignore /run/docker to avoid that alert noise
[22:17:06] <mutante>	 22:16 <+icinga-wm> RECOVERY - Disk space on releases1002 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=releases1002&var-datasource=eqiad+prometheus/ops
[22:17:12] <mutante>	 fixed now
[22:30:14] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10thcipriani) >>! In T276144#6887868, @Dzahn wro...
[22:41:53] <twentyafterfour>	 Amir1: that file is https://phabricator.wikimedia.org/diffusion/PHTO/browse/master/public_task_dump.py
[22:42:18] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Sergey.Trofimovsky.SF) From our perspective Gi...
[22:42:48] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: Define auth strategy for GitLab - https://phabricator.wikimedia.org/T274461 (10thcipriani) Aside question for outbound ports for {T276144}. Looking at https://apereo.github.io/cas/4.2.x/protocol/CAS-Pr...
[22:42:54] <Amir1>	 Thanks. Found it it's just hard to fix the warnings because the version in production seems to be different :/
[22:43:05] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) > much of production (i.e., phab) is no...
[22:43:23] <twentyafterfour>	 the version of that file is different? 
[22:43:25] <Amir1>	   File "/srv/deployment/phabricator/deployment-cache/revs/4547f31de8f69854e0cd9d3e0a802ce517360ee0/tools/wmfphablib/phabdb.py", line 1548, in sql_x
[22:43:40] <Amir1>	 we saw this in production
[22:43:50] <Amir1>	 there is no such line in the phabdb.py
[22:44:16] <twentyafterfour>	 weird
[22:45:41] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) >>! In T276144#6887970, @Sergey.Trofimo...
[22:49:04] <wikibugs>	 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) @Sergey.Trofimovsky.SF Do you agree we...
[23:42:32] <wikibugs>	 10Project-Admins: Create an #ai-governance tag - https://phabricator.wikimedia.org/T276639 (10ACraze)
[23:54:11] <wikibugs>	 10Phabricator, 10Project-Admins, 10phabricator maintenance bot: #phabricator_maintenance_bot should be under #tools, not #toolforge - https://phabricator.wikimedia.org/T273273 (10bd808) @Ladsgroup You made #phabricator_maintenance_bot and it looks like when you did you disregarded the existing conventions fo...