[00:05:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: ncmonitor.service on ncmonitor1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:53:04] <wikibugs>	 06Traffic, 10MediaWiki-Action-API, 10Notifications (Echo), 10Tool-quickcategories, 13Patch-For-Review: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11789022 (10DavidBrooks) Can the new permission be defaulted to true for...
[07:40:02] <wikibugs>	 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11789099 (10hashar) >>! In T421827#11785489, @Jdforrester-WMF wrote: > The git commands are done from the `zuul-...
[08:06:26] <wikibugs>	 06Traffic: Upgrade the CDN to HAProxy 3.0 - https://phabricator.wikimedia.org/T417291#11789116 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez
[08:17:00] <wikibugs>	 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11789124 (10Vgutierrez) >>! In T422030#11778885, @Fabfur wrote: > This is most probably due to a deprecation in haproxy configuration directives https://www.haproxy...
[09:33:32] <wikibugs>	 06Traffic, 10MediaWiki-Action-API, 10Notifications (Echo), 10Tool-quickcategories, 13Patch-For-Review: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11789248 (10Urbanecm_WMF) This issue should now be fixable on the bot ow...
[09:37:38] <wikibugs>	 06Traffic, 10MediaWiki-Action-API, 10Notifications (Echo), 10Tool-quickcategories, 13Patch-For-Review: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11789250 (10Aka) I can confirm that it works again after the permission...
[10:02:33] <wikibugs>	 06Traffic, 10MediaWiki-Action-API, 10Notifications (Echo), 10Tool-quickcategories, 13Patch-For-Review: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11789257 (10Novem_Linguae) 05Open→03Resolved a:03Urbanecm_WMF...
[10:38:25] <wikibugs>	 06Traffic, 07OKR-Work, 13Patch-For-Review, 10Test Kitchen (Test Kitchen (Experiment Platform Sprint 22)): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11789301 (10Sfaci) @ssingh Are you ok with a new increment of the traffic...
[10:38:47] <wikibugs>	 06Traffic, 07OKR-Work, 10Test Kitchen (Test Kitchen (Experiment Platform Sprint 22)): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11789302 (10Sfaci)
[11:41:38] <taavi>	 hello friends. for T422040 I will need to allocate a new public LVS service IP in eqiad, do you have a preference between high-traffic1/2?
[11:41:38] <stashbot>	 T422040: Migrate clouddumps https/rsync interfaces behind LVS - https://phabricator.wikimedia.org/T422040
[12:27:32] <vgutierrez>	 taavi: high-traffic2 plz
[12:28:22] <wikibugs>	 06Traffic, 10Data-Services, 10Datasets-General-or-Unknown, 06tools-infrastructure-team: Migrate clouddumps https/rsync interfaces behind LVS - https://phabricator.wikimedia.org/T422040#11789526 (10Vgutierrez)
[15:20:07] <wikibugs>	 06Traffic, 10MediaWiki-Action-API, 10Notifications (Echo), 10Tool-quickcategories, 13Patch-For-Review: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11790091 (10DavidBrooks) I realize this task is closed, but I still...
[17:21:15] <wikibugs>	 06Traffic, 10Data-Services, 10Datasets-General-or-Unknown, 06tools-infrastructure-team: Migrate clouddumps https/rsync interfaces behind LVS - https://phabricator.wikimedia.org/T422040#11790589 (10xcollazo) CC @BTullis
[19:01:26] <wikibugs>	 06Traffic, 10MediaWiki-Action-API, 10Notifications (Echo), 10Tool-quickcategories: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11790940 (10matmarex) @DavidBrooks Indeed, "bot passwords" is a bit of a misnomer, and som...