[01:16:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[01:21:52] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[03:10:25] <wikibugs>	 06Traffic, 06Commons, 06DBA, 06SRE: Unable to save edits or delete pages on Commons – database lag - https://phabricator.wikimedia.org/T402749#11122138 (10Zache) > However, I remain concerned that a determined attacker or a widely used non-compliant script could create the same load again. This risk hi...
[04:19:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[04:24:51] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[04:46:40] <wikibugs>	 06Traffic, 06SRE, 10Wikidata, 10Wikidata-Query-Service: Find a solution for SPARQL federation that is blocked by stricter user agent policy enforcement - https://phabricator.wikimedia.org/T402959#11122213 (10Abbe98) Affected SPARQL backends appear to at least include Fuseki and Virtuoso.
[05:06:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[05:11:51] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[05:39:53] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations: Take advantage of 10Gb NICs in the new network stack - https://phabricator.wikimedia.org/T360297#11122271 (10ayounsi) 05Open→03Resolved a:03ayounsi Closing that task as more specific subtasks exists.
[06:56:16] <wikibugs>	 06Traffic, 06Commons, 06DBA, 06SRE: Unable to save edits or delete pages on Commons – database lag - https://phabricator.wikimedia.org/T402749#11122371 (10Josve05a) (I meant to edit my comment but deleted it… ugh)
[07:55:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[07:59:34] <wikibugs>	 06Traffic, 06Data-Engineering: Reduce noise from duplicate sequence-gap alerts on HaProxy-webrequests - https://phabricator.wikimedia.org/T401383#11122421 (10Antoine_Quhen) 05Resolved→03Open The sequence for bad request id is no more 0. That's good news.  But I'm still observing a gap after each bad reques...
[08:00:51] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[08:26:36] <wikibugs>	 06Traffic, 06Commons, 06DBA, 06SRE: Unable to save edits or delete pages on Commons – database lag - https://phabricator.wikimedia.org/T402749#11122458 (10Aklapper) @Josve05a: I could re-post your comment from my bugmail copy, if you want me to?
[08:28:48] <stevemunene>	 Hello, planning to Add dse-k8s-codfw nodes to BGP in the next 10 mins would this be an appropriate time or should I do it a bit later on?
[08:41:37] <vgutierrez>	 WDYM with adding dse-k8s-codfw nodes to BGP?
[08:43:56] <stevemunene>	 the config referenced on https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/Add_or_remove_nodes#Step_1:_Add_node_to_BGP ie. set the BGP value to true on netbox and run homer from the cumin host
[08:44:36] <stevemunene>	 for the ctrl hosts and worker hosts
[08:46:05] <vgutierrez>	 oh right.. that's for netops then :)
[08:47:14] <stevemunene>	 ack, thanks vgutierrez 
[08:50:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[08:55:51] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[09:38:23] <wikibugs>	 06Traffic, 06Commons, 06DBA, 06SRE: Unable to save edits or delete pages on Commons – database lag - https://phabricator.wikimedia.org/T402749#11122675 (10Bugreporter) >>! In T402749#11122138, @Zache wrote: >> However, I remain concerned that a determined attacker or a widely used non-compliant script...
[09:39:39] <wikibugs>	 06Traffic, 06SRE, 10Wikidata, 10Wikidata-Query-Service: Find a solution for SPARQL federation that is blocked by stricter user agent policy enforcement - https://phabricator.wikimedia.org/T402959#11122678 (10Lucas_Werkmeister_WMDE)
[09:52:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[09:57:51] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[10:01:27] <wikibugs>	 06Traffic, 06Data-Engineering: Request for a new request dataset for caching research - https://phabricator.wikimedia.org/T401331#11122841 (10yazhuoz) @ssingh Thanks for your reply! I'm not sure if it's possible, but it would be super helpful if the logs could include some user-agent and referer info (even in...
[10:04:43] <jinxer-wm>	 FIRING: [5x] HaproxyKafkaSocketDroppedMessages: Unexpected rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[10:09:43] <jinxer-wm>	 RESOLVED: [8x] HaproxyKafkaSocketDroppedMessages: Unexpected rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[11:45:20] <wikibugs>	 06Traffic, 06Commons, 06DBA, 06SRE: Unable to save edits or delete pages on Commons – database lag - https://phabricator.wikimedia.org/T402749#11123289 (10Zache) >>! In T402749#11122675, @Bugreporter wrote: > I believe we should revert {T194864} so autopatroller, patroller and image reviewer will no lo...
[11:51:08] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11123297 (10elukey) >>! In T392851#11068425, @elukey wrote: > 3) Last but not least, `late_command.sh` fails during the Bookworm debian install, and the reason...
[11:52:16] <elukey>	 vgutierrez, sukhe o/ I added a comment to --^, I may have found how to fix late_command.sh for the new cp nodes, but I am not 100% sure what's best to do in late_command
[11:52:31] <elukey>	 like another if for the new cp hosts, or something more elaborate
[11:53:01] <wikibugs>	 06Traffic, 06Commons, 06DBA, 06SRE: Unable to save edits or delete pages on Commons – database lag - https://phabricator.wikimedia.org/T402749#11123303 (10Ladsgroup) I suggest something even more radical: Move CAL (and HotCat) to core. These two gadgets are one of the most widely used and installed gad...
[11:53:11] <elukey>	 the first option is simpler but we'll need to remember to update late command every time a new host is racked
[11:53:14] <elukey>	 that is not ideal
[11:55:47] <elukey>	 the alternative could be to run something like `in-target /usr/sbin/nvme id-ns /dev/nvme0n1`, grep for lba and see what number to use for -l
[13:07:15] <sukhe>	 elukey: noted, will check soon!
[13:08:53] <wikibugs>	 06Traffic, 06SRE, 13Patch-For-Review, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): hCaptcha: Ensure GeoIP and WMF-Uniq cookies are removed in proxied requests - https://phabricator.wikimedia.org/T402713#11123784 (10kostajh)
[13:13:17] <wikibugs>	 06Traffic, 06SRE, 13Patch-For-Review, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): hCaptcha: Ensure GeoIP and WMF-Uniq cookies are removed in proxied requests - https://phabricator.wikimedia.org/T402713#11123816 (10kostajh) 05Open→03Resolved a:03kostajh
[13:13:28] <wikibugs>	 06Traffic, 06SRE, 13Patch-For-Review, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): hCaptcha: Ensure GeoIP and WMF-Uniq cookies are removed in proxied requests - https://phabricator.wikimedia.org/T402713#11123818 (10kostajh) a:05kostajh→03mszabo
[13:20:22] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11123850 (10Papaul) Diff on mr1-ulsfo ` +   bgp { +       group Production { +           type external; +           import BGP_Default; +           expo...
[13:55:38] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Investigate using BGP addpath for unicast IBGP spine/leaf pods - https://phabricator.wikimedia.org/T402640#11124076 (10cmooney) >>! In T402640#11121128, @ayounsi wrote: > If I understand correctly we currently get some "per rack" load balancing, where `E3` might...
[14:24:21] <wikibugs>	 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11124248 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host durum1002.eqiad.wmnet with OS trixie
[14:32:05] <sukhe>	 elukey: that looks promising, nice debugging!
[14:32:24] <sukhe>	 so that also means the firmware upgrade was a red herring?
[14:32:32] <sukhe>	 in this case though, 
[14:32:34] <sukhe>	 Aug 27 11:44:16 in-target: lbaf  1 : ms:0   lbads:12 rp:0
[14:32:48] <sukhe>	 like you said, -l1 should get what we want?
[14:32:52] <elukey>	 exactly yes
[14:33:01] <sukhe>	 and we can differentiate based on 
[14:33:02] <sukhe>	 case $(hostname) in cp[1-9][0-9][0-9][0-9]|sretest2002)
[14:33:20] <elukey>	 the final proof should be to run the nvme command on nodes with nvme and verify that there are three options
[14:33:41] <sukhe>	 that should be enough? or we even probably differentiate on nvme list output or something
[14:34:37] <elukey>	 elukey@cp2027:~$ sudo /usr/sbin/nvme id-ns /dev/nvme0n1
[14:34:37] <elukey>	 [..]
[14:34:37] <elukey>	 lbaf  0 : ms:0   lbads:9  rp:0x1 
[14:34:37] <elukey>	 lbaf  1 : ms:8   lbads:9  rp:0x3 
[14:34:37] <elukey>	 lbaf  2 : ms:0   lbads:12 rp:0 (in use)
[14:34:37] <elukey>	 lbaf  3 : ms:8   lbads:12 rp:0x2 
[14:34:42] <elukey>	 all right confirmed :)
[14:34:56] <sukhe>	 sukhe@cp7001:~$ sudo nvme id-ns /dev/nvme0n1 | grep ^lbaf
[14:34:56] <sukhe>	 lbaf  0 : ms:0   lbads:9  rp:0x1 
[14:34:56] <sukhe>	 lbaf  1 : ms:8   lbads:9  rp:0x3 
[14:34:56] <sukhe>	 lbaf  2 : ms:0   lbads:12 rp:0 (in use)
[14:34:56] <sukhe>	 lbaf  3 : ms:8   lbads:12 rp:0x2 
[14:34:58] <sukhe>	 lbaf  4 : ms:64  lbads:12 rp:0x3 
[14:35:01] <sukhe>	 yeah
[14:35:42] <elukey>	 so we can definitely differentiate for hostname, the only drawback is that we'll need to keep that updated for new hosts etc..
[14:35:57] <sukhe>	 one thing I don't understand, why the same lbads with different lbaf?
[14:36:10] <sukhe>	 what is rp?
[14:37:20] <elukey>	 I think it is "relative performance"
[14:37:38] <sukhe>	 so in current case, w have lbads 12, so 4K block size (2^12)
[14:37:39] <elukey>	 it may indicate some extra perf setting to use IIUC
[14:37:48] <elukey>	 exactly this is my understanding as well
[14:39:21] <sukhe>	 ok, so then -l1 on the new host sounds good, at least my +1
[14:39:37] <sukhe>	 hostname: for now we can just separate based on cp20[43-58]?
[14:43:21] <sukhe>	 10:35:42 < elukey> so we can definitely differentiate for hostname, the only drawback is that we'll need to keep that updated for new hosts etc..
[14:43:30] <elukey>	 +1 yes I think it is the easiest, do your team have time for the late_command change or do you prefer me to do it ?
[14:43:33] <sukhe>	 correct but do note that we only have the codfw and drmrs refresh upcoming
[14:43:40] <sukhe>	 elukey: we will take care of it, thanks :)
[14:43:44] <sukhe>	 you did the most work here anyway
[14:43:44] <elukey>	 <3
[14:58:01] <sukhe>	 fabfur: brett: ^ we need to add an exception for the new cp hosts in codfw, to do this:
[14:58:08] <sukhe>	 in-target /usr/sbin/nvme format "$nvmedev" -l 1
[14:58:40] <sukhe>	 instead of the current in-target /usr/sbin/nvme format "$nvmedev" -l 2, line 71 in modules/install_server/files/autoinstall/scripts/late_command.sh
[14:58:52] <sukhe>	 see the above discussion for context and eluke.y's findings
[15:01:15] <fabfur>	 ok
[15:23:44] <wikibugs>	 06Traffic, 06SRE, 10API Platform (RESTBase Deprecation Roadmap): Block non-browser requests that use generic user agent (UA) headers - https://phabricator.wikimedia.org/T319423#11124709 (10Tgr) Obsoleted by {T400119}, probably?
[15:28:23] <brett>	 ack
[15:40:04] <swfrench-wmf>	 just in case it got lost in the noise: we did have to restart pybals on lvs1018 (high-traffic2) and lvs1019 (low-traffic) today following the conf1007 nginx restart
[15:40:04] <swfrench-wmf>	 (some aspect of connection shutdown didn't work as expected, so the client was unable to converge on a new wait index)
[15:40:04] <swfrench-wmf>	 in any case, all clear now - just wanted to mention it in case it was noticed after the fact :)
[15:52:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[15:57:51] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[16:01:58] <wikibugs>	 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11125078 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host durum1002.eqiad.wmnet with OS trixie completed: - durum1002 (**PASS**)   - Downtimed on Icinga/Aler...
[16:08:53] <wikibugs>	 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11125111 (10hashar)
[16:08:59] <wikibugs>	 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11125114 (10hashar) This prevent us from updating the CI Jenkins. That is done over the API https://integration.wikimedia.org/ci/ using authentication.  Ca...
[16:10:26] <wikibugs>	 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11125121 (10ssingh)
[16:19:01] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125167 (10ssingh)
[16:20:40] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125188 (10ssingh) We are looking into this, thanks for flagging.
[16:24:35] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125199 (10ssingh) @hashar: As the message indicates, it would be preferable if we can set a user-agent as a start, even for an internal traffic requirement like this. Is...
[16:33:04] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125236 (10ssingh) Reading into `./jjb-update` and `https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master`, it seems like this is being run...
[16:34:27] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125243 (10hashar) The code base is legacy and I am not sure how to monkey patch it ([[https://opendev.org/jjb/python-jenkins/src/branch/master/jenki...
[16:35:38] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125272 (10CDanis) As one option for a workaround, running `tunnelencabulator -s` after applying the above patchset will allow you to bypass this tem...
[16:59:00] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125364 (10ssingh) >>! In T403089#11125243, @hashar wrote: > The code base is legacy and I am not sure how to monkey patch it ([[https://opendev.org/...
[17:10:58] <wikibugs>	 06Traffic, 06SRE: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11125437 (10Krinkle)
[17:12:59] <wikibugs>	 06Traffic, 06Data-Engineering, 06SRE: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11125446 (10Krinkle) I believe this used to work in Turnilo via the `webrequest_sampled_128` dataset. Phabricator history seems to confir...
[17:20:22] <wikibugs>	 06Traffic, 06Data-Engineering, 06SRE: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11125465 (10CDanis) I'm not sure if there's an easy way to re-use the logic already in Refine given the current setup of webrequest_sampl...
[17:21:19] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125467 (10hashar) I have filed that as an unbreak now cause that prevented me from deploying updates to jobs and I wasn't aware we would start block...
[17:21:33] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125468 (10hashar) a:03hashar
[17:45:55] <wikibugs>	 06Traffic, 10envoy, 06serviceops, 06SRE: Envoy config updates from v1.26 - https://phabricator.wikimedia.org/T403101 (10RLazarus) 03NEW
[17:54:34] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125681 (10Jdforrester-WMF) 05Open→03Resolved Should now be Resolved.
[17:56:06] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11125686 (10ssingh) >>! In T403089#11125467, @hashar wrote: > I have filed that as an unbreak now cause that prevented me from deploying updates t...
[18:45:09] <wikibugs>	 06Traffic, 06Data-Engineering, 06Infrastructure-Foundations: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11125977 (10CDanis) > Open questions / next steps: > * Is configuring the Puppet server with HDFS access acceptable?  I'll make s...
[18:51:30] <wikibugs>	 06Traffic, 06SRE: Add unique error IDs to 4xx responses - https://phabricator.wikimedia.org/T330973#11126015 (10ssingh) 05Open→03Resolved a:03ssingh This is now done in https://gitlab.wikimedia.org/repos/sre/hiddenparma/-/commit/9f1bd99010be4d84e643bc6ef038c28efb98a092 automatically so I am taking th...
[18:54:12] <wikibugs>	 06Traffic, 10DNS: Central and South American countries in geo-maps - https://phabricator.wikimedia.org/T301605#11126041 (10ssingh) With some recent changes in `operations/dns.git` but especially with `magru` in South America coming online (and thanks to the work of @JameelKaisar and @CDanis), we feel that this...
[18:58:49] <wikibugs>	 06Traffic, 06Infrastructure-Foundations, 06SRE, 10SRE-tools: Abstract LVS restart using cookbook - https://phabricator.wikimedia.org/T334166#11126075 (10ssingh) Aug 2025 update: With Liberica rolled out to everywhere except `eqiad` and `codfw` (pending T352956 for the core sites), `cookbook sre.loadbalance...
[19:00:40] <wikibugs>	 06Traffic, 06Commons, 06Infrastructure-Foundations, 10WMF-General-or-Unknown: Upload to Commons fails with a common ADSL connection in Taiwan - https://phabricator.wikimedia.org/T205619#11126082 (10ssingh) Hi @Jidanni: It's been a while since we last followed up on this. At that time, we were unable to pin...
[19:08:10] <wikibugs>	 06Traffic, 13Patch-For-Review: Upgrade pdns-recursor to 5.x on all prod DNS hosts (all C:dnsrecursor and so possibly WMCS) - https://phabricator.wikimedia.org/T381608#11126121 (10ssingh) 05Open→03In progress
[19:08:27] <wikibugs>	 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11126126 (10ssingh) 05Open→03In progress
[19:10:54] <wikibugs>	 06Traffic, 10DNS, 06SRE: Set mediawiki.gr, wikipedia.pt, and wiktionary.org.uk NS records to WMF - https://phabricator.wikimedia.org/T401438#11126132 (10ssingh) @BCornwall: I think this is done but I will wait for your confirmation. And it seems like we are not moving `wikimedia.pt` (since it points to the s...
[19:20:35] <wikibugs>	 06Traffic, 10DNS: Central and South American countries in geo-maps - https://phabricator.wikimedia.org/T301605#11126166 (10RLazarus) 05Open→03Resolved a:03RLazarus LGTM, thank you for the work!
[19:24:23] <wikibugs>	 06Traffic, 10DNS: Central and South American countries in geo-maps - https://phabricator.wikimedia.org/T301605#11126200 (10RLazarus) a:05RLazarus→03None
[19:24:49] <wikibugs>	 06Traffic, 10DNS, 06SRE: Set mediawiki.gr, wikipedia.pt, and wiktionary.org.uk NS records to WMF - https://phabricator.wikimedia.org/T401438#11126211 (10BCornwall) `wikimedia.pt` redirects to the local chapter but `wikipedia.pt` redirects to to `pt.wikipedia.org`. I've tried emailing the representatives of t...
[19:26:14] <wikibugs>	 06Traffic, 10DNS, 06SRE: Set mediawiki.gr, wikipedia.pt, and wiktionary.org.uk NS records to WMF - https://phabricator.wikimedia.org/T401438#11126215 (10ssingh) >>! In T401438#11126211, @BCornwall wrote: > `wikimedia.pt` redirects to the local chapter but `wikipedia.pt` redirects to to `pt.wikipedia.org`. I'...
[19:29:19] <wikibugs>	 06Traffic, 10DNS, 06SRE: Set mediawiki.gr, wikipedia.pt, and wiktionary.org.uk NS records to WMF - https://phabricator.wikimedia.org/T401438#11126237 (10Mike_Peel) Maybe @Alchimista can help with the .pt domain?
[20:00:22] <wikibugs>	 06Traffic, 10Continuous-Integration-Infrastructure, 07Jenkins: Cannot update CI Jenkins jobs - https://phabricator.wikimedia.org/T403089#11126321 (10hashar) The script used to deploy CI job uses python-jenkins. We found out it reads the environment variable `JENKINS_API_EXTRA_HEADERS` which, as the name...
[22:55:16] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11126833 (10Papaul) BGP is up and running between mr1-ulsfo and both cr3 and cr4 ` Peer                     AS      InPkt     OutPkt    OutQ   Flaps Las...