[01:40:53] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12579616 (10bd808) `lang=shell-session,counterexample bd808@deployment-cp-upload09:~$ sudo run-puppet-agent Info: Using environment '... [01:46:13] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12579626 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/95a730d04a511a77945c03306b8a05c8504061ad%5E... [01:46:29] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12579627 (10bd808) p:05Triage→03High [01:50:43] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12579670 (10ssingh) @Fabfur: can you look into this please? I think we just need to do this but I may be wrong. ` <%- if @use_kapow_... [01:51:21] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12579671 (10bd808) The `before => [Service['haproxy'], $site_resource],` constraint on `file { $kapow_score_file:` seems to be the... [02:01:58] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#12579759 (10ssingh) >>! In T122097#12495291, @SBurnett-WMF wrote: > @ssingh - would Q3 work for this? Ouch, I am very s... [08:52:31] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780#12580470 (10ayounsi) Looks like they found something ! Not sure how that integrates in the big picture, but worth looking into. > After carefully... [08:52:34] 06Traffic: Upgrade to HAProxy 3.2.27 - https://phabricator.wikimedia.org/T445475 (10dkertesz) 03NEW [09:05:21] 06Traffic: Upgrade to HAProxy 3.2.27 - https://phabricator.wikimedia.org/T445475#12580527 (10dkertesz) 05Open→03In progress [09:13:01] 06Traffic: Upgrade to HAProxy 3.2.27 - https://phabricator.wikimedia.org/T445475#12580553 (10Vgutierrez) [09:15:41] hi folks, is there anyone today who might be able to help me with review and deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1326930? [09:18:03] 06Traffic: Upgrade to HAProxy 3.2.27 - https://phabricator.wikimedia.org/T445475#12580576 (10Vgutierrez) [09:19:33] bjensen: let me see [09:26:32] bjensen: looks good to me [09:27:01] vgutierrez: thanks! i'm not sure how to go about deployment, is that something i ought to do? [09:28:22] bjensen: so if you're extra cautious you'd disable puppet on `A:cp-text` then depool and run puppet on a single node, test that test.wp.o works as expected, repool and reenable puppet [09:28:37] if you're feeling brave, merge it and let puppet do its thing [09:28:50] happy to assist with the first approach :) [09:30:10] i would strongly prefer the first approach :D i'm not exactly sure how to go about the testing, how do i ensure that my test request is pointed at the node that's run puppet? [09:34:39] bjensen: you can ssh into any text node and run curl in there [09:35:11] bjensen: what's your DC? esams or drmrs? [09:35:28] the DC i'm closest to? probably esams [09:35:37] if it's esams, it's pretty easy to do it from your local connection cause we have single backend in there [09:36:07] curl https://en.wikipedia.org -v -so /dev/null 2>&1 |grep -i x-cache [09:36:08] < x-cache: cp6016 hit, cp6010 hit/1199 [09:36:46] ah i just looked in the dev tools in firefox :D cp3075, so esams? [09:36:52] bjensen: that's right :) [09:37:02] so you could target cp3075 for your test [09:37:17] but if you depool it to avoid impacting other users you need to ssh into the host [09:37:29] and I'd do that given that esams is our busiest DC :) [09:37:42] ahhhh, okay, makes sense [09:39:56] so, i ought to disable puppet on the text nodes via cumin, merge the change, depool cp3075, run puppet on cp3075, and test via curl on that host? [09:40:07] yes, that's right [09:40:20] the cumin alias you are looking for is A:cp-text [09:41:33] alright, doing that now [09:45:13] how do i go about depooling cp3075? [09:47:04] (puppet is disabled on the text nodes and the change has been merged) [09:48:03] vgutierrez: ^ (sorry, it's occurring to me that perhaps the depool should have happened first) [09:49:47] bjensen: `sudo -i depool` on the node or `sudo -i confctl select name=cp3075.esams.wnet set/pooled=no` in a puppetserver [09:50:31] thanks! that's now done, proceeding to run puppet on that node [09:53:02] BTW given that the VIP is in the loopback interface of every cp node, curl https://test.wikipedia.org is enough to target the current cp server [09:53:09] no need to use `--connect-to` or similar options [09:53:21] ah that's extremely nice [09:57:57] okay, i think we're good [09:58:53] so would the thing to do be to repool first, test again from my connection, and then proceed to re-enable puppet on the cp nodes? [10:03:09] vgutierrez: ^ [10:03:29] hmmm [10:03:37] repooling won't change anything but sure [10:03:55] it won't? i'm not sure i understand [10:04:04] in how ATS behaves I mean [10:04:22] if locally it was ok, from your connection should be fine as well [10:04:32] just double check that after repooling cp3075 you get the same cp node [10:04:40] oh, right, sounds good [10:05:54] yup, that seems fine, proceeding to re-enable puppet [10:05:58] thanks very much for the help! [10:06:16] np [10:17:30] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance, 06MediaWiki-Core-Platform-Team (Kanban), 13Patch-For-Review: Implement Varnish URL rewriting for internal ESI fragment endpoint - https://phabricator.wikimedia.org/T443703#12580817 (10Vgutierrez) as discussed with @Catrope yesterday, mediawiki won't set a... [10:20:14] 06Traffic: Upgrade to HAProxy 3.2.27 - https://phabricator.wikimedia.org/T445475#12580823 (10Vgutierrez) [10:36:01] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12580936 (10ayounsi) Is that an ongoing issue? I noticed a bit of discarded packets in the past, most likely due to burst saturation (the server only have a 1G NIC)... [10:43:49] 06Traffic: Upgrade to HAProxy 3.2.27 - https://phabricator.wikimedia.org/T445475#12580966 (10Vgutierrez) [10:50:26] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12581012 (10Fabfur) a:03Fabfur [11:12:26] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12581185 (10Michael) >>! In T444597#12580936, @ayounsi wrote: > Is that an ongoing issue? I noticed a bit of discarded packets in the past, most likely due to burst... [12:30:09] o/ i have some more nrpe puppet cleanup that affects lvs, not urgent but would appreciate double-checks that i'm not missing anything obvious for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1370493 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1370494 [12:38:48] looks good, thanks! [13:05:33] 06Traffic: Upgrade to HAProxy 3.2.27 - https://phabricator.wikimedia.org/T445475#12582068 (10Vgutierrez) [13:05:40] 06Traffic: Upgrade to HAProxy 3.2.27 - https://phabricator.wikimedia.org/T445475#12582070 (10Vgutierrez) 05In progress→03Resolved a:03Vgutierrez [13:15:38] FIRING: [6x] LVSRealserverMSS: Unexpected MSS value on 185.15.59.224:443 @ cp3076 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=esams&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [13:20:38] RESOLVED: [6x] LVSRealserverMSS: Unexpected MSS value on 185.15.59.224:443 @ cp3076 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=esams&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [13:27:19] ty! [15:06:47] sukhe: is there a host I could play with to test a fix for https://phabricator.wikimedia.org/T434806 ? [15:07:32] XioNoX: feel free to depool dns7001. confctl select 'name=dns7001.*' set/pooled=no [15:10:51] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780#12582804 (10ayounsi) > At this time, I cannot confirm whether a fix is planned for the 24.4R2-Sx branch or whether the recommended Junos release f... [15:11:09] sukhe: thx, I'll try to get to it tomorrow, after a depool I can reboot it, right? [15:11:37] yep, once depooled for everything, it's all up for experimentation :) [15:11:53] before repooling, you can follow https://wikitech.wikimedia.org/wiki/DNS#DNS_Host_Maintenance [15:11:56] or ask us here [15:12:02] I was going to ask, perfect [15:12:03] thx! [15:12:23] thanks for fixing it. not a big deal but nice to have to not worry about it [15:13:17] yup [15:32:46] 06Traffic, 10MediaWiki-REST-API, 10RESTBase, 10RESTBase-Sunsetting, and 4 others: Mark Mathoid endpoints as deprecated - https://phabricator.wikimedia.org/T431372#12582997 (10Krinkle) Despite the default having changed in MediaWiki 1.43.0 (released Dec 2024), there's a trickle of sites running a curren... [16:00:29] 06Traffic, 10Wikidata, 10Wikidata-Query-Service: WDQS HTTP 403 — request for access review for ZUNQELU - https://phabricator.wikimedia.org/T443372#12583323 (10ssingh) Hi. Thanks for the report. Can I please ask you to reach out to bot-traffic@wikimedia.org with this request? The reason is that we coordinate... [16:05:31] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12583390 (10Eevans) >>! In T444597#12580936, @ayounsi wrote: > Is that an ongoing issue? I noticed a bit of discarded packets in the past, most likely due to burst... [16:13:12] 06Traffic, 07Sustainability (Incident Followup): Add option to depool DNS hosts when DNS depooling a site - https://phabricator.wikimedia.org/T440168#12583455 (10ssingh) This has been done above in the CR but I am not very happy with (my own) solution after hearing some feedback. So keeping this open for now. [16:14:11] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12583462 (10ssingh) @Dzahn : Thanks for taking care of these. Is this something you would like to handle yourself or you prefer Traffic would do it? (I am asking to be clear.) [17:04:01] FIRING: PurgedHighBacklogQueue: Large backlog queue for purged on cp6003:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=drmrs%20prometheus/ops&var-instance=cp6003 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [17:04:15] uh [17:13:13] it should recover soon(TM) [17:14:01] RESOLVED: [2x] PurgedHighBacklogQueue: Large backlog queue for purged on cp6003:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=drmrs%20prometheus/ops&var-instance=cp6003 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [17:14:14] vgutierrez: you should get a lottery ticket today [18:51:55] 06Traffic, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12584272 (10ssingh) This should now be resolved. Thanks to @Fabfur for working on it and @Vgutierrez for review... [18:55:51] 06Traffic, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12584280 (10ssingh) 05Open→03Resolved [19:26:48] 10netops, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 06Infrastructure-Foundations, and 2 others: Slow connections from Cloud VPS to cdn.cypress.io make CI builds take up to 50 minutes - https://phabricator.wikimedia.org/T445584 (10pwangai) 03NEW [20:32:14] 10netops, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 06Infrastructure-Foundations, and 2 others: Slow connections from Cloud VPS to cdn.cypress.io make CI builds take up to 50 minutes - https://phabricator.wikimedia.org/T445584#12584573 (10pwangai) 05Open→03Invalid CI seems to be healing... [20:58:24] 06Traffic, 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: Let kapow_estimate be available in Turnilo and datalake - https://phabricator.wikimedia.org/T445104#12584667 (10JAllemandou) I had to revert the change due to the fact that the field is not encoded as a number in the...