[06:22:28] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12144395 (10Bbbub) Hi @ssingh: In our case I reached out on 29.4.2026 to bot-traffic@wikimedia.org and haven't heard back yet. [06:42:44] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12144433 (10Bbbub) In the meantime I've put a 100k+ of our most frequently used images from wikimedia on a CDN to mitigate this issue for our users. I'd pre... [09:44:21] fabfur: good morning! Effie mentioned that you might have some time today to help us submit https://gerrit.wikimedia.org/r/c/operations/dns/+/1311856, could we set up a time to do that? [09:47:47] bjensen: lgtm, do you know the procedure ? [09:49:12] fabfur: no, i'm not familiar [09:49:22] fabfur: we have drafted the patches, they are ready for the LVS dance [09:49:38] so you may review and merge [09:49:54] the ticket is for the DNS change only, do you have the other ones? [09:49:57] thanks effie! :D i wasn't sure if there were other steps [09:50:48] the follow up patches are linked off of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1311859 [09:51:39] I added fab as reviewer on #3 and #4, I missed #2, I sorted it out [09:52:34] bjensen: the procedure is described here: https://wikitech.wikimedia.org/wiki/LVS#Remove_a_load_balanced_service [09:52:54] fabfur: thanks :) [09:54:39] effie: wasn't this scheduled for this afternoon along w/ suk.he ? [09:56:52] fabfur: my understanding was that sometime during your early afternoon would work [09:57:40] if you're in a rush we can do this now, or alternatively wait for some hours when other from Traffic will join [09:57:53] bjensen: all good w/ the procedure ? [09:58:15] fabfur: since traffic will be merging and doing the LVS dance, just let us know what time works [09:58:23] yes, no questions on my end [09:58:44] ok we can start then with the dns [10:00:13] I'd say let's merge the DNS one and see if someone screams :) [10:02:40] is the service silenced already? [10:04:33] one moment, currently looking into silencing [10:12:44] bjensen: what about actually deploying this around 13.00 UTC ? is that good for you? [10:13:07] there will be more people around to help in case something goes south [10:14:25] yes, that time works for me, thanks [10:14:38] 👍 [10:15:05] we'll do the procedure together given it's your first time with something like that [10:15:21] so in the meantime if you have questions or doubts about the procedure pls ask! [11:55:23] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12145480 (10Nylki) >>! In T413570#12143858, @ssingh wrote: >>>! In T413570#12135347, @Nylki wrote: >> @ssingh Sorry for another ping, but as bbub, I was won... [13:19:54] shall we proceed with the change? [13:20:19] bjensen: yes go ahead please [13:20:23] let us know if we can help [13:21:52] sukhe: will traffic merge https://gerrit.wikimedia.org/r/c/operations/dns/+/1311856, or should I do that? [13:23:23] bjensen: no preference from our side. if you want to do this so that you can do it again, please feel free to [13:23:26] if not, just ask us [13:23:33] ah, okay [13:23:48] we offer both options since some people want to do it themselves, others don't want to :) [13:23:54] some help would be great, i don't want to break things :D [13:24:05] unlikely but sure. should I go ahead? [13:24:11] yes please! [13:24:57] if this does cause alerts for api-gateway, i intend to silence them [13:25:07] it might [13:25:12] https://wikitech.wikimedia.org/wiki/LVS#1._Silence_network_probes [13:25:19] generally we recommend the silencing as the first step [13:25:49] right - i spent a fair bit of time this morning looking through docs to find out how to silence for this service, but wasn't able to find anything [13:26:58] cccccbefefillgbninkglnlidbvjkvlfecgllhffkthf [13:27:00] argh [13:27:09] bjensen: done [13:27:14] next step lvs_setup [13:27:48] sukhe: thanks! i'll proceed to merge that [13:27:57] bjensen: oh ok, I just did [13:28:01] oh you've got it grand [13:28:21] yeah let me know if you want to do some of the next steps or otherwise I will just merge [13:28:33] i'd be happy to take it from here [13:28:45] awesome [13:28:48] ping us if we can help :) [13:28:53] will do, thank you! [13:28:54] https://wikitech.wikimedia.org/wiki/LVS#3._Remove_network_probes_/_monitoring [13:29:00] you need to run agent on A:dnsbox [13:29:06] as the continuation step [13:29:28] will do - has puppet-merge already been run? [13:29:35] yep [13:29:38] groovy [13:35:43] okay, puppet agent has been run on dnsboxes [13:36:12] next step is submitting and merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1311860, iiuc? [13:36:47] yes [13:36:51] it involves pybal restarts so let us know [13:37:14] the docs also mention "remove the service stanza from profile::lvs::realserver::pools", which is not a thing i recall doing [13:37:37] is that in puppet somewhere? [13:38:46] not required for this, since this is a k8s service [13:38:54] ah, okay, cool [13:38:55] 06Traffic, 06Data-Engineering, 06Data-Persistence: Data Lake - mediarequests - adapt HQL query to include counting mediarequests from webrequest_text - https://phabricator.wikimedia.org/T431475#12146002 (10Ottomata) a:03Ahoelzl [13:40:44] the change you did for hieradata/role/common/kubernetes/worker.yaml is what we need [13:40:48] (and you did that already) [13:42:37] right on, thanks! okay, so that's been merged, and puppet-agent has been run on the lvs balancers [13:42:56] assistance with the next parts would be great [13:44:45] bjensen: both fabrizio and I are in a meeting now [13:44:52] we can take it ~20 or so [13:44:54] does that sound fine? [13:45:13] ah, understood, that should work [13:45:27] I will ping you when done, sorry [13:48:48] i should probably have asked about timing before starting, apologies - the docs do indicate that this bit causes alerts (though they don't seem to be paging alerts, at least) [13:49:18] no worries, we will take care of that [14:21:04] cccccbefefilgcthekufljknnderedbfnclvjulevbek [14:21:08] wow twice in one day [14:21:45] bjensen: sorry, almost done [14:21:48] our 1:1 went over a bit [14:21:54] no worries! [14:22:30] back [14:22:44] let me check where we left [14:22:46] back me too if you need help [14:22:48] bjensen: fwiw that functionality can be disabled on an yubikey, or be moved behind a long tap [14:23:16] taavi: ahhh, thanks, i'll look into it [14:23:43] sukhe: at the moment, i believe we're here: https://wikitech.wikimedia.org/wiki/LVS#4._Remove_the_service_from_the_load-balancers_and_the_backend_servers [14:23:57] and puppet agent has been run on the balancers [14:24:04] yep, restarting pybal [14:31:05] bjensen: restarts done [14:31:24] we now need to run the ipvsadm commands to delete the VIPs [14:31:29] do you want to come up with those? [14:31:56] hm, those i am very unsure about [14:33:06] no worries [14:33:26] ipvsadm --delete-service --tcp-service 10.2.2.55:8087 for eqiad [14:33:33] ipvsadm --delete-service --tcp-service 10.2.1.55:8087 for codfw [14:33:39] can you confirm that from the service definition? [14:33:44] essentially, we are deleting the VIP and port [14:34:01] ah i was just about to ask :D so the ips and ports which are in service.yaml [14:34:36] yes, that looks right to me [14:35:05] ok [14:35:26] so we basically run those on the LVSes to remove them from IPVS since the service has been turned down [14:35:39] running those [14:41:11] bjensen: all done [14:41:21] checking a few things before we move on [14:42:10] sounds good, thanks [14:42:24] ok looks good, everything recovered [14:42:29] you can merge the last patch [14:42:32] that's baout it [14:42:33] *about [14:43:28] great, thanks sukhe and fabfur! i appreciate your patience with my new-ness :) [14:43:59] bjensen: no worries :) [14:44:48] thanks to sukhe who stepped in! [14:46:52] that last patch has now been merged as well [14:47:50] cool, that should be it then [17:33:45] 06Traffic, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 2 others: codfw: rack B8 maintenance - Thursday July 23rd 14:00 UTC - https://phabricator.wikimedia.org/T430929#12147336 (10CWilliams-WMF) [18:00:52] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12147454 (10ssingh) >>! In T413570#12145480, @Nylki wrote: >>>! In T413570#12143858, @ssingh wrote: >>>>! In T413570#12135347, @Nylki wrote: >>> @ssingh Sor... [18:02:38] 06Traffic, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 2 others: codfw: rack B8 maintenance - Thursday July 23rd 14:00 UTC - https://phabricator.wikimedia.org/T430929#12147459 (10ssingh) [18:03:08] 06Traffic, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 2 others: codfw: rack B8 maintenance - Thursday July 23rd 14:00 UTC - https://phabricator.wikimedia.org/T430929#12147460 (10ssingh) [18:18:43] 06Traffic: Provide a way to allow firewall to be bypassed - https://phabricator.wikimedia.org/T432510#12147522 (10andrea.denisse) Hi, I tried this with ` https://upload.wikimedia.org/wikipedia/commons/thumb/f/f8/Long_Island_duck%2C_Long_Island%2C_New_York_LCCN2017708800_%28cropped%29.jpg/3840px-Long_Island_duck...