[08:06:53] 10netops, 06Infrastructure-Foundations, 06SRE: cloudsw1-d5-eqiad instability Aug 6 2024 - https://phabricator.wikimedia.org/T371879#10050198 (10dcaro) That's to be expected when moving lots of data around, we can try to be smarter and/or limit the backfill throughput (and/or add QoS, coming soon!), but it's... [11:58:40] 10netops, 06Infrastructure-Foundations, 06SRE: cloudsw1-d5-eqiad instability Aug 6 2024 - https://phabricator.wikimedia.org/T371879#10050758 (10cmooney) >>! In T371879#10049699, @Dzahn wrote: > We got paged at 20:19 UTC for "primary outbound port utilisation over 80%" on both cloudsw1-d5 and cloudsw1-f4 toda... [12:37:36] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Core router error logs: "sshd: Did not receive identification string" from prometheus hosts - https://phabricator.wikimedia.org/T368513#10050973 (10ayounsi) This went very well until it didn't. Changes fully rolled back. The cookbook chang... [12:43:10] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Core router error logs: "sshd: Did not receive identification string" from prometheus hosts - https://phabricator.wikimedia.org/T368513#10050996 (10fgiunchedi) Thank you @ayounsi for the write up! I agree with your preferred option, and arg... [14:25:05] 10netops, 06Infrastructure-Foundations, 06SRE: Add link from cloudsw1-e4-eqiad to cloudsw1-f4-eiqad - https://phabricator.wikimedia.org/T372061 (10cmooney) 03NEW p:05Triage→03Low [14:32:11] 10netops, 06Infrastructure-Foundations, 06SRE: Add link from cloudsw1-e4-eqiad to cloudsw1-f4-eiqad - https://phabricator.wikimedia.org/T372061#10051491 (10cmooney) [14:40:48] 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 10Release-Engineering-Team (Radar): implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10051529 (10Jelto) [14:46:27] 10netops, 06Infrastructure-Foundations, 06SRE: Add link from cloudsw1-e4-eqiad to cloudsw1-f4-eiqad - https://phabricator.wikimedia.org/T372061#10051545 (10cmooney) [16:34:54] 06Traffic, 10MediaWiki-User-login-and-signup: HTTP 429 error on collabwiki when creating user accounts - https://phabricator.wikimedia.org/T371732#10051988 (10JLam-WMF) Hiya, still getting this error. Here's the latest one: {F57178231} [16:40:51] 06Traffic: Move ncmonitor ignorelist over to configuration parameter - https://phabricator.wikimedia.org/T372076 (10BCornwall) 03NEW [16:58:45] 06Traffic, 10MediaWiki-User-login-and-signup: HTTP 429 error on collabwiki when creating user accounts - https://phabricator.wikimedia.org/T371732#10052136 (10DSeyfert_WMF) Hi all, this error is becoming a blocker for ITS as we're unable to complete onboarding and offboarding processes without addressing Colla... [17:03:51] 06Traffic, 10MediaWiki-User-login-and-signup: HTTP 429 error on collabwiki when creating user accounts - https://phabricator.wikimedia.org/T371732#10052152 (10JLam-WMF) My Collab Wiki username is JLam-WMF, fyi. [17:34:48] 06Traffic, 13Patch-For-Review: Move ncmonitor ignorelist over to configuration parameter - https://phabricator.wikimedia.org/T372076#10052275 (10BCornwall) 05Open→03In progress p:05Triage→03Medium [17:35:34] 06Traffic, 10MediaWiki-User-login-and-signup: HTTP 429 error on collabwiki when creating user accounts - https://phabricator.wikimedia.org/T371732#10052288 (10ssingh) Hi folks: Can you describe the workflow a bit? Do you create a collab account by going to https://collab.wikimedia.org/wiki/Special:CreateAccou... [17:36:29] 06Traffic, 10MediaWiki-User-login-and-signup, 05WMF-NDA: HTTP 429 error on collabwiki when creating user accounts - https://phabricator.wikimedia.org/T371732#10052291 (10ssingh) [17:37:38] yeah I forgot that doesn't help [17:39:10] 06Traffic, 10MediaWiki-User-login-and-signup: HTTP 429 error on collabwiki when creating user accounts - https://phabricator.wikimedia.org/T371732#10052293 (10ssingh) [19:23:55] 06Traffic, 10conftool, 06Tech-Docs-Team, 07Documentation: Create new content structure for requestctl tool documentation - https://phabricator.wikimedia.org/T372095 (10TBurmeister) 03NEW [19:25:38] 06Traffic, 10conftool, 06Tech-Docs-Team, 07Documentation: Create new content structure for requestctl tool documentation - https://phabricator.wikimedia.org/T372095#10052604 (10TBurmeister) p:05Triage→03Medium [20:05:19] 06Traffic: ncmonitor: Initialize tld suffix list on resource creation - https://phabricator.wikimedia.org/T372103 (10BCornwall) 03NEW [20:05:35] 06Traffic: ncmonitor: Initialize tld suffix list on resource creation - https://phabricator.wikimedia.org/T372103#10052764 (10BCornwall) p:05Triage→03Low