[03:39:14] 10Traffic, 10Operations, 10observability: prometheus2004 not scraping lvs2007 & lvs2008 - https://phabricator.wikimedia.org/T246860 (10CDanis) [03:39:32] 10Traffic, 10Operations, 10observability: prometheus2004 not scraping lvs2007 & lvs2008 - https://phabricator.wikimedia.org/T246860 (10CDanis) p:05Triage→03High [03:44:25] 10Traffic, 10Operations, 10observability: prometheus2004 not scraping lvs2007 & lvs2008 - https://phabricator.wikimedia.org/T246860 (10CDanis) Target definitions as expanded by puppet are identical on both servers too. Very weird. [06:27:16] 10Traffic, 10Operations, 10observability: prometheus2004 not scraping lvs2007 & lvs2008 - https://phabricator.wikimedia.org/T246860 (10CDanis) 05Open→03Resolved a:03CDanis netstat confirmed that prom2004 wasn't even trying {P10604} A restart fixed it. I have no idea. [06:27:44] 10Traffic, 10Operations, 10observability: prometheus2004 not scraping lvs2007 & lvs2008 - https://phabricator.wikimedia.org/T246860 (10Vgutierrez) tcpdump & netstat confirmed that prometheus2004 wasn't even trying to connect to lvs2007|8:9100, a restart has fixed it. [06:38:32] 10Traffic, 10Operations, 10observability: prometheus2004 not scraping lvs2007 & lvs2008 - https://phabricator.wikimedia.org/T246860 (10CDanis) Oh, also NB that a simple `reload` was **not** sufficient to fix (tried that first). [09:24:11] 10Traffic, 10Operations, 10observability: prometheus2004 not scraping lvs2007 & lvs2008 - https://phabricator.wikimedia.org/T246860 (10fgiunchedi) Curious indeed, thanks for investigating! I took a quick look and it looks like prometheus2004 didn't even know about lvs2007 before today at ~6.23, so I'm suspec... [09:39:15] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10aborrero) Also sent https://salsa.debian.org/openstack-team/services/neutron-dynamic-routing/-/merge_requests/2 [11:23:46] 10netops, 10Operations, 10fundraising-tech-ops: DHCP routing issue with civi2001 - https://phabricator.wikimedia.org/T246812 (10ayounsi) Configuration looks good on the switch and router. This will need to be live troubleshooted, when the client is sending DHCP requests. First step is to run `show ethernet... [11:44:37] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: introduce filtering for neutron BGP addresses - https://phabricator.wikimedia.org/T246887 (10aborrero) [13:07:48] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: introduce filtering for neutron BGP addresses - https://phabricator.wikimedia.org/T246887 (10aborrero) This is my initial proposal: `lang=diff diff -u router.org router.new --color --- router.org 2020-03-04 12:45:50.139827978 +0100 +++ route... [13:09:55] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: introduce filtering for neutron BGP addresses - https://phabricator.wikimedia.org/T246887 (10aborrero) a:03aborrero [14:49:10] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Xqt) >>! In T243701#5884801, @Ladsgroup wrote: > I have been thinking about this and I think I have a s... [15:56:20] 10netops, 10Operations, 10fundraising-tech-ops: DHCP routing issue with civi2001 - https://phabricator.wikimedia.org/T246812 (10ayounsi) a:03Papaul At bootup that MAC shows up on ge-1/0/9 ` ayounsi@fasw-c-codfw> show ethernet-switching table | match 4c:d9:8f:aa:77:b4 frack-payments-codfw 4c:d9:8f:aa:7... [16:24:13] 10Wikimedia-Apache-configuration, 10Commons, 10SDC General, 10WikibaseMediaInfo, and 4 others: Make /entity/ alias work for Commons - https://phabricator.wikimedia.org/T222321 (10TJones) 05Open→03Resolved [16:52:45] 10Traffic, 10Cleanup, 10Operations, 10Release-Engineering-Team-TODO, and 3 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10MusikAnimal) > [x] Remove references from the CentralAuth database. This item is checked but I still see it in `meta_p.wiki` on the Toolforge... [17:07:47] 10Traffic, 10Cleanup, 10Operations, 10Release-Engineering-Team-TODO, and 3 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Jdforrester-WMF) >>! In T238803#5941793, @MusikAnimal wrote: >> [x] Remove references from the CentralAuth database. > > This item is checked... [17:12:02] 10Traffic, 10Operations: switch to irate() instead of rate() for traffic graphs - https://phabricator.wikimedia.org/T246902 (10DannyS712) Guessing this is about traffic? [17:49:18] 10Traffic, 10Operations: switch to irate() instead of rate() for traffic graphs - https://phabricator.wikimedia.org/T246902 (10CDanis) Yes, but about Observability too. This is/was a placeholder task so I don't forget to write a proper description later :) [17:49:44] 10netops, 10Operations: eqiad 10G ports needs - https://phabricator.wikimedia.org/T190364 (10ayounsi) 05Open→03Resolved a:03ayounsi I don't think there is much value anymore for this task (it was for last year). The spreadsheet for next FY capex has a 10G column. [19:33:48] 10netops, 10Operations, 10fundraising-tech-ops: DHCP routing issue with civi2001 - https://phabricator.wikimedia.org/T246812 (10Papaul) @ayounsi thanks for the troubleshooting I will look into this tomorrow. [19:34:31] 10netops, 10Operations, 10fundraising-tech-ops: DHCP routing issue with civi2001 - https://phabricator.wikimedia.org/T246812 (10Papaul) p:05Triage→03Medium [20:21:05] 10Traffic, 10Operations, 10Phabricator: Phabricator is inaccessible from Egypt - https://phabricator.wikimedia.org/T246923 (10Aklapper) Hi, this might be intended if you use certain providers / IP addresses which were also used by a persistent vandal (non-public reference: T218589#5033515 ). [21:09:17] 10Traffic, 10Operations, 10Phabricator: Phabricator is inaccessible from Egypt - https://phabricator.wikimedia.org/T246923 (10Krenair) If it is that I would not expect HTTP 501 responses. [21:16:25] 10Traffic, 10Cleanup, 10Operations, 10Release-Engineering-Team-TODO, and 3 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10bd808) [21:17:22] 10Traffic, 10Cleanup, 10Operations, 10Release-Engineering-Team-TODO, and 3 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Jdforrester-WMF) [21:47:23] 10Traffic, 10Operations, 10Phabricator: Phabricator is inaccessible from Egypt - https://phabricator.wikimedia.org/T246923 (10ahmad) Shpuldn't this be a 403/Forbidden response? [21:48:44] 10Traffic, 10Operations, 10Phabricator: Phabricator is inaccessible from Egypt: HTTP 501 error - https://phabricator.wikimedia.org/T246923 (10Aklapper) [22:00:41] 10Traffic, 10Operations, 10Phabricator: Phabricator is inaccessible from Egypt: HTTP 501 error - https://phabricator.wikimedia.org/T246923 (10Urbanecm) >>! In T246923#5943004, @Krenair wrote: > If it is that I would not expect HTTP 501 responses. >>! In T246923#5943149, @ahmad wrote: > Shpuldn't this be a 4...