[09:23:46] 10Traffic, 10Varnish, 10Continuous-Integration-Infrastructure, 10Operations, 10Patch-For-Review: Make CI run Varnish VCL tests - https://phabricator.wikimedia.org/T128188 (10ema) >>! In T128188#5143265, @ema wrote: > @hashar can we run it in CI? The catalog needs to be compiled against a given hostname.... [09:23:58] 10Traffic, 10Varnish, 10Continuous-Integration-Infrastructure, 10Operations, 10Patch-For-Review: Make CI run Varnish VCL tests - https://phabricator.wikimedia.org/T128188 (10ema) a:05ema→03None [09:58:01] 10Traffic, 10Operations: check_trafficserver_log_fifo: false positives when changing log format - https://phabricator.wikimedia.org/T248067 (10ema) [09:58:10] 10Traffic, 10Operations: check_trafficserver_log_fifo: false positives when changing log format - https://phabricator.wikimedia.org/T248067 (10ema) p:05Triage→03Medium [10:15:27] 10netops, 10Operations, 10observability: Migrate role::netmon to Buster - https://phabricator.wikimedia.org/T247967 (10fgiunchedi) [11:21:48] 10netops, 10Operations, 10ops-esams: 2*10G optics down on cr2-esams - https://phabricator.wikimedia.org/T245520 (10ayounsi) Replacing the optics on both sides didn't help, and light levels are correct. Service Request ID 2020-0319-0197 has been created. [12:59:04] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10aborrero) I hope what @ayounsi said help clarify the situation @faidon. Some additional info about the setup we tested can be found here: https://wikite... [13:31:50] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: IPv6 early PoC - https://phabricator.wikimedia.org/T245495 (10aborrero) [13:55:45] 10netops, 10DC-Ops, 10Operations: Juniper network device audit - all sites - https://phabricator.wikimedia.org/T213843 (10faidon) >>! In T213843#5913332, @ayounsi wrote: > Ok! From https://wikitech.wikimedia.org/wiki/Server_Lifecycle#States I thought that if a device was not in netbox it was not in our posse... [14:56:04] 10netops, 10Operations: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929 (10aborrero) [14:56:07] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: IPv6 early PoC - https://phabricator.wikimedia.org/T245495 (10aborrero) [16:00:07] 10Wikimedia-Apache-configuration, 10Operations, 10serviceops, 10Patch-For-Review: Build a black-box httpd testing framework - https://phabricator.wikimedia.org/T236699 (10RLazarus) 05Open→03Resolved Marking this done: https://gerrit.wikimedia.org/r/581616 deleted apache-fast-test, as httpbb is now comp... [16:04:45] 10netops, 10Operations: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929 (10aborrero) I like option 2) the most. Are those ranges actual data? Regarding coding the vlan id: I don't think we should do it. We might eventually move away from the prod VLAN thing, or have addresses where the VLAN par... [17:31:11] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: IPv6 early PoC - https://phabricator.wikimedia.org/T245495 (10ayounsi) The steps I have in mind here are: 1/ Setup v6 on the transport network (most likely `2620:0:860:fe0a::/64`) 2/ Assign a /48 for cloud codfw, see T187929 (Here we won't go... [17:55:50] hi, can i finish decom of cp1099? wanna review? -> https://gerrit.wikimedia.org/r/q/topic:%2522decom-eqiad%2522+(status:open+) [17:56:01] just ran across it when removing mw stuff to reduce power usage [17:56:24] seems like basically decom'ed just needs the actual cookbook and removal [18:02:43] 10netops, 10Operations: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929 (10ayounsi) a:03faidon I agree that option 2 is the way to go. The complication is how to subnet them properly for both the short term (T245495 PoC) and the longer term. I couldn't find much subnetting recommendation doc... [18:13:27] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: IPv6 early PoC - https://phabricator.wikimedia.org/T245495 (10aborrero) Mostly agree with everything, some inlined comments >>! In T245495#5984709, @ayounsi wrote: > The steps I have in mind here are: > 1/ Setup v6 on the transport network (m... [18:22:31] 10netops, 10Operations: eqord - ulsfo Telia link down - IC-313592 - https://phabricator.wikimedia.org/T221259 (10Dzahn) 05Open→03Resolved Wed, 18 Mar 2020 01:09 : Our technician performed cleaning to FMP in Denver and the issue resolved , the traffic is up now . apologize for the inconvenience caused.... [18:22:33] 10netops, 10Operations: eqord - ulsfo Telia link down - IC-313592 - https://phabricator.wikimedia.org/T221259 (10Dzahn) Icinga is all green again. [18:55:38] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Vojtech.dostal) This is also extremely important for OpenRefine users. We cannot push our reconciliatio... [18:59:04] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Ladsgroup) >>! In T243701#5973374, @Dvorapa wrote: > Any news? From possible solutions like T238751, T2... [19:16:11] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Dvorapa) Is there any short-term temporary solution until RFC gets done? Perhaps hardcode maxlag=4? Or... [21:42:50] 10Traffic, 10Operations, 10Product-Infrastructure-Team-Backlog: Increase in 503 responses to POST reqs to api.php since 2020-03-15 - https://phabricator.wikimedia.org/T248132 (10Mholloway) [21:48:52] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Ladsgroup) >>! In T243701#5985113, @Dvorapa wrote: > Is there any short-term temporary solution until R... [21:55:23] 10Traffic, 10Operations, 10Product-Infrastructure-Team-Backlog: Increase in 503 responses to POST reqs to api.php since 2020-03-15 - https://phabricator.wikimedia.org/T248132 (10Mholloway) Actually, this appears nearly identical even without filtering for POST or api.php. [21:57:48] 10Traffic, 10Operations, 10Product-Infrastructure-Team-Backlog: Increase in 503 responses since 2020-03-15 - https://phabricator.wikimedia.org/T248132 (10Mholloway) [21:58:15] 10Traffic, 10Operations, 10Product-Infrastructure-Team-Backlog: Increase in 503 responses since 2020-03-15 - https://phabricator.wikimedia.org/T248132 (10Mholloway)