[09:19:06] I see there was some fun in eqsin while I was eating oysters and sipping cask ales [09:22:09] 10Traffic, 10Operations: ATS production-ready as a backend cache layer - https://phabricator.wikimedia.org/T207048 (10ema) [09:22:12] 10Traffic, 10Operations, 10Patch-For-Review: Define and deploy Icinga checks for ATS backends - https://phabricator.wikimedia.org/T204209 (10ema) [09:23:15] 10Traffic, 10Operations: ATS production-ready as a backend cache layer - https://phabricator.wikimedia.org/T207048 (10ema) [09:23:18] 10Traffic, 10Operations, 10Patch-For-Review: Package and deploy ATS v8.x - https://phabricator.wikimedia.org/T204232 (10ema) [09:24:19] 10Traffic, 10Operations: ATS: log inspection at runtime - https://phabricator.wikimedia.org/T204225 (10ema) [09:24:22] 10Traffic, 10Operations: ATS production-ready as a backend cache layer - https://phabricator.wikimedia.org/T207048 (10ema) [09:39:46] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596 (10aborrero) >>! In T174596#4681328, @ayounsi wrote: > This is also the reason we have to have the... [11:18:33] 10Traffic, 10Operations, 10Patch-For-Review: Define and deploy Icinga checks for ATS backends - https://phabricator.wikimedia.org/T204209 (10ema) [13:21:43] 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review, 10User-Joe: Re-organize the apache configuration for MediaWiki in puppet - https://phabricator.wikimedia.org/T196968 (10Joe) 05Open>03Resolved [13:22:14] 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review, 10User-Joe: Re-organize the apache configuration for MediaWiki in puppet - https://phabricator.wikimedia.org/T196968 (10Joe) This is now fully done in both deployment-prep (beta) and production. [13:39:15] 10Traffic, 10Operations: Varnish won't purge thumbnails of specific file - https://phabricator.wikimedia.org/T207615 (10Gilles) a:03BBlack [14:08:42] 10Traffic, 10Operations, 10Patch-For-Review: Define and deploy Icinga checks for ATS backends - https://phabricator.wikimedia.org/T204209 (10ema) [14:25:15] 10netops, 10Operations, 10Goal: Increase network capacity (2018-19 Q1 Goal) - https://phabricator.wikimedia.org/T199142 (10ayounsi) 05Open>03Invalid Marking it as invalid as we did not reach that goal fully: Done: - eqiad: 1 rows with 3*10G racks - ulsfo: replace routers - eqdfw: replace router Not d... [14:28:06] afternoon vgutierrez [14:39:10] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596 (10Krenair) >>! In T174596#4685019, @aborrero wrote: > * I don't really understand what means the c... [14:42:45] 10netops, 10Cloud-Services, 10Operations: Renumber cloud-instance-transport1-b-eqiad to public IPs - https://phabricator.wikimedia.org/T207663 (10ayounsi) p:05Triage>03Normal [14:47:57] 10netops, 10Cloud-Services, 10Operations: Consider renumbering Labs to separate address spaces - https://phabricator.wikimedia.org/T122406 (10ayounsi) Here are the tasks for the record: > - Deprecation of the old space, which is equivalent at this point to deprecation of nova-network. This is already planned... [15:57:36] 10Traffic, 10Varnish, 10Analytics, 10Operations, 10User-Elukey: Sort out analytics service dependency issues for cp* cache hosts - https://phabricator.wikimedia.org/T128374 (10Milimetric) The fix for this would be high risk and low gain. So keeping around to just have context in case the problem does ma... [16:00:56] 10Traffic, 10netops, 10Operations, 10Goal: Increase network capacity (2018-19 Q2 Goal) - https://phabricator.wikimedia.org/T207668 (10ayounsi) p:05Triage>03Normal [16:03:40] 10netops, 10Operations, 10fundraising-tech-ops: add icinga1001 to send_nsca and pfw rules in FRACK - https://phabricator.wikimedia.org/T207175 (10ayounsi) 05Open>03Resolved [16:11:28] 10netops, 10Cloud-Services, 10Operations: Renumber cloud-instance-transport1-b-eqiad to public IPs - https://phabricator.wikimedia.org/T207663 (10aborrero) I don't see any problem with this from the top of my head. I would ask @chasemp to see if he can see any issue with this new setting. Please note that d... [16:46:30] 10netops, 10Cloud-Services, 10Operations: Renumber cloud-instance-transport1-b-eqiad to public IPs - https://phabricator.wikimedia.org/T207663 (10chasemp) No technical blockers to this VLAN have public IPs that I know of. Agreed that the switchover would be difficult to make transparent to users. It's poss... [17:26:49] 10netops, 10Operations, 10ops-eqiad: Interface errors on cr2-eqiad:xe-4/0/0 - https://phabricator.wikimedia.org/T203719 (10Cmjohnson) I replaced the optics on asw-a-eqiad:xe-2/1/2 {#3455} and cleaned the fiber. [17:31:57] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596 (10ayounsi) Talked to Arturo on IRC, replying to my own questions. I thought dmz_cidr were only the... [17:34:14] 10netops, 10Cloud-VPS, 10Operations, 10cloud-services-team (Kanban): dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596 (10faidon) >>! In T174596#4685786, @Krenair wrote: >>>! In T174596#4685019, @aborrero wrote: >> * I... [17:54:17] 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review, 10User-Joe: Re-organize the apache configuration for MediaWiki in puppet - https://phabricator.wikimedia.org/T196968 (10cscott) w00t! Now we can do https://gerrit.wikimedia.org/r/368248 (T117845) ? [18:55:26] 10netops, 10Cloud-Services, 10Operations: Renumber cloud-instance-transport1-b-eqiad to public IPs - https://phabricator.wikimedia.org/T207663 (10ayounsi) > Out of curiosity, I would expect some shortage of public IPv4 addressing, is not the case? We need to be careful with our public IPs indeed, but this is... [19:00:18] 10netops, 10Operations, 10fundraising-tech-ops: add icinga1001 to send_nsca and pfw rules in FRACK - https://phabricator.wikimedia.org/T207175 (10Dzahn) @cwdent has the send_nsca part also been done? [19:19:49] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10ayounsi) We currently have the following relevant vlans: - private: 10/8 IPs, not reachable from cloud instances and the... [19:20:36] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10ayounsi) a:05ayounsi>03Ottomata [19:24:02] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10faidon) How many servers are we talking about both right now, as well as in the mid-term e.g. in the next year or two? H... [19:35:24] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Ottomata) Right now, 8: 3 ganeti instances and 5 bare metal worker nodes. We wouldn't be adding more nodes in within thi... [19:36:34] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Ottomata) Where are the labsdb hosts going to live if they are being moved out of the labs-support VLAN? Likely these cl... [20:26:23] 10netops, 10Cloud-Services, 10Operations: Renumber cloud-instance-transport1-b-eqiad to public IPs - https://phabricator.wikimedia.org/T207663 (10faidon) It's sad to hear that's a major disruption :( Would it make sense to do this now when it's early in the migration and relatively few projects have migrated... [21:32:56] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Operations, 10Patch-For-Review: Add punjabi.wikimedia.org to DNS and Apache - https://phabricator.wikimedia.org/T207583 (10Krenair) It's the Sat, Oct 20, 16:58 UTC comment in the parent task [21:34:18] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Operations, 10Patch-For-Review: Add punjabi.wikimedia.org to DNS and Apache - https://phabricator.wikimedia.org/T207583 (10Dzahn) [21:34:36] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Operations, 10Patch-For-Review: Add punjabi.wikimedia.org to DNS and Apache - https://phabricator.wikimedia.org/T207583 (10Dzahn) [21:34:53] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Operations, 10Patch-For-Review: Add punjabi.wikimedia.org to DNS and Apache - https://phabricator.wikimedia.org/T207583 (10Dzahn) p:05Triage>03Normal [21:38:35] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Operations, 10Patch-For-Review: Add punjabi.wikimedia.org to DNS and Apache - https://phabricator.wikimedia.org/T207583 (10Dzahn) thanks! edited ticket description, added comment on Gerrit, deployed, added to DNS [21:39:39] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Operations, 10Patch-For-Review: Add punjabi.wikimedia.org to DNS and Apache - https://phabricator.wikimedia.org/T207583 (10Dzahn) a:03Dzahn