[00:54:13] 06Traffic: Discovery: Deprecation of TLS 1.2 - https://phabricator.wikimedia.org/T367821#10083440 (10Bugreporter) "able to view Wikimedia projects" means Grade C, not Grade A. [07:50:11] 06Traffic: liberica triggers one DNS query per HTTPCheck execution/realserver - https://phabricator.wikimedia.org/T372731#10083788 (10Vgutierrez) 05Open→03Resolved [08:44:31] 10netops, 06DBA, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate codfw row C & D database hosts to new Leaf switches - https://phabricator.wikimedia.org/T370852#10083924 (10ABran-WMF) after a quick chat with @cmooney, I've taken inventory of the 87 servers to handle: |**rack**|**node**|**clus... [09:07:30] 06Traffic: Remove katran blockers for low-traffic non-k8s based services - https://phabricator.wikimedia.org/T373020#10083984 (10Vgutierrez) [09:09:13] 06Traffic: Remove katran blockers for low-traffic non-k8s based services - https://phabricator.wikimedia.org/T373020#10083987 (10Vgutierrez) [09:29:31] 06Traffic: Remove katran blockers for low-traffic non-k8s based services - https://phabricator.wikimedia.org/T373020#10084042 (10Vgutierrez) [09:31:06] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 10MW-on-K8s, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10084045 (10Clement_Goubert) At first I thought it could be due to [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1052745 | thi... [09:44:05] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 10MW-on-K8s, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10084087 (10Ladsgroup) Maybe it's an urban myth but I always though if index.php/index.html exists in a directory, apache automatically... [10:00:38] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 10MW-on-K8s, 13Patch-For-Review, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10084137 (10Clement_Goubert) >>! In T373048#10084087, @Ladsgroup wrote: > Maybe it's an urban myth but I always th... [10:06:26] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 10MW-on-K8s, 13Patch-For-Review, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10084150 (10Clement_Goubert) p:05Triage→03Medium a:03Clement_Goubert [10:12:09] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 10MW-on-K8s, 13Patch-For-Review, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10084163 (10Clement_Goubert) As expected: ` cgoubert@cumin1002:~$ httpbb /srv/deployment/httpbb-tests/appserver/te... [10:28:47] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 10MW-on-K8s, 13Patch-For-Review, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10084194 (10Clement_Goubert) 05Open→03Resolved All good now `lang=html cgoubert@cumin1002:~$ httpbb /srv/d... [10:29:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: cr1-eqiad: disk failure - https://phabricator.wikimedia.org/T372781#10084199 (10ayounsi) Disk is gone : `name=show vmhost hardware re0 re0: [...] Item Capacity Part number... [10:31:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: cr1-eqiad: disk failure - https://phabricator.wikimedia.org/T372781#10084200 (10ayounsi) a:03ayounsi [11:19:20] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10084337 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [11:36:28] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs2011: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370891#10084369 (10cmooney) a:05cmooney→03None [11:41:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10084371 (10cmooney) [11:45:02] FIRING: [2x] PurgedHighEventLag: High event process lag with purged on cp2029:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [11:50:00] FIRING: [32x] PurgedHighEventLag: High event process lag with purged on cp2027:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [12:07:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw rack C1 from asw-c1-codfw to lsw1-c1-codfw - https://phabricator.wikimedia.org/T373095 (10cmooney) 03NEW p:05Triage→03Medium [12:08:51] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10084514 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [12:09:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw rack C1 from asw-c1-codfw to lsw1-c1-codfw - https://phabricator.wikimedia.org/T373095#10084515 (10cmooney) [12:09:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10084516 (10cmooney) [12:13:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks C2 & C3 from asw to lsw - https://phabricator.wikimedia.org/T373096 (10cmooney) 03NEW p:05Triage→03Medium [12:13:20] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks C2 & C3 from asw to lsw - https://phabricator.wikimedia.org/T373096#10084536 (10cmooney) [12:13:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10084537 (10cmooney) [12:16:43] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks C4 & C5 from asw to lsw - https://phabricator.wikimedia.org/T373097 (10cmooney) 03NEW p:05Triage→03Medium [12:16:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10084559 (10cmooney) [12:16:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks C4 & C5 from asw to lsw - https://phabricator.wikimedia.org/T373097#10084558 (10cmooney) [12:22:21] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10084589 (10Clement_Goubert) [12:25:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks C6 & C7 from asw to lsw - https://phabricator.wikimedia.org/T373101 (10cmooney) 03NEW p:05Triage→03Medium [12:25:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks C6 & C7 from asw to lsw - https://phabricator.wikimedia.org/T373101#10084619 (10cmooney) [12:25:53] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10084620 (10cmooney) [12:26:01] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10084622 (10Clement_Goubert) [12:26:13] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10084623 (10Clement_Goubert) [12:28:18] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D1 & D2 from asw to lsw - https://phabricator.wikimedia.org/T373102 (10cmooney) 03NEW p:05Triage→03Medium [12:28:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D1 & D2 from asw to lsw - https://phabricator.wikimedia.org/T373102#10084644 (10cmooney) [12:30:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D3 & D4 from asw to lsw - https://phabricator.wikimedia.org/T373103 (10cmooney) 03NEW p:05Triage→03Medium [12:31:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D3 & D4 from asw to lsw - https://phabricator.wikimedia.org/T373103#10084667 (10cmooney) [12:31:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10084668 (10cmooney) [12:36:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104 (10cmooney) 03NEW p:05Triage→03Medium [12:37:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10084691 (10cmooney) [12:37:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10084692 (10cmooney) [12:39:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D7 & D8 from asw to lsw - https://phabricator.wikimedia.org/T373105 (10cmooney) 03NEW p:05Triage→03Medium [12:40:00] FIRING: [34x] PurgedHighEventLag: High event process lag with purged on cp2027:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [12:40:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10084727 (10cmooney) [12:40:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D7 & D8 from asw to lsw - https://phabricator.wikimedia.org/T373105#10084726 (10cmooney) [12:45:00] FIRING: [64x] PurgedHighEventLag: High event process lag with purged on cp2027:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [12:50:00] FIRING: [15x] PurgedHighEventLag: High event process lag with purged on cp2027:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [12:55:00] FIRING: [7x] PurgedHighEventLag: High event process lag with purged on cp2027:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [13:00:00] FIRING: [4x] PurgedHighEventLag: High event process lag with purged on cp2028:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [13:01:20] 06Traffic: Error on publishing edits on wiki pages - https://phabricator.wikimedia.org/T373108#10084792 (10Bugreporter) Please send a screenshot of **whole webpage** instead of photograph. This may be something #Varnish related. Anyway such error message is not pretty. [13:04:48] 06Traffic: Error on publishing edits on wiki pages - https://phabricator.wikimedia.org/T373108#10084811 (10Bugreporter) See also: {T372473} [13:05:00] FIRING: [5x] PurgedHighEventLag: High event process lag with purged on cp2028:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [13:05:39] 06Traffic: Error message says "%error_body_content%" - https://phabricator.wikimedia.org/T371424#10084814 (10Bugreporter) 05Resolved→03Open See {T372473} and {T373108} [13:10:00] RESOLVED: [6x] PurgedHighEventLag: High event process lag with purged on cp2028:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [13:17:00] FIRING: PurgedHighBacklogQueue: Large backlog queue for purged on cp2039:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=codfw%20prometheus/ops&var-instance=cp2039 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [13:22:00] RESOLVED: PurgedHighBacklogQueue: Large backlog queue for purged on cp2039:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=codfw%20prometheus/ops&var-instance=cp2039 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [13:27:51] 06Traffic: Error on publishing edits on wiki pages - https://phabricator.wikimedia.org/T373108#10084876 (10Aklapper) Hi @Kuldeepburjbhalaike, thanks for taking the time to report this! For future reference, please include text as text, so text can be searched for and found. Text in images cannot be searched for. [13:28:13] 06Traffic: Error publishing edits on wiki pages: "%error_body_content%" - https://phabricator.wikimedia.org/T373108#10084877 (10Aklapper) [13:28:51] 06Traffic: Error publishing edits on wiki pages: "%error_body_content%" - https://phabricator.wikimedia.org/T373108#10084878 (10Aklapper) @Kuldeepburjbhalaike Which exact website(s) is this about? Please provide a full link. [13:34:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw rack C1 from asw-c1-codfw to lsw1-c1-codfw - https://phabricator.wikimedia.org/T373095#10084905 (10cmooney) [13:35:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks C2 & C3 from asw to lsw - https://phabricator.wikimedia.org/T373096#10084907 (10cmooney) [13:35:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks C2 & C3 from asw to lsw - https://phabricator.wikimedia.org/T373096#10084911 (10cmooney) [13:36:12] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks C4 & C5 from asw to lsw - https://phabricator.wikimedia.org/T373097#10084920 (10cmooney) [13:36:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks C6 & C7 from asw to lsw - https://phabricator.wikimedia.org/T373101#10084921 (10cmooney) [13:36:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D1 & D2 from asw to lsw - https://phabricator.wikimedia.org/T373102#10084925 (10cmooney) [13:36:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D3 & D4 from asw to lsw - https://phabricator.wikimedia.org/T373103#10084926 (10cmooney) [13:37:15] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D5 & D6 from asw to lsw - https://phabricator.wikimedia.org/T373104#10084928 (10cmooney) [13:37:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate servers in codfw racks D7 & D8 from asw to lsw - https://phabricator.wikimedia.org/T373105#10084929 (10cmooney) [14:09:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10085064 (10cmooney) [14:48:42] 06Traffic: Error message says "%error_body_content%" - https://phabricator.wikimedia.org/T371424#10085230 (10CDobbins) I'm looking into this now. Sorry that this wasn't resolved by the earlier fix. [15:29:19] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2013: move uplink to lsw1-c2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370927#10085381 (10cmooney) [15:30:04] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2013: move uplink to lsw1-c2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370927#10085386 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=4888a1d9-ee36-415c-a204-98c84040effe) set by... [15:30:14] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10085351 (10Papaul) 05Open→03Resolved a:03Papaul [15:30:25] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2013: move uplink to lsw1-c2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370927#10085387 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=028cbb12-db86-4824-9084-463287cc8911) set by... [15:50:49] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10085491 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host lvs201... [16:36:08] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10085669 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host lvs2013.co... [16:45:32] 06Traffic, 06SRE, 13Patch-For-Review: Disable acceptance of IPv6 router-advertisement on non-default LVS interface - https://phabricator.wikimedia.org/T358260#10085694 (10cmooney) If anyone is rebooting lvs hosts and has to deal with this the below shell script will make sure a working route is in use: ` #!/... [16:50:16] 06Traffic: Error publishing edits on wiki pages: "%error_body_content%" - https://phabricator.wikimedia.org/T373108#10085700 (10Kuldeepburjbhalaike) hi @Aklapper and @Bugreporter, i got this error while editing pages on pawikipedia, pawikisource and sawikipedia. sorry i have not screenshot of the full webpage wi... [16:52:27] 06Traffic: Error publishing edits on wiki pages: "%error_body_content%" - https://phabricator.wikimedia.org/T373108#10085718 (10ssingh) >>! In T373108#10085700, @Kuldeepburjbhalaike wrote: > hi @Aklapper and @Bugreporter, i got this error while editing pages on pawikipedia, pawikisource and sawikipedia. sorry i... [17:24:00] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2013: move uplink to lsw1-c2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370927#10085767 (10cmooney) a:05cmooney→03None All work completed, no issues to report. @Jhancock.wm @Papaul these two cros... [17:24:48] 06Traffic, 06Commons, 13Patch-For-Review: Error: 503, Backend fetch failed while editing Commons - https://phabricator.wikimedia.org/T372473#10085775 (10CDobbins) A patch to fix this was merged 30 minutes ago, though it hasn't yet been deployed. [18:57:05] or Monday but I leave that to you! [19:00:13] sukhe: if you're still around in 40' I think we'll be ready at that point. or if not maybe another US-timezone teammate as well [19:00:22] fwiw i've done this before although it's been at least a year since the last time I had to do it [19:00:59] sukhe: oh just saw the "back later but not in an hour" msg, if we're not rdy by the time you're leaving for daycare then that works for me as well, I'll be on pretty late [19:01:15] I am not sure if bret.t is around right now and hence [19:01:24] but yeah, just ping us [19:01:47] let me pre-empty the LVS restart thing by telling you the exact commands to run if that helps [19:01:52] or bonus points if you have done it before! [19:05:48] Yeah, I'm here [19:05:52] ok great! [19:11:49] brett for context, we are working from https://gerrit.wikimedia.org/r/1064831 andT364364 [19:12:32] inflatador: do you have the service change patch ready as well? [19:12:44] that probably is the other thing that Traffic could help with the review [19:13:02] basically the change to hieradata/common/service.yaml for the service definition [19:16:17] ryankemper^^ can you confirm that https://gerrit.wikimedia.org/r/c/operations/puppet/+/1064829 is the correct patch for traffic to review? [19:16:37] nah we don't need review on that one [19:16:49] I was gonna merge the dns change before working on the service patch but now that I think about it I should just get that patch up now [19:16:54] sukhe: will have that ready in 5' [19:17:18] no worries! and yeah, it might help you keep the order better as well and hence [19:17:23] (order of changes) [19:38:55] I think this would be better suited for monday before EOD - LVS is a delicate beast [19:39:14] inflatador: ryankemper: ^ [19:40:08] brett ACK, will follow y'all's advice on this one. Sorry to put you on the stop [19:40:12] or spot, as it were [19:40:25] No problem at all! Sorry I didn't push back sooner. I hope you didn't gear up too much [19:41:29] +1 but get the patches in and we can review [19:41:35] nah, we'll put the extra time to good use [19:41:44] best to do this end to end in a way once we start [19:41:55] so reviewed patches and just merge and go [19:47:38] sukhe: brett: initial service.yaml patch is up: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1064841/1/hieradata/common/service.yaml [19:48:04] thanks, will review shortly [19:57:38] would it be a safer operation if we used ipip for the pools? [19:58:00] we did that for cloudelastic, although it was more about avoiding the missing VLAN issue [19:59:43] avoiding L2 as much as possible always seems safer to me ;) [20:04:23] inflatador: that is a question for vg! [20:07:20] ACK, I'll ask tomorrow, but let's plan on doing it the most common way I guess [20:14:22] inflatador: it's not a bad idea to do anyway, at some point soon I think vg wants to require IPIP for all services [20:30:58] Small questions about the subsection `Add discovery/DNS resources` of the LVS wikitech page [20:31:10] it currently says the following: [20:31:14] > Switching from lvs_setup → production does not require anything more than a puppet run. Move forward and change state of your service to production. Also run puppet on the auth dns servers, which will add DNS discovery entries for your service to the zones they serve [20:31:42] Would it make more sense for it to mention merging the dns disc patch(es) after switching lvs_setup? (Assuming I'm understanding correctly) [20:32:08] i.e. as written it kind of makes it sound like just flipping the `state` to `production` is going to automagically add the dns discovery entries [20:32:29] yes that goes after the lvs setup switch [20:32:37] the merge [20:32:45] will update the docs