[06:22:08] 10netops, 10Operations, 10SRE-tools, 10Goal, 10Patch-For-Review: Configuration management for network operations - https://phabricator.wikimedia.org/T228388 (10Volans) [06:22:55] 10Traffic, 10Operations: ATS fails to log the used SSLCurve when the SSL session is being reused - https://phabricator.wikimedia.org/T234011 (10Vgutierrez) [06:52:54] 10Traffic, 10Core Platform Team, 10Operations, 10Performance-Team, and 6 others: RFC: Serve Main Page of Wikimedia wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10awight) [08:39:45] 10Traffic, 10Operations: ATS fails to log the used SSLCurve when the SSL session is being reused - https://phabricator.wikimedia.org/T234011 (10Vgutierrez) p:05Triage→03Normal [12:03:27] 10Traffic, 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10MoritzMuehlenhoff) I've bui... [14:17:41] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10elukey) 05Open→03Resolved ` elukey@asw2-a-eqiad> show ethernet-switching interface xe-4/0/37 Routing Ins... [14:26:39] 10netops, 10Operations: Extend firewall rules for new corp LDAP replicas - https://phabricator.wikimedia.org/T234047 (10MoritzMuehlenhoff) [16:21:35] Is traffic aware of https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=tools.wmflabs.org&service=HTTPS-wmflabs and https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=en.planet.wikimedia.org&service=HTTPS-planet is there a task that can be used to ACK them? they have been alerting for a while now [16:26:46] wmflabs isn't us, although the alerts is similar :) [16:27:17] I'll make a ticket re: the production globalsign one [16:29:51] bblack: fyi: https://phabricator.wikimedia.org/T234061 just made to link to icinga alerts =] [16:30:16] i'll need you to let me konw changes from last year and i can get updated pricing, etc... =] [16:30:23] (no rush, we have 60 days) [16:30:47] oh ok [16:31:02] thanks! [16:32:32] maybe this one is related https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=phab.wmfusercontent.org&service=HTTPS-wmfusercontent ? [16:33:49] yeah I hit it [16:34:45] cleaning icinga christmas tree, one line at a time [16:34:53] 10Traffic, 10Mail, 10Operations: Set up basic email infra for w.wiki domain - https://phabricator.wikimedia.org/T216172 (10BBlack) Ping @herron can we move on this? Any current blockers? [16:41:00] 10netops, 10Operations: Instability of the Level3 link between cr2-eqiad and cr2-esams - https://phabricator.wikimedia.org/T228827 (10ayounsi) Another one (scheduled as 17144179) 2019-09-26 23:32:28 xe-4/1/3 ifOperStatus: down -> up 2019-09-26 22:12:28 xe-4/1/3 ifOperStatus: up -> down [16:48:01] 10netops, 10Operations: Extend firewall rules for new corp LDAP replicas - https://phabricator.wikimedia.org/T234047 (10ayounsi) There is only a mention of dubnium.wikimedia.org (208.80.154.13) in the analytics firewall filter. If that task if for network devices only, feel free to close it. If it's for all ty... [16:48:55] XioNoX: while you're here, the router side MTU stuff is all cleaned up now right? can close ticket? [16:49:30] bblack: yep, I wasn't 100% sure it was cleaned on all the servers that's why I kept it open [16:50:05] 10Traffic, 10Operations: GRE MTU mitigations - Tracking - https://phabricator.wikimedia.org/T232602 (10BBlack) 05Open→03Resolved [17:16:16] 10Traffic, 10Mail, 10Operations: Set up basic email infra for w.wiki domain - https://phabricator.wikimedia.org/T216172 (10herron) 05Open→03Resolved a:03herron Thanks for the ping/reminder! Basic aliasing for w.wiki has been deployed and successfully tested. [17:16:59] 10Traffic, 10Mail, 10Operations: Set up basic email infra for w.wiki domain - https://phabricator.wikimedia.org/T216172 (10BBlack) Awesome, thank you! [17:37:05] bblack: "Switch most production hosts to using anycast recdns @ 10.3.0.1" this is done, right? [18:29:18] 10netops, 10Operations: configure BGP route damping on IX sessions - https://phabricator.wikimedia.org/T222424 (10ayounsi) Maybe @jbond too! [18:34:49] XioNoX: I haven't had time to follow up and analyze much (e.g. tcpdump on some recdns and see what's still hitting it, might be some services have the old resolv.conf info cached until next restart, etc...) [18:35:06] XioNoX: but yeah, basically all prod hosts puppet config points at it [18:35:21] bblack: that's where the "most" come at play :) [18:35:26] yup! :) [19:19:44] 10Traffic, 10Operations, 10Puppet: Puppet systemd::mask is an anti pattern that has unwanted side effect - https://phabricator.wikimedia.org/T233839 (10herron) p:05Triage→03Normal [20:18:28] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team (Development services), and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10greg) >>! In T226044#5527864, @CDanis wrote: > Just curious -- what's the expected timefr... [20:52:06] 10Traffic, 10Operations, 10serviceops, 10Puppet: Puppet systemd::mask is an anti pattern that has unwanted side effect - https://phabricator.wikimedia.org/T233839 (10Dzahn) [22:10:14] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) [22:13:03] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) >>! In T210411#5496180, @Vgutierrez wrote: > Please note that the docker-registry certificate is missing the public hostname: `docker-registry.wikimedia.org` Per I... [22:13:27] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) https://performance.wikimedia.org switch to https://performance.discovery.wmnet as backend.