[07:20:40] 10netops, 10Operations, 10fundraising-tech-ops: Automate diff and commit of frack ACL - https://phabricator.wikimedia.org/T260655 (10ayounsi) p:05Triage→03Low [09:01:38] 10netops, 10Operations, 10observability, 10Patch-For-Review: Add alert[12]001 to network ACLs - https://phabricator.wikimedia.org/T260533 (10ayounsi) 05Open→03Resolved a:03ayounsi Deployed! [09:12:34] 10netops, 10Operations, 10observability: Add alert[12]001 to network ACLs - https://phabricator.wikimedia.org/T260533 (10fgiunchedi) 05Resolved→03Open Thanks @ayounsi ! Reopening since we'll need to add these hosts to pfw devices as well, cc @Jgreen and @Dwisehaupt could you help with that ? Thanks! [09:13:59] 10netops, 10Operations, 10fundraising-tech-ops, 10observability: Add alert[12]001 to network ACLs - https://phabricator.wikimedia.org/T260533 (10fgiunchedi) a:05ayounsi→03None [09:42:23] 10netops, 10Operations, 10fundraising-tech-ops, 10observability: Add alert[12]001 to network ACLs - https://phabricator.wikimedia.org/T260533 (10fgiunchedi) @ayounsi looks like mgmt access isn't permitted yet, can't ping `mgmt.eqiad.wmnet` e.g. ` alert1001# ping ps1-c1-eqiad.mgmt.eqiad.wmnet PING ps1-c1-e... [11:39:22] 10netops, 10Operations, 10ops-eqiad: (Need by: 2019-09-30) upgrade msw1-eqiad from EX4200 to EX4300 - https://phabricator.wikimedia.org/T225121 (10faidon) Ping? Besides the issues identified by @ayounsi just above, I see that in another comment above @ayounsi mentioned "wipe the switch" but then I saw the sw... [15:06:33] 10HTTPS, 10Cloud-VPS, 10Patch-For-Review, 10cloud-services-team (Kanban): Set "https_upgrade" configuration flag for domainproxy to enforce HTTPS upgrade for GET|HEAD requests - https://phabricator.wikimedia.org/T120486 (10bd808) Change and timeline announced to community: https://lists.wikimedia.org/piper... [15:08:33] 10HTTPS, 10Cloud-VPS, 10Patch-For-Review, 10cloud-services-team (Kanban): Set "https_upgrade" configuration flag for domainproxy to enforce HTTPS upgrade for GET|HEAD requests - https://phabricator.wikimedia.org/T120486 (10bd808) [15:09:18] 10HTTPS, 10Cloud-VPS, 10cloud-services-team (Kanban): Set "https_upgrade" configuration flag for domainproxy to enforce HTTPS upgrade for GET|HEAD requests - https://phabricator.wikimedia.org/T120486 (10bd808) [15:31:00] 10netops, 10Operations, 10fundraising-tech-ops: Automate diff and commit of frack ACL - https://phabricator.wikimedia.org/T260655 (10Jgreen) Is this a process that would be prompted over ssh at the same time as we push the policy to /var/tmp? Or would there be a separate process that watches for a new policy... [15:38:20] 10Traffic, 10Operations: Analyze custom varnish 5.1 patches considering the migration to varnish 6 - https://phabricator.wikimedia.org/T260702 (10Vgutierrez) [15:38:41] 10Traffic, 10Operations: Analyze custom varnish 5.1 patches considering the migration to varnish 6 - https://phabricator.wikimedia.org/T260702 (10Vgutierrez) p:05Triage→03Medium [16:20:27] 10Traffic, 10Operations, 10Patch-For-Review: Analyze custom varnish 5.1 patches considering the migration to varnish 6 - https://phabricator.wikimedia.org/T260702 (10Vgutierrez) |patch|backport/custom|available on varnish 6.0 |available on varnish 6.4 |can be removed? |0002-exp-thread-realtime.patch| custom... [17:02:41] 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Operations, 10Patch-For-Review: TY pages in a subdomain of wikipedia and set hide banner cookie - https://phabricator.wikimedia.org/T251780 (10Pcoombe) Hi all, I've copied across the Thank You pages content to the new wiki. The pag... [17:04:53] 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Operations, 10Patch-For-Review: TY pages in a subdomain of wikipedia and set hide banner cookie - https://phabricator.wikimedia.org/T251780 (10DStrine) Thanks @Pcoombe for all your help!!!one!1 I want to over-communicate this with... [17:29:04] 10netops, 10Operations, 10ops-eqiad: cloudflare CLF-20200806 dmarc to router patch - https://phabricator.wikimedia.org/T259923 (10Cmjohnson) 05Open→03Resolved fixed [17:56:46] 10netops, 10Operations, 10ops-codfw: (Need by: ) codfw:rack/setup/new management switches - https://phabricator.wikimedia.org/T253154 (10Papaul) [18:32:55] irccloud's UI is being a bit finnicky and not showing me if my messages yesterday actually sent, so going to re-post, sorry if the messages already came through: [18:33:07] I'm seeing an e-mail from letsencrypt about certificate expirations for `cloudelastic.wikimedia.org`, as well as `cloudelastic100[1-4].wikimedia.org`. Certs expire 25 Aug 20 (9 days) [18:33:13] anyone have any context on that? I am guessing since we're getting an e-mail that we don't have automated renewal configured, but not super familiar with how we manage certs (yet) [18:33:21] (e-mail subject is `Let's Encrypt certificate expiration notice for domain "cloudelastic.wikimedia.org" (and 4 more)` if anyone needs to go looking for it) [18:34:04] ryankemper: https://wikitech.wikimedia.org/wiki/Acme-chief is the relevant noun, but, I don't know too much more than that [18:39:56] We do have automatic renewal [18:40:49] We usually get those after the cert config changes.. a SNI is added or removed.. cause at that point LE treats them as different certs [18:40:54] we get those emails every once in a while. i think the issue is the mails are sent at 25 days but auto-renewal at 15 or so [18:43:32] nope [18:43:37] https://github.com/wikimedia/puppet/blob/f2f2e968a7285cb933f1f11b687c8ffc6a46d217/hieradata/role/common/acme_chief.yaml#L25 [18:43:56] We get those cause 1005 and 1006 have been added there [18:44:24] acme-chief renews the cert 1 month before expiration date [18:44:43] Or after 2 months it's been issued [18:45:52] Those were added by me on Jun 4: https://github.com/wikimedia/puppet/commit/822259dc2de2eb5dc9669b6ad6f5f04b24dba2ba#diff-596e077704c5eb3d886cc4afb8acce21 [18:46:01] That's OK [18:46:10] So based off the above, shouldn't it already have renewed (1 month before expiration), or is there a step somewhere that I'm missing [18:46:27] That already happened [18:46:45] But from let's encrypt point of view we have 2 certs [18:47:07] One including 1001-1006 and one with 1001-1004 [18:47:35] The latter is going to expire (as expected) and that's why LE warns us [18:53:30] Ah! That makes perfect sense. Thanks for all the help guys [18:53:55] No problem :)