[15:52:54] ema: can you take a sanity peek at https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/526714/ ? [15:59:48] bblack: looks sane [16:00:24] gotta go, see you tomorrow! [17:14:57] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10faidon) a:03JAufrecht [17:26:51] 10Traffic, 10Operations: All files giving 404 for some people - https://phabricator.wikimedia.org/T229434 (10Aklapper) 05Resolved→03Invalid [17:27:29] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10greg) >>! In T226044#5380759, @BBlack wrote: > 2. Configure a new ph... [17:34:43] 10Traffic, 10Operations: All files giving 404 for some people - https://phabricator.wikimedia.org/T229434 (10CDanis) FTR this was likely another instance of {T207340} [17:39:38] 10Traffic, 10Operations, 10Patch-For-Review: Roll out Anycast RecDNS to more servers - https://phabricator.wikimedia.org/T228190 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts: ` ['cp1008.wikimedia.org'] ` The log can be found in `/var/log/wmf-auto-re... [17:54:15] 10Traffic, 10Operations, 10Patch-For-Review: Roll out Anycast RecDNS to more servers - https://phabricator.wikimedia.org/T228190 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp1008.wikimedia.org'] ` Of which those **FAILED**: ` ['cp1008.wikimedia.org'] ` [18:09:52] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10JAufrecht) > Decide on a public pretty domainname for these more pub... [18:10:21] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10JAufrecht) a:05JAufrecht→03greg passing to Greg for step 2, configuring the blog and vanity name. [18:13:45] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10BBlack) >>! In T226044#5381138, @JAufrecht wrote: > I think it should be techblog.wikimedia.org, because ev... [18:14:02] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10Krinkle) I tested the "Move Post" feature in Phame today - on the foresight that some people will likely tr... [18:19:25] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10BBlack) Replying to myself earlier: apparently they're datestamped URIs beginning with `/yyyy/mm/`, example... [18:27:14] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10Krinkle) >>! In T226044#5381193, @BBlack wrote: > [..] apparently they're datestamped URIs beginning with `... [18:32:32] 10Traffic, 10Operations: Roll out Anycast RecDNS to more servers - https://phabricator.wikimedia.org/T228190 (10BBlack) Rollout status update: things that are using anycast recdns resolv.conf in production as of 2019-07-31: * All hosts in edge DCs (esams, ulsfo, eqsin) * All cp edge cache hosts globally * All... [18:32:50] 10Traffic, 10Operations: Roll out Anycast RecDNS to more servers - https://phabricator.wikimedia.org/T228190 (10BBlack) [18:50:40] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10BBlack) Heh, apparently I can't even remember things I read and said before even when they're right above m... [19:08:11] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10Aklapper) >>! In T226044#5381170, @Krinkle wrote: > I tested the "Move Post" feature in Phame today [...] f... [20:34:53] 10netops, 10Operations, 10ops-codfw: update RE-S-X6-64G-S in cr[12]-codfw - https://phabricator.wikimedia.org/T226422 (10ayounsi) First JTAC suggestion is to re-seat the SCB. We didn't do that today as the doc wasn't clear if it could be done with the router online. JTAC is looking into the logs. [20:38:55] 10netops, 10Operations, 10ops-codfw: update RE-S-X6-64G-S in cr[12]-codfw - https://phabricator.wikimedia.org/T226422 (10ayounsi) Also noticed the following while looking at the doc again today: >4. Use the request chassis routing-engine master switch command to make the Routing Engine RE-S-X6-64G (RE1) the... [22:35:43] 10netops, 10Operations: Instability of the Level3 link between cr2-eqiad and cr2-esams - https://phabricator.wikimedia.org/T228827 (10ayounsi) > This circuit has been impacted by multiple planned maintenances and higher-level network events. They have all been different troubles that have been restored so the... [22:41:40] 10Traffic, 10Operations, 10RESTBase, 10Core Platform Team Legacy (Later), and 2 others: Split slash decoding from general percent normalization in Varnish VCL - https://phabricator.wikimedia.org/T127387 (10Pchelolo) 05Open→03Resolved a:03Pchelolo doesn't seem to be any more movement or things to be d... [22:41:46] 10Traffic, 10Mobile-Content-Service, 10Operations, 10RESTBase, and 2 others: Varnish not purging RESTBase URIs - https://phabricator.wikimedia.org/T127370 (10Pchelolo)