[09:28:34] 10Traffic, 10Operations, 10Patch-For-Review: Traffic Server - Prometheus integration - https://phabricator.wikimedia.org/T202381 (10ema) [10:36:36] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts: ``` ['cp2010.codfw.wmnet', 'cp4030.ulsfo.wmnet'] ``` The log can be found in `/var/l... [11:09:35] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['cp2010.codfw.wmnet', 'cp4030.ulsfo.wmnet'] ``` and were **ALL** successful. [13:33:42] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts: ``` ['cp2011.codfw.wmnet', 'cp4029.ulsfo.wmnet'] ``` The log can be found in `/var/l... [13:40:44] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) After one day: ``` elukey@stat1005:~$ grep https ipv6_after_changes.log| while read line; do endpoint=$(echo $line | cut -d" "... [14:00:40] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['cp4029.ulsfo.wmnet'] ``` Of which those **FAILED**: ``` ['cp4029.ulsfo.wmnet'] ``` [14:06:06] 10Traffic, 10Operations, 10Performance-Team, 10Wikimedia-General-or-Unknown, and 2 others: Search engines continue to link to JS-redirect destination after Wikipedia copyright protest - https://phabricator.wikimedia.org/T199252 (10Imarlier) It appears that a sitemap for the mobile site isn't going to be he... [14:12:38] marlier: ^ or we could switch away from the "separate URLs" sort of scheme entirely, and make all the m-dot stuff just legacy redirects at the cache layer for old links afterwards. [14:13:14] "dynamic serving" has its downsides, but since we currently redirect to mobile based on UA parsing anyways, it wouldn't be any worse than that. [14:15:00] bblack: Only downside to that is that I couldn't use the mobile site by choice by going direct to the .m site :-) [14:15:12] 10Traffic, 10Operations, 10ops-esams: cp3036 PS Redundancy Lost - https://phabricator.wikimedia.org/T202627 (10ema) [14:15:15] 10Traffic, 10Operations, 10ops-esams: cp3036 PS Redundancy Lost - https://phabricator.wikimedia.org/T202627 (10ema) p:05Triage>03Normal [14:15:26] But yes, that would also be a legit option. [14:16:20] we could always tack on some kind of "force mobile view" cookie and have a clicky to set it [14:16:31] (or something like that) [14:16:44] or go full on responsive, but that seems like a much larger effort [14:16:53] Right [14:16:59] With other costs [14:17:14] Like having to push enough CSS/JS to handle whatever the user decides to do. [14:17:23] Which has other penalties [14:18:14] yeah, makes sense [14:19:08] make the default anonymous view lightweight enough that responsive is reasonable and nobody feels the need to force a mobile view on desktop, and only do the heavyweight version of the site for users that log into a session (incl anonymous-editing session)? [14:20:54] The main things that I'm discovering here: 1) Google is actually fine with treating the desktop and mobile sites as being the same site even if they have different URLs, but we're (ideally) supposed to be providing some smarter hints; and 2) Google wants those hints to use CSS selectors if possible, so that they can switch which URL to return based on information that they have at query time (viewport width, interaction type, etc) [14:21:26] Our UA filters are effective for us, but make things hard for them. [14:21:50] they're also probably not very well maintained [14:23:24] the last update to that block of regexen was a little over 2 years ago to exclude "(SMART-TV.*SamsungBrowser)", before that it's 2013/2014 timeframe [14:26:37] 10Traffic, 10Operations, 10Performance-Team, 10Wikimedia-General-or-Unknown, and 2 others: Search engines continue to link to JS-redirect destination after Wikipedia copyright protest - https://phabricator.wikimedia.org/T199252 (10Imarlier) Outside of the scope of this ticket, but I wanted to note it. Thi... [15:04:33] bblack: That's not terribly surprising, TBH :-) [15:05:02] My sense is that most people use CSS selectors at this point, as Google and others suggest. UA is tough to get right. [15:07:00] But, I don't think that it's a big deal. From what I can see, Google does a fine job of indexing our stuff. including mobile, under the current scheme. We'd probably be better off giving them sitemaps for every wiki, but we don't need to worry about providing separate sitemaps for mobile URLs, which simplifies things considerably. [16:07:07] I'd have a question about let's encrypt certs if anybody has time. I am working on migrating archiva from meitnerium to archiva1001 (new OS, jvm, archiva version, etc..) and I'd need a lot of testing from people before switching archiva.wikimedia.org from meitnerium to archiva1001, to avoid any disruption in people's build/deployments/etc.. Initially I thought to use a self signed cert for the ne [16:07:13] w archiva host, but now I am wondering if I could request a LE cert like "archiva-new.wikimedia.org" and then let people test it properly during the next weeks [16:07:44] eventually moving archiva.w.o to the new host and revoking the -new one (but having a super easy rollback if the new host's version is buggy for some reason) [16:07:52] is it something doable or completely wrong? [16:26:26] 10netops, 10Operations, 10ops-eqdfw: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941 (10ayounsi) [16:28:42] elukey: seems reasonable [16:28:59] you'll want to set up archiva-new in DNS as well, etc [16:29:13] ah yes yes [16:29:28] (put it in dns, configure it as the proper server name of the new installs http config, etc) [16:29:41] and then yes, deploy an archiva-new LE cert to it [16:29:54] ack, thanks for the review :) [16:30:36] archiva::proxy uses letsencrypt::cert::integrated that seems straightforward to handle the LE workflow [17:02:37] 10netops, 10Operations, 10ops-eqdfw: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941 (10Papaul) [17:15:27] 10netops, 10Operations, 10ops-eqdfw: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941 (10ayounsi) [17:43:23] 10netops, 10Operations, 10ops-eqdfw: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941 (10ayounsi) [18:30:51] 10netops, 10Operations, 10ops-eqdfw, 10Patch-For-Review: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941 (10ayounsi) [21:22:20] 10netops, 10Operations, 10ops-eqdfw, 10Patch-For-Review: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941 (10Papaul) [22:14:25] interesting project of the day: https://github.com/batfish/batfish [22:39:32] 10Traffic, 10Operations, 10Parsing-Team, 10RESTBase, 10Services (next): Improve multi-content-bucket designA - https://phabricator.wikimedia.org/T202682 (10Pchelolo) p:05Triage>03Normal [22:40:21] 10Traffic, 10Operations, 10Parsing-Team, 10RESTBase, 10Services (next): Improve Accept header normalization in VCL for REST API - https://phabricator.wikimedia.org/T202682 (10Pchelolo) [23:16:39] 10netops, 10Operations: cr1/2-eqiad PFE_FW_SYSLOG_IP6_GEN log entries - https://phabricator.wikimedia.org/T201149 (10ayounsi) 05Open>03Resolved I disabled logging a couple weeks ago, so no more `PFE_FW_SYSLOG_IP6_GEN` logs. `Next-hop resolution requests from interface XXX throttled` are probably due to de... [23:26:14] 10Traffic, 10netops, 10Operations, 10ops-ulsfo, 10Patch-For-Review: Rack/cable/configure ulsfo MX204 - https://phabricator.wikimedia.org/T189552 (10ayounsi) [23:26:18] 10Traffic, 10netops, 10Operations, 10ops-ulsfo: troubleshoot cr3/cr4 link - https://phabricator.wikimedia.org/T196030 (10ayounsi) 05stalled>03Resolved We will use them for ulsfo. If the new FS test optics (with new firmware) work, then we could consider using them in future deployments. [23:34:34] 10netops, 10Operations: Intermitent connectivity issues in eqiad's row C - https://phabricator.wikimedia.org/T201139 (10ayounsi) p:05Unbreak!>03Normal Lowering the priority, as as far I know this didn't happen again. Focusing on the row A/B issues. [23:48:31] 10netops, 10Operations, 10ops-eqdfw, 10Patch-For-Review: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941 (10ayounsi) [23:48:38] 10netops, 10Operations, 10Goal: Increase network capacity (2018-19 Q1 Goal) - https://phabricator.wikimedia.org/T199142 (10ayounsi) [23:48:41] 10netops, 10Operations, 10ops-eqdfw, 10Patch-For-Review: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941 (10ayounsi) 05Open>03Resolved [23:51:44] 10netops, 10Operations, 10fundraising-tech-ops: NAT and DNS for fundraising monitor host - https://phabricator.wikimedia.org/T198516 (10ayounsi) 05Open>03Resolved a:03ayounsi I believe we're good here. Please re-open if not (or need firewall policies).