[07:58:08] (Manual test - added "notifempty" to /etc/logrotate.d/nginx on cp1008, it should remove cronspam for access_pipe.log.1.gz) [07:58:18] ah now I can log it! [07:58:25] !log added "notifempty" to /etc/logrotate.d/nginx on cp1008, it should remove cronspam for access_pipe.log.1.gz [07:58:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:59:00] ahhh it goes to the SAL, not to a specific traffic log [07:59:02] makes sense [07:59:03] :) [08:36:44] 10netops, 06Operations: cr2-knams<->asw-esams GBLX fiber down - https://phabricator.wikimedia.org/T158647#3162767 (10ayounsi) 05Open>03Resolved From Vancis: > I've traced the whole route coming from L3 towards your cabinet. > The port is now up. The issue was the XFP, I've reseated the optic and the port... [09:07:04] 10Domains, 10Traffic, 06Operations, 06WMF-Legal, 13Patch-For-Review: Using wikimedia.ee mail address as Google account - https://phabricator.wikimedia.org/T158638#3162783 (10Beetlebeard) Thank you!! [09:07:14] 10Domains, 10Traffic, 06Operations, 06WMF-Legal, 13Patch-For-Review: Using wikimedia.ee mail address as Google account - https://phabricator.wikimedia.org/T158638#3162784 (10Beetlebeard) 05Open>03Resolved [11:58:31] 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3163583 (10Dvorapa) For me every day new images are broken and those br... [13:04:21] 10netops, 06Analytics-Kanban, 06Operations, 15User-Elukey: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#3163723 (10elukey) [13:34:49] 10netops, 06Operations: Slight packet loss observed on the network starting Nov 2016 - https://phabricator.wikimedia.org/T154507#3163841 (10elukey) ``` 14:49 not sure if this makes any sense but I did the following 14:49 mtr tegmen.wikimedia.org from netmon1001 14:50 (one of the t... [14:20:11] 10netops, 06Operations: cr2-knams<->asw-esams GBLX fiber down - https://phabricator.wikimedia.org/T158647#3163967 (10ayounsi) Attaching extra informations from Vancis staff about the X-connect: {F7344302} [17:30:35] 10Traffic, 10DNS, 06Operations: mediawiki.org DNS TXT entry for Google Search Console site verification - https://phabricator.wikimedia.org/T161343#3164332 (10dr0ptp4kt) That did the trick. Thanks! [18:17:28] 10Traffic, 10DNS, 06Operations: mediawiki.org DNS TXT entry for Google Search Console site verification - https://phabricator.wikimedia.org/T161343#3164407 (10Dzahn) 05Open>03Resolved a:03Dzahn ok, cool :) thanks for confirming and your patience. resolving ticket [18:47:04] 10Traffic, 10DNS, 06Operations: mediawiki.org DNS TXT entry for Google Search Console site verification - https://phabricator.wikimedia.org/T161343#3164498 (10dr0ptp4kt) Thank you as well for your patience. I forgot about that delegation distinction (assuming there has always been one) ! [20:13:42] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2240511 (10grin) Just as a sidenote: be aware that wildcards are only wildcard **one** level up, not **any**; `*.wikimedia.org` matches robh.wikimedia.org but not server01.robh.wikimedia.org (... [21:08:45] 10Traffic, 06Operations, 10media-storage, 13Patch-For-Review, 15User-Urbanecm: Some PNG thumbnails and JPEG originals delivered as [text/html] content-type and hence not rendered in browser - https://phabricator.wikimedia.org/T162035#3165052 (10Aklapper) >>! In T162035#3163583, @Dvorapa wrote: > For me e... [21:33:07] 10Traffic, 06Operations, 10ops-codfw: lvs2002 random shut down - https://phabricator.wikimedia.org/T162099#3165182 (10Papaul) The HP guy called around 12:30pm to let me know that he was at UPS and haven't received the main board yet they told him that the truck will be there between 1 and 3 pm. The appointme... [21:40:30] 10Traffic, 06DC-Ops, 06Operations, 10ops-codfw: lvs2002 Embedded Flash/SD-CARD iLO errors - https://phabricator.wikimedia.org/T126321#3165192 (10Papaul) 05Open>03Resolved closing this task since it is the same Flash card issue in T162099