[15:50:21] 10Traffic, 10Varnish, 10Analytics, 10Operations, 10User-Elukey: Sort out analytics service dependency issues for cp* cache hosts - https://phabricator.wikimedia.org/T128374#4097473 (10Milimetric) p:05Normal>03Triage [15:51:04] 10Traffic, 10Varnish, 10Analytics, 10Operations, 10User-Elukey: Sort out analytics service dependency issues for cp* cache hosts - https://phabricator.wikimedia.org/T128374#2072088 (10Milimetric) p:05Triage>03Low [16:10:37] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Define turn-up process and scope for eqsin service to regional countries - https://phabricator.wikimedia.org/T189252#4097558 (10BBlack) [16:11:21] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Define turn-up process and scope for eqsin service to regional countries - https://phabricator.wikimedia.org/T189252#4036653 (10BBlack) [17:07:36] 10Traffic, 10Operations, 10ops-codfw: cp[2006,2008,2010-2011,2017-2018,2022].codfw.wmnet: Uncorrectable Memory Error - https://phabricator.wikimedia.org/T190540#4097768 (10RobH) >>! In T190540#4093023, @Papaul wrote: > cp2022 SEL after test > > "Normal","Thu Mar 29 2018 20:11:58","Log cleared." > "Warning",... [17:10:04] 10Traffic, 10Operations, 10ops-codfw: cp[2006,2008,2010-2011,2017-2018,2022].codfw.wmnet: Uncorrectable Memory Error - https://phabricator.wikimedia.org/T190540#4097772 (10RobH) Next steps: * reboot cp2022 in serial console a dozen times and watch it ** if it has the error, it shows during post AND pushes t... [17:42:42] 4/win 12 [17:43:47] 10Traffic, 10Operations, 10ops-codfw: cp[2006,2008,2010-2011,2017-2018,2022].codfw.wmnet: Uncorrectable Memory Error - https://phabricator.wikimedia.org/T190540#4097889 (10RobH) So I've rebooted cp2022 12 times, attempting to re-create the memory error that @ema experienced on this machine (and is demonstrat... [17:57:29] 10Traffic, 10Operations, 10ops-codfw: cp2006 memory replacement - https://phabricator.wikimedia.org/T191223#4097930 (10RobH) p:05Triage>03High [17:57:32] 10Traffic, 10Operations, 10ops-codfw: cp2008 memory replacement - https://phabricator.wikimedia.org/T191224#4097944 (10RobH) p:05Triage>03Normal [17:57:36] 10Traffic, 10Operations, 10ops-codfw: cp2010 memory replacement - https://phabricator.wikimedia.org/T191225#4097958 (10RobH) p:05Triage>03Normal [17:57:48] 10Traffic, 10Operations, 10ops-codfw: cp2011 memory replacement - https://phabricator.wikimedia.org/T191226#4097972 (10RobH) p:05Triage>03Normal [17:57:56] 10Traffic, 10Operations, 10ops-codfw: cp2017 memory replacement - https://phabricator.wikimedia.org/T191227#4097986 (10RobH) p:05Triage>03Normal [17:58:00] 10Traffic, 10Operations, 10ops-codfw: cp2018 memory replacement - https://phabricator.wikimedia.org/T191228#4098000 (10RobH) p:05Triage>03Normal [17:58:11] 10Traffic, 10Operations, 10ops-codfw: cp2022 memory replacement - https://phabricator.wikimedia.org/T191229#4098015 (10RobH) p:05Triage>03Normal [18:08:31] 10Traffic, 10Operations, 10ops-codfw: cp2006 memory replacement - https://phabricator.wikimedia.org/T191223#4098059 (10RobH) [18:20:27] 10Traffic, 10Operations, 10ops-codfw: cp2006 memory replacement - https://phabricator.wikimedia.org/T191223#4098091 (10RobH) [18:21:23] 10Traffic, 10Operations, 10ops-codfw: cp2006 memory replacement - https://phabricator.wikimedia.org/T191223#4097930 (10RobH) [18:25:38] i would like to turn "misc_static_sites" active:active [18:25:51] i have prepared the codew equivalent and added as backend [18:26:01] and renamed the director to not be named after some host [18:26:14] now i would simply enable codfw like so. https://gerrit.wikimedia.org/r/#/c/423080/1/hieradata/role/common/cache/misc.yaml easy enough? [18:26:40] i know i should not switch over in a single step, but this is just adding codfw.. is it sane [18:32:57] 10Traffic, 10Operations, 10ops-codfw: cp2008 memory replacement - https://phabricator.wikimedia.org/T191224#4098122 (10RobH) p:05Normal>03High [18:38:50] 10Traffic, 10Operations, 10ops-esams: cp3034: Uncorrectable Memory Error - https://phabricator.wikimedia.org/T189305#4098146 (10RobH) a:03RobH [18:43:09] 10Traffic, 10Operations, 10ops-codfw: cp2010 memory replacement - https://phabricator.wikimedia.org/T191225#4098168 (10RobH) p:05Normal>03High [18:43:28] 10Traffic, 10Operations, 10ops-codfw: cp2010 memory replacement - https://phabricator.wikimedia.org/T191225#4097958 (10RobH) [18:51:13] 10Traffic, 10Operations, 10ops-codfw: cp2011 memory replacement - https://phabricator.wikimedia.org/T191226#4098198 (10RobH) p:05Normal>03High [19:05:34] 10Traffic, 10Operations, 10ops-codfw: cp2017 memory replacement - https://phabricator.wikimedia.org/T191227#4098223 (10RobH) p:05Normal>03High [19:15:12] 10Traffic, 10Operations, 10ops-codfw: cp2018 memory replacement - https://phabricator.wikimedia.org/T191228#4098243 (10RobH) [19:15:22] 10Traffic, 10Operations, 10ops-codfw: cp2018 memory replacement - https://phabricator.wikimedia.org/T191228#4098000 (10RobH) p:05Normal>03High [19:17:50] 10Traffic, 10Operations, 10ops-codfw: cp[2006,2008,2010-2011,2017-2018,2022].codfw.wmnet: Uncorrectable Memory Error - https://phabricator.wikimedia.org/T190540#4098249 (10RobH) I've reviewed the next steps with @bblack via IRC. @Papaul ran memtest on all of the machines reporting failed memory, and they al... [19:18:07] bblack: when you have a moment need feedback on https://phabricator.wikimedia.org/T190540#4098249 [19:44:19] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 4 others: Thumbor incorrectly normalizes .jpe and .jpeg into .jpg for Swift thumbnail storage - https://phabricator.wikimedia.org/T191028#4098317 (10Gilles) thumb_handler.php is for private wikis, though, afaik, public wikis hit thumb.php [20:00:37] 10Wikimedia-Apache-configuration, 10Discovery, 10Zero, 10Mobile, and 2 others: m.wikipedia.org and zero.wikipedia.org should redirect how/where - https://phabricator.wikimedia.org/T69015#730153 (10LGoto) Hi @DFoy, checking on this - do you have any follow up? Thanks! [20:05:55] 10Traffic, 10Analytics, 10Operations, 10Research, and 6 others: Referrer policy for browsers which only support the old spec - https://phabricator.wikimedia.org/T180921#3773297 (10gh87) I was using the Apple Safari 11.0.3 (13604.5.6). Then I enabled the "Develop" tab in the Menu, then right-clicked to open... [20:12:35] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 4 others: Thumbor incorrectly normalizes .jpe and .jpeg into .jpg for Swift thumbnail storage - https://phabricator.wikimedia.org/T191028#4091369 (10Tgr) `thumb.php` is for URLs which contain the filename and size as query parameters (eit... [20:17:00] 10Traffic, 10Analytics, 10Operations, 10Research, and 6 others: Referrer policy for browsers which only support the old spec - https://phabricator.wikimedia.org/T180921#4098467 (10Tgr) Apparently it doesn't implement fallbacks as per spec (the last fallback value is `origin`). I don't think there's anythin... [20:17:45] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4098474 (10ayounsi) [20:46:47] 10Traffic, 10DNS, 10Operations, 10Release-Engineering-Team, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4098592 (10Varnent) Just to be clear around expectations - Foundation Wiki is managed by the Foundation and is... [21:24:37] i'll add the "apache-fast-test" script to cache::misc puppet profile [21:24:52] because if i want test a backend of some misc http service behind it [21:25:09] i can only run it from there, since nowadays ferm rules say http only from cache::misc to the backend [21:25:34] so the part that it's on the deployment_server, historically, doesn't help anymore