[04:28:48] 10Traffic, 10Operations: Perform HTTPS redirect without crossing domain boundaries for non canonical domains - https://phabricator.wikimedia.org/T231513 (10Vgutierrez) [04:29:30] 10Traffic, 10Operations: Perform HTTPS redirect without crossing domain boundaries for non canonical domains - https://phabricator.wikimedia.org/T231513 (10Vgutierrez) p:05Triage→03Normal [04:31:09] 10Traffic, 10Operations: Enable HSTS for non canonical domains using the ncredir service - https://phabricator.wikimedia.org/T231514 (10Vgutierrez) [04:31:38] 10Traffic, 10Operations: Enable HSTS for non canonical domains using the ncredir service - https://phabricator.wikimedia.org/T231514 (10Vgutierrez) p:05Triage→03Normal [04:49:03] 10netops, 10Operations: Review switches ACL to connect from tools-bastion to dbproxy1018 - https://phabricator.wikimedia.org/T231418 (10Marostegui) Works! Thank you! [05:10:03] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, and 2 others: Picture from Commons not found from Singapore - https://phabricator.wikimedia.org/T231086 (10aaron) >>! In T231086#5434295, @CDanis wrote: > I suggest: > * when ATS gets a 404 response from one cluster, retry on the other activ... [08:28:11] cp1085 is a special case. when i just ran puppet on cp_text_eqiad it did not finish [08:28:21] Waited 120 seconds and a preceding puppet run is still ongoing, aborting [08:29:43] maybe we need to delete the lock file manually [08:30:37] :? [08:30:49] so where it was blocked? [08:31:55] the log says half an hour ago it worked normal in about 25 seconds but now it doesnt seem to finish in minutes [08:32:25] 51 Aug 29 08:23:03 cp1085 puppet-agent[20349]: Loading facts [08:32:26] 52 Aug 29 08:28:41 cp1085 puppet-agent[21866]: Run of Puppet configuration client already in progress; [08:32:46] but earlier: 42 Aug 29 07:53:43 cp1085 puppet-agent[11169]: Applied catalog in 24.39 seconds [08:33:23] no other hosts had this, just 1085 [08:34:14] because there is a puppet running AFAICT [08:34:26] could be stuck at /usr/sbin/ipmi-config --category=core -o -S Lan_Conf [08:34:53] it's the puppet run triggered from the cumin command vs local agent ? [08:35:02] it's the cron one [08:35:12] hm [08:35:50] yea, but they should not be taking this long. i am tempted to kill stuff and delete the lock file and run puppet agent -tv [08:36:59] at most kill the stuck process, let puppet finish and clear the lock itself [08:37:16] I see that host has ipmi sensor status unknown ack-ed with a comment "known", cc ema [08:37:21] no link to a task [08:37:23] aha! [08:37:26] but might be related [08:40:24] killed 2 x 2 PIDs. running manually, seems stuck after Loading facts again [08:40:37] your theory sounds right [08:42:23] hrmm.. i just want it to get my config change [08:44:56] try https://wikitech.wikimedia.org/wiki/Management_Interfaces [08:48:32] ok. the "does IPMI work locally" part already fails with the "driver timeout" that is described as the typical error [08:51:58] mgmt does not let me connect because there is an existing session by somebody. i will create a ticket for it [08:59:12] ..and while i was doing that the puppet run now worked and applied the config change [08:59:34] still creating it to fix the known IPMI issue then [09:03:22] 10Traffic, 10Operations, 10ops-eqiad: cp1085 - IPMI not working - https://phabricator.wikimedia.org/T231525 (10Dzahn) [09:08:50] mutante: have you tried to follow the wiki to fix it? [09:10:02] volans: yes, tried. but cant use mgmt because somebody already on it [09:10:27] while both local and remote ipmi fails [09:11:31] ack [09:19:02] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) [09:43:41] 10Traffic, 10Operations: Improve ATS prometheus metrics - https://phabricator.wikimedia.org/T231533 (10Vgutierrez) [10:26:13] 10Traffic, 10Operations: Allow blocking requests from specific networks on the edge - https://phabricator.wikimedia.org/T231063 (10ema) 05Open→03Resolved This is now done. [10:28:42] 10Traffic, 10Operations, 10Patch-For-Review: cergen fails signing CSR - https://phabricator.wikimedia.org/T231423 (10ema) @Ottomata: I understand that this is now fixed, can you confirm and close the task if so? [10:48:47] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Error pulling image from docker registry - https://phabricator.wikimedia.org/T231388 (10ema) 05Open→03Resolved We have managed to generate a proper certificate for the docker-registry origin servers, and cp1075 is now back to using TLS to conne... [10:50:35] 10Traffic, 10Operations, 10Patch-For-Review: Improve ATS prometheus metrics - https://phabricator.wikimedia.org/T231533 (10ema) p:05Triage→03Normal [10:50:46] 10Traffic, 10Operations, 10ops-eqiad: cp1085 - IPMI not working - https://phabricator.wikimedia.org/T231525 (10ema) p:05Triage→03Normal [11:08:52] 10Traffic, 10Operations: Unexpectedly received mobile version of an article while logged out - https://phabricator.wikimedia.org/T231504 (10ema) p:05Triage→03High [11:19:03] 10Traffic, 10Operations: Unexpectedly received mobile version of an article while logged out - https://phabricator.wikimedia.org/T231504 (10ema) Thanks for filing this bug and for providing all request/response headers, very useful! I'm currently trying to reproduce the issue with [[ https://en.wikipedia.org/... [11:38:16] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) [11:38:39] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) [13:22:41] 10Traffic, 10Operations: Unexpectedly received mobile version of an article while logged out - https://phabricator.wikimedia.org/T231504 (10ema) a:03ema [13:42:53] 10Traffic, 10Operations, 10Patch-For-Review: cergen fails signing CSR - https://phabricator.wikimedia.org/T231423 (10Ottomata) 05Open→03Resolved [13:53:37] does anyone remember the link for that ticket that had to do with WM EE and/or wikimedia.ee [13:53:47] I swear there was one, but phab search is amazin [13:54:16] I always use google [13:54:23] wikimedia.ee site:phabricator.wikimedia.org [13:54:31] that gives https://phabricator.wikimedia.org/T204056 [13:55:22] ah! [13:55:23] thanks [13:55:33] yw! phab search would have NEVER found it