[07:17:06] good morning :) [07:17:42] I have a lot of analytics alerts due to failures in using the webproxy, and I noticed the switch between install1002/1003 2002/2003 [07:17:58] I was about to update the analytics-in4/6 filters but then I noticed more [07:17:59] https://gerrit.wikimedia.org/r/#/c/operations/homer/public/+/579131/ [07:18:26] let me know if this is too broad or not, I can just rework it to be analyitcs-related --^ [07:22:10] (or if good we can merge, push changes only in eqiad to unblock my use case and then wait for Arzhel) [07:24:13] what do you mean by too broad? [07:26:39] XioNoX: hello! I meant that it was probably related to more routers than my area of competence, and since I wanted to push the analytics change asap (some jobs are failing and people will complain etc..) I preferred to split the change in two [07:26:44] analytics vs the rest [07:26:51] also I didn't know you were around :D [07:27:37] I created https://gerrit.wikimedia.org/r/#/c/579140/ [07:28:09] XioNoX: --^ [07:29:10] just woke up and trying to figure out what the travel ban means for me 🤔 [07:29:11] for the other one, do you prefer to keep install[12]00[23] IPs together or just replace 1002/1003 2002/2003 ? (in the rules rejecting traffic for those ips) [07:29:18] ahhahaha sorry [07:29:38] I'll ping Alex later on [07:30:20] change lgtm, didn't check thr IPs though [07:31:50] dunno if the old install servers/proxies are still in use [07:36:10] yes this is a good point, I also noticed other terms referencing install1002/2002 in the analytics filters.. Since we are moving to buster I suppose that 1002/2002 will not be around for a lot [07:48:18] the only install1002/2002 servers currently only dropped the web proxy part, the other bits they carry (DHCP/apt) will be migrated soon [07:54:56] ack so I can only move the webproxy part for the moment [08:17:19] do you remember the 2 conferences that alex and effie presented recently? I am trying to find the slides/video? [08:24:05] found it: https://www.usenix.org/conference/srecon19emea/presentation/mouzeli [15:29:11] o/ [18:24:43] 4% of flee with puppet failures, known? [18:24:49] mostly appservers and parsoid [18:24:54] *fleet [18:30:15] Exec[verify-envoy-config] failure [18:30:36] 65 nodes for now according to puppetboard [18:30:49] akosiaris: ^^^ related to you work? [18:30:59] seems to be going down now [18:31:15] maybe already fixed, dunno [18:31:58] trying to run puppet on one of the failed ones [18:32:21] jynus: no, but i can confirm on one of them it's gone [18:32:48] it's successful, so I guess it's already fixed and they are recovering [18:33:00] - cluster: eventgate-analytics-http [18:33:00] + cluster: eventgate-analytics [18:47:10] <_joe_> volans: yes it was reverted [18:47:27] <_joe_> sorry we were talking in serviceops [19:00:53] since we switched webproxy* to buster backends there is some minor cronspam. that's because "hierarchy_stoplist" became obsolete in squid config. [19:01:29] afaict we don't really need to tell squids to "not ask neighbor squids if an URL contains cgi" though ? https://gerrit.wikimedia.org/r/c/operations/puppet/+/579373 [19:01:43] per http://www.squid-cache.org/Doc/config/hierarchy_stoplist/