[02:16:02] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments, 10User-brennen: 1.37.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T281145 (10DannyS712) ### Risky Patch! 🚂🔥 * **Changes**: ** https://gerrit.wikimedia.org/r/c/mediawiki/core/+/678414 (removing hooks with Revision obje... [07:11:16] (03PS1) 10Umherirrender: build: Update php-parallel-lint/php-parallel-lint [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/684060 [07:13:15] (03CR) 10Umherirrender: "Last canary for phplinter for libup, but with a weight of 0. Without this last canary updated, libup would not update the phplinter for al" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/684060 (owner: 10Umherirrender) [07:19:10] (03CR) 10DannyS712: [C: 03+2] build: Update php-parallel-lint/php-parallel-lint [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/684060 (owner: 10Umherirrender) [07:20:46] (03Merged) 10jenkins-bot: build: Update php-parallel-lint/php-parallel-lint [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/684060 (owner: 10Umherirrender) [07:23:53] (03PS9) 10DannyS712: Allow configuring prefixes in PrefixedGlobalFunctionsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/661937 (https://phabricator.wikimedia.org/T274007) [07:27:17] (03PS10) 10DannyS712: Allow configuring prefixes in PrefixedGlobalFunctionsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/661937 (https://phabricator.wikimedia.org/T274007) [07:27:55] (03CR) 10DannyS712: "> Patch Set 8:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/661937 (https://phabricator.wikimedia.org/T274007) (owner: 10DannyS712) [08:27:47] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [08:30:23] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 4 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [12:41:35] (03CR) 10Daimona Eaytoy: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/684019 (https://phabricator.wikimedia.org/T257301) (owner: 10Daimona Eaytoy) [18:37:54] !log Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - https://gerrit.wikimedia.org/r/c/operations/puppet/+/684117 [18:37:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:41:23] !log Run `puppe agent -tv` on deployment-cache-text06 and deployment-mediawiki11 [18:41:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:42:39] !log Cherry-pick "mediawiki: Remove 'deployment.wikimedia' vhost from Beta Cluster" - , ref T198673 [18:42:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:42:42] T198673: Remove deployment.wikimedia.beta.wmflabs.org wiki (deploymentwiki) - https://phabricator.wikimedia.org/T198673 [18:43:20] Majavah: I notice that upload in beta is at upload.beta instead of upload.wikimedia.beta [18:43:46] do you know if there's been discussion about aligning that to use a more similar set up as well? [18:44:03] for performance.wikimedia.org our team recently changed it so that we use varnish/ats the same as in prod and thus no special domain [18:44:14] wondering what it would take to do the same for upload [18:45:26] maybe a brute-force approach of replacing all 'upload.beta' with 'upload.wikimedia.beta' will work, but could also easily go wrong and then need to revert. [18:46:13] Krinkle: likely not intentional, I'd imagine fixing would take adding cache-upload config for both domains at the same time, pointing the domain to the cache-upload floating ip, and then changing all the references [18:50:15] !log adjust deployment-cache* hieradata to treat upload.wikimedia.beta.wmflabs.org like upload.beta.wmflabs.org [18:50:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:50:38] Majavah: hm.. I don't know this config very well, but it seemed like `upload_domain` references are used in various places that are non-triival to support multiple values for [18:50:50] but maybe that's all on top of hte main entry point [18:50:54] so that's ok to change later [18:54:54] https://upload.wikimedia.beta.wmflabs.org/?SDfsdfa [18:54:58] ha, that worked already by itself [18:55:00] awesome [18:55:26] interesting, so the upload host doesn't check vhost at all, it just serves at whatever? [18:56:10] no, https://phabricator.wikimedia.org/rCLIP22d283e12ea56ae685fa628888ed7fde38c69c88 [18:56:25] but for some reason cache-text has the rule, so I added it there too [18:58:06] !log add dns record upload.wikimedia.beta.wmflabs.org. -> 185.15.56.35 (deployment-cache-upload floating address) [18:58:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:59:48] 10Beta-Cluster-Infrastructure: Move upload.beta.wmflabs.org to upload.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T281650 (10Krinkle) [19:00:04] random image from beta commons works on the new domain, https://upload.wikimedia.beta.wmflabs.org/wikipedia/commons/1/11/FourthCaption_test_.jpg [19:01:06] 10Beta-Cluster-Infrastructure: Move upload.beta.wmflabs.org to upload.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T281650 (10Krinkle) * 18:50 !log adjust deployment-cache* hieradata to treat upload.wikimedia.beta.wmflabs.org like upload.beta.wmflabs.org * 18:58 !log add dns... [19:02:05] Krinkle: I get nerd sniped way too easily, I should learn to resist that :/ [19:04:47] Majavah: sorry :) [19:06:08] This was exactly the puzzle piece I was missing though [19:08:02] (03CR) 10Jforrester: "Do we want ef… to be allowed generally, rather than added on a per-repo basis? In general they should each be replaced." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/661937 (https://phabricator.wikimedia.org/T274007) (owner: 10DannyS712) [19:11:52] (03CR) 10DannyS712: "> Patch Set 10:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/661937 (https://phabricator.wikimedia.org/T274007) (owner: 10DannyS712) [19:16:07] Majavah: I notice that the varnish hieradata containing 'upload_domain' etc is set in six places (three files in puppet.git, three files via Horizon). [19:16:08] https://codesearch.wmcloud.org/search/?q=%5Cbupload%5C.beta%5Cb&i=nope&files=deployment-prep%7Chieradata%2Fcloud&excludeFiles=&repos= [19:16:45] I suspect "traffic-cache-atsupload-buster.yaml" or at least the "profile::cache::varnish::frontend::fe_vcl_config:" part of it, is not used [19:16:50] does that make sense? [19:17:24] e.g. overridden by deployment-cache-text.yaml and deployment-cache-upload.yaml prefix-based overrides [19:20:45] Krinkle: horizon always overrides ops/puppet hiera, but I don't see horizon hiera for the instances in the traffic project [19:30:50] oh, traffic project [19:30:54] right [19:31:04] I was trying to find the instances and couldn't find them [19:31:36] hm.. I guess maybe these were tested with local web request then since that project doesn't have beta dns afaik [19:31:42] probably no longer used anyway [19:31:48] I'll just update it for now [19:33:53] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:35:02] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Move upload.beta.wmflabs.org to upload.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T281650 (10Krinkle) Updated Hieradata in Horizon for `deployment-cache-text*` and `deployment-cache-upload*`: https://gerrit.wikimedia.org/r/plugins/gitiles/... [19:38:10] ugh, and we have both deployment-cache-upload.yaml and deployment-cache-upload06.deployment-prep.eqiad1.wikimedia.cloud.yaml [19:41:27] * Krinkle merges [21:30:13] (03PS3) 10Reedy: makerelease2.py: Remove use of "sanity" [tools/release] - 10https://gerrit.wikimedia.org/r/683634 [21:30:17] (03CR) 10Reedy: [C: 03+2] makerelease2.py: Remove use of "sanity" [tools/release] - 10https://gerrit.wikimedia.org/r/683634 (owner: 10Reedy) [21:31:18] (03Merged) 10jenkins-bot: makerelease2.py: Remove use of "sanity" [tools/release] - 10https://gerrit.wikimedia.org/r/683634 (owner: 10Reedy)