[00:42:52] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11080846 (10Scott_French) Alright, the first ImageSuggestions job (cawiki) seems to have completed without issue after a typical ~ 10m run durat... [09:33:09] 06serviceops, 10MW-on-K8s: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#11081798 (10Urbanecm_WMF) Hello hello! I heard @Clement_Goubert saying that "The "old" foreachwiki/mwscript wrappers WILL be deprecated completely soon". As far as I understand thing... [09:34:25] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#11081806 (10Urbanecm_WMF) FYI: {T401803} seems to be another slight issue with `mwscript-k8s`. [09:59:20] 06serviceops, 10MW-on-K8s: Custom dblists for mwscript-k8s - https://phabricator.wikimedia.org/T401737#11081905 (10Urbanecm_WMF) My 2c on the solutions: **(1) upload dblists to the container with `--file`** Definitely workable, but it means the resulting syntax will be quite long and complex: I'd need to spec... [10:04:10] 06serviceops, 10MW-on-K8s: Custom dblists for mwscript-k8s - https://phabricator.wikimedia.org/T401737#11081920 (10Urbanecm_WMF) > One downside of all three approaches is opacity: all we'd be able to tell from examining the Kubernetes object is the dblist's filename, not its contents. To tell what wikis it's o... [10:09:17] 06serviceops, 10MW-on-K8s: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#11081933 (10Clement_Goubert) >>! In T377497#11081797, @Urbanecm_WMF wrote: > Hello hello! I heard @Clement_Goubert saying that "The "old" foreachwiki/mwscript wrappers WILL be deprec... [11:22:50] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 3 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#11082227 (10MoritzMuehlenhoff) [13:39:03] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11082618 (10Eevans) >>! In T368096#11080846, @Scott_French wrote: > Alright, the first ImageSuggestions job (cawiki) seems to have completed wit... [14:25:51] 06serviceops, 06Abstract Wikipedia team: Cannot deploy function-orchestrator due to deprecated appArmor field - https://phabricator.wikimedia.org/T401833 (10cmassaro) 03NEW [14:26:45] 06serviceops, 06Abstract Wikipedia team: Cannot deploy function-orchestrator due to deprecated appArmor field - https://phabricator.wikimedia.org/T401833#11082821 (10cmassaro) [14:49:59] 06serviceops, 06Abstract Wikipedia team: Cannot deploy function-orchestrator due to deprecated appArmor field - https://phabricator.wikimedia.org/T401833#11082897 (10akosiaris) Those are warning (note the W prefix). They wouldn't stop the deployment from happening. The actual reason is this https://logstash.... [14:54:50] 06serviceops, 06Abstract Wikipedia team: Cannot deploy function-orchestrator in staging environment due to insufficient quotas - https://phabricator.wikimedia.org/T401833#11082906 (10akosiaris) [15:16:02] 06serviceops, 06Abstract Wikipedia team, 13Patch-For-Review: Cannot deploy function-orchestrator in staging environment due to insufficient quotas - https://phabricator.wikimedia.org/T401833#11082991 (10Jdforrester-WMF) >>! In T401833#11082897, @akosiaris wrote: > Those are warning (note the W prefix). They... [15:24:59] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#11083016 (10Vgutierrez) >>! In T352956#11016142, @akosiaris wrote: > Yup, scheduling it for the weeks of either... [15:30:22] 06serviceops, 06Abstract Wikipedia team, 13Patch-For-Review: Cannot deploy function-orchestrator in staging environment due to insufficient quotas - https://phabricator.wikimedia.org/T401833#11083052 (10akosiaris) >>! In T401833#11082991, @Jdforrester-WMF wrote: >>>! In T401833#11082897, @akosiaris wrote: >>... [15:30:47] 06serviceops, 06Abstract Wikipedia team, 13Patch-For-Review: Cannot deploy function-orchestrator in staging environment due to insufficient quotas - https://phabricator.wikimedia.org/T401833#11083055 (10akosiaris) Patches deployed, you should be good to retry @cmassaro [15:53:02] Thank you! We'll try during our next deployment window today [16:45:02] 06serviceops, 06Abstract Wikipedia team, 13Patch-For-Review: Cannot deploy function-orchestrator in staging environment due to insufficient quotas - https://phabricator.wikimedia.org/T401833#11083335 (10cmassaro) >>! In T401833#11083055, @akosiaris wrote: > Patches deployed, you should be good to retry @cmas... [16:48:53] 06serviceops, 10MW-on-K8s: Custom dblists for mwscript-k8s - https://phabricator.wikimedia.org/T401737#11083344 (10RLazarus) >>! In T401737#11081905, @Urbanecm_WMF wrote: > However, I am unsure whether ConfigMap would accept such a special file This would work fine (the Python wrapper reads the file and create... [16:55:09] 06serviceops, 10MW-on-K8s: Custom dblists for mwscript-k8s - https://phabricator.wikimedia.org/T401737#11083355 (10RLazarus) One other note about all three ConfigMap approaches: they would insert the file into `dblists/` but wouldn't update `dblists-index.php`. I don't //think// that's a problem for the mwscri... [17:07:40] 06serviceops, 06Abstract Wikipedia team, 13Patch-For-Review: Cannot deploy function-orchestrator in staging environment due to insufficient quotas - https://phabricator.wikimedia.org/T401833#11083397 (10Jdforrester-WMF) Unfortunately the new 20GiB limit doesn't seem to have taken: https://logstash.wikimedia.... [18:13:17] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11083575 (10Scott_French) Summarizing some discussion out of band with @Cparle - To evaluate how things are working, it may be useful to be abl... [18:20:58] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11083616 (10Krinkle) [18:21:14] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11083617 (10Krinkle) [19:28:42] 06serviceops, 10MW-on-K8s: Running maintenance scripts in screen on `deploy1003` appears to fail, but is still running - https://phabricator.wikimedia.org/T400962#11083855 (10jrbs) Thanks for the help folks! So for this task, I don't think anything is needed - there is no bug here. But perhaps the failure to s... [20:59:47] 06serviceops, 10MediaWiki-extensions-WikimediaMaintenance, 10MW-on-K8s: mwscript-k8s does not include an environment variable with the username of the executing user - https://phabricator.wikimedia.org/T401803#11084107 (10RLazarus) [21:16:50] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11084134 (10Scott_French) The manual `--dry-run` mode `SendNotificationsForUnillustratedWatchedTitles.php` run ([[ https://sal.toolforge.org/log... [22:13:02] 06serviceops, 13Patch-For-Review: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#11084257 (10thcipriani) @akosiaris looks like you had a lot of changes for making a new registry, is that registry ready? Or is it still in the testing phase? [22:31:30] 06serviceops, 13Patch-For-Review: Upgrade mw-debug/next to PHP 8.3 - https://phabricator.wikimedia.org/T401254#11084268 (10Scott_French) Following up on T401254#11074136, the umbrella task for tracking issues that arise on 8.3 is now {T401855}. The one thing we're still assessing is the processing around //mo... [22:32:25] 06serviceops, 13Patch-For-Review: Upgrade mw-debug/next to PHP 8.3 - https://phabricator.wikimedia.org/T401254#11084270 (10Scott_French)