[00:04:10] (03PS2) 10Nik Gkountas: add support for pagination for single page collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206409 (https://phabricator.wikimedia.org/T384485) [00:04:45] (03PS4) 10Nik Gkountas: Page collections caching: Use sitematrix lang code for all articles [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206883 (https://phabricator.wikimedia.org/T410387) [00:04:50] (03CR) 10CI reject: [V:04-1] Page collections caching: Use sitematrix lang code for all articles [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206883 (https://phabricator.wikimedia.org/T410387) (owner: 10Nik Gkountas) [00:05:50] (03CR) 10CI reject: [V:04-1] add support for pagination for single page collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206409 (https://phabricator.wikimedia.org/T384485) (owner: 10Nik Gkountas) [00:22:56] (03PS5) 10Nik Gkountas: Page collections caching: Use sitematrix lang code for all articles [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206883 (https://phabricator.wikimedia.org/T410387) [00:22:56] (03PS3) 10Nik Gkountas: add support for pagination for single page collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206409 (https://phabricator.wikimedia.org/T384485) [00:23:35] (03CR) 10CI reject: [V:04-1] Page collections caching: Use sitematrix lang code for all articles [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206883 (https://phabricator.wikimedia.org/T410387) (owner: 10Nik Gkountas) [00:24:35] (03CR) 10CI reject: [V:04-1] add support for pagination for single page collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206409 (https://phabricator.wikimedia.org/T384485) (owner: 10Nik Gkountas) [00:27:54] (03PS6) 10Nik Gkountas: Page collections caching: Use sitematrix lang code for all articles [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206883 (https://phabricator.wikimedia.org/T410387) [00:27:54] (03PS4) 10Nik Gkountas: add support for pagination for single page collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206409 (https://phabricator.wikimedia.org/T384485) [00:28:30] (03CR) 10CI reject: [V:04-1] Page collections caching: Use sitematrix lang code for all articles [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206883 (https://phabricator.wikimedia.org/T410387) (owner: 10Nik Gkountas) [00:29:28] (03CR) 10CI reject: [V:04-1] add support for pagination for single page collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206409 (https://phabricator.wikimedia.org/T384485) (owner: 10Nik Gkountas) [00:56:47] (03PS5) 10Nik Gkountas: add support for pagination for single page collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206409 (https://phabricator.wikimedia.org/T384485) [00:57:26] (03CR) 10CI reject: [V:04-1] add support for pagination for single page collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206409 (https://phabricator.wikimedia.org/T384485) (owner: 10Nik Gkountas) [02:22:04] FIRING: KubernetesDeploymentUnavailableReplicas: ... [02:22:04] Deployment aya-llm-predictor-00006-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=aya-llm-predictor-00006-deployment - ... [02:22:04] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [05:01:51] (03PS1) 10Kevin Bazira: locust: update revertrisk-wikidata load test results [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1207023 (https://phabricator.wikimedia.org/T406179) [05:04:21] (03CR) 10Kevin Bazira: "These results were generated on the `stat1008` machine as shown in: https://phabricator.wikimedia.org/P85371" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1207023 (https://phabricator.wikimedia.org/T406179) (owner: 10Kevin Bazira) [06:22:04] FIRING: KubernetesDeploymentUnavailableReplicas: ... [06:22:04] Deployment aya-llm-predictor-00006-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=aya-llm-predictor-00006-deployment - ... [06:22:04] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [08:37:05] o/ [08:37:05] whenever anyone gets a minute, please review: [08:37:05] 1. patch that updates rrwikidata load test results: https://gerrit.wikimedia.org/r/1207023 [08:37:05] 2. patch with rrwikidata LW prod deployment config: https://gerrit.wikimedia.org/r/1207027 [08:37:25] thanks in advance! [10:22:04] FIRING: KubernetesDeploymentUnavailableReplicas: ... [10:22:04] Deployment aya-llm-predictor-00006-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=aya-llm-predictor-00006-deployment - ... [10:22:04] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [10:54:14] (03CR) 10AikoChou: [C:03+1] locust: update revertrisk-wikidata load test results [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1207023 (https://phabricator.wikimedia.org/T406179) (owner: 10Kevin Bazira) [11:00:03] kevinbazira: o/ we have a revertrisk namespace, why don't we deploy the rrwikidata to that ns? the model is part of revertrisk family [11:01:21] (03CR) 10Kevin Bazira: [C:03+2] locust: update revertrisk-wikidata load test results [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1207023 (https://phabricator.wikimedia.org/T406179) (owner: 10Kevin Bazira) [11:01:50] (03Merged) 10jenkins-bot: locust: update revertrisk-wikidata load test results [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1207023 (https://phabricator.wikimedia.org/T406179) (owner: 10Kevin Bazira) [11:06:22] aiko: thanks for the review. I've updated the config to use the revertrisk namespace [11:19:46] thanks! could you also update the commit message? [11:23:29] (03CR) 10Nik Gkountas: "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206883 (https://phabricator.wikimedia.org/T410387) (owner: 10Nik Gkountas) [11:29:13] (03CR) 10Nik Gkountas: "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206409 (https://phabricator.wikimedia.org/T384485) (owner: 10Nik Gkountas) [12:01:19] going to deploy rrwikidata in prod ... [12:33:06] pods up and running in prod: https://phabricator.wikimedia.org/P85386 [14:22:04] FIRING: KubernetesDeploymentUnavailableReplicas: ... [14:22:04] Deployment aya-llm-predictor-00006-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=aya-llm-predictor-00006-deployment - ... [14:22:04] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [15:11:35] (03PS1) 10Bartosz Wójtowicz: revise-tone-task-generator: Adapt code to page_change events. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1207175 (https://phabricator.wikimedia.org/T408538) [15:11:36] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Action-API, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: The filter damaging=verylikelybad not availble for API:Feedrecentchanges - https://phabricator.wikimedia.org/T410435#11388520 (10BPirkle) [16:13:37] (03CR) 10Sbisson: [C:04-1] "Looks good. Just a small thing about the tests." [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1206883 (https://phabricator.wikimedia.org/T410387) (owner: 10Nik Gkountas) [16:21:28] (03PS1) 10Sbisson: Log repr() of exceptions [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1207197 [18:22:04] FIRING: KubernetesDeploymentUnavailableReplicas: ... [18:22:04] Deployment aya-llm-predictor-00006-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=aya-llm-predictor-00006-deployment - ... [18:22:04] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [19:30:30] (03CR) 10Eamedina: [C:03+2] Log repr() of exceptions [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1207197 (owner: 10Sbisson) [19:31:07] (03Merged) 10jenkins-bot: Log repr() of exceptions [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1207197 (owner: 10Sbisson) [22:22:04] FIRING: KubernetesDeploymentUnavailableReplicas: ... [22:22:04] Deployment aya-llm-predictor-00006-deployment in llm at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=llm&var-deployment=aya-llm-predictor-00006-deployment - ... [22:22:04] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas