[00:02:49] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.26% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [01:02:49] RESOLVED: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.26% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [01:18:20] (03open) 10raymond-ndibe: [cli] support start and end params in logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/145 (https://phabricator.wikimedia.org/T400917) [01:22:34] (03open) 10raymond-ndibe: support start and end params in logs endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/267 (https://phabricator.wikimedia.org/T400917) [01:32:09] (03update) 10raymond-ndibe: support start and end params in logs endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/267 (https://phabricator.wikimedia.org/T400917) [01:50:14] (03update) 10raymond-ndibe: support start and end params in logs endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/267 (https://phabricator.wikimedia.org/T400917) [01:56:46] (03update) 10raymond-ndibe: support start and end params in logs endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/267 (https://phabricator.wikimedia.org/T400917) [01:59:11] (03open) 10raymond-ndibe: support start and end params in logs endpoints [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T400917) [02:01:36] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api] Allow customizing time to request Loki logs for - https://phabricator.wikimedia.org/T400917#11625954 (10Raymond_Ndibe) 05Open→03In progress [02:01:41] 10Toolforge (Toolforge iteration 25): jobs-api: omitting filelog field from jobs creation payload completely breaks jobs-api for the tool - https://phabricator.wikimedia.org/T417518#11625957 (10Raymond_Ndibe) 05Open→03In progress [08:31:26] 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736 (10Nemoralis) 03NEW [08:47:58] 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11626453 (10Aklapper) Hi, per https://meta.wikimedia.org/wiki/LimeSurvey we offer https://wikimediafoundation.limesurvey.net/ Why would you want to duplicate (?) the existing installation? [08:51:53] 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11626467 (10Nemoralis) >>! In T417736#11626453, @Aklapper wrote: > Hi, per https://meta.wikimedia.org/wiki/LimeSurvey we offer https://wikimediafoundation.limesurvey.net/ > Why would you want t... [08:59:29] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 13Patch-For-Review: Add tracing to understand Toolforge and CloudVPS usage and dependencies - https://phabricator.wikimedia.org/T399313#11626502 (10Volans) a:03Volans [09:01:04] (03PS1) 10Volans: wmcs: infra-tracing-nfs update hiera keys [labs/private] - 10https://gerrit.wikimedia.org/r/1240214 (https://phabricator.wikimedia.org/T399313) [09:34:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:59:08] (03CR) 10Filippo Giunchedi: [C:03+1] wmcs: infra-tracing-nfs update hiera keys [labs/private] - 10https://gerrit.wikimedia.org/r/1240214 (https://phabricator.wikimedia.org/T399313) (owner: 10Volans) [10:21:24] 06tools-infrastructure-team, 10CAS-SSO, 06Infrastructure-Foundations: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755 (10taavi) 03NEW [10:22:29] 06tools-infrastructure-team, 10CAS-SSO, 06Infrastructure-Foundations: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755#11626903 (10taavi) Seeing errors like: ` Feb 18 10:20:53 cloudidp2001-dev tomcat10[535471]: 2026-02-18 10:20:53,011 WARN [org.apereo.cas.util.func... [10:29:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [10:33:11] 06tools-infrastructure-team, 10CAS-SSO, 06Infrastructure-Foundations: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755#11626962 (10taavi) 05Open→03Resolved Fixed with d5611fc9f in private-puppet. [10:35:44] 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 06Traffic, 13Patch-For-Review: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11626966 (10ayounsi) On the netops side the main advantage is to not require real servers to use public I... [10:45:21] 06tools-infrastructure-team, 10CAS-SSO, 06Infrastructure-Foundations: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755#11627006 (10taavi) 05Resolved→03Open Eh, no, now it's complaining about: ` Feb 18 10:44:51 cloudidp2001-dev tomcat10[617265]: 2026-02-18 10:44... [11:22:17] FIRING: JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:27:17] RESOLVED: JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:32:26] 06tools-infrastructure-team, 10CAS-SSO, 06Infrastructure-Foundations: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755#11627197 (10taavi) Turning CAS logging up at least reveals which key it is complaining about: ` Feb 18 11:27:32 cloudidp2001-dev tomcat10[621906]:... [11:36:30] 06tools-infrastructure-team, 10CAS-SSO, 06Infrastructure-Foundations: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755#11627227 (10taavi) 05Open→03Resolved Replacing that key with an another one of the exact same length made CAS let me log in again to Horizon. [11:37:34] (03CR) 10Volans: [V:03+2 C:03+2] wmcs: infra-tracing-nfs update hiera keys [labs/private] - 10https://gerrit.wikimedia.org/r/1240214 (https://phabricator.wikimedia.org/T399313) (owner: 10Volans) [11:43:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [11:53:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [11:55:15] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, and 2 others: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp2001-dev - https://phabricator.wikimedia.org/T410294#11627269 (10taavi) 05Open→03Resolved This seems done so boldly closing [11:55:34] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Deleting ENC projects is broken - https://phabricator.wikimedia.org/T416588#11627272 (10taavi) p:05Triage→03Medium [12:02:49] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.27% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [12:04:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [12:09:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [12:18:49] 06cloud-services-team, 10Cloud-VPS: networking.service in failed state on Trixie cloudgw nodes - https://phabricator.wikimedia.org/T417075#11627399 (10taavi) `lang=shell-session taavi@cloudgw2003-dev ~ $ sudo ifup -v vlan2107 ifup: parsing file /etc/network/interfaces.d/cloudgw ifup: configuring interface vla... [12:33:38] 10Tools, 10Wikidata, 07TestMe: ListeriaBot returns "Last line: ERROR: Login failed" - https://phabricator.wikimedia.org/T329380#11627421 (10matej_suchanek) [13:21:21] 10Cloud-VPS (Quota-requests): Temporary instance increase for Bullseye servers deprecation - https://phabricator.wikimedia.org/T416803#11627578 (10dcaro) Thanks for updating it! +1 from me Each instance comes with it's own space for the OS and such, it's not counted in the volumes size (that is only for extern... [13:50:38] 06cloud-services-team, 10Cloud-VPS: networking.service in failed state on Trixie cloudgw nodes - https://phabricator.wikimedia.org/T417075#11627747 (10taavi) 05Open→03Resolved [14:09:03] 10Tool-redminbot: Grab list of modules from wiki replicas instead of file - https://phabricator.wikimedia.org/T417783 (10Redmin) 03NEW [14:46:06] 06cloud-services-team, 10Toolforge: Toolforge NFS tracing misses some dumps events - https://phabricator.wikimedia.org/T415199#11627992 (10Volans) >>! In T415199#11627619, @dcaro wrote: > For the paths with `..`, we are still catching the ones that have it after the mount prefix right? (like `/mnt/nfs/home/dat... [15:02:57] 06cloud-services-team, 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T415829#11628059 (10taavi) [15:03:08] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T415828#11628062 (10taavi) [15:03:51] 06tools-infrastructure-team, 10Datasets-General-or-Unknown: Reclaim public IPs from individual dumps distribution (clouddumps) hosts - https://phabricator.wikimedia.org/T417028#11628064 (10taavi) p:05Triage→03Low [15:03:56] 06cloud-services-team, 10Toolforge: [trove] MariaDB 10.6 support - https://phabricator.wikimedia.org/T417494#11628065 (10fnegri) p:05Triage→03Medium [15:04:25] 06cloud-services-team, 10Cloud-VPS: [trove] MariaDB 10.6 support - https://phabricator.wikimedia.org/T417494#11628066 (10taavi) [15:04:39] 06cloud-services-team, 10Cloud-VPS, 07Documentation: Document how to add SSH keys in codfw1dev LDAP deployment - https://phabricator.wikimedia.org/T416568#11628067 (10taavi) p:05Triage→03Low [15:05:35] 06cloud-services-team, 10Toolforge: Unhandled ChunkedEncodingError crash from `toolforge jobs logs -f` - https://phabricator.wikimedia.org/T417225#11628070 (10taavi) p:05Triage→03Medium [15:05:46] 06cloud-services-team, 10Cloud-VPS: Deleting a project should delete associated sudo rules in LDAP - https://phabricator.wikimedia.org/T416497#11628071 (10taavi) p:05Triage→03Medium [15:10:04] 06cloud-services-team, 10Wikidata, 07ContentSecurityPolicy: CSP violation using CloudVPS endpoint on Wikidata - https://phabricator.wikimedia.org/T416994#11628097 (10taavi) [15:10:25] 06cloud-services-team, 10Cloud-VPS: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11628101 (10taavi) [15:12:12] 06cloud-services-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11628104 (10taavi) [15:13:37] 06cloud-services-team, 10Toolforge, 07Documentation: Create a "my first React app" tutorial for Toolforge - https://phabricator.wikimedia.org/T231950#11628109 (10taavi) p:05Triage→03Medium [15:13:45] 06cloud-services-team, 10Data-Services, 07Documentation: Revise and reformat Portal:Data_Services - https://phabricator.wikimedia.org/T348024#11628113 (10taavi) p:05Triage→03Medium [15:26:50] 06cloud-services-team, 10Cloud-VPS, 07Puppet: Prune old reports from /var/lib/puppetserver/server_data/facts on Cloud VPS puppet servers - https://phabricator.wikimedia.org/T417795 (10taavi) 03NEW p:05Triage→03High [15:28:16] 06cloud-services-team, 10Cloud-VPS, 07Puppet: Prune old reports from /var/lib/puppetserver/server_data/facts on Cloud VPS puppet servers - https://phabricator.wikimedia.org/T417795#11628178 (10taavi) [15:29:10] 06cloud-services-team, 10Wikidata, 07ContentSecurityPolicy: CSP violation using CloudVPS endpoint on Wikidata - https://phabricator.wikimedia.org/T416994#11628182 (10fnegri) 05Open→03Invalid @NathanGavenski as @Lucas_Werkmeister_WMDE explained above, I think the warning you see is to be expected, and... [15:40:13] 10Tool-toolwatch: Toolwatch incorrectly reports non-web tools as unavailable - https://phabricator.wikimedia.org/T376451#11628223 (10TheresNoTime) a:05TheresNoTime→03None [15:52:52] 06cloud-services-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11628283 (10fnegri) > We can do the DB synchronisation using a user's private key to access the Cloud VPS machine/database. Is this actually allowed and would there be a user-independent solution to do t... [16:00:38] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review, 07Puppet: Prune old reports from /var/lib/puppetserver/server_data/facts on Cloud VPS puppet servers - https://phabricator.wikimedia.org/T417795#11628332 (10taavi) 05Open→03Resolved [16:03:04] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.48% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [16:05:36] (03PS1) 10Majavah: Add toolsbeta-acme-chief private key [labs/private] - 10https://gerrit.wikimedia.org/r/1240325 [16:05:36] (03PS1) 10Majavah: Add fake metricsinfra Grafana admin password [labs/private] - 10https://gerrit.wikimedia.org/r/1240326 [16:12:19] 10Cloud-VPS (Quota-requests): Temporary instance increase for Bullseye servers deprecation - https://phabricator.wikimedia.org/T416803#11628381 (10YochayCO) Ok cool! Do you know when it will be done? I'm a little worried about the wiki tech supporters shutting down my servers on 1.3 (though I think they won't, I... [17:15:17] 10Tool-curator: Handle large Mapillary sequences - https://phabricator.wikimedia.org/T412617#11628665 (10DaxServer) 05In progress→03Resolved [17:17:00] 10Tool-curator: Save presets in Curator - https://phabricator.wikimedia.org/T417683#11628679 (10DaxServer) p:05Triage→03Medium [20:03:04] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.65% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [20:51:32] 06cloud-services-team, 10Quarry: Add a possibility to delete a draft - https://phabricator.wikimedia.org/T135908#11629717 (10bd808) [21:48:28] 06cloud-services-team, 10Wikidata, 07ContentSecurityPolicy: CSP violation using Cloud VPS endpoint on Wikidata - https://phabricator.wikimedia.org/T416994#11629909 (10bd808) [23:40:32] 06cloud-services-team, 10Toolforge: Your template file contains unknown keys - https://phabricator.wikimedia.org/T417843 (10Huji) 03NEW [23:40:33] 06cloud-services-team, 10Toolforge: Your template file contains unknown keys - https://phabricator.wikimedia.org/T417843#11630318 (10Huji) Looks similar to T380141 [23:42:44] 06cloud-services-team, 10Toolforge: Your template file contains unknown keys - https://phabricator.wikimedia.org/T417843#11630321 (10bd808) That service.template looks like a point in time copy of a service.manifest. Those to files are related, but they use different schemas. [23:44:05] 06cloud-services-team, 10Toolforge: Your template file contains unknown keys - https://phabricator.wikimedia.org/T417843#11630322 (10bd808) https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Webservice_templates > A webservice template file is a YAML document. It can contain these settings: > * type: the t... [23:47:26] 06cloud-services-team, 10Toolforge: Your template file contains unknown keys - https://phabricator.wikimedia.org/T417843#11630324 (10bd808) A service.template matching the intent of the service.manifest format file given would be: `lang=yaml,name=$HOME/service.template type: php7.3 backend: kubernetes ` [23:48:00] 06cloud-services-team, 10Toolforge: Your template file contains unknown keys - https://phabricator.wikimedia.org/T417843#11630326 (10bd808) 05Open→03Invalid