[00:08:44] 06cloud-services-team, 10Catalyst (Kiwen): Grafana.wmcloud.org has project alerts for catalyst, route alerts catalyst/patchdemo maintainers - https://phabricator.wikimedia.org/T385330#10533206 (10jeena) We found a potential way to use grafana as mentioned in @EBomani's comment above, but this might not be reco... [03:18:12] (03open) 10samwilson: Update Translate user rights [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/1 [09:20:31] FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-5 is lagging behind the primary, the current lag is 3666 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [10:55:21] 06cloud-services-team, 10Catalyst (Kiwen): Grafana.wmcloud.org has project alerts for catalyst, route alerts catalyst/patchdemo maintainers - https://phabricator.wikimedia.org/T385330#10533579 (10taavi) >>! In T385330#10533206, @jeena wrote: > We found a potential way to use grafana as mentioned in @EBomani's... [11:10:21] 06cloud-services-team, 10Toolforge: [toolsdb] tools-db-4 switched to read-only - https://phabricator.wikimedia.org/T385900#10533596 (10fnegri) I don't have much time today, but I just checked and the primary hasn't crashed since the last restart at 22:54 UTC. The replica (tools-db-5) is lagging, but replicati... [11:13:28] 06cloud-services-team, 10Toolforge: [toolsdb] tools-db-4 switched to read-only - https://phabricator.wikimedia.org/T385900#10533598 (10fnegri) I/O in the primary is still much higher than usual, which is concerning: {F58376946} [12:00:10] 10Tools: PetScan returns "This web service cannot be reached" - https://phabricator.wikimedia.org/T384464#10533604 (10M2k_dewiki) Hello, since yesterday https://petscan.wmcloud.org/ again returns "This web service cannot be reached" Thanks a lot! [12:30:31] RESOLVED: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-5 is lagging behind the primary, the current lag is 3845 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [14:13:46] 10Tool-openstack-browser: Openstack Browser lists a user a user that horizon doesn't - https://phabricator.wikimedia.org/T382152#10533619 (10taavi) Yeah, seems like the usual case of accounts that had project memberships before being disabled in LDAP. Keystone has a filter for those that was originally meant to... [14:26:32] 06cloud-services-team, 10Toolforge (Toolforge iteration 17): [jobs-api] move jobs.toolforge.org/* labels to annotations - https://phabricator.wikimedia.org/T385904#10533622 (10Raymond_Ndibe) close this because it doesn't completely solve the problem I am facing [14:26:53] 06cloud-services-team, 10Toolforge (Toolforge iteration 17): [jobs-api] move jobs.toolforge.org/* labels to annotations - https://phabricator.wikimedia.org/T385904#10533623 (10Raymond_Ndibe) 05Open→03Declined [14:47:19] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10533629 (10DWYoungDLS) |**Wikitech account/LDAP:**| Hotaru Natsumi | |**SUL account**| Hotaru Natsumi | |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** | Y | |**I have visited... [15:27:28] (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) [16:03:31] FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-5 is lagging behind the primary, the current lag is 3676 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [17:29:12] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] (diff_job_runtime_method) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [18:24:16] 06cloud-services-team, 10Toolforge: [toolsdb] tools-db-4 switched to read-only - https://phabricator.wikimedia.org/T385900#10533760 (10fnegri) I/O in the primary is back to normal: {F58378449} The replica caught up but then started lagging again: {F58378453} `SHOW PROCESSLIST` shows it's the same tool as bef... [19:22:20] (03open) 10gam3: Draft: Gam3/dutch verbs [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/2 [19:27:31] (03update) 10gam3: Draft: Gam3/dutch verbs [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/2