[04:04:19] (03CR) 10Thcipriani: [C: 03+2] makedeploynotes.py: Use authors, not committers [tools/release] - 10https://gerrit.wikimedia.org/r/503126 (owner: 10Jforrester) [04:04:52] (03Merged) 10jenkins-bot: makedeploynotes.py: Use authors, not committers [tools/release] - 10https://gerrit.wikimedia.org/r/503126 (owner: 10Jforrester) [06:26:04] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: (Service Check Timed Out) [06:40:53] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [09:44:06] 10Phabricator, 10Project-Admins: Move #tool-extjsonuploader to a subproject of #tools - https://phabricator.wikimedia.org/T221540 (10Aklapper) p:05Triage→03Low [09:44:38] 10Phabricator, 10Project-Admins: Move #tool-extjsonuploader to a subproject of #tools - https://phabricator.wikimedia.org/T221540 (10Aklapper) [11:51:33] 10Phabricator, 10Project-Admins, 10Release-Engineering-Team (Next), 10Documentation: Document how to convert projects into subprojects/milestones etc - https://phabricator.wikimedia.org/T221112 (10Aklapper) [11:53:15] 10Phabricator, 10User-Ladsgroup: Add task to D3r1ck01's workboard when D3r1ck01 gets assigned to the task - https://phabricator.wikimedia.org/T221479 (10Aklapper) [13:18:31] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:24:32] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10akosiaris) >>! In T218750#5126229, @Urbanecm wrote: >> >> The same could be said of the ssh key. Temporary access to a browser session woul... [13:32:02] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10Paladox) Gerrit sends a email if a ssh key is added (not sure about if it's removed). Gerrit does not notify if a http password has changed... [15:31:26] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10Paladox) I've done 2 changes https://gerrit-review.googlesource.com/c/gerrit/+/221772/ and https://gerrit-review.googlesource.com/c/gerrit/+... [17:58:53] 10Phabricator, 10Mobile: LDAP and search icon doesn't let you just tap on it - https://phabricator.wikimedia.org/T221557 (10Reedy) [18:09:53] 10Phabricator, 10Mobile: LDAP and search icon doesn't let you just tap on it - https://phabricator.wikimedia.org/T221557 (10Paladox) This happens on iOS too (and has for years), i thought that it was a bug introduced into iOS. [18:27:14] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Add security sensitive nodes to our kubernetes cluster - https://phabricator.wikimedia.org/T220821 (10ayounsi) [18:37:16] 10Phabricator, 10Project-Admins: Move #tool-extjsonuploader to a subproject of #tools - https://phabricator.wikimedia.org/T221540 (10Tgr) [18:41:22] 10Phabricator, 10Project-Admins: Move #tool-extjsonuploader to a subproject of #tools - https://phabricator.wikimedia.org/T221540 (10Tgr) I don't see any way to do that, I guess you'd have to be a phab admin? But FWOW #tools has plenty of projects called `tool-XXX`; only the more popular ones seem to have unp... [18:41:44] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Add security sensitive nodes to our kubernetes cluster - https://phabricator.wikimedia.org/T220821 (10akosiaris) 05Open→03Resolved kubernetes1005, kubernetes1006, kubernetes2005, kubernetes2006 added with specific... [18:41:48] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10akosiaris) [18:50:11] 10Phabricator, 10Mobile: LDAP and search icon doesn't let you just tap on it - https://phabricator.wikimedia.org/T221557 (10Aklapper) [18:50:13] 10Phabricator (Upstream), 10Mobile, 10Upstream: Toggle buttons in Phabricator don't work on mobile (affects: Login, Search) - https://phabricator.wikimedia.org/T201480 (10Aklapper) [18:50:36] 10Phabricator, 10Mobile: LDAP and search icon doesn't let you just tap on it - https://phabricator.wikimedia.org/T221557 (10Aklapper) See {T201480}. For future reference, please follow https://www.mediawiki.org/wiki/How_to_report_a_bug and include basic steps to reproduce (URL). Thanks [19:10:24] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10akosiaris) I did some benchmarking and here's some first (rather impressive numbers) for kask This is with 750 simul... [19:27:26] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10akosiaris) Just for posterity's sake, at ~1500 artificially simulated users the service started to crumble and starte... [20:25:31] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [20:33:19] 10Phabricator, 10Project-Admins, 10Release-Engineering-Team (Next), 10Documentation: Document how to convert projects into subprojects/milestones etc - https://phabricator.wikimedia.org/T221112 (10Aklapper) >>! In T221112#5121789, @mmodell wrote: > A better documented version of the script is now available... [20:34:17] 10Phabricator, 10Project-Admins, 10Release-Engineering-Team (Next), 10Documentation, 10Patch-For-Review: Document how to convert projects into subprojects/milestones etc - https://phabricator.wikimedia.org/T221112 (10Aklapper) [20:38:37] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:19:03] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:26:51] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:54:21] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 46.15% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:04:47] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1