[10:17:49] !log bd808@tools-bastion-14 tools.etherpad-backup Built and deployed image from ecf5892c (T422958) [10:17:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.etherpad-backup/SAL [10:33:30] Can someone please approve https://toolsadmin.wikimedia.org/tools/membership/status/2206 ? I seem to have lost the rights at some point [10:35:45] i guess there is an "anonymous" option in the same page where u edit rights of group administrators (re @GroupAnonymousBot: Am I in some magic sudo mode) [10:36:54] oh it has been addressed. sorry! [10:44:05] done (re @tehreedy: Can someone please approve https://toolsadmin.wikimedia.org/tools/membership/status/2206 ? I seem to have lost the rights at som...) [10:44:13] ❤️ [12:27:22] can someone help me with getting access? I got my account (carandraug) approved, the ssh key added (~2h ago), yet ssh fails. ssh -v shows that it's sending the correct key, yet it fails to connect [12:28:39] log at https://bpa.st/TXDA [12:39:22] carandraug: yes I'll take a look [12:43:10] carandraug: the public key on record is the one ending in 8GtcUh/b I'm not sure which one of the keys in the log is the correct one, can you try sending only the right key with -i ? [12:45:28] actually the fingerprint is SHA256:ivoF9rd9gIXRcnESCOG0j7CFIm72TMLYSdxz+qiVvdU for the ldap key so that should work with id_rsa [12:53:00] carandraug: someone or something put your user in project bastion so now access should work [13:03:15] !log bd808@tools-bastion-14 tools.etherpad-backup Built image from bab09762 and deployed (T422958) [13:03:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.etherpad-backup/SAL [13:15:17] godog: I got things fixed for carandraug. The problem was that their membership in the bastion group existed in keystone, but was missing from LDAP. My quick fix was to remove and then re-add them to in the bastion project via horizon. [13:28:08] How do I use log in the SAL for my tool? [13:42:32] @Gerges404: https://wikitech.wikimedia.org/wiki/Cloud_VPS_Server_Admin_Log#Add_from_Toolforge and https://wikitech.wikimedia.org/wiki/Cloud_VPS_Server_Admin_Log#Add_from_IRC [14:09:14] bd808: I'm sorry you keep having to chase after T379550, I do look at it off and on but at the moment I'm largely thwarted by T421911 [14:09:14] T379550: openstack: keystone may be failing to add users to the bastion project in Keystone and/or LDAP - https://phabricator.wikimedia.org/T379550 [14:09:14] T421911: Keystone logs no longer appearing in logstash - https://phabricator.wikimedia.org/T421911 [14:29:14] andrewbogott: just another one for uid=semantic where the keystone data was good, but project-bastion missing from ldap. [14:29:39] yeah, I have a theory for that one which I'm testing now although I don't have high hopes for reproducing [14:29:53] Have you seen the 'not even in keystone' version lately? [14:30:04] not today :) [14:38:10] bah, of course everything works while I'm watching [14:39:30] https://www.irccloud.com/pastebin/mHVs5EzD/ [14:40:20] andrewbogott: My guess is that this is like a 1-2% misfire [14:40:34] I guess we could try to run some sort of audit [14:40:51] I have an audit script for finding users who aren't in bastion but are in other projects. [14:41:09] But I don't think it checks ldap so that would be the next step for measuring today's problem [14:41:31] yeah. what I saw twice today was user in keystone, not in ldap [14:41:50] I continue to suspect this is either some race with writing to ldap, or something to do with paging... [14:42:17] both seem possible, yes [14:42:29] It does a full sync each time, so if a user was missed the first time they should get added on a subsequent run even if there's a race... have you ever noticed the problem fixing itself with time? [14:44:13] andrewbogott: I only have the two data points from today sadly. [14:44:58] one of them was a freshly added to any keystone project user (uid=semantic) [14:45:11] I guess another way of asking that is: can you think of a time when someone has had this issue has said "I created my account last week" rather than "I created my account a couple hours ago"? [14:45:17] I'm not sure about uid=carandraug [14:58:49] welp, I have added yet more noisy log messages, hopefully next time we find this issue I'll be able to tell what didn't happen. [15:05:33] !log bd808@tools-bastion-14 tools.etherpad-backup Built image from db847baa and deployed (T425111) [15:05:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.etherpad-backup/SAL [15:06:30] /me wonders aloud if there’s anyone at the hackathon who I could ask how push-to-deploy is going [15:07:26] francesco probably [15:13:15] mutante: is T240266 fixed? I still can't clone from Gerrit (re @wmtelegram_bot: probably https://phabricator.wikimedia.org/T240266) [15:39:07] Any Phabricator API? Can I find the tasks with most subscribers? [15:40:33] lucaswerkmeister: yes come talk to me! tl;dr slowly progressing [15:41:14] good to hear, I’ll see if I run into you at some point :) [15:45:19] @Yetkin Phabricator does have an API (https://phabricator.wikimedia.org/conduit/), but I don't know if there's an easy way to sort tasks by the number of subscribers they have (but other folks might have ideas) :) [15:54:25] imo no. there isn't a column recording the count of subscribers in the maniphest_task table, so finding such a task would have to do slow database queries (re @Yetkin: Any Phabricator API? Can I find the tasks with most subscribers?) [15:55:10] which database? [15:55:37] phabricator_maniphest [15:56:54] How can I query that? Quarry does not recognize it [16:01:01] I'm not aware of the Phab database having public replicas, so (if that is indeed the case) it might not be possible [16:01:15] IIUC there used to be a Phabricator dump, but (again IIUC) it was somewhat-recently removed in T417824 [16:01:26] T417824: Should we remove the Phabricator dump? - https://phabricator.wikimedia.org/T417824 [16:35:37] !log gergesshamon@tools-bastion-15 tools.wikimonitor setup [16:35:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor/SAL [18:15:38] !log jeanfred@tools-bastion-15 tools.integraality Deploy 19b856d, 473fbeb, 0cb0915 [18:15:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [18:16:46] !log jeanfred@tools-bastion-15 tools.integraality Deploy 01e97d6 (Extract grouping link strategies into GroupingLink hierarchy), 359770d (Remove pass-through methods for grouping link in GroupingConfiguration), 54e4a7b (Add lang parameter to get_label_for_variable), cccd661 (Simplify SPARQL for mul language in get_label_for_variable), 548383d (Support any label language for grouping links), d571596 (Support QID-based grouping links), cfd7c68 (Support for property-value-based grouping links), e7e6b83 (Extract SparqlGroupingLink intermediate class), 7b208f4 (Support external links in grouping link rendering) for T237276 [18:16:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL