[11:30:11] (03PS1) 10Alexandros Kosiaris: maps: Add tileratorui passes [labs/private] - 10https://gerrit.wikimedia.org/r/247811 [11:31:03] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] maps: Add tileratorui passes [labs/private] - 10https://gerrit.wikimedia.org/r/247811 (owner: 10Alexandros Kosiaris) [11:34:57] 6Labs, 10Tool-Labs: 'become' broken in Tool Labs ("sudo: a password is required") - https://phabricator.wikimedia.org/T116148#1741869 (10russblau) 3NEW [11:38:27] 6Labs, 10Tool-Labs: 'become' broken in Tool Labs ("sudo: a password is required") - https://phabricator.wikimedia.org/T116148#1741869 (10Luke081515) My tools are affected too, so I guess all tools are affected. [11:45:05] oh my [11:45:20] good that i have 4 terminals open already … [12:32:48] Uh, Somethings broken, Im getting an error when I try to become my tool [12:32:59] sudo: a password is required [12:33:11] same here [12:33:12] I concur, sudo is not working for me on any instance in my project. It worked fine this morning [12:34:31] Hm. [12:34:50] 6Labs, 10Tool-Labs: 'become' broken in Tool Labs ("sudo: a password is required") - https://phabricator.wikimedia.org/T116148#1741995 (10valhallasw) I also cannot sudo (to root), which suggests there's an issue with LDAP, but `ldaplist -l servicegroups` works without issues. Unfortunately, I cannot ssh in as... [12:35:45] What val just said backs up what I'm thinking actually. [12:38:30] I also concur! [12:40:40] 6Labs, 10Tool-Labs: Allow scfc and valhallasw to ssh in as root - https://phabricator.wikimedia.org/T116156#1742020 (10valhallasw) 3NEW [13:01:22] 6Labs, 10Tool-Labs, 5Patch-For-Review: 'become' broken in Tool Labs ("sudo: a password is required") - https://phabricator.wikimedia.org/T116148#1742087 (10akosiaris) puppet ran with the only change being: ``` -SUDOERS_BASE ou=sudoers,cn=tools,ou=projects,dc=wikimedia,dc=org \ No newline at end of file +... [13:03:39] 6Labs, 10Tool-Labs, 5Patch-For-Review: 'become' broken in Tool Labs ("sudo: a password is required") - https://phabricator.wikimedia.org/T116148#1742094 (10hashar) ``` hashar@tools-bastion-01:~$ sudo -niu tools.morebots sudo: a password is required ``` Funilly at 13:00:04 puppet ran: ``` Oct 21 13:00:04 too... [13:04:05] akosiaris: the nscd reload did not fix it apparently [13:04:37] I am not sure how puppet added the newline at the end of /etc/ldap/ldap.conf [13:05:28] ah https://gerrit.wikimedia.org/r/#/c/247826/1/modules/ldap/templates/open_ldap.erb [13:05:29] .. [13:05:38] hashar: that would be https://gerrit.wikimedia.org/r/247826 [13:05:51] there was some speculation that might be the reason [13:05:57] (03PS1) 10Addshore: Add wikidata/* to wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/247831 [13:06:32] akosiaris, hashar, thanks for looking into it [13:07:32] is there anything useful in /var/log/auth.log? As mentioned, I cannot sudo to root and I can't ssh in as root directly :( [13:07:34] (03PS1) 10Addshore: Add WMDE-Analytics-Engineering to #wikimedia-de-tech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/247832 [13:08:30] akosiaris: I don't get it. that change is the only one that has had any possible impact on labs within the past 24 hours. if this issue was older than 24 hours, it would have been picked up already :? [13:08:44] valhallasw`cloud: I don't have access [13:09:07] that was what I commented- the reload potentially applying a different, older change [13:09:20] jynus: hm, true [13:10:04] deployment-bastion asks for a sudo password, and because I run mailman on its own puppet master, it has not been affected. nice nail down at least [13:10:04] Oct 21 13:04:48 tools-bastion-01 sudo: pam_unix(sudo:auth): auth could not identify password for [valhallasw] [13:10:05] Oct 21 13:04:48 tools-bastion-01 sudo: valhallasw : user NOT authorized on host ; TTY=pts/52 ; PWD=/home/valhallasw ; USER=root ; COMMAND=/bin/su [13:11:43] valhallasw`cloud: I have sudoed as you in tools-bastion-01 and trying btw [13:11:51] ok! [13:12:00] akosiaris: doesn't ldap have anything in its logs? since we're leaning on it being an ldap issue atm? [13:12:39] so compared to toolsbeta-bastion, BINDDN and BINDPW are missing from /etc/sudo-ldap.conf [13:12:53] what ? [13:13:02] that should not have been touched [13:13:49] argh, it's a symlink [13:14:07] https://github.com/wikimedia/operations-puppet/commit/fba5006123579909681ca10fdf176d4c8ad4e2f2 ? [13:14:18] OpenLDAP does not honor BINDDN in ldap.conf. It will only honor it in .ldaprc files and we don't populate these right now. [13:14:34] yes, but OpenLDAP is not pam_ldap [13:14:39] pam_ldap honors those settings [13:15:00] ahhh [13:15:34] although that's sudo-ldap not pam_ldap anyway [13:15:41] plus there are 2 pam_ldaps IIRC [13:16:11] we use this one https://gerrit.wikimedia.org/r/247826 [13:16:14] http://arthurdejong.org/nss-pam-ldapd/ [13:16:16] ^ [13:16:47] ok, full backpedal, I 'll fully revert the change [13:17:19] you can also first apply manually on tools-bastion to check if that's indeed the cause [13:18:35] yeah, already did [13:18:40] that's obviously the issue [13:18:56] and whoever merged those 2 configurations with a symlink should not have done that ... sigh [13:19:00] and I should have noticed [13:25:52] I'm a bit surprised ldaplist still functioned, though. That uses ldapsupportlib.py which also gets the password from /etc/ldap/ldap.conf [13:26:50] 6Labs, 10Tool-Labs, 5Patch-For-Review: 'become' broken in Tool Labs ("sudo: a password is required") - https://phabricator.wikimedia.org/T116148#1742141 (10coren) Works for me: ```marc@tools-bastion-01:~$ sudo -niu tools.csbot tools.csbot@tools-bastion-01:~$ ``` Indeed, sudo -l confirms that ```[...] User... [13:27:42] Coren: ^ [13:28:26] 6Labs, 10Tool-Labs, 5Patch-For-Review: 'become' broken in Tool Labs ("sudo: a password is required") - https://phabricator.wikimedia.org/T116148#1742153 (10akosiaris) https://gerrit.wikimedia.org/r/#/c/247834/ submitted and merged. Applying it for all labs VMs will take a bit of time but tools-bastion-01 see... [13:28:48] 6Labs, 10Tool-Labs, 5Patch-For-Review: 'become' broken in Tool Labs ("sudo: a password is required") - https://phabricator.wikimedia.org/T116148#1742155 (10valhallasw) 5Open>3Resolved a:3valhallasw @akosiaris has already fixed it ;-) See https://gerrit.wikimedia.org/r/#/c/247834/ [13:29:29] akosiaris: thanks for the help :-) [13:29:45] valhallasw`cloud: I created the problem, don't thank me [13:30:11] also ldapsupportlib.py is defaulting to /etc/ldap.conf [13:30:18] which I made sure to not touch [13:30:33] and only uses /etc/ldap/ldap.conf as a fallback [13:30:35] akosiaris: Hah. It'd figure that akosiaris would merge a fix /just/ before I test and start boggling why it seems to be working. :-) [13:30:40] aaaah. Yes, you're right :-) [13:31:16] tbh, this was a weird bug [13:31:21] I was always able to sudo [13:31:27] presumably because I am ops [13:31:37] yes, there's a specific entry in /etc/sudoers for ops [13:31:41] sudoers.d [13:31:51] for a reason :-) [13:31:53] which lead us completely off trail trying to figure out the per project problems [13:32:00] akosiaris: Ah, right, ops has its own magic so my original test was invalid (though the sudo -l would have shown the issue) [13:33:38] so, I 'll submit a comment fix up for that, but to be honest, that situations with /etc/ldap.conf /etc/ldap/ldap.conf and /etc/sudo-ldap.conf is confusing to say the least [13:34:09] and I made sure to not touch /etc/ldap.conf cause it was nss_ldap and would cause problems but that sudo-ldap.conf symlink I did not think of [14:07:53] 6Labs, 10Tool-Labs, 5Patch-For-Review: 'become' broken in Tool Labs ("sudo: a password is required") - https://phabricator.wikimedia.org/T116148#1742359 (10zhuyifei1999) a:5valhallasw>3akosiaris [14:15:16] 6Labs, 10Tool-Labs: Mail is not delivered - https://phabricator.wikimedia.org/T116176#1742380 (10scfc) 3NEW [14:21:01] 6Labs, 10Tool-Labs: Mail is not delivered - https://phabricator.wikimedia.org/T116176#1742417 (10scfc) NB: My test of `localuser scfc` should have been `localuser tools scfc`: ``` root@tools-mail:~# localuser tools scfc ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) root@tools-mail:~# ``` [14:32:53] 6Labs, 10Tool-Labs: Mail is not delivered - https://phabricator.wikimedia.org/T116176#1742458 (10akosiaris) ``` root@tools-mail:~# localuser tools scfc tim@tim-landscheidt.de ``` This seems to work fine. Since it uses /etc/ldap.conf and not /etc/ldap/ldap.conf should not have been impacted by rOPUPfba50061235... [14:53:45] 6Labs, 10Tool-Labs: Mail is not delivered - https://phabricator.wikimedia.org/T116176#1742490 (10scfc) 5Open>3Resolved a:3scfc Indeed it works now and the queue is empty. Hmmm. [14:53:53] 6Labs, 10Tool-Labs: Mail is not delivered - https://phabricator.wikimedia.org/T116176#1742493 (10scfc) a:5scfc>3None [16:29:39] 6Labs, 10Labs-Infrastructure, 10Wikimedia-Apache-configuration, 6operations, and 2 others: wikitech-static sync broken - https://phabricator.wikimedia.org/T101803#1742711 (10Andrew) 5Open>3Resolved All green, and I've made the test less touchy. So, closing this bug for now. [18:31:29] (03CR) 10Legoktm: [C: 032] Add WMDE-Analytics-Engineering to #wikimedia-de-tech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/247832 (owner: 10Addshore) [18:31:44] (03Merged) 10jenkins-bot: Add WMDE-Analytics-Engineering to #wikimedia-de-tech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/247832 (owner: 10Addshore) [18:31:49] cheers legoktm [18:32:29] !log tools.wikibugs Updated channels.yaml to: b4e285f9673929b8547902be466e04d903d3237d Add WMDE-Analytics-Engineering to #wikimedia-de-tech [18:32:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [21:38:19] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-111: Labs virt capacity expansion - https://phabricator.wikimedia.org/T107624#1743987 (10RobH)