[00:27:29] leloiandudu, um [00:27:36] what instance/project is this? [00:29:10] Krenair: fountain [00:29:18] https://tools.wmflabs.org/fountain/ [00:29:27] that's part of the tools project [00:29:32] I'm not touching that [00:29:42] Sorry [00:30:00] Krenair: ok, thanks! but am I on the right channel? [00:30:05] yes [00:30:11] thanks again :) [00:30:21] but it's not tools-specific so if you want tools-specific help you have to say so [00:30:41] I see... thanks [04:23:34] 10Tool-Labs-tools-Pageviews: Restrict Topviews to showing data only for individual days, weeks, or months - https://phabricator.wikimedia.org/T142403#2551505 (10MusikAnimal) p:05Normal>03High [04:24:13] 10Tool-Labs-tools-Pageviews: Improve Topviews interface - https://phabricator.wikimedia.org/T142802#2551507 (10MusikAnimal) p:05Triage>03Normal [04:53:04] leloiandudu: tool labs has tons of hosts, hacking a dir for one use case is not usually done. Besides, chmod ugo+rw looks like a security vulnerability to me (chmod 1777 is better but still argh...) Most likely you need to find someway to change that /etc/mono/registry to somewhere under your home dir. (sorry but not a Java person here) [04:55:56] (Java => ASP.NET, they look so similar o.O) [05:01:45] leloiandudu: http://lists.ximian.com/pipermail/mono-list/2008-July/038996.html [05:43:18] zhuyifei1999_: 1) I never asked for the chown, I specifically said "granting special rights shouldn't be required". The directory just should be there [05:44:32] IMHO, it shouldn't [05:44:33] zhuyifei1999_: 2) thanks for your link, how do I set the env var for lighttpd? [05:44:36] I don't see why registry should be shared globally on a shared hosting environment [05:45:27] lighttpd? I'm not a lightttpd expert either. search google? [05:46:12] zhuyifei1999_: I agree, this is stupid. but for mono to work the (empy) dir just should be there. The framework I use for my tool isn't going to write there (thus, no chmod is required) [05:47:14] please file a ticket http://phabricator.wikimedia.org/ [05:47:47] zhuyifei1999_: thanks, I'll try the env var approach first. if it is not working - I will file a ticket [05:47:50] thanks again :) [05:48:00] np [06:59:01] valhallasw`vecto, Back in Feb, I submitted a task (https://phabricator.wikimedia.org/T127633) to install a package on exec servers. The package has its .deb repo so it should be very straightforward. It's a morphological analyser of Polish words, potentially useful for Wiktionarians (but also others who do NLP). Is there any way of bumping the task up? I'd help if only I could [07:09:13] After several cycles of doing 'webservice start' I now get this error: http://pastebin.com/Cpp3hM4m Something about permssions. I didn't change anything. What do I do now? [07:35:29] suddenly it works again. very weird [08:22:20] 06Labs, 10Tool-Labs: install Morfeusz (morphological analyser) and Python bindings - https://phabricator.wikimedia.org/T127633#2049173 (10valhallasw) Unfortunately, we don't install packages from ppa's due to security concerns (the author of a ppa effectively gets root access). However, as far as I can see,... [08:27:24] leloiandudu: could you file a bug explaining what happened, and where this error showed? [08:27:53] as well as which tool user? [08:29:47] leloiandudu: from the looks of it, something is wrong with /var... somewhere [08:40:14] valhallasw`vecto, thanks for looking at the task. Should I just google for installing .egg packages and try on virtualenv, or is there a recommended way to do it on tools? [08:40:46] Alkamid: pip should be able to install them [10:16:25] leloiandudu: and re: mono; https://bugzilla.xamarin.com/show_bug.cgi?id=26024 suggests you could set the MONO_REGISTRY_PATH environment variable [10:16:59] I don't know mono well enough to judge the security implications of creating a writable-for-all directory there [10:23:19] leloiandudu: for questions such as those, please create a task in phabricator; that makes it easier to keep track of requests and responses for us [10:38:44] hi valhallasw`vecto! zhuyifei1999_ have already suggested using MONO_REGISTRY_PATH and it worked perfectly, thanks! [10:40:09] valhallasw`vecto: will file a bug regarding webservice restart [10:42:14] valhallasw`vecto: also, regarding mono issue, please note that I never suggested creating a writable-for-all directory [10:46:52] leloiandudu: ah, that's what thr link suggested, hence my confusion [10:47:05] leloiandudu: thank you for filing a webservice task [10:52:19] 06Labs, 10Tool-Labs: error starting webservice - https://phabricator.wikimedia.org/T142932#2551666 (10Leloiandudu) [10:52:40] done :) [11:46:53] 06Labs, 10Tool-Labs: error starting webservice - https://phabricator.wikimedia.org/T142932#2551691 (10valhallasw) 05Open>03Resolved a:03valhallasw The job is currently running on `tools-webgrid-lighttpd-1402`; the accounting log shows the errors are probably from `tools-webgrid-lighttpd-1412`. There is... [12:59:05] 06Labs, 06Operations, 06Release-Engineering-Team, 10wikitech.wikimedia.org: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2551726 (10lfschenone) [13:00:21] 06Labs, 06Operations, 06Release-Engineering-Team, 10wikitech.wikimedia.org: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2250492 (10lfschenone) I modified my rename request. The requested username change would now be: - At Gerrit, from lfs to Sophivorus -... [14:41:09] /deopme/28 [15:08:59] Platonides: ? [15:09:18] bad command, Luke081515 [15:12:00] ah, ok [17:02:56] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2551901 (10faidon) p:05Normal>03High I've pinged @Andrew a few times last week on IRC about Labs cron spam. Some of it was fixed but Labs still accounts for the vast... [17:21:44] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Josve05a was created, changed by Josve05a link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Josve05a edit summary: Created page with "{{Tools Access Request |Justification=Run SQL queries. |Completed=false |User Name=Josve05a }}" [17:44:36] Hi all :) Is this enough, or do I need to add more details, or transclude this page somewhere else to list the request? https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Josve05a [17:50:48] Josve05a, no that's all that's needed [17:50:56] ok, good :) [17:59:32] PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:01:20] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Josve05a was modified, changed by Alex Monk link https://wikitech.wikimedia.org/w/index.php?diff=817296 edit summary: [18:02:44] PROBLEM - Puppet run on tools-merlbot-proxy is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:02:46] ooh :D [18:02:48] Josve05a, you should also know that you can use quarry to run sql queries against replicas [18:03:20] yes I know, but it has a 30 min kill limit, and ..well...I may not be within that limit [18:08:11] Josve05a, basically you can ssh to tools-login.wmflabs.org now [18:08:14] PROBLEM - Puppet run on tools-precise-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:08:18] See https://wikitech.wikimedia.org/wiki/Help:Tool_Labs [18:08:39] you can ask for help here or labs-l@lists.wm.o, just say you're using the tools project [18:08:49] Krenair: yes, I've "logged in" there now :) [18:09:47] oh wait..no I haven't :P wrong insance :p [18:10:21] It asks for password...ugh [18:10:32] thanks, I'll try and figure this out :) [18:11:03] Josve05a, you need to use ssh keys, not passwords [18:11:35] It says "Warning: Permanently added 'tools-login.wmflabs.org' (ECDSA) to the list of known hosts. [18:11:36] Password:" [18:11:46] so...password... [18:12:03] PROBLEM - Puppet run on tools-redis-1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:13:53] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2198026 (10AlexMonk-WMF) >>! In T132422#2551901, @faidon wrote: > I've pinged @Andrew a few times last week on IRC about Labs cron spam. Some of it was fixed but Labs st... [18:14:18] Josve05a, you get a password prompt if your ssh key login fails [18:14:23] you can't do anything with the password [18:14:54] why would my ssh key fail...hmm.. [18:15:20] have you used other labs hosts before? [18:15:54] no. not to my knowledge [18:18:12] it seems we need to get some fail2ban on tools-bastion [18:18:24] hundreds of repeated failed logins for root [18:18:52] Josve05a: Aug 14 18:06:09 tools-bastion-03 sshd[30093]: pam_unix(sshd:session): session opened for user josve05a by (uid=0) [18:18:58] so I guess it was successful in the end? [18:19:21] yes, I think so... [18:45:02] 06Labs, 10Tool-Labs, 05WMF-NDA: Repeated ssh login attempts for root@tools-login from two Chinese IPs - https://phabricator.wikimedia.org/T142947#2552049 (10valhallasw) [18:50:33] 06Labs: Get wmflabs.org on the Public Suffix List - https://phabricator.wikimedia.org/T142948#2552064 (10yuvipanda) [18:53:02] yuvipanda, been wondering about that one for a long time [18:53:07] you think we should actually do it? [18:58:59] 10Tool-Labs-tools-Other, 06Wikisource, 06Wikisource-Community-User-Group: ia-upload convert pdf to djvu and uploaded as 0 × 0 pixels file - https://phabricator.wikimedia.org/T142939#2552081 (10Danny_B) [19:00:34] 06Labs: Get wmflabs.org on the Public Suffix List - https://phabricator.wikimedia.org/T142948#2552082 (10yuvipanda) https://github.com/publicsuffix/list/pull/284 [19:37:19] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:18:53] Is there any guide on how to setup our Pywikibot to run on WMF labs? [20:19:53] also, I have access to tool labs, but I don't have any tool yet (ie, I can't "become" anything), do I need that as well? [20:23:32] mafk: IIRC, you got stewardbot? [20:23:44] for running a bot you need a tool, yep [20:24:07] Luke081515: yes, well, I have access to some tools, stewardbots indeed, and some others; but none of my own [20:24:17] then create one [20:24:24] Krenair: please block https://wikitech.wikimedia.org/wiki/Special:Contributions/Alcohol_of_Nagasaki [20:24:37] harassment [20:24:40] vandalism [20:24:54] krenair@bastion-01:~$ ldapsearch -x member=uid=maurelio,ou=people,dc=wikimedia,dc=org | grep dn: | grep "tools\." [20:24:54] dn: cn=tools.hat-collector,ou=servicegroups,dc=wikimedia,dc=org [20:24:54] dn: cn=tools.stewardbots,ou=servicegroups,dc=wikimedia,dc=org [20:24:54] krenair@bastion-01:~$ [20:25:16] mafk: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Developing#Using_the_shared_Pywikibot_files_.28recommended_setup.29 [20:25:50] mafk, done re. harassment/vandalism user [20:25:55] :) [20:26:20] so I need to create a tool first [20:26:25] exactly [20:26:41] service: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaServiceGroup&action=addservicegroup&projectname=tools [20:26:43] y'know, things were kind of easier when there was no Internet [20:26:47] :P [20:27:04] lol [20:27:06] when I was young there was no Internet nor mobile phones [20:27:11] * paladox likes the internet [20:27:21] * paladox all my things are internet connected. [20:27:34] that might be not only positive [20:27:35] including all the bedrooms have smart tvs lol [20:27:46] not mine [20:27:56] paladox: lemme hack and monitor you, then :P [20:27:59] LOL [20:28:11] but I have a cat at home, it counts? [20:28:12] I doint have camera [20:28:16] cameras [20:28:20] LOL [20:28:21] * Luke081515 activates paladox mircophones [20:28:29] LOL [20:28:39] it dosent have that either [20:28:39] paladox: I hope you have them firewalled [20:28:44] yeh [20:28:48] all behind the bt smart hub [20:28:57] Luke081515: mind I remind you, that pursuant some paragraphs of the StGB, hacking is illegal :P [20:29:20] seems to be firewalled since i did not set it to send traffic to any devices, ie no one can setup an apache server and run a website. [20:29:22] paladox can have fun with you later [20:29:27] mafk: not if no one discovers me :P [20:29:39] I have done that before, but now i use a test instance [20:29:54] but my pc was behind norton security which had also a firewall [20:30:13] https://www.productsandservices.bt.com/products/smart-hub [20:30:28] mafk Platonides Luke081515 ^^ [20:30:36] Can you please take this conversation elsewhere? [20:30:47] hm, we have #wikimedia-offtopic [20:30:51] :O [20:31:11] oh sorry [20:31:17] mafk: so after you created a tool, you can follow the instructions at wikitech, I linked [20:31:36] I guess they work. I can't test that, I'm not using phython for my bots [20:32:13] Luke081515: okay, so mabot might be appropriate for the toolname [20:32:22] all lowercase iirc [20:32:28] yep [20:33:49] Krenair: do you now, if blocking a user at wikitech blocks his shell access etc too? [20:34:08] shell access is not automatic [20:34:20] ie. unless someone gave him shell before [20:34:34] he wouldn't have it so no need to remove [20:34:37] ^ [20:35:03] Krenair: I think new users got shell access, but are not at the usergroup shell anymore? [20:35:20] what do you mean? [20:35:22] they get auto-added once they are added to a project [20:35:28] but adding a user to a project is still manual [20:35:35] ah, ok [20:35:59] Platonides: tu entiendes de esto del Labs? me echarías un cable en otro canal para no molestar aquí? [20:36:04] ok [20:36:15] Gracias [20:36:19] valhallasw`cloud: so if a users gets blocked, he should removed from the usergroup shell as well? [20:36:38] Luke081515: I'm not entirely sure [20:37:03] mafk and Platonides theres #wikimedia-offtopic channel [20:37:15] paladox: we have our own [20:37:20] Oh [20:37:25] whats that? [20:38:02] mafk ^^ [20:38:10] and that was not off topic, he's going to help me with labs elsewhere to avoid channel cluttering [20:38:25] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Merlijn van Deen - alternative was created, changed by Merlijn van Deen - alternative link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Merlijn_van_Deen_-_alternative edit summary: Created page with "{{Tools Access Request |Justification=Testing w/ alt non-root account |Completed=false |User Name=Merlijn van Deen - alternative }}" [20:38:33] Nope i didnt mean that but google translator seems to have translated it wrong [20:39:08] mafk ^^ [20:39:21] I don't speak Spanish but I didn't need google translate to tell me it was relevant to labs [20:39:21] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Merlijn van Deen - alternative was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817307 edit summary: [20:39:42] it was :) [20:40:18] but if we're going to speak spanish in a non-spanish channel other than a few words, I think we should do it elsewhere [20:54:03] tools.mabot created Luke081515 :) [20:54:31] ok :) [20:54:46] then use the instructions here: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Developing#Using_the_shared_Pywikibot_files_.28recommended_setup.29 [21:00:25] Luke081515: I think I'll use WinSCP to import the user-config.py I use [21:00:58] mafk: heh, if you need pywikibot specific help, I'm not the right person to speak with ;). My bots are written in PHP, not phython [21:13:53] mafk: make sure it is not world-readable [21:14:01] it contains a password [21:14:13] Platonides: chmod 600 ? [21:14:52] Platonides: it contains "just" OAuth tokens, but still better to keep them private [21:17:44] yes [21:17:49] ah, they are oauth tokens in the recent versions, ok [21:17:51] still, they allow login into the account [21:31:54] 06Labs, 10Continuous-Integration-Infrastructure: Request increased quota for contintcloud labs project - https://phabricator.wikimedia.org/T142877#2549610 (10hashar) The DNS leak is tracked by T115194 there are good indications that is an old issue and the 32K leftover entries are artifacts of an unknown issue... [21:35:33] 06Labs, 10Labs-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2552183 (10hashar) 05Resolved>03Open Quota has been lowered from 20 to 10 instances in an emergency on July 4th since t... [21:48:04] Platonides: I think I've denied myself from accessing my own folder <_< [21:48:55] chgrp: cannot read directory 'mabot': Permission denied [21:51:14] hm [21:51:15] drwxrwsr-x 5 tools.alex tools.alex 4096 Jun 29 13:49 . [21:51:18] drws--S--T 5 tools.mabot tools.mabot 4096 Aug 14 21:36 . [21:51:49] I just wanted myself to see what I hosted in my folder, and I've ended restricting everyone included myself [21:52:31] tools.mabot@tools-bastion-03:~$ ls -la [21:52:32] total 72 [21:52:34] drws--S--T 5 tools.mabot tools.mabot 4096 Aug 14 21:36 . [21:56:29] can you create a new file under ~? [21:57:00] I can access them through git bash, but using WinSCP it gives me the error [21:57:28] create, not read [21:57:49] on tools.mabot? [21:57:53] any file? [21:57:59] yes [21:58:25] created "test" [21:58:27] I saw [21:58:38] so you've got some access but WinSCP is very unhappy [21:59:34] want me to reset your stuff back to normal? [22:00:29] Krenair: please [22:00:34] and sorry for messing [22:00:42] I'll do that file by file next time [22:06:10] mafk, okay, hopefully your permissions are a bit more normal now :) [22:06:41] * mafk checks [22:07:12] at least enough that WinSCP should be happy? [22:08:01] * enterprisey hears WinSCP [22:08:47] El comando «cd "/data/project/mabot"» [22:08:49] ha fallado. Ha devuelto el código de error 1 y -bash: line 40: cd: /data/project/mabot: Permission denied [22:08:50] como mensaje de error. [22:09:11] oh I know [22:09:17] "The command (...) has failed and returned the error code (...) as error message" [22:09:55] now try [22:10:18] I'm betting this is because WinSCP runs as your user, and so relies upon the group permissions [22:10:58] works now [22:11:00] :D [22:11:14] no more chgrp [22:12:19] it's weird that .pywikibot doesn't appear on WinSCP, but if I type the route manually it does :) [22:12:50] El comando «cd "/data/project/mabot/.pywikibot"» [22:12:52] ha fallado. Ha devuelto el código de error 1 y -bash: line 47: cd: /data/project/mabot/.pywikibot: Permission denied [22:12:53] como mensaje de error. [22:13:05] Krenair: maybe you can delete that .pywikibot file so I can recreate it again [22:13:39] you could do this yourself you know [22:14:31] now try [22:15:44] .pywikibot might not show up by default because it begins with a . [22:15:52] there may be an option somewhere for hidden files [22:16:00] works now [22:16:26] rm -rf .pywikibot <-- I was about doing that before [22:18:38] yeah well [22:18:39] either works [22:19:57] logs/commands.logs gives me error as well [22:20:39] I should see if I can run WinSCP as group instead of user [22:21:03] rm: cannot remove ‘logs/commands.log’: Permission denied [22:22:23] you mean, as tools.mabot instead of maurelio? [22:22:39] yep [22:22:39] don't think you can log in as those users [22:22:46] have to log in as you and sudo [22:23:01] fwiw if possible reset permissions recursive [22:23:16] for all files in /data/projects/mabot [22:27:10] I didn't do any magic, I figured out what the permissions should look like by hand [22:29:03] Krenair hi im wondering with the grrrit-wm bot, can i blacklist a certain channel that is for example doing [22:29:04] operations.*: [22:29:04] branch: "^(?!betacluster)" [22:29:19] I need to blacklist operations/debs/wikistats [22:29:25] just mutante has asked me [22:30:06] I don't know [22:30:07] Krenair: I think it's best if I ask a Labs admin to delete tools.mabot entirely and I recreate it later, nothing will be lost [22:30:30] yeah... maybe ask someone else to do that :) [22:30:49] like perhaps valhallasw`cloud [22:31:58] Oh ok [22:32:40] valhallasw`cloud: please delete my tools.mabot so I can recreate from start, thank you [22:42:43] emailed [22:45:00] Mafk: we can't remove tools altogether [22:45:13] (03Draft1) 10Paladox: Blacklist channel #operations/debs/wikistats [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/304746 [22:45:16] valhallasw`cloud: I don't understand [22:45:35] valhallasw`cloud: you're removing inactive projects now so I guess you can delete them, right? [22:45:44] You can try fixing permissions, use take, or file a task for support [22:45:57] (03PS2) 10Paladox: Blacklist channel #operations/debs/wikistats [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/304746 [22:46:01] Project != Tool [22:46:24] valhallasw`cloud: so I basically want to restart tools.mabot [22:46:41] can't tools.mabot be deleted then? [22:46:49] No [22:47:14] But you can use 'take' to fix ownership [22:47:28] valhallasw`cloud: an example? [22:47:42] !log tools.lolrrit-wm cherry picking https://gerrit.wikimedia.org/r/#/c/304746/ (Blacklist channel #operations/debs/wikistats) [22:47:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master [22:48:00] valhallasw`cloud im wondering if you could merge https://gerrit.wikimedia.org/r/#/c/304746/ please? [22:48:02] See docs, I'm on mobile. [22:48:12] Tested it and it dosent seem to affect the other channesl [22:48:13] valhallasw`cloud: okay [22:48:13] channels [22:48:45] 10Labs-project-Wikistats, 13Patch-For-Review: wikistats (labs project): rename Image column to Files - https://phabricator.wikimedia.org/T142732#2552280 (10Dzahn) 05Open>03Resolved Done. The column and link has been changed as requested. [22:50:53] valhallasw`cloud: replica.my.cnf: Operation not permitted <-- is that suposed to happen? [22:51:25] Mafk: i don't know. [22:52:47] (03CR) 10Paladox: [C: 031] "I cherry picked this and seems to work and seems the other channels are unaffected to :)." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/304746 (owner: 10Paladox) [22:55:51] (03PS3) 10Paladox: Blacklist channel #wikimedia-operations/debs/wikistats [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/304746 [23:01:48] 06Labs, 10Labs-Infrastructure, 06Operations: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2552287 (10AlexMonk-WMF) >>! In T115194#2244874, @Andrew wrote: > there are relatively many ldap connection failures in the sink log. That fits with the fact that... [23:02:48] 06Labs, 10Labs-Infrastructure: Once we have Liberty: remove project-id logic from designate/ldap plugin, use project_id in metadata instead. - https://phabricator.wikimedia.org/T105891#2552290 (10AlexMonk-WMF) 05stalled>03Open We now have Liberty. [23:08:26] 06Labs, 10Labs-Infrastructure: Once we have Liberty: remove project-id logic from designate/ldap plugin, use project_id in metadata instead. - https://phabricator.wikimedia.org/T105891#2552294 (10AlexMonk-WMF) ```krenair@bastion-01:~$ curl -f http://169.254.169.254/openstack/2015-10-15/meta_data.json 2>/dev/n... [23:12:12] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Once we have Liberty: remove project-id logic from designate/ldap plugin, use project_id in metadata instead. - https://phabricator.wikimedia.org/T105891#2552298 (10AlexMonk-WMF) a:03AlexMonk-WMF [23:16:08] valhallasw`cloud: fixed everything with chmod [23:16:20] now I'll learn about the grid, etc [23:16:33] but later tomorrow [23:31:16] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Once we have Liberty: remove project-id logic from designate/ldap plugin, use project_id in metadata instead. - https://phabricator.wikimedia.org/T105891#2552305 (10AlexMonk-WMF) Basically off-topic but I thought I should write it down somewhere: Can we imp... [23:44:42] 06Labs, 10Labs-Infrastructure, 10Horizon: Disassociating floating IP does not show it as available in the horizon project quota summary - https://phabricator.wikimedia.org/T136872#2350559 (10AlexMonk-WMF) I think this is because when you disassociate an IP, it's still reserved for your project. See `nova flo... [23:48:22] 06Labs, 10Labs-Infrastructure: Can't create account "Bishoy Camel" (user with a former SVN account not migrated) - https://phabricator.wikimedia.org/T128833#2552310 (10AlexMonk-WMF) 05Open>03stalled a:03BishoyCamel Stalled and assigning to requester, this is blocked on more information from them. [23:55:25] 06Labs, 10Labs-Infrastructure: root@ should email project admins - https://phabricator.wikimedia.org/T61142#639898 (10AlexMonk-WMF) So we'd want MX records on each instance's internal domain, pointing to a mail server that knows how to interpret the domain, look up all project admins, and delive... [23:57:16] 06Labs, 10Labs-Infrastructure, 07Tracking: NFS overload is causing instances to freeze - https://phabricator.wikimedia.org/T124133#2552316 (10AlexMonk-WMF) 05Open>03Resolved >>! In T124133#2517843, @chasemp wrote: > This task encompassed a particular failure pattern we were seeing really often in late 20... [23:58:18] 06Labs, 10Labs-Infrastructure: Clean up leaked designate entries - https://phabricator.wikimedia.org/T120797#2552319 (10AlexMonk-WMF) a:03AlexMonk-WMF I'll write a script to verify and clean everything on the internal domains.