[00:35:35] 10Tool-Labs-tools-Xtools: Resolve intermittent 500 and 503 errors with xtools - https://phabricator.wikimedia.org/T144783#2610060 (10Matthewrbowker) [00:35:51] 10Tool-Labs-tools-Xtools: Resolve intermittent 500 and 503 errors with xtools - https://phabricator.wikimedia.org/T144783#2610072 (10Matthewrbowker) p:05Triage>03High [01:21:30] PROBLEM - Free space - all mounts on tools-web-static-02 is CRITICAL: CRITICAL: tools.tools-web-static-02.diskspace._srv.byte_percentfree (<100.00%) [01:54:50] (03CR) 10Alex Monk: [C: 032] Improve contrib-admin display of goals and repos [labs/striker] - 10https://gerrit.wikimedia.org/r/308667 (owner: 10BryanDavis) [01:55:57] 10Striker, 13Patch-For-Review, 15User-bd808: Admin console fails to add new diffusionrepo entries - https://phabricator.wikimedia.org/T144296#2610109 (10Krenair) 05Open>03Resolved a:03bd808 Seems fine now [01:55:59] (03Merged) 10jenkins-bot: Improve contrib-admin display of goals and repos [labs/striker] - 10https://gerrit.wikimedia.org/r/308667 (owner: 10BryanDavis) [02:03:36] 10Tool-Labs-tools-Xtools: Resolve intermittent 500 and 503 errors with xtools - https://phabricator.wikimedia.org/T144783#2610060 (10MusikAnimal) This is the web watcher I believe. It detected the service did not respond (at the time) with a 200, so it restarted it. This very brief period where it restarts will... [04:25:46] PROBLEM - SSH on tools-prometheus-02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:27:24] RECOVERY - Free space - all mounts on tools-prometheus-02 is OK: OK: tools.tools-prometheus-02.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) [07:14:21] 06Labs, 10Tool-Labs: Cannot delete job id #7523461 by tools.legobot on Tool Labs - https://phabricator.wikimedia.org/T138123#2610237 (10Legoktm) 05Resolved>03Open Uh, @yuvipanda , could you use your root magic powers again? It seems to have gotten stuck again: ``` tools.legobot@tools-bastion-03:~$ qdel -f... [07:32:39] !log tools depooled tools-exec-1219 and 1218, seem to be unresponsive, causing jobs that appear to run but aren't really [07:32:43] legoktm ^ [07:32:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [07:32:49] that got rid of your jobs as well [07:32:54] thanks [07:33:43] 06Labs, 10Tool-Labs: Gridengine nodes tools-exec-1218 and 1219 seem unreachable - https://phabricator.wikimedia.org/T144789#2610259 (10yuvipanda) [07:33:58] !log tools.wikiloves @Danilo pulled f71efdc from Git, solving T144669 [07:34:00] T144669: Error in links wikiloves statistics tool - https://phabricator.wikimedia.org/T144669 [07:34:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL, Master [07:34:06] 06Labs, 10Tool-Labs: Gridengine nodes tools-exec-1218 and 1219 seem unreachable - https://phabricator.wikimedia.org/T144789#2610276 (10yuvipanda) Noticed because of T138123#2610237 [07:34:17] 06Labs, 10Tool-Labs: Cannot delete job id #7523461 by tools.legobot on Tool Labs - https://phabricator.wikimedia.org/T138123#2390979 (10yuvipanda) 05Open>03Resolved Happened because of https://phabricator.wikimedia.org/T144789. fixed. [07:35:49] PROBLEM - Free space - all mounts on tools-services-01 is CRITICAL: CRITICAL: tools.tools-services-01.diskspace.root.byte_percentfree (<11.11%) [07:38:00] !log tools.wikiloves Pulled latest from Git: f79fd1a [07:38:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL, Master [08:06:42] (03PS4) 10Jean-Frédéric: Refactor database configuration handling [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 [08:06:44] (03PS3) 10Jean-Frédéric: Setup local development environment for ErfgoedBot [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303498 [08:07:24] (03CR) 10jenkins-bot: [V: 04-1] Refactor database configuration handling [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 (owner: 10Jean-Frédéric) [08:09:05] (03PS5) 10Jean-Frédéric: Refactor database configuration handling [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 [08:09:07] (03PS4) 10Jean-Frédéric: Setup local development environment for ErfgoedBot [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303498 [08:10:27] (03CR) 10Jean-Frédéric: "> Looks good. The only thing I'm unsure about is whether this will correctly load the db username/password (does pywikibot load .my.conf a" (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 (owner: 10Jean-Frédéric) [08:16:55] 06Labs, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Wikimedia-Incident: Nodepool instance instance creation quota management - https://phabricator.wikimedia.org/T143016#2610325 (10hashar) The https://horizon.wikimedia.org/project/instances/ shows proper values / instances now. [08:40:43] 10Labs-Kubernetes: Locale on Kubernetes - https://phabricator.wikimedia.org/T144794#2610377 (10Magnus) [08:51:41] 10Labs-Kubernetes, 13Patch-For-Review: Locale on Kubernetes - https://phabricator.wikimedia.org/T144794#2610377 (10yuvipanda) I've rebuilt the containers, will push them tomorrow to have better locale support! [09:00:01] 10Labs-Kubernetes, 13Patch-For-Review: Locale on Kubernetes - https://phabricator.wikimedia.org/T144794#2610421 (10Magnus) Great, thanks! [09:43:22] (03PS6) 10Jean-Frédéric: Refactor database configuration handling [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 [09:43:24] (03PS5) 10Jean-Frédéric: Setup local development environment for ErfgoedBot [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303498 [09:45:46] (03CR) 10Jean-Frédéric: "I also added tests, which I had overlooked initially ^__^'" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 (owner: 10Jean-Frédéric) [10:01:09] 06Labs: Request increased quota for reading-web-staging labs project - https://phabricator.wikimedia.org/T144623#2610510 (10Jhernandez) @phuedx We can safely **remove** reading-web-staging-2? =stack @bd808 stack is taking a full vm of the smallest size. I'm happy to move it somewhere else if needed. The minimu... [10:04:57] 06Labs: Request increased quota for reading-web-staging labs project - https://phabricator.wikimedia.org/T144623#2610516 (10phuedx) >>! In T144623#2610510, @Jhernandez wrote: > @phuedx We can safely **remove** reading-web-staging-2? Yes. Sorry. [10:12:15] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2610543 (10hashar) 05Open>03Resolved The issue has... [10:12:55] 06Labs: Request increased quota for reading-web-staging labs project - https://phabricator.wikimedia.org/T144623#2610546 (10Jhernandez) I've triggered termination of reading-web-staging-2 per convo on irc with @phuedx If there's something we need to test like CN we'll work to set it up in reading-web-staging-3... [10:13:53] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:35:38] 06Labs, 06Operations, 13Patch-For-Review: grafana-labs.wikimedia.org doesn't reflect grafana-labs-admin.wikimedia.org - https://phabricator.wikimedia.org/T143556#2610559 (10fgiunchedi) a:05fgiunchedi>03None unassigning as I don't think I'll have time to work on the last bit (add `Anonymous` viewer rights... [10:53:51] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:36:54] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:38:26] 06Labs, 06Operations, 10wikitech.wikimedia.org: Can't login wikitech - https://phabricator.wikimedia.org/T144805#2610832 (10Shizhao) [13:54:07] 06Labs, 06Operations, 10wikitech.wikimedia.org: Can't login wikitech - https://phabricator.wikimedia.org/T144805#2610832 (10Krenair) It's two-factor authentication. You must've enabled it on your account. [14:42:01] PROBLEM - Puppet run on tools-exec-1201 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:22:00] RECOVERY - Puppet run on tools-exec-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [15:25:29] PROBLEM - Free space - all mounts on tools-web-static-01 is CRITICAL: CRITICAL: tools.tools-web-static-01.diskspace._srv.byte_percentfree (<100.00%) [15:44:21] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2611272 (10Andrew) @MaxBioHazard, if this is still working today then we can close this [15:46:53] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2611278 (10MaxBioHazard) Works for me now. [15:48:54] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: I can't use crontab on Labs - https://phabricator.wikimedia.org/T144731#2611281 (10Andrew) 05Open>03Resolved a:03Andrew Great, thanks! [15:49:38] (03CR) 10Lokal Profil: [C: 031] "It looks fine to me and I would be happy to have it merged." (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303498 (owner: 10Jean-Frédéric) [16:06:50] madhuvishy: does this matter? 'PROBLEM - Free space - all mounts on tools-web-static-01 is CRITICAL: CRITICAL: tools.tools-web-static-01.diskspace._srv.byte_percentfree (<100.00%)' [16:09:31] 10Striker, 13Patch-For-Review, 15User-bd808: Need pretty page for uwsgi proxy errors - https://phabricator.wikimedia.org/T144040#2611351 (10bd808) 05Open>03Resolved a:03bd808 [16:20:29] (03CR) 10Lokal Profil: [C: 031] "Looks fine. One (inline) question though." (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 (owner: 10Jean-Frédéric) [16:37:22] 06Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, 10wikitech.wikimedia.org, 07Epic: [EPIC] Make wikitech more friendly for the multiple audiences it supports - https://phabricator.wikimedia.org/T123425#2611498 (10Dereckson) [16:37:24] 06Labs, 06Operations, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Enable math extension on wikitech - https://phabricator.wikimedia.org/T126338#2611496 (10Dereckson) 05Open>03Resolved @Halfak All works fine. You can add back your equations on https://wikitech.wikimedia.o... [16:38:05] 06Labs: Request increased quota for reading-web-staging labs project - https://phabricator.wikimedia.org/T144623#2611501 (10bd808) >>! In T144623#2610510, @Jhernandez wrote: > = stack > @bd808 stack is taking a full vm of the smallest size. I'm happy to move it somewhere else if needed. > > The minimum requirem... [16:40:48] PROBLEM - Free space - all mounts on tools-services-01 is CRITICAL: CRITICAL: tools.tools-services-01.diskspace.root.byte_percentfree (<11.11%) [16:42:11] chasemp: no, I'm going to remove those instances [16:42:26] we switched over to tools-static-10 and 11 [16:44:45] thought so, thanks [16:44:57] only 10 and 11 huh? didn't make it to the 30's I guess [16:49:22] chasemp: yeah thankfully [17:07:29] (03CR) 10Lokal Profil: "@JeanFred: Any thoughts?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/302887 (owner: 10Lokal Profil) [17:14:59] !log google-api-proxy Replacing manual config with role::labs::google_api_proxy [17:15:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Google-api-proxy/SAL, Master [17:20:50] (03CR) 10Yuvipanda: [C: 031] "Code looks good to me. Have you tested this?" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/307461 (https://phabricator.wikimedia.org/T143282) (owner: 10BryanDavis) [17:22:02] (03CR) 10BryanDavis: "> Code looks good to me. Have you tested this?" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/307461 (https://phabricator.wikimedia.org/T143282) (owner: 10BryanDavis) [17:24:44] bd808 do you think you'll have time to do that? [17:24:50] I usually test with a virtualenv... [17:25:54] I can test it, yeah. I'm not sure how to test jsub with a venv but I can copy it to my ~ and test there [17:26:26] thanks! [17:59:35] (03CR) 10Jean-Frédéric: "Sorry for the delay in reviewing this." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/302887 (owner: 10Lokal Profil) [18:08:03] 06Labs, 10Tool-Labs, 13Patch-For-Review, 07Wikimedia-Incident: Tune nginx config parameters for tools / labs proxies - https://phabricator.wikimedia.org/T143637#2611983 (10yuvipanda) a:03madhuvishy [18:08:14] 06Labs, 10Labs-Infrastructure, 10Tool-Labs, 07Wikimedia-Incident: Write a simple script that handles failovering proxies - https://phabricator.wikimedia.org/T143639#2611984 (10yuvipanda) a:03madhuvishy [18:08:22] 06Labs, 10Tool-Labs, 07Wikimedia-Incident: Setup a simple service that pages when it is unreachable - https://phabricator.wikimedia.org/T143638#2611985 (10yuvipanda) a:03madhuvishy [18:12:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:12:36] PROBLEM - Puppet run on tools-checker-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:13:03] PROBLEM - Puppet run on tools-exec-1201 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:21:52] 06Labs, 13Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#2612016 (10Andrew) Updated list of remaining globals: deployment_server_override labs_mediawiki_hostname lvm_mount_point node_dedicated_tool puppetmaster restricted_from restricted_to... [18:47:38] RECOVERY - Puppet run on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:52:36] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [18:53:02] RECOVERY - Puppet run on tools-exec-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [19:41:27] !log google-api-proxy Puppet disabled on google-api-proxy-02 until https://gerrit.wikimedia.org/r/#/c/308780/ is merged [19:41:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Google-api-proxy/SAL, Master [20:15:51] (03CR) 10BryanDavis: [V: 031] "Tested on tools-dev by copying the jsub file from this patch to my home directory and running it with some test jobs:" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/307461 (https://phabricator.wikimedia.org/T143282) (owner: 10BryanDavis) [20:27:25] tgr: In some of your sentry instances you have a variable 'sentry_server_name' set. [20:27:32] It looks to me like that doesn't do anything, do you agree? [20:27:47] (I'm trying to clean up stray globals as part of standardizing our puppet config system) [20:28:41] andrewbogott: probably, I seem to recall bd808 fixing that to use intelligent defaults [20:29:11] great, I'm going to unset those vars from your instances then [20:29:12] thanks [20:29:39] thanks for the cleanup :) [20:30:55] andrewbogott: btw, not sure where to list this: https://wikitech.wikimedia.org/wiki/Purge_2016#In_use_authmanager [20:31:02] the project can be deleted [20:31:32] tgr: great! I will mark it for deletion. thanks [20:35:44] matt_flaschen: I'm making a slight refactor of the old mediawiki_singlenode class. There are four instances in the 'editor-engagement' project that use this: mwui ee-flow toro piramido [20:35:49] Do you if any of those can just be deleted? [20:35:57] (Not a big deal, just trying to avoid maintaining dead hosts) [20:41:17] andrewbogott: thanks for the review of the change that removed "show: true" from Nodepool :] [20:41:36] hashar: I'm seeing CI images on wikitech now… do you know if that's related? [20:41:43] andrewbogott: that made me mad for a good hour and I was quite happy when I found out "show" is no more needed [20:41:47] (Maybe they were always there, I haven't created instance son wikitech recently) [20:42:12] I dont use wikitech for the nodepool instances [20:42:25] I use either the CLI tool on labnodepool1001 or Horizon!!!!!!! [20:42:41] me too [20:43:05] !log tools drain and reboot tools-exec-1410 for testing [20:43:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [20:46:16] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [20:56:15] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [21:45:46] !log tools reboot tools-prometheus-02. nova diagnostics shows no vda activity. [21:45:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:50:35] RECOVERY - SSH on tools-prometheus-02 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [21:53:25] PROBLEM - Free space - all mounts on tools-prometheus-02 is CRITICAL: CRITICAL: tools.tools-prometheus-02.diskspace.root.byte_percentfree (<66.67%) [21:54:53] PROBLEM - Puppet staleness on tools-prometheus-02 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [43200.0] [21:59:54] RECOVERY - Puppet staleness on tools-prometheus-02 is OK: OK: Less than 1.00% above the threshold [3600.0] [22:03:26] RECOVERY - Free space - all mounts on tools-prometheus-02 is OK: OK: All targets OK [22:04:30] 10Tool-Labs-tools-Other, 15User-bd808: Disable Stashbot paste elements resolve on #wikidata - https://phabricator.wikimedia.org/T144810#2612927 (10Dereckson) a:03bd808 [22:06:47] 06Labs, 06Operations, 10wikitech.wikimedia.org: Can't login wikitech - https://phabricator.wikimedia.org/T144805#2612937 (10Peachey88) @Shizhao When was the last time you logged into wikitech successfully? There was a mistake at one stage that resulted in a number of people accidently having 2FA activated,... [22:07:58] 10Labs-Kubernetes, 13Patch-For-Review: Locale on Kubernetes - https://phabricator.wikimedia.org/T144794#2612940 (10yuvipanda) I've pushed it now. The default LOALE should be set to C.UTF-8 i think. See if the default works ok for you now? [22:10:29] !log tools Deleted instance tools-web-static-01 and tools-web-static-02 (T143637) [22:10:30] T143637: Tune nginx config parameters for tools / labs proxies - https://phabricator.wikimedia.org/T143637 [22:10:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [22:11:09] PROBLEM - Host tools-web-static-02 is DOWN: CRITICAL - Host Unreachable (10.68.16.216) [22:13:09] PROBLEM - Host tools-web-static-01 is DOWN: CRITICAL - Host Unreachable (10.68.18.78) [22:13:10] yuvipanda: where is this check from? ^ [22:13:25] madhuvishy it's doing ping from shinken-01 [22:13:30] shinken.wmflabs.org [22:13:34] you can't silence it [22:13:56] !log tools.stashbot Restarted bot to apply af577b0 (Stop looking for Phab pastes in messages) for T144810 [22:13:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [22:14:15] yuvipanda: but it'll realize the instance is gone and stop alerting? [22:14:19] !log tools got pbuilder off tools-services-01, was taking up too much space. [22:14:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [22:14:25] madhuvishy in about 15min yeah [22:14:30] yuvipanda: okay cool [22:14:34] madhuvishy unless LDAP continues to have stale entries for it [22:14:37] !log tools.stashbot Restarted bot to apply af577b0 (Stop looking for Phab pastes in messages) for T144810 [22:14:39] T144810: Disable Stashbot paste elements resolve on #wikidata - https://phabricator.wikimedia.org/T144810 [22:14:39] in which case it'll alert every day [22:14:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [22:14:42] we have 3 instances doing that rn [22:14:50] hmmm [22:18:40] 10Tool-Labs-tools-Other, 15User-bd808: Disable Stashbot paste elements resolve on #wikidata - https://phabricator.wikimedia.org/T144810#2613013 (10bd808) 05Open>03Resolved Fixed in https://github.com/bd808/tools-stashbot/commit/af577b05209d3c2b58bc36303d5136dfa65d4302 by just getting rid of `P\d+` detectio... [22:18:52] yuvipanda, I know about deployment-parsoid05, what are the others? [22:19:01] analytics30something? [22:19:12] krenair there's a bunch of tools-secgrouptest or something along those lines [22:19:15] I see 'em every day here [22:19:19] oh, just 3 in tools? [22:19:20] heh [22:20:24] yeah [22:25:48] RECOVERY - Free space - all mounts on tools-services-01 is OK: OK: All targets OK [22:33:06] 10Striker, 06Project-Admins: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111#2588823 (10Legoktm) In general I think prefixing and namespacing Phabricator projects is a good idea. In this case I think it's mostly important because we can have wikibugs auto... [22:51:09] andrewbogott, piramido and toro should be okay to delete, ee-flow not yet, mwui ask design and Volker_E. [22:53:09] ok [22:53:58] !log editor-engagement deleting instances toro and piramido [22:54:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Editor-engagement/SAL, dummy [22:54:52] andrewbogott: I'm here, what's the topic? [22:55:10] Volker_E: wondering if the 'mwui' instance in editor-engagement can be deleted [22:55:55] andrewbogott: what URL is pointing to it? Haven't touched it via ssh so far [22:56:02] Volker_E: no idea [22:56:22] I mean, you should be able to ssh as mwui.editor-engagement.wmflabs.org [22:56:26] but I don't know what it's doing webwise [22:56:47] andrewbogott: ok, will take a look. Prob not earlier than tomorrow [22:56:52] you mean mwui.editor-engagement.eqiad.wmflabs? [22:57:02] yes, that's what I meant [22:58:14] no rush, just, there are a whole lot of instances in that project and for the most part no one will admit to wanting them :) [22:58:52] andrewbogott: alright :) [23:04:43] 10Striker, 06Project-Admins: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111#2613188 (10bd808) >>! In T144111#2613084, @Legoktm wrote: > In general I think prefixing and namespacing Phabricator projects is a good idea. In this case I think it's mostly imp... [23:11:07] 10Striker, 06Project-Admins: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111#2588823 (10AlexMonk-WMF) If we decided to change it to `#Tool-{toolname}`, #Tool-Labs would look pretty weird, though maybe we could rename it to `#Labs-Tools` or something [23:35:02] 06Labs, 13Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#2613251 (10Andrew) All globals are now eliminated from ldap. Next is removing the gui from wikitech [23:57:14] PROBLEM - SSH on tools-exec-1219 is CRITICAL: Server answer