[00:11:30] Krinkle: found a fix :) [00:49:18] 6Labs, 5Patch-For-Review: lighttpd redirects URLs of directories without a trailing slash from https to http - https://phabricator.wikimedia.org/T66627#1235653 (10yuvipanda) After some false starts, looks like http://redmine.lighttpd.net/projects/1/wiki/Docs_ModMagnet might be the way to fix this. but that emb... [01:01:34] PROBLEM - Host tools-login is DOWN: CRITICAL - Host Unreachable (10.68.16.7) [01:01:37] wtf [01:01:41] oh [01:01:42] right [01:01:44] tools-login [01:01:45] let me delete that [01:02:09] !log tools deleted tools-login, tools-bastion-01 has been running for long enoug [01:02:13] Logged the message, Master [01:05:20] YuviPanda: So the shinken alerts I’m getting are you, not a VM problem? [01:05:30] andrewbogott: just tools-login, right? [01:05:33] andrewbogott: or are there more? [01:05:49] earlier two static nodes sent down/up messages [01:05:51] andrewbogott: tools-login is dead because I killed it a few days ago (shutoff) but forgot to remove. [01:05:52] * andrewbogott checks which [01:06:00] andrewbogott: ah yeah, but that’s because I just created those. [01:06:11] ok! [01:06:21] Great, then I will go back to not working. [01:06:26] andrewbogott: yeah! sorry about that [01:06:31] shinken has nothing to ack tho [01:06:48] np, I’m just extra touchy today :) [01:22:47] 6Labs, 5Patch-For-Review: lighttpd redirects URLs of directories without a trailing slash from https to http - https://phabricator.wikimedia.org/T66627#1235691 (10yuvipanda) Looks like the only solution is to write a small lighttpd module. [01:31:16] andrewbogott: I’m going to delete tools-static now, and it’ll complain. do not worry! [01:32:10] :) [01:32:55] !log tools deleted tools-static, tools-static-01 has taken over [01:32:59] Logged the message, Master [01:34:11] PROBLEM - Host tools-static is DOWN: CRITICAL - Host Unreachable (10.68.17.119) [03:43:14] 10Tool-Labs, 3ToolLabs-Goals-Q4: Make tools-static redundant - https://phabricator.wikimedia.org/T96966#1235806 (10yuvipanda) Done https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin [03:43:22] 10Tool-Labs, 3ToolLabs-Goals-Q4: Make tools-static redundant - https://phabricator.wikimedia.org/T96966#1235807 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [03:43:23] 6Labs, 10Tool-Labs, 3ToolLabs-Goals-Q4, 7Tracking: Make sure that toollabs can function fully even with one virt* host fully down - https://phabricator.wikimedia.org/T90542#1235809 (10yuvipanda) [03:43:57] 6Labs, 10Tool-Labs: Make webservice2 write out a bigbrotherrc file - https://phabricator.wikimedia.org/T90574#1235811 (10yuvipanda) [03:43:59] 10Tool-Labs: Add 'file a bug' link to tool labs error pages - https://phabricator.wikimedia.org/T90570#1235812 (10yuvipanda) [03:44:01] 6Labs, 10Tool-Labs, 7Tracking: Make toollabs reliable enough (Tracking) - https://phabricator.wikimedia.org/T90534#1235810 (10yuvipanda) [03:44:42] 6Labs, 10Tool-Labs, 7Tracking: Make toollabs reliable enough (Tracking) - https://phabricator.wikimedia.org/T90534#1061596 (10yuvipanda) [03:44:44] 10Tool-Labs: Configure web services in such a way that users don't have to (re)start it ever - https://phabricator.wikimedia.org/T94883#1235813 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done with service manifests. No announcement yet, however - let's hope I can fix that on Monday. Basically, if you do a... [03:51:05] 6Labs, 5Patch-For-Review: lighttpd redirects URLs of directories without a trailing slash from https to http - https://phabricator.wikimedia.org/T66627#1235823 (10yuvipanda) ok, so ^ does it at the nginx level, which only handles the tool level redirect. [03:52:03] 6Labs, 5Patch-For-Review: lighttpd redirects URLs of directories without a trailing slash from https to http - https://phabricator.wikimedia.org/T66627#1235824 (10yuvipanda) Nope, it doesn't. I give up. Fuck lighttpd. [05:45:13] 10Wikimedia-Labs-General, 10Wikidata: Need a way to test with data set reasonably close to production - https://phabricator.wikimedia.org/T91131#1235912 (10TechDragon) Are we even able to get a 'full dump' as a starting point for creating this dataset? I was able to install the stack and browse to the interf... [06:52:40] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:22:39] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [07:41:12] 6Labs, 7Tracking: WikiBrain - https://phabricator.wikimedia.org/T96950#1235919 (10Shilad) Than you! [08:54:02] 10Tool-Labs: Configure web services in such a way that users don't have to (re)start it ever - https://phabricator.wikimedia.org/T94883#1235940 (10Multichill) Awesome Yuvi. Sounds excellent. Can you leave a link here to the documentation about these service manifests and maybe other relevant information? [08:55:26] 10Tool-Labs: Configure web services in such a way that users don't have to (re)start it ever - https://phabricator.wikimedia.org/T94883#1235941 (10yuvipanda) Will do when done. These have been running for about a week now, and I think everything has been stable in that time period. Bigbrother will be retired onc... [09:16:58] 6Labs, 10Tool-Labs, 3Labs-Q4-Sprint-2, 3ToolLabs-Goals-Q4, 7Tracking: Replace bigbrother and ssh-cron-thingy with service manifests - https://phabricator.wikimedia.org/T90561#1235949 (10yuvipanda) [09:17:00] 6Labs, 10Tool-Labs, 3Labs-Q4-Sprint-2, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Review and productionize webservice manifest monitor - https://phabricator.wikimedia.org/T95210#1235947 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [09:17:38] 10Tool-Labs, 3ToolLabs-Goals-Q4: Make tools-login / bastion hosts redundant and move them to trusty - https://phabricator.wikimedia.org/T91863#1235950 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done :) [09:17:40] 10Tool-Labs, 3ToolLabs-Goals-Q4: Phase out precise instances from toollabs - https://phabricator.wikimedia.org/T94790#1235953 (10yuvipanda) [09:19:25] 10Tool-Labs, 3ToolLabs-Goals-Q4: Switchover Labs NFS server to labstore1002 - https://phabricator.wikimedia.org/T97219#1235954 (10yuvipanda) 3NEW a:3coren [10:01:36] I will let you know when I see Magog_the_Ogre around here [10:01:36] @notify Magog_the_Ogre [10:01:46] @seen Magog_the_Ogre [10:01:46] YuviPanda: Last time I saw Magog_the_Ogre they were quitting the network with reason: Ping timeout: 245 seconds N/A at 4/23/2015 12:40:48 AM (2d9h20m58s ago) [10:36:37] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Daniel78 was created, changed by Daniel78 link https://wikitech.wikimedia.org/wiki/Nova+Resource%3aTools%2fAccess+Request%2fDaniel78 edit summary: Created page with "{{Tools Access Request |Justification=To run FPCBot (https://commons.wikimedia.org/wiki/User:FPCBot). I have ran it several years from my own machine but it should really not..." [12:12:33] is it intended that ProxyCommand to tools-login no longer works? [13:42:39] PROBLEM - Puppet failure on tools-webgrid-07 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:12:39] RECOVERY - Puppet failure on tools-webgrid-07 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:38] PROBLEM - Puppet failure on tools-webgrid-07 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:53:37] RECOVERY - Puppet failure on tools-webgrid-07 is OK: OK: Less than 1.00% above the threshold [0.0] [15:35:31] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Daniel78 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=155589 edit summary: [16:24:15] 6Labs, 5Patch-For-Review: lighttpd redirects URLs of directories without a trailing slash from https to http - https://phabricator.wikimedia.org/T66627#1236065 (10scfc) Ah, sorry, I forgot to jot down how I intended it to fix. [16:26:43] 10Tool-Labs, 5Patch-For-Review: lighttpd redirects URLs of directories without a trailing slash from https to http - https://phabricator.wikimedia.org/T66627#1236070 (10scfc) p:5Triage>3Normal [18:08:52] PROBLEM - Puppet failure on tools-bastion-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:13:50] gifti: yes. use tools.wmflabs.org - should work. [18:15:05] :( [18:15:33] ok [18:24:47] gifti: err [18:24:51] gifti: tools-login.wmflabs.org [18:24:52] I meant [18:24:57] tools-login.eqiad.wmflabs wont' [18:30:02] 10Tool-Labs, 5Patch-For-Review: lighttpd redirects URLs of directories without a trailing slash from https to http - https://phabricator.wikimedia.org/T66627#1236149 (10yuvipanda) 5Open>3Resolved @scfc <3 I did not know about proxy_redirect and did not look in that direection at all. Thank you very much :D... [18:30:53] YuviPanda: now i'm confused [18:31:34] gifti: ssh tools-login.wmflabs.org should just work, yeah. [18:41:12] 10Wikimedia-Labs-Other: Set appropriate 404 status code for etherpad.wmflabs.org instead of 200 - https://phabricator.wikimedia.org/T58887#1236157 (10Aklapper) 5Open>3declined a:3Aklapper >>! In T58887#621318, @Andrew wrote: > My temptation, though, is to resolve this by deleting the instance entirely...... [18:47:41] 10Wikimedia-Labs-Other: Icinga should only croak in #wikimedia-labs-nagios on actual changes - https://phabricator.wikimedia.org/T62898#1236168 (10Aklapper) I joined #wikimedia-labs-nagios and clicked the topic link (from 2013) but http://nagios.wmflabs.org/ does not exist (anymore). @scfc: How is this task rel... [18:48:35] 10Wikimedia-Labs-General, 6operations, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1236169 (10Aklapper) [18:49:18] 10Wikimedia-Labs-General: Create web version of metainformation table on replica servers - https://phabricator.wikimedia.org/T56703#1236171 (10Aklapper) p:5Low>3Lowest [19:25:36] 10Gerrit-Patch-Uploader: Wrap commit message - https://phabricator.wikimedia.org/T97229#1236178 (10Fomafix) 3NEW [19:28:16] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Sealnk was created, changed by Sealnk link https://wikitech.wikimedia.org/wiki/Nova+Resource%3aTools%2fAccess+Request%2fSealnk edit summary: Created page with "{{Tools Access Request |Justification=project |Completed=false |User Name=Sealnk }}" [19:29:03] 10Gerrit-Patch-Uploader: Wrap commit message - https://phabricator.wikimedia.org/T97229#1236186 (10valhallasw) Line wrapping should be done by the author of a patch, not by a tool. I'm not quite sure how you get wrapping at 45 characters, though. The point of having a textarea with cols="75" (which maybe should... [19:32:11] 10Tool-Labs, 3ToolLabs-Goals-Q4: Support for truly 'generic' webservices via service manifests - https://phabricator.wikimedia.org/T97230#1236187 (10yuvipanda) 3NEW [19:32:35] 10Tool-Labs, 3ToolLabs-Goals-Q4: Support for truly 'generic' webservices via service manifests - https://phabricator.wikimedia.org/T97230#1236194 (10yuvipanda) 'webservice generic start ' maybe? [19:46:08] 10Wikimedia-Labs-Other: Set appropriate 404 status code for etherpad.wmflabs.org instead of 200 - https://phabricator.wikimedia.org/T58887#1236200 (10scfc) 5declined>3Resolved Well, the purpose was to have a status code that conveys "does not exist", and that has been done IMHO. [19:56:33] 10Wikimedia-Labs-Other: Icinga should only croak in #wikimedia-labs-nagios on actual changes - https://phabricator.wikimedia.org/T62898#1236210 (10scfc) 5Open>3Invalid a:3scfc It was related to Labs as [[http://icinga.wmflabs.org|Labs's Icinga]] monitors the status of all Labs instances and that status was... [19:59:25] 10Wikimedia-Labs-General: multimedia-alpha.eqiad.wmflabs is not recognized (but works via bastion2) - https://phabricator.wikimedia.org/T65035#1236213 (10Aklapper) [19:59:31] 10Wikimedia-Labs-General: Create web version of metainformation table on replica servers - https://phabricator.wikimedia.org/T56703#1236217 (10scfc) Does this refer to the `meta_p` database (cf. https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Metadata_database)? [19:59:33] 10Wikimedia-Labs-General: multimedia-alpha.eqiad.wmflabs is not recognized (but works via bastion2) - https://phabricator.wikimedia.org/T65035#663766 (10Aklapper) @mtraceur: Is that still an issue? [20:02:21] 10Wikimedia-Labs-Other: "Edit with form" doesn't put page on watchlist (already fixed in Semantic Forms) - https://phabricator.wikimedia.org/T63104#1236221 (10Aklapper) 5Open>3Resolved a:3Aklapper >>! In T63104#653982, @Yaron_Koren wrote: > This was a bug in Semantic Forms, that has since been fixed - if W... [20:19:05] 10Wikimedia-Labs-Other: [tracking] OSM on Labs - https://phabricator.wikimedia.org/T60797#1236235 (10Aklapper) This task has no blocking tasks left - what is left to do here? Also note the #OpenStreetMap project created in T92562 - is this related? Or not? [20:32:28] 10Wikimedia-Labs-Other, 6Phabricator: phab-01.wmflabs.org test instance's statuses are out of date - https://phabricator.wikimedia.org/T76943#1236243 (10Aklapper) Currently the only admin on https://phab-01.wmflabs.org/ is "admin" (potentially also because that instance was re-set up at some point). And I admi... [20:36:26] 10Wikimedia-Labs-Other: "Edit with form" doesn't put page on watchlist (already fixed in Semantic Forms) - https://phabricator.wikimedia.org/T63104#1236245 (10scfc) 5Resolved>3Open It still doesn't work for me; I just edited https://wikitech.wikimedia.org/wiki/Shell_Request/Hchuying via https://wikitech.wiki... [20:44:14] 10Wikimedia-Labs-wikitech-interface: Upgrade SMW to 1.9 or later - https://phabricator.wikimedia.org/T62886#1236254 (10Aklapper) [20:45:08] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Wikimedia-Labs-wikitech-interface: OpenStackManager 'configure instance' clears any settings not visible in the configure GUI. - https://phabricator.wikimedia.org/T94851#1236259 (10Aklapper) [20:55:47] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, 10Tool-Labs-tools-Article-request, and 9 others: Labs' Phabricator tags overhaul - https://phabricator.wikimedia.org/T89270#1236279 (10Aklapper) So to make me finally start with at least some of the many many things in this task (which all sound... [20:59:46] 6Labs, 10LabsDB-Auditor, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, and 10 others: Labs' Phabricator tags overhaul - https://phabricator.wikimedia.org/T89270#1236285 (10Aklapper) [21:01:54] 10Wikimedia-Labs-wikitech-interface, 6operations, 7Regression: Some wikitech.wikimedia.org thumbnails broken (404) - https://phabricator.wikimedia.org/T93041#1236287 (10Aklapper) @Andrew: Any news? :-/ [21:02:35] tools-bastion-01 overloaded by puppet, it seems... even ssh takes ages [21:03:30] 10Wikimedia-Labs-wikitech-interface: Can't reset password on wikitech (Unicode passwords not accepted), due to LDAP/opendj? - https://phabricator.wikimedia.org/T58114#1236288 (10Aklapper) [21:04:07] 10Wikimedia-Labs-wikitech-interface: Transient "Unable to allocate memory for pool." - https://phabricator.wikimedia.org/T72435#1236291 (10Aklapper) @scfc: Have you ever faced this again since then? [21:04:47] 10Gerrit-Patch-Uploader: Wrap commit message - https://phabricator.wikimedia.org/T97229#1236292 (10Fomafix) >>! In T97229#1236186, @valhallasw wrote: > I'm not quite sure how you get wrapping at 45 characters, though. I can enter 45 **M** or 161 **l**, then I am at the end of the line. [21:05:03] 10Wikimedia-Labs-wikitech-interface: Stop using an older release of SMW and get back to tracking their master branch - https://phabricator.wikimedia.org/T75940#1236293 (10Aklapper) [21:05:12] 10Wikimedia-Labs-wikitech-interface: Upgrade SMW to 1.9 or later - https://phabricator.wikimedia.org/T62886#659361 (10Aklapper) [21:05:47] 10Wikimedia-Labs-wikitech-interface, 6Project-Creators: Migrate shell access request process to Phabricator - https://phabricator.wikimedia.org/T72627#1236299 (10Aklapper) @Andrew, @coren, @yuvipanda, can you please describe what "shell access" in the context of Labs is (and what it will be in the future)? [21:17:54] Hi, I have gotten tools access today but I can't login to either bastion or login.tools. Anyone able to help out ? [21:18:29] hey Daniel78 [21:18:50] Daniel78: Make sure you have setup your key pair and uploaded it [21:19:10] I did but it still say "Permission denied (publickey)" [21:19:24] and I uploaded a few hours ago [21:19:25] Daniel78, are the public key you're putting into preferences in the right format? [21:19:40] I copied the exact output from cat id_rsa.pub [21:20:20] Daniel78: set ssh to use the key with -i [21:20:32] ok, I'll try that [21:20:46] ssh -i [21:21:26] hm no, didn't help [21:22:04] Daniel78: run ssh-add and the path to id_rsa [21:22:29] Daniel78: what is your username? [21:22:38] make sure you have username@tools-login.wmflabs.org [21:25:13] Seem to be the same server, but it also fails. [21:26:10] Daniel78: are you running "ssh Daniel78@tools-login.wmflabs.org" as opposed to "ssh @tools-login.wmflabs.org" [21:26:29] 10Wikimedia-Labs-wikitech-interface, 6Project-Creators: Migrate shell access request process to Phabricator - https://phabricator.wikimedia.org/T72627#1236308 (10yuvipanda) Right now it is a manual user right granted that allows you access to the bastion hosts. I personally am not sure if we need that to be... [21:26:41] ("ssh tools-login.wmflabs.org" i mean, not @) [21:27:22] Daniel78: ah, so Daniel78 isn’t your shell name I guess [21:27:32] small 'd' [21:27:33] Daniel78: your shell name you should find in your wikitech preferences [21:27:42] I’m tailing the ssh log now [21:27:43] Yes, I tried with both Daniel78 and daniel78 [21:27:59] daniel78 is listed on the prefs page [21:28:03] hmm [21:28:26] Daniel78: try now? [21:28:33] with daniel78@tools-login.wmflabs.org [21:28:37] * YuviPanda just fudged something [21:28:39] just did [21:30:20] Daniel78: ah, different error now - says public key failure :) [21:31:20] > ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8q1R+K7O6SqpsIfjV+nCy2Tg9+oX3vJnsRPa9/SjRRf5DiMt8vUotPhhEz120U4hHfFIkgITQHMh1P5c2B8RA7/fPQpKlJ/YzJuuGs5hq6VVaVWDxPExdIGPpisrM+AO2PaVHWpAMThLCBe5j68pr3YmxXEzWq21BlWSUnGbj1B27FvhCjRZs5Aw5uIAXkkYJjE2MqRtIKXH93IRLyKepaygOxS2LrA7GjZI/dK/ [21:31:20] 7gAHjHsLcB/A7KZRqJ+xlqT063RU5DdeOXqpmCqxfDzJaGFSaoV0ebV5ThgdG9JSUa2qcUBFjDuIIqG6wzTAzcC5yrpKLErrdgQsxSpyLFhRZ danielb@linux-2lth [21:31:26] Daniel78: are these slashes part of the key? [21:32:39] what about the "> " at the start? [21:33:24] There are slashes in the key, so I think it looks right [21:33:42] I have no > in the start in the prefs view of the key [21:33:57] the > i added [21:34:03] well my copy pasting added [21:34:30] ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8q1R+K7O6SqpsIfjV+nCy2Tg9+oX3vJnsRPa9/SjRRf5DiMt8vUotPhhEz120U4hHfFIkgITQHMh1P5c2B8RA7/fPQpKlJ/YzJuuGs5hq6VVaVWDxPExdIGPpisrM+AO2PaVHWpAMThLCBe5j68pr3YmxXEzWq21BlWSUnGbj1B27FvhCjRZs5Aw5uIAXkkYJjE2MqRtIKXH93IRLyKepaygOxS2LrA7GjZI/dK/7gAHjHsLcB/A7KZRqJ+xlqT063RU5DdeOXqpmCqxfDzJaGFSaoV0ebV5ThgdG9JSUa2qcUBFjDuIIqG6wzTAzcC5yrpKLErrdgQsxSpyLFhRZ danielb@linux-2lth [21:34:42] that is from "more ~/.ssh/id_rsa.pub" [21:34:57] Daniel78: oh damn, I think there might be a stray newline in the keys you pasted on wikitech? [21:35:10] I see /\n at the end [21:35:34] oh, ok. Can you edit it or should I try to reupload ? [21:35:57] Daniel78: I can’t edit it sadly. can you go fix? [21:35:59] there’s a bug for it [21:36:56] 10Wikimedia-Labs-wikitech-interface: Transient "Unable to allocate memory for pool." - https://phabricator.wikimedia.org/T72435#1236318 (10scfc) 5Open>3Invalid a:3scfc No, and I have opened a similar amount of tabs since then multiple times. I assume moving the DB off the web server and similar work freed... [21:37:24] ok, I'll try [21:39:08] YuviPanda: Yay, that seem to have been it. Thanks ! [21:39:12] yw! [21:39:13] :) [21:39:13] I am in [21:39:16] that needs to be fixed [21:39:35] Daniel78, I've already added you to fpcbot [21:39:42] Yes would be good if the input was validated ofc :) [21:39:52] KTC: great, thanks [21:40:14] I might look at the rest tomorrow - but one step closer at least [21:40:59] Daniel78: yeah am fixing that now [21:41:09] YuviPanda: nice :) [21:41:31] Daniel78, I'm trying to figure out scheduling jobs on the grid, once I do that, you'll only need to provide the password and we can set the bot off :) [21:43:00] 10MediaWiki-extensions-OpenStackManager: SSH key import should accept stray whitespace before / after key - https://phabricator.wikimedia.org/T97235#1236321 (10yuvipanda) 3NEW [21:43:27] KTC: sounds great [22:21:15] 10Wikimedia-Labs-wikitech-interface: Strip linebreaks in uploaded ssh keys - https://phabricator.wikimedia.org/T51553#1236362 (10Krenair) [22:21:17] 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: SSH key import should accept stray whitespace before / after key - https://phabricator.wikimedia.org/T97235#1236361 (10Krenair) [22:22:17] 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: Strip linebreaks in uploaded ssh keys - https://phabricator.wikimedia.org/T51553#1236366 (10Krenair) [22:22:51] 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: Strip linebreaks in uploaded ssh keys - https://phabricator.wikimedia.org/T51553#1236370 (10yuvipanda) Note that this is broader than T97235 - that only does a trim :) [22:23:48] 10MediaWiki-extensions-OpenStackManager: Possible to register mail addresses with trailing newlines - https://phabricator.wikimedia.org/T73692#1236372 (10Krenair) [22:25:06] 10MediaWiki-extensions-OpenStackManager: cannot create foo.foo.wmflabs.org domain - https://phabricator.wikimedia.org/T56664#1236373 (10Krenair) [23:25:39] PROBLEM - Puppet failure on tools-webgrid-07 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:50:40] RECOVERY - Puppet failure on tools-webgrid-07 is OK: OK: Less than 1.00% above the threshold [0.0]