[06:32:58] PROBLEM - Puppet run on tools-exec-1428 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:42:19] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3149784 (10Marostegui) I am not going to import anything this week most likely, but I am advancing on the task to import s5, by starting the compression on... [07:12:58] RECOVERY - Puppet run on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [11:58:19] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/NeoAct was created, changed by NeoAct link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/NeoAct edit summary: Created page with "{{Tools Access Request |Justification=Learn |Completed=false |User Name=NeoAct }}" [13:27:16] 10Tool-Labs-tools-Article-request, 10ArticleFeedbackv5, 06Collaboration-Team-Triage, 10Notifications: Notification Extension for Public-sourced Article Request - https://phabricator.wikimedia.org/T162038#3150548 (10BoozyVonDrunkathon) [13:48:57] !log tools enable puppet on gridmaster [13:49:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:01:37] RECOVERY - Puppet staleness on tools-grid-master is OK: OK: Less than 1.00% above the threshold [3600.0] [14:42:13] I just created a page for the Wikimedia Tool Developers User Group on meta. https://meta.wikimedia.org/wiki/Wikimedia_Tool_Developers [14:43:54] that's pretty cool Freddy2001, I wonder if that has ever been tried before [15:16:06] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#3150959 (10madhuvishy) [15:16:08] 06Labs: Create replica.my.cnf for bkeegan on tools - https://phabricator.wikimedia.org/T134074#3150956 (10madhuvishy) 05Open>03Resolved a:03madhuvishy Closing since the user hasn't responded in a week, and this should be working. Please feel free to reopen if you run into any trouble with the replica file. [15:16:53] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#2315661 (10madhuvishy) [15:16:55] 06Labs, 10Tool-Labs: User sdesabbata has no replica.my.cnf - https://phabricator.wikimedia.org/T157176#3150960 (10madhuvishy) 05Open>03Resolved a:03madhuvishy Closing this as it's done and the user hasn't responded in a week. Please feel free to reopen if you run into any problems with the replica file. [15:17:59] 06Labs, 10Tool-Labs: Restore replica.my.cnf for toolsbeta.admin - https://phabricator.wikimedia.org/T109807#3150977 (10madhuvishy) 05Open>03declined Closing this as declined since user specific creds now exist for this usecase. [15:48:35] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#3151141 (10madhuvishy) 05Open>03Resolved a:03madhuvishy [15:50:24] I'm about to merge a patch that affects Horizon logins. It should be a no-op but please let me know if you see any bad behavior. [16:00:12] Well, apparently I broke Horizon logins. Working on it... [16:08:21] ok, logins working again [16:57:21] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3151459 (10madhuvishy) Hi @Cmjohnson, apologies for the delay here, we were working through the possibilities of what the next step... [17:01:26] 10Tool-Labs-tools-Article-request, 10ArticleFeedbackv5, 06Collaboration-Team-Triage, 10Notifications: Notification Extension for Public-sourced Article Request - https://phabricator.wikimedia.org/T162038#3151487 (10Matthewrbowker) 05Open>03Invalid a:05BoozyVonDrunkathon>03Matthewrbowker So... TL;DR... [17:26:25] 06Labs, 10Horizon, 13Patch-For-Review: Keystone is weirdly case-sensitive when checking 2fa creds - https://phabricator.wikimedia.org/T154860#3151573 (10Andrew) 05Open>03Resolved I believe this to be fixed. [18:22:04] 10Tool-Labs-tools-Article-request, 10ArticleFeedbackv5, 06Collaboration-Team-Triage, 10Notifications: Notification Extension for Public-sourced Article Request - https://phabricator.wikimedia.org/T162038#3150548 (10Niharika) >>! In T162038#3151487, @Matthewrbowker wrote: > So... TL;DR your bug report first... [18:49:34] (03CR) 10Legoktm: [C: 032] "This will take effect the next time the bot is restarted. I'm not going to do it now since this is just a trivial change." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/345087 (https://phabricator.wikimedia.org/T161421) (owner: 10MtDu) [18:50:04] (03Merged) 10jenkins-bot: Wikibugs realname should use HTTPS over HTTP [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/345087 (https://phabricator.wikimedia.org/T161421) (owner: 10MtDu) [18:50:15] (03CR) 10jenkins-bot: Wikibugs realname should use HTTPS over HTTP [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/345087 (https://phabricator.wikimedia.org/T161421) (owner: 10MtDu) [18:50:16] (03PS10) 10Paladox: Connect wikibugs to irc over ssl [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089) [19:10:26] PROBLEM - Puppet run on tools-checker-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:16:46] paladox: wikibugs able to do sasl? [19:16:56] Zppix not yet. [19:17:03] https://gerrit.wikimedia.org/r/328663 will do it though [19:17:10] sasl is better than ssl i believe [19:17:35] ssl is more secure as it is https [19:18:02] 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 05MW-1.27-release (WMF-deploy-2016-04-05_(1.27.0-wmf.20)), 13Patch-For-Review: Clean up after ldap->mysql keystone migration - https://phabricator.wikimedia.org/T126758#3151944 (10Andrew) On labtest I ran ``` ldapdelete -x -r -D 'u... [19:20:25] RECOVERY - Puppet run on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:40] 06Labs, 10Labs-Infrastructure: labvirt-star.eqiad.wmnet.crt expiring soon - https://phabricator.wikimedia.org/T162085#3152009 (10Andrew) [20:24:37] 06Labs, 06Operations: Investigate alternative RAID strategies for labstore1001/2 - https://phabricator.wikimedia.org/T162090#3152197 (10madhuvishy) [20:52:14] 06Labs, 06Operations, 13Patch-For-Review: Instance creation fails before first puppet run around 1% of the time - https://phabricator.wikimedia.org/T160908#3152276 (10Andrew) Oddly, labs instances seem to be getting their dhcp leases from install1001: lease { interface "eth0"; fixed-address 10.68.21.59;... [21:12:16] Anyone know why an irc bot would disconnect due to write error broken pipe? [21:15:19] connection closed [21:15:44] Platonides: by k8s im assuming? [21:15:58] ? [21:16:05] why would it do that [21:16:21] It never has done it before [21:16:27] And ive not changed anything recently [21:17:44] connections do disconenct on the internet… [21:18:14] Platonides: ive never seen that msg before [21:18:34] Usually if it is that its ping timeout or connection closed by remote host etc [21:21:30] Zppix: in this case the bot has a message pending [21:21:40] but was unable to send it because the connection was closed [21:27:28] 06Labs, 10MediaWiki-User-login-and-signup, 10wikitech.wikimedia.org: Fatal exception when attempting to log into Wikitech - https://phabricator.wikimedia.org/T160171#3152350 (10Zppix) Alright thanks for at least looking into it for me @bd808 [21:31:13] Ack thanks Platonides [21:33:23] 06Labs, 10MediaWiki-User-login-and-signup, 10wikitech.wikimedia.org: Fatal exception when attempting to log into Wikitech - https://phabricator.wikimedia.org/T160171#3090987 (10Reedy) I did make https://gerrit.wikimedia.org/r/#/c/322204/1/LdapAuthenticationPlugin.php in an attempt to fix stuff like this before [21:36:16] 06Labs, 10MediaWiki-User-login-and-signup, 10wikitech.wikimedia.org: Fatal exception when attempting to log into Wikitech - https://phabricator.wikimedia.org/T160171#3152373 (10Zppix) Maybe it just needs some updating @reedy? [21:52:07] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#3152417 (10Andrew) [21:55:18] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Create a Horizon panel for managing per-project sudo policies - https://phabricator.wikimedia.org/T162097#3152421 (10Andrew) [22:00:56] 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 05MW-1.27-release (WMF-deploy-2016-04-05_(1.27.0-wmf.20)), 13Patch-For-Review: Clean up after ldap->mysql keystone migration - https://phabricator.wikimedia.org/T126758#3152454 (10Andrew) >>! In T126758#3151944, @Andrew wrote: > ```... [22:04:12] Hello! I have a question about those webservers for the tools with URLs like https://tools.wmflabs.org/persondata/test.txt [22:04:34] These are usually pretty fast [22:05:31] But very often when I drink my breakfast coffee, which is about 7-8 MES (GMT-2) it takes minutes [22:05:54] Any known reason? [22:06:57] Wurgl: that's a very general question but there isn't a reason I know of for a scheduled slow period, you have to elaborate if it's more than one tool and the same requests and all kinds of things but broad strokes no [22:07:00] no reason I know of [22:08:22] chasemp: I tried others which I found here: https://tools.wmflabs.org/?status but during that thime, those others respond fast [22:10:03] Wurgl: sounds like an issue with this specific tool and it would be best to ask the maintainers [22:10:58] Hmm � Well, maybe I tried just my pages _with_ database access and the database is the reason. Maybe I shall try with nonsense text files, like that example [22:11:25] * Wurgl is the maintainer (at least since january) [22:11:34] Okay, junior maintainer [22:12:31] heh :) [22:13:45] I just found out that the tool is used, about 10-20k pages are requested each day, so that downtime is not so nice [22:15:21] performance debugging is one of the most time consuming activities you can get into [22:15:50] start a ticket and begin keeping track of things consistently and maybe a pattern will emerge or the next time it's slow see what the state of the tool is or compare request types [22:16:31] I will try, but start without a ticket [22:16:57] time wget https://tools.wmflabs.org/persondata/test.txt <-- Things like this to have numbers [22:17:14] the "time" at the beginning is important [22:57:25] chasemp: A script like this one should give enough information and a pair of requets every 10 Minutes ist fair enough? http://pasted.co/4312d906 [22:58:37] kind of ironic, isn't it? [23:01:22] Wurgl, also, GMT-2? where are you? [23:02:34] Germany [23:03:06] is germany trying to move somewhere into the middle of the atlantic ocean now too? :P [23:03:51] Aha! GMT+2 [23:03:52] Okay [23:04:02] not many places actually use GMT-2, I was secretly hoping you were on some obscure island :D [23:07:01] Okay, okay. But seems that these (most of) islands do not use the word "winter" https://upload.wikimedia.org/wikipedia/commons/e/e8/Standard_World_Time_Zones.png [23:07:07] Wurgl: mach echt besser direkt alles auf nen Ticket, spart Zeit. muss man sonst alles nachher wieder aus den logs rausklauben und wegen den ganzen Zeitzonen lesen die Leute auch nicht alles im Channel [23:07:37] * mutante disappears again after that random German comment :p [23:07:38] nohup ./wikitimes � copy paste [23:43:33] 06Labs, 10DBA: Prepare and check storage layer for khw.wikipedia - https://phabricator.wikimedia.org/T160870#3152669 (10Dereckson) 05Open>03stalled We aren't currently sure this wiki will be created soon per [[ https://lists.wikimedia.org/pipermail/langcom/2017-April/001207.html | this langcom announcement... [23:45:26] 06Labs, 10DBA: Prepare and check storage layer for kbp.wikipedia.org - https://phabricator.wikimedia.org/T160869#3152675 (10Dereckson) 05Open>03stalled Language engineering wants more translations before this wiki's language is added to MediaWiki, and we don't want to create a wiki before the language is a... [23:47:16] 06Labs, 10DBA: Prepare and check storage layer for dty.wikipedia.org - https://phabricator.wikimedia.org/T162102#3152679 (10Dereckson) [23:55:42] andrewbogott: Is the script for https://phabricator.wikimedia.org/T152043 running? Looks like it wasn't applied yet from a quick check on a few random wikis [23:55:50] But I imagine it might take a while