[00:06:37] 06Labs, 06Release-Engineering-Team, 10wikitech.wikimedia.org: Replace SemanticForms with PageForms - https://phabricator.wikimedia.org/T149866#2766962 (10Reedy) [00:07:48] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitech: Switch over from using extension SemanticForms to PageForms - https://phabricator.wikimedia.org/T149749#2766979 (10Paladox) [00:07:52] 06Labs, 06Release-Engineering-Team, 10wikitech.wikimedia.org: Replace SemanticForms with PageForms - https://phabricator.wikimedia.org/T149866#2766981 (10Paladox) [00:12:05] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitech: Switch over from using extension SemanticForms to PageForms - https://phabricator.wikimedia.org/T149749#2766987 (10Reedy) >>! In T149749#2761592, @Paladox wrote: > @Dereckson SemanticMediaWiki is no longer needed for... [00:21:05] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitech: Switch over from using extension SemanticForms to PageForms - https://phabricator.wikimedia.org/T149749#2761568 (10Krenair) I don't think we can deploy PageForms to the cluster given https://gerrit.wikimedia.org/r/#/... [00:43:33] (03CR) 10Dzahn: "So was it possible restart the ssh gerrit connection without also killing the IRC connection or was it not. Since the status before all th" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [00:48:38] (03CR) 10Dzahn: "also i think the whole whitelist.js plan with parsing a wiki page and creating a db might be overkill and we'd be fine with a simple list " [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [01:03:28] (03CR) 10Dzahn: [WIP] Adds a grrrit-wm restarting command for you to type in irc (035 comments) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [01:06:07] (03CR) 10Dzahn: [WIP] Adds a grrrit-wm restarting command for you to type in irc (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:07:59] (03PS18) 10Paladox: [WIP] Adds a grrrit-wm restarting command for you to type in irc [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) [02:13:43] (03PS19) 10Paladox: Adds a grrrit-wm restarting command for you to type in irc [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) [02:13:52] (03CR) 10Paladox: "Tested on the test bot and it worked." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:17:22] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitech: Switch over from using extension SemanticForms to PageForms - https://phabricator.wikimedia.org/T149749#2767127 (10Yaron_Koren) @Reedy - SF/PF's support for SMW (in terms of handling old versions) hasn't changed in a... [02:23:16] (03PS20) 10Paladox: Adds a grrrit-wm restarting command for you to type in irc [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) [02:24:48] marktraceur hi, the problem is all fixed now, https://gerrit.wikimedia.org/r/#/c/318976/ works now, done the testing. [02:24:48] ssh connection isent duplicated either too [02:24:49] I will be the judge of that [02:24:50] https://github.com/mscdex/ssh2/issues/491 [02:26:17] paladox => #wikimedia-bot-gerrit grrrit-wm: restart [02:26:17] info: Connecting to gerrit.. [02:26:18] Client disconnected [02:26:18] info: Connecting to gerrit.. [02:26:18] info: Connected; requesting stream-events [02:26:19] re-connecting to gerrit [02:26:19] info: Connected to event stream! [02:26:19] reconnected to gerrit [02:26:26] markteaceur ^^ [02:26:29] thats from the log [02:27:30] Yes, hello. [02:27:39] I did the survey. [02:31:53] (03CR) 10MarkTraceur: [C: 04-1] "This is what a non-WIP patch should look like. Fantastic. Thanks. A few more small issues, mostly relating to code style, and we can move " (032 comments) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:33:40] paladox: That's not my name [02:34:50] (03CR) 10Paladox: "@MarkTraceur" (032 comments) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:35:08] marktraceur not sure what you mean? [02:35:35] (03CR) 10MarkTraceur: Adds a grrrit-wm restarting command for you to type in irc (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:36:44] Debra: sorry for all the email spam on that. We don't collect email addresses for the survey and Research suggested that 1 week and 3 day reminders increased the response rate quite a bit. [02:36:54] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitech: Switch over from using extension SemanticForms to PageForms - https://phabricator.wikimedia.org/T149749#2761568 (10Dereckson) [02:37:42] (03CR) 10MarkTraceur: Adds a grrrit-wm restarting command for you to type in irc (032 comments) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:38:34] paladox: Also it looks like your test code is still deployed to the production instance, based on the above behaviour of grrrit-wm [02:39:22] marktraceur, yeh, when ive tested on the test bot, i try it on the prod bot since well some changes that work on the test bot doint work on the prod bot [02:39:49] paladox: So what, you might ask, is the point of having the test bot, if you also test on the production bot? [02:40:25] marktraceur well i am just making sure it starts on the prod bot, ive done all the testing on the test bot so i would think it works [02:40:56] (03CR) 10Paladox: Adds a grrrit-wm restarting command for you to type in irc (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:42:05] (03PS21) 10Paladox: Adds a grrrit-wm restarting command for you to type in irc [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) [02:42:54] (03CR) 10MarkTraceur: Adds a grrrit-wm restarting command for you to type in irc (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:43:53] (03CR) 10Paladox: "@MarkTraceur" (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:45:14] (03CR) 10MarkTraceur: [C: 04-1] Adds a grrrit-wm restarting command for you to type in irc (032 comments) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:45:47] (03CR) 10MarkTraceur: Adds a grrrit-wm restarting command for you to type in irc (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [02:52:36] 06Labs, 10Labs-Infrastructure: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4 - https://phabricator.wikimedia.org/T149870#2767164 (10madhuvishy) [02:54:37] marktraceur doing it your way dosen't allow the ssh connection to automatically try and reconnect [02:55:38] paladox: I have no idea why it would be any different [02:56:02] Because this part [02:56:02] }).on('close', function() { [02:56:03] console.log('Client disconnected'); [02:56:07] needs to call startRelay [02:56:15] so i have moved it down where you said [02:56:34] so when it disconnects, it dosen't call startRelay. [02:56:40] marktraceur ^^ [02:56:50] What? [02:57:07] paladox: I'm saying, in subscribeToGerritStream, check if there's an active connection, and call sshConn.end() there [02:57:16] I did [02:57:27] paladox: It doesn't affect anything else, it just makes sure the SSH connection is closed before it...okay, so why would that change anything else? [02:58:02] paladox: Oh, I'm sorry, I see what you mean [02:58:06] Yep [02:58:35] paladox: Not only do you have to move the if ( sshConn ) { sshConn.end(); } bit, you also have to put startRelay(); in its place [02:58:46] I have [02:58:52] OK, in that case I'm lost again. [02:58:53] (03PS22) 10Paladox: Adds a grrrit-wm restarting command for you to type in irc [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) [02:58:58] * marktraceur looks [02:59:27] Ah. [02:59:31] Yep [03:00:52] paladox: OK, it might not look like it's saving any effort, but keep startRelay() in that end handler, just put sshConn = null; in front of it. [03:01:06] Ok [03:01:32] marktraceur not sure what you mean? [03:01:33] paladox: The whole point of this is to prevent duplicate SSH connections, and let people (re)start the connection with startRelay no matter what the situation [03:01:39] Yes [03:01:45] Which is what i did and it worked [03:01:59] but then you told me to move startRelay and the sshcon [03:02:16] paladox: No, I told you to move sshConn.end() and *add* startRelay [03:02:22] yep [03:02:28] which is what i did [03:02:29] paladox: Admittedly I didn't think about the infinite loop problem, but now I see that's an issue [03:02:43] paladox: No, you removed the startRelay call in .on( 'close' [03:02:54] Yep, so i re add it to close? [03:03:05] paladox: Put it back, but right before you call it, do sshConn = null; so there's no infinite loop. [03:03:46] so [03:03:47] }).on('close', function() { [03:03:47] console.log('Client disconnected'); [03:03:48] sshConn = null; [03:03:48] startRelay(); [03:03:48] ? [03:03:49] Pretty sure the bot would crash almost immediately due to infinite recursion if you tested that [03:03:54] paladox: Exactly, yes [03:03:57] ok [03:04:10] paladox: It may or may not be worthwhile to add "; restarting" to the console entry [03:04:25] Ok [03:06:11] marktraceur testing it [03:06:13] fails with error: Error: No response from server [03:06:29] I might as well go back to how i set it before which worke [03:06:30] workd [03:06:33] woked [03:06:35] worked [03:08:10] bd808: I liked the final e-mail. It made me finally respond. :P [03:08:33] paladox: Where did the error come from, though? [03:08:49] marktraceur came from because [03:08:49] }).on('close', function() { [03:08:49] console.log('Client disconnected'); [03:08:50] sshConn = null; [03:08:50] startRelay(); [03:08:53] we did that ^^ [03:09:00] paladox: use a pastebin [03:09:05] Sorry [03:09:13] Why would that prevent a response from the server? [03:09:25] I mean, maybe I'm missing something, but I'm pretty sure that wouldn't affect anything [03:09:31] Oh [03:09:44] There must be something else causing "no response from server" [03:09:47] It failed at [03:09:47] throw err; [03:10:00] That's a super useful traceback [03:10:47] marktraceur https://phabricator.wikimedia.org/P4358 [03:11:23] paladox: thank you, and sorry for not saying please before [03:11:32] Your welcome :) [03:11:51] paladox: That happens consistently? [03:12:08] marktraceur well not that i noticed [03:12:11] it was working before [03:12:20] paladox: I mean, it happens consistently with the new code? [03:12:34] Yes [03:12:42] marktraceur it was working with https://gerrit.wikimedia.org/r/#/c/318976/20/src/relay.js [03:12:51] I understand that [03:12:54] Ok [03:13:07] I'm trying to figure out what the problem is with the new code, which to my mind shouldn't cause the server to simply not respond. [03:13:32] oh [03:13:48] sorry, i have to go now - 03:13 am, have to get up at 7 [03:13:58] paladox: OK, let's talk about it tomorrow :) [03:14:08] yep thanks :) [03:25:28] 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review, 07Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2767234 (10Andrew) Apologies if I'm repeating previous comments... This issue is produced in two stages: 1) designa... [03:28:35] 06Labs, 10Tool-Labs: Linkwatcher spawns many processes without parent - https://phabricator.wikimedia.org/T123121#2767235 (10Beetstra) @valhallasw The bot moved two days ago again .. it is now on 1218. [03:56:15] 10Labs-project-Wikistats: new possible wikifarms / hives detected - check for lists - https://phabricator.wikimedia.org/T38570#2767270 (10MarkAHershberger) RobiH writes: > Maybe even potential to join forces with them. The current long-term plan is to move WikiApiary to labs and to share bot infrastructure. A... [04:10:44] 10Labs-project-Wikistats: new possible wikifarms / hives detected - check for lists - https://phabricator.wikimedia.org/T38570#2767305 (10Dzahn) >>! In T38570#2767270, @MarkAHershberger wrote: > but we don't have any really solid action items yet. > (The first would be "Move WikiApiary"...) [ ] create task to... [04:16:10] 10Labs-project-other: move WikiApiary to Labs - https://phabricator.wikimedia.org/T149874#2767311 (10Dzahn) [04:17:00] 10Labs-project-other: move WikiApiary to Labs - https://phabricator.wikimedia.org/T149874#2767326 (10Dzahn) [04:20:00] 10Labs-project-other: move WikiApiary to Labs - https://phabricator.wikimedia.org/T149874#2767329 (10Dzahn) [04:39:20] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4 - https://phabricator.wikimedia.org/T149870#2767164 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by madhuvishy on neodymium.eqiad.wmnet for hosts: ``` ['labstore2003.c... [05:04:36] !log deployment-prep beginning to move the rest of beta to the new puppetmaster [05:04:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [05:04:54] Fun [05:05:35] Wait does every tool acct gets its own log? [05:05:41] Sal rather [05:07:39] Zppix|mobile, service groups in tools can have their own SALs, yes [05:07:48] as if they were proper labs projects [05:08:03] I mean logs within the sal [05:08:34] what? [05:26:04] Zppix|mobile: yes. !log tools. ... works for every shared tool account (which is the less correct name for "service groups" as mentioned above) [05:26:55] so like tools.zppixbot? [05:27:09] yes [05:27:55] see https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL for example [05:42:48] 10Labs-project-other: move WikiApiary to Labs - https://phabricator.wikimedia.org/T149874#2767367 (10MarkAHershberger) You are the best. Thanks, @Dzahn! [06:08:10] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4 - https://phabricator.wikimedia.org/T149870#2767395 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by madhuvishy on neodymium.eqiad.wmnet for hosts: ``` ['labstore2004.c... [06:40:37] PROBLEM - Puppet run on tools-exec-1417 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:56:13] PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:12:37] 06Labs, 10Labs-Infrastructure, 10Tool-Labs, 10DBA, 10Wikimedia-Developer-Summit (2017): Labsdbs for WMF tools and contributors: get more data, faster - https://phabricator.wikimedia.org/T149624#2767439 (10jcrespo) [07:20:39] RECOVERY - Puppet run on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [07:31:14] RECOVERY - Puppet run on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [07:59:36] 06Labs, 10Labs-Infrastructure, 10DBA, 07Availability: Decide between proxysql and haproxy for labsdbproxy service - https://phabricator.wikimedia.org/T149844#2767472 (10Marostegui) I will also add my thoughts about this. As I expressed during the meeting, if possible I would try to go for proxySQL. Jaime... [08:10:29] Hi [08:10:41] can a root please run "chown -R kelson:wikidev mwoffliner/" on the nfs drive [08:10:42] ? [08:10:55] Looks like I have lost the control of my files during the migration [08:11:36] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4 - https://phabricator.wikimedia.org/T149870#2767493 (10madhuvishy) The WMF auto reimage script didn't work out due to a puppet issue we discovered in this process (ticket coming soon... [08:21:56] 10Labs-project-Wikistats: new possible wikifarms / hives detected - check for lists - https://phabricator.wikimedia.org/T38570#2767555 (10Kghbln) I guess the domain should be kept as primary instead of . I dunno if this will be possible with labs though I do not think tha... [08:23:08] 10Labs-project-Wikistats: new possible wikifarms / hives detected - check for lists - https://phabricator.wikimedia.org/T38570#2767558 (10Kghbln) Never mind, will post at T149874 [08:24:23] 10Labs-project-other: move WikiApiary to Labs - https://phabricator.wikimedia.org/T149874#2767559 (10Kghbln) > request wikiapiary.com domain to be added to labs domains / (transfer domain ?) I also think that the domain should be kept as primary instead of . I dunno if t... [08:43:06] Hi, I'm trying to run "select page_title, rev_user_text, count(*) from revision join page on rev_page=page_id where page_title="Q27655022" and rev_timestamp >= "20161002120000" group by rev_user_text order by count(*) desc;" at wikidatawiki_p from toollabs. It seems it works correctly but after a two hours of real time I killed it. It threw "ERROR 1030 (HY000): Got error 1152 "Unknown error... [08:43:08] ...1152" from storage engine TokuDB". Could anybody explain me what it means? [08:45:05] I would like to have a list of users who edited Q27655022 later than after 2. 10. 2016 12:00 (timestamp 20161002120000) ordered by number of edits. [08:45:27] I group by rev_user_text simply because if I group by rev_user only it won't works for anonymous users. [09:19:30] madhuvishy: hi, can you help me with the permissions problem on mwoffliner directory? [09:21:28] Kelson: Hi! sorry it's past 2am here, could you file a ticket, I'll look at it in the morning? [09:21:38] is this on labs or tools? [09:22:00] madhuvishy: oops, labs [09:22:15] mhashemi: as far as I understand the difference beetween both of them [09:22:22] mhashemi: wil try to open a ticket [09:22:24] thx [09:24:31] there was no labs or tools data migration so i'm not sure what happened. [09:24:32] thanks [09:27:24] madhuvishy: a few months ago the nfs serveur was moved/upgraded in labs AFAIK [09:28:16] 06Labs, 10wikitech.wikimedia.org, 07Puppet: Puppet failure e-mail from labs/wikitech contains wrong url to wikitech - https://phabricator.wikimedia.org/T149883#2767649 (10Mvolz) [09:53:44] 06Labs, 10Tool-Labs, 15User-Addshore, 15User-bd808: Possible use of tools-lab-elasticsearch cluster - https://phabricator.wikimedia.org/T149709#2767733 (10Addshore) [10:45:04] 06Labs, 10MediaWiki-extensions-Newsletter: Clear 'nl_*' tables in http://newsletter-test.wmflabs.org/ - https://phabricator.wikimedia.org/T149651#2767845 (10Qgil) I think the best way to find this answer is to ask the Labs people on IRC... [10:53:34] 06Labs, 10MediaWiki-extensions-Newsletter: Clear 'nl_*' tables in http://newsletter-test.wmflabs.org/ - https://phabricator.wikimedia.org/T149651#2758891 (10jcrespo) @01tonythomas could you please provide the database name? [11:04:39] 06Labs: Fix mwoffliner directory permission problem on Labs NFS - https://phabricator.wikimedia.org/T149888#2767882 (10Kelson) [11:57:33] !log tools.zppixbot updating public_html from gerrit repo [11:57:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [12:00:19] 06Labs, 10MediaWiki-extensions-Newsletter: Clear 'nl_*' tables in http://newsletter-test.wmflabs.org/ - https://phabricator.wikimedia.org/T149651#2767986 (1001tonythomas) >>! In T149651#2767873, @jcrespo wrote: > @01tonythomas could you please provide the database name? > > **Edit**: Oh, this is labs, labs an... [12:31:59] Kelson: I'm guessing that's on the scratch share? [12:32:32] chasemp: I have open a bug with the whole path (I see on my VM) https://phabricator.wikimedia.org/T149888#2767882 [12:33:26] chasemp: not knowing much more than that [12:33:45] Kelson: what's your user id? [12:33:45] chasemp: do you want the mount cmd? [12:33:49] nah [12:33:53] kelson [12:33:56] if you type 'id' at a terminal [12:33:59] what number is there? [12:34:09] first number on the left [12:34:34] chasemp: uid=2381(kelson) [12:35:20] ok Kelson I know who you are so I just went ahead and fixed it up :) sorry, that must have lost perms in the move? not sure [12:35:26] give it a whirl and let me know when you can [12:41:32] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 - https://phabricator.wikimedia.org/T140452#2465396 (10Marostegui) How are you planning to migrate (or populate data) in this new servers? [12:57:59] chasemp: thx very much, seems to work perfectly [12:59:01] 06Labs: Fix mwoffliner directory permission problem on Labs NFS - https://phabricator.wikimedia.org/T149888#2768123 (10Kelson) 05Open>03Resolved This has been fixed by @Chase [12:59:33] 06Labs: Fix mwoffliner directory permission problem on Labs NFS - https://phabricator.wikimedia.org/T149888#2767882 (10chasemp) actually @chasemp :) We get mixed up so much @chase :D [13:00:00] chasemp: just remarked my error, have updated the ticket. [13:00:12] heh no worries [13:00:21] I feel bad for whoever @chase is they are often ping'd on my stuff [13:00:46] chasemp: chase has been an employee of WMUK for many years (until this summer) [13:01:00] didn't realize [13:09:23] hello is "atop" required on instances (e.g. for graphite etc) I notice it's log file size growing and I'm very constrained with that partition [13:11:14] atop -a -w /var/log/atop.log 600 is the command that runs upon boot [13:13:53] chippy: no it's a standalone tool, there is a cleanup cron for it iirc but you're welcome to one-time remove things as you need within your project [13:14:05] chasemp thanks [13:14:25] Im reading a couple of tickets in phabricator about similar lack of space in /var partitions [13:48:47] 10Labs-project-other, 06Developer-Relations: move WikiApiary to Labs - https://phabricator.wikimedia.org/T149874#2768326 (10Qgil) [14:40:31] (03PS1) 10MarcoAurelio: Add privacy disclaimer per WMF Labs ToS [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319577 [14:42:10] (03CR) 10MarcoAurelio: [C: 032] Add privacy disclaimer per WMF Labs ToS [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319577 (owner: 10MarcoAurelio) [14:42:42] (03Merged) 10jenkins-bot: Add privacy disclaimer per WMF Labs ToS [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319577 (owner: 10MarcoAurelio) [14:45:56] (03Draft2) 10MarcoAurelio: Fix privacy statement link [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319580 [14:46:03] (03Draft1) 10MarcoAurelio: Fix privacy statement link [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319580 [14:46:49] (03CR) 10MarcoAurelio: [C: 032] Fix privacy statement link [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319580 (owner: 10MarcoAurelio) [14:48:03] (03Merged) 10jenkins-bot: Fix privacy statement link [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319580 (owner: 10MarcoAurelio) [14:49:36] !log tools.stewardbots Added privacy statement per Labs terms of use [14:49:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [14:51:10] (03PS2) 10MarcoAurelio: Add COPYING file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319018 (owner: 10BryanDavis) [14:51:29] my instance is suddenly missing several files and some look like their older versions, how is this possible? [14:57:34] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 - https://phabricator.wikimedia.org/T140452#2768500 (10chasemp) >>! In T140452#2768084, @Marostegui wrote: > How are you planning to migrate (or populate data) in these new servers... [14:59:33] (03CR) 10MarcoAurelio: [C: 032] "Thanks! We may revisit the licensing in the future, but for now that's the license we seem to be using." [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319018 (owner: 10BryanDavis) [14:59:54] (03Merged) 10jenkins-bot: Add COPYING file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319018 (owner: 10BryanDavis) [15:12:13] bd808: I'm about to deploy those rate-limits for stewardbot if you're okay [15:12:56] mafk: works for me. yell at me if you find that it misbehaves [15:13:14] + we have a "revert" button so... [15:13:34] that bot class needs a bit of cleanup generally and a large does of PEP8 formatting help [15:13:42] *dose [15:14:42] bd808: yep, the code is very old; but since I only do small changes I can't really help cleanup that [15:15:21] my knowledge of python is reduced to starting and stopping pwb scripts and the like [15:15:29] :) [15:15:50] I'm mostly acting as the janitor: typo fix here and there, that stuff [15:15:56] I have somehow learned a lot about python irc bots in the last year or so [15:16:16] (03PS4) 10MarcoAurelio: StewardBot: ratelimit @steward pings [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319019 (https://phabricator.wikimedia.org/T148110) (owner: 10BryanDavis) [15:17:20] (03CR) 10MarcoAurelio: [C: 032] StewardBot: ratelimit @steward pings [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319019 (https://phabricator.wikimedia.org/T148110) (owner: 10BryanDavis) [15:18:16] (03Merged) 10jenkins-bot: StewardBot: ratelimit @steward pings [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319019 (https://phabricator.wikimedia.org/T148110) (owner: 10BryanDavis) [15:18:57] now merging to labs [15:20:20] !log tools.stewardbots [[gerrit:319019|Ratelimit [!@]steward IRC command]] (T148110) [15:20:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [15:23:33] seems it's not working? :) [15:28:24] ah, right [15:28:39] 06Labs, 06Operations, 10netops, 05Prometheus-metrics-monitoring: Firewall rules production/labs for prometheus-node-exporter - https://phabricator.wikimedia.org/T149253#2768566 (10fgiunchedi) [15:28:42] that's because the copy in public_html is not the one working [15:28:57] I shall update the shadow copy which we do really use [15:29:06] <_< [15:29:58] mafk: sneaky! Is that because of the weak way that secrets are handled in the configuration? [15:30:41] maybe [15:31:13] if you have saw our directory, public_html seems more like a directory where people can freely copy our code [15:31:31] and there are some other internal folders where the working copies of our tools are stored [15:34:44] there's a lot of stuff that needs to be put in order there :( [15:35:34] 10Tool-Labs-tools-Database-Queries, 06Project-Admins: Archive Tool-Labs-tools-Database-Queries project - https://phabricator.wikimedia.org/T107699#2768592 (10Aklapper) For the records, creation of this project was originally requested in T51825 by @liangent (and @valhallasw). >>! In T107699#1502332, @Bugrepor... [15:53:13] bd808: seems there's a syntax error [15:53:17] https://phabricator.wikimedia.org/T148110#2768647 [15:56:08] mafk: ugh. I'll take a look [15:56:24] stewardbot.err is flooded with that [15:57:44] woah. something messed up there [15:57:56] did I touch that? [15:58:35] nope [15:59:52] that elif is indented wrong or something removed an if before it [16:00:12] line 838 is the result of your additions/removals of content (obviously) :) [16:00:20] but the content there is the same as before [16:02:24] this code is a bit of a mess [16:03:39] The if True: on 836 has no else... [16:06:27] mafk: do you know what version of the code was running before you updated? [16:07:02] bd808: yes, I think so: https://phabricator.wikimedia.org/diffusion/TSTW/history/master/StewardBot/StewardBot.py [16:07:12] the one before your change [16:09:51] mafk: that version has the same bug, just on line 817 instead [16:10:10] so... that wasn't reunning [16:10:15] *running [16:11:47] but the bot did joined the channels last time :? [16:12:37] I think your tab->spaces patch may be how things got strangely aligned [16:16:15] I'm going to revert, see what error messages I get from the .err file [16:16:27] mafk: *nod* [16:16:36] is that an ok? [16:16:49] yes. revert and see what happens [16:17:59] (03PS1) 10MarcoAurelio: Revert "StewardBot: ratelimit @steward pings" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319599 [16:19:27] well, no jenkis for now, so I'll push it myself [16:19:46] (03CR) 10MarcoAurelio: [C: 032 V: 032] Revert "StewardBot: ratelimit @steward pings" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319599 (owner: 10MarcoAurelio) [16:20:13] you can just `git reset --hard 3b1b988` in the live clone [16:20:29] that's the sha1 before my patch [16:24:17] now it gives the same error message, but on line 817 <_< [16:24:22] * mafk wtf's [16:24:51] yeah. so ... the tab to spaces cleanup patch you did looks to be the culprit [16:25:06] not sure why it did worked before [16:26:19] do a `git reflog` and figure out what hash was active before [16:27:36] I think 5738f90 expanded tabs to 4 spaces instead of 8 spaces and that changed things [16:30:49] mafk: ok I see what happened in 5738f90. It is the tab expansion [16:31:21] so, is if fix-able? [16:31:22] the line that is throwing the error is supposed to match to the prior elif on line 765 [16:32:20] elif gublock? [16:32:20] I think so, yes. The easiest thing is probably to do a commit that bascially redoes 5738f90 with 8 space tab expansion instead of 4 [16:32:30] match w/ that? [16:33:37] yeah. on line 812-815 things were indented with 2 tabs == 16 spaces. That got changed to only 8 spaces [16:34:19] there may be more places that the expansion caused issues too. I haven't reviewed all of the lines yet [16:35:16] an easy way to do so? [16:36:48] mafk: I'm going through line by line. NOt fast by should be effective [16:41:22] bd808: maybe resetting before my tab-changes? [16:41:44] mafk: for now, yeah. do that and get the bot back up [16:42:03] I'll do a new whitespace patch and then my patch back on top of it [16:48:21] not working either [16:48:32] people will kill me [16:48:39] well... [16:48:47] mafk: it was just that one block. patch incoming [16:49:38] (03PS1) 10BryanDavis: Fix indenting from 5738f90 [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319611 [16:50:46] mafk: ^ the tab to 4 space expansion worked everywhere except this small section where indent was double tab and not mixed tab+spaces [16:52:28] (03CR) 10MarcoAurelio: [C: 032 V: 032] Fix indenting from 5738f90 [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319611 (owner: 10BryanDavis) [16:59:40] (03PS1) 10BryanDavis: Revert "Revert "StewardBot: ratelimit @steward pings"" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319615 [16:59:48] (03CR) 10jenkins-bot: [V: 04-1] Revert "Revert "StewardBot: ratelimit @steward pings"" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319615 (owner: 10BryanDavis) [17:00:08] 06Labs, 06Operations, 10netops, 05Prometheus-metrics-monitoring: Firewall rules production/labs for prometheus-node-exporter - https://phabricator.wikimedia.org/T149253#2768875 (10akosiaris) 05Open>03Resolved a:03akosiaris After a couple of rounds, finally done. Tested with telnet on a few hosts and... [17:02:18] bd808: I need an old working copy with the SQL dbs and passwords [17:02:29] I'll manage to get one [17:02:35] matanya has [17:04:34] !log ores deployed ores-wmflabs-deploy:74699db to staging [17:04:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [17:15:35] (03CR) 10MarcoAurelio: "recheck" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319611 (owner: 10BryanDavis) [17:16:07] mafk: there's a bug in the revert-revert that I need to fix [17:16:07] !log ores deployed ores-wmflabs-deploy:ffb5dbb to staging (fixes logging bug) [17:16:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [17:16:45] (03CR) 10MarcoAurelio: "recheck" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319615 (owner: 10BryanDavis) [17:17:10] bd808: don't worry, I'm not merging anything for now [17:17:17] mafk: zuul is down actually [17:17:20] (03PS2) 10BryanDavis: Revert "Revert "StewardBot: ratelimit @steward pings"" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319615 [17:17:26] is it still? [17:17:43] https://integration.wikimedia.org/ throws 403 [17:17:54] and as log as it does, I guess it's not available [17:19:07] it's v+2 some changes... [17:20:05] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 - https://phabricator.wikimedia.org/T140452#2768940 (10jcrespo) [17:23:31] !log ores deployed ores-wmflabs-deploy:ffb5dbb [17:23:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [17:26:30] (03PS3) 10BryanDavis: Restore "StewardBot: ratelimit @steward pings" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319615 [17:29:40] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 - https://phabricator.wikimedia.org/T140452#2768980 (10jcrespo) Socket authentication allow for easier administration: ``` $ sudo mysql Welcome to the MariaDB monitor. Commands... [17:34:14] bd808: I've managed to get the bot back, with an old 2014 file copy I had stored elsewhere [17:34:23] lol [17:34:47] much better than nothing :) [17:36:12] !log manually ran "sudo mkdir /srv/log; sudo mkdir /srv/log/ores; sudo service celery-ores-worker restart" on ores-worker-(05|06|07|08|09|10) [17:36:12] Unknown project "manually" [17:36:18] !log ores manually ran "sudo mkdir /srv/log; sudo mkdir /srv/log/ores; sudo service celery-ores-worker restart" on ores-worker-(05|06|07|08|09|10) [17:36:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [17:38:37] 06Labs, 07Documentation: Update documentation for the various phab Labs tags - https://phabricator.wikimedia.org/T129043#2769067 (10Aklapper) > * Add a description of the various Labs tags[1] that come up in autocomplete for "labs", at https://phabricator.wikimedia.org/project/profile/832/ [[ https://phabricat... [17:38:43] 06Labs, 07Documentation: Update documentation for the various phab Labs tags - https://phabricator.wikimedia.org/T129043#2769069 (10Aklapper) > * Add a description of the various Labs tags[1] that come up in autocomplete for "labs", at https://phabricator.wikimedia.org/project/profile/832/ [[ https://phabricat... [17:44:52] 10Tool-Labs-tools-Zppixbot, 07Documentation, 07Tracking: Write documentation - https://phabricator.wikimedia.org/T149500#2769103 (10Zppix) [17:44:53] 06Labs, 07Documentation: Update documentation for the various phab Labs tags - https://phabricator.wikimedia.org/T129043#2769102 (10Zppix) [17:57:40] 06Labs, 10Beta-Cluster-Infrastructure: Move deployment-prep to role::puppetmaster::standalone - https://phabricator.wikimedia.org/T149620#2769165 (10AlexMonk-WMF) [18:14:03] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Implement a frontend failover solution for labsdb replicas - https://phabricator.wikimedia.org/T141097#2769216 (10jcrespo) [18:14:05] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 - https://phabricator.wikimedia.org/T140452#2769214 (10jcrespo) 05Open>03Resolved Monitoring of the new hosts can be seen at: https://grafana.wikimedia.org/dashboard/db/mysql?... [18:17:45] 06Labs, 10Labs-Infrastructure, 10DBA: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2769241 (10jcrespo) [18:21:52] (03PS1) 10BryanDavis: PEP8 fixes for StewardBot.py [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319636 [18:21:54] (03PS1) 10BryanDavis: PEP8 fixes for SULWatcher/ini2mysql.py [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319637 [18:21:56] (03PS1) 10BryanDavis: PEP8 fixes for SULWatcher/SULWatcher.py [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319638 [18:21:58] (03PS1) 10BryanDavis: Fix basic PEP8 issues using autopep8 [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/319639 [18:38:53] 06Labs, 10Beta-Cluster-Infrastructure: Move deployment-prep to role::puppetmaster::standalone - https://phabricator.wikimedia.org/T149620#2769393 (10AlexMonk-WMF) 05Open>03Resolved I shut down the old instance, will probably delete it in a week or two [19:15:30] !log deployment-prep test [19:15:33] nope it broke [19:17:29] !log tools.stashbot restarted [19:17:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [19:20:30] bd808: he managed to fix it [19:21:33] matanya: good. It sounded like the state of that tool is a bit messy [19:22:22] bd808: very, old code base from 2004 i patched and patched, but didn't refactor, cause no time [19:22:49] and the public part overwrites the private part with creds etc [19:22:59] it needs some cleanup. I'm glad to submit a few more patches [19:24:16] it would be nice to get a newer version of irclib in use too. The globally installed one is pretty out of date. If we get it all properly using a virtualenv then it will be easier to move to kubernetes if desired too [19:25:05] bd808: i'd really like that [19:25:20] can i give you owner rights there, and you'll play around as you see fit ? [19:25:26] Krenair: did you notice what was wrong with stashbot or just that it was somehow dead? [19:25:34] matanya: spooky! [19:25:41] i only have time to bring it up when it crashes, sadly [19:25:51] the story of too many useful tools [19:26:53] matanya: if you want to add me I can at least take a look and maybe write up a migration plan from what I see running [19:27:25] yeah [19:27:40] will do so shortly [19:34:41] bd808: added [19:44:32] bd808, just that it was running according to kubectl but was not responding on irc [19:45:27] ok. The current logging is ephemeral, so once the pod is restarted I can't see what happened. I guess I should look into changing that [19:46:37] it's supposed to notice irc disconnects and rejoin automatically but that takes a few minutes. [19:47:36] hmmm... looks like maybe it was still connected but messed up in come other way based on the part + join in my scrollback [20:09:39] 06Labs, 06Operations, 07Tracking: Add config option in tools webservice debian package to write logs to /dev/null - https://phabricator.wikimedia.org/T149946#2769686 (10madhuvishy) [20:10:46] 06Labs, 10Labs-Infrastructure, 06Services (watching): Novaproxy in labs strips out the ETag header if gzip is enabled - https://phabricator.wikimedia.org/T148676#2769701 (10Pchelolo) 05Open>03Resolved a:03Pchelolo We've worked around this my not gzipping the content for requests to #mobile-content-serv... [20:20:57] andrewbogott, can you put me in the services project? [20:22:38] yep, one second [20:23:27] 06Labs, 10wikitech.wikimedia.org, 07Puppet: Puppet failure e-mail from labs/wikitech contains wrong url to wikitech - https://phabricator.wikimedia.org/T149883#2769728 (10Krenair) 05Open>03Invalid Looks fine to me in puppet, https://gerrit.wikimedia.org/r/#/c/271978/ would've fixed this if that instance... [20:23:28] Krenair: done [20:23:34] thanks [20:24:45] andrewbogott, um, it doesn't work [20:24:55] I can't log into the instances [20:25:01] https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Services&action=history doesn't show a change [20:26:32] I added you on the commandline because I was there already… are you not appearing in the Members list? [20:26:34] I see you there [20:27:42] krenair@bastion-01:~$ ldapsearch -x cn=project-services | grep krenair [20:27:42] krenair@bastion-01:~$ [20:27:46] nope, definitely didn't work [20:30:01] andrewbogott, what command did you run? [20:30:17] openstack role add --user krenair --project services user [20:30:25] but, ok, I'll remove/replace you in wikitech [20:30:27] okay that doesn't actually let people log into instances [20:30:35] oh you're right :) [20:30:36] instances check ldap, they still can't connect to keystone [20:30:44] PROBLEM - Puppet run on tools-puppetmaster-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:31:46] Krenair: added you via wikietch [20:31:50] yep [20:31:52] that worked [20:31:54] thank you andrewbogott [20:32:10] -rw-r--r-- 1 root root 0 Mar 16 2016 /var/lib/puppet/state/agent_catalog_run.lock [20:32:11] :| [20:32:57] it would actually have broken before that because we know it didn't pick up https://gerrit.wikimedia.org/r/#/c/271978/ [20:34:48] 06Labs, 10wikitech.wikimedia.org, 07Puppet: Puppet failure e-mail from labs/wikitech contains wrong url to wikitech - https://phabricator.wikimedia.org/T149883#2769745 (10Krenair) Okay, I just literally logged in, found a `/var/lib/puppet/state/agent_catalog_run.lock` file from March, deleted it, ran puppet,... [20:43:45] !log services fixed puppet in 'citoid' instance by removing /var/lib/puppet/state/agent_catalog_run.lock from March [20:43:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Services/SAL [20:47:14] (03CR) 10Zppix: [C: 031] "Tested via test instance" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/318976 (https://phabricator.wikimedia.org/T149609) (owner: 10Paladox) [20:49:05] Zppix: which testinstance? [20:49:56] Sagan tools, we have a channel #wikimedia-bot-gerrit [20:49:59] #wikimedia-bot-gerrit [20:50:01] a test channel [20:50:45] ...which is not registered [20:51:05] working on it [20:52:38] !log services fixed puppet on 'appservice' instance by 'apt-get install --only-upgrade' on the exim packages [20:52:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Services/SAL [20:53:00] Sagan there its registered [20:53:19] I saw it [20:55:18] bd808: you have rights to the project in labs ? [20:55:32] he does [20:55:38] he create one for me some time ago [20:56:08] Sagan: context is key :) i am talking about a specific project :) [20:56:21] oh, sry [20:56:32] matanya: but heh, IIRC he can assign it himself if he needs that :P [20:57:15] :D [20:57:40] matanya: question lacks specific context. ;) [20:58:31] matanya: I don't see any new tools in my groups list either... did the add take? [20:59:59] I am talking about the stewardsbot project [21:00:09] bd808: i granted you commit right to it in gerrit [21:00:28] but thought you might want access to the labs project as well [21:00:55] ah, ultimate power. Yeah I could technically add myself but it would be nicer if you took care of that [21:01:32] I don't like to self-join unless circumstances demand it [21:01:55] but I guess I don't have to explain that idea to a steward ;) [21:08:48] 06Labs, 10Continuous-Integration-Infrastructure: Request increased quota for contintcloud labs project - https://phabricator.wikimedia.org/T142877#2769904 (10hashar) [21:08:51] 06Labs, 10Labs-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2769905 (10hashar) [21:14:20] bd808: yeah, good point :D [21:17:59] bd808: no user on wikitech ? [21:18:12] matanya: oh "BryanDavis" [21:18:28] added [22:05:15] (03Draft1) 10Paladox: Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 [22:05:20] (03Draft2) 10Paladox: Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 [22:06:03] (03CR) 10Paladox: "Will deploy it to prod, if it has any affect on anything else I wont merge otherwise I will merge in a day or two." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [22:06:05] (03CR) 10Paladox: [C: 031] Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [22:06:37] !log tools.lolrrit-wm deploying https://gerrit.wikimedia.org/r/#/c/319705/ to fix bot crashing [22:06:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL [22:06:47] (03CR) 10jenkins-bot: [V: 04-1] Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [22:11:59] (03CR) 10Reedy: "No. Self. Merging." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [22:12:40] (03PS3) 10Paladox: Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 [22:13:02] (03CR) 10Paladox: "Ok, will wait for someone who is probably a wmf employee or someone else who has +2 rights." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [22:19:03] (03CR) 1020after4: "try / catch around the whole method might prevent it from dying but it would be better to test for unset values so that we avoid the speci" (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [22:22:59] (03CR) 10Paladox: Fix grrrit-wm crashing on specific git push commands (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [22:25:28] Why is grrit gone [22:35:54] (03PS4) 10Paladox: Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 [22:36:24] (03CR) 10jenkins-bot: [V: 04-1] Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [22:37:12] (03PS5) 10Paladox: Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 [22:37:59] (03PS6) 10Paladox: Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 [22:38:33] (03CR) 10jenkins-bot: [V: 04-1] Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [22:39:59] (03CR) 10Paladox: "recheck" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [22:40:42] (03PS7) 10Paladox: Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 [22:47:36] !log tools.stashbot Restarted to pick up config change for https://github.com/bd808/tools-stashbot/issues/27 [22:47:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [23:28:14] (03CR) 10Paladox: [C: 031] "@20after4 you fixed it thanks, tested and works. And gives the correct message too." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [23:28:54] (03CR) 1020after4: [C: 032] Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [23:29:29] (03Merged) 10jenkins-bot: Fix grrrit-wm crashing on specific git push commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319705 (owner: 10Paladox) [23:31:30] !log tools.lolrrit-wm deploying https://gerrit.wikimedia.org/r/#/c/319705/ again (a more proper fix) [23:31:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL [23:31:59] !log tools.lolrrit-wm forgot to say twentyafterfour helped to fix. [23:32:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL [23:57:37] PROBLEM - Puppet run on tools-docker-registry-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]