[00:21:57] 10Tool-Labs-tools-Other, 06Operations: Jouncebot: Add functionality to change Nick from Jouncebot_ to Jouncebot automatically - https://phabricator.wikimedia.org/T150916#2801201 (10Zppix) [00:22:21] 10Tool-Labs-tools-Other, 06Release-Engineering-Team: Jouncebot: Add functionality to change Nick from Jouncebot_ to Jouncebot automatically - https://phabricator.wikimedia.org/T150916#2801214 (10Zppix) [00:35:30] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.68 ms [00:39:26] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [00:42:20] Zppix: lets talk here to keep out of the way of the SWATers [00:42:38] so you are doing the equivalent of `git clone ssh://bd808@gerrit.wikimedia.org:29418/wikimedia/bots/jouncebot` and getting some error? [00:43:49] Zppix: lets debug the ssh parts. What does `ssh $USER@gerrit.wikimedia.org -p 29418` say to you? [00:45:22] one sec [00:45:34] ssh:// or ssh? [00:49:16] for git clone, ssh:// [00:49:25] for the second command, just ssh [00:53:01] bd808 i get the usage message bd808 [00:53:04] :P [00:54:11] ok. so you ssh key is working [00:54:34] do you get an error message when the git clone fails? [00:57:01] Cloning into 'jouncebot'... [00:57:01] fatal: Could not read from [00:57:02] Please make sure you have t [00:57:02] and the repository exists. [00:57:05] @ bd808 [00:57:54] wow that was weird let me just screenshot it [00:58:25] bd808 https://gyazo.com/999c95a5f8e171365165a813d6420790 [00:59:02] show the command you typed as well? [00:59:23] git clone ssh://zppix1@gerrit.wikimedia.org:29418/wikimedia/bots/jouncebot [01:01:23] what about the second command? [01:02:32] weird. if I replace your user name with mine that works perfectly from here [01:02:33] try this: GIT_SSH_COMMAND="ssh -v" git clone ssh://zppix1@gerrit.wikimedia.org:29418/wikimedia/bots/jouncebot [01:02:33] i know for fact gerrit and wikitech have the key i use (i havent change the key i use for gerrit since i first started dev) [01:02:33] works now [01:02:34] weird [01:02:34] gremilins! [01:02:34] maybe gerrit maintence was going on and it was on read-only? [01:02:40] Krenair we got it its fine now [01:02:46] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 2.45 ms [01:02:48] no [01:02:53] read only wouldn't prevent cloning [01:03:12] hmm idk [01:04:20] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [01:13:07] bd808 quick question what code is used to make jouncebot to send private message (im using it in the process of automate nick changing [01:14:22] Zppix: literally you just need to cut-n-paste the parts that do this from stashbot ;) [01:15:19] https://github.com/bd808/tools-stashbot/blob/master/stashbot/bot.py#L100 [01:15:20] https://github.com/bd808/tools-stashbot/blob/master/stashbot/bot.py#L215-L218 [01:15:49] teh config vars may be slightly different in jouncebot, but that's the logic needed [01:16:38] the "self.reactor.scheduler" is a new version of the irc lib... let me find the older syntax [01:17:24] ok [01:17:56] "conn.execute_delayed(...)" is the older syntax [01:18:57] is the stuff in the parens the same? [01:19:50] yeah, delay in seconds & function to call [01:22:12] anything else need replaced? [01:27:21] bd808 [01:27:49] Zppix: ? [01:28:04] you want me to upload the patch so you can put your name on it? ;) [01:28:38] no [01:28:58] i am just making sure if i new what lib it used i would find out myself [01:29:04] knew* [02:04:40] there bd808 only took 20 mins to figure out patch files because git review is being a dick [02:10:45] (03PS1) 10Filippo Giunchedi: add dummy material for restbase201[012] [labs/private] - 10https://gerrit.wikimedia.org/r/322038 [02:11:49] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] add dummy material for restbase201[012] [labs/private] - 10https://gerrit.wikimedia.org/r/322038 (owner: 10Filippo Giunchedi) [02:18:16] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 1.57 ms [02:28:13] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [04:30:43] PROBLEM - Puppet staleness on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0] [04:35:38] PROBLEM - Puppet staleness on tools-elastic-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [04:37:54] PROBLEM - Puppet staleness on tools-puppetmaster-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [04:38:46] PROBLEM - Puppet staleness on tools-proxy-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0] [04:41:15] PROBLEM - Puppet staleness on tools-mail-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0] [04:44:23] PROBLEM - Puppet staleness on tools-exec-gift is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [04:45:15] PROBLEM - Puppet staleness on tools-k8s-master-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0] [04:49:02] PROBLEM - Puppet staleness on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [04:49:04] PROBLEM - Puppet staleness on tools-elastic-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [04:52:50] PROBLEM - Puppet staleness on tools-elastic-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0] [04:54:13] PROBLEM - Puppet staleness on tools-docker-registry-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [04:54:43] PROBLEM - Puppet staleness on tools-logs-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [04:54:43] PROBLEM - Puppet staleness on tools-services-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [04:58:23] PROBLEM - Puppet staleness on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0] [04:59:33] PROBLEM - Puppet staleness on tools-proxy-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [05:02:26] PROBLEM - Puppet staleness on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [05:53:37] 10Tool-Labs-tools-Xtools, 07I18n: xTools not translating some strings - https://phabricator.wikimedia.org/T150931#2801648 (10Matthewrbowker) [06:37:42] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:14:05] chasemp: I guess ^ all didn't get re-enabled? [07:14:06] * yuvipanda lets it be [07:17:42] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [07:41:41] Change on 12www.mediawiki.org a page Wikimedia Labs/Things to fix in beta was modified, changed by Jkmartindale link https://www.mediawiki.org/w/index.php?diff=2287193 edit summary: Use translated templates. [08:50:24] 06Labs, 10Labs-Infrastructure, 07LDAP: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#2801794 (10MoritzMuehlenhoff) >>! In T63967#2799842, @Andrew wrote: > For my part: I can't think of any reason why it would hurt to have the nslcd regex be more permissive than the user-creation regex... [09:01:33] RECOVERY - Host tools-puppetmaster-01 is UP: PING OK - Packet loss = 0%, RTA = 0.89 ms [09:34:00] 06Labs, 10Labs-Infrastructure, 07LDAP: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#2801858 (1080686) I'd love to use it again, currently it doesn't work. I had SVN commit access but since the move I cannot log into Git. I have complained a few times about it but as this ticket is sti... [09:39:10] PROBLEM - Host tools-puppetmaster-01 is DOWN: CRITICAL - Host Unreachable (10.68.22.61) [10:14:53] 06Labs, 10Tool-Labs, 10Datasets-General-or-Unknown, 10Wikidata, 07Privacy: Information leak on wikidata-externalid-url - https://phabricator.wikimedia.org/T150803#2797140 (10Esc3300) T150939 [11:24:02] 06Labs, 10Tool-Labs: Tool creation fails? - https://phabricator.wikimedia.org/T150946#2802158 (10Magnus) [12:19:08] bd808: this is what I actually needed yesterday (but I doubt it reflects the truth) -- https://quarry.wmflabs.org/query/14103 [12:27:05] 06Labs, 10Labs-Infrastructure, 10DBA: LabsDB replica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#2802317 (10jcrespo) [12:42:37] 06Labs, 10Tool-Labs: Tool creation fails? - https://phabricator.wikimedia.org/T150946#2802332 (10Magnus) //Update:// Tried to create the same service group again, just in case, but that fails with "Failed to create service group." So some part of the system knows it exists... [13:02:30] Change on 12www.mediawiki.org a page Wikimedia Labs/Tool Labs/Migration of Toolserver tools was modified, changed by Jkmartindale link https://www.mediawiki.org/w/index.php?diff=2287503 edit summary: Use translated templates. [13:06:13] 06Labs, 10Tool-Labs: Tool creation fails? - https://phabricator.wikimedia.org/T150946#2802361 (10Magnus) Update 2: It is visible in [[ https://toolsadmin.wikimedia.org/tools/id/quickstatements | toolsadmin ]]. [13:27:51] RECOVERY - Puppet staleness on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [3600.0] [13:28:45] RECOVERY - Puppet staleness on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [3600.0] [13:29:02] RECOVERY - Puppet staleness on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [13:29:42] RECOVERY - Puppet staleness on tools-logs-02 is OK: OK: Less than 1.00% above the threshold [3600.0] [13:30:16] RECOVERY - Puppet staleness on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [13:32:28] RECOVERY - Puppet staleness on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [3600.0] [13:34:41] 06Labs, 10Tool-Labs: Tool creation fails? - https://phabricator.wikimedia.org/T150946#2802386 (10chasemp) Thanks @Magnus. I'm not sure if it's all of it but maintain-kubeusers was down (and seems to have died once again) on the k8s master and it has a role in tool creation even if not in k8s land. @yuvipanda... [13:34:50] 06Labs, 10Tool-Labs: Tool creation fails? - https://phabricator.wikimedia.org/T150946#2802388 (10chasemp) p:05Triage>03High [13:35:44] RECOVERY - Puppet staleness on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:00:36] RECOVERY - Puppet staleness on tools-elastic-03 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:04:02] RECOVERY - Puppet staleness on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:16:06] 10Tool-Labs-tools-Other, 06Release-Engineering-Team, 13Patch-For-Review: Jouncebot: Add functionality to change Nick from Jouncebot_ to Jouncebot automatically - https://phabricator.wikimedia.org/T150916#2802503 (10Zppix) Please see https://gerrit.wikimedia.org/r/#/c/322037/ [16:03:21] 06Labs, 10Tool-Labs: Tool creation fails? - https://phabricator.wikimedia.org/T150946#2802737 (10Magnus) I see the directory and .kube dir have been created now, but no DB replica data yet, no public_html. Exciting! ;-) [16:04:49] 06Labs, 10Labs-Infrastructure, 10DBA: Initial data tests for db1095 - https://phabricator.wikimedia.org/T150960#2802738 (10Marostegui) [16:11:44] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: labstore1003 - RAID fail - https://phabricator.wikimedia.org/T149156#2802757 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson [16:12:23] 06Labs, 10Labs-Infrastructure, 07LDAP: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#2802759 (10Andrew) @80686, If git doesn't work for you either then we have more serious problems :( Is there another ticket associated with this? [16:50:21] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:24:41] Hi, what a version of crontab is running on tools? [17:25:22] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:45:44] yuvipanda: do you know, what a version of crontab you are running on labs? [18:05:02] doctaxon: version? The package is 3.0pl1-124ubuntu2 from Ubuntu Trusty [18:08:41] jynus: tested COUNT(*) instead of COUNT (*) and now it works [18:09:05] :-) [18:09:14] there is an sql mode [18:09:29] that I think allows that, but I do not recommend you that [18:09:43] well I did SELECT COUNT(*) not just COUNT(*) [18:09:55] but I think it'll not let me COUNT(*) [18:11:35] mafk, don't do this: https://phabricator.wikimedia.org/P4465 [18:12:13] won't do [18:12:21] if you mean the ignore spaces part [18:12:29] unless you want the ancient god of mysql to curse you [18:12:51] for doing things non-mysqly [18:12:54] :-) [18:13:20] I'm testing some queries I'm making, learning things [18:13:38] you should come to [18:13:57] https://phabricator.wikimedia.org/T149624 [18:14:15] it may get cancelled if I get not interest by community members [18:29:12] !log tools.wikibugs killed stray wikibugs on tools-exec-1404 [18:29:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [18:57:06] Bd808 i will work on your fixes when i get home in few hours [19:16:42] 06Labs, 06Community-Tech, 10DBA, 10MediaWiki-extensions-PageAssessments, 13Patch-For-Review: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2798143 (10chasemp) I'm not sure how to know this is OK to expose. Is there anywhere that someo... [19:24:31] 10Tool-Labs-tools-Database-Queries, 06Project-Admins: Archive Tool-Labs-tools-Database-Queries project - https://phabricator.wikimedia.org/T107699#2803461 (10Aklapper) Any comments / input to my last comment, please? [19:31:29] 06Labs, 06Community-Tech, 10DBA, 10MediaWiki-extensions-PageAssessments, 13Patch-For-Review: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2803491 (10kaldari) @dpatrick: Any chance you could OK replicating this data to Tool Labs? The d... [19:33:52] 06Labs, 06Community-Tech, 10DBA, 10MediaWiki-extensions-PageAssessments, 13Patch-For-Review: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2803498 (10chasemp) once @dpatrick gives this a once over you can assign to me and I'll knock it... [19:57:29] PROBLEM - Puppet run on tools-exec-1204 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:07:12] 06Labs, 10Tool-Labs: Hashtag tool 500 internal server error - https://phabricator.wikimedia.org/T150984#2803640 (10Ciell) [20:11:10] 06Labs, 10Tool-Labs: Hashtag tool 500 internal server error - https://phabricator.wikimedia.org/T150984#2803664 (10chasemp) p:05Triage>03Normal @mahmoud @slaporte Both listed as maintainers I believe :) I restarted it to seemingly no effect [20:22:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:29:46] RECOVERY - Puppet staleness on tools-services-02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:37:31] RECOVERY - Puppet run on tools-exec-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [21:06:57] 06Labs, 10Tool-Labs: Hashtag tool 500 internal server error - https://phabricator.wikimedia.org/T150984#2803873 (10Slaporte) Looks up to me now -- I think @chasemp's restart worked? [21:07:12] 06Labs, 10Tool-Labs: Hashtag tool 500 internal server error - https://phabricator.wikimedia.org/T150984#2803874 (10mahmoud) Hey all, just got back to my laptop, the service appears to be running, no 500s for me. Maybe it was something transient related to maintenance, or @chasemp's restarts fixed it with some... [21:07:48] 06Labs, 10Tool-Labs: Hashtag tool 500 internal server error - https://phabricator.wikimedia.org/T150984#2803875 (10chasemp) huh, it possibly just took a long time to restart and I'm impatient :) [21:26:22] bd808 i am now going to fix your issues mentioned on that change for jouncebot :) [21:34:19] bd808: i just edited it is it better now? [21:40:24] 10Tool-Labs-tools-Other, 06Release-Engineering-Team, 13Patch-For-Review: Jouncebot: Add functionality to change Nick from Jouncebot_ to Jouncebot automatically - https://phabricator.wikimedia.org/T150916#2803986 (10Zppix) p:05Triage>03Normal [21:49:16] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Epantaleo was created, changed by Epantaleo link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Epantaleo edit summary: Created page with "{{Tools Access Request |Justification=I would like to visualize the etymological tree of words. I have a demo of the software here: http://www.epantaleo.com/wp-content/uplo..." [22:05:36] (03PS10) 10Zppix: Replacing swig with swig-templates [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320294 (owner: 10Paladox) [23:05:32] yuvipanda: been thinking about using tools-redis for nagf (as proof of concept before using in other tools). So far I've logically blocked it on being able to run things locally etc. but maybe there's another way. [23:05:43] yuvipanda: Maybe we can expose some of those services via an environment variable? [23:06:08] that way, on my local host it would just not be set and I can have the configuration instantiate a different cache object instead of redis (e.g. filesystem cache) [23:06:23] as a way of discovery [23:06:55] Krinkle: hmm [23:06:58] Krinkle: that doesn't sound like a bad idea [23:07:03] Krinkle: can you file a bug? [23:07:12] Krinkle: it might be a k8s only thing tho [23:07:18] yuvipanda: Even if we use dns for discovery of the exact service, I'd still prefer not to hard code that hostname. [23:07:22] Yeah, totally. [23:07:32] and its existance [23:07:43] Krinkle: yeah, I agree [23:07:49] although it makes sense that in the future we'd just make nagf responsible for having that service by using a Docker file that creates it [23:07:53] Krinkle: tools-redis is already a 'service' name [23:07:54] but since we backed away from that.. [23:07:55] okay [23:08:00] the actual host has changed names many times [23:08:11] Yeah, sure. [23:08:25] yuvipanda: labs-infra? [23:08:34] Krinkle: 'tool-labs' [23:11:05] k [23:11:11] 06Labs, 10Tool-Labs: Expose tool-labs service names via environment variables - https://phabricator.wikimedia.org/T151002#2804324 (10Krinkle)