[03:13:11] who runs the xtools-ec tool? [03:27:05] comets: Cyberpower678 · MusikAnimal · Technical 13 all seem to be active, although i am unsure on T13 [03:27:41] i think T13 got a very very long ban.. [03:27:54] comets: On enwiki yeah, indef arbcom ban [03:28:24] yo [03:28:26] jsut wanted to point out that their tool is somewhat broken, the links in the "Latest edit (global) " sections has an extra "https" in the middle .. [03:28:37] Oh yeah [03:28:46] elee is also a manitainer :P [03:28:47] I wonder if I have access [03:28:49] example : > https://https//en.wikipedia.org/wiki/Jeremy_Bobb [03:28:53] (to tool labs's) [03:28:57] hold on lets see [03:29:17] this is the version on tools.wmflabs, not xtools, right? [03:29:58] comets: ^ [03:30:00] https://tools.wmflabs.org/xtools-ec/ [03:30:09] alright hrm [03:31:12] okay uh Cyberpower will have to be trouted [03:31:22] join us in #xtools for more drama [03:31:55] the links itself are broken too but that isn't the main problem :P [03:32:06] yeah xtools on tool labs is effectively sad [03:32:13] this has apparently been the case for a long time now or something? unsure [03:32:27] where is the cooler more hip version? :( [03:32:39] I came on the project just recently (in the order of a few days) to help them set up xtools.wmflabs.org (that is, the servers) [03:33:02] * TheMesquito trouts CP678|around [03:33:20] kek [03:33:24] let me get in on that action [03:33:30] * elee hands a trout to TheMesquito [03:33:36] * elee then points at CP678|around [03:34:00] Huh [03:34:11] * elee gestures for TheMesquito to trout CP678|around [03:53:32] 6Labs: High load on idle machines - https://phabricator.wikimedia.org/T104416#1416423 (10Physikerwelt) 3NEW [04:14:36] hi is there a way to copy files from one labs instance to another using scp... if I try ssh I always get permission denied [04:23:40] 6Labs: Lost a file in my home directory "/home/hcohl/purgemathcache.sql" - https://phabricator.wikimedia.org/T104348#1416457 (10Physikerwelt) @yuvipanda I could not even manage to scp a file from one machine within the project to another. if I try ssh I always get permission denied [04:30:01] 6Labs, 3Labs-Sprint-104: Recover files from old corrupted file system (Tracking) - https://phabricator.wikimedia.org/T104334#1416459 (10dschwen) What is the "old corrupted filesystem"? I have instances I cannot access anymore (like maps-wma1). Were those using the "old corrupted filesystem"? If so I need every... [06:03:28] those ntp patches for leap second 2015 actually worked on my machine :D [06:58:13] PROBLEM - Puppet failure on tools-submit is CRITICAL 44.44% of data above the critical threshold [0.0] [07:23:16] RECOVERY - Puppet failure on tools-submit is OK Less than 1.00% above the threshold [0.0] [07:42:22] YuviPanda: morning [07:42:37] I am missing some files on tools [08:07:20] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Recover some sql queries - https://phabricator.wikimedia.org/T104134#1416632 (10Multichill) Thanks Yuvi, but you took all of them, not just the ones that were changed and all the recovered files now have a timestamp of "Jun 29 21:21" so no way of kn... [08:33:34] multichill: even diffen met je queries-map? [08:41:59] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Recover some sql queries - https://phabricator.wikimedia.org/T104134#1416697 (10valhallasw) ``` for i in `find -name "*.` ../recovered/$i; diff -u `find -name $i` ../recovered/$i || echo "NEW FILE: $i";); done > ~/difflijst ``` New files are: ``` n... [08:42:00] multichill: ^ [08:42:22] not exactly a trivial diff, I noticed, but with some horrible bash.... [08:49:53] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103, 3Labs-Sprint-104: Labs: increase size of the volume for the maps project and restore - https://phabricator.wikimedia.org/T103358#1416712 (10Kghbln) Is a time estimation possible until this will be done and the tiles return to projects like Wikivoy... [08:52:23] matanya: if it's because of the NFS incident, please file a subtask on https://phabricator.wikimedia.org/T103265 [08:52:33] sorry, https://phabricator.wikimedia.org/T104334 [08:53:12] valhallasw`cloud: thanks, copied from my own backup already [08:53:18] matanya: ok :-)] [09:08:43] 6Labs, 10Continuous-Integration-Infrastructure: "puppet-compiler02" Jenkins slave is no more connected - https://phabricator.wikimedia.org/T104428#1416715 (10hashar) 3NEW a:3hashar [09:08:49] 6Labs, 10Continuous-Integration-Infrastructure: "puppet-compiler02" Jenkins slave is no more connected - https://phabricator.wikimedia.org/T104428#1416723 (10hashar) p:5Triage>3Unbreak! [09:10:48] 6Labs, 10Continuous-Integration-Infrastructure: "puppet-compiler02" Jenkins slave is no more connected - https://phabricator.wikimedia.org/T104428#1416715 (10hashar) Console has a bunch of entries related to mounting NFS labstore. Seems the instance hasn't been recovered following the recent labs NFS crash. R... [09:14:24] 6Labs, 10Continuous-Integration-Infrastructure: "puppet-compiler02" Jenkins slave is no more connected - https://phabricator.wikimedia.org/T104428#1416727 (10hashar) labstore.svc.eqiad.wmnet: Temporary failure in name resolution Seems /etc/resolv.conf needs to be adjusted to the new DNS server and DNS sea... [09:16:42] 6Labs, 10Continuous-Integration-Infrastructure: "puppet-compiler02" Jenkins slave is no more connected - https://phabricator.wikimedia.org/T104428#1416728 (10hashar) a:5hashar>3None I can't ssh to the instance so need some labs root to connect to it and fix the instance :-/ [09:16:53] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: "puppet-compiler02" Jenkins slave is no more connected - https://phabricator.wikimedia.org/T104428#1416730 (10hashar) [09:18:56] YuviPanda: whenever you come around. puppet-compiler02.puppet3-diffs.eqiad.wmflabs doesn't respond to ssh despite a reboot. Can use some manual fixing of it ( ref: https://phabricator.wikimedia.org/T104428 ) [10:20:59] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: "puppet-compiler02" Jenkins slave is no more connected - https://phabricator.wikimedia.org/T104428#1416869 (10yuvipanda) See also T103760 [10:27:25] hi hashar . [10:27:45] I saw _joe_'s response on -operations, and he's right :) I've been pestering him about it for a week now, he needs to respond... [10:47:03] ok, something else is going on. stand by [10:50:02] 6Labs: Remove NFS mounts from project cephtest - https://phabricator.wikimedia.org/T102381#1416924 (10yuvipanda) It has no instances now! I suppose this means I can disable NFS? @andrew? [10:50:15] valhallasw`cloud: Thanks, I'll have a look at it later [10:50:18] (working) [10:51:47] 6Labs: Investigate why novaadmin was no longer projectadmin of the puppet3-diffs project - https://phabricator.wikimedia.org/T104440#1416925 (10yuvipanda) 3NEW [11:08:34] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103, 3Labs-Sprint-104: Labs: increase size of the volume for the maps project and restore - https://phabricator.wikimedia.org/T103358#1416951 (10scfc) [12:15:37] 6Labs: Lost a file in my home directory "/home/hcohl/purgemathcache.sql" - https://phabricator.wikimedia.org/T104348#1417008 (10Hcohl) @yuvipanda: Should I submit a task in https://phabricator.wikimedia.org/T104334 even though I have already opened this task? Has there been any luck in finding the file? [12:16:36] 6Labs: Lost a file in my home directory "/home/hcohl/purgemathcache.sql" - https://phabricator.wikimedia.org/T104348#1417009 (10yuvipanda) I just finished copying your entire older home folders into /data/project/home on math project instances - all your files should be there. Can you verify? [12:22:04] 6Labs: Lost a file in my home directory "/home/hcohl/purgemathcache.sql" - https://phabricator.wikimedia.org/T104348#1417013 (10Hcohl) The file was there. Thanks so much. [12:23:26] 6Labs: Lost a file in my home directory "/home/hcohl/purgemathcache.sql" - https://phabricator.wikimedia.org/T104348#1417019 (10yuvipanda) 5Open>3Resolved yw! [12:24:05] 6Labs: Lost a file in my home directory "/home/hcohl/purgemathcache.sql" - https://phabricator.wikimedia.org/T104348#1417022 (10yuvipanda) [12:24:07] 6Labs, 3Labs-Sprint-104: Recover files from old corrupted file system (Tracking) - https://phabricator.wikimedia.org/T104334#1417021 (10yuvipanda) [13:08:59] 6Labs: Login into fastcci-master instance not possible - https://phabricator.wikimedia.org/T104418#1417088 (10Aklapper) [13:15:13] 6Labs, 6Discovery, 10Maps: WikiMiniAtlas (wma.wmflabs.org) is still down - https://phabricator.wikimedia.org/T104417#1417100 (10Aklapper) [14:47:09] YuviPanda|food: is there a phabricator ticket about that new instances don't get mpount points of data/project [15:58:19] hello, I'm trying to figure out why our xtools-articleinfo tool decided to die a week or two ago... I think it was the move to trusty? perhaps the other tools were not as affected by the move [15:58:40] as I understand it this very much outdated code requires php 5.3, and I see we have php 5.5 [15:59:36] the error log complains about an "array to string conversion" which sounds like it could be some deprecated code or the like [15:59:54] is there any way to get xtools-articleinfo back on precise? [15:59:56] MusikAnimal: you can switch it back to trusty [15:59:57] err [15:59:58] precise [16:00:03] do tell [16:00:05] webservice stop [16:00:08] webservice --release precise start [16:01:21] hmm, still no dice. How could I confirm it's running on precise and not trusty? [16:01:39] MusikAnimal: qstat -xml [16:01:43] if the instance it is running on [16:01:51] is tools-webgrid-lighttpd-12xxx [16:01:54] then it's precise [16:01:55] else trusty [16:04:54] ah, so I guess that's not it, then [16:05:16] hmm it's weird because we've seen this issue before, where it reports "no revisions found" but somehow later it will work again [16:05:50] anyway thank you for helping YuviPanda! [16:06:12] :) [16:06:49] we're working on a rewrite, or to port over the one the German Wikipedia uses, which is lightning fast [16:27:33] 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1417872 (10Legoktm) 1 [16:29:45] Krenair: you can run those commands locally and on tools-login [16:30:28] Krenair: but the problem was that the bot's config was hardcoding a specific freenode server [16:31:43] legoktm, https://phabricator.wikimedia.org/P858 [16:32:47] Krenair: fab only works from your laptop. you can run the raw "qmod -rj wb2-*" command on tools-login [16:45:27] legoktm: you had extdist on toollabs at some point? [16:45:44] YuviPanda: only as a prototype yeah [16:55:07] YuviPanda: what does this mean man " I respectfully disagree - setting it up in labs / prod is no different if puppet is being used" [16:55:14] are you saying you want labs only things in ops/puppet? [16:55:30] chasemp: means that if they're going to use puppet to deploy it then it doesn't matter if it's in labs or prod... [16:55:38] chasemp: there are plenty of labs only things in ops/puppet [16:55:43] manifets/role/labs* [16:55:52] there's practically no other place for them atm :) [16:55:58] labs projects like run by individual people? [16:56:02] yes [16:56:22] modules/wdq-mm, modules/quarry, modules/ores, modules/ircyall, modules/extdist [16:57:11] chasemp: the alternative is to have a 'special' VM you setup by hand, and that's terrible. [16:57:14] i see you are equating any puppet with ops puppet I guess? where I was saying, the least they can do is make this a repeatable puppet thing [16:57:27] not really, a puppet repo for their stuff which they use puppet apply for? [16:57:40] we don't really have anything of that sort atm [16:57:59] are toollabs folks responsible for their own stuff? [16:58:21] the toollabs folks as in the admins? [16:58:25] or the people who run tools on toollabs? [16:58:29] second [16:58:44] we provide a very restricted set of things that toollabs users can do [16:58:57] well, not 'very' but they don't have root, yeah. [16:59:03] and afaik this is on Labs itself and not Tool Labs... [16:59:22] how would you determine which it should be on? [17:00:08] ah, so the ideal would be - if it can fit on toollabs and you are ok with your applicationrunning along with the hundreds of others in a shared environment that goes down whenever NFS is down and has its own idosyncracies, then toollabs is ok [17:00:14] (a *lot* of tools fit that bill) [17:00:29] hat seems the same for labs or toollabs tho [17:00:32] if you want to do puppet *anything*, you want labs. [17:00:44] you mean if you want to use ops/puppet anything? [17:00:51] I guess, why would toollabs preclude puppet use [17:00:54] no, any puppet at all [17:01:09] because puppet needs root and you don't get root on toollabs? [17:01:09] toollabs is prevented from puppetizing their setups? [17:01:13] yes [17:01:44] the tldr on this is there is some dashboard wmde has made [17:01:48] some wmf teams want to use it [17:01:52] no one is available to support it [17:01:56] releng has disavowed [17:02:03] ops has no real involvment [17:02:13] and it's a awhile away from prod ready [17:02:14] right, so it's a hot potato :P [17:02:18] so...labs [17:02:27] for them to get it together [17:02:29] yes essentially [17:02:32] sure labs, but not 'labs people' [17:02:47] you took that way wrong, they just asked who would know how to run things in labs [17:02:56] and I said, idk typical labs folks? [17:03:31] ah, ok :) [17:03:52] this was not you run it or ops labs folks [17:03:52] chasemp: I guess if they know they aren't going to get any more support than what any normal labs user gets, that's ok... [17:04:04] like, I don't think any of us can review that puppet patch [17:04:11] they need a space to live and semi-sane setup [17:04:42] so if they want to use puppet at all, they need someone from ops to merge it and if not they are kind of screwed. [17:04:53] no they don't, they can use puppet in labs right on their own? [17:04:57] no [17:05:06] regular labs folks don't get root on their vm's? [17:05:10] they can use a self hosted puppetmaster, but that doesn't move them out of the 'screwed' category [17:05:15] yes but it runs operations/puppet.git [17:05:20] you dnon't need a master to use puppet [17:05:27] and I wouldn't recommened it for them [17:05:39] I'm talking like super slimmed down it's just for repeatability nothing to do with our puppet setup at all [17:05:56] ah, so that's something I don't think any other project does atm. [17:05:57] puppet != master oriented or ops/puppet [17:06:02] indeed, I agree. [17:06:20] I did try it before I had root and gave up at some point. [17:06:46] anyway, thanks for the clarifications chasemp :) we're all on the same page/sentence/word [17:06:54] the idea here is just some handholding to keep it within reason [17:08:01] I don't think we can commit any of the labs folks to helping with any handholding outside of setting them up with a project (or tools account) is what I'm saying, I guess... [17:20:48] 6Labs, 6Discovery, 3Discovery-Maps-Sprint: Upgrade postgres on labsdb1004 / 1005 to 9.4, and PostGis 2.1 - https://phabricator.wikimedia.org/T101233#1418068 (10MaxSem) [17:21:00] 6Labs, 6Discovery: Upgrade postgres on labsdb1004 / 1005 to 9.4, and PostGis 2.1 - https://phabricator.wikimedia.org/T101233#1333262 (10MaxSem) [17:23:36] 6Labs, 6Discovery, 10Maps: Enable OSM Postgres machine access in labs - https://phabricator.wikimedia.org/T98382#1418091 (10MaxSem) [17:38:08] (03PS1) 10Sitic: Add option to show traditional oneline watchlist [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/222146 [17:38:27] (03CR) 10Sitic: [C: 032 V: 032] Add option to show traditional oneline watchlist [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/222146 (owner: 10Sitic) [17:43:34] Coren|Away, YuviPanda: Do you happen to know the IP addresses that MediaWiki sees for requests from Labs? I know it's in 10.0.0.0/8, but is there something narrower that's only Labs (or only Tool Labs) and not e.g. Parsoid? [17:44:05] anomie: toollabs bots run from exec nodes have their own external IPs, I think. I can get you a list, moment. [17:52:20] anomie: so it's in http://wikitech.wikimedia.org/w/api.php?action=query&list=novainstances&niproject=tools&niregion=eqiad&format=json except that fails to load atm >_> [17:53:51] YuviPanda: I'm struggling with MySQL error message. Does this tell you anything? [17:54:05] File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 249, in __exit__ [17:54:05] self.rollback() [17:59:32] marmick: please paste the entire message to a pastebin [17:59:36] hi [17:59:40] i can't login wikitech [17:59:44] it loads forever [17:59:56] andrewbogott: ^ [18:00:27] Steinsplitter: wikitech or horizon? Wikitech is working fine for me… [18:00:38] oh, wait, login — let me try [18:01:19] valhallasw`cloud: http://pastebin.com/4Fre1K7t it's a mysql gone away. i can stop it sometimes but not always... [18:01:41] i tried a) try and except, which worked in some occasions. [18:01:49] b) mysql_cur.execute("SET SESSION wait_timeout=28800") [18:01:53] increasing the timeout. [18:01:54] Steinsplitter: try now? [18:02:18] c) mysql_con.ping(True) pinging in between.... [18:02:28] works thanks [18:02:48] anomie: fwiw, all toollabs hosts seem to be 10.68.[16,17,18].* [18:03:01] it appears unpredictably for simple queries like "truncate" an empty table, but also big ones. it does appear only in db from big wikis. [18:03:29] anomie: but the IP pool is shared between the different labs projects iirc, so you wouldn't be able to easily distinguish toollabs from another project [18:03:30] valhallasw`cloud: do you know if they report internal or external IP? [18:03:33] for the exec nodes? [18:03:43] YuviPanda: to WMF servers, the internal IP [18:03:46] ok! [18:03:48] so to wikipedia et al [18:04:28] marmick: ping, and reconnect if that fails? [18:05:43] valhallasw`cloud: Thanks! [18:05:55] valhallasw`cloud: i tried it. look, it points me a line which is not called. http://pastebin.com/bxmizrQf [18:06:38] does this line has anything to do? [18:06:54] marmick: not called in what sense? the queryreconnect line triggers an Exception, and the with: block you're in then tries to roll back, which fails because hte connection is closed [18:07:25] i mean that it is not in the code flow [18:07:32] it cannot access that line [18:07:43] it is in an if whose conditions are not met [18:07:54] so it makes no sense it shows this line with the error [18:08:37] two options: the code changed after the script was started, or the conditions /are/ met [18:08:56] what do u mean the code changed after the script started? O_O [18:09:26] marmick: if you changed cira_filter_db_conserv.py after starting the script, the lines shown can be incorrect [18:09:48] but it cannot jump to that line [18:10:00] and before it, inside the if-block there is a print [18:10:03] which i would see [18:10:20] besides, it's an if at the end of the with mysql_con: block [18:10:26] * anomie guesses that Labs as a whole is 10.68.0.0/16, based on that. [18:10:27] right after other calls to functions [18:12:51] marmick: as I said, there's only one reason I know that causes incorrect lines to be shown: changing the source file after starting it. [18:14:11] let's see. i'm executing again. [18:14:54] ERROR 1044 (42000) at line 2: Access denied for user 's52299'@'%' to database 's52299_p' [18:18:00] Krenair: you might need two underscores, not sure. [18:18:48] yeah, works with two [18:19:39] Krenair: I think the idea is the name would be u1234_dbname_p [18:23:17] Krenair, yes username__anything [18:34:53] YuviPanda: will replica.my.cnf created immediataly or it is needed to wait some hours? [18:37:57] 6Labs, 7Database: Provision a labsdb useraccount that can be used to run replica-addusers.pl - https://phabricator.wikimedia.org/T104476#1418287 (10yuvipanda) 3NEW [18:39:47] 6Labs, 10Labs-Infrastructure: Add replica.my.cnf to pagepile tool - https://phabricator.wikimedia.org/T104453#1418304 (10yuvipanda) Caused partially by T104476. You can use another tool's credentials in the meantime - I hope to have the situation resolved shortly. [18:40:32] Steinsplitter: it's broken atm, will hopefully be back tomorrow [18:40:47] ah, ok :/ [18:40:48] thx [18:42:32] 6Labs, 10Labs-Infrastructure: replica.my.cnf creation broken - https://phabricator.wikimedia.org/T104453#1418312 (10yuvipanda) [18:42:40] 6Labs, 10Labs-Infrastructure: replica.my.cnf creation broken - https://phabricator.wikimedia.org/T104453#1417449 (10yuvipanda) p:5Triage>3High [19:44:44] !log toolsbeta restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login [19:44:48] Logged the message, Master [19:47:24] !log toolsbeta still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it? [19:47:28] Logged the message, Master [19:48:53] YuviPanda|zzz: ^ :( no mailserver fixes today, then *sigh* [20:57:33] 6Labs, 10MediaWiki-extensions-OATHAuth, 10MobileFrontend, 10wikitech.wikimedia.org, 3Reading-Web: MF Special:Login doesn't have a field for 2FA - https://phabricator.wikimedia.org/T103771#1418696 (10Florian) @Jdlrobson: sorry for not responding here, but i'm in vacation and a bit busy :) But yeah, that w... [21:06:00] 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1418736 (10Andrew) [21:06:03] 6Labs, 7Tracking: Create labs project for analysis of recent changes and user contributions - https://phabricator.wikimedia.org/T104144#1418733 (10Andrew) 5Open>3Resolved a:3Andrew done [21:44:30] Hey labs, where does the beta cluster's interwiki map come from? Did someone build it by hand with Special:Interwiki ? [21:45:34] maybe imported form meta? [21:47:00] Luke081515: maybe. Much is customized, like b -> http://en.wikibooks.beta.wmflabs.org/wiki/$1 and all the interwiki language codes [21:48:19] or not updated since 2012? [22:04:32] beta cluster lacks mw interwiki, I filed https://phabricator.wikimedia.org/T104504 [23:19:05] Bot24: hi [23:19:05] Hello! [23:19:15] Bot24: help [23:19:15] Bot24 - Help [23:19:15] [23:19:16] Commands: [23:19:16] help - this help text [23:19:16] [23:19:16] Trusted user commands: [23:19:17] stop - shutdown bot [23:19:17] restart - restart bot [23:19:24] T2873 [23:19:44] oh its unhappy [23:28:58] T2873 [23:29:59] nm. Obviously needs more testing. Sorry for bothering [23:33:11] Negative24: What should this bot do? [23:33:20] Luke081515: lookup phab stuff [23:38:28] Luke081515: I shall demonstrate if Bot24 plays nice this time [23:38:36] T1892 [23:38:38] T1892: Doesn't exist [23:38:44] well bad example [23:38:47] T2871 [23:38:48] T2871: persistent site effect with link [[:en:User:Gangleri/tests/wikivar|en]] more details - https://phabricator.wikimedia.org/T2871 [23:38:52] yes! [23:39:46] looks good [23:40:03] plus it does urls as well [23:40:06] https://phabricator.wikimedia.org/T2871 [23:40:07] T2871: persistent site effect with link [[:en:User:Gangleri/tests/wikivar|en]] more details [23:40:53] 6Labs, 6Collaboration-Team: Investigate and remove NFS from editor-engagement project - https://phabricator.wikimedia.org/T102663#1419321 (10Catrope) p:5Triage>3High a:3Mattflaschen [23:40:54] T1026631419321: Doesn't exist [23:41:09] oh that's not good [23:41:13] Bot24: stop [23:41:13] Stopping... [23:41:29] alpha testing, yea!