[00:11:10] <wikibugs>	 6Labs, 10Wikidata: wdq.wmflabs.org does not update (data week old) - https://phabricator.wikimedia.org/T89583#1085136 (10Magnus) I haven't had time for a detailed look yet. Yuvi updated WDQ to the latest dump, which is where the timestamp comes from. He says both VMs are running 1 CPU at 100%, but don't reply...
[02:45:48] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy-02 is CRITICAL: CRITICAL: tools.tools-webproxy-02.diskspace.root.byte_percentfree.value (<14.29%)  
[02:55:43] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy-02 is OK: OK: All targets OK  
[04:38:44] <wikibugs>	 10Tool-Labs-tools-Other: Move Gerrit reports crontabs to Tools - https://phabricator.wikimedia.org/T88384#1085430 (10MZMcBride) Thanks for taking care of this! As briefly discussed on IRC, I've now disabled my local cron jobs. \o/
[06:06:13] <Niharika>	 Hello. I'm trying to update git on a Labs instance. On the labs-vagrant wiki page, it's suggested that I do so by changing user to vagrant. That lets me update the mediawiki repository, but gives me permission denied errors when I try updating any other repo. In my case, iegreview. 
[06:42:35] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0]  
[06:55:12] <wikibugs>	 10Wikimedia-Labs-Other, 6operations, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1085467 (10Springle)
[06:55:13] <wikibugs>	 10Tool-Labs, 7Tracking: Missing Toolserver features in Tools (tracking) - https://phabricator.wikimedia.org/T60791#1085468 (10Springle)
[06:55:14] <wikibugs>	 10Tool-Labs-tools-Other, 7Tracking: merl tools (tracking) - https://phabricator.wikimedia.org/T69556#1085469 (10Springle)
[07:07:42] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:06:45] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy-02 is CRITICAL: CRITICAL: tools.tools-webproxy-02.diskspace.root.byte_percentfree.value (<22.22%)  
[08:10:17] <wikibugs>	 6Labs, 10Tool-Labs: Monitor that the redundant webproxies have same state in terms of what they are proxying to whom - https://phabricator.wikimedia.org/T91484#1085566 (10yuvipanda) 3NEW a:3yuvipanda
[08:26:45] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy-02 is OK: OK: All targets OK  
[08:27:47] <wikibugs>	 6Labs, 10Tool-Labs, 5Patch-For-Review: Retire 'tomcat' node, make Java apps run on the generic webgrid - https://phabricator.wikimedia.org/T91066#1087224 (10yuvipanda) @Sitic: Can you also add -l release=trusty? That should transition it immediately, since otherwise jstart defaults to precise, and the newer...
[08:37:07] <sitic_>	 YuviPanda: those trusty generic webnodes seem to miss some common python packages: ImportError: No module named datetime
[08:37:46] <YuviPanda>	 sitic: oh, that’s.. strange.
[08:37:51] <YuviPanda>	 sitic: they should have them all.
[08:37:56] <YuviPanda>	 sitic: I’m looking
[08:42:23] <YuviPanda>	 >>> import datetime
[08:42:23] <YuviPanda>	 >>> 
[08:42:27] <YuviPanda>	 sitic: ^ works for me...
[08:42:34] <YuviPanda>	 sitic: are you using the system python?
[08:42:58] <sitic>	 in a virtualenv
[08:43:13] <sitic>	 that seems to be the problem
[08:44:27] <YuviPanda>	 sitic: right. you should maybe use the trusty bastion node - trusty.tools.wmflabs.org
[08:44:53] <sitic>	 ah thanks, I was looking for that. I'll test things there
[08:49:33] <sitic>	 YuviPanda: I get "ssh: connect to host trusty.tools.wmflabs.org port 22: Connection timed out" when I ssh from tools-login and "Permission denied (publickey,hostbased)." when I ssh directly for that trusty node
[08:51:14] <YuviPanda>	 sitic: I’m tailing the log file, can you try sshing in again?
[08:51:21] <YuviPanda>	 tools-trusty.wmflabs.org also works, I think
[08:51:49] <YuviPanda>	 > input_userauth_request: invalid user siric [preauth]
[08:52:02] <YuviPanda>	 what’s your shell username?
[08:52:44] <sitic>	 sitic
[08:53:00] <YuviPanda>	 right
[08:53:07] <YuviPanda>	 so try sshing to sitic@trusty.tools.wmflabs.org
[08:53:30] <sitic>	 ahh yes sorry, forgot to copy a line in my ssh config
[08:54:24] * sitic needs some coffee
[08:54:29] <sitic>	 thanks a lot, works now
[08:55:08] <YuviPanda>	 sitic: :D
[08:56:51] <marcmiquel>	 hi guys! I am trying to connect to a database in tool labs and I am having problems with it. Would you please indicate me what I do wrong?
[08:57:46] <YuviPanda>	 hey marcmiquel 
[08:58:01] <YuviPanda>	 marcmiquel: so this is Kolossos’ db, so he will have to create a user account there and give you creds
[08:58:29] <marcmiquel>	 he told me i could use my creds from myconf
[08:59:18] <marcmiquel>	 To access it:
[08:59:19] <marcmiquel>	 Databasehost "cawiki.labsdb" looks good,
[08:59:19] <marcmiquel>	 database for coordinates should be "p50380g50921__ghel_p"
[08:59:20] <marcmiquel>	 User and Password should be in your /replica.my.cnf or .my.cnf
[08:59:22] <marcmiquel>	 But for database access for Python I can't help you.
[08:59:24] <marcmiquel>	 Connection on terminal should be easy for a beginning "sql cawiki"
[09:04:02] <marcmiquel>	 I'm wondering how I am supposed to connect to the db..
[09:04:20] <YuviPanda>	 marcmiquel: you should be able to use http://mysql-python.sourceforge.net/MySQLdb.html to connect to it
[09:04:28] <marcmiquel>	 i do!
[09:04:54] <marcmiquel>	 but, my q is... does this db contains info from all langs?
[09:05:14] <marcmiquel>	 do i connect to it like it is cawiki, itwiki, ptwiki,...?
[09:08:41] <YuviPanda>	 marcmiquel: only Kolossos knows that.
[09:08:46] <YuviPanda>	 you can also go inside and look around
[09:08:49] <YuviPanda>	 is’t his database...
[09:08:55] <YuviPanda>	 it’s not provided by the WMF or something
[09:08:57] <YuviPanda>	 he maintains it
[09:21:56] <marcmiquel>	 morning livnetata
[09:40:27] <YuviPanda>	 sitic: is it working now?
[09:40:32] <YuviPanda>	 I think you might be the last person doing this :)
[09:40:39] <sitic>	 YuviPanda: yes thanks
[09:40:47] <YuviPanda>	 sitic: sweeet :) yw!
[09:40:54] <sitic>	 rebuild my virtualenv, that's it
[09:41:02] <YuviPanda>	 sitic: ah. does it work yet?
[09:41:13] <sitic>	 yes :-)
[09:41:15] <YuviPanda>	 sitic: we also have first class uwsgi support now, but dunno if tornado runs through that yet.
[09:41:24] <YuviPanda>	 (https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web)
[09:41:35] <YuviPanda>	 nodejs support too :)
[09:41:44] <sitic>	 yeah saw that, I don't think so, tornado does some strange things
[09:41:50] <YuviPanda>	 yeah
[09:42:04] <sitic>	 also pip is missing on the trusty nodes, had to install it manually.
[09:42:24] <sitic>	 so if you're going to use that image for generic exec node please add pip :-)
[09:46:08] <YuviPanda>	 sitic: you mean on the trusty generic node?
[09:46:23] <YuviPanda>	 or on trusty.tools
[09:46:23] <YuviPanda>	 ?
[09:50:47] <sitic>	 i mean when you switch everthing (normal exec nodes, tools-login etc., not web) to trusty
[09:51:57] <sitic>	 it's currently installed there
[10:30:03] <ananthrk_>	 YuviPanda: Hi. Created a new database in tools.toolsdb. Wrote a program to insert 20 million click stream data into this table
[10:31:09] <ananthrk_>	 Need to run the program again and hence wanted to clean up existing records. Initially tried "delete from", but since it was taking longer, killed it midway to try "drop table" command
[10:31:31] <ananthrk_>	 however I see that the "drop table" command is still waiting for a table metadata lock and it has been waiting for sometime now
[10:32:12] <ananthrk_>	 and am not able to query the engine status (no privilege)
[10:46:26] <YuviPanda>	 ananthrk_: oh wow. didn’t expect 20million entries :)
[10:46:41] <YuviPanda>	 you should probably test with smaller amounts
[10:46:54] <ananthrk_>	 YuviPanda: the table is gone now and am re-importing the data
[10:47:19] <ananthrk_>	 20 million is just for one month and this table is supposed to house data across months
[10:48:39] <YuviPanda>	 ananthrk_: oh, I see. I didn’t fully understand the scale when you meant ‘scale’ :)
[10:48:57] <YuviPanda>	 ananthrk_: can you open a bug about it and cc me / dario? i dunno if we’ll have capacity for multiple months of this..
[10:49:06] <YuviPanda>	 as in, literally, I do not know how much capacity we have :)
[10:52:13] <wikibugs>	 6Labs, 10Tool-Labs, 5Patch-For-Review: Retire 'tomcat' node, make Java apps run on the generic webgrid - https://phabricator.wikimedia.org/T91066#1087804 (10yuvipanda) Ok, so @sitic's tool has been transitioned away!
[11:25:32] <wikibugs>	 6Labs, 10Tool-Labs, 10Analytics: Make anonymized clickstream data available to the public - https://phabricator.wikimedia.org/T91495#1087859 (10yuvipanda) 3NEW
[12:00:59] <wikibugs>	 6Labs, 10Tool-Labs, 5Patch-For-Review: Move uwsgi jobs to be run on generic hosts, retire uwsgi hosts - https://phabricator.wikimedia.org/T91065#1088104 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Nodes removed, queues deleted, exec host deleted, and submit host deleted \o/
[12:01:00] <wikibugs>	 6Labs, 10Tool-Labs, 7Tracking: Make sure that toollabs can function fully even with one virt* host fully down - https://phabricator.wikimedia.org/T90542#1088107 (10yuvipanda)
[12:12:14] <wikibugs>	 10Tool-Labs, 10Wikimedia-Hackathon-2015: Conduct a Tool Labs workshop at Lyon Hackathon - https://phabricator.wikimedia.org/T91058#1088126 (10Qgil) Just checking, is this a dupe of {T90656}?
[12:28:45] <petan>	 YuviPanda: what happened to icinga.wmflabs.org?
[12:38:00] <petan>	 !channels
[12:38:00] <wm-bot>	 | #wikimedia-labs-nagios #wikimedia-labs-offtopic #wikimedia-labs-requests
[13:22:30] <YuviPanda>	 petan: it has been unmaintained for a long time. I *think* I killed it?
[13:22:37] <YuviPanda>	 Use shinken.wmflabs.org now
[13:22:46] <petan>	 what is that
[13:23:05] <petan>	 it wants password
[13:23:14] <YuviPanda>	 guest/guest
[13:23:53] <petan>	 shinken looks like yet another monitoring tool... besides hundreds that are already out :D
[13:46:04] <ananthrk_>	 YuviPanda: Are there any other logs besides the uwsgi.log in the home directory? I made some chagnes to the API and the new version refuses to load. the messages in uwsgi.log is not helpful ("unable to open app.py")
[13:57:00] <ananthrk_>	 YuviPanda: Please ignore. I managed to fix it myself
[14:00:10] <YuviPanda>	 ananthrk_:D please do document issues you run I to into the doc page so others can benefit :D
[14:05:42] <ananthrk_>	 YuviPanda: sure
[14:31:36] <matanya>	 YuviPanda: have a sec? 
[14:38:01] <anomie>	 YuviPanda: Is "webservice" going to eventually be migrated to "webservice2"?
[14:38:45] <YuviPanda>	 anomie: yup. Still a few bugs to work out
[14:38:59] <YuviPanda>	 matanya: in transit now...
[14:39:15] <matanya>	 some other time then :)
[14:45:42] <YuviPanda>	 anomie: most definitely within end of month.
[14:46:17] <YuviPanda>	 anomie: big question is that webservice2 defaults to trusty. Should I change it when it is called webservice or not
[14:47:29] <anomie>	 YuviPanda: I was just wondering if I could just wait for webservice to become trusty
[14:47:35] <YuviPanda>	 Several tools are written in ways that break badly when not using legacy php versions 
[14:47:57] <YuviPanda>	 anomie: you should start migrating :)
[14:48:01] <YuviPanda>	 The bugs are minor 
[14:48:18] <YuviPanda>	 Like webservice2 stop sometimes not working 
[14:48:32] <YuviPanda>	 And lack of webservice2 status command 
[14:49:52] * anomie sighs and does the 2 minutes of work to change
[14:50:55] <YuviPanda>	 :)
[15:12:48] <wikibugs>	 10Tool-Labs, 10Wikimedia-Hackathon-2015: Conduct a Tool Labs workshop at Lyon Hackathon - https://phabricator.wikimedia.org/T91058#1088500 (10coren) I don't think that it is; I think Yuvi is more of a tutorial on writing tools whereas mine is aimed at existing toolwriters about reinforcing their tools with mor...
[15:18:06] <wikibugs>	 6Labs, 6operations: Rename specific account in LDAP, Wikitech, Gerrit and Phabricator - https://phabricator.wikimedia.org/T85913#1088529 (10Tobi_WMDE_SW) @Chad @yuvipanda @coren so, nothing seemed to has happened here for almost a month now. can you tell us what people might be relevant for getting this done....
[15:19:57] <wikibugs>	 10Tool-Labs, 10Wikimedia-Hackathon-2015, 7Documentation: Re-organize Tool Labs documentation - https://phabricator.wikimedia.org/T91509#1088547 (10yuvipanda) 3NEW
[16:11:30] <wikibugs>	 10MediaWiki-extensions-OpenStackManager: Special pages should link to in-wiki documentation - https://phabricator.wikimedia.org/T36500#1088832 (10matmarex)
[16:11:39] <wikibugs>	 10MediaWiki-extensions-OpenStackManager: Special pages should link to in-wiki documentation - https://phabricator.wikimedia.org/T36500#388005 (10matmarex)
[16:11:49] <wikibugs>	 10MediaWiki-extensions-OpenStackManager: OpenStackManager special pages should link to in-wiki documentation - https://phabricator.wikimedia.org/T36500#1088835 (10matmarex)
[16:18:24] <marcmiquel>	 good afternoon
[16:19:01] <marcmiquel>	 I am having several 'broken pipe' today...much more often than usual. I know it might be redundant since there this channel status...but is everything ok? :)
[16:21:18] <wikibugs>	 10MediaWiki-extensions-OpenStackManager: OpenStackManager special pages should link to in-wiki documentation - https://phabricator.wikimedia.org/T36500#1088893 (10matmarex)
[16:52:52] <wikibugs>	 6Labs, 7Tracking: MediaWiki Extension "ImportArticles" project - https://phabricator.wikimedia.org/T89208#1089036 (10Andrew) Can you tell me a bit more about what you'll be doing with this?  e.g. will you need more than one instance?
[17:09:42] <wikibugs>	 6Labs: Upgrade Labs Compute nodes to Trusty - https://phabricator.wikimedia.org/T90822#1089074 (10Andrew) 1012 is running Trusty now -- the upgrade was mostly painless although it did require a reboot.
[17:31:25] <Coren>	 marcmiquel: Where are you getting the broken pipes from?
[17:32:03] <wikibugs>	 6Labs, 7Tracking: MediaWiki Extension "ImportArticles" project - https://phabricator.wikimedia.org/T89208#1089183 (10Cblair91) I am intending to implement some more stuff regarding it to handle the loading better (and not through the API possibly). A single instance is fine :)
[17:32:22] <mforns>	 bd808, hi
[17:34:55] <mforns>	 what should I do to get added to deployment labs project? I'd like to access deployment-eventlogging02
[17:40:53] <mforns>	 hi Coren
[17:41:03] <Coren>	 o/
[17:41:10] <mforns>	 quick question
[17:41:15] <mforns>	  what should I do to get added to deployment labs project?
[17:42:41] <Coren>	 mforns: You have to request it from one of the team leads, I suppose.  In general, projects are managed by their own teams and not the labs admin.
[17:42:45] <Coren>	 (Including membership)
[17:44:21] <jzerebecki>	 !log Wikidata-build resurecting this project for automatically building wikidata again
[17:44:22] <labs-morebots>	 Wikidata-build is not a valid project.
[17:44:37] <jzerebecki>	 !log wikidata-build resurecting this project for automatically building wikidata again
[17:44:40] <labs-morebots>	 Logged the message, Master
[17:44:47] <mforns>	 Coren ok, do you know where I can find the lead of a project? of deployment project?
[17:45:17] <mforns>	 searched in wikitech but did not find it
[17:46:47] <jzerebecki>	 mforns: https://www.mediawiki.org/wiki/Beta_cluster
[17:47:06] <mforns>	 thanks jzerebecki 
[17:47:19] <Coren>	 That's Antoine Musso, IIRC.  Aka hashar
[17:48:09] <mforns>	 thanks Coren!
[17:53:41] <wikibugs>	 6Labs, 10MediaWiki-extensions-OpenStackManager: Wikitech 'manage instances' displays "PHP Fatal error:  Call to a member function getImageName() on a non-object" - https://phabricator.wikimedia.org/T89856#1089365 (10Andrew) This should be fixed by https://gerrit.wikimedia.org/r/#/c/194358/
[17:53:56] <marcmiquel>	 Coren: I'm getting using terminal ssh connection and python I meant.
[17:54:05] <marcmiquel>	 i'm getting several "broken pipes"
[17:54:10] <marcmiquel>	 at half execution.
[17:56:11] <andrewbogott>	 marcmiquel: do you think that your network connection is just timing out in the middle of running a command?
[17:56:22] <andrewbogott>	 that wouldn’t be unusual if you have flaky wifi for example
[17:56:49] <marcmiquel>	 i don't think so, it happened to me while i'm connected in two diff places
[17:56:58] <marcmiquel>	 but it could be...
[17:58:29] <andrewbogott>	 You can try using the ‘screen’ tool, which gives you a persistent session that you can reconnect to if you get kicked off
[17:58:44] <marcmiquel>	 where can i get this tool?
[17:58:51] <andrewbogott>	 just type ‘screen’ :)
[17:59:16] <andrewbogott>	 This is on tools-login, right?
[17:59:24] <andrewbogott>	 Try running screen and then intentionally breaking your connection...
[17:59:29] <marcmiquel>	 screen ssh x@...?
[17:59:33] <andrewbogott>	 Ah, no --
[17:59:43] <andrewbogott>	 ssh to tools-login, and then run ‘screen’ once you’re connected.
[18:00:14] <andrewbogott>	 Then when you reconnect, I think just ‘screen -r’ should reconnect you
[18:00:16] * andrewbogott tries it
[18:00:32] <marcmiquel>	 what should i see?
[18:00:50] <marcmiquel>	 reconnect? if there's broken pipe my python execution dies
[18:01:00] <marcmiquel>	 maybe i should use qsub or jsub instead
[18:01:05] <andrewbogott>	 maybe!
[18:01:19] <andrewbogott>	 But screen is still useful :)  Screen makes a terminal connection that’s hosted locally on tools-login
[18:01:28] <marcmiquel>	 i see :)
[18:01:30] <andrewbogott>	 So if your connection (between laptop and tools-login) is broken, the python job doesn’t notice
[18:01:30] <marcmiquel>	 thanks!
[18:01:36] <andrewbogott>	 because it’s attached to the screen session
[18:01:45] <andrewbogott>	 Then, later /you/ reconnect to the screen session and you can see what happened.
[18:02:09] <andrewbogott>	 It’s a pretty standard tool for handling a bad network connection.
[18:02:09] <marcmiquel>	 :)
[18:02:19] <marcmiquel>	 just typing is enough
[18:02:21] <andrewbogott>	 mosh is the newer way to handle that, but mosh support isn’t great in labs yet
[18:03:04] <Coren>	 marcmiquel: The broken pipe normally happens when something writes to a pipe or socket but nothing is reading.
[18:03:45] <Coren>	 marcmiquel: If you do, for instance, "something | otherthing" and otherthing dies or exits before something is done writing to stdout, it'll get that error.
[18:04:35] <marcmiquel>	 the connection dies before python has time to finish?
[18:05:41] <Coren>	 marcmiquel: I'd need to know more precisely what you are trying to do.  In general, it means that whatever you are writing to goes away before you're done.
[18:06:18] <Coren>	 andrewbogott: mosh is newer indeed, but it's also very different in what it's doing and has its own set of quirks.
[18:06:19] <marcmiquel>	 my python code reads from mw database and writes out to the terminal
[18:06:30] <marcmiquel>	 in the end it will write in a file too
[18:06:34] <marcmiquel>	 but it didn't get till the end
[18:06:42] <marcmiquel>	 not even showed all that needed to show in the terminal output
[18:07:36] <Coren>	 You shouldn't get a broken pipe, ever, unless you are writing to a pipe or socket.  Possibly, what you are getting this error for is because your code is trying to send a command to the db server and the connection is gone.
[18:07:53] <andrewbogott>	 Ah, my advice may be useless if the problem is the pipe between python and terminal (and not between your local machine and the remote)
[18:08:04] <marcmiquel>	 Coren: this is very possible
[18:08:23] <marcmiquel>	 that the database is having some cuts, right?
[18:08:35] <marcmiquel>	 or gone for secs
[18:09:26] <marcmiquel>	 i have to leave for few min.
[18:09:30] <marcmiquel>	 brb
[18:15:28] <wikibugs>	 10Wikimedia-Labs-wikitech-interface: some labs / nova projects exist but are not listed in all projects / SWM ask / by property search - https://phabricator.wikimedia.org/T62288#1089394 (10JanZerebecki) 5Open>3Resolved
[18:31:27] <wikibugs>	 10MediaWiki-extensions-OpenStackManager, 10Wikimedia-Labs-General: adding/removing project members updates project page, changing projectadmins does not - https://phabricator.wikimedia.org/T73164#1089451 (10greg) See: https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Deployment-prep&diff=next&oldi...
[18:32:59] <wikibugs>	 6Labs: Sanity check submitted SSH keys on Labs - https://phabricator.wikimedia.org/T77902#1089455 (10Krenair)
[18:33:23] <wikibugs>	 10MediaWiki-extensions-OpenStackManager: Sanity check submitted SSH keys on Labs - https://phabricator.wikimedia.org/T77902#831412 (10Krenair)
[18:33:46] <wikibugs>	 10MediaWiki-extensions-OpenStackManager: Sanity check submitted SSH keys - https://phabricator.wikimedia.org/T77902#831412 (10Krenair)
[19:01:01] <PiRSquared>	 hi
[19:01:09] <PiRSquared>	 can I ask about labs DB here?
[19:02:02] <JohnFLewis>	 PiRSquared: this is the labs channel so shoot :p
[19:02:40] <PiRSquared>	 SELECT family FROM meta_p.wiki WHERE dbname="sourceswiki";
[19:02:53] <PiRSquared>	 should probably give "wikisource" not "special"
[19:03:15] <PiRSquared>	 JohnFLewis: ?
[19:04:31] <PiRSquared>	 why is it "special"?
[19:11:17] <Coren>	 PiRSquared: Hm, because it didn't fit the standard database naming scheme.
[19:12:53] <Coren>	 (i.e. *wikisource)
[19:13:33] <Coren>	 It'll need to be special cased.
[19:14:03] <PiRSquared>	 downstream or can you fix it in the DB itself?
[19:14:54] <Coren>	 I have to fix the script that maintains the db, but I can hotfix the value in the db if it's causing you issues now.
[19:15:10] <PiRSquared>	 nothing serious
[19:16:43] <wikibugs>	 6Labs: Labs replica DBs incorrectly classify sourceswiki as 'special' - https://phabricator.wikimedia.org/T91534#1089566 (10coren) 3NEW a:3coren
[20:36:25] <PiRSquared>	 legoktm: should it be added to wikisources dblist?
[20:36:31] <PiRSquared>	 or will that break stuff
[20:49:32] <legoktm>	 PiRSquared: off the top of my head I have no idea
[20:50:11] <PiRSquared>	 I guess it kind of is "special"... but it's also a wikisource
[21:13:12] <Howie>	 andrewbogott: I wonder if you can help me. I ran importDump.php what seemed to be successful and as far as I can tell the MediaWiki pages aren't changing
[21:13:36] <andrewbogott>	 Howie: can I get some context?
[21:13:45] <andrewbogott>	 Ran it where?
[21:13:46] <Howie>	 drmf.eqiad.wmflabs
[21:13:58] <Howie>	 The xml file is located in /home/hcohl
[21:14:22] <Howie>	 it is called 01outb.mmd2015-03-4T13_06_42Zdrmf.xml
[21:14:45] <andrewbogott>	 and you’re using labs-vagrant?
[21:16:00] <Howie>	 yes
[21:17:00] <andrewbogott>	 what project is this?
[21:17:01] <Howie>	 The reason I know it's not working is 
[21:17:06] <Howie>	 (math)
[21:17:15] <Howie>	 http://drmf.wmflabs.org/w/index.php?title=Stieltjes-Wigert&mathpurge=true
[21:17:47] <Howie>	 I can look in the xml file and some of the Wikitext is different on the Wiki than what is in the xml file
[21:18:23] <Howie>	 http://pastebin.com/2Ddcwrkv
[21:18:32] <Howie>	 shows the importDump worked
[21:19:00] <andrewbogott>	 Howie: if you look at https://wikitech.wikimedia.org/wiki/Special:NovaProxy you will see something interesting...
[21:19:18] <andrewbogott>	 drmf.wmflabs.org points to the drmf-beta instance
[21:19:28] <andrewbogott>	 and drmf-beta.wmflabs.org points to drmf
[21:19:53] <Howie>	 physikerwelt set that up.
[21:20:36] <andrewbogott>	 ok, but — do you see what I mean?  You are importing on one machine and the looking at a different one
[21:20:37] <Howie>	 we want to point to the drmf instance, but all our development was on the drmf-beta instance, he said that will work
[21:20:45] <andrewbogott>	 So it’s no surprise that there’s no change on the one you’re looking at...
[21:20:54] <Howie>	 where should I look?  on beta?
[21:21:48] <andrewbogott>	 um…
[21:22:10] <andrewbogott>	 Can you look at the page I just linked?  You should be able to see what web addresses correlate to what instances.
[21:24:40] <Howie>	 I'm looking at it,....
[21:25:12] <andrewbogott>	 It looks to me like if you are modifying the host drmf.eqiad.wmflabs, that corresponds to the url drmf-beta.wmflabs.org
[21:26:18] <Howie>	 so strange.
[21:26:29] <Howie>	 yes if I look there , it looks ok.
[21:26:45] <Howie>	 thank you
[21:27:02] <andrewbogott>	 sure thing.  I don’t know why the names are crossed, it seems… intentionally confusing :)
[21:27:19] <Howie>	 We are presenting a demo for CICM 2015
[21:27:28] <Howie>	 he doesn't want to point to drmf-beta in the paper.
[21:27:55] <Howie>	 so he set it up so that we access drmf.wmflabs.org
[21:28:05] <Howie>	 to get to the demo wiki
[21:28:20] <Howie>	 at least that's what I thought it was supposed to do.
[21:28:59] <Howie>	 Can you help me make it so that the wiki we have the demo on which works is viewed from drmf.wmflabs.org ?
[21:30:10] <andrewbogott>	 sure… so, you just want http://drmf.wmflabs.org to point to the drmf.eqiad.wmflabs instance?
[21:30:29] <Howie>	 This is the working instance
[21:30:30] <Howie>	 http://drmf-beta.wmflabs.org/wiki/Main_Page
[21:30:52] <Howie>	 we would like to be able to access this by typing the url http://drmf.wmflabs.org
[21:31:11] <andrewbogott>	 and you’re ok if the site currently referenced as drmf-beta falls off the internet?
[21:31:28] <Howie>	 I don't want to lose the correct wiki though.
[21:32:00] <Howie>	 I can tell it's the correct wiki because the page "Orthogonal Polynomials" is empty.
[21:32:02] <andrewbogott>	 can I just make the ‘beta’ url point to the ‘beta’ instance and the non-beta url point to the non-beta instance?
[21:32:05] <Howie>	 also
[21:32:22] <Howie>	 this page loads with no errors: http://drmf-beta.wmflabs.org/wiki/Stieltjes-Wigert
[21:34:27] <andrewbogott>	 !log math moved http://drmf-beta.wmflabs.org to point to the drmf-beta instance, and http://drmf.wmflabs.org to point to the drmf instance.  Because previously it was the other way around which was super confusing.
[21:34:33] <labs-morebots>	 Logged the message, dummy
[21:34:56] <andrewbogott>	 Howie: give it a few minutes, things should sort themselves out
[21:35:07] <Howie>	 However, my MediaWiki:Common.css doesn't look like it's loading on the working instance
[21:35:19] <andrewbogott>	 !log math (and also because Howie requested it)
[21:35:23] <labs-morebots>	 Logged the message, dummy
[21:36:05] <andrewbogott>	 I can’t really account for the behavior of labs-vagrant, but at least you should be able to see what you’re doing now :)
[21:36:23] <Howie>	 If I navigate to drmf.wmflabs.org, I will see the working Wiki?
[21:37:15] <andrewbogott>	 you will see the wiki hosted on drmf.eqiad.wmflabs
[21:37:22] <andrewbogott>	 as soon as DNS updates
[21:37:29] <andrewbogott>	 Oh, actually, nm, it should change immediately.
[21:37:30] <andrewbogott>	 So, yes.
[21:37:51] <Howie>	 ok great, thanks
[21:38:00] <Howie>	 Can you help me understand why my Common.css file isn't loading?
[21:38:28] <Howie>	 Also now my Admin password isn't working
[21:38:54] <Howie>	 I really appreciate this.
[21:39:09] <Howie>	 I see it on drmf.wmflabs.org now
[21:39:27] <andrewbogott>	 Howie: I really don’t know much about mediawiki so I’m probably the wrong person to ask.  Are there any mw developers around?
[21:40:42] <JohnFLewis>	 andrewbogott: hey, when you have a spare second, could you look into a ssh issue for me?
[21:40:42] <Howie>	 I don't know.
[21:41:56] <Howie>	 Can anybody help me reset my Admin password?
[21:45:17] <JohnFLewis>	 andrewbogott: basically mailman-02 is refusing any attempts to ssh into it (running, unfortunately Ubuntu Lucid too)
[21:46:22] <bd808>	 Howie: I got in with the default mediawiki-vagrant admin password
[21:48:04] <andrewbogott>	 JohnFLewis: looking...
[21:48:13] <andrewbogott>	 bd808: can you help Howie sort out his css issue?
[21:48:18] <JohnFLewis>	 thanks
[21:48:28] <YuviPanda>	 andrewbogott: JohnFLewis I think puppet doesn’t run on lucid at all, so you have to get in with root...
[21:48:42] <andrewbogott>	 My root key doesn’t work either
[21:48:46] <YuviPanda>	 ah
[21:48:49] <bd808>	 andrewbogott: I can try. Howie what's busted for you?
[21:48:51] <YuviPanda>	 then it might be *fully* screwed
[21:49:01] * JohnFLewis sighs
[21:49:45] <JohnFLewis>	 really need a duplicate of production to test on which unfortunately is running Ubuntu Lucid :/
[21:50:33] <Howie>	 I figured out how to change the Admin password, thanks!
[21:51:07] <wikibugs>	 6Labs, 6operations: Puppetize labstore1003 - https://phabricator.wikimedia.org/T91573#1090306 (10yuvipanda) 3NEW
[21:52:17] <andrewbogott>	 YuviPanda: I think that puppet sort of works but NFS does not.  So if puppet is installing keys that should sort of work...
[21:52:23] * andrewbogott digs
[21:52:37] <YuviPanda>	 right
[21:52:56] <JohnFLewis>	 andrewbogott: I think you said that was the issue when you helped me set it up a few months ago. thanks for looking into it :)
[21:52:57] <YuviPanda>	 andrewbogott: there’s deployment-lucid-salt (or somesuch) that’s also running lucid
[21:53:01] <YuviPanda>	 dunno if that works at all tho
[21:54:08] <JohnFLewis>	 YuviPanda: that doesn't (can't ssh into it)
[21:54:43] <YuviPanda>	 http://shinken.wmflabs.org/host/deployment-lucid-salt
[21:55:25] <andrewbogott>	 JohnFLewis: mind if I reboot it so I can watch the boot log?
[21:55:36] <JohnFLewis>	 andrewbogott: sure
[21:55:54] <JohnFLewis>	 it's just running mailman on it so I don't mind what you do to it
[21:56:00] <andrewbogott>	 Also, if you could direct me to a public key that you’d like to use, I can see about mounting the drive and installing by hand
[21:58:40] <JohnFLewis>	 andrewbogott: https://api.github.com/users/JohnFLewis/keys
[21:59:51] <JohnFLewis>	 that's my current one (according to GitHub anyway :p)
[22:07:24] <JohnFLewis>	 !log mailman created mailman-jessie
[22:07:28] <labs-morebots>	 Logged the message, Master
[22:14:40] <andrewbogott>	 JohnFLewis: well, my attempts seem to have killed the instance.  If this is important we can start anew...
[22:16:07] <JohnFLewis>	 andrewbogott: if possible. My intents is to have a working mailman-production dupe (so, Lucid), a trusty version (only because it holds some stuff needed) and I'm setting up a jessie version. So, if we can recreate a Lucid, please :)
[22:17:27] <JohnFLewis>	 also how does labs handle the ability to manage incoming mail? (as this is mailman, having the ability to use the email side of it would be a major plus)
[22:19:23] <YuviPanda>	 JohnFLewis: I set that up for beta at some point. Have to fiddle with LDAP manually to create an MX record
[22:20:27] <JohnFLewis>	 YuviPanda: is it something I can convince you guys to do? (if I set up what ever is needed on the project side)
[22:20:42] <YuviPanda>	 JohnFLewis: yup, totally
[22:20:50] <YuviPanda>	 just a matter of filing bugs, etc.
[22:21:10] <JohnFLewis>	 YuviPanda: under the general labs project right?
[22:21:20] <YuviPanda>	 JohnFLewis: yup
[22:21:32] <YuviPanda>	 JohnFLewis: the instance would also need a public IP, and associated hostname...
[22:22:11] <JohnFLewis>	 they'll have associated hostnames though I don't believe we have any public IPs to use/assign
[22:22:48] <JohnFLewis>	 I'll file a bug and you can just list off what needs to be done and we can work form there
[22:22:56] <YuviPanda>	 sure thing
[22:22:59] <YuviPanda>	 probably not today tho...
[22:25:42] <wikibugs>	 6Labs: Ability to handle mail (mailman project) - https://phabricator.wikimedia.org/T91576#1090401 (10JohnLewis) 3NEW
[22:26:06] <JohnFLewis>	 YuviPanda: ^^ just put there what needs to be done/asked for and I'll get around to it
[22:33:16] <andrewbogott>	 JohnFLewis: sorry, I can’t get a lucid instance to spin up at all.  you might better reach your goal by petitioning ops to upgrade their server :)
[22:33:48] <JohnFLewis>	 andrewbogott: trust me, it's on their list but currently codfw is marked as a blocker :/
[22:34:23] <JohnFLewis>	 I guess just working with jessie will be fine for now as that is the planned upgrade target
[22:35:03] <wikibugs>	 6Labs: Ability to handle mail (mailman project) - https://phabricator.wikimedia.org/T91576#1090452 (10JohnLewis)
[22:35:40] <wikibugs>	 6Labs: Ability to handle mail (mailman project) - https://phabricator.wikimedia.org/T91576#1090456 (10yuvipanda) From a DNS perspective, you will need to have an MX record set up (via LDAP directly, afaik), and a public IP / A record set up as well. This should allow incoming mail handling.
[22:37:46] <JohnFLewis>	 andrewbogott / YuviPanda: could one of you assign a floating IP to the mailman project please? :)
[22:38:08] <andrewbogott>	 yep, one second…
[22:38:29] <YuviPanda>	 thanks andrewbogott 
[22:38:34] <YuviPanda>	 Coren: did you see my email about tomcat / generic?
[22:39:00] <Coren>	 YuviPanda: You mean the one to labs-l?  Aye.
[22:39:14] <YuviPanda>	 Coren: I’ve killed the special uwsgi nodes as well.
[22:40:06] <Coren>	 YuviPanda: Hm.  That means there's going to be an increase on memory pressure; we'll have to keep an eye on 'em/
[22:40:34] <YuviPanda>	 Coren: yes, with shinken alerts! :)
[22:40:44] <YuviPanda>	 and adding more instances :D
[23:06:54] <wikibugs>	 6Labs, 10Tool-Labs: Request to be added as maintainer to abandoned bibleversefinder/ tool - https://phabricator.wikimedia.org/T91585#1090568 (10Technical13) 3NEW
[23:09:59] <wikibugs>	 6Labs, 10Tool-Labs: Request to be added as maintainer to abandoned bibleversefinder/ tool - https://phabricator.wikimedia.org/T91585#1090580 (10yuvipanda) I've started it back up and put it on bigbrotherrc so it should stay up. Can you test to see if it works?
[23:13:52] <JohnFLewis>	 YuviPanda: I've assigned a public IP + a host name
[23:13:56] <JohnFLewis>	 andrewbogott: thanks
[23:17:35] <Technical_13>	 @link [[Phab:T91585]] Coren YuviPanda 
[23:17:35] <wm-bot>	 https://wikitech.wikimedia.org/wiki/Phab:T91585
[23:18:00] <YuviPanda>	 Technical_13: I already responded
[23:18:08] <Technical_13>	 :)
[23:20:05] <Technical_13>	 YuviPanda: https://www.biblegateway.com/passage/?search=job%203:1-3:1&version=KJV looks good.
[23:20:17] <YuviPanda>	 Technical_13: sweet
[23:20:43] <Technical_13>	 I realize that the ticket will probably be marked as "stalled" until the RfC has concluded, and that is fine by me.  I'd still like to be added per the process. :)
[23:21:30] <wikibugs>	 6Labs: Ability to handle mail (mailman project) - https://phabricator.wikimedia.org/T91576#1090614 (10JohnLewis) Assigned a public IP and a host for mailman-jessie.
[23:21:32] <wikibugs>	 6Labs, 10Tool-Labs: Request to be added as maintainer to abandoned bibleversefinder/ tool - https://phabricator.wikimedia.org/T91585#1090615 (10yuvipanda) ```<Technical_13>  YuviPanda: https://www.biblegateway.com/passage/?search=job%203:1-3:1&version=KJV looks good.```  Given that it seems ok, are there any c...
[23:24:50] <wikibugs>	 6Labs, 10Tool-Labs: Request to be added as maintainer to abandoned bibleversefinder/ tool - https://phabricator.wikimedia.org/T91585#1090624 (10Technical13) No other complaints other than "it's down" that I'm aware of.  I'd still like to be added as a maintainer if even it means that I must wait for a conclusi...
[23:25:28] <wikibugs>	 6Labs, 10Tool-Labs: Request to be added as maintainer to abandoned bibleversefinder/ tool - https://phabricator.wikimedia.org/T91585#1090625 (10yuvipanda) 5Open>3stalled :) ok!
[23:25:32] <Betacommand>	 How do you do a search in the database when the value we are working with is UTF8 and the database encoding is latin-l ?
[23:25:57] <Technical_13>	 Now I need to head to bible study (oddly enough).