[00:02:23] <Dispenser>	 Getting dab solver running involves about 3/4 of the work of a simple port my tools
[00:05:15] <Dispenser>	 It kind of specifically architected for Toolserver, Web browsers, and the weird rules
[00:06:19] <legoktm>	 If you need help porting, I'd be more than willing to spend some time on it
[00:07:11] <Dispenser>	 Have you ported TS query-killer yet?  Yeah, it relies on that.
[00:07:30] <legoktm>	 labs has it's own query killer iirc
[00:07:35] <legoktm>	 or did you need something specific?
[00:07:48] <legoktm>	 in which case, I'd be willing to help port that.
[00:10:27] <tan_>	 hi
[00:10:42] <tan_>	 I am new to wikimedia tool labs
[00:11:17] <tan_>	 I am trying to access the tool labs but getting this connection error: debug1: Authentications that can continue: publickey,hostbased debug1: Next authentication method: publickey debug1: Offering RSA public key: /Users/sayantanm/.ssh/id_rsa debug1: Authentications that can continue: publickey,hostbased debug1: Trying private key: /Users/sayantanm/.ssh/id_dsa debug1: read PEM private key done: type RSA debug1: Authentications t
[00:11:33] <Dispenser>	 Really, I thought I read "Toolserver is the only place where read queries increase replication lag" someplace.  It needs to support /* related.py LIMIT:3 NM */
[00:12:34] <Dispenser>	 I wrote my own query-killer (TS's wasn't quick enough killing), but screen deamon's aren't allowed here
[00:12:49] <tan_>	 I regenerated my RSA keys but no luck
[00:17:37] <legoktm>	 Dispenser: what does the full query look like?
[00:18:14] <Dispenser>	 That's just the relevant part
[00:18:27] <legoktm>	 can your daemon just run in the grid?
[00:22:46] <Dispenser>	 May be able to.  It was in auto-restarting bash loop and it printed out the current status in case I needed diagnose things.
[01:57:46] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Template search API request in Beta Labs finds nothing - 10https://bugzilla.wikimedia.org/66981#c3 (10James Forrester) 5REOP>3NEW a:3None Yes, appears to be the same Beta Labs flakiness again; moving there.
[04:43:56] <SigmaWP>	 Where is the socket for mysql located?
[05:11:04] <SigmaWP>	 never mind, found it
[09:06:22] <hashar>	 !log deployment-prep Adding cawiki and eswiki for cxserver testing {{gerrit|Ibbcbd43d43f878099788518d9e952fa7b20d8ca3}}
[09:06:24] <labs-morebots>	 Logged the message, Master
[09:10:39] <hashar>	 !log deployment-prep used addwiki.php to create the wiki. manually triggered the Jenkins job that update the databases https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/2319/
[09:10:40] <labs-morebots>	 Logged the message, Master
[09:11:16] <hashar>	 kart_: the dbs are upgrading
[09:11:16] <hashar>	 https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/2319/label=deployment-bastion-eqiad,wikidb=eswiki/console
[09:11:20] <hashar>	 https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/2319/label=deployment-bastion-eqiad,wikidb=cawiki/console
[09:11:31] <kart_>	 nice :)
[09:12:19] <kart_>	 beer for hashar!
[09:16:04] <hashar>	 the stupid addwiki script did not create the index
[09:16:05] <hashar>	 ba
[09:17:27] <kart_>	 :(
[09:22:38] <hashar>	 !log deployment-prep Blow up ElasticSearch indices for cawiki and eswiki with: <tt>mwscript extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php --wiki cawiki --startOver --indexType content  && mwscript extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php --wiki cawiki --startOver --indexType general</tt>
[09:22:40] <labs-morebots>	 Logged the message, Master
[09:24:14] <hashar>	 !log deployment-prep Reindexed ElasticSearch index for cawiki/eswiki with: mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki {cawiki,eswiki} --batch-size=50
[09:24:16] <labs-morebots>	 Logged the message, Master
[09:24:18] <hashar>	 kart_: should be good now
[09:24:31] <hashar>	 http://ca.wikipedia.beta.wmflabs.org/wiki/Pàgina_principal http://es.wikipedia.beta.wmflabs.org/wiki/Página_principal
[09:40:40] <kart_>	 hasteur: Thanks, again!
[10:09:13] <YuviPanda>	 DispenserAFK: printing out things land up in the 'out' log on the tool's home dir
[11:12:00] <wikibugs>	 3Wikimedia Labs / 3tools: Provide wiki metadata in the databases similar to toolserver.wiki - 10https://bugzilla.wikimedia.org/48626 (10merl)
[11:12:01] <wikibugs>	 3Wikimedia Labs / 3tools: Missing Toolserver features in Tools (tracking) - 10https://bugzilla.wikimedia.org/58791 (10merl)
[11:12:02] <wikibugs>	 3Wikimedia Labs / 3tools: Add is_sensitive to meta_p.wiki - 10https://bugzilla.wikimedia.org/67476 (10merl) 3NEW p:3Unprio s:3enhanc a:3Marc A. Pelletier on toolserver.wiki there was a column is_sensitive. This is needed by to tools parsing user input oder langlinks. E.g first letter of titles for de...
[11:14:53] <YuviPanda>	 !log
[11:21:46] <wikibugs>	 3Wikimedia Labs / 3tools: Add some of the missing tables in commonswiki_f_p - 10https://bugzilla.wikimedia.org/59683#c12 (10nosy) Seems to be a general problem.
[11:57:21] <a930913>	 How can I restrict a query to pages created within the last month? Whatever I do seems to be slow. "SELECT page_title,   (SELECT COUNT(*)   FROM imagelinks   WHERE il_to = page_title) AS imagelinks,   (SELECT COUNT(*)   FROM pagelinks   WHER
[11:57:25] <a930913>	 E pl_namespace = 6     AND pl_title = page_title) AS links FROM page WHERE page_namespace = 6   AND page_is_redirect = 1 HAVING imagelinks + links <= 1;"
[12:07:01] <wikibugs>	 3Wikimedia Labs / 3tools: Add some of the missing tables in commonswiki_f_p - 10https://bugzilla.wikimedia.org/59683#c13 (10Silke Meyer (WMDE)) Any chance we get this resolved? Please increase priority so that Merl can finish his migration, too.
[12:07:34] <YuviPanda>	 meh
[12:13:31] <valhallasw>	 a930913: lazy man's solution: first look up the page id of a page created a month ago (from revision, I guess?) then use that as min(page_id)?
[12:14:15] <a930913>	 valhallasw: Now /that/ is programming :D
[12:14:40] <valhallasw>	 well, let's see if it's fast enough first :-p
[12:16:52] <a930913>	 valhallasw: Oh, 4 seconds to go back a day :(
[12:18:09] <valhallasw>	 a930913: maybe use recentchanges?
[12:40:56] <Betacommand>	 a930913: I would also suggest what valhallasw says but also use the rev_prev or what ever its called
[12:41:37] <Betacommand>	 Coren: enwiki is about 45 minutes lagging
[13:04:00] <a930913>	 Betacommand: Help? :p
[13:06:10] <Betacommand>	 a930913: sure if you give me some help :P
[13:06:24] * a930913 hands Betacommand some help.
[13:06:53] <Betacommand>	 a930913: Give me a sec and Ill dig up the RC query
[13:07:14] <Betacommand>	 a930913: there is a new query at VPT about a whois
[13:08:16] <Betacommand>	 http://tools.wmflabs.org/betacommand-dev/cgi-bin/SIL?ip= 101.60.209.117 or more generichttp://tools.wmflabs.org/betacommand-dev/SIL.html 
[13:08:25] <Betacommand>	 http://tools.wmflabs.org/betacommand-dev/cgi-bin/SIL?ip=101.60.209.117 or more generichttp://tools.wmflabs.org/betacommand-dev/SIL.html 
[13:08:54] <Betacommand>	 a930913: can you post a note for me?
[13:12:08] <Betacommand>	 a930913: select * from recentchanges where rc_last_oldid = 0 ; gives you all new pages in the last 30 days
[13:12:30] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Import some content on beta.wmflabs.org wikis - 10https://bugzilla.wikimedia.org/66402 (10Antoine "hashar" Musso)
[13:16:38] <a930913>	 Betacommand: I should stop this whole meatsocking attribution thing, and just claim the glory for myself :p
[13:16:54] <Betacommand>	 a930913: might cause fewer headaches
[13:17:05] <a930913>	 Betacommand: I have a query, and I want to limit it to pages created in the last month.
[13:17:09] <Betacommand>	 a930913: I really dont care, as long as that info is given
[13:17:19] <Betacommand>	 a930913: see above
[13:17:36] <Betacommand>	 that query just gives new pages in the last month
[13:17:38] <a930913>	 I don't want a list of new pages though :/
[13:17:45] <a930913>	 Oh!
[13:17:52] <a930913>	 You mean like that.
[13:17:59] <Betacommand>	 a930913: join it to your query
[13:18:10] <a930913>	 Sorry, I was still thinking of valhallasw's hack :)
[13:18:25] <Betacommand>	 a930913: which is what I used :P
[13:18:51] <a930913>	 Betacommand: Erm, what type of join is that?
[13:19:18] <Betacommand>	 pastebin your current query
[13:19:42] <Glaisher>	 http://tools.wmflabs.org/betacommand-dev/SIL.html doesn't provide full WHOIS details...
[13:20:06] <Betacommand>	 Glaisher: what is is missing?
[13:20:22] <Glaisher>	 lots...
[13:20:24] <Glaisher>	 http://whois.domaintools.com/127.0.0.1
[13:20:37] <Glaisher>	 somethnig similar to overlordq's would be nice
[13:20:42] <a930913>	 Betacommand: https://en.wikipedia.org/wiki/Wikipedia:Database_reports/Unused_file_redirects/Configuration
[13:22:31] <Betacommand>	 a930913: what are you trying to do
[13:23:41] <a930913>	 Betacommand: Limit that query to those created within the $last_month.
[13:25:34] <a930913>	 Coren: What should my licensing be? :p
[13:25:37] <Carmela>	 a930913: Deleting unused file redirects is kind of silly.
[13:26:02] <Carmela>	 I'm not sure what the virtue of that database report is.
[13:26:06] <Carmela>	 a930913: Public domain!
[13:26:20] <a930913>	 Carmela: It's just cruft, no?
[13:26:25] <Coren>	 a930913: That's really a personal choice.  Most people either go GPL of some version or the 2-clause BSD, with dozen of options inbetween.
[13:26:55] <Carmela>	 a930913: Maybe.
[13:27:10] <Coren>	 Personally, I tend to slap a 2-clause BSD on stand alone stuff and GPL on large collaborative projects.
[13:27:27] <a930913>	 Coren: What licence are you going to assign all my works? :p
[13:27:40] <Carmela>	 I use public domain for most code. Copyright is evil.
[13:28:14] <a930913>	 Carmela: My code is essentially public domain, in that I have never specified any limitation :p
[13:28:31] <Carmela>	 That's not quite how it works.
[13:28:33] <Coren>	 a930913: I'm not going to assign any.  I'm talking with Legal about providing a default license for stuff that doesn't *have* an explicit one in Labs, but you can always override that default with licenses for any file, project or globally just by stating so.
[13:28:49] <Carmela>	 Works are assumed to be copyrighted unless explicitly marked otherwise.
[13:29:23] <a930913>	 Coren: I haven't explicitly licenced any :p
[13:29:24] <Carmela>	 In the U.S., anyway.
[13:29:37] <Coren>	 a930913: You should, regardless of whether we implement a default or not.
[13:29:42] <Carmela>	 ^
[13:29:56] <a930913>	 Coren: I know, but I can never decide which :)
[13:30:05] <a930913>	 So if you chose for me... :D
[13:30:10] <Coren>	 Heh.
[13:31:03] <a930913>	 Carmela: It's marked as open source, given the code to anybody who asked, with no limitations specified. So where does that put it?
[13:31:43] <Coren>	 a930913: Nowhere specific, and it puts reusers in trouble potentially.  :-)
[13:32:07] <Coren>	 a930913: If you really don't care what people do with your code, the simples is the 2-clause BSD:  http://opensource.org/licenses/BSD-2-Clause
[13:34:27] <Betacommand>	 a930913: http://pastebin.com/8TE87U8A
[13:35:21] <Betacommand>	 coren enwiki replication is stuck
[13:39:25] <a930913>	 Betacommand: <3
[13:40:50] <Betacommand>	 a930913: had to look up rc_cur_id not sure why its called that
[13:41:00] <Betacommand>	 rc_page_id
[13:41:09] <Betacommand>	 would be a better name
[13:42:24] * a930913 hands Betacommand a trout to slap somebody with.
[13:42:43] <Betacommand>	 a930913: I have enough of them
[13:42:59] <Betacommand>	 a930913: Remember when I used to and them out on AN/ANI?
[14:23:31] <wikibugs>	 3Wikimedia Labs / 3tools: Add some of the missing tables in commonswiki_f_p - 10https://bugzilla.wikimedia.org/59683#c14 (10Marc A. Pelletier) There's a complicated issue with schema variation between databases that is causing probems.  The one caused by rc_moved_to_ns is gone, I'm working on the other one n...
[14:37:36] <Betacommand>	 !log tools replication for enwiki is halted current lag is at 9876
[14:37:38] <labs-morebots>	 Logged the message, Master
[14:45:47] <YuviPanda>	 a930913: if you're looking for a license, I reccomend WTFPL
[14:48:04] <a930913>	 YuviPanda: :)
[15:12:39] <a930913>	 Hmm, [[toollabs:external_link_made_to_look_internal/?http://google.com|Google]] ?
[15:25:53] <Coren>	 YuviPanda: The WTFPL isn't (strangely enough) a proper open source license.
[15:29:07] <a930913>	 legoktm: Make ^ for {{noping}}?
[15:30:21] <a930913>	 Oh, noping got Lua'd.
[15:35:28] <a930913>	 Coren: Can a tool have a colon in its name?
[15:36:03] <Coren>	 a930913: No; [a-zA-Z_][a-zA-Z_0-9]*
[15:36:26] <a930913>	 Because [[toollabs:http://foo.bar]] -> /data/project/http:/public_html//foo.bar :p
[15:36:29] <Coren>	 Though I would recommend against using uppercase.
[15:37:18] <a930913>	 Dispenser: o/
[15:37:47] * a930913 mooos at Dispenser ;)
[15:38:00] <Dispenser>	 hi a930913 
[15:38:17] * a930913 sees what Dispenser did. :)
[15:38:50] <a930913>	 Dispenser: Can I see a photo of your 24TB array? :3
[15:41:34] <Dispenser>	 Only got 6 TB on my home computer and 81 % full :-(
[15:42:07] <a930913>	 Dispenser: So where is the other 19TB?
[15:42:41] <Dispenser>	 Not yet purchased and I might have to revise that 24 TB figure upward
[15:43:30] <a930913>	 Dispenser: I thought you had 24TB of data?
[15:44:10] <Dispenser>	 Sorry
[15:44:50] <Dispenser>	 The request was for 24 TB to improve Checklinks and Reflinks
[15:48:40] <a930913>	 Can I help you revise a plan, so it can be submitted to Coren to review, and hopefully approve? Because, while you might need 24TB of data one day, you don't actually need that much yet, do you?
[15:51:52] <Dispenser>	 Likely take 3 months to fill.  I'm sourcing numbers (likely deduplication and compression rates).  I'm thinking of going independent, already got Reflinks somewhat running in a VM.
[15:52:49] <Betacommand>	 Dispenser: how are you forwarding to your remote server from labs?
[15:52:58] <Dispenser>	 php script
[15:53:59] <Dispenser>	 There are other strategies and we could include a purge-non-essential data if Labs quickly need the space.
[15:56:39] <andrewbogott>	 aude: Do you know much about the wikidata-test instance?  Or can you refer me to someone who does?
[15:56:58] <aude>	 why?
[15:57:31] <andrewbogott>	 Two questions:  1)  is it OK if I force a refresh of the puppet config there?  and 2) does that box really need to be self-hosted puppet?
[15:57:45] <aude>	 1) maybe/probably
[15:57:48] <andrewbogott>	 2) is curiosity, not an action item
[15:57:49] <a930913>	 Dispenser: What about the legality of it? Running your own removes liability from wikimedia, but what's its status?
[15:57:57] <aude>	  2) need to make a role so it doesn't need to be self-hosted
[15:58:11] <aude>	 do you need me to pull latest puppet
[15:58:12] <aude>	 ?
[15:58:27] <andrewbogott>	 oh, it has custom patches now?  Paravoid just claimed that it didn't but I haven't verified for myself.
[15:58:38] <andrewbogott>	 Anyway, yes please, if you're able to fetch/rebase/rerun puppet
[15:58:40] <andrewbogott>	 that'd be great.
[15:58:44] <aude>	 they are on github at the moment
[15:59:14] <aude>	 idk if it makes sense maintaining them in wmf operations/puppet
[15:59:21] <andrewbogott>	 That box is running with a slightly broken exim config (which I don't understand the details of yet, but it should be easy to fix)
[15:59:25] <aude>	 ok
[15:59:51] <aude>	 we may want to use labsvagrant (probably) at some point
[15:59:55] <andrewbogott>	 it's totally fine for it to be self-hosted.  Just wondering if it was a mistake or an obsolete change.
[16:00:09] <aude>	 ok
[16:00:11] <aude>	 one sec
[16:00:28] <andrewbogott>	 aude: yeah, I'm trying to train myself to use labsvagrant for my labs MW installs.  Seems to work fine in most cases.
[16:00:44] <aude>	 we don't have vagrant for wikibase yet
[16:00:55] <aude>	 that's the issue but definitely would like to have it
[16:01:31] <andrewbogott>	 aude, if you don't mind, maybe update puppet on the other two as well?  Looks like wdjenkins and wdjenkins-node1 are self-hosted as well.
[16:01:42] <Dispenser>	 a930913: In American anyone can sue for any reason, valid and invalid.  There plenty of web crawlers on the net, standard practice is to respect robots.txt.  I imagine that most webmaster will be happy over some of the features I have planned.
[16:01:46] <aude>	 might have to do tomorrow
[16:01:56] <andrewbogott>	 aude:  ok, no rush on the other two
[16:01:58] <aude>	 how urgent?
[16:02:16] <aude>	 i might need tobi to do those, since i'm afraid there might be important stuff overwrittent
[16:02:25] <aude>	 and he's on holiday
[16:02:41] <andrewbogott>	 aude: that's fine -- you can do wikidata-test right now, right?
[16:02:51] <a930913>	 Coren: That's a point, google and archival sites cache this stuff, so why can't we?
[16:02:53] <aude>	 ok
[16:03:09] <aude>	 doing
[16:03:20] <andrewbogott>	 many thanks!
[16:03:53] <paravoid>	 thanks aude!
[16:04:27] <bd808>	 !log deployment-prep Updated scap to ff04431
[16:04:29] <labs-morebots>	 Logged the message, Master
[16:04:40] <aude>	 Error 400 on SERVER: Could not find class passwords::mysql::phabricator 
[16:04:45] <aude>	 what to do?
[16:04:54] <andrewbogott>	 huh.
[16:05:00] <andrewbogott>	 Oh, you need to rebase the private repo too
[16:05:03] <Coren>	 a930913: There are bazillion of caveats in how it's done and why.  Like I said, none of it is insurmountable, but it does need to be addressed beyond "need X terabytes".  Who has access, what are the archival criteria, who handles takedown notices (and how), etc.
[16:05:07] <andrewbogott>	 it's in /var/lib/git/labs/private I believe.
[16:05:12] <aude>	 k
[16:10:35] <a930913>	 Dispenser: Sounds quite reasonable, being an official body and all. I'm sure the community would approve of your application being fast tracked. What say you?
[16:11:32] <Dispenser>	 sounds good
[16:14:17] <a930913>	 Dispenser: So, moving forward, can we get the ball rolling at the end of the day?
[16:14:50] <Dispenser>	 Mine you I can't write proposals
[16:15:03] <aude>	 andrewbogott: done
[16:15:22] <aude>	 there's some stuff incompatible in our puppet code but can deal with it later
[16:15:34] <aude>	 and can probalby remove the self thing
[16:15:55] <a930913>	 Coren: What format of a proposal are you looking for?
[16:20:17] <Betacommand>	 a930913: Im already on that
[16:20:32] <a930913>	 Betacommand: On what?
[16:20:48] <Betacommand>	 the proposal
[16:21:37] <a930913>	 Betacommand: So what do we have so far?
[16:21:55] <Betacommand>	 Not much
[16:22:05] <Betacommand>	 waiting to hear from Dispenser 
[16:25:29] <andrewbogott>	 aude: actually, switching a host from self-hosted to master-hosted is… hard.  Like, I don't think anyone has ever done it without breaking the instance.
[16:25:47] <andrewbogott>	 So it's only a good option if you can safely build a fresh replacement.
[16:25:49] * a930913 stares intently at Dispenser.
[16:26:33] <aude>	 we can scrap it and make a new one
[16:26:46] <aude>	 our puppet stuff is designed to just work [tm]
[16:27:10] <aude>	 just not now and not tomorrw probably
[16:27:45] <andrewbogott>	 aude: sure; there's no pressing reason to do that, it's just good housekeeping.
[16:27:45] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Template search API request in Beta Labs finds nothing - 10https://bugzilla.wikimedia.org/66981#c4 (10Chad H.) 5NEW>3RESO/FIX Config error on beta labs, fixed in gerrit 143902.
[16:45:25] <Betacommand>	 !ping
[16:45:25] <wm-bot>	 !pong
[16:49:11] <Betacommand>	 Coren: enwiki replication has stopped
[16:49:31] <Coren>	 I'll go see (fully expecting catscan2 to be at it again)
[16:49:49] <Betacommand>	 looks like commonswiki is down too
[16:52:10] <Coren>	 transactions over 30ks killed; the DB should start catching up soon.
[16:53:38] <Coren>	 It may take a while though; what the catscan queries was blocking is a DDL query changing the schema and that pauses replication.
[16:55:13] <Coren>	 Ah, and I see the same 'alter table' in the queue for commons.
[16:55:56] * Coren expects the same will hold for a couple of other databases as well. Yeah schema changes.
[16:58:16] <Betacommand>	 Coren: those are the only two with major lag
[16:58:54] <Betacommand>	 !log tools Coren: transactions over 30ks killed; the DB should start catching up soon.
[16:58:56] <labs-morebots>	 Logged the message, Master
[16:59:11] <Betacommand>	 !log tools Coren: It may take a while though; what the catscan queries was blocking is a DDL query changing the schema and that pauses replication.
[16:59:12] <labs-morebots>	 Logged the message, Master
[17:00:05] <Coren>	 Betacommand: Possibly the schema changes already when through on the other databases; they're not as problematic when the DB is much smaller.
[17:11:13] <Coren>	 Dispenser: Dude, you can't just redirect from to Tool Labs tool to a server you own without an intersitial explaining that by proceeding forward users are giving you their IP and browser info.
[17:11:40] <Coren>	 https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use#If_my_tools_collect_Private_Information...
[17:11:49] <Coren>	 ^^ this has the message you have to display
[17:12:32] <Coren>	 Well, actually, you need to adapt it somewhat because it's not a Labs server but yours, but you get the point.
[17:14:03] <Dispenser>	 I'm not getting IP info
[17:14:17] <Dispenser>	 And are you saying hot linking is forbidden?
[17:14:45] <YuviPanda>	 in a way, yes.
[17:14:55] <YuviPanda>	 you can't use Google Analytics or Google Maps, for example. 
[17:15:07] <Dispenser>	 or linking to favicons
[17:15:09] <YuviPanda>	 or hot linking anything from 3rd party server (including yours?)
[17:15:24] <YuviPanda>	 that as well, assuming they are in 3rd party domains, directly from the client
[17:17:15] <Coren>	 Dispenser: You have a web server -> you are getting IP info (and logging it, by default).  So yeah, hotlinking is specifically forbidden because it discloses third parties' IPs.
[17:17:48] <Dispenser>	 Once a Labs tool, always a Labs tool?  Cuz' there no redirects out
[17:18:33] <Coren>	 Dispenser: You're not getting the point.  People follow a link to the Tool Labs, which has the WMF privacy policy, and they end up quietly disclosing their IPs in violation of that policy.
[17:19:13] <Dispenser>	 Once a Labs tool, always a Labs tool
[17:19:15] <Coren>	 Dispenser: You /can/ redirect users to a server you own, but you gotta warn them first.
[17:20:01] <Coren>	 (I.e.: beyond this point the WMF privacy policy doesn't apply and you're giving your IP to something not the WMF)
[17:21:04] <Dispenser>	 Well it a proxy, so unless everyone's IP address is 208.80.155.255 there shouldn't be a problem
[17:21:45] <Betacommand>	 Coren: I think because its all done back end and all the tool sees is the webserver address info
[17:22:03] <Coren>	 Ah, a proxy would be okay if there are no inline images, etc, that hit your server directly.
[17:22:23] <Coren>	 (I.e.: all relative paths, or rewriting).
[17:23:21] <Dispenser>	 So you're telling me my tools are boned anyway
[17:23:36] <Betacommand>	 Dispenser: no
[17:24:05] <Coren>	 No, I'm telling you that you can't quietly break the privacy policy without warning the editors that follow a link to your tool.  If that doesn't happen, then you're golden.
[17:24:33] <Dispenser>	 Because they hotlink favicon, open web pages in iFrames, use Google Charts API
[17:24:43] <Betacommand>	 Dispenser: as long as IP info is stripped before it reaches your server (which I think is already done) it shouldnt be an issue
[17:25:06] <Coren>	 There's a different, interresting question, about whether a tool that depends on a service you keep closed source on a third party server complies with the open source requirement, but I'm not interested in debating that one one.
[17:25:21] <Betacommand>	 Coren: Im in the process of writing that proposal, keep in mind its going to be rough and may lack some 
[17:25:49] <Coren>	 And if someone uses the Google Charts API in a way that the endusers' IPs end up at Google on project please tell me, because that's very very prohibited.
[17:25:57] <valhallasw>	 Dispenser: well, you could use wmflabs as a proxy, by getting the data from your server when the user requests a page on wmflabs
[17:26:03] <a930913>	 Coren: Sure, google is not open source, but I can still use their results can't I?
[17:26:30] <valhallasw>	 but the question is how much sense it makes to use a wmflabs domain in that case
[17:26:31] <Betacommand>	 specific details as those will be discovered as we develop the process
[17:26:31] <YuviPanda>	 a930913: depends on how you use them. if you're doing a serverside API call that is ok, but doing a *client* side call is nono, I think.
[17:26:33] <Coren>	 a930913: If you do it in a way that obeys /their/ TOS sure.  Just don't send the endusers there quietly.  :-)
[17:27:05] <Coren>	 Betacommand: Obviously; there's nothing wrong with prototypes and exploratory coding.  In fact, that's kinda expected.
[17:27:13] <valhallasw>	 Dispenser: and in the end, it's all allowed, as long as you show the user a warning before doing that
[17:27:24] <a930913>	 You have to think of Dispenser's tools as a large API :p HTTP request goes in, data comes out :)
[17:28:08] <Coren>	 valhallasw: Allowed but /discouraged/ unless you have to; but that's philosophical not technical.  But yeah, if you have to do things that disclose client info you gotta warn.
[17:29:44] <Dispenser>	 Coren: grep api.charts.google.com / -R :-)
[17:36:33] <Dispenser>	 whoops, grep chart.apis.google.com / -R
[18:01:21] <spagewmf>	 chrismcmahon et al: beta labs failure "http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Sandbox"
[18:01:33] <spagewmf>	 DB error Error: 1146 Table 'centralauth.renameuser_status' doesn't exist (10.68.16.193)
[18:01:47] <chrismcmahon>	 Coren: ^^
[18:02:17] <chrismcmahon>	 spagewmf: Coren there were some db changes being rolled out a little while ago I think?
[18:03:16] <a930913>	 Wait, if every external link /was/ cached, any deadlink could be replaced with the cached version :o
[18:03:19] <Coren>	 chrismcmahon: I've seen a couple schema changes go through recently; I don't know where in the process the beta labs lives.
[18:07:03] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): db error on beta labs "centralauth.renameuser_status' doesn't exist" - 10https://bugzilla.wikimedia.org/67485 (10Chris McMahon) 3NEW p:3Unprio s:3major a:3None Seen at http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Sandbox
[18:12:30] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): db error on beta labs "centralauth.renameuser_status' doesn't exist" - 10https://bugzilla.wikimedia.org/67485#c1 (10Tomasz W. Kozlowski) p:5Unprio>3Normal Also affecting http://commons.wikimedia.beta.wmflabs.org/      A database query error has occurred. This ma...
[18:37:50] <Betacommand>	 Coren: what address should I send the proposal to ?
[18:39:15] <Coren>	 Betacommand: It'd probably be even better on-wiki somewhere; easy to point people at and easy to comment.
[18:39:33] <Coren>	 Otherwise, you can mail it to me (marc@wikimedia.org)
[18:40:21] <Betacommand>	 Coren: sent
[18:41:07] <Coren>	 I'll see about finding a good on-wiki place for it later this afternoon.  You okay with it?
[18:42:10] <Betacommand>	 yeah
[18:42:22] <Betacommand>	 Its a rough draft
[18:42:37] <Coren>	 kk.  I'll post it and point the relevant people at it; I'm pretty sure many people in Engineering will offer refinements to it.
[18:43:10] <Betacommand>	 Coren: tools should really have there own wiki
[18:43:31] <Betacommand>	 merging to wiki tech is confusing as hell at times
[18:43:39] <Coren>	 Betacommand: It's been considered; just not beyond the "toying with the idea" phase yet.
[18:43:59] <Betacommand>	 and using a combination of meta and wikitech......
[18:44:32] <Coren>	 Betacommand: Also, this proposal is probably going to revive the archive.org discussion which, IIRC, kinda stalled because we didn't really have the resources to develop it fully back then.
[18:45:06] <Betacommand>	 Coren: this is basically doing just that
[18:45:28] <Betacommand>	 Along with a crapton of metadata harvesting
[18:45:52] <Coren>	 Yeah, but by using the WMF resources to host the mirror, which may or may not be the best scenario.  Restarting that discussion is worthwhile in itself IMO
[18:46:35] <Betacommand>	 Coren: it does solve a lot of issues though if we bring it in house
[18:47:16] <Coren>	 Potentially yes.  Personally, I like the idea, but it has impact and (opportunity) cost and requires consideration.
[18:48:08] <Betacommand>	 Coren: drop me an email with the link in case Im afk, Im going to be busy
[18:48:18] <Coren>	 kk
[18:59:38] <legoktm>	 !log deployment-prep manually created centralauth.renameuser_status table
[18:59:41] <labs-morebots>	 Logged the message, Master
[19:05:05] <Dispenser>	 If I'm correctly reading the rules, Google Analytics is allowed if the Private Information message is shown milliseconds before its loaded
[19:05:48] <valhallasw>	 oh god, techies trying to find 'smart ways to circumvent rules'
[19:06:28] <Betacommand>	 Dispenser: No
[19:09:39] <Dispenser>	 I never gotten a chance to use Google Analytics.  If I get my users to click "I agree to Google Analytics snooping on me" can I use it then?
[19:11:18] <valhallasw>	 Dispenser: Yes, that's the point of the rule.
[19:11:31] <Betacommand>	 Dispenser: yes if you notify them that their IP address is being exposed
[19:11:49] <harej>	 Dispenser, what are you using Google Analytics for?
[19:12:23] <Dispenser>	 Nothing yet!  But I'd sure like know what it provides!
[19:12:54] <Betacommand>	 Dispenser: not much
[19:13:33] <^d>	 I always enjoy when I visit a site and Ghostery shows me that I'm not contributing to people's google or facebook analytics :)
[19:15:53] <YuviPanda>	 ^d: :) me too.
[19:16:40] <^d>	 Also, I get a warm fuzzy feeling when it visits WMF wikis and shows that there's nothing to block :D
[19:17:33] <^d>	 wikia has 8 :(
[19:17:52] <Dispenser>	 I don't.  I get a disturbing feeling they're using something custom and more invasive
[19:18:33] <^d>	 Well, maybe $randomSite is.
[19:18:36] <^d>	 WMF wikis don't.
[19:18:58] <YuviPanda>	 yeah, we don't even have 'how many people have visited the site?' info
[19:19:16] <^d>	 We do, based on the aggregate squid logs.
[19:19:33] <^d>	 *varnish, whatev
[19:19:33] <YuviPanda>	 ^d: not very accurate, especially for mobile
[20:46:16] <wikibugs>	 3Wikimedia Labs / 3tools: Move wiki.toolserver.org to WMF - 10https://bugzilla.wikimedia.org/60220#c33 (10Krinkle) I too would recommend against setting up a new wiki. A wiki named "Toolserver" certainly wouldn't make sense as that as of late is no longer going to be operational. Perhaps set up its content a...
[20:47:25] <Nettrom>	 I'm getting an odd permission error when trying to copy a file from my tool's directory to my own, anyone around who could help with that?
[20:51:18] <scfc_de>	 Nettrom: What file are you trying to copy from where to where?
[20:52:06] <Nettrom>	 scfc_de: opentasks.py, from /data/project/suggestbot/projects/opentask to /home/nettrom
[20:52:58] <Nettrom>	 scfc_de: somehow I get a "Permission denied" error when trying to do that
[20:53:23] <scfc_de>	 Nettrom: Are you executing the copy as your user account or your tool account?  The former should work, the latter not.
[20:53:34] <Nettrom>	 scfc_de: as my user account
[20:53:52] <Nettrom>	 hmm... is the problem that I've used "become" to be the tool account first?
[20:53:56] <Nettrom>	 let me check
[20:54:14] <Nettrom>	 nope, still permission denied
[20:56:18] <scfc_de>	 Nettrom: Ah, okay, one moment.
[20:58:09] <scfc_de>	 So the permission error is that user nettrom can't read the file.  nettrom is member of tools.suggestbot and that group has read access for file and x rights for all directories leading up to it.  Strange.
[20:58:50] <Nettrom>	 scfc_de: that's what I found too and why I couldn't understand it
[21:00:18] <scfc_de>	 User nettrom can read /data/project/suggestbot/access.log ...
[21:00:55] <scfc_de>	 And /data/project/suggestbot/logs/inlink-update.err ...
[21:01:40] <scfc_de>	 And it can read the containing directory, so the perm problem should be with the file only?!
[21:02:40] <scfc_de>	 Nettrom: Ah, you went for "foo" for testing :-).
[21:02:40] <Nettrom>	 yep, I can create files in the directory
[21:02:46] <Nettrom>	 :D
[21:03:46] <Nettrom>	 and even if I take the 'foo' file I can edit it with the right permissions
[21:03:51] <Nettrom>	 must be something weird with this file
[21:04:21] <Nettrom>	 copy the file to a new filename and I can read that
[21:04:48] <scfc_de>	 Yep, stat for /data/project/suggestbot/projects/opentask/{opentasks.py,test} are the same (0664/51172/51172), yet it can only read test.  Perhaps NFS does some weird negative caching?
[21:05:33] <Nettrom>	 yeah, I have no idea... but looks like the copy is readable just fine, so I solved it that way
[21:05:33] <Nettrom>	 thanks for looking into it scfc_de !
[21:07:08] <scfc_de>	 Nettrom: No problem.  If the issue disappears in an hour or so, just drop a note here, then I think it might have been the NFS server caching some failure.  Otherwise I have no idea why.
[21:17:01] <wikibugs>	 3Wikimedia Labs / 3tools: Move wiki.toolserver.org to WMF - 10https://bugzilla.wikimedia.org/60220#c34 (10Tim Landscheidt) Eh, the intention was always to set it up as a public, (almost) read-only archive.  Of course it would describe obsolete infrastructure, just as https://wikimania2005.wikimedia.org/ is o...
[21:20:29] <andrewbogott>	 Coren: on labstore1001, is /usr/local/sbin/sync-exports a hotfix or is it part of a package or a manifest someplace?
[21:20:52] <Coren>	 ... it /should/ be in a manifest.
[21:21:32] <Coren>	 and it's clearly not.  I probably have the commit stashed in my repo.  @&#^
[21:22:04] <YuviPanda>	 :)
[21:23:08] <Coren>	 Yeah, there it is.  Hang on, lemme swap it out with my WIP and review
[21:24:26] * andrewbogott hangs on
[21:25:00] <andrewbogott>	 Coren:  Sorry to interrupt, I was just going to offer to puppetize it :)
[21:25:25] <Coren>	 Yeah, don't worry about it, I already have it stashed.  It's just a short rebase away.
[22:21:20] <chrismcmahon>	 Coren: seems like the beta labs db has been read-only for some time now, do you know an ETA for that to be over?
[22:21:40] <grrrit-wm>	 (03PS1) 10Yuvipanda: Move analytics product into #wikimedia-analytics [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/144075 
[22:21:48] <YuviPanda>	 milimetric: ^ thought I should let you know :)
[22:23:14] <Coren>	 No; I know of schema changes (possibly still in progress) but not their nature nor duration.  legoktm did/does them afaik, asking him might help?
[22:23:28] <Coren>	 chrismcmahon: ^^
[22:24:48] <chrismcmahon>	 thanks Coren 
[22:26:39] <YuviPanda>	 valhallasw: I don't know your gerrit id :| https://gerrit.wikimedia.org/r/#/c/144075/
[22:26:58] <YuviPanda>	 valhallasw: aha, foudn you
[22:27:01] <YuviPanda>	 added as reviewer
[22:39:12] <milimetric>	 oh, thanks for letting me know Yuvi, I wonder if the new noise will be annoying
[22:40:38] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032 V: 032] Move analytics product into #wikimedia-analytics [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/144075 (owner: 10Yuvipanda)
[22:41:09] <valhallasw>	 YuviPanda: ^ done
[22:41:09] <YuviPanda>	 milimetric: you'll find out soon enough :) Am just cleaning up -dev, since you guys seem to be using it for stories as well
[22:41:30] <YuviPanda>	 milimetric: and there's no analytics people on -dev anyway, and people on -dev don't know much about that either
[22:41:37] * valhallasw is off to bed
[22:41:40] <YuviPanda>	 valhallasw: ty
[22:41:47] <valhallasw>	 but YuviPanda, you have +2 rights there, right?
[22:42:28] <valhallasw>	 merging simple stuff is fine with me, as long as you fix it when it breaks :-p
[22:42:44] <YuviPanda>	 valhallasw: yeah :)
[22:42:47] * valhallasw is off to bed
[22:43:15] <milimetric>	 YuviPanda: no, totally fair, I was feeling bad about the noise there, but yeah, we're using Bugzilla for Everything now, it's crazy
[22:43:39] <YuviPanda>	 milimetric: :) yeah. we can suppress things easily based on conditionals if you want. if you see the patch, it's just a plain python lambda
[22:44:12] <milimetric>	 yep, i like it
[22:44:14] <milimetric>	 thanks again
[22:44:53] <YuviPanda>	 milimetric: :) yw!
[22:45:02] <YuviPanda>	 milimetric: feel free to poke me or valhallas.w for review/merge/deploys
[22:47:08] <milimetric>	 lol stop pinging the poor guy he's been trying to get to sleep for like 6 minutes
[22:48:37] <YuviPanda>	 milimetric: I didn't ping him! I put a . there, to not ping :)
[23:28:40] <Dispenser>	 I love how the upstream proxy hijacks my custom 404 and 503 error messages
[23:34:17] <legoktm>	 chrismcmahon: errr, my schema change took about half a second, I was just adding a table. I don't think that's caused by me...
[23:36:46] <chrismcmahon>	 legoktm: bummer.  thanks for the notice
[23:51:11] <phe>	 sge can't start job, I got:
[23:51:16] <phe>	 Unable to run job: error writing object "2054684" to spooling database
[23:51:16] <phe>	 aborting transaction (rollback)
[23:51:17] <phe>	 job 2054684 was rejected cause it couldn't be written.
[23:51:17] <phe>	 Exiting.
[23:55:47] <gifti>	 mh, i have the same