[00:14:45] Who here is good with SQL? I'm terrible at it, and I'd like to write a query to make use of the database on Tool Labs [00:15:41] shoot the description [00:48:29] Danny_B, I want to get every file which is in two separate categories [00:49:46] you mean union or intersection of them? [00:57:27] Danny_B, intersection [01:04:12] Danny_B, I am away, but if you get back and are in the mood to provide answers, feel free to send it to me on the channel or privately. [01:05:49] Magog_the_Ogre: you around? I know both an easy way and a hard way [01:05:56] yeah I'm still here [01:06:19] Magog_the_Ogre: the easy way is to make two queries and do the intersection in your code [01:06:19] is there a reason to do it the hard way? [01:06:33] yeah good point [01:06:35] Magog_the_Ogre: yes [01:06:59] otherwise you could write a single sql query, it will be a lot slower and a lot more complex [01:07:26] slower? I thought it would be faster. [01:07:30] I am terrible with databases [01:07:32] I do a ~6 million comparison daily and it takes less than 10 seconds in code [01:08:19] I am so bad with databases that literally none of my code uses it. I do everything through API queries. [01:08:44] Magog_the_Ogre: depends on the programming language. In python list (or set ) operations (union, intersections and subtractions/additions ) are dam fast [01:09:18] well the one category is 13000 files [01:09:19] Magog_the_Ogre: if you want some help rewriting them I can give a hand. I do my stuff in python and I can do a whole lot with the right queries [01:09:45] Magog_the_Ogre: the comparisons I do daily involve categories in the millions [01:09:50] I suppose I can do a max query on that for categories on each file. Shouldn't take more than a minute of requests. [01:10:11] Magog_the_Ogre: lets do it via the database, only takes 1-2 seconds [01:10:18] OK [01:10:26] Magog_the_Ogre: what language do you code in? [01:10:32] I will have to write a new class DataBaseQuery{} [01:10:33] PHP [01:10:48] ouch, I cant help there, but i can provide the query [01:11:04] give me the query, I'll RTFM on how to do it in PHP [01:12:06] select page_namespace, page_title from categorylinks left join page on cl_from = page_id where cl_to ="CAT_NAME"; [01:12:24] that is a basic query for getting the contents of a category [01:12:44] returns all pages in a namesapce, title tuple [01:13:16] use _ instead of spaces in the category name [01:14:16] um, Betacommand that is only one category [01:14:30] Magog_the_Ogre: http://tools.wmflabs.org/betacommand-dev/reports/CATCSD.html is the basic results of "select page_namespace, page_title, cl_timestamp from categorylinks left join page on cl_from = page_id where cl_to ="Candidates_for_speedy_deletion;"" [01:14:54] or is two categories too hard? [01:15:02] Magog_the_Ogre: yeah get the results from each query and then do a list set on them. Let me look up the PHP code [01:15:14] Magog_the_Ogre: I guess you want the hard way then :P [01:15:29] No, I can do it the easy way [01:16:07] * Betacommand points to PHP docs "array array_intersect ( array $array1 , array $array2 [, array $... ] )" [01:16:24] 3 seconds for 130000 rows. That is like a million times faster than the API [01:16:37] Magog_the_Ogre: yeah I know [01:17:16] Magog_the_Ogre: then do it on the second, store both in arrays, and use the above function to get the results [01:17:23] now, maybe Betacommand can magic up a way to do the queries in PHP [01:18:02] I can't find the documentation :( [01:19:46] Magog_the_Ogre: https://wiki.toolserver.org/view/Database_access#PHP [01:19:57] Magog_the_Ogre: http://php.net/manual/en/function.array-intersect.php [01:20:20] Magog_the_Ogre: And Im not a PHP programmer. :P [01:20:34] oh I know how to intersect [01:20:48] to be fair, neither am I. I taught myself for the sole purposes of writing Wikimedia bots. [01:21:01] Magog_the_Ogre: why???? [01:21:17] python is a much more robust/supported framework [01:22:03] Peachy was written in PHP, and it was the first pre-written framework I found. [01:23:05] Magog_the_Ogre: ouch, pywikipedia /pywiki is what I use/recommend [01:23:44] that is all water under the bridge [01:24:12] Magog_the_Ogre: If you ever want to migrate I can give a hand and provide far more support [01:25:07] my code is massive and poorly written. [01:25:33] which is the product of the fact that I am a very good and fast programmer and had absolutely no real world experience when I started. [01:25:58] Magog_the_Ogre: Like I said I can help port/improve the code [01:26:29] does anyone have documentation on my.cnf? [01:26:45] Magog_the_Ogre: Ill provide what help I can in PHP though it will very limited [01:26:57] Magog_the_Ogre: its a standard linux config file [01:27:14] OK [01:27:27] Magog_the_Ogre: http://dev.mysql.com/doc/refman/5.1/en/option-files.html [01:28:57] what do I use for the username and password? [01:29:33] Magog_the_Ogre: just read the file [01:29:52] Magog_the_Ogre: $ts_mycnf = parse_ini_file($ts_pw['dir'] . "/.my.cnf") [01:30:00] that reads the needed config data [01:30:09] but I have to create .my.cnf [01:30:24] Magog_the_Ogre: it should already exist in the home of your tool [01:30:32] it totally doesn't [01:30:59] Magog_the_Ogre: note there is a . before the my [01:31:11] I noted it, and it's not there [01:31:20] Magog_the_Ogre: what the tool name? [01:31:29] magog [01:31:39] unless you have admin access, you won't see it anyway [01:31:44] https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver [01:31:51] New tools don't get the .my.cnf file, but the replica credentials will work. [01:32:13] I have to use the replica.my.cnf [01:32:43] ah [01:33:03] Magog_the_Ogre: I can see the directory listing [01:33:15] Magog_the_Ogre: I was using the toolserver docs [01:33:20] servers me right [01:52:16] does anyone know what is the connection parameter for the database in SQL? is it commonswiki? [02:27:04] Magog_the_Ogre: commonswiki.labsdb probably [04:34:22] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks - 10https://bugzilla.wikimedia.org/69182#c1 (10Sean Pringle) labsdb1001 has now been switched onto SSD and labsdb1002 is in progress. Have also set tokudb_empty_scan=disabled [1] which... [04:34:53] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks - 10https://bugzilla.wikimedia.org/69182 (10Sean Pringle) a:3Sean Pringle [07:53:37] !log integration pointing integration-slave1006-trusty to local puppetmaster and installing Jenkins related material on it [09:18:50] !log integration polled integration-slave1006-trusty (4 CPUs). Deleted integration-slave1004-trusty (2 CPUs) [11:09:35] !ping [11:09:35] !pong [11:13:47] yuvipanda: pooooooooooong [11:14:02] enough plagiarism for today, time to walk around :-) [11:15:05] valhallasw`mania: :D [11:15:19] https://github.com/valhallasw/plagiabot/blob/master/README.md SO COOL [11:15:23] Coren_WM2014: ^ [11:33:08] valhallasw`mania: turnitin.com is evil, but that looks super awesome [11:45:35] ori: is that stream thing for beta working? [11:45:53] I am trying it now but it's silent [11:47:06] petan: that's because no-one edits beta :-p [11:47:15] I did [11:47:25] so this argument is moot [11:47:49] petan: in that case, let me say 'it worked for me yesterday' [11:48:03] yesterday whole beta was down [11:48:27] do you just want to annoy people or do you want your problem fixed? [11:48:33] ?? [11:48:38] are you kidding me? of course I want it fixed [11:49:29] DEBUG:socketIO_client.transports:[packet received] 5::/rc:{"args":[{"comment":"[[WP:AES|\u2190]]Created page with [11:49:29] 'rctest'","wiki":"enwiki","server_name":"en.wikipedia.beta.wmflabs.org","title":"User:Valhallasw2","timestamp":1407412158,"server_script_path":"/w","namespace":2,"server_url":"http://en.wikipedia.beta.wmflabs.org","length":{"new":6,"old":null},"user":"Valhallasw2","bot":false,"patrolled":false,"type":"new","id":103735,"minor":false,"revision":{"new":118878," [11:49:29] old":null}}],"name":"change"} [11:49:29] Valhallasw2 edited User:Valhallasw2 [11:49:32] works for me. [11:49:50] python test.py [11:49:51] No handlers could be found for logger "socketIO_client" [11:49:55] doesn't work for me [11:51:18] petan: my crystal ball is broken. [11:51:30] yes, that is why I asked ori [11:51:44] I'm telling you it's not the server, /because it's working for me/ [11:52:00] so yes, the stream thing is working. [11:52:24] why /the code you're using/ is not working, I don't know, because you're telling us nothing about that code, or what it does [11:52:46] this is my code: https://wikitech.wikimedia.org/wiki/RCStream#Python [11:53:02] instead of commons I have * there [11:53:10] and you changed stream.wikimedia.org to stream.wmflabs.org? [11:54:01] if you add [11:54:03] import logging [11:54:03] logging.basicConfig(level='DEBUG') [11:54:23] yes I did [11:54:26] just after import socketIO_client, you'll get a lot more information [11:55:04] petrb@huggle:/tmp$ python test.py [11:55:05] INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): stream.wmflabs.org [11:55:59] let me guess: you're trying to connect from within labs? [11:56:04] yes [11:56:05] try using the internal IP instead, then [11:56:22] which one is it? [11:56:25] I don't know name of instance [11:56:36] @labs-project-instances deployment-prep [11:56:36] Following instances are in this project: deployment-apache02, deployment-cache-bits01, deployment-cache-mobile03, deployment-memc02, deployment-memc03, deployment-cache-text02, deployment-parsoid04, deployment-eventlogging02, deployment-bastion, deployment-apache01, deployment-salt, deployment-logstash1, deployment-parsoidcache01, deployment-redis01, deployment-upload, deployment-fluoride, deployment-db1, deployment-jobrunner01, deployment-videoscaler01, deployment-elastic01, deployment-elastic02, deployment-elastic03, deployment-elastic04, deployment-analytics01, deployment-cache-upload02, deployment-graphite, deployment-rsync01, deployment-memc04, deployment-memc05, deployment-db2, deployment-stream, deployment-pdf01, deployment-cxserver01, deployment-mediawiki01, deployment-mediawiki02, deployment-sentry2, deployment-sandbox, deployment-soa-cache01, deployment-mathoid, [11:57:42] https://wikitech.wikimedia.org/wiki/Stream.wikimedia.org [11:57:58] oh, wait, that's the live LB stuff [11:58:30] hey valhallasw`mania, petan [11:58:43] ori: what is IP of wmflabs instance for that strem thing [11:58:53] there is some bug i haven't had the chance to chase down that causes the service to be unreliable sometimes [11:59:02] so it's not insane for it to be working for valhallasw`mania but not working for petan [11:59:09] doesn't necessarily mean that petan is doing something wrong [11:59:13] ori: the beta rcstream is pretty stable, it's just the deployment one that's broken [11:59:17] I thik problem is that it just can't connect there [11:59:25] I use it from labs [11:59:28] using external IP [11:59:29] the issue here seems to be the standard NAT problems in labs [12:00:51] valhallasw`mania: are you at wikimania, by the way? (going by your nick) [12:01:08] i literally just arrived in london a couple of hours ago [12:01:15] yep. I'm at the central plaza in the greenhouse [12:01:16] if you want to hack on this together i'd be happy to :) [12:01:22] cool! [12:01:54] i'm going to find something to eat first but i'll try to find you afterwards [12:02:18] there's also lunch at the venue [12:02:32] although you're supposed to have a hackathon ticket for that, I think :-p [12:03:20] petan: i'm going to make it my personal goal to get the rcstream to a state where you feel good about it :P [12:03:28] petan-approved™ [12:03:42] I would feel good about it if I knew the name of instance where it run [12:03:56] i think it's delployment-stream? [12:03:58] or something like that [12:04:02] it's in the deployment-prep project [12:04:13] @labs-instance-info deployment-stream [12:04:22] @labs-instance deployment-stream [12:04:24] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 2.6.0.2 [libirc v. 1.0.1] my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [12:04:24] @help [12:04:43] @labs-info deployment-stream [12:04:43] [Name deployment-stream doesn't exist but resolves to I-00000388.eqiad.wmflabs] I-00000388.eqiad.wmflabs is Nova Instance with name: deployment-stream, host: virt1004, IP: {unknown} of type: m1.medium, with number of CPUs: 2, RAM of this size: 4096M, member of project: deployment-prep, size of storage: 40 and with image ID: ubuntu-14.04-trusty (testing) [12:04:55] IP unknown o.O [12:05:19] @labs-info deployment-rcstream [12:05:19] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 1 seconds ago [12:05:31] !deployment-stream is IP: deployment-stream [12:05:31] Key was added [12:05:33] omg [12:05:37] !deployment-stream del IP: deployment-stream [12:05:38] Successfully removed deployment-stream [12:05:51] !deployment-stream is IP: 10.68.17.106 [12:05:51] Key was added [12:55:32] !log deployment-prep Cleared git modules under deployment-bastion: /srv/scap-stage-dir/php-master/skins That broke https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ All changes losts [14:42:46] !log deployment-prep `git reset --hard origin/master` for srv/scap-stage-dir/php-master and srv/scap-stage-dir/php-master/skins [15:47:20] I need a new IP address for the services project [15:48:03] * gwicke tries to find Coren_WM2014 IRL [15:50:20] Coren_WM2014: I am back from my badly-timed vacation; anything happening labswise that needs immediate attention? [15:52:18] gwicke: use proxy! [15:52:57] gwicke: if it's just web access then the proxy will work fine. If other services, let me know and I'll raise your quota. [15:59:17] I'm in shell group but I'm not member of bastion project: https://wikitech.wikimedia.org/wiki/Shell_Request/Stemd [16:06:27] stemdA: looking... [16:06:33] stemdA: what is your username on wikitech? [16:06:37] stemd [16:06:39] Oh, nm, I guess it's clearly Stemd :) [16:06:40] :) [16:07:37] stemdA: how long ago did you create your account on wikitech? [16:07:44] (Just curious for bug-hunting purposes, fixing your account is easy.) [16:08:01] during/on Wikimania in Washington 2012 [16:08:21] and never had time to go further... [16:08:27] Ah, that's OK then, this wasn't automated back then. [16:08:31] Should be fixed now. [16:08:41] I'm going to mark your request as resolved. [16:10:56] thx, I've not tested as I'm now on different comp, but I suppose it's OK now ;) [16:13:34] stemdA: Yeah, probably you'll have to fight with keys and ssh a bit, but it's pretty well documented here: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [16:14:09] andrewbogott: just setting ~/.ssh/config [16:17:58] addshore: highlight! [16:18:01] addshore: highlight! [16:18:10] pong valhallasw`mania [16:18:13] (sorry, I can see the popups from here) [16:23:01] oh wait, valhallasw`mania your here? :P [16:23:18] yuvipanda: which methods does the proxy support? [16:24:39] I need at least PUT and DELETE [16:26:00] /cc andrewbogott [16:26:05] gwicke: it should just work, it relays the whole command. [16:26:16] Anyway, it's easy enough to try :) [16:27:19] Syntax: proxy_cache_methods GET | HEAD | POST ...; [16:27:20] Default: [16:27:22] proxy_cache_methods GET HEAD; [16:27:25] according to http://nginx.org/en/docs/http/ngx_http_proxy_module.html [16:28:24] gwicke: that's for the cache [16:28:26] gwicke: this is not a caching proxy [16:28:30] gwicke: and I know that POST works, and I think DELETE and PUT should too [16:28:30] gwicke: try [16:28:38] ah, okay [16:28:58] I'll give it a try, thx! [16:29:08] gwicke: \o/ yw [16:29:14] ori: stream.wmflabs.org resolves to 208.80.155.138 which manifests/openstack.pp maps to deployment-eventlogging02 = 10.68.16.52 for internal Labs DNS requests. Above it was said that deployment-stream = 10.68.17.106 was the receiving end for stream.wmflabs.org. Is that correct? [16:50:47] Someone else around who could look at https://wikitech.wikimedia.org/wiki/Special:NovaAddress for Deployment-prep? [16:52:27] scfc_de: Is your question about what that page looks like, or what that page /should/ look like? [16:57:01] andrewbogott: I want to compare the page's internal/external IPs for Deployment-prep to manifests/openstack.pp, in particular what internal IP the external IP for stream.wmflabs.org (208.80.155.138) is directed to. [16:57:55] stream.wmflabs.org = 208.80.155.138 (as you say) which is bound to instance deployment-stream [16:58:09] which is at 10.68.17.106 [16:58:13] that's what you need? [16:58:20] andrewbogott: Exactly. Thanks! [16:59:04] andrewbogott: Is there any entry for deployment-eventlogging02? [16:59:57] Looks like not [17:00:22] btw, in theory the resource page for a given instance displays the public IPs bound to it. In practice, that only works some of the time. [17:00:33] But it's correct for deployment-stream [17:01:47] Ah! Forgot about that. I think you can even query for that in SMW. [17:02:18] At the moment there's no callback for changes to IP. So that page is only updates when e.g. the instance reboots. [17:02:27] It's a long-standing bug on my list... [17:58:29] andrewbogott: Can you look at my project request? [17:59:08] Everybody has been gone because of Wikimania. [18:00:09] petan: ^ you around? can you help with this? :) [18:00:21] andrewbogott: Nevermind. Happening. [18:00:29] Negative24: granted -- please fill in the docs here https://wikitech.wikimedia.org/wiki/Nova_Resource:Performance and clean up your old tool if necessary. [18:00:42] andrewbogott: Will do. Thanks. [18:01:08] sorry for noise [18:01:37] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks - 10https://bugzilla.wikimedia.org/69182#c2 (10metatron) While normal queries now perform greatly with SSD, there's still no cure for that problem. I removed every fancy stuff like GRO... [18:31:50] legoktm: is possible to use mwparserfromhell to access template parameters positionally? [18:32:01] yes [18:32:41] legoktm: https://pythonhosted.org/mwparserfromhell/api/mwparserfromhell.nodes.html#module-mwparserfromhell.nodes.template here seems necessary a name in get(name). How can I access the mpositionally? [18:32:47] get(1) [18:32:54] '1' works as well I think [18:33:05] legoktm: thankyou :) [18:33:27] :) [18:58:36] I just created an instance but it still says "BUILD (scheduling)" and nothing is showing up in the console output. Is this normal? [19:05:08] Negative24: it takes a few minutes… what instance and project? [19:06:07] Instance name: performance-testing, Project: performance, Instance ID: i-0000050c.eqiad.wmflabs [19:06:34] ah, of course -- short attention span, sorry :) [19:06:41] :) [19:08:36] I created it at 12:45 MDT which was more than 25 min. ago [19:09:44] Negative24: something seems wrong, I'm investigating [19:13:27] Negative24: I'm not sure what's happening, but probably if you try again it will work. I'm going to delete that instance and let you retry... [19:13:34] and then I'll investigate in a place that won't bother you. [19:13:45] Sure. [19:14:28] andrewbogott: Wait. Now there are 3 other instances. [19:14:38] yep, that's me experimenting [19:15:13] Okay. Retrying... [19:15:44] Sent the instance for creation... [19:16:39] w00t! Instance i-0000050f.eqiad.wmflabs now "ACTIVE" [19:16:57] Negative24: ok. ssh won't work until it finishes the initial puppet run, maybe 5 minutes. [19:17:17] Yep. Receiving console output... [19:17:52] One time fluke? [19:21:39] It might be that one of the VM hosts is messed up, I'm going to test. [19:23:59] andrewbogott: I'm not seeing any more console messages. It stopped at '[0;36mnotice: Did not receive certificate[0m' [19:24:44] Negative24: I can log in just fine, have you tried? [19:25:26] I haven't tried yet. [19:25:32] I guess it is done. [19:33:45] Negative24: well, I started a new instance on that same host (virt1008) and it worked fine. So that failed instance remains a mystery… let me know if it happens again. [21:23:56] 3Wikimedia Labs / 3deployment-prep (beta): Make the password for logstash-beta.wmflabs.org available to all users with sudo - 10https://bugzilla.wikimedia.org/69267 (10Bryan Davis) 3NEW p:3Unprio s:3enhanc a:3None All users who have access to the logs on deployment-bastion via tailing log files shoul... [21:43:09] 3Wikimedia Labs / 3deployment-prep (beta): Give all members of the deployment-prep project sudo - 10https://bugzilla.wikimedia.org/69269 (10Bryan Davis) 3NEW p:3Unprio s:3normal a:3None In a recent irc conversation {{citation needed}} it was determined that the requirement for people to have signed a... [21:43:22] 3Wikimedia Labs / 3deployment-prep (beta): Give all members of the deployment-prep project sudo - 10https://bugzilla.wikimedia.org/69269 (10Bryan Davis) [21:43:38] 3Wikimedia Labs / 3Infrastructure: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - 10https://bugzilla.wikimedia.org/48501 (10Bryan Davis) [23:56:22] 3Wikimedia Labs / 3deployment-prep (beta): Beta cluster job queue not running - 10https://bugzilla.wikimedia.org/69272#c1 (10Kunal Mehta (Legoktm)) legoktm@deployment-bastion:~$ mwscript showJobs.php --wiki=enwiki --group htmlCacheUpdate: 74 queued; 0 claimed (0 active, 0 abandoned); 0 delayed enotifNotify:...