[00:03:01] 3Tool Labs tools / 3Quentinv57's tools: Empty query in /data/project/quentinv57-tools/public_html/tools/sulin fo.php on line 207 - 10https://bugzilla.wikimedia.org/70874 (10Technical 13) 3NEW p:3Unprio s:3normal a:3None SUL Info Warning: mysqli::query(): Empty query in /data/project/quentinv57-tools... [00:20:34] andrewbogott: what you mean by relocate? [00:21:09] The server that hosts those instances is suspect -- we're clearing it off to rebuild. [00:21:44] i was sayin that from the beginning ;-) [00:21:58] let me check [00:22:01] Moving them requires a brief shutdown while I copy the disks over. [00:22:54] jlocal spam? [00:24:21] you can move the large right now, the other one in a moment [00:27:05] spam spam spam [00:36:52] Danny_B: dannyb-large is done now, can I move 'dannyb'? [00:37:10] in a minute, will let you know [00:37:13] ok [00:38:53] seriously where is all this spam coming from? [00:39:01] andrewbogott: you seeing it? [00:39:47] jeremyb: my mailer was closed. checking... [00:40:04] !log multimedia Added Dduvall as a project admin [00:40:06] Logged the message, Master [00:40:14] marxarelli: ^ {{done}} [00:40:28] bd808: excellent [00:41:18] jeremyb: I see the spam, seems like it's all from one user [00:41:32] hm, no, nevermind, I take that back. [00:41:44] why did it just start though? [00:41:46] coren, is this spam flood possibly a result of rebooting tools-submit? [00:42:13] spam flood? I don't see it. Where? [00:42:55] So far about 20 of these: https://dpaste.de/aoXZ [00:43:18] Och! My fault! (But why am I not getting them?) [00:43:44] Coren: remember spambucket? :) [00:44:04] Yeah, just found them. Sorry about this -- that was my bad. Fixed now (I think) [00:44:31] * Coren tries to remember to not enable verbose debugging on a script being run in cron. [00:44:47] hah [00:45:23] andrewbogott: shoot [00:45:34] Also, remove rlane's past email from there. [00:54:06] Danny_B|webchat: ok, all done. Thanks for accomodating. [00:55:12] no prob thanks for care [00:58:44] 3Tool Labs tools / 3Quentinv57's tools: Empty query in /data/project/quentinv57-tools/public_html/tools/sulinfo.php on line 207 - 10https://bugzilla.wikimedia.org/70874#c1 (10Cometstyles) Luxo's GUC tool is failing to search IP contributions as well...may be related.. http://tools.wmflabs.org/guc/index.php <... [01:20:55] Coren: andrewbogott_afk: lots of spam in syslog (but not new): nslcd[1241]: [b58ea4] error writing to client: Broken pipe [01:21:00] can we do something about that? [01:58:44] 3Tool Labs tools / 3Quentinv57's tools: Empty query in /data/project/quentinv57-tools/public_html/tools/sulinfo.php on line 207 - 10https://bugzilla.wikimedia.org/70874 (10Technical 13) p:5Unprio>3High [02:15:38] (03PS1) 10coren: Tool labs: The new and improved tool list! [labs/toollabs] - 10https://gerrit.wikimedia.org/r/160590 [02:16:22] (03CR) 10coren: [C: 032] "Yeay progress!" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/160590 (owner: 10coren) [02:16:31] (03CR) 10coren: [V: 032] "Yeay progress!" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/160590 (owner: 10coren) [02:22:42] "more smartest" O_O [02:29:28] Yes. Is more smartest now. This many! [02:46:32] !log deployment-prep updated OCG to version 188a3c221d927bd0601ef5e1b0c0f4a9d1cdbd31 [02:46:37] Logged the message, Master [04:17:56] Magnus is ridiculously prolific. [04:39:37] (03PS1) 10coren: Tool Labs: more tweaks to list.php [labs/toollabs] - 10https://gerrit.wikimedia.org/r/160606 [04:48:25] too bad he refuses to fix/upgrade his tools :\ [04:54:00] warpath: Most point to the source: https://tools.wmflabs.org/?list#toollist-magnustools [04:59:07] speaking of labs, can you find out why all of a sudden today, all SUL related tools stopped working? [04:59:21] a change in api somewhere? [05:18:33] andrewbogott_afk: probably kartik or runa are using those [05:31:02] andrewbogott_afk: I was told last two are safe to reboot, first one probably too [05:48:29] 3Wikimedia Labs / 3deployment-prep (beta): deployment-salt can't talk to itself, git deploy hangs - 10https://bugzilla.wikimedia.org/70868#c3 (10jeremyb) 5NEW>3RESO/FIX 2 prerequisites before booting salt-minion: * kill all the existing salt-minion/grain-ensure/salt-call/etc. procs * make sure /etc/salt... [05:49:52] bug 70076 is happening again [05:56:14] The DPL extension doesn't scale to big wikis. Does building a similar tool at WMLabs have any merit? I.e. is it slow because the db is big (in which case no), or because the db is used by the wiki actively (in which case running such tool on WMLabs has some merit), or because an extension query resources are limited by the wiki software artificially to avoid high load on the db (in which case again yes)? [05:59:45] 3Wikimedia Labs / 3tools: watchlist table not available on labs - 10https://bugzilla.wikimedia.org/57617 (10Dispenser) [05:59:45] 3Tool Labs tools / 3[other]: Migrate http://toolserver.org/~dispenser/* to Tool Labs - 10https://bugzilla.wikimedia.org/66868 (10Dispenser) [05:59:50] Define "big"? [05:59:58] It's in use on wikis with millions pages [06:02:40] Nemo_bis: en.wp refused to install it for this reason I believe. [06:04:25] Also, if I'd like to patrol a category at en.wp, I need to also look through subcats. Does DPL do subcats? [06:05:02] I have "fresh category members" thought in mind for new page patrol at English Wikipedia, but I'm not sure how to best approach it without JavaScript. [06:15:29] No it doesn't [07:35:10] Coren: I have a single toolinfo.cgi for all my tools :/ [07:37:44] 3Wikimedia Labs / 3deployment-prep (beta): Search is sometimes slow - 10https://bugzilla.wikimedia.org/70869#c4 (10Antoine "hashar" Musso) Created attachment 16480 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16480&action=edit Elastic search instances load average [07:42:14] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862#c1 (10Antoine "hashar" Musso) Yuvi, I am not sure how familiar you are with diamond. Would it make sense to write a basic collector that list the rejected/unsigned keys on the salt master, send that... [07:44:59] 3Wikimedia Labs / 3deployment-prep (beta): Setup monitoring for Beta cluster - 10https://bugzilla.wikimedia.org/51497#c8 (10Antoine "hashar" Musso) Thank you Yuvi for the monitoring! Do we have a way to tweak the body of email notifications? I find them hard to read :-D [07:49:07] Nemo_bis, context: https://www.mediawiki.org/wiki/Thread:Extension_talk:DynamicPageList_%28Wikimedia%29/Performance_concerns_regarding_the_Intersection_extension [07:52:43] Svetlana: there are 1063 categories with queries that show up in Reasonator. They are all categories about people and who is in them based on what the category is about [07:52:55] http://tools.wmflabs.org/wikidata-todo/autolist.html?q=CLAIM%5B31%3A4167836%5D%20AND%20CLAIM%5B360%3A5%5D%20 [07:53:10] Click on the reasonator icon for best results [07:53:53] I didn't parse what you said. What is Reasonator? Does it mean that some wikis already use Wikidata for categories? [07:53:54] obviously they are project agnostic [07:54:27] Reasonator is a tool that makes information out of the Wikidata data [07:55:05] reasonator uses the gear like icon [07:57:17] What you will find is that Wikidata / Reasonator knows about items Wikipedias do not know about in their categories [07:58:01] I regularly add statements that are reflected in the implied queries [07:59:00] one problem with categories is that not everything is of the same instance / subclass [08:01:31] wikidata does not have that problem [08:03:44] say I create page [[svetlana]] on english wikipedia and add it to [[category:dummy]] -- how does it reflect on the reasonator tool? [08:10:14] 3Wikimedia Labs / 3deployment-prep (beta): deployment-salt can't talk to itself, git deploy hangs - 10https://bugzilla.wikimedia.org/70868#c4 (10Antoine "hashar" Musso) all are now salty except: * deployment-saio * deployment-parsoidcache02 * deployment-soa-cache01 Those instances haven't been migrated to t... [08:10:22] GerardM-, ^ [08:31:19] Eeeeeeek. People, say something I don't understand, and then vanish!? [08:34:58] 3Wikimedia Labs / 3tools: androidsdk::dependencies doesn't work on Trusty - 10https://bugzilla.wikimedia.org/69423#c3 (10Antoine "hashar" Musso) p:5Unprio>3Low Low priority, we no more build any Android app. Can be revisited later on. [08:38:35] could we please have wikibugs use /notice instead of /msg to this channel? i keep thinking humans are talking (/notice is a lower activity level in my irc client) [09:08:27] Svetlana: fill a bug [09:08:30] please :] [09:09:08] Svetlana: the bug component is under https://bugzilla.wikimedia.org/enter_bug.cgi?product=Tool%20Labs%20tools [09:09:29] look for "wikibugs IRC bot". That will CC the wikibugs authors/maintainers [09:13:15] Bug 70881 [09:13:16] 3Tool Labs tools / 3wikibugs IRC bot: Bot should use notices in-channel - 10https://bugzilla.wikimedia.org/70881 (10Gryllida) 3NEW p:3Unprio s:3normal a:3Merlijn van Deen 18:35:45 Wikimedia Labs / tools: androidsdk::dependencies doesn't work on Trusty - https://bugzilla.wikimedia.org/69423... [09:22:57] Hiya, are there any plans to add geodata-specific exports to dumps.wikimedia? [09:23:44] cuz the api is a bit limiting^^ [09:31:44] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c6 (10Aude) reproduced these issues in vagrant, which uses hhvm: Fatal error: Argument 1 passed to Wikibase\\ItemContent::__construct() must be an instance of Wik... [09:32:39] PROBLEM - ToolLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: tools.tools-exec-03.puppetagent.failed_events.value (11.11%) [09:36:44] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c7 (10Aude) i then restarted hhvm and the error is gone! before restarting, i did composer install of all of wikibase etc., which maybe exceeded some limit or such? [09:43:49] RECOVERY - ToolLabs: Puppet failure events on labmon1001 is OK: OK: All targets OK [09:47:04] PROBLEM - ToolLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: tools.tools-exec-03.puppetagent.failed_events.value (10.00%) [09:47:47] 3Wikimedia Labs / 3Infrastructure: Wrong URLs in wikitech.wikimedia.org watchlist notifications - 10https://bugzilla.wikimedia.org/70882 (10Kelson [Emmanuel Engelhart]) 3NEW p:3Unprio s:3major a:3None Here is a notification email I have received: ==== The Wikitech page OCG has been changed on 15 Sep... [09:55:04] RECOVERY - ToolLabs: Puppet failure events on labmon1001 is OK: OK: All targets OK [10:09:29] 3Wikimedia Labs / 3tools: androidsdk::dependencies doesn't work on Trusty - 10https://bugzilla.wikimedia.org/69423#c4 (10Yuvi Panda) 5NEW>3RESO/FIX I fixed this a while ago in I76c5b3dff7905d94adb4d46929f09250dff26bd7 [10:10:35] rakkaus does that :D [10:13:29] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862#c2 (10Yuvi Panda) Indeed, that seems ok to do. *Ideally* we would just do this in icinga instead of with diamond, but considering icinga status on labs I'd say go ahead with doing it in diamond. We... [10:16:29] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862#c3 (10Antoine "hashar" Musso) I already have too many things to complete which are long overdue. So I am unlikely to look at writing a diamond collector anytime soon. If you have some spare bandwid... [10:17:29] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862#c4 (10Yuvi Panda) Alright, I'll put it on my 'spare bandwidth TODO' list :) In the meantime, if anyone else wants to step in, please do! I'll be happy to help. [10:17:44] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862 (10Yuvi Panda) [10:59:38] 3Tool Labs tools / 3Database Queries: DBQ-200 Edit history and protection log of semi-protected articles - 10https://bugzilla.wikimedia.org/59482#c1 (10This, that and the other (TTO)) s:5major>3normal This query request is old. If you no longer require this query to be run, please close this bug as WONTF... [10:59:38] 3Tool Labs tools / 3Database Queries: DBQ-197 User preferences usage on Portuguese Wikipedia - 10https://bugzilla.wikimedia.org/59480#c2 (10This, that and the other (TTO)) s:5major>3normal This query request is old. If you no longer require this query to be run, please close this bug as WONTFIX. Thanks! [10:59:39] 3Tool Labs tools / 3Database Queries: DBQ-205 Number of power users using enhanced recentchanges - 10https://bugzilla.wikimedia.org/59487#c2 (10This, that and the other (TTO)) s:5major>3normal This query request is old. If you no longer require this query to be run, please close this bug as WONTFIX. Than... [10:59:39] 3Tool Labs tools / 3Database Queries: DBQ-198 Create and update database reports on fr.wikipedia - 10https://bugzilla.wikimedia.org/59481#c1 (10This, that and the other (TTO)) s:5major>3normal This query request is old. If you no longer require this query to be run, please close this bug as WONTFIX. Than... [10:59:40] 3Tool Labs tools / 3Database Queries: DBQ-214 Basic translation stats for Meta - 10https://bugzilla.wikimedia.org/59497#c1 (10This, that and the other (TTO)) s:5major>3normal This query request is old. If you no longer require this query to be run, please close this bug as WONTFIX. Thanks! [10:59:41] 3Tool Labs tools / 3Database Queries: DBQ-196 one side categories on en.wiki - 10https://bugzilla.wikimedia.org/59479#c2 (10This, that and the other (TTO)) s:5major>3normal This query request is old. If you no longer require this query to be run, please close this bug as WONTFIX. Thanks! [10:59:42] 3Tool Labs tools / 3Database Queries: DBQ-208 List of all composer or songwriter names, in sorted alphabetical order - 10https://bugzilla.wikimedia.org/59491#c1 (10This, that and the other (TTO)) s:5major>3normal This query request is old. If you no longer require this query to be run, please close this... [10:59:43] 3Tool Labs tools / 3Database Queries: DBQ-215 List of JavaScript/CSS editors - 10https://bugzilla.wikimedia.org/59498#c1 (10This, that and the other (TTO)) s:5major>3normal This query request is old. If you no longer require this query to be run, please close this bug as WONTFIX. Thanks! [10:59:44] 3Tool Labs tools / 3Database Queries: DBQ-204 SQL query for pages created on dates between October 2011 and March 2013 - 10https://bugzilla.wikimedia.org/59486#c1 (10This, that and the other (TTO)) s:5major>3normal This query request is old. If you no longer require this query to be run, please close thi... [10:59:45] 3Tool Labs tools / 3Database Queries: DBQ-206 List of English Wikipedia articles along with creation date, length, and categories - 10https://bugzilla.wikimedia.org/59488#c1 (10This, that and the other (TTO)) s:5major>3normal This query request is old. If you no longer require this query to be run, pleas... [11:03:15] 3Wikimedia Labs / 3Infrastructure: 404 URLs in wikitech.wikimedia.org watchlist notifications (pointing to labs.wikimedia.org) - 10https://bugzilla.wikimedia.org/70882 (10Andre Klapper) [11:04:14] 3Tool Labs tools / 3Database Queries: DBQ-204 SQL query for pages created on dates between October 2011 and March 2013 - 10https://bugzilla.wikimedia.org/59486#c2 (10Yuvi Panda) I'll note that for most bugs in this component, someone can setup a query at quarry.wmflabs.org and that should be enough. [11:04:29] 3Tool Labs tools / 3Erwin's tools: Migrate http://tools.wmflabs.org/erwin85/catanalyzer.php to Tool Labs - 10https://bugzilla.wikimedia.org/60868#c2 (10This, that and the other (TTO)) 5NEW>3RESO/WON (In reply to Andre Koopal from comment #1) > I'm afraid you'll probably have to consider this tool as bein... [11:07:46] 3Tool Labs tools / 3Database Queries: DBQ-197 User preferences usage on Portuguese Wikipedia - 10https://bugzilla.wikimedia.org/59480#c3 (10Helder) Ideally, these stats should be updated periodically, but I would like an update before closing this. This is mostly useful to keep an updated table on https://pt... [11:08:49] andrewbogott_afk: hmm, wikitech lost VE in the migration? [11:10:00] 3Tool Labs tools / 3Database Queries: DBQ-214 Basic translation stats for Meta - 10https://bugzilla.wikimedia.org/59497#c2 (10Nemo) 5NEW>3RESO/DUP (In reply to Bugzilla Bug Importer (valhallasw) from comment #0) > [[Special:SupportedLanguages]] and CleanChanges have both been disabled, so > it's impossib... [11:12:59] 3Tool Labs tools / 3Database Queries: DBQ-215 List of JavaScript/CSS editors - 10https://bugzilla.wikimedia.org/59498#c2 (10Nemo) p:5Unprio>3Normal This is still relevant. It would be easy to do, if only I knew how to list all dbnames with a query: a bash loop similar to for dbname in $WIKIS; do... [11:40:59] 3Tool Labs tools / 3Database Queries: DBQ-205 Number of power users using enhanced recentchanges - 10https://bugzilla.wikimedia.org/59487#c3 (10Nemo) p:5Unprio>3Normal Still relevant. [11:43:43] !log phabricator run /srv/phabricator/bin/storage upgrade on phab-01 to setup mysql database tables for phabricator [11:43:44] Logged the message, Master [13:42:17] SUL related tools are not working all of a sudden today, this also includes global tools (tools which search for certain things within multiple wikis) [13:48:57] Suddenly, success? [14:00:02] a930913: My collector doesn't know how to gather that though. It understands toolinfo.json per-tool; though I suppose I could also have it try a cgi if we standardize it. [14:04:57] Coren, can you log into tools-exec-03 for more than 20 seconds at a time? I can log in but I get a broken pipe before I can do anything [14:06:09] andrewbogott: Yeah, though it behaves oddly. [14:06:27] -07 is doing fine, right? [14:06:34] Coren: andrewbogott https://shinken.wmflabs.org :D (admin/admin) [14:06:35] * Coren nods. [14:06:55] It wouldn't surprise me if I gave it brain-damage yesterday, but… such strange behavior [14:06:59] andrewbogott: That looks like there are two (different) places where the packets are routed to. [14:07:36] andrewbogott: So you get odd packet loss, and occasionally a RST [14:07:39] um… how do I log into shinken? [14:08:35] Oh, was that link to shinken unrelated to -03? [14:08:46] andrewbogott: completely unrelated, yes :) [14:09:00] andrewbogott: exec-03 did report puppet failures earlier today, but they resolved themselves quickly [14:09:33] andrewbogott: If I had to venture a guess, I'd say that virt1006 still tries to respond to the IP of the instance. [14:09:46] hm... [14:10:03] Are you able to see if -07 has the same problem? [14:10:29] andrewbogott: It doesn't. It's been put back in the queues and works fine. [14:13:36] ok… is it better now? [14:15:10] andrewbogott: At first glance, the issue seems to be gone. [14:15:18] andrewbogott: What did you change? [14:15:40] I deleted /etc/libvirt/nwfilter/nova-instance-i-000000d6-fa163eb43307.xml on virt1006 [14:15:46] Luxo's guc tool working for you Nemo_bis ? [14:15:52] (and restarted compute, but I bet that didn't matter) [14:16:11] andrewbogott: Ah, crumbs left behind by the move? [14:16:39] I guess… I'm pretty sure nova is supposed to actively maintain those files in Havana. But since that move failed midway through a couple of times, maybe that step was missed... [14:16:50] At any rate, -03 seems to be healthy now. I'll check on it a bit later and will put it back in the queues if it's still good. [14:16:58] Anyway, we should keep an eye out fo see if that happens with future migrations. [14:17:22] andrewbogott: You're probably right that this was due to the failed migration. [14:17:33] Of course, since nova generates those files… it may recreate that one. We'll see [14:17:54] I just want to be sure that I'm not generating that problem with /every/ migration [14:18:24] Because, for instance, I also see nova-instance-i-000000db-fa163e61a8b2.xml in that same place. And 0db is -07 [14:18:30] And yet, that file seems harmless in that case [14:18:47] andrewbogott: Neither -07 nor -submit have that issue; and those migrations worked right the first time. [14:19:04] andrewbogott: Possibly the file wasn't causative; but removing it it forced nova to finish some cleanup. [14:19:14] yeah [14:21:56] Ima leave an arping up; if something else responds to that IP I'll see it soon enough. [14:22:30] (I should have tried that before you fixed the issue to be safe) [14:24:38] Coren: Your collector is not following the standard :p [14:32:05] a930913: My collector is interested in metainformation about tools on Labs. [14:55:25] Coren: Hmm, include only those with labs URLs? [15:00:47] how do I get to OAuth upload tool? [15:04:14] Coren: Is tools-db CPU bound again? [15:11:01] a930913: More importantly, include only those /not/ pointed at by a wiki page but demonstrably under the control of the tool maintainers. [15:11:44] a930913: There's no reason you can't have your json per-tool and point to /them/ from to Wiki page (that is, things work perfectly fine with both schemes at the same time) [15:21:24] Coren: Yeah, but I don't like having to edit wikis :p That's why I make tools to do that for me. [15:44:20] !log deployment-prep testing scap change from https://gerrit.wikimedia.org/r/#/c/160668/ [15:44:22] Logged the message, Master [15:48:44] 3Wikimedia Labs / 3Infrastructure: 404 URLs in wikitech.wikimedia.org watchlist notifications (pointing to labs.wikimedia.org) - 10https://bugzilla.wikimedia.org/70882#c3 (10Andrew Bogott) This looks fixed to me! Kelson, can you confirm? [15:54:05] YuviPanda: I'd appreciate it if you could scan https://wikitech.wikimedia.org/wiki/Virt1006_rebuild and let me know which of those instances are yours and safe for immediate reboot (e.g. the 'design' ones). It's going to take ages to do all of them, hoping to get a head start. [15:57:44] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c8 (10Bryan Davis) I wonder if this is related to the hhvm cache issues that Ori has been looking into? I've only been following that via irc eavesdropping, but I... [16:34:44] 3Wikimedia Labs / 3deployment-prep (beta): Search is sometimes slow - 10https://bugzilla.wikimedia.org/70869#c5 (10Antoine "hashar" Musso) Chad / Nik are the best point to investigate ElasticSearch related issue. Maybe someone imported a bunch of articles on beta which caused a lot of indexing on ElasticSea... [16:37:59] 3Wikimedia Labs / 3deployment-prep (beta): Search is sometimes slow - 10https://bugzilla.wikimedia.org/70869#c6 (10Nik Everett) I can have a look at it soon - yeah. The Elasticsearch cluster in beta isn't designed for performance - just to be there and functional. [17:13:29] Does labs have a bittorrent? [17:13:51] "a bittorrent"? [17:13:55] Client? [17:14:18] It could seed wiki dumps, toolstuff and other free softwares we use. [17:14:39] a930913: It would probably affect network speed, though [17:14:46] Labs is slow as it is [17:14:51] marktraceur: Can't it run with low priority? [17:15:08] I.e. use spare capacity. [17:20:16] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c9 (10Aude) @bryan I think quite certain it is related. this happens less often on test.wikidata, but I think because hhvm has higher limits in production. (JitAC... [17:26:45] hi, i'd like to ask who is maintaining the localized OSM tiles at http://c.tiles.wmflabs.org/osm-multilingual [17:27:45] toolserver.org used to offer completely localized OSM tiles [17:28:39] e.g. if I want to see a Russian-language of Ukraine, I'd for example use http://{s}.www.toolserver.org/tiles/osm-labels-ru [17:29:06] however, the rendering seems to have changed since the server move [17:29:15] street names don't get localized anymore [17:29:30] which is really stupid in many cases [17:32:00] does anyone know whom I should contant about this? [17:33:17] * a930913 pokes Coren to provide his omniscience for Matur10n. [17:34:53] :) [17:36:20] So coren_ is the person I should contact? :) [17:46:11] Matur10n: He solves 90% of my problems :) [17:53:49] I see :) [17:55:10] He's a qualified sysadmin and also not half bad as a therapist. [18:00:31] Matur10n: I know very little about the OSM project; our (WMF) contact is Alexandros (akosiaris on IRC) but you may want to contact Kai Krueger who seems to be the one with the technical know-how (http://wiki.openstreetmap.org/wiki/User:Amm) [18:00:58] thank you Coren! :) [18:04:36] also, i hear MaxSem is pretty involved in OSM [18:06:51] I see [18:07:19] basically the rendering of maps just changed massively since the server move [18:07:30] so I wonder who is in charge now [18:08:39] Matur10n: I think it's mostly the same people "in charge"; but from my understanding the tile thing wasn't just /moved/ so much as rebuilt in a new way. The idea was to allow for more production-level scalability, but that may have changed a couple of things in /how/ it's used. [18:09:44] I see. Well, as I said, street maps don't get translated anymore, even in the specifically localized map versions [18:09:51] which is quite bad in many cases [18:10:32] I wouldn't know about that; you really need to ask the OSM people. Maybe it's just done differently now? Sorry I can't be more helpful. [18:10:41] I see ;) [18:12:01] 3Wikimedia Labs: meta_p.wiki and meta_p.legacy tables not filled on all servers - 10https://bugzilla.wikimedia.org/70893 (10Peter Schlömer (dapete)) 3NEW p:3Unprio s:3major a:3None On the database replica servers, the meta_p.wiki and meta_p.legacy tables only contain one entry - for centralauth - on s1... [18:12:57] for example, the country of belarus has two official languages, belarussian and russian. default name tags in OSM for this country use Russian only. So now there's no possibility anymore to see the Belarussian names "live in action". and there are many other examples [18:13:13] well, I'll try to contact the people who are in charge :) [18:15:43] so, maybe, if you happen to stumble across one of them, you can tell them too :) [18:29:49] Coren: Current query is over 100ks :( [18:30:18] a930913: Have you checked that your query /itself/ isn't problematic? [18:30:35] a930913: Like, not doing silly things like full row scans? [18:30:36] Coren: All it is, is a normalisation between two other tables. [18:30:51] Coren: It's the INFILE. [19:12:29] 3Wikimedia Labs / 3Infrastructure: 404 URLs in wikitech.wikimedia.org watchlist notifications (pointing to labs.wikimedia.org) - 10https://bugzilla.wikimedia.org/70882#c4 (10Kelson [Emmanuel Engelhart]) OK to me. [19:15:40] Coren: How does the rep db work? [19:16:04] That's... a hopelessly vague question. [19:17:54] Coren: Ok, could we replicate the CORE database inside labs? [19:18:23] What's a "CORE database"? [19:19:22] Coren: A not edge database? ;-) [19:19:29] 3Wikimedia Labs / 3Infrastructure: 404 URLs in wikitech.wikimedia.org watchlist notifications (pointing to labs.wikimedia.org) - 10https://bugzilla.wikimedia.org/70882 (10Andrew Bogott) 5PATC>3RESO/FIX [19:20:09] Coren: core.kmi.open.ac.uk [19:20:26] aude: are wikidata-mobile-tester and wikidata-test and wikidata-builder3 yours? [19:21:07] a930913: Then no. It's not our databases, we can't really setup slave-masters with them. [19:21:53] andrewbogott: they are [19:22:03] They do seem to offer an api and SPARQL (yuck) [19:22:04] we probably don't care about the mobile one now [19:22:12] Coren: If they are willing to build stuff on their side? [19:22:28] what do you need us to do? [19:22:31] aude: They're in the list that I emailed about earlier. Mind if I migrate them now? That will mean a bit of downtime and a reboot. [19:22:46] what does migrate mean? [19:22:50] * aude missed the mail [19:23:04] aude: They need to fly south for the winter. [19:23:13] a930913: Then it might be possible; but the question is whether the WMF - as an organization - wants to mirror that data. That's a question best asked of Erik Möller. [19:23:14] should be ok though [19:23:17] :) [19:23:34] a930913: It's not something that's conceptually impossible; we do that for (part of) the OSM dataset for instance. [19:23:54] both are nearly 100% puppetized [19:25:34] aude: OK, any preference regarding what order I restart them in? I'm going to move them now. [19:25:52] no preference [19:26:12] ok! [19:27:08] Coren: Apparently Ed knows him :) [19:33:08] aude: all done, they should be starting up again now [19:33:52] wow, quick [19:34:41] YuviPanda: 'boiledegg' -- that yours? [19:34:56] * andrewbogott happy to note that he's not the only one who names his servers after breakfasts [19:36:57] !log math moving and rebooting mws instance [19:36:59] Logged the message, dummy [19:47:37] multichill: Do you know SPARQL? [19:48:36] I played with it in university back in 2000(?). Seems to be a synoniem for slow and broken these days. [19:51:38] multichill: Could you help me try make a query? :/ [19:51:44] 3Tool Labs tools / 3wikibugs IRC bot: Bot should use notices in-channel - 10https://bugzilla.wikimedia.org/70881#c1 (10Merlijn van Deen) 5NEW>3RESO/WON Unfortunately, for the rest of the world, /notice shows up as a highlight in the channel, and thus attracts *more* attention -- I don't think that's an i... [19:53:21] a930913: I would try http://core.kmi.open.ac.uk/api/doc (if it would respond). I'm going offline in a moment [19:53:59] multichill: It 404s or otherwise doesn't load. [19:54:15] It's probably SPARQL based :P [19:55:03] :p [20:01:09] andrewbogott: ya, boiled egg, sugaredfrosties are mine [20:01:32] YuviPanda: mind if I shut down, move, reboot boiledegg? [20:01:38] andrewbogott: nope, go ahead [20:01:43] thanks [20:02:09] andrewbogott: quarry-web-test is theonly other one, and it has live data. if you want to do that next, I can take a backup now [20:02:33] That'd be great, thanks. Let me know when it's safe to move. [20:04:46] andrewbogott: moment [20:08:16] andrewbogott: I presume NFS is going to be alright [20:08:38] The instance shouldn't know the difference. [20:09:02] andrewbogott: cool, feel free to migrate. let me know when done, since two other instances depend on this one and would need to be restarted [20:09:19] thanks, I'll do it right now [20:15:13] when will you do dwl? [20:17:34] gifti: now, if you want, otherwise in a batch next week [20:18:37] YuviPanda: quarry-web-test should be moved and booting… can you access it? [20:18:41] now would be great [20:18:52] andrewbogott: checking [20:19:09] andrewbogott: can't login yet [20:19:17] gifti: ok, here goes... [20:19:24] YuviPanda: what about boiledegg? That one ok? [20:19:35] andrewbogott: yaa [20:19:41] hm [20:19:57] well, let's give -web-test a few minutes to recover from the shock [20:20:06] ok [20:20:29] it is like 'wat? what happened mannn!?! everything feeeellssss difffferent' [20:20:34] * YuviPanda gives quarry-web-test a beer [20:20:53] heh [20:25:06] YuviPanda: what's the command for give a beer? [20:25:21] jeremyb: /me gives a beer [20:25:22] I think [20:25:23] ;) [20:25:24] YuviPanda, quarry-web-test may have died on the operating table. I'm trying a few more things... [20:25:32] ow damn [20:25:41] maybe it was alcohol poisoning? [20:25:41] it was fully puppetized, though [20:25:58] sudo give me a beer [20:28:40] * sudo gives mutante a beer [20:34:21] hmm [20:35:05] andrewbogott: did you already do it? [20:35:18] gifti: no, sorry, distracted by quarry-web-test misbehaving [20:35:28] ah, ok [20:36:04] Guest93328: :) thanks [20:36:20] mutante: yw :) [20:38:22] YuviPanda: I am failing to revive it :( Do you mind deleting and rebuilding? [20:38:33] I don't know what happened, I've migrated 20 other instances without ill effect [20:39:05] awwwwwwwww [20:39:13] * YuviPanda nicks to being sadpanda [20:39:23] let me [20:39:24] try [20:39:26] sorry :( [20:40:19] andrewbogott: 'tis ok, I don't have anything precious on there, except the data which I dumped [20:40:21] let me rebuild one [20:43:38] gifti2: dwl is moved and back up [20:46:01] andrewbogott: 'failed to create instance'? [20:46:11] hmm, I might be out of quota [20:46:18] * YuviPanda goes to check [20:46:27] andrewbogott: I suppose new instances will automatically be put on the 'good' machine? [20:46:28] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (33.33%) [20:46:38] YuviPanda: yeah, virt1006 is out of the pool [20:46:55] andrewbogott: can you increase quota for quarry? [20:47:02] You're hitting your ram quota. Want me to raise? [20:47:08] andrewbogott: ya [20:47:16] andrewbogott: enough for an xlarge? [20:47:53] try now [20:49:18] andrewbogott: yup, added [20:49:18] ty [20:50:55] petan: if you are there, can I reboot the wm-bot instance? [20:54:05] andrewbogott: why? [20:54:25] petan: so I can move it to a different host [20:54:46] could you wait 20 min [20:55:14] sure [20:55:22] or sec [20:55:30] I will shut the bots down now ok? [20:55:38] YuviPanda: dammit, the scheduler is putting new things on virt1006. Including the instance you just built. Gotta sort that out before I do anything else :( [20:55:43] petan: sure [20:55:48] aw shit [21:09:39] * YuviPanda does [21:09:43] In theory I'm on purpose leaving the dead husks of instances there to make it look full. But also the damn scheduler is supposed to... [21:11:23] andrewbogott: created one moe [21:11:25] *more [21:11:28] quarry-web-01 [21:11:43] I'm still seeing a bunch of errors in the scheduler log [21:11:49] probably the new instance has state of 'error' [21:13:50] andrewbogott: ya [21:16:43] and of course now something unrelated has broken, just because I looked at this logfile [21:17:43] hehehe [21:22:38] andrewbogott: any luck? if it's a gone case, I can probably repurpose one of my other instances temporarily. [21:22:44] but I'd like it to be its own instance [21:24:25] It's going to take a while for me to sort out the scheduler. [21:24:54] andrewbogott: hmm, let me see what it'll take to repurpose one of the other ones [21:27:16] andrewbogott: ok, if ETA is >30m, I can repurpose one temporarily, but I'd like to get a new VM back tomorrow. should I repurpose? [21:27:30] I can't predict. I would hope to have it sorted in 30 [21:27:56] andrewbogott: alright. it's fairly trivial to repurpose, so I'll wait 15m, see how it goes, and then if not done repurpose. [21:44:14] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [21:48:08] Can someone explain why the statistics stage is taking so long? https://tools.wmflabs.org/paste/view/d1965f10 [21:48:44] (It's a simple SELECT * FROM x WHERE ID=y LIMIT 1) [21:50:20] andrewbogott: alright, I'm going to repurpose a machine temporarily for now [21:50:34] ok [21:50:53] andrewbogott: this is quarry-runner-test, and it wasn't on your list, so I presume it's on the non-terrible virt [21:51:12] You can tell, instances have a 'host' field on their info page [21:52:23] aaah [21:52:24] ok [21:52:26] TIL [21:53:20] !log quarry appllying db, web and redis roles to quarry-runner-test, will act as db and web host until labs issues clear up [21:53:22] Logged the message, Master [22:11:52] !log testlabs created a zillion disposable instances on virt1006 to block other new instances being scheduled there. [22:11:55] Logged the message, dummy [22:13:42] andrewbogott: I'm running into problems of mysql being a piece of shit, now it's stuck in a stupid state where it won't install nor uninstall since the initial start fails o_O [22:14:08] * YuviPanda bangs head on wall [22:14:18] YuviPanda: I have just done the dumbest possible thing (^) which should allow you to create a new instance that doesn't land on virt1006 [22:14:24] hmm [22:14:25] let me try [22:15:17] andrewbogott: ok, created. waiting [22:16:00] andrewbogott: hmm, current host and original host 'none' in https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000005f5.eqiad.wmflabs? [22:16:21] Remember how I said there was an unrelated issue that suddenly appeared when I looked at this? [22:16:22] That's it [22:16:26] lol [22:16:34] * YuviPanda hopes horizon is on the horizon [22:16:41] but you're on virt1001, not to worry [22:16:53] This isn't OSM's fault, actually. I don't know what happened. [22:17:55] hmm, have beta labs log files moved? I used to find them in /data/project/logs/*, but only seeing Cirrus-Search logs there atm [22:18:04] (from deployment-bastion.eqiad.wmflabs) [22:18:26] maybe they are in elasticsearch now? [22:21:36] YuviPanda: ok, now I think /that/ bug is fixed as well. Maybe [22:22:15] andrewbogott: they're still none [22:22:23] If you click 'reboot' it should refresh [22:22:33] ah, probably not [22:22:37] let me wait for it to come up [22:22:59] I should be in bed now (GF has early day tomorrow), but have moved to couch instead. Should've waited to migrate :( [22:23:02] * YuviPanda learns lessons [22:23:19] sorry. Like I said, yours is the only instance that didn't survive the move :( [22:23:31] I'm curious why it died, but not /that/ curious [22:23:31] https://tools.wmflabs.org/?status also mega much moar fast. This many! *holds two fingers* [22:23:32] yeah, it's not particularly special either :( [22:23:47] andrewbogott: heh, was it because it was its own puppetmaster? [22:24:28] No, I've really no idea. Just don't want to spend a whole day investigating "Why did a an instance on a broken server fail?" Because the answer will be "because the server is broken" and we're already trying to take measures for that... [22:24:48] heh [22:24:57] andrewbogott: yeah, I'm glad I took the backup [22:27:32] (03PS1) 10coren: Tool Labs: fix status.php so that it sucks less [labs/toollabs] - 10https://gerrit.wikimedia.org/r/160847 [22:34:19] anomiebot-4 is not at 1225h53m but it /has/ been running nonstop since March 4. [22:36:59] For that matter, the anomiebots are stunningly stable. [22:43:06] YuviPanda: are you pretty much unblocked now? [22:43:14] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (22.22%) [22:43:15] andrewbogott: seems so, ya [22:43:31] Sorry about all the breakings [22:43:47] andrewbogott: 'tis ok :) [22:44:21] andrewbogott: all working now [22:44:37] speaking of, andrewbogott, I put -03 back in the queues now that it is working fine. [22:44:44] * YuviPanda goes to sleep [22:44:47] thanks andrewbogott! :) [22:44:54] and I have more quota now! [22:45:05] g'night! [22:48:26] * andrewbogott is going. Will he take the breaking with him? [22:51:40] ToolLabs: Low disk space on /var [22:57:25] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [22:58:20] Coren: ^ that , btw [22:59:02] Yeah, I'm just boggled by the check having ever triggered in the first place. [22:59:44] I'll need to bug yuvi to make sure I actually can do more than just look at the damn thing too. :-) [23:00:11] hmm.. here's the history thing [23:00:13] https://icinga.wikimedia.org/cgi-bin/icinga/history.cgi?host=labmon1001&service=ToolLabs%3A+Low+disk+space+on+%2Fvar [23:00:26] yea, let's ping him :) [23:01:01] He just went to sleep. [23:27:44] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (11.11%) [23:34:46] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK