[00:18:42] Qcoder00: yes, enwiki has 6:13:54 replag [00:53:28] Coren, petan: s1 replication stopped for over 6 hours [00:57:50] 3Wikimedia Labs / 3tools: Replication for enwiki has stopped - 10https://bugzilla.wikimedia.org/64154#c2 (10Tim Landscheidt) Replication for enwiki seems to have stopped again: | MariaDB [enwiki_p]> SELECT MAX(rc_timestamp) FROM recentchanges; | +-------------------+ | | MAX(rc_timestamp) | | +-------------... [00:59:08] yeah, that too.... [01:00:59] springle ^^ [01:15:54] I'll go take a look; if it's a suck write I can help. [01:17:11] Coren: pong [01:17:33] Eloquence: looking [01:19:04] springle: It was catscan2 eating up locks again. [01:19:29] We need to sit down with its maintainer and find a long-term solution. [01:19:37] yeah i saw it just before you killed [01:19:44] same old story [01:20:13] springle: On the more substantial matter, I have to run maintain-replicas tonight for some schema changes and I understand you wanted to add some as well? [01:20:34] springle: Plus we need to replicate two new tables or so; lemme dig my list up. [01:20:52] Coren: yes, but I havn't had time to test them yet. So not yet [01:21:04] (the schema changes, not the tables) [01:21:50] 3Wikimedia Labs / 3tools: separate /tmp and /var/tmp volumes - 10https://bugzilla.wikimedia.org/64697 (10Marc A. Pelletier) p:5High>3Normal s:5critic>3normal [01:22:34] 3Wikimedia Labs / 3tools: Replication for enwiki has stopped - 10https://bugzilla.wikimedia.org/64154#c3 (10Sean Pringle) This blocked replication: ---TRANSACTION D3668010, ACTIVE 28563 sec fetching rows mysql tables in use 3, locked 3 132316 lock struct(s), heap size 13384120, 1331972 row lock(s), undo log... [01:29:20] 3Wikimedia Labs / 3tools: add wb_property_info table on tool labs - 10https://bugzilla.wikimedia.org/56180 (10Marc A. Pelletier) 5PAT>3RES/FIX [01:36:49] 3Wikimedia Labs / 3tools: querycache and querycachetwo tables aren't available on labs sql dbs - 10https://bugzilla.wikimedia.org/63782#c1 (10Marc A. Pelletier) This will require an okay from legal (though I do not anticipate difficulties). [01:37:35] 3Wikimedia Labs / 3tools: Create views for user_daily_contribs table - 10https://bugzilla.wikimedia.org/61300#c1 (10Marc A. Pelletier) This request will require evaluation from Legal. [01:39:34] 3Wikimedia Labs / 3tools: expose page_links_update on page table in tool labs dbs - 10https://bugzilla.wikimedia.org/60506#c1 (10Marc A. Pelletier) 5NEW>3RES/WOR Presuming you mean page_links_updated, that column is already exposed. [01:44:21] 3Wikimedia Labs / 3tools: Provide filearchive table with fa_storage_key or, if it exists and is sufficiently indexed and populated, fa_sha1 for commonswiki - 10https://bugzilla.wikimedia.org/57697#c1 (10Marc A. Pelletier) I can see several issues with this, not least of which the ability to identify whether... [02:09:04] 3Wikimedia Labs / 3tools: Wikibase tables are missing on testwikidatawiki - 10https://bugzilla.wikimedia.org/57527#c1 (10Marc A. Pelletier) 5NEW>3RES/FIX Those tables are now available. [02:11:20] 3Wikimedia Labs / 3tools: Make betafeatures_user_counts table available - 10https://bugzilla.wikimedia.org/57491#c6 (10Marc A. Pelletier) 5NEW>3RES/FIX That table has been added to replication. For the record, while the patch to maintain-replicas.pl is 'easy', applying it to the replicas proper requires... [02:12:34] 3Wikimedia Labs / 3tools: Table 'wikidatawiki_p.user' doesn't exist - 10https://bugzilla.wikimedia.org/54959#c3 (10Marc A. Pelletier) 5ASS>3RES/FIX Long since resolved. [02:13:34] 3Wikimedia Labs / 3tools: Missing or wrong information in meta_p.wiki table - 10https://bugzilla.wikimedia.org/54962#c5 (10Marc A. Pelletier) 5ASS>3RES/FIX There was a typo that prevented the size column from being properly set. Fixed since. [06:34:05] 3Wikimedia Labs / 3tools: Provide filearchive table with fa_storage_key or, if it exists and is sufficiently indexed and populated, fa_sha1 for commonswiki - 10https://bugzilla.wikimedia.org/57697#c2 (10Rainer Rillke @commons.wikimedia) (In reply to Marc A. Pelletier from comment #1) https://bugzilla.wikimed... [06:56:05] 3Wikimedia Labs / 3tools: Create views for user_daily_contribs table - 10https://bugzilla.wikimedia.org/61300#c2 (10Liangent) I think it's just an aggregated revision table, isn't it? [06:57:57] tools of Magnus are still failing [07:14:06] **FRUSTRATION** [10:33:35] 3Wikimedia Labs / 3tools: Move wiki.toolserver.org to WMF - 10https://bugzilla.wikimedia.org/60220 (10TeleComNasSprVen) [10:33:35] 3Wikimedia Labs / 3tools: Transfer domain toolserver.org to WMF - 10https://bugzilla.wikimedia.org/60864 (10TeleComNasSprVen) [14:24:01] 3Wikimedia Labs / 3tools: commonswiki_p.hashs view is broken on Wikimedia Labs - 10https://bugzilla.wikimedia.org/59684#c1 (10Tim Landscheidt) 5NEW>3ASS Kind of a duplicate of bug #55708. The table "hashs" was dropped per bug #52927 and the corresponding view just needs to be deleted. [15:05:02] how to create new mysql database on tool-labs (for my gsoc project) ? [15:11:37] rohit-dua: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Creating_new_databases [15:14:50] scfc_de: thank you, can you tell what are replica servers, and host of the replica (e.g., enwiki.labsdb) ?? [15:16:16] 3Wikimedia Labs: GlusterFS readonly on integration project - 10https://bugzilla.wikimedia.org/61141#c2 (10Antoine "hashar" Musso) 5NEW>3RES/FIX Fixed by the migration to EQIAD. We are now using NFS. [15:17:31] 3Wikimedia Labs / 3Infrastructure: packages needs upgrade on bastion instances - 10https://bugzilla.wikimedia.org/60895#c1 (10Antoine "hashar" Musso) 5NEW>3RES/WON The instances got migrated to eqiad and some are still not upgraded. I am forgetting this bug though, no point in polluting Bugzilla lists i... [15:18:31] rohit-dua: Replica servers are the ones that host (replicated) copies of the Wikipedias & Co. They are numbered from s1.labsdb to s7.labsdb and there are aliases for the individual projects as well (enwiki.labsdb => s1.labsdb). You can create your own databases on those servers as well. This is mainly useful for applications where you need to JOIN between your data and Wikipedia. Apart from that, there is a dedicated DB server [15:18:31] "tools-db" for anything else. [15:22:31] 3Wikimedia Labs / 3Infrastructure: bastion1 and bastion3 on the same host (virt10) - 10https://bugzilla.wikimedia.org/57833#c2 (10Antoine "hashar" Musso) 5NEW>3RES/FIX Seems all instances are on different hosts now. Though bastion2-eqiad is on the same host has bastion1. Assuming this is good enough and... [15:25:10] scfc_de: thank you.... so dbs for individual projects (like mine, for gsoc) will be hosted on replica servers? right? [15:28:04] rohit-dua: What's your project? And what do you want to do with the database? [15:30:25] scfc_de: My project is related to upload to Google-books to internet archive and commons. I need the DB to store user details like email, google-books url, commons-name, etc. [15:30:45] scfc_de: https://www.mediawiki.org/wiki/Google_Books,_Internet_Archive,_Commons_upload_cycle [15:30:55] manybubbles: still interested in help? [15:31:24] matanya: yeah! I've literally just finished the work that should make beta better. one moment while I get it in beta [15:31:34] sure [15:33:44] !log deployment-prep rolling restart of Elasticsearch servers in beta to pick up new highlighter plugin to fix bugs found when we fixed hebrew analysis. and to implement phrase highlighting. [15:33:45] is not a valid project. [15:33:48] !log deployment-prep rolling restart of Elasticsearch servers in beta to pick up new highlighter plugin to fix bugs found when we fixed hebrew analysis. and to implement phrase highlighting. [15:33:50] Logged the message, Master [15:33:51] spaces [15:36:46] matanya: ok! everything should be ready for you [15:37:04] rohit-dua: If you need to query the Wikipedia DBs as part of your projects, you should probably put the DB on the replica servers. But you should have your mentors confirm that, because they are more knowledgeable about your project. [15:37:16] the last time we talked I found out that I was using the wrong analyzer for queries in hebrew which caused craziness. It should be different now [15:37:17] manybubbles: cool, what type of searches you think would be best to test? [15:37:19] hopefully better! [15:37:30] same test? [15:37:35] matanya: any are fine [15:37:37] sure! [15:37:41] scfc_de: ok thank you. [15:37:43] doing [15:38:02] I just added support for "quoted phrases" only highlighting the phrase. Atleast, they should only highlight the phrase [15:38:20] there are some weirdness is that that I'll have to fix (soon) around phrase sloo [15:38:22] slop [15:38:26] too jittery today [15:39:32] manybubbles: just a little improvement [15:39:56] matanya: thats a shame! [15:40:02] can you point me to what isn't improved? [15:40:03] there are some really great suggestions, but some are really off [15:40:19] I searched : http://he.wikipedia.beta.wmflabs.org/w/index.php?search=%D7%9E%D7%95%D7%A8%D7%94&title=%D7%9E%D7%99%D7%95%D7%97%D7%93%3A%D7%97%D7%99%D7%A4%D7%95%D7%A9&go=%D7%9C%D7%93%D7%A3 [15:40:26] which is a teacher in hebrew [15:40:42] see the second result for example [15:40:53] it returns Maria as a result [15:41:22] hashar: Do you know what's up with http://ganglia.wmflabs.org/ ("There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused")? [15:41:27] that has nothing to do with the search term expect for some letters being common between the two [15:41:49] matanya: hmmmm.... lets try another setting [15:41:56] scfc_de: I still have to overhaul that lame setup sorry [15:42:02] scfc_de: the instance is too small [15:42:25] scfc_de: we would need the frontend and the aggregator and two different instances [15:43:45] scfc_de: I will reboot it [15:43:58] !log ganglia rebooting aggregator, it is stalled somehow: There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused [15:43:59] Logged the message, Master [15:44:03] hashar: Thanks (*fingers-crossed*). [15:45:54] 502 Bad Gateway [15:45:54] [15:45:55] :( [15:46:36] You use the Labs web proxy? So the webserver on the instance isn't up? [15:46:48] ah yeah probably [15:48:20] Ah! Working. So I should hurry up to debug my reporter script before it crashes again :-). Merci bien! [15:48:32] :-] [15:57:11] matanya: tweaked a setting. can you check again? [15:57:20] sure [15:57:56] well, the order changed, but it is still there [15:58:13] moreover only the fourth result is a valid one [16:00:05] matanya: so it got worse? [16:00:09] I can try another tweak [16:01:26] yes manybubbles [16:07:40] matanya: before I go flailing around I want to confirm: the word in the search box and the highlighted word in the first title are different (on this page: http://he.wikipedia.beta.wmflabs.org/w/index.php?search=%D7%9E%D7%95%D7%A8%D7%94&title=%D7%9E%D ) [16:08:28] yes manybubbles [16:08:49] the search term is a "teacher" and the title is a sur name [16:09:17] which has a duplicate meaning as well that means: my teacher [16:10:47] matanya: ah. it looks like my last tweak wasn't going to fix it. I'm going to have to go deeper. which will take some time. [16:11:10] ok manybubbles i'll be afk a bit [16:11:21] bbl - about an hour or [16:11:47] so, you can leave ,messages here, i'll read back scroll when back [16:40:43] !log tools Moved ~scfc/bin/replagstats to ~tools.admin/bin/ and enabled as a continuous job (cf. also bug #48694). [16:40:45] Logged the message, Master [16:43:01] 3Wikimedia Labs / 3tools: Show replication lags in Ganglia - 10https://bugzilla.wikimedia.org/48694#c5 (10Tim Landscheidt) (I moved ~scfc/bin/replagstats to ~tools.admin/bin/, rewrote it from a cron to a continuous job and started it with jstart.) [17:37:16] 3Wikimedia Labs / 3Infrastructure: bastion1 and bastion3 on the same host (virt10) - 10https://bugzilla.wikimedia.org/57833#c3 (10Ryan Lane) When Labs upgrades to the Icehouse release they should use the new scheduler feature to ensure they can't live on the same host. [17:40:20] I mistakenly deleted my replica.my.cnf file in tool-labs. How do I re-get it ?? [17:41:20] it will reappear after some while [17:41:27] it did for me [17:42:14] gifti: will error.log and access.log appear too? [17:42:22] yes [17:42:30] but empty i guess [17:43:32] is there any backup going on for tool-labs? [17:43:51] not atm afair [17:53:36] !log deployment-prep Logstash in beta hasn't recorded any events since 2014-05-04T04:32:36. [17:53:39] Logged the message, Master [17:53:50] !log tools Don't think replagstats is really working ... [17:53:52] Logged the message, Master [17:54:20] !log deployment-prep Restarted logstash on deployment-logstash1 [17:54:23] Logged the message, Master [18:23:18] 3Wikimedia Labs: /etc/mailname is set to "labs-vmbuilder-precise.eqiad.wmflabs" - 10https://bugzilla.wikimedia.org/64962 (10Tim Landscheidt) 3NEW p:3Unprio s:3normal a:3None On a freshly created instance, /etc/mailname is set to "labs-vmbuilder-precise.eqiad.wmflabs", and that doesn't get fixed by subs... [18:34:35] 3Wikimedia Labs / 3Infrastructure: Monitor Puppet "stability" - 10https://bugzilla.wikimedia.org/64964 (10Tim Landscheidt) 3NEW p:3Unprio s:3enhanc a:3None At the moment, Puppet status in [[wikitech:Special:NovaInstance]] is either "unknown", "ok" (last Puppet run succeeded), "stale" (last Puppet run... [18:34:57] what should be the value of host when using Mysql module in python : MySQLdb.connect(host,......) [18:35:19] in tools-lab [18:37:52] rohit-dua: depends on what database you are trying to access [18:38:20] my database is on tools-db [18:43:15] rohit-dua: host='tools-db' seems to work? [18:47:21] Nettrom, yes it does work. my mistake [18:47:35] rohit-dua: no problem, happy we got it figured out :) [19:31:14] !log tools replagstats fixed; Ganglia graphs are now under the virtual host "tools-replags" [19:31:16] Logged the message, Master [20:38:31] 3Wikimedia Labs / 3tools: Show replication lags in Ganglia - 10https://bugzilla.wikimedia.org/48694#c6 (10Tim Landscheidt) (I needed to group the statistics at Ganglia under the virtual host "tools-replags".) [21:05:20] matanya: if you are around, I've tweaked another setting for hebrew [21:50:35] Coren: you around? I need help debugging an issue [21:55:54] Coren: Where did my crontab go after the move too tools-submit? [21:55:59] * my tools' [21:58:02] hoo: tools-submit :P [21:58:07] Betacommand: nope [21:58:11] hoo: what are you trying to do? [21:58:23] Betacommand: Get my cron back... don't care onw hich host [21:58:43] Oh, there where about 50 tools that had their crontab deleted [21:58:55] hoo: What's the name of the tool? [21:59:00] Of course mine was one of them :P [21:59:09] scfc_de: "hoo" [21:59:44] hoo: is that the tool name or your login name? [22:00:12] Betacommand: both [22:00:26] :P [22:01:51] hoo: You don't have a copy from the move to eqiad in your tool's home directory, so I /think/ the crontab's lost for good. [22:02:22] hoo: thats evil [22:02:47] also you cant have a crontab on a non-tool account [22:03:13] scfc_de: File name? [22:05:31] hoo: It would have been ...DATA.something. [22:06:12] tools.hoo@tools-login:~$ find -name '*DATA*' [22:06:12] ./...DATA.olduser [22:06:12] ./...DATA.tools-db-dump.sql [22:06:19] so, nope :/ [22:06:27] that's only DB stuffs [22:06:40] ...DATA.crontab, to be exact. [22:07:19] nope, don't have that... think I deleted it [22:59:36] Coren: if you are not already eating dinner… could you log into labs-vmbuilder-trusty and see if you can tell why puppet complains about idmapd? [22:59:48] Looks to me like the relevant packages are installed, but maybe they're scrambled somewhat on trusty... [23:11:03] 3Wikimedia Labs: Project-wide Puppet classes and variables - 10https://bugzilla.wikimedia.org/64980 (10Tim Landscheidt) 3NEW p:3Unprio s:3enhanc a:3None It would be nice if Puppet classes and variables could not only be chosen on an instance level as now, but also for the containing project. Thus, if... [23:34:01] 3Wikimedia Labs: Project-wide Puppet classes and variables - 10https://bugzilla.wikimedia.org/64980#c1 (10Ryan Lane) When puppet 3 is available we'll be able to do this via hiera.