[00:33:40] 6Labs, 10Tool-Labs: missing database on replica server - https://phabricator.wikimedia.org/T105713#1452833 (10Springle) So, I need to get some more information from Jaime about what occurred on the weekend (he is unwell and on leave since then), but looking over the logs I see: ``` 150711 2:00:17 [ERROR] mys... [01:56:08] PROBLEM - Puppet staleness on tools-bastion-01 is CRITICAL 55.56% of data above the critical threshold [43200.0] [02:07:28] can i have NFS back on project wikistats? [02:07:37] i use labsdebrepo for packages [02:07:49] so:/etc/apt/sources.list.d# cat labsdebrepo.list [02:07:49] deb [trusted=yes] file:///data/project/repo/ / [02:08:10] and i dont have the repo anymore then [02:14:26] mutante: You can set it in puppet? [02:14:28] Or hiera. [02:19:18] John pointed me to modules/labstore/files/nfs-mounts.yaml [02:19:22] looks like it [02:19:25] thanks both [02:19:53] now just need the gid for the project [02:20:44] You can also do it from the project's hiera on-wiki [02:21:06] See eg: https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep at the end [02:22:50] ah, Ok :) [02:55:58] !log wikistats - apt-get upgrade [02:56:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL, Master [02:56:08] interesting, i need to configure the LDAP server names [02:56:13] when doing a regular package upgrade [02:57:05] and whether i want "check server SSL cert" to be never,allow,try or demand [02:57:13] tries the defaults [05:29:41] PROBLEM - Free space - all mounts on tools-webgrid-lighttpd-1404 is CRITICAL tools.tools-webgrid-lighttpd-1404.diskspace.root.byte_percentfree (<30.00%) [06:24:15] Hi all [06:24:17] Hi andrewbogott [06:41:12] 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1453118 (10yuvipanda) [06:41:13] 6Labs, 7Tracking: Instance for running OpenOCR (OCR as a service) in a Docker container - https://phabricator.wikimedia.org/T105584#1453115 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Created project openocr with abartov as member :) [06:44:41] RECOVERY - Free space - all mounts on tools-webgrid-lighttpd-1404 is OK All targets OK [08:25:36] 6Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: Remove [?] links from Special:NovaInstance - https://phabricator.wikimedia.org/T105770#1453211 (10Nemo_bis) The cases where they work are negligible and I saw no progress in years towards an increase of functioning links. Having nearly-always broken lin... [08:42:19] 6Labs: Should keystone endpoints specify api version? - https://phabricator.wikimedia.org/T102806#1453235 (10hashar) Thanks for the double check @Andrew [09:35:33] 6Labs, 10Tool-Labs: missing database on replica server - https://phabricator.wikimedia.org/T105713#1453319 (10Superyetkin) >>! In T105713#1452833, @Springle wrote: > Do we know if the tables were MyISAM or ARIA? I recall creating the database via MySQL Workbench, so the storage engine should be the default va... [11:10:30] hi, is on toolabs a quota/ratelimit? [11:41:15] !log bots retarted wm-bot throwing errors and rc hosed [11:41:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/SAL, Master [12:26:02] some plans for ssl on beta cluster? [12:26:12] I don't like exposing my passwords :/ [12:41:47] (03PS1) 10Sitic: Reduce font size to 1.4rem, improved traditional watchlist layout [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224776 (https://phabricator.wikimedia.org/T104606) [12:42:50] (03CR) 10Sitic: [C: 032 V: 032] Reduce font size to 1.4rem, improved traditional watchlist layout [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224776 (https://phabricator.wikimedia.org/T104606) (owner: 10Sitic) [14:47:41] 6Labs, 3Labs-Sprint-103, 3Labs-Sprint-104, 3Labs-Sprint-105: In openstack upstream, add project_id to instance metadata - https://phabricator.wikimedia.org/T103384#1453988 (10Andrew) This was merged upstream, finally. [14:47:55] 6Labs, 3Labs-Sprint-103, 3Labs-Sprint-104, 3Labs-Sprint-105: In openstack upstream, add project_id to instance metadata - https://phabricator.wikimedia.org/T103384#1453990 (10Andrew) 5Open>3Resolved [14:48:31] 6Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: Remove [?] links from Special:NovaInstance - https://phabricator.wikimedia.org/T105770#1453996 (10scfc) What default link? And what is the harm caused? [14:49:23] petan: You shouldn't be using anything but a garbage password for beta. [14:50:11] 6Labs, 10Labs-Infrastructure: Once we have Liberty: remove project-id logic from designate/ldap plugin, use project_id in metadata instead. - https://phabricator.wikimedia.org/T105891#1454004 (10Andrew) 3NEW [14:50:12] meh... but still imagine how many people actually use same pw there as on prod [14:50:27] 6Labs, 10Labs-Infrastructure: Once we have Liberty: remove project-id logic from designate/ldap plugin, use project_id in metadata instead. - https://phabricator.wikimedia.org/T105891#1454011 (10Andrew) 5Open>3stalled [14:50:29] it's a security hole at some point [14:50:39] self signed would be fine [14:51:28] 6Labs, 10Labs-Infrastructure: Once we have Liberty: remove project-id logic from designate/ldap plugin, use project_id in metadata instead. - https://phabricator.wikimedia.org/T105891#1454004 (10Andrew) p:5Triage>3Normal a:3Andrew [14:52:30] petan: Not arguing (and there are other reasons to do SSL - not least of which is that's what we do elsewhere) [14:53:14] I will let you know when I see hashar and I will deliver that message to them [14:53:14] @notify hashar pls do some ssl on beta <3 tvm [14:53:59] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1454022 (10Steinsplitter) 5Open>3Resolved [14:56:48] andrewbogott: btw, when you were talking with CristianCantoro you probably meant role::labs::lvm::srv [14:57:12] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1454029 (10Superyetkin) 5Resolved>3stalled [14:57:36] What did I say? (I thought I copy/pasted whatever I suggested) [14:57:44] role::labs::lvm::lvm [14:57:50] hm [14:58:07] I was thinking that as well. Out of date docs? [14:58:28] nah, I was looking at the ‘configure instance’ page. So I don’t know why I typed the wrong thing. [14:58:46] 6Labs, 10Tool-Labs: missing database on replica server - https://phabricator.wikimedia.org/T105713#1454038 (10Superyetkin) [14:58:52] :) [15:15:46] 6Labs, 10Labs-Infrastructure: create ldap record and metadata entry synchronously with instance creation - https://phabricator.wikimedia.org/T102905#1454085 (10Andrew) [15:15:47] 6Labs, 10Labs-Infrastructure: Add tenant_id to instance metadata service - https://phabricator.wikimedia.org/T103097#1454084 (10Andrew) 5Open>3Resolved [15:17:11] Coren|MX, YuviPanda, what’s going on backupwise? Working? Need code review for backup scripts? [15:21:00] petan: we have bugs for SSL on beta cluster. But I am not working on it [15:21:20] s/not/now [15:21:21] :P [15:22:40] andrewbogott: not sure - I left comments on Coren|MX's patch [16:31:54] 6Labs, 6operations, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1454350 (10Andrew) I just ran a simple test on labvirt1005 (with 3.13), and was able to make it lock up on the first try. So now I'm ready to try a different kernel. [16:36:28] 6Labs, 6operations, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1454377 (10yuvipanda) Install linux-generic-lts-vivid package and reboot? [16:44:41] andrewbogott: Yuvi did a round of code review, I'm going to push a new changeset today. [16:44:58] andrewbogott: That said, we can start another - the changes to the script are mostly stylistic and it does work. [16:48:58] andrewbogott: YuviPanda: I did just that - started a new backup. screen session on 1002 [16:49:25] great! [16:50:55] !log tools.heritage Checked out pywikibot-core [16:50:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [16:51:27] YuviPanda: Got 10 minutes to talk nfs? [16:53:41] Coren|MX: in 15mins? [16:54:39] halfak: https://phabricator.wikimedia.org/T76375 [16:54:55] Sure thing. [16:55:59] 6Labs, 7Tracking: New project: ORES - https://phabricator.wikimedia.org/T105908#1454465 (10Halfak) 3NEW [17:00:15] 6Labs, 6operations, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1454502 (10Andrew) Starting with a 3.13 system... # apt-get install linux-generic-lts-vivid # apt-get install linux-image-3.19 linux-headers-3.19 # apt-get dist-upgrade # puppet ag... [17:00:42] 10Tool-Labs-tools-Other: tools-info connecting to wrong database server - https://phabricator.wikimedia.org/T105911#1454503 (10Sitic) 3NEW [17:01:57] YuviPanda: After food, then? [17:02:47] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1454531 (10Nemo_bis) 5stalled>3Resolved Superyetkin, your issue is not about replication and is already tracked in its own report; please don't reopen this one. [17:36:11] 6Labs, 6operations, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1454617 (10Andrew) labvirt1005 with 3.19.0-22-lowlatency has survived quite a few cycles of suspend/resume. So I'm convinced that it does not exhibit that particular bug, at least. [17:37:40] 6Labs: something has regressed with cmdline/horizon instance boot - https://phabricator.wikimedia.org/T105916#1454619 (10Andrew) 3NEW a:3Andrew [17:38:01] 6Labs: something has regressed with cmdline/horizon instance boot - https://phabricator.wikimedia.org/T105916#1454628 (10Andrew) p:5Triage>3High [17:46:27] Hi guys! [17:46:27] I was just running an script I made in march-april which runs into several wp languages databases, separately and at the same time using "union all". [17:46:27] I am surprised because I notice it is much slower than before. What have happened? And, could u speed it up somehow by changing some user parameters? Otherwise it is impossible to run the tests... [17:56:34] marmick: it would be best if you create a ticket for that and paste the entire SQL query there [17:57:35] mutante: it is not one query but many. hundreds. maybe one of the queries is taking longer than the others...my experience is with the entire script. [17:58:00] what used to take 5 min, now it's taking 1h. [17:58:06] or even more [17:59:14] marmick: then paste the entire script [17:59:44] or well,, link to the repo [17:59:49] repo? [18:00:04] isn't the script checked in somewhere? [18:00:05] i will create the ticket [18:00:16] ok, coolk [18:00:31] who should I refer to? [18:01:08] don't worry about the "who", just describe the problem with as much detail as possible [18:01:15] then people will add tags [18:46:30] I want to reset my Admin password from the command line for my MediaWiki installation ...does anybody have any advice? [18:48:26] Howie_, maintenance/changePassword.php ? [18:49:03] Krenair: much thanks. This needs to be run by root right? [18:49:57] Howie_, no [18:52:12] I get the following http://pastebin.com/1wuipDyB ... deprecated etc. but it looks like it works. [18:55:39] find /etc/php5/cli/conf.d/ -name "*.ini" -exec sed -i -re 's/^(\s*)#(.*)/\1;\2/g' {} \; [18:55:45] this should fix it :) [18:55:53] no, i didn't write it myself, but it's a common thing [18:56:08] what does that do? [18:56:50] Howie_: it replaces # with ; in all files ending in .ini in that directory [18:56:57] you can also safely ignore it [18:57:09] it's just that PHP wants you now use ; instead of # for comment lines [18:57:34] 6Labs, 10Labs-Infrastructure: Instance names with underscores are weirdly broken - https://phabricator.wikimedia.org/T105927#1454841 (10Andrew) 3NEW [18:57:39] you could also open /etc/php5/cli/conf.d/20-xhprof.ini in a text editor and just replace the # with ; [18:57:42] on line 2 [18:58:38] http://stackoverflow.com/questions/14074101/getting-comments-starting-with-are-deprecated-message-via-cli [18:59:19] ok lol [19:00:03] Ok. but then what is the ---> PHP Warning: Module 'xhprof' already loaded in Unknown on line 0 [19:00:37] somehow there are 2 places where it gets loaded [19:02:03] so when it is told to do it another time it just says "already got it" [19:02:16] i don't know where specifically [19:02:37] but i'd grep for "xhprof" through the entire /etc/php5/ [19:02:57] andrewbogott: Do we have some black magic we can recover files from an image? [19:03:06] 6Labs: something has regressed with cmdline/horizon instance boot - https://phabricator.wikimedia.org/T105916#1454855 (10Andrew) This turns out to have been T105927. [19:03:18] Coren|MX: ‘image’? [19:03:21] andrewbogott: Presuming the instance is dead enough to not start SSH (but its userspace is alive) [19:03:28] oh — yes. [19:03:29] one minute... [19:03:49] Coren|MX: https://wikitech.wikimedia.org/wiki/OpenStack#Mounting_an_instance.27s_disk [19:03:52] like that? [19:04:30] Coren|MX: you also might try to use salt to prod it back to life if it’s up and running [19:04:54] Aaah. the salt master might be up! Didn't think of that. ty [19:05:45] labcontrol1001 is the salt master these days. [19:15:43] andrewbogott: I'm a little worried by what I see. sshd on that instance no longer starts after a sshd_config change applied by puppet; but the instance owner doesn't actually use puppet. [19:16:00] andrewbogott: Did you do something in puppet that could affect the sshd config in general? [19:16:19] I don’t think so, although it could have to do with resolv.conf. [19:16:23] What’s the project and instance? [19:17:00] maybe by "not use" he means "actively disabled"? [19:17:57] mutante: No, just no puppet config for the instance beyond the base classes. [19:18:15] sshd -d gives: Unsupported KEX algorithm "curve25519-sha256@libssh.org" [19:18:15] [19:18:20] (03PS1) 10Jean-Frédéric: Prettify files using AutoPEP8 [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/224867 [19:18:32] So cipher changes. [19:21:05] andrewbogott: from /etc/ssh/sshd_config line 22 [19:21:14] * Coren|MX tries to find the changeset [19:22:13] ehm.. that's been a while since we changed that [19:22:21] unless there was another one [19:23:32] https://gerrit.wikimedia.org/r/#/c/185321/ [19:24:09] <% if scope.function_os_version(['debian >= jessie || ubuntu >= trusty']) %> [19:24:14] Coren|MX: which distro? [19:24:24] it should only be applied on those above, and works there [19:24:43] unless something happened with the "function_os_version" itself [19:39:02] 6Labs, 6operations, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1454934 (10Andrew) Oh, except on 3.19.0, resuming an instance doesn't work. It says it's resuming but actually never works again. [19:41:08] mutante: That might just be a case of outdated sshd [19:41:57] Internet here is teh suxx0rz [19:44:19] 6Labs, 7Tracking: New project: ORES - https://phabricator.wikimedia.org/T105908#1454964 (10yuvipanda) A sacrificial haiku dedicated to @yuvipanda is required before a project can be created [19:46:31] Coren|MX: it should definitely work on jessie and trusty, and if another version it should not set that option [19:49:35] mutante: Definitely a precise box [19:50:12] So it's not clear why that gets applied then. It also means that it's possible every precise vm is currently in a precarious position [19:50:25] * Coren|MX checks, out of paranoia [19:50:57] 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1455010 (10yuvipanda) [19:50:58] 6Labs, 7Tracking: New project: ORES - https://phabricator.wikimedia.org/T105908#1455007 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done [19:54:23] mutante: It's not set on other precise instances that I can see. Oddness. [19:57:24] Coren|MX: that's weird, we had the OS check in there from the beginning [19:58:02] i'd manually set it to: [19:58:11] KexAlgorithms diffie-hellman-group-exchange-sha256 [19:58:24] then see if puppet keeps it that way [19:59:23] mutante: All I gots is salt access, and even that is intermittent for some reason. Ima try a sed [20:00:06] oh, i forgot you cant get on it [20:00:44] maybe apt-get remove and install ssh again via salt? [20:01:00] eh, with purge so the config comes fresh from package [20:01:58] you can probably get away with s/KexAlgorithms/#KexAlgorithms/ too [20:02:09] just dont specify it [20:12:50] Yeah, I'm trying that now [20:18:14] Things are made more fun by salt being intermitent [20:21:45] Negative24: about? [20:22:11] 6Labs, 10wikitech.wikimedia.org: Cannot select different project in Special:NovaProject - https://phabricator.wikimedia.org/T105945#1455114 (10scfc) 3NEW [20:37:48] 6Labs: Recurrent annual survey for Labs/Tool Labs - https://phabricator.wikimedia.org/T105948#1455149 (10leila) 3NEW a:3leila [20:40:21] 6Labs, 10wikitech.wikimedia.org: Cannot select different project in Special:NovaProject - https://phabricator.wikimedia.org/T105945#1455167 (10Krenair) WFM [20:44:29] 6Labs: Recurrent annual survey for Labs/Tool Labs - https://phabricator.wikimedia.org/T105948#1455176 (10scfc) [20:44:31] 6Labs, 10Tool-Labs, 6Learning-and-Evaluation: Organize a (annual?) toollabs survey - https://phabricator.wikimedia.org/T95155#1455177 (10scfc) [21:05:39] 6Labs, 10Tool-Labs: "Fatal Error: Database query failed" on "related changes" tool - https://phabricator.wikimedia.org/T105953#1455246 (10He7d3r) 3NEW a:3Erwin [21:07:34] 6Labs: Recurrent annual survey for Labs/Tool Labs - https://phabricator.wikimedia.org/T105948#1455263 (10leila) thanks @scfc for deduplication. sorry that I missed the other one. [21:07:49] 6Labs, 10Tool-Labs, 6Learning-and-Evaluation: Organize a (annual?) toollabs survey - https://phabricator.wikimedia.org/T95155#1455268 (10leila) a:3leila [21:27:39] Hi all [21:30:45] 6Labs, 10wikitech.wikimedia.org: Cannot select different project in Special:NovaProject - https://phabricator.wikimedia.org/T105945#1455338 (10scfc) Tested again (including logging out and in) and it still doesn't allow me to change the project filter. [21:44:49] 10Tool-Labs-tools-Database-Queries, 7Database: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#1455384 (10marcmiquel) 3NEW [22:14:47] (03CR) 10Multichill: [C: 032 V: 032] "Awesome" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/224867 (owner: 10Jean-Frédéric) [22:27:33] chasemp: hmm? [22:31:37] chasemp: I was going to move the differential role over to the module (I completely agree). Didn't get time to do it however. [22:51:07] RECOVERY - Puppet staleness on tools-bastion-01 is OK Less than 1.00% above the threshold [3600.0] [22:56:41] 6Labs, 10Tool-Labs: support python3 uwsgi apps - https://phabricator.wikimedia.org/T104374#1455630 (10Ricordisamoa) {meme, src=votecat, above=Ricordisamoa, below="needs py3k"} [23:56:27] (03PS1) 10Sitic: Add option to hide own edits [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224980 (https://phabricator.wikimedia.org/T105937)