[00:21:57] Hey folks. Looks like quarry is having an outage. Is this a known thing? [00:22:57] I'm seeing a "Can't connect to MySQL server on 'enwiki.labsdb' ((1045, u"Access denied for user 'u2029'@'10.68.17.11' (using password: YES)"))" [00:23:39] halfak: yikes. that may have something to do with db perms things that madhuvishy was working on today [00:24:15] I bet she is afk right now [00:24:26] Ah hmmm [00:25:01] I'll look in a bit halfak [00:25:18] OK thanks :) [00:25:56] Want me to file a task quick? [00:26:20] halfak: tasks are always good [00:26:25] * halfak works on one [00:27:09] 10Quarry: Quarry can't connect to DB (permission denied) - https://phabricator.wikimedia.org/T161182#3123890 (10Halfak) [00:29:55] hi [00:30:05] bd808: halfak madhuvishy fixed my hack and so quarry is broken :) [00:30:10] am going to fix it!!! [00:30:19] <3! [00:30:47] I really need to move Quarry to tools [00:31:22] yuvipanda: sounds like a good plan [00:31:31] Yeah I invoked yuvipanda since I'm at gym :) [00:32:40] I think that Yvette 'fixed' the docs removing the potential issues that kept quarry out of tools before [00:33:26] https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Tools/Rules&diff=prev&oldid=1726430 [00:33:55] lol [00:33:58] white lists [00:34:08] I suppose that it's better than a white list [00:34:15] bd808: yeah, that was my last locker but now it's just time [00:34:25] * halfak wants to know what command invokes yuvipanda [00:35:10] $ sudo fixthings -yuvi "panda" [00:35:14] pip install -U yuvipanda && python -m yuvipanda wake [00:35:19] :D [00:35:57] 06Labs, 10Quarry, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry and PAWS are allowed to be hosted there - https://phabricator.wikimedia.org/T152212#3123937 (10bd808) 05Open>03Resolved a:03MZMcBride [00:36:16] bd808: halfak I think it involves ancient telephony technolog [00:36:17] y [00:36:41] 867-5309? [00:37:13] bd808: halfak should be fixed now. can one person verify? [00:37:30] yay! seems to work! [00:38:19] halfak: cool! thanks :) [00:40:28] 10Quarry: Quarry can't connect to DB (permission denied) - https://phabricator.wikimedia.org/T161182#3123963 (10Halfak) 05Open>03Resolved p:05Triage>03Unbreak! [00:40:31] Thanks yuvipanda. Also thanks madhuvishy for responding even when you are at the gym. :) [00:40:54] This is one of my favorite IRC channels :) [00:41:41] 10Quarry: Quarry can't connect to DB (permission denied) - https://phabricator.wikimedia.org/T161182#3123965 (10Halfak) a:03yuvipanda [00:42:00] 10Quarry: Quarry can't connect to DB (permission denied) - https://phabricator.wikimedia.org/T161182#3123890 (10Halfak) @yuvipanda did something to fix it. Thanks! [00:42:19] Sorry for the spam. Trying to keep the phab history happy. [00:42:22] Np :) also thanks yuvipanda [00:42:23] halfak: :) np! [00:42:43] * halfak gets back to halfaking [00:43:22] yuvipanda: i assume your replica.my.cnf works? :) [00:44:17] madhuvishy: ah, not sure :) I just switched quarry to use the quarry tool's creds [00:44:44] Ah okay [00:58:53] 10Quarry: Quarry can't connect to DB (permission denied) - https://phabricator.wikimedia.org/T161182#3123980 (10yuvipanda) I had used my personal user credentials for quarry, and @madhuvishy asked me a few hours ago if she can reset it and I said yes :) Quarry now uses the credentials of tool 'quarry', which is... [00:59:36] \o/ for documentation [03:44:28] yuvipanda: Hm.. do you know what https://phabricator.wikimedia.org/T152043 is blocked on? [03:44:45] I think it's in limbo right now, even if it can't be priority, it should at least be in someone's backlog :) [03:44:49] is it legal again? [03:45:43] I assume not, since they were already approved. Even in non-anonymous fashion. That was rectified. [03:47:18] Krinkle: have you seen our backlog :/ [03:47:45] 751 tasks in the triage column [03:55:14] 06Labs, 10Tool-Labs, 10DBA, 07Regression: Tool Labs: Add skin, language, and variant to user_properties_anon - https://phabricator.wikimedia.org/T152043#2836353 (10bd808) Does anybody know how to accomplish this change to the user_properties_anon view? [04:12:41] Where's maintain-replicas gone now? [04:15:46] bd808: The table you're looking for is properties_anon_whitelist seemingly [04:15:52] https://github.com/wikimedia/puppet/blob/production/modules/role/files/labs/db/views/maintain-meta_p.py#L104-L105 [04:16:22] There was a line in the maintain-replicas.py that did up_property like pw_property [04:16:51] * bd808 has no idea how any of this magic works [04:17:48] Ooh [04:17:49] ops.write_execute("INSERT INTO meta_p.properties_anon_whitelist VALUES ('gadget-%');") [04:17:56] yeah. just spotted that [04:18:24] so apparently we need to patch that and then run it for everything? [04:18:39] looks to be, yeah [04:19:39] Reedy: want to drop a note on the task so my feeble old brain remembers this in 12 hours? [04:19:41] Patching the script looks easy [04:19:54] or be the hero and make the patch :) [04:22:16] done [06:33:33] PROBLEM - Puppet run on tools-exec-1423 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:13:35] RECOVERY - Puppet run on tools-exec-1423 is OK: OK: Less than 1.00% above the threshold [0.0] [10:15:52] 06Labs, 10wikitech.wikimedia.org: Requesting 'content administrator' access for hashar - https://phabricator.wikimedia.org/T161202#3124491 (10hashar) [10:18:48] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Rebalance tools exec nodes with an eye towards CPU usage - https://phabricator.wikimedia.org/T161006#3124503 (10hashar) The scheduler now spread the Nodepool instances across multiple Compute nodes and CI was responsive again yesterday. Looks like the hack... [11:38:43] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/AKBYS was created, changed by AKBYS link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/AKBYS edit summary: Created page with "{{Tools Access Request |Justification=To develop new tools. |Completed=false |User Name=AKBYS }}" [12:33:40] !log wikispeech Deploy latest from Git master: e41c0c2 (T146459, T158954) [12:33:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL [12:33:45] T146459: Wikispeech.playOrStop() triggers twice if button is in focus - https://phabricator.wikimedia.org/T146459 [12:33:45] T158954: Use XPath to get text nodes related to utterances - https://phabricator.wikimedia.org/T158954 [12:54:50] what should i do when creating the new instance to obtain the replica.my.cnf file in my ~ ? [12:59:04] Danny_B: can you explain in more detail? we don't issue replica.my.cnf files for anything other than the tools project [12:59:24] and we acknowledge that means people will need to use those creds for their person or their tool in another context [12:59:33] as long as it's kept to your use only it's just the way it is atm [13:50:55] 06Labs, 15User-Addshore: Delete Cognate labs project - https://phabricator.wikimedia.org/T161217#3124997 (10Addshore) [13:53:40] 06Labs, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Create new labs project for development & testing of the FileImport & FileExport extensions - https://phabricator.wikimedia.org/T161218#3125019 (10Addshore) [15:53:47] 10Tool-Labs-tools-Other: Migrate http://toolserver.org/~dispenser/* to Tool Labs - https://phabricator.wikimedia.org/T68868#3125424 (10Dispenser) [15:53:51] 06Labs, 10DBA, 06Operations, 07Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#3125426 (10Dispenser) [15:53:55] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Tracking: Missing Toolserver features in Tools (tracking) - https://phabricator.wikimedia.org/T60791#3125425 (10Dispenser) [15:53:59] 06Labs, 10Tool-Labs: Provide namespace IDs and names in the databases similar to toolserver.namespace - https://phabricator.wikimedia.org/T50625#3125422 (10Dispenser) 05Resolved>03Open Use View as suggested by Coren: `mysql -h tools.labsdb toolserverdb_p` > ERROR 1049 (42000): Unknown database 'toolserverd... [16:00:14] 06Labs, 10DBA, 06Operations, 07Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#3125440 (10jcrespo) [16:00:16] 06Labs, 10Tool-Labs: Provide namespace IDs and names in the databases similar to toolserver.namespace - https://phabricator.wikimedia.org/T50625#3125439 (10jcrespo) [16:37:53] !log tools migrating tools-webgrid-lighttpd-1402 and 1407 to labvirt1001 (testing labvirt1001 and easing CPU load on labvirt1010) [16:37:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:42:53] PROBLEM - Host tools-webgrid-lighttpd-1402 is DOWN: CRITICAL - Host Unreachable (10.68.16.35) [16:54:38] marxarelli|afk: where is that etherpad? :) [17:07:05] RECOVERY - Host tools-webgrid-lighttpd-1402 is UP: PING OK - Packet loss = 0%, RTA = 1.54 ms [17:10:07] PROBLEM - Host tools-webgrid-lighttpd-1407 is DOWN: CRITICAL - Host Unreachable (10.68.17.251) [17:22:12] (03CR) 10Paladox: Add phabricator-upstream to #wikimedia-dev and #wikimedia-devtools (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333553 (owner: 10Paladox) [17:22:15] (03PS5) 10Paladox: Add phabricator-upstream to #wikimedia-dev and #wikimedia-devtools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333553 [17:46:59] RECOVERY - Host tools-webgrid-lighttpd-1407 is UP: PING OK - Packet loss = 0%, RTA = 0.61 ms [19:01:59] !log tools migrating tools-exec-1407 to labvirt1001 [19:02:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:05:44] PROBLEM - Host tools-exec-1407 is DOWN: CRITICAL - Host Unreachable (10.68.18.16) [19:50:47] RECOVERY - Host tools-exec-1407 is UP: PING OK - Packet loss = 0%, RTA = 1.41 ms [19:56:04] !log tools migrating tools-exec-1408 to labvirt1001 [19:56:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:59:20] PROBLEM - Host tools-exec-1408 is DOWN: CRITICAL - Host Unreachable (10.68.18.14) [20:12:13] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ederporto was created, changed by Ederporto link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Ederporto edit summary: Created page with "{{Tools Access Request |Justification=I'm working with NeuroMat, and my activities now include developing less generalized tools for the project. |Completed=false |User Name=E..." [20:35:45] RECOVERY - Host tools-exec-1408 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [20:38:50] !log tools migrating tools-exec-1401 to labvirt1001 [20:38:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:43:42] PROBLEM - Host tools-exec-1401 is DOWN: CRITICAL - Host Unreachable (10.68.17.202) [20:44:01] 10Wikibugs: Improve tags used in IRC messages - https://phabricator.wikimedia.org/T161249#3126402 (10Mattflaschen-WMF) [21:22:14] RECOVERY - Host tools-exec-1401 is UP: PING OK - Packet loss = 0%, RTA = 1.10 ms [21:28:11] 06Labs, 10Labs-Infrastructure, 10DNS, 06Discovery, and 2 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256#3126688 (10grin) [21:41:25] Has anyone ever encountered this error on wmflabs: [21:41:27] django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: libpq.so.5: cannot open shared object file: No such file or directory [21:42:53] psycopg2 loads fine in a python shell, but not when using "webservice start" [21:56:00] tobias47n9e: virtualenv / memory / working dir / pythonpath would be my first guesses [21:56:54] valhallasw`cloud: Yeah I was thinking about that too. Probably need another setting. The file is at /usr/lib/libpq.so.5 [22:00:41] valhallasw env = LD_LIBRARY_PATH="/usr/lib" might be the correct setting :thinking: [22:01:55] tobias47n9e: you shouldn't need to do that [22:02:11] are you using a virtualenv? If so, try recreating it [22:02:36] check if the file is also there on the webgrid node where it's running [22:03:07] (libpq.so.5) [22:03:44] I'm not sure if you can easily increase the memory allocation, so that might be somewhat harder to check (qacct will be able to tell you, but its syntax is... arcane) [22:04:51] valhallasw`cloud: Yes virtualenv. I wrote down all the steps here: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/My_first_Django_OAuth_tool#Login_and_deploy [22:05:09] oh, you're running it on kubernetes [22:05:20] in that case, postgres might not be installed [22:05:58] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:05:59] (I'm also not sure why it requires postgres -- I assume you're using mysql) [22:06:52] you could try just uninstalling psycopg2 [22:06:59] (in the venv) [22:07:06] valhallasw`cloud: I actually need postgres [22:07:19] Or rather I would like to use postgres [22:07:49] Postgres is running on pgsql.eqiad.wmnet [22:08:01] Ok. I think then the best thing to do is to file a bug that the postgres libraries are not installed in the kubernetes container [22:08:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:09:42] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:10:07] valhallasw`cloud: Ok I can do that. Or I could just settle for sqlite. Will have to do some thinking [22:10:34] depends on the workload, as sqlite would be on nfs [22:10:47] if it's more than trivial loads, it will make our server unicorns cry [22:12:53] valhallasw`cloud: It is a webservice that the Commons app would use to look up requested pictures. That is around 10k installs of semi-active users. But I don't know enough about scaling to estimate the workload that will produce. [22:13:27] I'd go for the real database [22:15:07] Ok. Then I will file the bug and see what can be done. Thanks for your help again! [22:24:10] 06Labs, 10Tool-Labs: Add dependencies for Postgresql to Kubernetes container - https://phabricator.wikimedia.org/T161266#3126960 (10Tobias1984) [22:36:26] 06Labs, 10Tool-Labs, 06Operations, 13Patch-For-Review: ssl certificate/key update: *.tools.wmflabs.org (expires on 2017-03-24) - https://phabricator.wikimedia.org/T160187#3127011 (10madhuvishy) 05Open>03Resolved Is done now! [22:40:58] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:44:43] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:48:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0]