[00:05:46] yuvipanda: I have a problem with the grid. [00:06:01] For some reason none of the PHP scripts are starting. [00:06:40] yuvipanda: http://pastebin.com/dArGgzYQ [00:07:49] It would seem my script is only getting 1.25MB allocated. [00:08:02] Though I have the script allocate 128M. [00:08:58] bd808: ^ [06:00:56] Cyberpower678: allocate more ram I guess [06:48:41] PROBLEM - Puppet run on tools-exec-gift-trusty-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:50:15] PROBLEM - Puppet run on tools-exec-1424 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:50:39] PROBLEM - Puppet run on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:23:43] RECOVERY - Puppet run on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:30:16] RECOVERY - Puppet run on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [07:30:42] RECOVERY - Puppet run on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:41] !log ores restart ores-web-03 [07:39:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [09:06:49] Is there a private channel where I can report a vulnerability of a tool? [10:15:45] 10Tool-Labs-tools-Xtools: "not valid wiki" error when I check Commons' contributions i X!s Tools - https://phabricator.wikimedia.org/T161427#3131912 (10Liuxinyu970226) [12:51:13] 10Tool-Labs-tools-Xtools: "not valid wiki" error when I check Commons' contributions in X!s Tools - https://phabricator.wikimedia.org/T161427#3131978 (10Ivanhercaz) [12:58:47] zhuyifei1999_: that is absolutely brilliant. :p. The problem is I don't know what's restricting the application to 1.25MB. The JSUB task has been increased to 256MB and the script itself is set to 128MB. [12:59:11] And then I get this error. 1.25MB isn't even enough to start up PHP. [12:59:47] the memory controlling mechanism is oge is weird [13:00:42] are you using the -mem option? [13:00:55] *in oge [13:06:29] oge? [13:07:01] Is that grid engine your are referring to? [13:07:05] If so yes. [13:10:41] yeah open grid engine [13:11:37] I guess you just allocate more ram (eg. to 512 mb) [13:12:24] I remember one of the reasons we are switching to k8s is that oge memory management is just weird [13:13:57] So 256m restricts to 1.25MB? That's not weird but just plan buggy/ [14:28:03] zhuyifei1999_: what if I just removed the -mem parameter? [14:28:08] I only need 128MB [14:28:15] there is a default [14:28:25] What is this default? [14:30:48] parser.add_argument( [14:30:48] '-mem', type=argparse_memkb, default=524288, [15:03:11] zhuyifei1999_: so doubling the memory requirements to 512m seems to have worked. [15:03:25] k [18:37:09] 06Labs, 10wikitech.wikimedia.org: Stop using an older release of SMW and get back to tracking their master branch - https://phabricator.wikimedia.org/T75940#3132221 (10bd808) [18:37:11] 06Labs, 10wikitech.wikimedia.org: Upgrade SMW to 1.9 or later - https://phabricator.wikimedia.org/T62886#3132223 (10bd808) [18:37:50] 06Labs, 10wikitech.wikimedia.org: Stop using an older release of SMW and get back to tracking their master branch - https://phabricator.wikimedia.org/T75940#786392 (10bd808) 05Open>03declined We are going to pursue solutions for {T53642} instead. [18:50:38] 06Labs, 10Tool-Labs: Add dependencies for Postgresql to Kubernetes container - https://phabricator.wikimedia.org/T161266#3126960 (10bd808) Why not switch to a pure python postgres driver like https://pypi.python.org/pypi/py-postgresql or https://pypi.python.org/pypi/pg8000 ? Usage of postgres in tool labs is g... [18:51:04] 06Labs, 10Tool-Labs: Add dependencies for Postgresql to Kubernetes container - https://phabricator.wikimedia.org/T161266#3132232 (10bd808) p:05Triage>03Normal [18:52:16] 06Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 06Operations, 10hardware-requests: Eqiad: Hardware request for labstore1006/7, dataset1002/3 - https://phabricator.wikimedia.org/T161311#3132234 (10bd808) p:05Triage>03Normal [18:54:30] 06Labs, 10Labs-Infrastructure, 10DNS, 06Discovery, and 3 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256#3132236 (10bd808) p:05Triage>03Normal [18:58:34] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Requesting 'content administrator' access for hashar - https://phabricator.wikimedia.org/T161202#3132241 (10bd808) 05Open>03Resolved a:03bd808 `{{done}}`. Glad to have more help. [19:09:18] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 07Tracking: Make webservice backend default to kubernetes - https://phabricator.wikimedia.org/T154504#2913589 (10bd808) [19:09:21] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Switch default backend for tools webservices to be kubernetes (Tracking) - https://phabricator.wikimedia.org/T149762#2762071 (10bd808) [19:11:06] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, 13Patch-For-Review: Unify k8s roles between prod and tools - https://phabricator.wikimedia.org/T158452#3132268 (10bd808) [19:11:08] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, 07kubernetes: Coordinate Kubernetes efforts between Tool Labs and Production - https://phabricator.wikimedia.org/T153943#3132267 (10bd808) [19:11:27] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, and 2 others: Coordinate Kubernetes efforts between Tool Labs and Production - https://phabricator.wikimedia.org/T153943#2895796 (10bd808) [19:16:29] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: `jsub` is not available to Tool labs webservice running on Kubernete - https://phabricator.wikimedia.org/T161089#3132273 (10bd808) 05Open>03declined >>! In T161089#3121209, @Tpt wrote: > Ok! Is it planned to have an easy way to submit job to k8s (I don't find any... [19:17:09] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: `jsub` is not available to Tool labs webservice running on Kubernete - https://phabricator.wikimedia.org/T161089#3132280 (10Tpt) Ok, thank you! [19:18:11] 06Labs, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Create new labs project for development & testing of the FileImport & FileExport extensions - https://phabricator.wikimedia.org/T161218#3132282 (10bd808) [19:18:16] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#3132283 (10bd808) [19:28:19] 06Labs, 10wikitech.wikimedia.org: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3132303 (10bd808) [19:31:19] Hi all, I just created a tool at tools-db but when I tried to use sql I got access denied error. My new toolname is mostlinkedmissing, service number is s53348. I tried sql local and sql cswiki. [19:31:24] Should I fill a ticket? [19:31:54] (mysql --defaults-file="${HOME}"/replica.my.cnf -h enwiki.labsdb enwiki_p do not work too) [19:32:39] see http://pastebin.com/UmvVuHSG too [19:37:14] 06Labs, 15User-Addshore: Delete Cognate labs project - https://phabricator.wikimedia.org/T161217#3132309 (10bd808) [19:38:16] 06Labs, 10Labs-Infrastructure: Investigate instances with high "steal" CPU - https://phabricator.wikimedia.org/T161118#3132311 (10bd808) p:05Triage>03Normal [19:39:07] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, 07kubernetes: Fully document process for building a new version of Kubernetes debs - https://phabricator.wikimedia.org/T161031#3132313 (10bd808) [19:39:10] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, and 2 others: Coordinate Kubernetes efforts between Tool Labs and Production - https://phabricator.wikimedia.org/T153943#3132314 (10bd808) [19:40:10] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, 07kubernetes: Fully document process for building a new version of Kubernetes debs - https://phabricator.wikimedia.org/T161031#3119203 (10bd808) p:05Triage>03Normal [19:46:42] Urbanecm: please file a bug. I'm pretty sure that as soon as the credentials file exists it should work. [19:47:19] bd808, I have replica.my.cnf file, it do not work. [19:47:25] But okay, going to file a bug. [19:48:22] we have done some work recently on the process that creates the replica.my.cnf files, so it is possible that there is some regression that is breaking things [19:49:49] 06Labs, 10Tool-Labs, 07Regression, 15User-Urbanecm: I have no access to DB with my recently created tool mostlinkedmissing - https://phabricator.wikimedia.org/T161466#3132333 (10Urbanecm) [19:50:01] 06Labs, 10Tool-Labs, 07Regression, 15User-Urbanecm: I have no access to DB with my recently created tool mostlinkedmissing - https://phabricator.wikimedia.org/T161466#3132346 (10Urbanecm) p:05Triage>03High [19:50:11] bd808, created ^^^^ [19:50:50] thanks [19:50:55] yw [19:51:12] I don't have access to the db servers to look for error logs there unfrotunately [19:52:23] 06Labs, 10Tool-Labs, 07Regression, 15User-Urbanecm: I have no access to DB with my recently created tool mostlinkedmissing - https://phabricator.wikimedia.org/T161466#3132347 (10Urbanecm) [19:56:21] 06Labs, 10Tool-Labs, 07Regression, 15User-Urbanecm: I have no access to DB with my recently created tool mostlinkedmissing - https://phabricator.wikimedia.org/T161466#3132352 (10bd808) @madhuvishy could you take a look and see if you can figure out the reason this is failing? I have a tool with credentials... [19:56:51] Urbanecm: ^ not just your problem. looks to be a bug somewhere [19:57:57] PROBLEM - Puppet run on tools-proxy-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:59:28] 10Labs-project-other, 10Math: Instances in math project show high system CPU usage - https://phabricator.wikimedia.org/T160824#3132359 (10bd808) [20:04:08] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Requesting 'content administrator' access for hashar - https://phabricator.wikimedia.org/T161202#3132360 (10hashar) Thanks. I have cleaned up the few leaded Nova Resource pages. Will help as I can. [20:05:21] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: Shut down "cewbot" - https://phabricator.wikimedia.org/T160264#3132361 (10bd808) [20:08:22] 06Labs, 15User-bd808: Request creation of getstarted labs project - https://phabricator.wikimedia.org/T160884#3132366 (10bd808) 05Open>03Resolved a:03bd808 https://wikitech.wikimedia.org/wiki/Nova_Resource:Getstarted [20:08:25] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#3132369 (10bd808) [20:13:28] 06Labs, 10DBA, 13Patch-For-Review, 07Regression: Tool Labs: Add skin, language, and variant to user_properties_anon - https://phabricator.wikimedia.org/T152043#3132374 (10bd808) [20:17:23] 06Labs, 10Tool-Labs: Add dependencies for Postgresql to Kubernetes container - https://phabricator.wikimedia.org/T161266#3132376 (10Tobias1984) The postgis database bacckend of Django is based on psycopg2. Probably it would need to be re-implemented in py-postgresql. Not sure how much effort that would take. W... [20:18:07] 06Labs, 10DBA, 10MediaWiki-extensions-Babel, 06Security-Team, 06WMF-Legal: Replicate babel db table on Labs - https://phabricator.wikimedia.org/T160713#3132379 (10bd808) ``` > describe babel; +-------------+------------------+------+-----+---------+-------+ | Field | Type | Null | Key |... [20:24:49] 06Labs, 10DBA, 10MediaWiki-extensions-Linter: Make "linter" table available on Labs - https://phabricator.wikimedia.org/T160611#3132386 (10bd808) ``` > describe linter; +---------------+------------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Ext... [20:31:53] 06Labs, 10Tool-Labs: Setup postgres credentials for all tool users - https://phabricator.wikimedia.org/T76167#792350 (10bd808) It would seem to me that if we did this we would be promoting postgres as a 'first class' service that we maintain and recommend for general usage. From what I have seen thus far our p... [20:33:36] 06Labs, 10DBA: Expose ar_content_format and ar_content_model columns of archive table on Labs replicas - https://phabricator.wikimedia.org/T89741#3132397 (10bd808) [20:37:55] RECOVERY - Puppet run on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:47:04] 06Labs, 10Tool-Labs: Still issues with node.js webservice - https://phabricator.wikimedia.org/T160353#3132430 (10bd808) * The python stack trace problem is tracked in {T156626} * Related to @valhallasw's comment about using `webservice restart`, `webservice stop` clears the contents of the "service.manifest" f... [20:52:17] 10Tool-Labs-tools-Other, 10Wikidata: Add interstitial to wikidata-externalid-url - https://phabricator.wikimedia.org/T160205#3132436 (10bd808) [20:58:41] 06Labs, 10MediaWiki-User-login-and-signup, 10wikitech.wikimedia.org: Fatal exception when attempting to log into Wikitech - https://phabricator.wikimedia.org/T160171#3132439 (10bd808) 05Open>03declined The error logged for this was: ``` [9005c35bbc1b3d296bacb04f] /w/index.php?title=Special:UserLogin&retu... [21:10:29] 06Labs: Lost Wikitech 2FA details, recovery needed - https://phabricator.wikimedia.org/T159521#3070340 (10bd808) @mschwarzer per https://wikitech.wikimedia.org/wiki/Password_reset#Reset_two_factor_authentication, your next step would be to create a text file in your $HOME directory on a Labs bastion server (e.g.... [21:16:45] 06Labs, 10Horizon: Show Already Used Hostnames in OpenStack Horizon - https://phabricator.wikimedia.org/T159460#3132453 (10bd808) p:05Triage>03Low As far as I know OpenStack does not track historical VM instance names, so to implement this we would need to make our own tracking system and connect it to the... [21:21:06] 06Labs, 10Tool-Labs, 07I18n: Internationalize Tool Labs' homepage - https://phabricator.wikimedia.org/T105590#3132459 (10bd808) p:05Triage>03Low [21:23:12] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: Build toollabs trusty 'catch all' container - https://phabricator.wikimedia.org/T152089#2837904 (10bd808) @yuvipanda should we decline this? As I understand it your attempt at making such a container were unsuccessful. [21:51:16] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: Provide namespace IDs and names in the databases similar to toolserver.namespace - https://phabricator.wikimedia.org/T50625#3132467 (10bd808) This is quite a feat of thread necromancy you have performed @Dispenser. It looks like you are probably reporting two sep... [21:51:42] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: Provide namespace IDs and names in the databases similar to toolserver.namespace - https://phabricator.wikimedia.org/T50625#3132469 (10bd808) a:05coren>03None [21:55:18] 06Labs, 10Tool-Labs, 07Upstream: exim breaks if current directory doesn't exist - https://phabricator.wikimedia.org/T160654#3106750 (10bd808) p:05Triage>03Low [22:16:37] 06Labs, 10Tool-Labs, 07Documentation, 15User-bd808: Document /data/project/.system/config/$TOOLNAME.web-memlimit for raising grid engine webservice memory limit - https://phabricator.wikimedia.org/T160189#3132480 (10bd808) 05Open>03Resolved a:03bd808 [22:18:58] 06Labs, 10Tool-Labs, 06Developer-Relations: Make sure abandoned useful tools are properly advertised so potentially interested new maintainers could find them - https://phabricator.wikimedia.org/T159595#3072793 (10bd808) I would suggest a workboard column or subproject of #tool-labs-standards-committee for t... [22:21:52] 06Labs: Close wmve-techteam - https://phabricator.wikimedia.org/T159589#3132488 (10bd808) [22:21:55] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#3132489 (10bd808) [22:30:44] 06Labs, 10Tool-Labs, 06translatewiki.net: update node.js on tools.telegrambot - https://phabricator.wikimedia.org/T159368#3065513 (10bd808) [[https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes|Migrating the tool to Kubernetes]] will get you node v6.9.1 instead of the very old version that we h... [22:39:45] 06Labs, 10Tool-Labs: Access Denied - https://phabricator.wikimedia.org/T160585#3132509 (10Tomthirteen) Hello, I'm trying to run a query: sql simplewiki_p -e "select rev_comment, rev_page from revision where rev_timestamp > 20160101000000 and rev_timestamp < 20170101000000" > VanIP.txt; But I need it to sa... [22:39:48] 06Labs, 07LDAP: Difficult to set up mock LDAP server in Labs - https://phabricator.wikimedia.org/T159277#3062676 (10bd808) Since labs instances use LDAP for authentication and authorization this is probably always going to be very tricky to impossibly hard. MediaWiki-Vagrant does offer `role::ldapauth` which s... [22:40:20] 06Labs, 07LDAP: Difficult to set up mock LDAP server in Labs - https://phabricator.wikimedia.org/T159277#3132512 (10bd808) p:05Triage>03Low [22:42:34] 06Labs: Delete test-spm-1.project-proxy - https://phabricator.wikimedia.org/T159257#3061977 (10bd808) @yuvipanda, the instance was created by you according to the nova metadata, so if you have no idea what it is for it can probably be deleted. ;) [22:53:17] 10MediaWiki-extensions-OpenStackManager, 05MW-1.29-release (WMF-deploy-2017-02-28_(1.29.0-wmf.14)), 13Patch-For-Review: MW OpenStackManager: add support for ED25519 SSH keys - https://phabricator.wikimedia.org/T159070#3132548 (10bd808) [22:59:04] 06Labs, 10Labs-Infrastructure: Wikitech error when adding users to projects - https://phabricator.wikimedia.org/T159021#3054591 (10bd808) The logged exception was: ``` UnexpectedValueException from line 196 of /srv/mediawiki/php-1.29.0-wmf.13/includes/user/UserGroupMembership.php: UserGroupMembership::insert()... [23:01:47] 06Labs: 'prometheus' service user vs. actual human account 'prometheus' - https://phabricator.wikimedia.org/T158968#3132573 (10bd808) [23:01:49] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Add "prometheus" to https://wikitech.wikimedia.org/wiki/MediaWiki:Titleblacklist - https://phabricator.wikimedia.org/T158971#3132570 (10bd808) 05Open>03Resolved a:03bd808 https://wikitech.wikimedia.org/w/index.php?title=MediaWiki:Titleblacklist&diff=17522... [23:08:12] 06Labs, 10Wikimedia-Labs-General, 07Upstream: tool-labs is not configured correctly to run phpunit tests - https://phabricator.wikimedia.org/T52222#3132589 (10bd808) 05stalled>03declined Install PHP dependencies using composer when and where needed. [23:08:55] 06Labs: Investigate multiarch differences between precise / trusty - https://phabricator.wikimedia.org/T86294#3132592 (10bd808) 05Open>03declined Precise is gone. Trusty things have been running fine for many years. [23:18:42] 06Labs, 10MediaWiki-extensions-OpenStackManager, 07Documentation, 07Puppet: doc.wikimedia.org puppet documentation for labs/lvm/srv.html gives a 404 - https://phabricator.wikimedia.org/T119329#3132598 (10bd808) 05Open>03declined https://doc.wikimedia.org/puppet/puppet_classes/role_3A_3Alabs_3A_3Alvm_3A...