[00:01:50] 06Labs, 10Labs-Infrastructure: Wikitech: SAL transclusion on project pages should be limited - https://phabricator.wikimedia.org/T50065#2552322 (10AlexMonk-WMF) 05Open>03Resolved a:03Krinkle https://wikitech.wikimedia.org/w/index.php?title=Template:Nova_Project&diff=prev&oldid=120278 [00:02:22] 06Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: Make wikitech an openid provider - https://phabricator.wikimedia.org/T49067#2552328 (10AlexMonk-WMF) [00:06:35] 06Labs, 10Labs-Infrastructure: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#2552331 (10AlexMonk-WMF) [00:07:16] PROBLEM - Puppet run on tools-k8s-master-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:08:03] 06Labs, 10Labs-Infrastructure, 10media-storage: Provide Swift object store(s) for the labs projects - https://phabricator.wikimedia.org/T114998#1712021 (10AlexMonk-WMF) This is now out of date as the beta cluster has it's own internal swift install. [00:09:21] 06Labs, 10Labs-Infrastructure: puppet::self broken - https://phabricator.wikimedia.org/T128930#2090480 (10AlexMonk-WMF) Has this happened lately? [00:09:46] 06Labs, 10Tool-Labs: Install pdf2djvu for Wikisource DjVu aid - https://phabricator.wikimedia.org/T130138#2552338 (10AlexMonk-WMF) [00:10:07] 06Labs, 10Quarry: Long-running Quarry query (querry?) produces strangely incorrect results - https://phabricator.wikimedia.org/T135087#2552339 (10AlexMonk-WMF) [00:11:09] 06Labs, 10Labs-Infrastructure: Abolish use of labs proxies in domains other than .wmflabs.org - https://phabricator.wikimedia.org/T131290#2162138 (10AlexMonk-WMF) I think we should go in the opposite direction - e.g. use LE to get certs for any domains outside of *.wmflabs.org necessary, and abolish the use of... [00:13:47] 06Labs, 10Adminbot: Get a cloak for morebots & labs-morebots - https://phabricator.wikimedia.org/T140547#2552343 (10AlexMonk-WMF) [00:16:59] 06Labs, 10Labs-Infrastructure: Delete staging labs project - https://phabricator.wikimedia.org/T137726#2376776 (10AlexMonk-WMF) >>! In T137726#2376890, @yuvipanda wrote: > The domain staging.wmflabs.org needs to be deleted manually, according to @andrew `designate --os-tenant-name staging domain-delete stagin... [00:18:44] 06Labs, 10Labs-Infrastructure: Avoid indefinite growing of apt caches and old kernel images - https://phabricator.wikimedia.org/T127374#2552346 (10AlexMonk-WMF) [00:23:16] 06Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#727373 (10AlexMonk-WMF) I haven't seen this issue occur in a long while [00:32:31] (03PS1) 10BryanDavis: Move staticfiles submodule mount point [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/304753 [00:44:04] (03PS1) 10BryanDavis: Allow configuration of static_root [labs/striker] - 10https://gerrit.wikimedia.org/r/304754 [00:46:17] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Public IPs not being updated from OpenStack Nova plugin - https://phabricator.wikimedia.org/T52620#577207 (10AlexMonk-WMF) A quick look through the nova source code reveals that the notification types we listen for in wikistatus (`compute.instance.create_ip.... [00:46:27] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Public IPs not being updated from OpenStack Nova plugin - https://phabricator.wikimedia.org/T52620#2552353 (10AlexMonk-WMF) 05stalled>03Open [00:46:49] 06Labs, 10Labs-Infrastructure: Move tenant management out of ldap (after Kilo but before Mitaka) - https://phabricator.wikimedia.org/T100213#2552354 (10AlexMonk-WMF) [00:49:36] (03CR) 10BryanDavis: [C: 032] Allow configuration of static_root [labs/striker] - 10https://gerrit.wikimedia.org/r/304754 (owner: 10BryanDavis) [00:50:22] 06Labs, 10Labs-Infrastructure, 06Operations: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#2552368 (10AlexMonk-WMF) Was this completed? ```krenair@bastion-01:~$ ldapsearch -x "(&(objectClass=novauser)(!(loginShell=/bin/bash)))" # extended LDIF # # LDAPv3 # base... [00:54:23] 06Labs, 10Labs-Infrastructure, 06Operations: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#2552371 (10AlexMonk-WMF) Ugh, wrong objectClass: ```krenair@bastion-01:~$ ldapsearch -x "(&(objectClass=person)(!(loginShell=/bin/bash)))" | grep dn: | grep ou=people | gr... [00:59:03] 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Set up LVS for labs dns recursors - https://phabricator.wikimedia.org/T119660#2552373 (10AlexMonk-WMF) See also T133389 [01:00:54] (03PS1) 10BryanDavis: Use static rather than bootstrap_css and bootstrap_javascript tags [labs/striker] - 10https://gerrit.wikimedia.org/r/304755 [01:02:33] (03CR) 10BryanDavis: [C: 032] Use static rather than bootstrap_css and bootstrap_javascript tags [labs/striker] - 10https://gerrit.wikimedia.org/r/304755 (owner: 10BryanDavis) [01:02:47] 06Labs, 10Labs-Infrastructure, 10Salt: update salt key monitoring scripts for labs to new nova api version - https://phabricator.wikimedia.org/T123607#1933624 (10AlexMonk-WMF) This appears to refer to modules/openstack/files/monitor_labs_salt_keys.py [01:16:34] 06Labs, 10Labs-Infrastructure, 10Salt: update salt key monitoring scripts for labs to new nova api version - https://phabricator.wikimedia.org/T123607#2552379 (10AlexMonk-WMF) Try this? 100% untested ```lang=diff diff --git a/modules/openstack/files/monitor_labs_salt_keys.py b/modules/openstack/files/monitor... [01:24:57] 10MediaWiki-extensions-OpenStackManager: "The specified resource does not exist" when you try to configure an instance and are not a projectadmin - https://phabricator.wikimedia.org/T67379#2552382 (10AlexMonk-WMF) [01:28:26] 06Labs, 10Labs-Infrastructure: Automatically updated list of all configured domains - https://phabricator.wikimedia.org/T45580#2552383 (10AlexMonk-WMF) Maybe we could just make a cron that updates a wikitech page, or leaves a text file accessible under wikitech.wikimedia.org or something. [01:41:26] 06Labs, 10Labs-Infrastructure, 06Operations, 07IPv6: Enable ipv6 on labs - https://phabricator.wikimedia.org/T37947#399081 (10AlexMonk-WMF) >>! In T37947#399351, @scfc wrote: > http://permalink.gmane.org/gmane.org.wikimedia.labs/2651: > > | > Of particular interest would be to hear if there are plans to I... [01:43:02] 06Labs, 10Labs-Infrastructure: Add more network nodes - https://phabricator.wikimedia.org/T41787#435174 (10AlexMonk-WMF) This is quite an old task. Quantum appears to be an old name for Neutron, but of course we're not using Neutron right now (I think there's some other relevant tasks lying around about this s... [01:52:20] !log wikistats major cleanup of backend data. deleted and fixed hundreds (redirects, duplicates, broken wikis, convert to API, ..) [01:52:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL, Master [02:17:20] !log wikistats deleted 'nostalgia' from wikipedias table per request (it's in the 'specials' table already) [02:17:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL, Master [02:18:54] 06Labs, 10Labs-Infrastructure: Don't rely on wikitech API for production services - https://phabricator.wikimedia.org/T104575#1421040 (10AlexMonk-WMF) modules/labstore/files/nfs-exports-daemon: url = ("https://wikitech.wikimedia.org/w/api.php" + Labs hiera stuff: modules/wmflib/lib/hiera/mwcache.rb: @... [02:26:33] 06Labs, 10Labs-Infrastructure, 10Monitoring, 06Operations: Have a paging check for Nova API accessible - https://phabricator.wikimedia.org/T133656#2238563 (10AlexMonk-WMF) It's not just as simple as checking whether http://labnet1002.eqiad.wmnet:8774 is up, is it? [02:30:32] 06Labs, 10Labs-Infrastructure: Empty default security group for newly created project - https://phabricator.wikimedia.org/T136871#2350514 (10AlexMonk-WMF) ```krenair@silver:~$ nova secgroup-list-default-rules +-------------+-----------+---------+----------+ | IP Protocol | From Port | To Port | IP Range | +---... [02:43:59] (03CR) 10BryanDavis: [V: 031] Use static rather than bootstrap_css and bootstrap_javascript tags [labs/striker] - 10https://gerrit.wikimedia.org/r/304755 (owner: 10BryanDavis) [02:45:09] (03CR) 10BryanDavis: Use static rather than bootstrap_css and bootstrap_javascript tags [labs/striker] - 10https://gerrit.wikimedia.org/r/304755 (owner: 10BryanDavis) [02:45:17] (03CR) 10BryanDavis: [C: 032 V: 031] Use static rather than bootstrap_css and bootstrap_javascript tags [labs/striker] - 10https://gerrit.wikimedia.org/r/304755 (owner: 10BryanDavis) [02:46:35] (03Merged) 10jenkins-bot: Allow configuration of static_root [labs/striker] - 10https://gerrit.wikimedia.org/r/304754 (owner: 10BryanDavis) [02:47:13] (03Merged) 10jenkins-bot: Use static rather than bootstrap_css and bootstrap_javascript tags [labs/striker] - 10https://gerrit.wikimedia.org/r/304755 (owner: 10BryanDavis) [02:49:09] 06Labs, 10Labs-Infrastructure, 10Horizon: Disassociating floating IP does not show it as available in the horizon project quota summary - https://phabricator.wikimedia.org/T136872#2552424 (10bd808) >>! In T136872#2552307, @AlexMonk-WMF wrote: > I think this is because when you disassociate an IP, it's still... [02:57:36] 06Labs, 10Labs-Infrastructure, 10labs-sprint-117, 13Patch-For-Review: Give 'novaobserver' keystone account rights to read everything, everywhere, write or change nothing - https://phabricator.wikimedia.org/T104588#1421660 (10AlexMonk-WMF) Would we want to allow labs instances to access keystone (or even de... [02:58:54] (03PS1) 10BryanDavis: Bump Striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/304757 [03:02:19] 06Labs, 10Labs-Infrastructure, 10Horizon: Disassociating floating IP does not show it as available in the horizon project quota summary - https://phabricator.wikimedia.org/T136872#2552439 (10AlexMonk-WMF) >>! In T136872#2552424, @bd808 wrote: >>>! In T136872#2552307, @AlexMonk-WMF wrote: >> I think this is b... [03:02:47] 10Labs-project-Wikistats: wikistats (labs project): rename Image column to Files - https://phabricator.wikimedia.org/T142732#2552440 (10Dzahn) [03:19:35] 06Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: Make wikitech an openid provider - https://phabricator.wikimedia.org/T49067#2552443 (10bd808) This is probably something that we could do via #striker a bit easier than via wikitech. If we can come up with a list of things that it would allow/unblock I... [03:22:15] 06Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: Make wikitech an openid provider - https://phabricator.wikimedia.org/T49067#2552445 (10bd808) On the other hand, having a stable and trusted MediaWiki extension that implements an OpenID provider could allow some really interesting things such as makin... [03:30:08] (03CR) 10BryanDavis: [C: 032] Bump Striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/304757 (owner: 10BryanDavis) [03:30:27] (03CR) 10BryanDavis: [C: 032] Move staticfiles submodule mount point [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/304753 (owner: 10BryanDavis) [03:30:33] (03Merged) 10jenkins-bot: Move staticfiles submodule mount point [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/304753 (owner: 10BryanDavis) [03:30:36] (03Merged) 10jenkins-bot: Bump Striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/304757 (owner: 10BryanDavis) [03:55:45] (03PS1) 10BryanDavis: Change public_html symlinks to point to new static location [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/304760 [03:59:51] (03CR) 10BryanDavis: [C: 032] Change public_html symlinks to point to new static location [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/304760 (owner: 10BryanDavis) [03:59:57] (03Merged) 10jenkins-bot: Change public_html symlinks to point to new static location [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/304760 (owner: 10BryanDavis) [04:02:14] 10Labs-project-Wikistats, 13Patch-For-Review: fix broken links in largest_html (was: Update lietuvai.lt statistics URLs) - https://phabricator.wikimedia.org/T136183#2552457 (10Dzahn) for ELIP it now links to http://lietuvai.lt/wiki/Special:Recentchanges , http://lietuvai.lt/wiki/Special:Listadmins etc. Th... [04:02:29] 10Labs-project-Wikistats, 13Patch-For-Review: fix broken links in largest_html (was: Update lietuvai.lt statistics URLs) - https://phabricator.wikimedia.org/T136183#2552459 (10Dzahn) 05Open>03Resolved [04:16:05] 06Labs, 10Labs-Infrastructure: Delete staging labs project - https://phabricator.wikimedia.org/T137726#2552462 (10demon) 05Open>03declined Actually, don't delete this. I'm using it again ;-) [05:43:21] LALALALALLALALA [05:43:24] JE [05:43:25] M [05:43:28] TUVO SEXO [05:44:25] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:54:56] PROBLEM - Puppet staleness on tools-exec-1211 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:59:00] PROBLEM - Puppet staleness on tools-exec-1213 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:59:12] PROBLEM - Puppet staleness on tools-exec-1204 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [06:33:13] PROBLEM - Puppet run on tools-worker-1015 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:03:20] 06Labs, 06Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Rename specific account in LDAP, Wikitech, Gerrit and Phabricator - https://phabricator.wikimedia.org/T85913#2552603 (10adrianheine) @hashar: Since I left WMDE, both addresses will be bouncing. Just remove me from the list. [07:08:14] RECOVERY - Puppet run on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [07:12:43] 10Labs-project-Wikistats, 13Patch-For-Review: fix broken links in largest_html (was: Update lietuvai.lt statistics URLs) - https://phabricator.wikimedia.org/T136183#2552606 (10Nemo_bis) Very nice, most wikis are good now. [07:17:51] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:52:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:57:04] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL - No data received from host [08:02:05] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.046 second response time [08:11:15] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [08:13:14] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [08:14:22] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [08:51:35] PROBLEM - Puppet staleness on tools-k8s-master-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [10:44:07] 06Labs, 10Tool-Labs, 10WLX-Jury, 13Patch-For-Review, 10Wiki-Loves-Monuments (2016): Figure out a way to support java 1.8 on tool labs (For WLX Jury) - https://phabricator.wikimedia.org/T124903#2552899 (10intracer) 05Open>03Resolved [11:46:53] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:58:31] Hello, [11:59:02] When I try to connect to my database in tool labs using mysql, I get this error [11:59:05] PHP Warning: mysql_connect(): Connection refused in /data/project/durl-shortener/public_html/php-scripts/db-connection.php on line 14 [11:59:22] All my credentials are correct, what could be the problem [11:59:25] ? [11:59:32] d3r1ck: what server are you connecting to? [12:00:19] valhallasw`cloud: 10.68.23.58 [12:00:27] thats the hostname I am using [12:00:47] or should it be localhost? [12:01:09] Because I checked that from the mariadb consolve and got that host name [12:01:24] 10.68.23.58 is tools-bastion-03.tools.eqiad.wmflabs. [12:01:39] 12:00:46 0 ✓ zhuyifei1999@tools-bastion-02: ~$ host 10.68.23.58 [12:01:39] 58.23.68.10.in-addr.arpa domain name pointer tools-bastion-03.tools.eqiad.wmflabs. [12:01:39] 12:00:54 0 ✓ zhuyifei1999@tools-bastion-02: ~$ host tools-bastion-03.tools.eqiad.wmflabs [12:01:39] tools-bastion-03.tools.eqiad.wmflabs has address 10.68.23.58 [12:02:03] ohh, ok [12:02:05] d3r1ck: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Connecting_to_the_database_replicas [12:02:16] my tool is durl-shortener. [12:02:39] how can i know the hostname? [12:02:59] .labsdb [12:03:14] eg enwiki.labsdb [12:03:18] in your case, you're probably looking for https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#User_databases [12:03:37] although I'm not sure how an url shortener fits within the intended use case of tool labs [12:04:50] valhallasw`cloud: I deployed it to test something online [12:06:02] zhuyifei1999_: I didn't understand your point. Is that suppose to be the hostname? [12:06:19] yes [12:07:15] or you can use tools.labsdb [12:07:48] afaik, there're no automatic backups, so you gotta handle backups yourself [12:07:57] ok [12:08:26] try not to hardcode ip addresses, they are subject to change [12:08:43] zhuyifei1999_: so my database is like this: url_shortener [12:08:45] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [12:09:10] so the host should be url_shortener.labsdb [12:09:20] see https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#User_databases as valhallasw`cloud suggested [12:09:25] no [12:10:23] Ok [12:10:41] .labsdb is only for replicated databases, tools.labsdb, s.labsdb (idk if there are more) [12:11:18] if you don't intend to do anything with replicas you can try tools.labsdb [12:11:57] ok [12:13:42] then CREATE DATABASE __url_shortener [12:19:12] is the one in your replica.my.cnf, usually in the form of s [12:21:52] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:44:13] 06Labs, 15User-Luke081515: Revert: Request increased quota for rcm labs project - https://phabricator.wikimedia.org/T142311#2553417 (10chasemp) >>! In T142311#2550415, @Luke081515 wrote: > (Claiming, that I don't forget this task). Actually I set up a new instance, and will now clone from the old. When I'm fin... [15:22:42] 06Labs, 10Dumps-Generation: Put pageviews dataset in labs /public/dumps - https://phabricator.wikimedia.org/T142671#2553486 (10Ottomata) [15:23:30] 06Labs, 10Dumps-Generation: Put pageviews dataset in labs /public/dumps - https://phabricator.wikimedia.org/T142671#2553491 (10Nuria) Puppet code that needs to change: https://github.com/wikimedia/operations-puppet/blob/f4f87efce1fcaab1296f2b9ee31a429b391af6c3/modules/role/manifests/labs/nfs/misc.pp [15:23:41] 06Labs, 10Analytics, 10Dumps-Generation: Put pageviews dataset in labs /public/dumps - https://phabricator.wikimedia.org/T142671#2553492 (10Nuria) [15:24:12] 06Labs, 10Analytics: Put pageviews dataset in labs /public/dumps - https://phabricator.wikimedia.org/T142671#2553498 (10Ottomata) [15:45:37] does OAuth work with Labs? I always get: ERROR: OAuth authentication not supported: No module named requests_oauthlib [15:45:56] but if I run it directly through command line, it works [15:45:56] mafk use a virtualenv :) [15:46:17] what do you mean by 'Labs'? where is it running? [15:46:18] yuvipanda: I'm using the shared python files, do I still need virtualenv? [15:46:26] oh, pywikibot? [15:46:33] yep [15:46:41] ah, then I have no idea, sorry. [15:46:51] in general I'd suggest everyone use a virtualenv for everything [15:47:01] * mafk headesks [15:47:08] if you are using jsub you might also need '-l release=trusty' so that it doesn't run on super old precise [15:47:22] oh, I will test that [15:47:40] jsub -N test -l release=trusty versiontest.sh ? [15:47:52] yeah [15:48:23] * mafk tests [15:50:44] yeah, I think requests_oauthlib is installed on trusty but not precise [15:51:06] valhallasw`cloud precise EOL is coming up sooon [15:51:48] yes, april 17 [15:51:57] we had a whole plan for this at some point [15:52:09] valhallasw`cloud yeah, https://phabricator.wikimedia.org/T94792 [15:52:11] and then got lost in fires :| [15:52:46] yuvipanda: still writes to test.err [15:53:07] ah. [15:53:39] mafk: pywikibot outputs most stuff to stderr, and stderr gets written to .err [15:53:54] or do you mean you still get the error...? [15:55:20] valhallasw`cloud: http://pastebin.com/kwHLXi8G [15:55:28] that I got [15:55:30] I think it's fine now? [15:55:35] mafk: I think so [15:55:54] PYWIKIBOT2_DIR: Not set isn't concerning right? [15:56:18] I'll try to run another script [15:56:23] and see what happens [15:56:38] No. (note that pywikibot-specific questions are better for #pywikibot, more people with the relevant knowledge there) [15:57:21] okay :D [15:57:52] yuvipanda: any idea what might be scary about things switching from precise to trusty? python venvs are the only thing that comes to mind for me [15:58:51] bd808 php 5.3 -> 5.5 [15:59:03] bd808 there's a bunch of deprecated functions that are removed finally [15:59:07] that should be fine really [15:59:17] right [15:59:21] oh ... I guess there are some. really really old shit [15:59:40] * yuvipanda nods [16:02:10] looks like 5.3 -> 5.4 had most of the breaking stuff [16:21:00] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 15User-bd808: Make jsub / qsub default to trusty instances - https://phabricator.wikimedia.org/T94792#2553739 (10bd808) a:03bd808 Updated the draft plan that @yuvipanda had written at https://wikitech.wikimedia.org/wiki/Tools_Precise_deprecation with new d... [16:23:33] 06Labs, 10Tool-Labs, 10Mail: Move tools-mail to trusty - https://phabricator.wikimedia.org/T96299#1213660 (10bd808) Note: tools-mail is a OpenGridEngine submit host, so we can't migrate it to jessie. [16:30:07] 06Labs, 10Tool-Labs, 10Mail: Move tools-mail to trusty - https://phabricator.wikimedia.org/T96299#2553774 (10bd808) [16:33:12] 06Labs, 10DBA, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2553792 (10DStrine) [16:35:09] (03CR) 10Dzahn: [C: 031] "Yes, as long as this just excludes operations/debs/wikistats and not the other operations/debs, yes that is what i wanted and thank you fo" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/304746 (owner: 10Paladox) [16:45:09] 06Labs, 10DBA, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2297843 (10AndyRussG) >>! In T135405#2374114, @Base wrote: > The tables in the desc looks like indeed the most useful ones, but I would like all the... [16:46:35] 06Labs, 10DBA, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2553836 (10Jseddon) [16:47:07] 06Labs, 10DBA, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2297843 (10Jseddon) [17:51:32] 06Labs, 10Continuous-Integration-Infrastructure: Investigate upgrade of OpenStack python module for labnodepool1001 - https://phabricator.wikimedia.org/T143013#2554094 (10thcipriani) [17:51:59] Labs Instance creation still broken for me (https://phabricator.wikimedia.org/T142742). Should I assign this to someone or will it get triaged in time? [17:54:45] jaufrecht have you tried using horizon.wikimedia.org? [17:56:09] No, trying it now. [17:56:16] I had thought it was just for DNS or something. [17:56:23] 06Labs, 10Continuous-Integration-Infrastructure: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2554147 (10thcipriani) [17:56:47] It says I'm at my limit of CPUs, even though I have instances left in quota. [17:57:08] yep, so horizon produces better error messages at least. [17:57:19] each quota is individual [17:57:20] you can't exceed any quota [17:57:28] yeah, so you need to have quota available in all the aspects (CPU, RAM) to be able to create new instances [17:57:41] apologies for the unclear error message on wikitech, we're in the process of deprecating it completely in favor of horizon [17:59:49] I can juggle instances to get back below 9 CPUs, but I need some quota space in order to do so. [18:00:09] also maxed out on ram [18:00:19] could someone bump me up? [18:01:02] jaufrecht we stopped doing them ad hoc because that was causing issues, there's a process for that now, documented at https://phabricator.wikimedia.org/T140904 [18:01:07] sorry that's all new [18:01:48] "Requests are processed by the Labs team during the Labs team meeting every Monday (8:30 AM PST) that the meeting is held." grumble grumble [18:03:48] 06Labs, 10Continuous-Integration-Infrastructure: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2554186 (10Paladox) Are we looking for https://github.com/openstack-infra/nodepool/blob/877265912853e0f7a6a21e3771fe9250f71e9c26/nodepool/nodepool.py#L57 has it been configured to be lo... [18:05:01] 06Labs, 10Continuous-Integration-Infrastructure: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2554189 (10thcipriani) [18:06:52] 06Labs, 10Continuous-Integration-Infrastructure: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2554218 (10Paladox) [18:06:55] 06Labs, 10Continuous-Integration-Infrastructure: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2554219 (10Paladox) [18:06:58] 06Labs, 10Continuous-Integration-Infrastructure: Investigate upgrade of OpenStack python module for labnodepool1001 - https://phabricator.wikimedia.org/T143013#2554220 (10Paladox) [18:08:44] 06Labs: Request increased quota for labs project - https://phabricator.wikimedia.org/T143020#2554228 (10JAufrecht) [18:09:00] 06Labs: Request increased quota for labs project - https://phabricator.wikimedia.org/T143020#2554242 (10JAufrecht) [18:09:23] Filed https://phabricator.wikimedia.org/T143020. Is it possible to have this reviewed early, since I just missed the cutoff? [18:09:39] 06Labs: Request increased quota for Phlogiston labs project - https://phabricator.wikimedia.org/T143020#2554228 (10JAufrecht) [18:09:56] 06Labs: Request increased quota for Phlogiston labs project - https://phabricator.wikimedia.org/T143020#2554228 (10JAufrecht) [18:09:58] 06Labs, 10Phlogiston (Interrupt): Create new Phlogiston-01 instance - https://phabricator.wikimedia.org/T142277#2554245 (10JAufrecht) [18:10:15] 06Labs, 10Phlogiston (Interrupt): Create new Phlogiston instance for production - https://phabricator.wikimedia.org/T142277#2529844 (10JAufrecht) [18:20:44] 06Labs, 10Continuous-Integration-Infrastructure: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2554306 (10Paladox) We are using nodepool version 0.1.1, we should update to 0.3.0. [18:43:06] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2554426 (10RobH) a:05yuvipanda>03madhuvishy I signed via hangout, and @madhuvishy listed off her key fingerprint. I've signed and pushed to keyserv... [18:43:44] 06Labs, 10Labs-Infrastructure, 10DBA: labsdb* has no automatic failover solution - https://phabricator.wikimedia.org/T141097#2554432 (10chasemp) Brief proposal for the discussion: * new labsdb boxes * haproxy (somewhere?) * service url for reaching these new labsdb boxes to access replicas * service url for... [18:58:02] 06Labs, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2554504 (10greg) [18:58:12] 06Labs, 10Labs-Infrastructure, 06Operations, 07Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2554505 (10greg) [18:58:24] 06Labs, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: Investigate upgrade of OpenStack python module for labnodepool1001 - https://phabricator.wikimedia.org/T143013#2554506 (10greg) [18:58:33] 06Labs, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2554512 (10greg) [19:12:53] !log tools kill unused tools-merlbot-proxy [19:12:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:14:09] PROBLEM - Host tools-merlbot-proxy is DOWN: CRITICAL - Host Unreachable (10.68.16.208) [19:47:43] 06Labs, 06Operations, 06Release-Engineering-Team, 10wikitech.wikimedia.org, 07LDAP: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2554836 (10demon) [20:04:19] 06Labs, 06Operations, 10Ops-Access-Requests: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2554886 (10RobH) [20:09:49] !log tools.xtools Restarted xtools webservice [20:09:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.xtools/SAL, Master [20:22:45] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2554964 (10AlexMonk-WMF) I've bumped labtestcontrol2001 mysql's max_connections from 151 to 500, to match real-labs' value of 500 on m5-master (db1009). You should no lo... [20:26:11] musikanimal hey [20:26:20] do you know we're collecting pageview stats for tools individually now? [20:27:12] where do you store them? [20:27:29] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2554985 (10AlexMonk-WMF) If anything else in labtest spams root@, let me know and I'll take a look. [20:27:43] gifti graphite [20:27:57] for tools on k8s you can see that (and memory / CPU / network usage) on https://grafana-labs-admin.wikimedia.org/dashboard/db/kubernetes-tool-combined-stats [20:29:59] nice, unfortunately i don't use php [20:30:12] ? [20:30:13] oh [20:30:17] for kubernetes you mean? [20:30:31] for webservice what do you use? [20:30:45] python, node, java are available already [20:30:47] tcl [20:31:00] ah, right [20:31:07] what's the name of the tool (for webservice, that is) [20:31:17] giftbot [20:31:37] ok [20:31:47] I'm going to make a trusty image soon [20:31:50] that'll have tcl (and all the things) in it [20:31:55] woo [20:33:46] when will we use kubernetes for things other than webservices? [20:34:29] gifti depending on what level of use you want to use it for, you can already [20:34:34] you can already use kubectl [20:34:36] oh [20:34:38] we don't have a jsub backend yet [20:34:46] tom29739 has been doing a lot of work with it [20:35:25] gifti for your webservice, do you use only tcl? [20:35:29] or tcl and other languages as well? [20:35:51] tcl only [20:36:09] and the directory listing [20:36:19] and fcgi [20:36:30] right [20:36:39] maybe I can make a tcl specific container image [20:40:38] gifti https://gerrit.wikimedia.org/r/#/c/304889/ I am making that now [20:41:06] thx [20:43:03] yuvipanda: can you delete draft queries in quarry? [20:43:13] nope [20:43:17] you can't really delete anything in quarry [20:43:23] duh, ok [20:47:53] gifti: you can clear the content of a quarry, IIRC there is no history, and use that draft later [20:48:12] gifti I'm going to try to switch giftbot to k8s now [20:48:28] Luke081515: nice idea, haha [20:48:51] * Luke081515 did this a few times [20:49:03] there is history actually. it's just not exposed [20:56:44] gifti I'm moving it now [20:56:52] ok [20:58:56] found a bug, fixing [21:05:12] gifti ah, I see you're using a version of tcl you compiled yourself. any spefic reason? [21:05:26] unicode reasons [21:05:33] hmhm [21:05:51] gifti since this is a much newer build of TCL (jessie), can we try it with the system tcl in here first? [21:06:45] my compile uses a specific branch of the tcl source code, still [21:07:18] gifti so /shared is /data/project/shared now, will work across both gridengine and k8s. [21:07:20] this is a headache [21:07:28] ok [21:07:38] so if you want to continue using that, can you update the path? [21:07:46] yes [21:08:05] gifti is there also a reason for it being in /data/project/shared instead of in your tool's homedir? [21:08:41] i provide it for tools.taxonbot, too [21:08:50] ah ok [21:09:20] i wonder if i could use ~shared [21:10:25] ok, that doesn't work ^^ [21:10:36] yeah, I think that'll at best resolve to /home/shared [21:10:37] which isn't there [21:11:01] oh, right, that would be ~tools.shared [21:13:40] gifti I Just changed it for the .fcgi files, let's see if that works [21:14:47] gifti ok, it isn't crashing instantly now. can you check it out and tell me if the webservice works? [21:15:05] let's see [21:15:38] directory listing works, scripts don't [21:15:45] 403 and 500 [21:17:46] I see [21:17:47] can't find package Fcgi [21:17:48] while executing [21:17:48] "package require Fcgi" [21:17:53] do you know where it is searching for these packages? [21:18:50] ah, i guess it might be in /data/project/shared [21:18:58] i have to adapt the config file [21:19:41] * yuvipanda nods [21:19:43] can you do that? [21:20:15] um, ok, i remembered that wrong [21:20:31] let me search for Fcgi [21:21:33] sorry for all the hassle! this hopefully ends in a nicer enviornment in the end [21:21:45] (and once you have this started you can already use kubernetes via kubectl!) [21:22:49] yuvipanda: is there a possibility to use k8s interactively? [21:23:03] for debugging purposes >.> [21:23:41] yeah [21:23:49] try 'webservice shell' [21:23:57] thx [21:24:51] i might have to recompile tclsh [21:25:04] % set auto_path [21:25:05] /shared/tcl/lib/tcl8.6 /shared/tcl/lib /data/project/shared/tcl/lib [21:25:05] oh I see [21:25:31] let's restart and see [21:26:24] hm, ok, Fcgi doesn't seem to come from my side (/shared/tcl) [21:27:42] hmm [21:32:19] sorry, it does, finally found it [21:33:08] woot! [21:35:17] % lappend auto_path /data/project/shared/tcl/lib/tcl8.6 [21:35:17] /usr/share/tcltk/tcl8.6 /usr/share/tcltk /data/project/shared/tcl/lib /usr/local/lib/tcltk /usr/local/share/tcltk /usr/lib/tcltk/x86_64-linux-gnu /usr/lib/tcltk /usr/lib/tcltk/tcl8.6 /usr/lib /data/project/shared/tcl/lib/tcl8.6 [21:35:21] % package require Fcgi [21:35:23] this is puzzling me [21:35:25] couldn't load file "/data/project/shared/tcl/lib/tcl8.6/tclfcgi0.8/libtclfcgi0.8.so": libtcl8.5.so.0: cannot open shared object file: No such file or directory [21:36:00] yuvipanda: maybe we need a tcl8.5 install? [21:37:31] or i could try to compile it with tcllib8.6 [21:37:53] gifti you aren't using the system installed tcl at all [21:37:57] right [21:37:59] you're using your own tcl [21:38:09] so it shouldn't matter what's in the image [21:38:29] oh, it's just that tcl8.5 is hardcoded into the Fcgi Makefile [21:38:29] but yeah, not sure why it's trying to use tcl8.5. From looking at puppet even gridengine only has 8.6? [21:38:36] ah [21:39:07] but in fact there is a tclsh8.5 [21:39:56] 06Labs, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2555319 (10Paladox) Should this be high priority due to it affecting ci nodepool, preventing us from using nodepool until we find and fix mos... [21:39:57] indeed [21:39:58] I see that [21:40:03] I've no idea where that comes from tho [21:40:09] since only 8.6 is installed from puppet :| [21:41:53] ok, compiling against 8.6 doesn't work [21:42:36] :( [21:42:45] yuvipanda: can you pull tcl8.5 into that image? [21:42:58] 06Labs, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2555344 (10hashar) Nodepool has a 60 seconds **timeout** for the API requests. If OpenStack replies quickly with a 4xx/5xx it process with the next iteration in... [21:43:04] yeah, let me replace 8.6 with 8.5 [21:43:34] doing so now [21:43:52] 06Labs, 10Labs-Infrastructure: Clean up leaked designate entries - https://phabricator.wikimedia.org/T120797#2555349 (10AlexMonk-WMF) Since these domains are owned by noauth-project, I don't think you can use the usual clients, so I hit the REST API directly: http://developer.openstack.org/api-ref/dns/ This s... [21:44:22] yuvipanda: iirc regular tclsh8.6 in tools isn't even the upstream tclsh8.6 [21:44:37] gifti this one is definitely upstream 8.6 I think, coming from debian [21:44:53] ok, this is also troubling [21:45:04] 06Labs, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2555351 (10Paladox) @hashar we should update nodepool to see if it fixes some of our problems, including patching in some of your changes that still need merging... [21:45:08] because sometimes i have to use that [21:45:24] or not [21:45:30] i don't even know anymore [21:45:36] all the differences [21:46:12] i guess my own has a few more patches [21:46:20] that aren't in the repo [21:47:12] gifti if you do webservice restart now, new image will have 8.5 [21:54:03] 06Labs, 10Labs-Infrastructure, 06Operations, 07Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2555374 (10AlexMonk-WMF) I've written a script to hopefully purge the vast majority of problematic entries in T120797 [22:00:58] 06Labs, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2555388 (10hashar) What is the command to get that 32k instances result? The only thing I see approaching is the usage metric: ``` (openstac... [22:03:25] 06Labs, 10Labs-Infrastructure: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2555392 (10yuvipanda) Actual ssh failing just now in: ``` (clush)yuvipanda@fearless:~/code/clush$ clush -l root --hostfile=files -b 'echo hi' extdist-02.eqiad.wmflabs: ssh_exchange_identification: Connectio... [22:08:27] 06Labs, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: Investigate upgrade of OpenStack python module for labnodepool1001 - https://phabricator.wikimedia.org/T143013#2555399 (10hashar) Here are roughly the OpenStack related python modules that Nodepool depends on: ``` python-cinderclient... [22:10:18] !log tools depool tools-exec-1211 and 1205, seem to be out of action [22:10:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [22:10:25] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 15User-greg: Create incident report for CI outage on Aug 10th - https://phabricator.wikimedia.org/T142887#2555420 (10hashar) 05Open>03Resolved a:03greg Published at https://wikitech.wikimedia.org/wik... [22:15:43] yuvipanda: libfcgi.so.0: cannot open shared object file: No such file or directory [22:15:55] something seems still to be missing [22:15:57] let me see where libfcgi comes from [22:16:44] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Delete ci-trusty-wikimedia-278848 instance in contintcloud project - https://phabricator.wikimedia.org/T143058#2555452 (10hashar) [22:16:54] gifti where are you getting that from btw? [22:16:55] when running? [22:17:48] when i do 'package require Fcgi' from /data/project/shared/tcl/bin/tclsh8.6 in 'webservice shell' [22:17:58] right [22:18:50] am installing libfcgi-dev [22:18:53] now [22:18:54] moment [22:19:01] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Delete ci-trusty-wikimedia-278848 instance in contintcloud project - https://phabricator.wikimedia.org/T143058#2555452 (10Paladox) I think this may be fixed in a update to nodepool. I think I saw something that introduces the ability to... [22:20:00] yuvipanda: btw, are there direct links for the download button in quarry? [22:20:09] *download data [22:20:10] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Delete ci-trusty-wikimedia-278848 instance in contintcloud project - https://phabricator.wikimedia.org/T143058#2555477 (10hashar) That is unrelated to Nodepool. Using the openstack CLI client, the instance can not be deleted either :-/ [22:20:22] what do you mean? [22:20:50] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Delete ci-trusty-wikimedia-278848 instance in contintcloud project - https://phabricator.wikimedia.org/T143058#2555478 (10Paladox) Oh sorry, maybe it is fixed in openstack cli? [22:20:59] ugh [22:21:11] *uh [22:21:34] gifti try restarting again [22:21:48] when a query is complete there is a "download data" button with some options [22:22:10] oh, found what i need [22:22:12] nvm [22:24:10] yuvipanda: works! [22:24:17] \o/ [22:36:55] 06Labs, 10Labs-Infrastructure: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2555517 (10chasemp) breakdown: ```extdist-02 16a0fd3f-c751-4686-8c77-e84b58d42eed debian-8.0-jessie confd-precise 7966c5ea-ac06-47bd-9f0b-b6c142de1650 ubuntu-12.04-precise db2003 a1c342e1-8725-46a1-a116-74b... [22:54:00] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Delete ci-trusty-wikimedia-278848 instance in contintcloud project - https://phabricator.wikimedia.org/T143058#2555452 (10AlexMonk-WMF) @Paladox, if an instance is stuck in ERROR state that indicates a problem not on the client side :) [23:53:44] 10Tool-Labs-tools-Pageviews, 03Community-Tech-Sprint: Restrict Topviews to showing data only for individual days or months - https://phabricator.wikimedia.org/T142403#2555845 (10MusikAnimal)