[00:31:16] 10Labs-project-Wikistats: all kinds of mixed issues with miraheze table (was: allthetropes is not updating on wikistats) - https://phabricator.wikimedia.org/T146712#2859023 (10NDKilla) @Dzahn Please let me know if you need anything else from us. All APIs should be working at whatever they're $wgServer is set to... [00:35:37] 10Tool-Labs-tools-Pageviews: Query stats.grok.se for data older than July 2015 - https://phabricator.wikimedia.org/T149358#2859033 (10MusikAnimal) p:05Triage>03Normal [01:04:33] (03CR) 10BryanDavis: "recheck" [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/325972 (owner: 10BryanDavis) [01:04:53] (03CR) 10BryanDavis: [C: 032] contrib: Add ldapPublicKey when creating dummy users [labs/striker] - 10https://gerrit.wikimedia.org/r/325890 (owner: 10BryanDavis) [01:06:17] (03Merged) 10jenkins-bot: contrib: Add ldapPublicKey when creating dummy users [labs/striker] - 10https://gerrit.wikimedia.org/r/325890 (owner: 10BryanDavis) [01:06:39] (03CR) 10BryanDavis: [C: 032] profile: make less output nicer [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/325972 (owner: 10BryanDavis) [01:06:51] (03CR) 10BryanDavis: "recheck" [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/325973 (owner: 10BryanDavis) [01:07:13] (03Merged) 10jenkins-bot: profile: make less output nicer [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/325972 (owner: 10BryanDavis) [01:15:26] (03CR) 10BryanDavis: [C: 032] stashbot.sh: Add 'attach' command [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/325973 (owner: 10BryanDavis) [01:15:57] (03Merged) 10jenkins-bot: stashbot.sh: Add 'attach' command [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/325973 (owner: 10BryanDavis) [01:17:46] !log stashbot Switched git remote to https://gerrit.wikimedia.org/r/labs/tools/stashbot [01:17:47] Unknown project "stashbot" [01:17:47] Did you mean to say "tools.stashbot" instead? [01:17:55] !log tools.stashbot Switched git remote to https://gerrit.wikimedia.org/r/labs/tools/stashbot [01:17:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [01:27:36] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Software-Licensing: Expand the Tool Labs definition of "free license" to include FSF-approved and DFSG-compatible licenses - https://phabricator.wikimedia.org/T152581#2859075 (10ZhouZ) Hi bd808, I believe historically we only listed one-single reference (... [02:03:58] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Software-Licensing: Expand the Tool Labs definition of "free license" to include FSF-approved and DFSG-compatible licenses - https://phabricator.wikimedia.org/T152581#2859128 (10bd808) >>! In T152581#2859075, @ZhouZ wrote: > Are there particular licenses o... [02:19:28] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Software-Licensing: Expand the Tool Labs definition of "free license" to include FSF-approved and DFSG-compatible licenses - https://phabricator.wikimedia.org/T152581#2859136 (10Legoktm) I think the main ones in that list that people are going to want to u... [03:05:42] 06Labs, 10Tool-Labs: virtualenvs set umask to 022 - https://phabricator.wikimedia.org/T152759#2859178 (10scfc) [03:28:47] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Software-Licensing: Expand the Tool Labs definition of "free license" to include FSF-approved and DFSG-compatible licenses - https://phabricator.wikimedia.org/T152581#2859210 (10bd808) The OSI list also does not explicitly cover older versions of the appro... [04:17:21] 06Labs, 10Tool-Labs: Move aptly backups to a cron rather than puppet - https://phabricator.wikimedia.org/T150726#2794568 (10scfc) I don't think moving this to `cron` is a good solution. Looking at the Puppet log, it shows that on each run the same files are backed up again and again: ``` Dec 9 03:43:51 tool... [06:39:40] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:14:38] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [08:37:20] !log tools run delete-dbusers and force replica.my.cnf creation for all tools that did not have it [08:37:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:29:32] 06Labs, 06Operations: Missing Labs hiera entry in labs-private repo - https://phabricator.wikimedia.org/T152767#2859419 (10Volans) [09:30:50] (03PS1) 10Volans: Add missing Hiera for labspuppetbackend_mysql_password [labs/private] - 10https://gerrit.wikimedia.org/r/326082 (https://phabricator.wikimedia.org/T152767) [09:33:14] (03CR) 10Volans: [V: 032 C: 032] Add missing Hiera for labspuppetbackend_mysql_password [labs/private] - 10https://gerrit.wikimedia.org/r/326082 (https://phabricator.wikimedia.org/T152767) (owner: 10Volans) [09:48:57] 06Labs, 06Operations, 13Patch-For-Review: Missing Labs hiera entry in labs-private repo - https://phabricator.wikimedia.org/T152767#2859467 (10Volans) p:05High>03Normal a:05Volans>03None I've quickly added the missing one, the old one `labspuppetbackend::mysql_password` is still there and `hieradata/... [10:26:36] 06Labs, 10DBA: Querying the logging table on labs is slow - https://phabricator.wikimedia.org/T131266#2859500 (10jcrespo) > Is there a way to change the behavior of the query planner to use the correct index (in this case, page_time) when running select statements on the logging_userindex table/view? There wo... [11:30:46] tom29739: around? [11:46:23] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2859564 (10yuvipanda) As part of this, I've fixed all tools that didn'... [13:27:14] hi [13:28:05] I deployed a project named "lingua-libre" on tools, based on symphony [13:28:36] when I launch `php bin/console server:run` it seems to work [13:29:08] however I depoyed the project in ~/Lingua-Libre, not in ~/public_html [13:30:37] psychoslave: you can symlink ~/Linga-Libre to ~/public_html [13:30:41] probably the easiest thing to do [13:30:44] rmdir public_html [13:30:53] ln -s $PWD/Lingua-Libre $PWD/public_html [13:30:54] So currently the project is there and its embeded server can run, but it won't be browsable at tools.wmflabs.org/lingua-libre/ [13:31:13] then do 'webservice --backend=kubernetes start' [13:31:29] ok, I'm gonna try that thank you [13:32:03] psychoslave: np [13:33:24] hmm, 404 : http://tools.wmflabs.org/lingua-libre/ [13:34:57] psychoslave: so we use lighttpd to expose the webservices using PHP by default. it looks like the application you are trying to use might need additional rules to be usable with lighttpd [13:35:08] (I see no 'index.php' in that dir, for example) [13:35:17] so you'll probably have to ping the developers of the service to see how it'll work with lighttpd [13:38:06] ok, thank you, how do I switch back to lighttpd? [13:42:17] psychoslave: you're already using lighttpd [13:43:26] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web will have more info [13:43:33] ok, well, I don't know kubernetes, I had guess that it was on other http server [13:44:08] psychoslave: kubernetes makes no difference here. the default is just lighttpd, and it's running on kubernetes instead of gridengine [13:44:19] the full command there would've been [13:44:45] webservice --backend=kubernetes php5.6 start [13:44:49] php5.6 just happens to be the default [14:09:12] hey hello YuviPanda : } [14:09:39] I have a lame question for you. is the kubernetes setup we have build on top of the 'tools' labs project, eg on top of openstack? [14:10:00] and bonus question: are the puppet manifests reusable to build another kubernetes on top of another labs project? [14:10:23] hey hashar :) [14:10:38] hashar: 1. yes, it's on top of openstack, but not using any openstack specific features [14:10:38] (context is some meeting we had yesterday and we might look at moving CI to containers/kubernetes. At one point we will need to setup a proof of concept separated from tools) [14:10:40] hashar: 2. yes, but we haven't tested them yet. [14:10:56] so openstack is only used to provide a base set of empty instances [14:11:02] and you install kubernetes on top of those instances [14:11:14] PaaS on top of an IaaS ! [14:11:26] i wasn't sure [14:11:27] hashar: yes [14:11:36] I though we did a POC of k8s on tools in June [14:11:44] and eventually that got promoted to bare metal hardware [14:11:47] like the Ganetic cluster [14:11:51] Ganeti [14:12:30] thanks for the clarification. Guess we will talk about it on the ops list soonish ™ [14:12:52] hashar: ah, nope [14:13:00] hashar: there's people playing with a kubernetes cluster in production [14:13:07] hashar: for production stuff. unrelated to tools [14:13:23] yeah noticed that [14:13:34] and my understand is the whole thing is not yet complete / still being polished [14:13:47] so maybe it is too soon to spawn a k8s proof of concept for CI purposes [14:13:51] hashar: yeah, is ongoing. but the tools cluster has been in use for a while tho [14:14:00] hashar: >50% of our webrequests are to kubernetes [14:14:07] yeah I noticed the migration effort :} [14:14:10] hashar: and we've been running it for almost 9 months now in tools [14:14:13] and consider it stable [14:14:20] niceee [14:14:47] did you get any high level documentation written on wikitech ? Or is there some k8s 101 doc on upstream? [14:14:57] I am going to have to dig in it to see what that beast is about [14:15:34] hashar: there's lots of upstream documentation at https://k8s.io [14:18:05] will have to dig in there [14:18:09] YuviPanda: thanks :) [14:19:05] hashar: yw [14:53:31] 06Labs, 10Labs-Infrastructure, 10Monitoring: keystone: Monitor existence and membership for certain projects and accounts - https://phabricator.wikimedia.org/T152708#2859836 (10Andrew) [17:41:09] YuviPanda, hi. [18:48:54] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Gstupp was created, changed by Gstupp link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Gstupp edit summary: Created page with "{{Tools Access Request |Justification=I will create a tool that will allow you to create or retrieve a wikidata item that represents a scientific journal article given a pubme..." [19:30:45] (03PS2) 10Andrew Bogott: Add labs root key for bd808 [labs/private] - 10https://gerrit.wikimedia.org/r/325824 (https://phabricator.wikimedia.org/T152520) (owner: 10BryanDavis) [19:30:53] (03CR) 10Andrew Bogott: [V: 032 C: 032] Add labs root key for bd808 [labs/private] - 10https://gerrit.wikimedia.org/r/325824 (https://phabricator.wikimedia.org/T152520) (owner: 10BryanDavis) [19:32:46] !log tools upgrade / restart prometheus-node-exporter [19:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:38:39] PROBLEM - Puppet run on tools-exec-1417 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:39:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:43:48] (03CR) 10Andrew Bogott: [C: 032] Add support for authenticated Action API use [labs/striker] - 10https://gerrit.wikimedia.org/r/324637 (https://phabricator.wikimedia.org/T144712) (owner: 10BryanDavis) [19:45:13] (03Merged) 10jenkins-bot: Add support for authenticated Action API use [labs/striker] - 10https://gerrit.wikimedia.org/r/324637 (https://phabricator.wikimedia.org/T144712) (owner: 10BryanDavis) [19:53:15] 06Labs, 06Operations, 13Patch-For-Review: audit labs versus production ssh keys - https://phabricator.wikimedia.org/T108078#2860488 (10ArielGlenn) We are all agreed that jenkins failing on this would be pretty annoying, since it could block unrelated changes. BUT surely we can do some sort of regular audit. [19:56:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:03:57] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2860513 (10yuvipanda) When I finally got to actually creating the user... [20:18:40] RECOVERY - Puppet run on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [20:19:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [20:27:19] PROBLEM - Puppet run on tools-worker-1019 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:31:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:07:19] RECOVERY - Puppet run on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [21:08:08] Hi all, why https://tools.wmflabs.org/replag/ stopped working? I can see only s3 there and I don't think s3 is only for fiwikivoayge... [21:12:21] chasemp, ^ [21:12:31] Fascinating is I can't see any lag when looking directly into heartbeat_p... [21:12:36] MariaDB [meta_p]> select count(*) from wiki; [21:12:36] +----------+ [21:12:37] | count(*) | [21:12:37] +----------+ [21:12:37] | 1 | [21:12:38] +----------+ [21:12:39] 1 row in set (0.00 sec) [21:12:41] weird and interesting [21:12:47] ok [21:18:54] Urbanecm: we reduced replag to 0 by stopping most replication ;) [21:21:01] Why was replication stopped? bd808 [21:21:32] Urbanecm: it was a joke. there's just some issue with the meta_p.wiki table that we are looking into [21:21:56] chasemp, issue is only on labsdb1003 [21:21:57] not labsdb1001 [21:22:09] I just fixed it on labsdb1001 I believe is why [21:22:12] ah [21:22:19] or at least it's nearly fixed maybe an outlier or two I'm not sure yet [21:22:54] bd808, please don't měkké [21:23:15] Eh. Please don't make jokes like this :) [21:23:20] I used a winky ;) [21:24:59] What a winky is? [21:27:40] Seems it works again... [21:31:14] Krenair: as I understand it things are fixed now [21:31:51] yep [21:32:02] it certainly looks a lot better [21:38:09] Was the problem in replication? Or only in lag-reporting tools? [21:39:30] Urbanecm: it was with the meta_p wiki table Urbanecm and the fact that I replaced all entries with 1 instead of adding 1 entry [21:40:12] Thanks. [21:56:05] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2860727 (10chasemp) A small note owed as part of a previous conversati... [22:02:52] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Gstupp was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1115020 edit summary: [22:08:54] bd808 Hi im wondering do you know how i can get logstash working? im trying to get logstash working for gerrit but not sure how to get it working on https://logstash-beta.wmflabs.org/ [22:08:55] ? [22:09:40] this is https://gerrit.wikimedia.org/r/#/c/326177/4/modules/gerrit/templates/log4j.properties.erb how i currently have done it [22:09:48] following https://blog.lanyonm.org/articles/2015/12/29/log-aggregation-log4j-spring-logstash.html#log4j-over-tcp [23:02:08] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2860880 (10jcrespo) I have created the role: `CREATE ROLE labsdbuser;...