[00:01:14] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, and 2 others: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2584663 (10bd808) [00:01:17] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Striker, and 2 others: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2584660 (10bd808) 05Open>03Resolved https://toolsadmin.wikimedia.org/ Many thanks to @mmodell for Phab magic, @yuvipanda and @jcres... [00:02:12] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Striker, and 2 others: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2584664 (10mmodell) Awesome, congratulations! [00:02:47] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, and 2 others: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2226182 (10bd808) [00:03:02] 06Labs, 10DBA, 06Editing-Analysis: Replicate editor month table to Labs - https://phabricator.wikimedia.org/T143955#2584669 (10AlexMonk-WMF) I don't think anything from analytics databases ever gets replicated to labs... [00:03:44] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2584673 (10bd808) [00:03:48] 06Labs, 06Developer-Relations, 06WMF-Legal: Provide an easy way for Tool Labs tools to expose their source code - https://phabricator.wikimedia.org/T102081#2584675 (10bd808) [00:03:51] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2584674 (10bd808) [00:03:55] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, and 2 others: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2584671 (10bd808) 05Open>03Resolved https://toolsadmin.wikimedia.org/ [00:04:16] 06Labs, 10DBA, 06Editing-Analysis: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2584676 (10AlexMonk-WMF) [00:15:00] bd808: nice work with https://toolsadmin.wikimedia.org, I've just been trying it out [00:15:23] Finally self serve diffusion repos :) [00:15:37] there are a few kinks but yeah super happy [00:32:00] bd808: the tools admin console is nifty [00:36:50] 06Labs, 10Horizon, 15User-bd808: horizon.tables.FixedFilterAction failing in puppet gui - https://phabricator.wikimedia.org/T143960#2584718 (10Andrew) [00:57:55] bd808 congrats on getting it all up :) [01:00:34] yuvipanda, Matthew_: thanks! [01:00:50] I'll write up something for labs-l tomorrow [01:21:20] 06Labs, 10DBA, 06Editing-Analysis: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2584567 (10Neil_P._Quinn_WMF) p:05Triage>03Normal [03:52:45] (03PS1) 10BryanDavis: Map cn (common name) to ldapname [labs/striker] - 10https://gerrit.wikimedia.org/r/306868 (https://phabricator.wikimedia.org/T143956) [04:13:43] (03CR) 10BryanDavis: [C: 032] Map cn (common name) to ldapname [labs/striker] - 10https://gerrit.wikimedia.org/r/306868 (https://phabricator.wikimedia.org/T143956) (owner: 10BryanDavis) [04:14:42] (03Merged) 10jenkins-bot: Map cn (common name) to ldapname [labs/striker] - 10https://gerrit.wikimedia.org/r/306868 (https://phabricator.wikimedia.org/T143956) (owner: 10BryanDavis) [04:15:35] (03PS1) 10BryanDavis: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/306870 [04:15:48] (03CR) 10BryanDavis: [C: 032] Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/306870 (owner: 10BryanDavis) [04:15:54] (03Merged) 10jenkins-bot: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/306870 (owner: 10BryanDavis) [06:01:12] PROBLEM - Puppet staleness on tools-exec-1204 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [06:03:00] PROBLEM - Puppet staleness on tools-exec-1213 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0] [07:39:45] 06Labs, 10Wikimedia-Labs-General, 10DBA, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2585146 (10jcrespo) [07:39:48] 06Labs, 10DBA: s2 replag currently 8 hours - https://phabricator.wikimedia.org/T143934#2585143 (10jcrespo) 05Open>03Resolved a:03jcrespo s2 was under maintenance on production; This will happen several times on one shard at a time in order to propagate a production schema change. There is very little we... [08:09:08] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/VolkerE was created, changed by VolkerE link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/VolkerE edit summary: Created page with "{{Tools Access Request |Justification=Showcasing design-related work for interested people. |Completed=false |User Name=VolkerE }}" [08:11:02] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Enable access to Wikipedia Tulu (tcywiki) on labs replicas - https://phabricator.wikimedia.org/T142223#2585214 (10AlexMonk-WMF) [08:11:04] 06Labs, 10Tool-Labs: Document all tables missing in replicas - https://phabricator.wikimedia.org/T139289#2585216 (10AlexMonk-WMF) [08:11:07] 06Labs, 10Labs-Infrastructure, 10DBA: Add pp_propname/pp_value index to Labs replica - https://phabricator.wikimedia.org/T140609#2585215 (10AlexMonk-WMF) [08:11:11] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: maintain-replicas.pl unmaintained, unmaintainable - https://phabricator.wikimedia.org/T138450#2585217 (10AlexMonk-WMF) [08:11:14] 06Labs, 10DBA: Wrong page title in labs database replica enwiki page table - https://phabricator.wikimedia.org/T136618#2585218 (10AlexMonk-WMF) [08:11:27] 06Labs, 10DBA: Data missing from June 11/12 on s3.labsdb - https://phabricator.wikimedia.org/T115517#2585225 (10AlexMonk-WMF) [08:11:28] 06Labs: Replication issue with Fa WP replica - https://phabricator.wikimedia.org/T108032#2585226 (10AlexMonk-WMF) [08:11:32] 06Labs, 10Tool-Labs, 10DBA, 06Operations: Replicate wikimania2017wiki to labs - https://phabricator.wikimedia.org/T126096#2585224 (10AlexMonk-WMF) [08:11:46] 06Labs, 10Wikimedia-Labs-General, 10DBA, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2585212 (10AlexMonk-WMF) [08:13:54] 06Labs, 10Wikimedia-Labs-General, 10Technical-Tool-Request: Create web version of metainformation table on replica servers - https://phabricator.wikimedia.org/T56703#2585240 (10AlexMonk-WMF) I believe so @scfc [08:15:32] 06Labs, 10Tool-Labs, 10DBA, 06Operations: Replicate wikimania2017wiki to labs - https://phabricator.wikimedia.org/T126096#2004612 (10AlexMonk-WMF) Needs a maintain-replicas run [08:16:49] 06Labs, 10DBA, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2297843 (10AlexMonk-WMF) I suppose some views just need to be added to maintain-replicas for these [08:17:57] 06Labs, 10Tool-Labs: Document all tables missing in replicas - https://phabricator.wikimedia.org/T139289#2425693 (10AlexMonk-WMF) You should be able to `SHOW TABLES` on e.g. `metawiki`, then compare it to the same for `metawiki_p`. [09:46:21] Hello [09:46:50] I'm trying to create a self hosted Puppet VM [09:47:46] For now, it is still connected to labs-puppetmaster-eqiad.wikimedia.org [09:48:10] How can I make the VM use itself as puppetmaster [09:48:54] the VM is assigned the role::puppet::self [09:48:59] racodond: https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [09:49:33] valhallasw`cloud: Thanks! I'm trying to help racodond get started here. [09:50:38] We were reading that page and did assign the role. My understanding from the doc is that this should configure /etc/puppet/puppet.conf with server = localhost (or similar). But it does not seem to be the case [09:52:49] gehel: I don't know -- the notes about forcing puppet runs are probably important [09:53:16] valhallasw`cloud: I'm digging into the role::puppet::self code... [09:54:33] reading the code, it seems that we have a $::puppetmaster variable that is set. I'm wondering where it comes from... [09:56:03] hiera, most likely [09:56:15] operations/puppet -> hieradata/labs/... [09:56:52] the code talks about puppet classes configured via LDAP... [09:57:49] the applied roles are configured on wikitech and stored in ldap [10:45:49] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:53:10] 06Labs, 10Horizon, 15User-bd808: horizon.tables.FixedFilterAction failing in puppet gui - https://phabricator.wikimedia.org/T143960#2584718 (10AlexMonk-WMF) I've figured this out - horizon uses `categorize` to count the number of items in each, but the JS relies upon `'category-' + category` classes to do th... [11:20:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:53:51] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2585688 (10zhuyifei1999) Doesn't the `ssh` of Cygwin / MinGW / MSYS (not an expert here) work? [13:02:22] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2585693 (10tom29739) >>! In T143939#2585688, @zhuyifei1999 wrote: > Doesn't the `ssh` of Cygwin / MinGW / MSYS (not an expert here) work? Yes, but it's a big pain to set up + it doesn't work reliably. [13:11:42] 06Labs, 10Tool-Labs: Problem with utf-8 on Grid - https://phabricator.wikimedia.org/T143691#2575981 (10zhuyifei1999) `sys.getfilesystemencoding()`: On bastion =>`utf-8` On `jsub -l release=trusty` => `ascii` [[https://docs.python.org/3/library/sys.html#sys.getfilesystemencoding|Docs of that call]] points to... [13:23:27] 06Labs, 10Tool-Labs: Problem with utf-8 on Grid - https://phabricator.wikimedia.org/T143691#2585792 (10zhuyifei1999) My os.environ: ``` environ({'UPSTART_EVENTS': 'runlevel', 'LIBRARY_PATH': '/data/project/yifeibot/.local/lib', 'UPSTART_JOB': 'rc', 'UPSTART_INSTANCE': '', 'NSLOTS': '1', 'TMP': '/tmp/187097.1.t... [13:56:02] 06Labs, 10Tool-Labs: Problem with utf-8 on Grid - https://phabricator.wikimedia.org/T143691#2585880 (10valhallasw) SGE does not pass any environment variables unless explicitly specified, except for the ones you noted. Explicitly passing is possible, but will likely break when tools switches to kubernetes -- j... [13:58:03] valhallasw`cloud: just found http://bugs.python.org/issue13643 [14:00:34] zhuyifei1999_: yes? [14:01:05] No locale = C locale = ascii encoding [14:02:48] ? [14:19:52] zhuyifei1999_: I don't understand what I should do with that bug? [14:26:56] is there some sore of default env vars that sge pass through? [14:43:17] I don't know, and as noted, you should not depend on it [14:44:15] There's a better way: don't use SGE, use k8s instead [14:51:01] well, you can't use k8s here yet [14:51:12] and it's likely that you'll encounter comparable issues with k8s [14:52:07] (I'm not sure what the environment variables are for a process in a container) [14:53:21] i need help with Two problems [14:54:44] 06Labs, 10Tool-Labs: Document all tables missing in replicas - https://phabricator.wikimedia.org/T139289#2586152 (10valhallasw) Yes, that would already help a lot. I think that information should also be available from the scripts that generate the VIEWs. There is also the difference between a 'standard' medi... [14:55:43] anyone_: feel free to ask! [15:22:17] TE MATARE CON MIS PROPIAS MANOS HIJO DE PERRA ALVARO MOLINA [15:22:18] TE MATARE CON MIS PROPIAS MANOS HIJO DE PERRA ALVARO MOLINA [15:22:19] TE MATARE CON MIS PROPIAS MANOS HIJO DE PERRA ALVARO MOLINA [15:23:49] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2586262 (10Stevietheman) It has been over 4 months since the most recent popular pages updates, and the pages in projects in which I'm a member are getting stale. I... [15:39:07] I think I broke stools? [15:39:17] *xtools [15:39:35] Wait, it's back. [15:43:01] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2586356 (10valhallasw) @Stevietheman: Please create a seperate bug for that issue. [16:34:17] 06Labs, 10Horizon, 15User-bd808: horizon.tables.FixedFilterAction failing in puppet gui - https://phabricator.wikimedia.org/T143960#2586505 (10bd808) Nice sleuthing @AlexMonk-WMF [16:40:56] valhallasw`cloud for k8s I want to explicitly set it, and I think I might be already [16:41:00] although maybe that's only in specific containers [16:41:57] Explicitly set what? LANG? [16:42:25] To en_US.UTF-8? [16:42:36] think so [16:42:54] tom29739 valhallasw`cloud btw, you might like: https://kubernetesbootcamp.github.io/kubernetes-bootcamp/ [16:42:58] I suppose that's a sane default [16:43:26] Although I would enjoy setting it to fr_FR >:) [16:43:42] :P [16:43:49] you can still override it [16:44:28] yuvipanda: also it won't solve the python print issue, because the output is still piped [16:44:44] right. gotta .encode [16:44:50] There's the kubectl logs command [16:44:52] or PYTHONIOENCODING=... [16:45:31] (I think only Python cares about pipe encoding) [16:46:40] For some reason the webservices don't work with kubectl logs :/ [16:46:50] yes, because we override the output [16:46:52] to go to a file [16:46:55] and not to stdout [16:48:59] But then you have to logrotate so the files don't get too big with highly used tools. [16:49:16] yup. log management is aproblem [16:49:31] but didn't want to couple moving to k8s with changes in log management at the same time [16:50:05] It'd be nice to have some sort of web solution for logs. [16:50:20] Like logstash or something. [16:50:46] it's on the wishlist [16:51:21] we explored it earlier. the fact that it doesn't have any auth layer, and we'd have to build our own, is what makes it super complex [16:51:27] don't want one tool to see other tool's logs [16:51:31] so we'll need to build our own authz / authn [16:53:45] !log tools migrate tools-static-02 to labvirt1001 [16:53:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [16:53:55] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2586595 (10Stevietheman) I thought this was the bug. Also, what would my bug state and where would it go? [16:54:51] 06Labs, 10Tool-Labs: Clean up data in /data/scratch/mwoffliner - https://phabricator.wikimedia.org/T144025#2586596 (10madhuvishy) [16:55:35] PROBLEM - Host tools-static-02 is DOWN: CRITICAL - Host Unreachable (10.68.20.114) [16:55:44] 06Labs, 10Tool-Labs: Problem with utf-8 on Grid - https://phabricator.wikimedia.org/T143691#2586601 (10Vladis13) >>! In T143691#2583956, @valhallasw wrote: > On a sidenote, you should probably use subprocess.Popen instead of os.system. Unfortunately, it same breaks. It seems still called the "os" module. Pyth... [16:55:47] 06Labs: Clean up data in /data/scratch/mwoffliner - https://phabricator.wikimedia.org/T144025#2586602 (10yuvipanda) [16:57:57] yuvipanda, there's https://github.com/graylog2/graylog2-server [16:58:07] "It is possible to use an external LDAP or Active Directory server to perform user authentication in Graylog. Since version 1.2, you can also use LDAP groups to perform authorization by mapping them to Graylog roles." [16:58:15] nice [16:58:45] tom29739 https://phabricator.wikimedia.org/T127367 is the tracking ticket [16:59:34] tom29739 thanks for pointing out graylog server! I didn't know of it [17:01:11] Graylog is an alternative to Kibana I think [17:01:59] In that Graylog uses Elasticsearch as a backend. [17:02:39] And it replaces logstash too. [17:03:42] 06Labs, 10Tool-Labs: Problem with utf-8 on Grid - https://phabricator.wikimedia.org/T143691#2586657 (10Vladis13) >>! In T143691#2583956, @valhallasw wrote: > You can try setting LANG=en_US.UTF-8 instead, but I can't guarantee that'll work. Not work, see above. On bastion `locale`: ``` LANG=en_US.UTF-8 LANGUAGE... [17:11:17] RECOVERY - Puppet run on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [17:19:41] valhallasw`cloud ok, I merged the clush change! [17:19:57] valhallasw`cloud I wrote *some* docs at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin#Command_orchestration [17:20:22] valhallasw`cloud it doesn't have sudo yet, I want to make sure that nobody except root can actually ssh in before I grant it sudo [17:25:12] RECOVERY - Host tools-static-02 is UP: PING OK - Packet loss = 0%, RTA = 1.96 ms [17:30:04] PROBLEM - Puppet run on tools-static-02 is CRITICAL: CRITICAL: 83.33% of data above the critical threshold [0.0] [17:32:07] 06Labs, 10Tool-Labs: Problem with utf-8 on Grid - https://phabricator.wikimedia.org/T143691#2586804 (10Vladis13) Works here's how: Make myscript.sh: ``` #!/bin/bash LANG="ru_RU.UTF-8" ./myscript.py # "en_US.UTF-8" will same result ``` Make myscript.py: ``` #!/usr/bin/env python3 # coding: utf8 import os string... [17:32:24] PROBLEM - Free space - all mounts on tools-static-02 is CRITICAL: CRITICAL: tools.tools-static-02.diskspace._srv.byte_percentfree (<100.00%) [17:40:03] RECOVERY - Puppet run on tools-static-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:58] 06Labs, 10Horizon, 15User-bd808: horizon.tables.FixedFilterAction failing in puppet gui - https://phabricator.wikimedia.org/T143960#2586890 (10Andrew) 05Open>03Resolved This is so much better now! [17:44:59] yuvipanda: brainfart; could we put the host outside of the tools project? [17:45:25] yuvipanda: also we need to block the clush user in wikitech I think [17:46:31] yuvipanda: wait, clush just ssh's in, right? so we could just use it as our own user instead of the clush user? [17:46:42] valhallasw`cloud yeah, that's what I've been doing [17:46:43] from my local machine [17:46:49] I can write up how to do that too [17:47:00] but that should also work from tools-puppetmaster I think? [17:47:16] valhallasw`cloud not unless you do agent forwarding? [17:47:33] mmm. hba? [17:48:17] yuvipanda: I dislike having a central username for all ops, as that makes it harder to track who did what [17:48:23] auth.log is very helpful for that [17:49:47] valhallasw`cloud yeah, I agree. what do you suggest we do? [17:51:14] yuvipanda: not entirely sure; host based auth is one option, but that's roughly equivalent to having pub/priv key sets on tools for internal use [17:51:39] alternatively, clush could run 'sudo -u valhallasw sudo blah' on the host [17:52:27] valhallasw`cloud hmm, we could do that. restrict clush access to only use with sudo on the master, and log all accesses? [17:54:04] that would work [17:54:43] valhallasw`cloud yeah. [17:55:40] valhallasw`cloud I guess we could just wrap it in a script that does both logging and enforce sudo [17:55:51] sounds good [17:56:24] valhallasw`cloud in that case, should we also deny access to the original clush binary? [17:57:19] just deny access to the private key? [17:58:26] valhallasw`cloud ah, hmm. [17:58:38] I guess I can make access to the private key to tools.admin [17:59:55] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2586956 (10kaldari) @Stevietheman: The bug already exists: T141154. [17:59:58] or to root:root? [18:00:19] valhallasw`cloud yeah, but then the question is do you want to disallow just sudoing to root and then running them all [18:00:29] which will lose user info details [18:01:06] you can always do that I think? [18:01:48] this sounds like you are recreating what keyholder does for scap & scap3 [18:02:15] group based access with logging to shared ssh keys [18:04:38] 06Labs, 10Tool-Labs, 07Tracking: Overhaul logging setup for Tools (Tracking) - https://phabricator.wikimedia.org/T127367#2586979 (10tom29739) Graylog (https://github.com/graylog2/graylog2-server, https://www.graylog.org/) seems to cover most of these points: # Take load off NFS - logs are stored on an Elasti... [18:05:31] ^ yuvipanda - Graylog seems to cover all of those points except "7. Provide a filesystem based interface for log reading" which I interpreted as a means to download logs [18:08:33] tom29739: I think 7 means something along the lines of 'I can run cat /some/path/to/file.txt' [18:08:38] but that might just not be possible [18:09:04] the main thing is that the output of a script could be relevant for the next one [18:10:20] I thought the idea of the task was to get logs off NFS. [18:11:01] And having 7 kinda defeats the object [18:11:32] But if you were doing filesystem ingress you could tell it to leave the file there and not delete the file. [18:12:04] I think an option for jsub to write logs to NFS might be the easiest solution [18:12:31] (default = logstash/graylog, option = write to nfs) [18:13:24] It already writes to NFS now, so it would be easy to add an option for it. [18:19:54] crosswatch is still giving a 404, webservice down? [18:35:03] yuvipanda, genericwebservice.py", line 18, in run, os.execv('/bin/sh', ['/bin/sh', '-c', self.extra_args]): TypeError: execv() arg 2 must contain only strings [18:36:04] PROBLEM - Puppet run on tools-static-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:38:53] yuvipanda: oh, self.extra_args is None [18:40:09] so somehow it's not being passed anymore? [18:40:46] oh, I just forgot to add the parameter ._. [18:43:15] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2587092 (10Stevietheman) >>! In T133326#2586956, @kaldari wrote: > @Stevietheman: The bug already exists: T141154. Thanks! [18:43:48] 06Labs, 10Tool-Labs, 07Tracking: Overhaul logging setup for Tools (Tracking) - https://phabricator.wikimedia.org/T127367#2041636 (10valhallasw) Thank you for finding this, and taking the time to work out all the points @tom29739! I think 7) is not too important -- the use case (as far as I can think of) is u... [19:06:05] RECOVERY - Puppet run on tools-static-02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:13:11] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Yasbot was created, changed by Yasbot link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Yasbot edit summary: Created page with "{{Tools Access Request |Justification=Heloo Dear, I want to run a bot in fawiki to improve articles and help managers. also i had this access about 3years ago in toolserver..." [19:37:20] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:15:33] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:33:54] tools-bastion-03 seems to be really slow for me. [20:35:58] Maybe it's just my connection. [20:36:13] it seems ok on my end, but I just did a few simple things [20:39:50] Looks like people are running stuff on the bastion. Again. "/usr/bin/php ./scan.php" [20:39:59] It's spiking at 30% CPU. [20:40:08] that's not going to cause any issues [20:40:20] When multiple people run stuff, the bastion becomes unusuable. [20:40:25] it's a 8 cpu host [20:41:02] So why does it get unresponsive then? [20:41:14] it's not unresponsive for me [20:41:19] and the main cause of unresponsiveness is NFS [20:42:01] * tom29739 checks his connection [20:42:15] (the issue is typically not that your commands run slower, but that you don't get a prompt because bash can't get whatever information it needs from NFS) [20:47:10] we have NFS stats on this host in graphite we can pull up but I can't atm [20:48:18] * valhallasw`cloud reminds himself to strace bash when bastion is slow again [20:48:39] I just reordered my $PATH, I hope that might help with starting commands [20:55:34] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [21:25:47] 06Labs, 10Horizon, 15User-bd808: horizon.tables.FixedFilterAction failing in puppet gui - https://phabricator.wikimedia.org/T143960#2587539 (10AlexMonk-WMF) a:05bd808>03AlexMonk-WMF [21:27:14] bd808 btw, phabricator attachment works now :) [21:27:34] bd808 is there a phab component for it? we should add tool creation to this :D [21:28:30] https://phabricator.wikimedia.org/project/profile/2134/ [21:28:44] and that's on my roadmap :) [21:30:28] \o/ awesome [21:32:43] 06Labs, 10Horizon, 13Patch-For-Review: Horizon dashboard for managing instance puppet config - https://phabricator.wikimedia.org/T91990#2587546 (10AlexMonk-WMF) [21:32:47] 06Labs, 10Horizon, 15User-bd808: horizon.tables.FixedFilterAction failing in puppet gui - https://phabricator.wikimedia.org/T143960#2587545 (10AlexMonk-WMF) [21:33:01] you good citizen sob (project image is The Forge by Francisco Goya and reused as a public domain art reproduction.) :) [21:33:51] chasemp: :) got to know who butters your bread [21:39:03] bd808: should striker bugs be reported here? [21:41:36] valhallasw`cloud: on phab -- https://phabricator.wikimedia.org/project/profile/2134/ [21:42:00] bd808: ya, mostly whether they should be reported in this channel or somewhere else [21:42:14] oh I get you [21:42:34] (03PS1) 10Merlijn van Deen: Report Striker (toolsadmin.wikimedia.org) to #wikimedia-labs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307022 [21:43:26] (03CR) 10BryanDavis: [C: 031] Report Striker (toolsadmin.wikimedia.org) to #wikimedia-labs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307022 (owner: 10Merlijn van Deen) [21:43:37] (03CR) 10Merlijn van Deen: [C: 032] Report Striker (toolsadmin.wikimedia.org) to #wikimedia-labs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307022 (owner: 10Merlijn van Deen) [21:50:54] 06Labs, 10Tool-Labs: Document all tables missing in replicas - https://phabricator.wikimedia.org/T139289#2587587 (10AlexMonk-WMF) >>! In T139289#2586152, @valhallasw wrote: > Yes, that would already help a lot. I think that information should also be available from the scripts that generate the VIEWs. Nope, m... [21:53:12] (03Merged) 10jenkins-bot: Report Striker (toolsadmin.wikimedia.org) to #wikimedia-labs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307022 (owner: 10Merlijn van Deen) [21:54:29] 10PAWS, 06Research-and-Data-Backlog, 07Epic, 03Research-and-Data-2017-Q1: Launch Open Notebooks Infrastructure - https://phabricator.wikimedia.org/T140430#2587592 (10DarTar) [21:54:46] 10PAWS, 06Research-and-Data-Backlog, 07Epic, 03Research-and-Data-2017-Q1: Launch Open Notebooks Infrastructure - https://phabricator.wikimedia.org/T140430#2464537 (10DarTar) [22:11:42] 06Labs, 10Horizon, 13Patch-For-Review: Horizon dashboard for managing instance puppet config - https://phabricator.wikimedia.org/T91990#2587639 (10Luke081515) [22:22:26] 06Labs, 13Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1339137 (10AlexMonk-WMF) ```krenair@terbium:~$ ldapsearch -x -D "uid=novaadmin,ou=people,dc=wikimedia,dc=org" -w redacted objectClass=puppetClient | grep -o "puppetVar: .*=" | sort | un... [22:59:16] 06Labs, 06Operations, 13Patch-For-Review: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#2587703 (10yuvipanda) [23:00:08] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Jogo.obb was created, changed by Jogo.obb link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Jogo.obb edit summary: Created page with "{{Tools Access Request |Justification=I am already user of Tool Labs as [[User:JogoBot]] and want to use also my main SUL-Account here now for consistence (and be future proof..." [23:10:12] !log logstash deded. [23:10:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Logstash/SAL, Master [23:14:38] (03CR) 10Paladox: [C: 031] Fix extension project path [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/303858 (owner: 10Awight) [23:15:22] (03CR) 10Paladox: "thanks." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/304746 (owner: 10Paladox) [23:16:09] 06Labs, 06Operations, 13Patch-For-Review: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#2587726 (10yuvipanda) I'm going to pick up *some* of this next week, specifically stop allowing using role::puppet::self fr... [23:18:50] !log striker add self to project [23:18:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL, Master [23:19:06] 10PAWS, 07Upstream: PAWS cron functionality - https://phabricator.wikimedia.org/T124972#2587738 (10jayvdb) Patch merged into v1.4alpha [23:20:48] !log tools.lolrrit-wm cherry pick https://gerrit.wikimedia.org/r/#/c/303858/ [23:20:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master [23:22:05] (03CR) 10Paladox: "I cherry picked this." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/303858 (owner: 10Awight) [23:22:10] yuvipanda hi, could you merge https://gerrit.wikimedia.org/r/#/c/303858/ please? [23:22:19] i can deploy it. :) [23:22:43] (03PS2) 10Yuvipanda: Fix extension project path [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/303858 (owner: 10Awight) [23:22:46] (03CR) 10Yuvipanda: [C: 032] Fix extension project path [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/303858 (owner: 10Awight) [23:22:50] done paladox [23:22:56] thanks :) [23:23:21] yw [23:23:26] (03Merged) 10jenkins-bot: Fix extension project path [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/303858 (owner: 10Awight) [23:23:54] !log tools.lolrrit-wm deploying https://gerrit.wikimedia.org/r/#/c/303858/ [23:23:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master [23:37:02] PROBLEM - Puppet run on tools-static-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:37:48] There is a problem in a number of my edits Is it possible to be solved [23:39:45] the number of General statistics is Different than SUL editcounter [23:45:53] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]