[00:14:48] 06Labs, 10Tool-Labs, 13Patch-For-Review: Develop on separate branches for each target of Debian packages - https://phabricator.wikimedia.org/T156886#3056704 (10scfc) I have "documented" the two branches `master` and `ubuntu/precise` at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin#Deploy_new_... [00:52:19] 10Striker, 07Technical-Debt: Replace deprecated phabricator conduit api calls in phabricator.py file - https://phabricator.wikimedia.org/T159044#3056742 (10Paladox) [01:19:16] (03PS1) 10Tim Landscheidt: Cut release 1.19 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340055 [01:19:47] (03PS1) 10Tim Landscheidt: Cut release 1.19~precise+1 [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340056 [01:21:28] (03CR) 10Tim Landscheidt: [C: 032] Cut release 1.19~precise+1 [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340056 (owner: 10Tim Landscheidt) [01:21:32] (03CR) 10Tim Landscheidt: [C: 032] Cut release 1.19 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340055 (owner: 10Tim Landscheidt) [01:21:52] (03Merged) 10jenkins-bot: Cut release 1.19~precise+1 [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340056 (owner: 10Tim Landscheidt) [01:21:57] (03Merged) 10jenkins-bot: Cut release 1.19 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340055 (owner: 10Tim Landscheidt) [01:34:21] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:40:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:49:23] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [02:15:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:26:34] 06Labs, 10Tool-Labs, 05Prometheus-metrics-monitoring: Remove prometheus-node-exporter from Tool Labs apt repository (if no longer needed) - https://phabricator.wikimedia.org/T158824#3056800 (10scfc) I also had to remove `prometheus-blackbox-exporter` from `precise-tools` and `trusty-tools` (but not from `jes... [02:38:07] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:39:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:41:59] !log tools Deployed jobtools and misctools 1.19/1.19~precise+1 (T155787, T156886). [02:42:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [02:42:08] T156886: Develop on separate branches for each target of Debian packages - https://phabricator.wikimedia.org/T156886 [02:42:08] T155787: Mail from cron regarding a failure of jsub - https://phabricator.wikimedia.org/T155787 [02:42:12] !log Purged misctools from instances where not puppetized. [02:42:12] Unknown project "Purged" [02:42:20] !log tools Purged misctools from instances where not puppetized. [02:42:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [02:44:22] 06Labs, 10Tool-Labs, 13Patch-For-Review: Develop on separate branches for each target of Debian packages - https://phabricator.wikimedia.org/T156886#3056812 (10scfc) 05Open>03Resolved a:03scfc Building the packages, adding them to `aptly` and deploying them went without problems. [02:44:45] 06Labs, 10Tool-Labs: Mail from cron regarding a failure of jsub - https://phabricator.wikimedia.org/T155787#3056815 (10scfc) 05Open>03Resolved [02:53:07] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [03:09:09] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [03:13:33] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [03:13:37] PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:14:14] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [03:19:06] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [03:24:59] 06Labs, 10Tool-Labs, 13Patch-For-Review: tools_enable_php_mcrypt_module is always executed on some instances - https://phabricator.wikimedia.org/T159022#3056826 (10scfc) a:03scfc [03:48:32] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [03:48:39] RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [04:45:24] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#3056836 (10Andrew) [06:40:33] PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:54:48] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/340069 (owner: 10L10n-bot) [07:15:34] RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [09:07:14] 06Labs, 10MediaWiki-extensions-OpenStackManager, 13Patch-For-Review, 05WMF-deploy-2017-02-28_(1.29.0-wmf.14): MW OpenStackManager: add support for ED25519 SSH keys - https://phabricator.wikimedia.org/T159070#3057138 (10MoritzMuehlenhoff) @scfc: All SSH daemons in labs have fully-featured support for ed2551... [10:03:18] (03CR) 10Hashar: "recheck" [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340056 (owner: 10Tim Landscheidt) [10:03:23] (03CR) 10Hashar: "recheck" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340055 (owner: 10Tim Landscheidt) [10:43:14] 10Tool-Labs-tools-Quentinv57's-tools, 13Patch-For-Review: Please fix bugzilla link in Tools-Quentinv57 - https://phabricator.wikimedia.org/T158290#3057437 (10MarcoAurelio) a:03Cyberpower678 Pull request submitted: https://github.com/quentinv57-tools/tools/pull/4 [14:02:58] 06Labs: Request creation of wikidiff2-wmde-dev labs project - https://phabricator.wikimedia.org/T158645#3057873 (10jkroll) Great. Thanks @Andrew ! [14:44:22] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 05Prometheus-metrics-monitoring: Labs Prometheus not recording k8s stats since 2017-01-24T06:00 - https://phabricator.wikimedia.org/T157355#3057943 (10fgiunchedi) No AFAIK grafana isn't able to merge data sources like that. A panel can have multiple data sources, ea... [14:53:23] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:11:07] 06Labs: check on the nova-api upstart logs - https://phabricator.wikimedia.org/T159141#3058007 (10Andrew) [15:28:24] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:21] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3058399 (10madhuvishy) a:05madhuvishy>03None [16:30:06] 06Labs: Request creation of Wikimedia Incubator labs project - https://phabricator.wikimedia.org/T159068#3055862 (10Andrew) Can you explain more about needing a public IP for parsoid? Can't the parsoid service run behind a port-specific proxy? It's http right? [16:36:20] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3058414 (10Cmjohnson) I can make a cable to run anywhere in the data center so proximity is not an issue. I need to find a space t... [16:53:40] 06Labs: Request creation of Wikimedia Incubator labs project - https://phabricator.wikimedia.org/T159068#3058492 (10Andrew) (Project request is approved but we need more info re: the floating IP request) [17:29:01] 06Labs, 10Tool-Labs, 10Continuous-Integration-Config, 13Patch-For-Review: Make lintian warnings voting errors in labs/toollabs repository - https://phabricator.wikimedia.org/T95098#3058596 (10scfc) 05Open>03Resolved [17:37:53] 06Labs, 10Labs-Infrastructure, 06Revision-Scoring-As-A-Service, 07artificial-intelligence: GPU resources for Labs - https://phabricator.wikimedia.org/T159165#3058608 (10Halfak) [17:51:20] (03PS7) 10Tim Landscheidt: Package jmail [labs/toollabs] - 10https://gerrit.wikimedia.org/r/339920 (https://phabricator.wikimedia.org/T158722) [17:52:12] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3058682 (10Andrew) Email nag sent to labs-announce on 2017-02-27 [17:56:46] 06Labs, 10Huggle: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted - https://phabricator.wikimedia.org/T157710#3013987 (10Harej) I left a message on Wikipedia: https://en.wikipedia.org/wiki/Wikipedia:Huggle/Feedback#Labs_instance [18:00:06] 06Labs: Web proxies don't show up - https://phabricator.wikimedia.org/T159162#3058710 (10Paladox) [18:01:44] (03PS1) 10Tim Landscheidt: Move puppetdb::password variables to hieradata/labs.yaml [labs/private] - 10https://gerrit.wikimedia.org/r/340148 [18:04:08] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3058720 (10chasemp) [18:08:28] 06Labs: Web proxies don't show up - https://phabricator.wikimedia.org/T159162#3058731 (10scfc) 05Open>03Resolved The domain resolves for me now: ``` [tim@passepartout ~]$ host wmde-wikidiff2-patched.wmflabs.org wmde-wikidiff2-patched.wmflabs.org has address 208.80.155.156 [tim@passepartout ~]$ ``` What usu... [18:12:01] 06Labs, 10Huggle: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted - https://phabricator.wikimedia.org/T157710#3013987 (10Matthewrbowker) @Petrb and @Addshore are the managers of this instance. It is necessary for #wm-bot. I will attempt to reach out to Petr and get it converted, though I... [18:13:26] 06Labs, 10Huggle: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted - https://phabricator.wikimedia.org/T157710#3058748 (10Andrew) Thanks y'all [18:14:05] halfak: you still have in mind that snuggle-en needs rebuilding, right? [18:14:33] Totally legitimate question. I need to find 15 minutes to review and drop snuggle-en. [18:14:40] I'm 99% sure that instance can just die. [18:15:15] halfak: ok, I'll will keep nagging in the meantime [18:15:37] Of course it's always an option to shut it down (but not delete) and wait and see what breaks and/or who complains :) [18:15:44] Yeah. You're a saint [18:15:45] :) [18:15:53] Oh we could do that. Actually I'd like that. [18:15:59] It would be a fine test [18:16:08] Honestly though, I think I can get these 15 minutes in today. [18:16:19] ok — I'll shut it down tomorrow if you don't beat me to it [18:16:20] I'm conferencing but I'm sure I'll find a boring talk ;) [18:16:24] Sounds great [18:16:36] Thanks again for your patience with this shitty labs user :/ [18:16:44] Are you at cscw? [18:16:47] yup [18:17:03] sounds like fun [18:19:38] 06Labs: Web proxies don't show up - https://phabricator.wikimedia.org/T159162#3058767 (10jkroll) You're right, sorry for the confusion. Thanks! [18:26:07] 06Labs, 10Beta-Cluster-Infrastructure, 10Wikimedia-General-or-Unknown, 13Patch-For-Review: rename -labs.php to -beta.php - https://phabricator.wikimedia.org/T150268#2780096 (10demon) Why? I'm not seeing any real benefit and a ton of potential breakage. [18:30:44] (03CR) 10Tim Landscheidt: "Works for my use case." [labs/private] - 10https://gerrit.wikimedia.org/r/340148 (owner: 10Tim Landscheidt) [18:31:04] (03CR) 10Tim Landscheidt: "* Tested to work for my use case." [labs/private] - 10https://gerrit.wikimedia.org/r/340148 (owner: 10Tim Landscheidt) [18:32:56] 06Labs, 06Operations, 10wikitech.wikimedia.org: Can't create account "Trizek (WMF)" - https://phabricator.wikimedia.org/T158408#3058811 (10Trizek-WMF) 05Open>03Invalid The "paid coding" vs "paid editing" reason is convincing me. Thanks @bd808! [18:50:04] andrewbogott hi, im wondering would you be able to review https://gerrit.wikimedia.org/r/#/c/340026 please? [19:26:04] 10MediaWiki-extensions-OpenStackManager: Sudo Policies can't be displayed for Tools - https://phabricator.wikimedia.org/T70100#3058954 (10Andrew) It's not a memory issue, that page is just too damn big if 'tools' is selected in the filter. If I increase max_execution_time then it loads just fine [19:34:07] 06Labs, 10Tool-Labs: Delete "toolserver" tool - https://phabricator.wikimedia.org/T116389#3058967 (10scfc) [19:34:10] 06Labs, 10Tool-Labs, 07Tracking: Toolserver migration to Tools (tracking) - https://phabricator.wikimedia.org/T60788#3058968 (10scfc) [19:50:36] any known problem with ldap in labs? i was logged into a machine (relforge-search.eqiad.wmflabs) but couldn't sudo and it was reporting 'ldap_start_tls_s: Connection error'. Guessed something was just funky with the instance and rebooted it, but now can't login at all getting pubkey errors (which would coincide with it not being able to talk to ldap) [19:53:24] fwiw though, i can log into other other machines in the same project and they seem fine :S [19:57:14] ebernhardson: nothing known but I can try to take a look [19:57:59] chasemp: would appreciate it, thanks [19:58:27] ebernhardson: can you try to login? [19:58:38] chasemp: Permission denied (publickey). [19:58:57] (just now) [19:59:11] yeah I saw it [19:59:16] that box is failing ot connect to ldap [20:00:14] oh, i bet i broke that :S I needed ssl certs for talking to https://relforge1001.eqiad.wmnet/ from labs so installed some certs, but perhaps that blew away whatever cert is needed by ldap? [20:00:18] * ebernhardson just realized [20:00:45] that seems pretty possible atm [20:00:55] the login failure is the ssh key lookup script failing to find to ldap [20:01:00] s/find/bind [20:01:05] which is all tls iirc [20:01:11] puppet should be able to fix that, no? [20:01:28] puppet is also broken here in several ways [20:04:56] but I'm not entirely sure this isn't part of first run setup [20:04:57] I'm actualy kind of thinking it is which means puppet won't recover it either way [20:05:45] ebernhardson: gotta ask, how much of this instance is recreatable easily :) becuase fixing this would be a rabbit hole potentially [20:05:57] andrewbogott: can you take a peak at relforge-search.search.eqiad.wmflabs ldap bind issues [20:06:11] possibly related to other ssl work interferring [20:06:37] I'm not sure atm what the fix would be but a key lookup demonstrates the failure easily [20:06:38] via /usr/sbin/ssh-key-ldap-lookup [20:06:39] chasemp: well, ideally i would like to keep /srv/mediawiki-vagrant/settings.d/* which has a variety of custom config, and i have stuff i was in the middle of working on in /srv/mediawiki-vagrant/puppet, would be it be possible to mount nfs and copy those over? [20:07:00] I can grab that sure [20:07:25] it's accessible via root key (no ldap) [20:09:08] looking at another server, the problem seems to be related to me replacing/etc/ssl/certs/Puppet_Internal_Ca.pem with the production one. Another guess might be to copy that from somewhere else, rebuild the cert symlinks, and see if it all fixes itself? [20:09:21] * ebernhardson didn't realize the server already had one when copying it in.. [20:09:59] chasemp: Looking, although if this is a new instance then a rebuild might be in order. [20:10:16] yeah [20:10:34] it's an interesting problem in that I don't know exactly what teh right thing to do is :) [20:11:17] * ebernhardson is good at breaking things in interesting ways aparently :P [20:17:16] ebernhardson: can you just rebuild it? I looks like you clobbered some unpuppetized certs… I can work on it more if this instance is valuable. [20:19:28] i would like to save a couple directories mentioned above, if those can be copied to NFS everything else is recreatable [20:19:45] ebernhardson: well I have them locally [20:19:52] do you have another instance w/ nfs for this project? [20:19:54] or [20:20:00] spin up another one and I'll drop them off for you [20:20:14] nah we don't actually mount nfs anywhere, but you can drop off the files in sistersearch.search.eqiad.wmflabs and i'll move it back over after rebuilding [20:20:33] nfs was just a random guess of something available to copy to [20:25:50] ebernhardson: ok check puppet and settings_d folders there [20:26:02] sorry it's not an easily recoverable thing and godspeed :) [20:26:26] chasemp: looks to have all the stuff i was most recently working on. Thanks! [20:26:36] and no worries, i should have just stuck to mucking with certs inside the mwv instance .. [21:33:49] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: HTTP 500 on Special:NovaSudoers - https://phabricator.wikimedia.org/T158299#3059294 (10Andrew) It's not a memory issue, that page is just too damn big if 'tools' is selected in the filter. If I increase max_execution_time then it load... [21:49:06] (03PS1) 10Tim Landscheidt: Port sql to Python [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340233 [21:52:33] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: HTTP 500 on Special:NovaSudoers - https://phabricator.wikimedia.org/T158299#3059377 (10scfc) (Cf. also https://gerrit.wikimedia.org/r/#/c/339832/ for reducing the size of the page :-).) [21:56:56] (03CR) 10Tim Landscheidt: "I installed sql as /usr/local/bin/sql.I038d14fb73a9bfe633003a6ab89d712510f61f61 on tools-bastion-03 for easier testing." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340233 (owner: 10Tim Landscheidt) [22:46:39] Hi [22:46:59] some bot hosted at wmflabs is being used to spam my channels [22:47:05] it's called "Revolucio" [22:47:41] Pitky-: can you be more descriptive. What kind of channels do you mean, when did it start, what does the spam look like. [22:48:35] can I paste you logs? [22:49:03] is this an irc channel etc? and sre [22:49:03] sure [22:50:09] http://paste.dimichichachara.tk/?a377b7a1f5b778e1#aHPGfjZd39nY9NbKHbqZThXRwXgtIvQrlDfAR91Aa7c= [22:50:18] channel #viaplus [22:50:43] this bot owned by a french 15 year old trol kid, Simon2001 [22:50:47] Pitky-: do you know what IP it's coming from? [22:51:00] chasemp, yes [22:51:28] internal-server-nat.wmflabs.org [22:51:59] I'd want to request the account deletion from the wmf labs [22:54:29] chasemp [22:58:06] Pitky-: do you have an account on phabricator.wikimedia.org by chance? this will take awhile to track back [22:58:15] i can log a ticket for it, but for now what happens if you ban this user? [23:06:22] andrewbogott: can you help me with this? [23:06:23] ^ [23:06:34] seems this person applied for a tools account and bryan denied [23:06:34] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Simon-kempf [23:06:40] and now possibly they are running this somewhere in labs [23:06:45] https://wikitech.wikimedia.org/w/index.php?title=User_talk:Simon-kempf&oldid=1577616 [23:06:55] * andrewbogott catches up [23:24:17] Pitky-: if you are still there, please do open a ticket with all that you know [23:38:47] (03CR) 10BryanDavis: "> @bd808: As you created the initial tox.ini, perhaps you can assess" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/339920 (https://phabricator.wikimedia.org/T158722) (owner: 10Tim Landscheidt) [23:41:20] 10Tool-Labs-tools-Other, 06Commons, 06Community-Tech, 10Internet-Archive, 07Community-Wishlist-Survey-2016: Create a new DerivativeFX after the Toolserver shutdown [AOI] - https://phabricator.wikimedia.org/T110409#3059748 (10srishakatux) [23:46:18] (03CR) 10BryanDavis: [C: 031] "Untested, but the code is readable and seems to do what the bash version intended. scfs said he tested this (T158722#3055794) and I trust " [labs/toollabs] - 10https://gerrit.wikimedia.org/r/339920 (https://phabricator.wikimedia.org/T158722) (owner: 10Tim Landscheidt) [23:50:21] 06Labs, 10Labs-project-Phabricator, 10Phabricator, 13Patch-For-Review: Applying role role::phabricator::main causes errors on instances - https://phabricator.wikimedia.org/T138881#3059774 (10Paladox) p:05Low>03Normal [23:50:49] 06Labs, 10Labs-project-Phabricator, 10Phabricator: Applying role role::phabricator::main causes errors on instances - https://phabricator.wikimedia.org/T138881#2412999 (10Paladox)