[00:29:14] yuvipanda: Imma going to need a password for the nginx proxy in front of elasticsearch I guess. Does that mean I should apply the same role you are using for the k8s stuff to switch puppet masters? [00:29:47] Or should we do something other than basic auth for Elasticsearch write actions? [00:33:03] bd808: hmm, I think the former. [00:33:14] works for me [00:33:21] bd808: to begin with, let's just disallow all non-local writes, and then we can switch puppetmasters I guess [00:34:15] well if I provision an elasticsearch cluster with no way to write to it ... [00:34:35] or are you thinking the first use you want is some sort of side loaded dumps? [00:34:47] bd808: no I just meant in terms of puppet commits / steps, I guess [00:34:58] since puppetmaster switching is still slightly wonky... [00:35:04] hmm maybe that means we should do it first! [00:35:07] a [00:35:20] bd808: you're right, let's just switch puppetmasters. there's the tools client role we can use... [00:35:28] k [01:11:43] !log reading-web-staging Added Gergő Tisza as project admin [01:11:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Reading-web-staging/SAL, Master [01:11:47] tgr: ^ [01:17:29] yuvipanda: https://gerrit.wikimedia.org/r/#/c/256618/ -- this may be the first nginx config I ever wrote and it's totally untested at this point so ... yeah. [01:17:49] * bd808 wanders off to play with his dog and cook dinner [01:17:59] bd808: thanks; I went with multimedia though, seemed more relevant [01:18:22] tgr: *nod* figured it might come in handy at some point either way [01:20:33] bd808: looks good to me :D we can setup and test I guess. [02:34:37] PROBLEM - Puppet failure on tools-worker-04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:16:12] 6Labs, 6operations: Kill the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#1846901 (10yuvipanda) 3NEW [05:08:31] 6Labs, 6operations, 7Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#1847021 (10yuvipanda) 3NEW [05:48:38] 6Labs: Move ldap host-record creation out of OpenStackManager and into sink - https://phabricator.wikimedia.org/T96677#1847069 (10Andrew) 5Open>3Resolved a:3Andrew This has been done for ages. [05:53:09] 6Labs, 10wikitech.wikimedia.org: Discrepancies in public IP instance lists between different wikitech UIs - https://phabricator.wikimedia.org/T62883#1847095 (10Andrew) I just fixed a bunch of SMW updating; things should be slightly more consistent now. [08:29:41] 6Labs, 6operations, 7Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#1847316 (10yuvipanda) p:5High>3Triage [08:32:22] 6Labs, 6operations, 5Patch-For-Review: Kill the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#1847327 (10yuvipanda) Ok, so right now `role::puppet::self` can do one of 3 things: # Be 'self hosted puppetmaster' - you have a p... [08:32:32] 6Labs, 6operations, 5Patch-For-Review: Kill the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#1847328 (10yuvipanda) a:3yuvipanda Ok, so right now `role::puppet::self` can do one of 3 things: # Be 'self hosted puppetmaster'... [08:34:32] 6Labs, 6operations, 5Patch-For-Review: Kill the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#1847332 (10yuvipanda) Doing all these in a nice way should also allow us to unify the location of the SSL stuff for puppet. Right n... [08:52:12] 6Labs, 10MediaWiki-extensions-OpenStackManager: Exception after logging in on wikitech, due to invalid call signature in hook OpenStackNovaUser::DynamicSidebarGetGroups - https://phabricator.wikimedia.org/T120185#1847360 (10TheDJ) 3NEW [08:53:43] 6Labs, 10MediaWiki-extensions-OpenStackManager: Exception after logging in on wikitech, due to invalid call signature in hook OpenStackNovaUser::DynamicSidebarGetGroups - https://phabricator.wikimedia.org/T120185#1847372 (10Legoktm) [08:53:45] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-General-or-Unknown, 10wikitech.wikimedia.org: Weird error message when creating a wikitech account that already exists - https://phabricator.wikimedia.org/T117553#1847373 (10Legoktm) [08:57:31] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-General-or-Unknown, 10wikitech.wikimedia.org: Weird error message when creating a wikitech account that already exists - https://phabricator.wikimedia.org/T117553#1847383 (10Legoktm) p:5Low>3Unbreak! >>! In T117553#1838347, @Andrew wrote: > Th... [08:58:01] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-General-or-Unknown, 10wikitech.wikimedia.org: MWException after account creation on wikitech - https://phabricator.wikimedia.org/T117553#1847385 (10Legoktm) [09:30:17] anyone know why the /public/dumps/pagecounts-raw structure is almost always a day behind? Is the copying/download a manual process? [09:37:39] (03PS1) 10Jcrespo: Adding some empty files required for mysql TLS testing [labs/private] - 10https://gerrit.wikimedia.org/r/256649 [09:40:53] 6Labs, 6operations, 5Patch-For-Review: Kill the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#1847481 (10yuvipanda) Also the different reasons instances have role::puppet::self applied: # For testing puppet changes # To act... [09:49:44] (03CR) 10Jcrespo: [C: 032] Adding some empty files required for mysql TLS testing [labs/private] - 10https://gerrit.wikimedia.org/r/256649 (owner: 10Jcrespo) [09:49:52] (03CR) 10Jcrespo: [V: 032] Adding some empty files required for mysql TLS testing [labs/private] - 10https://gerrit.wikimedia.org/r/256649 (owner: 10Jcrespo) [09:57:40] can I view an execution summary for a job I scheduled using crontab? [10:11:46] has this been verified to work? [10:11:47] https://wikitech.wikimedia.org/wiki/User:Russell_Blau/Using_pywikibot_on_Labs [10:12:31] lbertolotti: it should [10:13:42] lbertolotti: I used this to make it work: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Developing#Using_the_shared_Pywikibot_files_.28recommended_setup.29 [10:15:22] very well [10:15:25] 6Labs, 6operations, 5Patch-For-Review: Kill the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#1847541 (10yuvipanda) Hah, it looks like someone else already did most of the work! I can actually just set the puppetmaster hiera... [10:15:36] my bash_profile [10:15:36] has [10:15:47] export PATH=$PATH:/shared/pywikipedia/core/scripts [10:15:56] is that necessary? [10:16:12] 6Labs, 6operations, 5Patch-For-Review: Kill the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#1847543 (10yuvipanda) (I have tested both 1 and 2 and they both work) [10:17:34] tools.portalbox@tools-bastion-01:~$ python /shared/pywikipedia/core/scripts/version.py Pywikibot: [https] r-pywikibot-core.git (47bb909, g2, 2015/12/02, 14:40:06, ok) Release version: 2.0b3 requests version: 2.2.1 cacerts: /etc/ssl/certs/ca-certificates.crt certificate test: ok Python: 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] PYWIKIBOT2_DIR: Not set PYWIKIBOT2_DIR_PWB: Not set PYWIKIBOT2_NO_USER_CONFIG [10:17:49] lbertolotti: not necessary, but I guess it makes it possible to call the scripts directly [10:18:31] by looking at the above output [10:18:40] can you guess what is wrong? [10:19:53] lbertolotti: seems like you are missing the user-config.py file? [10:20:06] lbertolotti: is there anything wrong? [10:20:10] that looks OK to me? [10:20:55] lbertolotti: indeed, I get the same output (more or less)... [10:21:30] If it is correct [10:21:42] then why python always asks me my password for the bot? [10:22:47] tools.portalbox@tools-bastion-01:~$ python bot.py Password for user Portal box bot on wikipedia:en (no characters will be shown): [10:23:21] lbertolotti: have you performed step 5 in the guide from Russel Blau? [10:23:38] y [10:27:41] lbertolotti: is the username the same in both user-config and the password file? including caps and spaces/underscores [10:28:03] first of all [10:28:39] how many logins do I have to provide? [10:28:43] my bot login [10:28:47] or more stuff? [10:29:47] lbertolotti: my password-file only contains one line: ("my bot name", "my bot password") [10:31:27] lbertolotti: and in the user-config I have two linjes, one pointing to the password-file and one like: usernames['wikipedia']['no'] = u'my bot name' [10:32:23] very well, it seems the bot is now back to work [10:39:01] have any of you used the wikipedia app in a cellphone? [11:25:07] hi yuvipanda. i see there are brief periods of downtime. but i am not recognized with my public key in tools-login.wmflabs.org. what can I do? [11:41:47] hello. anyone? :) [11:55:50] valhallasw`cloud: do you know anyone who can help? [11:59:45] marmick: check if you are using the right username and key? [12:04:46] same as i used to (i haven't connected for around 3 months) [12:05:09] i keep my transmit credentials for the ftp [12:05:09] but i can't connect neither using ssh [12:05:52] i'm using the same mac computer. no changes as far as i know (and i'm the only user of the machine) [12:11:32] valhallasw`cloud: :S [12:20:33] marmick: ftp....? [12:21:41] sftp [12:21:44] marmick: what's your user name? there is no user 'marmick' as far as I can see [12:22:27] nope, it's marcmiquel. [12:23:33] mm, i get spam from "no-reply@tools.wmflabs.org", or is it not? [12:23:34] marmick: no login attempts from marcmiquel on tools-login... [12:23:55] gifti: I can't look in your mailbox, so no idea [12:24:22] gifti: please create a bug in phab with the email, including headers? [12:24:37] subject is "Scanned image from MX-2600N" and attachment is a word document … [12:25:04] valhallasw`cloud: this is very very weird. [12:25:04] marcmiquel$ ssh marcmiquel@tools-login.wmflabs.org [12:25:05] Permission denied (publickey,hostbased). [12:25:08] valhallasw`cloud: ok [12:26:09] marmick: ping tools-login.wmflabs.org == 208.80.155.130 ? [12:26:50] if so, please ssh -vv marcmiquel@tools-login.wmflabs.org and paste the output to a pastebin [12:26:55] yes [12:28:33] hm, Dec 3 12:27:17 tools-bastion-01 sshd[15720]: Connection closed by [preauth] [12:28:49] so it's an issue before the actual authentication [12:29:54] it looks like my public key disappeared from my os x config [12:31:21] ah! yes, that might be what's happening [12:31:45] I don't see any 'Failed publickey' messages in the log, which suggests there is no pubkey auth being tried [12:33:53] i've got the .pub file. i kept it when i generated it [12:33:53] there must be a way to set it up in the os x config again [12:34:57] marmick: no, you can't regenerate the private key from the public key [12:35:24] so you'll have to create a new key, and add the new public key to wikitech [12:38:06] 6Labs, 10Tool-Labs: Weird e-mails from tool labs - https://phabricator.wikimedia.org/T120210#1847873 (10Giftpflanze) 3NEW [12:38:11] this is very weird, how might it have disappeared from the os x config... [12:38:43] ok, i'm back to class [12:45:03] 6Labs, 10Tool-Labs: Weird e-mails from tool labs - https://phabricator.wikimedia.org/T120210#1847894 (10valhallasw) This e-mail indeed passed through the tool labs mailserver. In principle, this is OK (we are supposed to forward mails to gifti@tools.wmflabs.org to you), but we should obviously not accept mails... [12:47:45] 6Labs, 10Tool-Labs: Weird e-mails from tool labs - https://phabricator.wikimedia.org/T120210#1847899 (10valhallasw) see https://github.com/Exim/exim/wiki/SPF and /etc/exim4/conf.d/acl/30_exim4-config_check_rcpt [13:00:18] valhallasw`cloud: i finally got back in :) thanks. [14:09:52] 6Labs, 6operations, 7Puppet: Self hosted puppetmaster is broken - https://phabricator.wikimedia.org/T119541#1848004 (10chasemp) a:5chasemp>3None [14:22:00] 6Labs, 10Tool-Labs: Weird e-mails from tool labs - https://phabricator.wikimedia.org/T120210#1848046 (10chasemp) p:5Triage>3Low [14:22:14] 6Labs, 6operations, 7Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#1848050 (10chasemp) p:5Triage>3Normal [14:22:33] 6Labs, 6operations, 5Patch-For-Review: Kill the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#1848051 (10chasemp) p:5Triage>3Normal [14:22:48] 6Labs, 10Labs-Infrastructure, 6operations, 7Icinga: icinga config broken due to duplicate labs-ns1 / labcontrol2001 - https://phabricator.wikimedia.org/T120050#1848053 (10chasemp) p:5Triage>3Normal [14:22:58] 6Labs, 10Labs-Infrastructure, 6operations, 7Icinga: icinga config broken due to duplicate labs-ns1 / labcontrol2001 - https://phabricator.wikimedia.org/T120050#1843587 (10chasemp) @andrew, I think you know the deal here? [14:23:08] 6Labs, 10Labs-Infrastructure, 6operations, 7Icinga: labtestcontrol2001 should not make Icinga page us - https://phabricator.wikimedia.org/T120047#1848056 (10chasemp) p:5Triage>3Normal [14:23:16] 6Labs, 10Labs-Infrastructure, 6operations, 7Icinga: labtestcontrol2001 should not make Icinga page us - https://phabricator.wikimedia.org/T120047#1843529 (10chasemp) all silenced for now but yes agreed. [14:24:02] 6Labs, 10MediaWiki-extensions-OpenStackManager: Prevent empty service groups - https://phabricator.wikimedia.org/T120022#1848061 (10chasemp) p:5Triage>3Normal [14:24:26] 6Labs: Investigate alternatives to NovaProxy - https://phabricator.wikimedia.org/T119936#1848064 (10chasemp) p:5Triage>3Normal [14:25:05] 6Labs, 10Tool-Labs, 6operations, 7Icinga: tool labs instance distribution monitoring is broken - https://phabricator.wikimedia.org/T119929#1848066 (10chasemp) p:5Triage>3High [14:26:25] I have noticed some portals are not displaying correctly when I use the wikipedia app [14:30:47] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-eqiad: Locate and assign some MD1200 shelves for proper testing of labstore1002 - https://phabricator.wikimedia.org/T101741#1848074 (10chasemp) a:3coren [14:31:03] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-eqiad: labstore1002 issues while trying to reboot - https://phabricator.wikimedia.org/T98183#1848075 (10chasemp) a:3coren [14:34:56] 6Labs: Document, explain, diagram labs vlans and network setup - https://phabricator.wikimedia.org/T100529#1848076 (10chasemp) @andrew, I thought you did this to some extent since? [14:35:30] hello [14:36:04] I noticed the wikipedia app is not displaying some portal pages correctly [14:36:09] 6Labs, 10Tool-Labs: Puppetize gridengine master configuration - https://phabricator.wikimedia.org/T95747#1848078 (10chasemp) @yuvipanda, whao is this still not in puppet? [14:36:52] (03CR) 10Alexandros Kosiaris: [C: 031] move gsbmonitoring to monitor/gsb.yaml [labs/private] - 10https://gerrit.wikimedia.org/r/256601 (owner: 10Dzahn) [14:39:44] 6Labs, 10Tool-Labs, 7Puppet: Fully puppetize Grid Engine (Tracking) - https://phabricator.wikimedia.org/T88711#1848087 (10chasemp) [14:39:45] 6Labs, 10Tool-Labs: Puppetize gridengine master configuration - https://phabricator.wikimedia.org/T95747#1848086 (10chasemp) [14:40:09] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Job queue broken for labswiki (jobs for wikitech.wikimedia.org are not running) - https://phabricator.wikimedia.org/T117394#1848089 (10Krinkle) [14:41:15] 6Labs: Neutron networking, with IPv6 at eqiad - https://phabricator.wikimedia.org/T85611#1848093 (10chasemp) 5Open>3declined a:3chasemp yes but this will be handled down the line with other tasks and can't be addressed now [14:41:20] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Job queue broken for labswiki (jobs for wikitech.wikimedia.org are not running) - https://phabricator.wikimedia.org/T117394#1772839 (10Krinkle) This is causing problems on wikitech since link updates are not running. E.g. pages added or r... [14:41:49] 6Labs, 10Tool-Labs: Trusty instances do not show the motd banners - https://phabricator.wikimedia.org/T85307#1848101 (10chasemp) a:3coren this should be fixed now I believe? [14:42:30] 6Labs: Document, explain, diagram labs vlans and network setup - https://phabricator.wikimedia.org/T100529#1848104 (10chasemp) [14:44:48] 6Labs, 10Tool-Labs: Weird e-mails from tool labs - https://phabricator.wikimedia.org/T120210#1848109 (10coren) It's not as clearcut as that - you'd expect people receiving email to their `*@tools.wmflabs.org` addresses to be able to reply with that same address as From:; so it's not immediately clear that such... [14:46:46] 6Labs, 10Tool-Labs: Trusty instances do not show the motd banners - https://phabricator.wikimedia.org/T85307#1848111 (10coren) 5Open>3Resolved It is, as a side effect of the PAM fix. [14:46:48] 6Labs, 10Beta-Cluster-Infrastructure, 10Labs-Infrastructure, 6operations: On deployment-prep, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837#1848113 (10coren) [14:53:59] 6Labs, 10Labs-Team-Backlog: Find an Ops-internal project to host on labs bare metal. - https://phabricator.wikimedia.org/T118588#1848127 (10chasemp) From T117095 @robh we stalled that task (I'm sorry we should have removed #hardware-requests as well but I'm jumping the specs layout over to this issue as it's a... [14:54:24] 6Labs, 10hardware-requests, 6operations: Get Ops bare metal test server - https://phabricator.wikimedia.org/T118588#1848131 (10chasemp) [14:58:04] 6Labs, 10hardware-requests, 6operations: Get Ops bare metal test server - https://phabricator.wikimedia.org/T118588#1848152 (10chasemp) [14:58:16] 6Labs, 10MediaWiki-Special-pages, 10wikitech.wikimedia.org, 7Regression, 7Wikimedia-log-errors: Special:MovePage throws MWException "Hook SMWParseData::onTitleMoveComplete has invalid call signature" - https://phabricator.wikimedia.org/T120218#1848153 (10Krinkle) [15:20:31] 6Labs, 10Tool-Labs: Weird e-mails from tool labs - https://phabricator.wikimedia.org/T120210#1848218 (10scfc) I think people using `tools.wmflabs.org` as their domain for outgoing mails should use the proper mail server (which is tricky, but doable if you //really// want to), so I don't have any problem with p... [15:21:27] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Job queue broken for labswiki (jobs for wikitech.wikimedia.org are not running) - https://phabricator.wikimedia.org/T117394#1848232 (10Joe) FWIW, labswiki is not supposed to use the cluster's jobqueue at all: https://github.com/wikimedia... [15:22:16] 6Labs, 10Tool-Labs: Puppetize gridengine master configuration - https://phabricator.wikimedia.org/T95747#1848236 (10scfc) 5duplicate>3Open This task only covers the master instance(s); T88711 covers all others as well. [15:26:37] 6Labs, 10Tool-Labs: Weird e-mails from tool labs - https://phabricator.wikimedia.org/T120210#1848249 (10valhallasw) > you'd expect people receiving email to their *@tools.wmflabs.org addresses to be able to reply with that same address as From //that// doesn't work, actually, because we publish an SPF record:... [15:30:19] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Job queue broken for labswiki (jobs for wikitech.wikimedia.org are not running) - https://phabricator.wikimedia.org/T117394#1848260 (10Joe) I just confirmed with tcpdump: silver (wikitech) is submitting jobs to the jobqueue even if it sho... [15:34:05] 6Labs, 10Tool-Labs: correctly envelope forwarded email - https://phabricator.wikimedia.org/T120225#1848287 (10valhallasw) 3NEW [15:35:21] 6Labs, 10Tool-Labs, 7Puppet: Fully puppetize Grid Engine (Tracking) - https://phabricator.wikimedia.org/T88711#1848306 (10chasemp) [15:35:22] 6Labs, 10Tool-Labs: Puppetize gridengine master configuration - https://phabricator.wikimedia.org/T95747#1848307 (10chasemp) [15:35:31] 6Labs, 10Tool-Labs: Puppetize gridengine master configuration - https://phabricator.wikimedia.org/T95747#1199397 (10chasemp) >>! In T95747#1848236, @scfc wrote: > This task only covers the master instance(s); T88711 covers all others as well. right gotcha, linking in as a blocker then to make the tracking tas... [15:36:28] 6Labs, 10Tool-Labs: correctly envelope forwarded email - https://phabricator.wikimedia.org/T120225#1848312 (10valhallasw) p:5Triage>3Low [15:37:30] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Job queue broken for labswiki (jobs for wikitech.wikimedia.org are not running) - https://phabricator.wikimedia.org/T117394#1848313 (10Joe) a:3Joe [15:38:05] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Job queue broken for labswiki (jobs for wikitech.wikimedia.org are not running) - https://phabricator.wikimedia.org/T117394#1772839 (10Joe) Found the problem - the jobqueue file gets included disregarding the fact that we're on labswiki... [16:00:30] is there a better channel for talking about the wikipedia app? [16:04:55] mobile app? [16:18:59] yes, the mobile app [16:19:22] I noticed some portal pages are not displaying correctly [16:19:28] #wikimedia-mobile works then [16:19:39] and/or file a task in phabricator [16:29:21] 6Labs, 10hardware-requests, 6operations: Get Ops bare metal test server - https://phabricator.wikimedia.org/T118588#1848510 (10chasemp) Desired recommendation: Dell PowerEdge R420, Dual Intel Xeon E5-2440, 32GB Memory, Dual 300GB SSD, Dual 500GB Nearline SAS [16:36:35] 6Labs, 10hardware-requests, 6operations: Get Ops bare metal test server - https://phabricator.wikimedia.org/T118588#1848542 (10mark) As this is a test server with limited life time, we can use an out of warranty spare for this. (Approved) [16:49:51] 6Labs, 10MediaWiki-Special-pages, 10wikitech.wikimedia.org, 7Regression, 7Wikimedia-log-errors: Special:MovePage throws MWException "Hook SMWParseData::onTitleMoveComplete has invalid call signature" - https://phabricator.wikimedia.org/T120218#1848581 (10mwjames) I added a similar issue with https://phab... [17:09:49] 6Labs, 10hardware-requests, 6operations: Get Ops bare metal test server - https://phabricator.wikimedia.org/T118588#1848673 (10chasemp) >>! In T118588#1848510, @chasemp wrote: > Desired recommendation: Dell PowerEdge R420, Dual Intel Xeon E5-2440, 32GB Memory, Dual 300GB SSD, Dual 500GB Nearline SAS >>! In... [17:11:31] 6Labs, 10wikitech.wikimedia.org: Can't reset password on wikitech (Unicode passwords not accepted), due to LDAP/opendj? - https://phabricator.wikimedia.org/T58114#1848685 (10Nemo_bis) I'll try to reproduce this again after the switch to openldap announced at https://lists.wikimedia.org/pipermail/wikitech-l/201... [17:17:26] 6Labs, 10Labs-Infrastructure, 5Patch-For-Review: Investigate decommissioning labcontrol2001 - https://phabricator.wikimedia.org/T118591#1848723 (10Andrew) a:5Andrew>3mark labcontrol2001 is now on, but not running any useful services. It can be shut down or renamed at any time. I propose that for the sh... [17:23:02] 6Labs, 10hardware-requests, 6operations: Get Ops bare metal test server - https://phabricator.wikimedia.org/T118588#1848764 (10RobH) If approved, please remove promethium from the spares page, and in the edit summary, please list this task #. Additionally, I typically resolve the #hardware-request once its... [17:44:06] paravoid: hhttps://gerrit.wikimedia.org/r/#/c/256693/ is the final cleanup/sanity round. [17:45:26] 6Labs, 10MediaWiki-Special-pages, 10wikitech.wikimedia.org, 7Regression, 7Wikimedia-log-errors: Special:MovePage throws MWException "Hook SMWParseData::onTitleMoveComplete has invalid call signature" - https://phabricator.wikimedia.org/T120218#1848897 (10bd808) Likely caused by https://gerrit.wikimedia.o... [17:52:00] 6Labs, 10Labs-Team-Backlog, 5Patch-For-Review: Labs: create a new scheme for /etc/security/access.conf customization - https://phabricator.wikimedia.org/T120106#1848999 (10coren) [17:53:22] Coren: oh wow [17:53:31] I have a few comments [17:53:35] but before we go into the details [17:53:39] perhaps we should split the commit into two? [17:53:50] one for the introduction of the module and one for switching existing use cases over? [17:54:30] paravoid: That was my original intent, but doing so required a lot of stopgap fixes here and there to avoid conflict. I can try to split them back up, but I don't think this gains in clarity all that much. [17:54:40] (See, iirc, changeset 3) [17:54:46] alright [17:55:58] changeset 4, actually. But yeah. Would you rather I still tried to split it up? Thankfully, the use cases are relatively few and all fairly straightforward. [17:56:07] no that's ok [17:57:12] I'm sure there's a lot of quibbles about the implementation details, but /man/ the manifests that use this are improved a lot. :-) [18:00:36] wow this is good work [18:02:07] ty [18:04:15] The ldap class is still a mess that needs attention, mind you. I didn't try to fix *that* too. :-) [18:04:22] s/class/module/ [18:10:20] (Actually, I think the next step would be to get rid of that ldap::role::client::labs kludge and put the relevant code in role::labs::instance where it makes sense. [18:10:37] That'd allow getting rid of ldap::client::pam [18:10:46] (Which really has no business in an ldap class) [18:11:35] At least the $::realm == 'labs' part anyways. [18:12:58] yeah, we were discussing the mess that the ldap module is with yuvipanda the other day [18:13:31] he said he was planning to fix it but is waiting for the opendj migration to be over [18:13:41] * Coren nods. [18:13:42] however I pointed out that ldap::client::* is completely orthogonal to opendj/ldap [18:13:51] so it might be a good idea to start small [18:14:05] and start refactoring it piece by piece [18:14:16] Yeah, I'm seeing a simple clean way to get rid of the $::real == 'labs' part which is going to simplify any future cleanup. [18:14:29] * Coren works on the other patch for now. [18:15:03] In re 'wikimedia-*' for names, I was mostly trying to avoid any possible conflict with package additions? [18:15:42] afaict, there is no provision to distinguish apt-added things from local policy otherwise. [18:17:50] yeah [18:18:49] (03CR) 10Dzahn: [C: 032] move gsbmonitoring to monitor/gsb.yaml [labs/private] - 10https://gerrit.wikimedia.org/r/256601 (owner: 10Dzahn) [18:18:57] (03CR) 10Dzahn: [V: 032] move gsbmonitoring to monitor/gsb.yaml [labs/private] - 10https://gerrit.wikimedia.org/r/256601 (owner: 10Dzahn) [18:34:09] Hmm I can't seem to hit nobelium.eqiad.wmnet:9200 from toollabs [18:37:36] hm, I don't remember what the original request for access was but afaik its houldn't be special in a bad way [18:39:41] paravoid: Latest changeset makes everything happy, I think. [18:41:01] paravoid: I've kept both under security:: as they are clearly interrelated in my mind, but if you really feel splitting them is better I've no beef about it. [18:41:30] (but I've organized them better under it) [18:58:45] 6Labs, 10Quarry, 10Labs-Infrastructure, 7HTTPS: Quarry should be HTTPS-only - https://phabricator.wikimedia.org/T107627#1849414 (10Dzahn) [18:59:30] 6Labs, 10Quarry, 10Labs-Infrastructure, 7HTTPS: Quarry should be HTTPS-only - https://phabricator.wikimedia.org/T107627#1849419 (10Dzahn) looks like https works just fine and only a redirect is missing from http->https, will look into it [18:59:35] 6Labs, 10Quarry, 10Labs-Infrastructure, 7HTTPS: Quarry should be HTTPS-only - https://phabricator.wikimedia.org/T107627#1849420 (10Dzahn) a:3Dzahn [19:12:14] Coren: paravoid I don't want to cookie lick the ldap module cleanup :) considering I've fallen into the puppet/ module hole now I might not have time for the ldap/ hole for a while [19:12:28] :-) [19:12:50] yuvipanda: That's okay, I think I'll tackle it gradually anyways as I touch related matters. [19:33:25] bd808: wanna do the es stuff today? :) [19:34:27] yuvipanda: I need to deploy scholarships in a couple of hours but then I could poke at elastic after that [19:34:35] bd808: \o/ awesome [19:34:37] cool [19:34:41] * bd808 should probably do other things but... shiny [19:35:00] yesss shiny [19:38:55] 6Labs: PoC bare-metal server allocation in labs -- bootstrap mode - https://phabricator.wikimedia.org/T95185#1849627 (10chasemp) [19:39:03] 6Labs: PoC bare-metal server allocation in labs -- bootstrap mode - https://phabricator.wikimedia.org/T95185#1182578 (10chasemp) [19:39:13] 6Labs: PoC bare-metal server allocation in labs -- bootstrap mode - https://phabricator.wikimedia.org/T95185#1182578 (10chasemp) [19:39:14] 6Labs, 10hardware-requests, 6operations: Get Ops bare metal test server - https://phabricator.wikimedia.org/T118588#1849633 (10chasemp) 5Open>3Resolved a:3chasemp [19:48:59] PROBLEM - Puppet staleness on tools-worker-04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [19:49:08] ^ is me I guess [20:08:38] * Coren hates daemons that just die quietly. [20:23:40] I'm trying to figure out a sane way to deal with sudo during package building. Package building needs sudo, but I moved everything related to packaging into a tool (tools.dpkg). But tools.dpkg cannot sudo, so it all becomes a big mess of permissions between valhallasw/tools.dpkg/root. [20:24:55] and mostly it incentivizes doing stuff as root that shouldn't be done as root [20:25:18] so... should we give tools.dpkg the right to sudo cowbuilder? [20:25:44] um [20:25:44] Why would... Oh, cowbuilder vs fakeroot. Gotcha. [20:25:48] hmm [20:25:56] what does the 'dpkg' tool give you? [20:25:58] just roles, I suppose [20:26:15] yuvipanda: just a place to put files in a central place, mostly [20:26:21] I would say -> move to a project. Tools is /really/ not designed with the optic that endusers can become root. [20:26:38] sorry, let me clarify. This is for packages that end up in tools aptly [20:26:49] could still be moved to a seperate project, I suppose [20:26:52] valhallasw`cloud: I think you should just build them as your own user and just use /data/project/dpkg :) [20:27:12] valhallasw`cloud: or, just setup cowbuilder on the aptly role itself, and just put it in /srv somewhere (and setup a backup job to NFS) [20:27:19] valhallasw`cloud: Then why cowbuilder rather than just fakeroot/debuild? [20:27:46] yuvipanda: yeah, and then someone can just `take` if the permissions mess up too badly [20:27:53] valhallasw`cloud: yeah [20:27:57] still, needs a place to store the .dsc et al files [20:28:30] Coren: because fakeroot/debuild require you to install system-wide packages during build [20:28:48] and I never quite could get a backport working sanely there [20:29:11] whereas cowbuiler (i.e. role::package_builder) mostly works out-of-the-box [20:29:35] Yeah, that makes sense. [20:29:50] also it allows us to build for other distros than the current one [20:30:05] so building for precise and trusty on the same host, rather than requiring to seperate building hosts [20:30:36] yuvipanda: I suppose we could move packaging off tools altogether; aptly also works cross-project. Hm. [20:31:15] valhallasw`cloud: could do [20:31:45] or do what I do and just use them directly if they will work or not do them if they need real backporting [20:33:18] hm, I actually thought I had to recompile stuff to not end up in dll hell [20:33:44] 6Labs, 10MediaWiki-Special-pages, 10wikitech.wikimedia.org, 7Regression, 7Wikimedia-log-errors: Special:MovePage throws MWException "Hook SMWParseData::onTitleMoveComplete has invalid call signature" - https://phabricator.wikimedia.org/T120218#1849834 (10Reedy) ``` $params = array( &$this->oldTitle, &$... [20:34:53] valhallasw`cloud: depends. on pure python / pure data stuff you can get away with it [20:34:58] anything requireing compilation yah [20:35:27] right [20:37:18] yuvipanda: oh, right, could you jot down a few lines on how to rebuild & restart grrrit-wm? [20:40:35] valhallasw`cloud: hmm, so the biggest issue there is how to push to the image [20:40:35] Oh, a beautiful logo, when I ssh to tools-bastion-01 :D [20:40:43] which is at yuvipanda/grrrit [20:40:54] oh, because that requires your password rather than just stuff in your $HOME [20:42:17] valhallasw`cloud: right [20:42:21] valhallasw`cloud: so the right solution is... [20:42:31] https://phabricator.wikimedia.org/T118758 [20:43:02] yuvipanda: in the meanwhile create a 'grrrit-wm' account on dockerhub and put that password somewhere? [20:43:22] valhallasw`cloud: I'm looking now to see if I can give other people push access [20:43:28] ah ok [20:43:56] valhallasw`cloud: I can! [20:44:00] do you have a dockerhub account? [20:44:51] 6Labs, 6operations, 10ops-eqiad: setup promethium in eqiad in support of T95185 - https://phabricator.wikimedia.org/T120262#1849886 (10chasemp) [20:46:02] yuvipanda: now I do [20:46:18] valhallasw`cloud: username? [20:46:32] yuvipanda: make a guess ;-D [20:46:34] valhallasw :-) [20:46:44] valhallasw`cloud: ok done [20:46:50] I would be so annoyed if someone had nicked my nick [20:46:51] valhallasw`cloud: you can push to yuvipanda/grrrit now [20:46:56] so this is pretty interesting https://hub.docker.com/r/continuse/openstack-controller/ [20:47:05] yuvipanda: mmmm [20:47:39] valhallasw`cloud: we should test that [20:47:50] yuvipanda: yes, because I see nothing in the web interface [20:47:57] yuvipanda: tell me how, I'll make notes and dump them online [20:47:58] valhallasw`cloud: also https://github.com/yuvipanda/paws/commit/da79955fe3627568fb89208f5f1fd4d42cfba104 never got merged heh [20:48:22] valhallasw`cloud: err, i meant https://gerrit.wikimedia.org/r/#/c/238622/ [20:48:48] (03CR) 10Merlijn van Deen: [C: 032] Add simple Dockerfile [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/238622 (owner: 10Yuvipanda) [20:49:01] valhallasw`cloud: ok, so 1. install docker!, 2. sudo docker login, 3. git clone the repo, 4. 'sudo docker build -t yuvipanda/grrrit .' 5. 'sudo docker push yuvipanda/grrrit' [20:49:03] done [20:49:09] repeat 4 / 5 as necessary [20:49:11] uuuh [20:49:20] and k8s automatically pulls the new version? [20:49:23] what is this black magic [20:49:31] no [20:49:34] it doesnt' yet :) [20:49:36] also can I do this on toollabs? [20:49:38] but this is the pushing step [20:49:55] valhallasw`cloud: temporarily, yes, if you are ok with putting your credentials there :) [20:50:02] valhallasw`cloud: there's tools-registry-01 [20:50:08] which has appropriate things [20:50:16] valhallasw`cloud: if you want to do it there [20:50:19] hm, docker is 70kB [20:50:23] I suppose I can spare that :-p [20:51:14] ...that's because it's not docker but osmething else. ugh [20:51:18] yeah [20:51:23] they have their own repo [20:51:27] or jessie-backports [20:51:36] let me try docker.io :P [20:51:50] it's actually in trusty-universe as well [20:52:11] also why sudo docker login? [20:52:15] and not as normal user? [20:52:44] because it needs to do weird stuff. ok... [20:52:49] (03Merged) 10jenkins-bot: Add simple Dockerfile [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/238622 (owner: 10Yuvipanda) [20:53:19] valhallasw`cloud: yeah docker group exists too [20:53:25] valhallasw`cloud: but that's root equivalent... [20:54:11] valhallasw@maeglin:grrrit$ sudo docker build -t yuvipanda/grrrit master 0 [20:54:11] docker: "build" requires 1 argument. See 'docker build --help'. [20:54:23] doesn't take build . [20:54:42] valhallasw`cloud: sudo docker build -t yuvipanda/grrrit . [20:54:45] oh, the . has to be after the parameters [20:54:45] ? [20:54:47] ugh. [20:54:48] yeah [20:55:11] ok, well, bye-bye hard drive space [20:55:23] heh [20:55:31] I hope my 1.1G free is enough :P [20:55:34] :P [20:55:36] should be [20:55:42] eventually we want to have a docker builder service [20:55:50] just push to a git repo and boom docker image built [20:56:28] are you guys sorting out grrrit-wm on k8s? [20:57:01] ...600M free.... [20:57:17] I love how efficient this docker thing is :P [20:57:45] ...300M free... [20:57:59] yes, that just fits. [20:58:28] ...and then it still has to install node. Dear lord :P [20:58:47] ok, built! [20:59:18] pushing.... [21:00:10] chasemp: grrrit-wm is already on k8s [21:00:20] chasemp: but only I could deploy, so giving valhallasw`cloud access too now [21:00:37] ah yes the first is what I thought and the second seems great [21:00:38] yuvipanda: I hope the next time is not going to be this horribly slow [21:00:53] valhallasw`cloud: yeah, rebuilds are faster since stuff is cached [21:01:21] I don't get why it needs to push 20 images [21:01:33] it's pusing 20 layers [21:01:39] and it usually finds that most are cached [21:01:45] why does it need to push them? because they already exist? [21:01:47] but this one probably not because I haven't pushed in a while [21:02:01] even then for things like [21:02:04] FROM debian:jessie [21:02:08] I think it's only pushing metadata [21:02:27] pushing 128MB [21:02:28] >_< [21:05:06] and the nodejs stuff [21:05:09] blame npm [21:05:13] it pulls in the whole universe [21:05:18] clearly :D [21:05:38] 10Wikibugs, 5Patch-For-Review: wikibugs - throttle output, don't get kicked for flooding - https://phabricator.wikimedia.org/T112032#1850012 (10Dzahn) note to self: i just got kicked from #wikimedia-releng [21:09:12] Digest: sha256:9a69c2d5b982e0e414963cd178804b0e1684e3d976df01b0ced4e1e203308ae0 [21:09:14] ok, pushed :-p [21:10:06] valhallasw`cloud: wooo [21:10:13] valhallasw`cloud: ok, now ssh to tools-k9s-master-01 :) [21:10:27] and sudo as me (mostly because I haven't puppetized .kube/config yet) [21:11:41] ok [21:12:20] valhallasw`cloud: so [21:12:22] kubectl --user=lolrrit-wm --namespace=lolrrit-wm get pods [21:12:25] will give you list of running pods [21:12:33] (ignore the one stuck in Terminating state, gotta fix that) [21:12:35] you can do [21:12:42] kubectl --user=lolrrit-wm --namespace=lolrrit-wm delete pods/ [21:12:46] so if you delete the running pod [21:12:52] it'll just pull in new container and run it [21:13:20] because of the replicationcontroller? [21:13:35] valhallasw`cloud: yup [21:13:58] valhallasw`cloud: the replication controller has image set to - image: yuvipanda/grrrit:latest [21:14:03] the :latest will pull the image down [21:14:10] ah [21:14:11] ok [21:14:17] you can look at the rc spec by [21:14:19] kubectl --user=lolrrit-wm --namespace=lolrrit-wm get rc -o yaml [21:14:21] so that's why it's waiting for termination? [21:14:21] the -o yaml outputs y aml [21:14:49] valhallasw`cloud: ok, so it's at crashloopbackoff [21:14:54] means it started and kept crashing [21:14:59] after the 5th attempt it is stopped [21:15:00] >_< [21:15:05] so your new image didn't work. [21:15:12] can I go back to fabric now? [21:15:20] no [21:15:23] :P [21:15:27] this is all super rough [21:15:28] this is horrible [21:15:30] indeed [21:15:36] which is why nobody except me should be doing this [21:15:39] how the heck do I get logs [21:15:48] kubectl --user=lolrrit-wm --namespace=lolrrit-wm logs pods/grrrit-rqtuh doesn't owrk [21:16:39] yeah, that's a bug I need to fix (logs only works when container is running) [21:16:42] let me get the logs [21:17:08]