[00:45:51] YuviPanda: I think we should make a phab task about singlenode, do an audit of who is using it and offer them help to migrate off. [01:01:51] 10PAWS, 6RevisionScoringAsAService: Install revscoring inside PAWS - https://phabricator.wikimedia.org/T120317#1879781 (10jayvdb) I'd like to see some documentation (just a wiki page?) on how to use revscoring with pywikibot before revscoring is installed into #PAWS, so the Pywikibot team has something to cons... [01:09:45] 10PAWS, 6RevisionScoringAsAService: Install revscoring inside PAWS - https://phabricator.wikimedia.org/T120317#1879785 (10Halfak) > how to use revscoring with pywikibot The docs are here http://pythonhosted.org/revscoring. Revscoring doesn't specifically integrate with pywikibot and it shouldn't need to. It... [01:10:38] 10PAWS, 6RevisionScoringAsAService: Install revscoring inside PAWS - https://phabricator.wikimedia.org/T120317#1879786 (10yuvipanda) I think it'd be nice to make it easy somehow to get scores for a revision when dealing with revisions in pwb. Maybe that can hit ores.wmflabs.org directly? [01:27:09] bd808: +1 [01:27:43] what project should I stuff that in? [01:28:03] bd808: http://tools.wmflabs.org/watroles/role/role::deprecated::mediawiki::install [01:28:06] bd808: just 'labs' I think [01:28:14] k. I'll start something [01:28:32] we should really do the same for labs-vagrant users too [01:28:41] bd808: +1 [01:29:05] bd808: I tried ~6months ago, but hit the 'editor engagement' team early on which has no clear owners nor clear users and gave up [01:29:39] *nod* it's not going to go fast or be easy likely [01:29:58] yeah [01:30:08] RIP ee [01:30:15] I suspect at some point 'so nobody is here to claim this I guess we'll shut it down for a fwe weeks and see how it goes' [01:40:12] 6Labs: Migrate projects using ::role::deprecated::labsvagrant to ::role::labs::mediawiki_vagrant - https://phabricator.wikimedia.org/T121477#1879857 (10bd808) 3NEW [01:46:43] 10PAWS, 7I18n, 7Upstream: i18n support - https://phabricator.wikimedia.org/T121478#1879871 (10jayvdb) 3NEW [01:56:03] PandaWithPopcorn: have you had a chance to look at the tools-elastic-* to see why they won't let me ssh in? [01:56:24] It feels like an ldap related thing [01:57:52] bd808: no, let me look now [01:58:02] too many things on fire now [01:58:22] sorry. it's not urgent [01:58:24] nah [01:58:29] nothing is on fire *right now* [01:58:40] > Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class ::role::toollabs::puppet::client for tools-elastic-01.tools.eqiad.wmflabs on node tools-elastic-01.tools.eqiad.wmflabs [01:58:42] Warning: Not using cache on failed catalog [01:58:58] ah ha [01:58:59] bd808: yeah, that class I killed last week. sorry didn't check for other users [01:59:07] what a jerk ;) [01:59:16] :D [03:08:04] bd808: the elastic stuff is sorted out now? [03:55:23] tools-bastion-01 has high cpu usage from time to time (caused by sshd?) [10:16:57] I'm getting 502 Bad gateway on ~25% of all pageloads at quarry.wmflabs.org... known fault or should I file a task? [10:25:31] 6Labs, 10Quarry: 502 Bad Gateway on HTTP-requests to quarry.wmflabs.org - https://phabricator.wikimedia.org/T121502#1880330 (10Stigmj) 3NEW [10:30:44] 6Labs, 10Tool-Labs: Inconsistent locale settings of different grid compute nodes. - https://phabricator.wikimedia.org/T121505#1880358 (10zhuyifei1999) 3NEW [10:32:15] 6Labs, 10Attribution-Generator, 6TCB-Team, 15User-bd808: Create labs projects for lizenzhinweisgenerator - https://phabricator.wikimedia.org/T120925#1880377 (10Addshore) Many thanks! :) [11:23:14] 6Labs, 10Tool-Labs: Inconsistent locale settings of different grid compute nodes. - https://phabricator.wikimedia.org/T121505#1880448 (10valhallasw) ``` */5 * * * * jsub -quiet -once -N jsubtest sh test.sh ``` ``` valhallasw@tools-bastion-01:/data/project/yifeibot$ cat test.sh #! /bin/sh date | tee jsubtest.t... [12:06:46] 6Labs, 10Tool-Labs: Inconsistent locale settings of different grid compute nodes. - https://phabricator.wikimedia.org/T121505#1880490 (10Stigmj) I have seen these as well.. Seems to be a german locale: rm: cannot remove `/data/project/pagecount/total-import-is-running': No such file or directory rm: cannot re... [12:17:40] 6Labs, 10Tool-Labs: Inconsistent locale settings of different grid compute nodes. - https://phabricator.wikimedia.org/T121505#1880526 (10valhallasw) Very strange. It seems connected to tools-exec-1201: (from tools.yifeibot) ``` tools-exec-1211: Tue Dec 15 11:30:12 UTC 2015 tools-exec-1219: Tue Dec 15 11:35:05... [13:11:36] 6Labs, 10Tool-Labs: Inconsistent locale settings of different grid compute nodes. - https://phabricator.wikimedia.org/T121505#1880609 (10valhallasw) ``` valhallasw@tools-exec-1201:/var/spool/gridengine/execd/tools-exec-1201$ sudo less /proc/10174/environ "/proc/10174/environ" may be a binary file. See it anyw... [13:18:18] 6Labs, 10Tool-Labs: Inconsistent locale settings of different grid compute nodes. - https://phabricator.wikimedia.org/T121505#1880613 (10zhuyifei1999) Not really relevant, but I changed the script to ``` echo `hostname`: `date` | tee jsubtest.txt >&2 ``` for the sake of outputting in a single line. [15:08:44] helllooooo [15:09:02] i belive i have messed up an instance in the analytics project (again) by forgetting to reenable puppet [15:09:06] kafka-jessie01 [15:09:10] help? :D [15:10:20] ottomata: want puppet re-enabled? [15:10:49] kafka-jessie01 is a Puppetmaster for itself (puppetmaster) [15:10:56] but puppet vomits on strange things [15:11:01] ConfigParser.NoOptionError: No option 'dbuser' in section: 'master' at /etc/puppet/modules/ssh/manifests/client.pp:9 on node kafka-jessie01.analytics.eqiad.wmflabs [15:11:54] yes please [15:11:58] hm [15:12:08] ah well [15:12:13] i mean, it is more a testbed node at the moment [15:12:17] mabye i'll just delete it :) [15:13:14] ok [15:14:14] thanks anyway :) [16:29:01] YuviPanda: sorry, wandered away last night for dinner and never came back. I'm still getting permission denied for my ssh key on tools-elastic-0[123]. [16:42:20] 6Labs, 10Labs-Infrastructure: maps-wma1 instance unresponsive (second time in 3 days) - https://phabricator.wikimedia.org/T121431#1880951 (10Andrew) 5Open>3Resolved a:3Andrew My key worked. The puppet cron was clearly disabled -- I started a puppet run and things seem reasonable now. Puppet still throw... [16:43:59] 6Labs, 10Labs-Infrastructure: maps-wma1 instance unresponsive (second time in 3 days) - https://phabricator.wikimedia.org/T121431#1880960 (10dschwen) Thanks Andrew. I was not aware that I disable puppet. I'll check out what I did there. [16:52:53] 10Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1880983 (10Edgars2007) OK, will get serious. I think the problem is still there, @yuvipanda. Or at least is related to this one. In last few days some 15 blank queries (those, which you get after pressing "New query") have appeare... [16:59:06] 6Labs: Thousands of duplicate /etc/pam.d/*.orig files which may be messing with our pam config - https://phabricator.wikimedia.org/T121533#1880999 (10Andrew) 3NEW [17:44:53] YuviPanda: ooh, more ldap tooling [17:45:01] (late response to watroles) [17:59:35] 6Labs: Thousands of duplicate /etc/pam.d/*.orig files which may be messing with our pam config - https://phabricator.wikimedia.org/T121533#1881426 (10chasemp) p:5Triage>3High [18:09:00] 6Labs, 5Patch-For-Review: Thousands of duplicate /etc/pam.d/*.orig files which may be messing with our pam config - https://phabricator.wikimedia.org/T121533#1881466 (10Andrew) I propose to merge https://gerrit.wikimedia.org/r/#/c/259296/ and then salt /etc/pam.d/*.orig to /etc/pambak Then it will be as if th... [18:53:51] !log rcm created rcm-3 [18:53:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL, Master [18:59:24] 10Gerrit-Patch-Uploader: Gerrit patch uploader should require people to supply valid-looking emails - https://phabricator.wikimedia.org/T121559#1881715 (10Legoktm) 3NEW [20:59:57] Krinkle: nagf gives a 404 actually. Can you look at it? [21:01:19] nagf-bh0p4 0/1 CrashLoopBackOff 8 7d [21:01:37] Seems k8s is having issues [21:02:23] yeah that's me [21:02:29] Krinkle: I built a new nagf image that doesn't work. hooray [21:02:33] let me revert to a working nagf image [21:02:38] k [21:03:17] ShiveringPanda: Is there docs somewhere on the basic steps for b8s in our environment? E.g. start/stop/restart service, current status (get pods?), and how to deploy/revert/see recent changes. [21:04:54] bbl [21:05:25] Krinkle: just general 'kubectl' documentation should work. I'll try to write up more 'howto' this week once rest of labs is stable [21:26:43] Krinkle: Luke081515 nagf back up btw [21:27:56] Thanks :) [21:30:18] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1882790 (10yuvipanda) The puppetmaster ldap variable is now gone from deployment-prep and integration hosts [21:32:39] !log rcm deleted rcm-3 and rcm-6; reboot of rcm-5 [21:32:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL, Master [21:36:02] Can someone help me? I disabled role::phabricator::labs for rcm-5, but when I reload the config page, this role is still active [21:38:48] Luke081515: Nova Resource or Nova Instance? [21:38:56] in the first case: takes a while to be updated [21:39:05] in the second case: try logging out and in again [21:39:06] NovaInstance [21:39:11] ok [21:39:38] https://tools.wmflabs.org/watroles/role/role::phabricator::labs does still list rcm-5, though [21:39:48] so maybe log out and in again, then try removing it again? [21:39:49] I tried it again now [21:39:55] with login and logout [21:40:01] is it still there? [21:40:28] try removing it again? [21:40:43] hm, ok, wait a moment [21:40:47] is there a save on that page... [21:41:02] yeah, and i pressed it [21:41:59] valhallasw`cloud: chasemp something odd is going on [21:42:07] watroles is seeing different results than ldapvi [21:42:42] Normaly all rcm-* instances should not be there, I disabled it on -2 and -4, tried -5 at the moment, but it seems like the agent don't want to let them go [21:44:31] Luke081515: when you run puppet on teh vm does it enforce that role? [21:44:55] chasemp: You mean with puppet agent -tv? [21:45:03] yes [21:45:29] ShiveringPanda: is watroles maybe pointed at old ldap? [21:45:33] he loads the data [21:45:34] yeah was tjinking [21:45:37] restarted it [21:45:39] let's see [21:49:07] chasemp: yep was pointing at wrong place [21:51:50] pointing it to the right thing promptly makes it 500 [21:51:51] yay [21:52:59] touching one thing makes 400 other things fail [21:59:11] fixed [22:00:45] 399 [22:01:16] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1882890 (10yuvipanda) all the old 'instancecreator*' ones are gone too now. [22:02:38] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1882891 (10yuvipanda) Removed 'basicpuppet' from instance 'basic' which was un-sshable anyway. [22:04:55] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1882897 (10yuvipanda) Removed 'hash_path_suffix' ldap variable, since it's used in only one instance and nowhere in ops/puppet [22:06:13] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1882918 (10yuvipanda) ```puppetVar: deployment_server_override= puppetVar: hadoop_cluster_name= puppetVar: hadoop_namenodes= puppetVar: ircecho_chans= puppetVar: ircecho_infile= puppetVar... [22:07:21] chasemp: 30 instances to go to kill $::puppetmaster [22:08:22] chasemp: https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#Set_up_project-wide_puppetmaster needs updating to point to the canonical way [22:09:07] that woudl be good too yeah [22:10:32] chasemp: I've edited it :D [22:10:35] https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#Set_up_a_multi-instance_self_hosted_puppetmaster [22:10:38] multi-instance 'just do not do it' [22:10:46] project-wide is already fine [22:14:27] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1882947 (10yuvipanda) Killed role::graphite::base::hostname as well since that too is a misnomer (confusion about hiera vs ldap I think?). [22:15:30] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1882953 (10yuvipanda) Killed the ircecho ldap stuff too. [22:18:18] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1882975 (10yuvipanda) Killed the ocg stuff too, since it's not being used in puppet anymore. [22:23:30] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1882996 (10yuvipanda) [22:24:21] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1339137 (10yuvipanda) Edited task description to match remaining ldap variables left. I've also removed the puppetmaster variable from the configure page so it can not be used easily anym... [22:26:37] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1883015 (10yuvipanda) kibana_authrealm and logstash_irc_name are gone, unused. [22:26:56] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1883016 (10yuvipanda) [22:54:27] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1883081 (10yuvipanda) [23:36:05] 6Labs: Create new labs project for dashiki - https://phabricator.wikimedia.org/T121598#1883316 (10madhuvishy) 3NEW a:3yuvipanda [23:36:40] 6Labs: Create new labs project for dashiki - https://phabricator.wikimedia.org/T121598#1883324 (10yuvipanda) 5Open>3Resolved Done [23:39:48] !log dashiki Added milimetric as projectadmin [23:39:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Dashiki/SAL, Master [23:51:12] 6Labs, 10MediaWiki-extensions-OpenStackManager: Permissions error on creating Hiera pages on wikitech - https://phabricator.wikimedia.org/T121602#1883360 (10madhuvishy) 3NEW [23:51:52] ShiveringPanda: I seem to have somehow broken the error logging in one of my project repos... [23:53:03] ShiveringPanda: I moved the error log to error.log.bak and created a new error.log, but it isn't collecting any errors. [23:53:21] Do I need to do something to make that work correctly? [23:53:22] kaldari: restart webservice? [23:53:32] kaldari: since otherwise the process just has old file handle [23:53:52] I thought I tried that, but maybe I didn't. I'll try it now... [23:54:31] 6Labs, 10MediaWiki-extensions-OpenStackManager: Permissions error on creating Hiera pages on wikitech - https://phabricator.wikimedia.org/T121602#1883378 (10Krenair) Does action=edit work anyway? [23:55:58] ShiveringPanda: hmm, well it's definitely writing to the new file now, but it's still not collecting any errors. Maybe I'm misunderstanding what the error.log is supposed to collect. Where are PHP errors logged? [23:56:42] php errors on a webservice should go to error.log IIRC [23:57:15] ShiveringPanda: Yeah, looks like it used to be doing that [23:57:48] ShiveringPanda: lemme test a few things and get back to you [23:59:05] kaldari: ok [23:59:13] kaldari: there might be a bit of a lag due to NFS