[01:43:17] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Queth was created, changed by Queth link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Queth edit summary: Created page with "{{Tools Access Request |Justification=In order to utilize the following production level anon instance for a few post-secondary institutions across Canada: https://tools.wm..." [03:42:40] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Maxeeder was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=865009 edit summary: [03:45:41] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Gabrieloli was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=865012 edit summary: [03:48:30] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Queth was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=865025 edit summary: [06:44:33] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:11:04] hi all, i've got a tool which previously been using a database on svwiki.labsdb with the name p50380g51020_perfectbot but it seem to have been (re?)moved lately, can i find out what happened with it? [07:24:33] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [08:00:26] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/逆襲的天邪鬼 was created, changed by 逆襲的天邪鬼 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/%e9%80%86%e8%a5%b2%e7%9a%84%e5%a4%a9%e9%82%aa%e9%ac%bc edit summary: Created page with "{{Tools Access Request |Justification=Learn to make a bot, and plan to do some useful things on Chinese Wikipedia. |Completed=false |User Name=逆襲的天邪鬼 }}" [08:03:14] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/逆襲的天邪鬼 was modified, changed by 逆襲的天邪鬼 link https://wikitech.wikimedia.org/w/index.php?diff=865243 edit summary: [08:07:15] !log tools tools-bastion-03:~# chmod 640 /var/log/syslog [08:07:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [08:09:32] PROBLEM - Puppet staleness on tools-checker-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [09:19:54] (03PS1) 10Lokal Profil: Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) [09:20:35] (03CR) 10jenkins-bot: [V: 04-1] Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [09:26:47] (03PS2) 10Lokal Profil: Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) [09:27:23] (03CR) 10jenkins-bot: [V: 04-1] Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [09:35:15] (03PS3) 10Lokal Profil: Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) [09:37:40] (03CR) 10Lokal Profil: "Not sure why Jenkins fails. Local tox deals with it fine." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [11:15:50] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:55:52] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:10:34] (03CR) 10Lokal Profil: "Could we not just use yamllint[1]" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309844 (owner: 10Jean-Frédéric) [12:40:37] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 4 others: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#2670440 (10Aklapper) p:05Low>03Normal [13:46:52] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:48:18] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [14:26:52] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:18:58] (03CR) 10Lokal Profil: "> Could we not just use yamllint[1]" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309844 (owner: 10Jean-Frédéric) [16:32:22] (03CR) 10Jean-Frédéric: "Please submit here Lokal_Profil :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309844 (owner: 10Jean-Frédéric) [16:48:26] 06Labs: Switch off a specific wmflabs instance - https://phabricator.wikimedia.org/T146466#2671132 (10AlexMonk-WMF) @JanZerebecki does not have time to turn to power off a labs instance? [16:59:02] (03CR) 10Lokal Profil: "Note that this gives warnings (but not errors) for the docker-compose .yml files." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309844 (owner: 10Jean-Frédéric) [18:33:14] <|L> can someone help me? At two of my instances puppet is dead [18:33:57] <|L> valhallasw`vecto: can you help me with puppet (again)? Now two of my hosts got problems :( [18:36:05] |L: try to run it again with: sudo puppet agent -tv [18:36:22] |L: you can also check the logs with sudo cat /var/log/puppet.log [18:36:53] <|L> Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Exec[jenkins-deploy kvm membership] is already declared in file /etc/puppet/modules/contint/manifests/packages/androidsdk.pp:19; cannot redeclare at /etc/puppet/modules/contint/manifests/packages/labs.pp:86 on node cac.rcm.eqiad.wmflabs [18:36:55] <|L> ehm [18:37:21] <|L> hashar: do you know, which role I need to turn off? [18:37:36] oh [18:37:49] that is for CI ! [18:38:15] modules/contint/manifests/packages/androidsdk.pp: exec {'jenkins-deploy kvm membership': [18:38:15] modules/contint/manifests/packages/labs.pp: exec {'jenkins-deploy kvm membership': [18:38:26] I guess you can remove any contint:: class you have [18:38:32] <|L> ok :) [18:38:35] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [18:38:35] they are not meant to be used on labs project :D [18:38:51] <|L> guess the other instance has a different problem ;) [18:39:35] <|L> hashar: Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Connection refused - connect(2) [18:39:52] bah that one ... good question :] [18:40:15] <|L> (I got a phabricator there, maybe because phab use ssh port 22 too)? [18:40:20] looks like an issue with the remote puppet master [18:41:14] you can check the puppet conf by looking at /etc/puppet/puppet.conf [18:41:25] the [agent] section would have a server = [18:41:46] <|L> server = labs-puppetmaster-eqiad.wikimedia.org [18:42:19] <|L> did that one changed? [18:43:16] <|L> hashar: ^ [18:43:28] <|L> seems like that only happens at that instance [18:45:37] |L: I cant remember the proper one. But yeah maybe that changed [18:45:41] you can look at the other instance ? [18:46:14] <|L> server = labs-puppetmaster-eqiad.wikimedia.org [18:46:20] <|L> has that server too [18:46:29] so that is something else :( [18:47:33] <|L> lemme try if that is phabricator [18:47:50] looks like it is the puppet role :/ [18:47:56] sometime running puppet agent -tv --debug [18:47:59] helps [18:48:08] <|L> hm, but xenon has no active roles [18:48:27] <|L> Info: Creating a new SSL key for xenon.rcm.eqiad.wmflabs [18:48:27] <|L> Error: Could not request certificate: getaddrinfo: Name or service not known [18:48:28] <|L> Exiting; failed to retrieve certificate and waitforcert is disabled [18:50:26] |L, you're logged in? [18:50:44] <|L> Krenair: At the instance, wikitech, or horizon? 3 times yep [18:51:11] I can't get in [18:51:22] I might not have access [18:52:01] |L, can you paste your puppet.conf? [18:53:17] <|L> give me a moment [18:55:05] <|L> Krenair: http://pastebin.com/c6E6PXX0 [18:55:25] yeah that's a known bug [18:55:49] <|L> so what can I do to solve that bug? [18:56:18] get rid of the first part of the file [18:56:29] everything before "# This file is managed by Puppet!" on line 25 [18:56:51] <|L> so, delete line 1-24? [18:57:56] yes [18:58:38] <|L> Warning: Unable to fetch my node definition, but the agent run will continue: [18:58:39] <|L> Warning: Connection refused - connect(2) [18:58:40] <|L> and [18:58:47] <|L> Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Connection refused - connect(2) [18:58:47] <|L> Error: /File[/var/lib/puppet/lib]: Could not evaluate: Connection refused - connect(2) Could not retrieve file metadata for puppet://localhost/plugins: Connection refused - connect(2) [18:59:02] <|L> Krenair: ^ [19:00:16] ohhh right [19:00:20] it's not quite the same thing as usual [19:00:23] close, but [19:00:30] is that instance supposed to be a puppetmaster? [19:00:37] <|L> it was one [19:00:46] but not anymore? [19:00:55] <|L> I used it, load a role, and then disabled both [19:01:00] ok [19:01:01] <|L> yep, so it has no roles at the moment [19:01:06] replace puppet.conf with the first 24 lines of that paste [19:01:09] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [19:02:00] <|L> Krenair: Yay, that works. Thanks :) [19:02:20] <|L> oh, lot's of changes now.... :D [19:02:50] puppet was broken there for how long? [19:03:27] <|L> 24.09 [19:03:53] <|L> (7253 minutes ago) was displayed as the last succesful run [19:06:27] only 5 days ago? [19:37:18] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:53:00] hi, i would like graphoid to be auto-updated with scap3 on beta cluster. where could i get started, or whom should i get chocolates for? [19:53:26] hashar or bd808? [19:53:42] (looking at the history of edits at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated [19:54:14] As far as I know there is no "autoupdate" with scap3 at the moment, so you'd need a custom jenkins job to script it [19:55:54] probably something like the beta-code-update-eqiad and beta-scap-eqiad jobs but you could squish that into one job that updated the clone on deployment-tin and then ran scap3 to push it out [19:56:21] then you'd wire the new job up to zuul to be triggered on each push to your repo [19:57:05] bonus points for making the job templated so that any other scap3 project could use it [20:01:03] * yurik is now so scared he hides in the bushes and keeps very very quiet ... considering that i cannot even connect to deployment-tin for some reason... only to deployment-fluorine02 [20:02:21] nice try yurik, but I will not paint this fence for you :) [20:02:31] hehe :) [20:02:53] bd808, seriously though, is ssh deployment-tin.deployment-prep.eqiad.wmflabs the right way to ssh? [20:03:19] my ssh config might have gotten out of date [20:03:37] hmmm.. I just got "No route to host" from that. [20:04:18] yep, same here [20:04:28] i am able to connect to ssh deployment-puppetmaster.eqiad.wmflabs [20:04:38] but i'm not sure if that's the one to use for scaping [20:05:11] there is /srv/home/jenkins deploy [20:05:19] yurik: apparently the deploy server is now deployment-mira.deployment-prep.eqiad.wmflabs [20:05:34] sigh... do we have any docs about it? :) [20:05:57] it's "just like tin" except that it is a jessie host instead of a trusty one [20:06:26] I think this is a work in progress on the last of the trusty->jessie migration [20:06:57] I found the host by checking where the jenkins job is running -- https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ [20:07:33] yurik: I would guess that the folks in #wikimedia-releng would have told us/you that right off [20:08:11] <|L> bd808: hi, my vagrant says, that I need to update composer, but when I ran composer update, it says, that there is no composer.json. What do I need to do? [20:08:12] ah, yes, sorry, i just realized that #labs is different from #relend :( [20:08:14] my bad :( [20:08:29] thans bd808 ! [20:09:23] |L: where are you being told to update composer? That may help me figure out what you need to do [20:09:41] <|L> bd808: mwscript update.php [20:09:54] <|L> (triggert by vagrant git-update) [20:10:05] ok, so your mediawiki/vendor directory is out of date. [20:10:15] hmmm git-update should have taken care of that for you [20:10:59] but irregardless, you need to run `composer update` in your $IP directory. That would be /vagrant/mediawiki from inside the vm [20:11:07] <|L> ok, thx :) [23:09:31] (03PS1) 10BryanDavis: Allow OAuth authentication for anon users [labs/striker] - 10https://gerrit.wikimedia.org/r/313137 (https://phabricator.wikimedia.org/T144710) [23:09:33] (03PS1) 10BryanDavis: Add account creation initial screen [labs/striker] - 10https://gerrit.wikimedia.org/r/313138 (https://phabricator.wikimedia.org/T144710) [23:09:38] (03PS1) 10BryanDavis: Collect data needed to create a new LDAP account [labs/striker] - 10https://gerrit.wikimedia.org/r/313139 (https://phabricator.wikimedia.org/T144710) [23:09:40] (03PS1) 10BryanDavis: Add check for unique sul account, username, and shell account [labs/striker] - 10https://gerrit.wikimedia.org/r/313140 (https://phabricator.wikimedia.org/T144710) [23:09:42] (03PS1) 10BryanDavis: Add confirmation step to account creation wizard [labs/striker] - 10https://gerrit.wikimedia.org/r/313141 (https://phabricator.wikimedia.org/T144710) [23:09:44] (03PS1) 10BryanDavis: Add client side registration form validation [labs/striker] - 10https://gerrit.wikimedia.org/r/313142 (https://phabricator.wikimedia.org/T144710) [23:09:46] (03PS1) 10BryanDavis: Create LDAP and Striker users from registration form data [labs/striker] - 10https://gerrit.wikimedia.org/r/313143 (https://phabricator.wikimedia.org/T144710) [23:09:51] (03PS1) 10BryanDavis: Add striker.labsauth.utils.oauth_from_session helper [labs/striker] - 10https://gerrit.wikimedia.org/r/313144 (https://phabricator.wikimedia.org/T144710) [23:09:54] (03PS1) 10BryanDavis: Use consistent naming for accounts [labs/striker] - 10https://gerrit.wikimedia.org/r/313145 [23:09:56] (03PS1) 10BryanDavis: Add a goal prompt for SSH public key upload [labs/striker] - 10https://gerrit.wikimedia.org/r/313146 (https://phabricator.wikimedia.org/T144710) [23:11:21] (in voice of [[Count von Count]]) 10, 10 patches! AH AH AH! [23:14:57] (03CR) 10jenkins-bot: [V: 04-1] Create LDAP and Striker users from registration form data [labs/striker] - 10https://gerrit.wikimedia.org/r/313143 (https://phabricator.wikimedia.org/T144710) (owner: 10BryanDavis) [23:15:16] (03CR) 10jenkins-bot: [V: 04-1] Add striker.labsauth.utils.oauth_from_session helper [labs/striker] - 10https://gerrit.wikimedia.org/r/313144 (https://phabricator.wikimedia.org/T144710) (owner: 10BryanDavis) [23:16:03] (03CR) 10jenkins-bot: [V: 04-1] Use consistent naming for accounts [labs/striker] - 10https://gerrit.wikimedia.org/r/313145 (owner: 10BryanDavis) [23:17:07] (03CR) 10jenkins-bot: [V: 04-1] Add a goal prompt for SSH public key upload [labs/striker] - 10https://gerrit.wikimedia.org/r/313146 (https://phabricator.wikimedia.org/T144710) (owner: 10BryanDavis) [23:23:42] (03PS2) 10BryanDavis: Create LDAP and Striker users from registration form data [labs/striker] - 10https://gerrit.wikimedia.org/r/313143 (https://phabricator.wikimedia.org/T144710) [23:23:44] (03PS2) 10BryanDavis: Use consistent naming for accounts [labs/striker] - 10https://gerrit.wikimedia.org/r/313145 [23:23:46] (03PS2) 10BryanDavis: Add striker.labsauth.utils.oauth_from_session helper [labs/striker] - 10https://gerrit.wikimedia.org/r/313144 (https://phabricator.wikimedia.org/T144710) [23:23:48] (03PS2) 10BryanDavis: Add a goal prompt for SSH public key upload [labs/striker] - 10https://gerrit.wikimedia.org/r/313146 (https://phabricator.wikimedia.org/T144710)