[07:26:23] 10PAWS: How to change the default uploading time in PAWS - https://phabricator.wikimedia.org/T147832#2704902 (10Info-farmer) [07:27:43] 10PAWS: How to change the default uploading time in PAWS - https://phabricator.wikimedia.org/T147832#2704902 (10Info-farmer) [07:33:20] 10PAWS: How to change the default uploading time in PAWS - https://phabricator.wikimedia.org/T147832#2704916 (10Info-farmer) [07:45:39] (03PS1) 10Giuseppe Lavagetto: Convert hiera to the form expected by the new backend [labs/private] - 10https://gerrit.wikimedia.org/r/315204 (https://phabricator.wikimedia.org/T147403) [08:08:41] 06Labs, 10Tool-Labs: Java Runtime on Toollabs - not up to date? - https://phabricator.wikimedia.org/T147833#2704935 (10Gradzeichen) [08:50:51] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:25:49] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:28:47] (03PS1) 10Jean-Frédéric: Lint SQL files using php-sqllint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/315226 (https://phabricator.wikimedia.org/T132641) [11:00:03] PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:00:36] PROBLEM - Puppet run on tools-exec-1213 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:01:38] PROBLEM - Puppet run on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:02:20] PROBLEM - Puppet run on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:02:32] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:03:22] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:03:36] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:04:12] PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:05:10] PROBLEM - Puppet run on tools-exec-1206 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:05:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:05:22] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:05:48] PROBLEM - Puppet run on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:05:50] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:06:52] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:08:35] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:12:41] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:13:03] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:13:05] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:13:33] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:15:19] PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:15:21] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:28:53] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Convert hiera to the form expected by the new backend [labs/private] - 10https://gerrit.wikimedia.org/r/315204 (https://phabricator.wikimedia.org/T147403) (owner: 10Giuseppe Lavagetto) [11:35:38] (03PS1) 10Giuseppe Lavagetto: Fix labtest-instances... [labs/private] - 10https://gerrit.wikimedia.org/r/315233 [11:37:06] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Fix labtest-instances... [labs/private] - 10https://gerrit.wikimedia.org/r/315233 (owner: 10Giuseppe Lavagetto) [11:51:39] RECOVERY - Puppet run on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [11:55:04] RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [11:55:23] RECOVERY - Puppet run on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [11:55:35] RECOVERY - Puppet run on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:31] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:58:35] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [11:59:11] RECOVERY - Puppet run on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [12:00:09] RECOVERY - Puppet run on tools-exec-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [12:00:21] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [12:00:49] RECOVERY - Puppet run on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [12:08:35] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [12:10:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [12:10:52] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:11:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:12:41] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:13:33] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [12:15:21] RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [12:18:03] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [12:18:05] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [12:36:51] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [12:37:49] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:37:57] (03CR) 10Lokal Profil: [C: 04-1] "Looks good and works when I test it." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/315226 (https://phabricator.wikimedia.org/T132641) (owner: 10Jean-Frédéric) [12:44:03] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:44:04] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:46:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:49:20] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:56:48] PROBLEM - Puppet run on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:00:22] (03PS2) 10Jean-Frédéric: Lint SQL files using php-sqllint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/315226 (https://phabricator.wikimedia.org/T132641) [13:01:03] PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:05:55] (03CR) 10Jean-Frédéric: "> But you should now also drop the "sed -i 's/^connect.*$//" bit in" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/315226 (https://phabricator.wikimedia.org/T132641) (owner: 10Jean-Frédéric) [13:06:13] (03CR) 10Lokal Profil: "> > > Flake8: "F821 undefined name 'unicode'"" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314860 (https://phabricator.wikimedia.org/T143573) (owner: 10Lokal Profil) [13:32:04] 06Labs, 10Tool-Labs: tools-exec-cyberbot in SHUTOFF state - https://phabricator.wikimedia.org/T147805#2705547 (10chasemp) p:05Triage>03Normal right, not putting this in the tools SAL is my mistake. I had verified with cyberpower678 awhile ago that this is all moved successfully to its own project. I shut... [13:49:04] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [13:51:20] RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [13:54:05] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [14:01:48] RECOVERY - Puppet run on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [14:06:02] RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:49] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Prepare and check production and labs-side filtering for olowiki - https://phabricator.wikimedia.org/T147302#2705670 (10chasemp) note: we should update this contact info https://wikitech.wikimedia.org/wiki/Add_a_wiki#Start [14:40:05] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:41:51] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:42:51] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:54:22] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [15:20:05] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [15:24:47] 10Tool-Labs-tools-Matthewrbowker's-tools: Typo at webpage - https://phabricator.wikimedia.org/T147758#2705871 (10Matthewrbowker) 05Open>03Resolved p:05Triage>03Low Fixed in https://github.com/Matthewrbowker/labs/commit/c9a68c4bb339d07cd47505a7ad8edc797859ae0f, will deploy live when I bump the version of... [15:31:36] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Prepare and check production and labs-side filtering for olowiki - https://phabricator.wikimedia.org/T147302#2705900 (10Marostegui) We have added olowiki filtering. We executed the following command ``` root@neodymium:/home/jynus/software/redactatron/scr... [15:37:50] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:53:41] yuvipanda: here we are :) [15:53:48] hello volans :) [15:55:04] volans: which project are you doing this in? [15:55:23] so, I'm trying to setup a puppetmaster and a host setup by this puppetmaster [15:55:28] in automation-framework [15:55:43] ok! [15:56:01] so let's look at https://wikitech.wikimedia.org/wiki/Standalone_puppetmaster [15:56:05] yep [15:56:40] should I setup hiera in advance? [15:58:05] volans: which hiera? [15:58:16] the puppetmaster global variable [15:58:40] volans: ah, nope, let's do it after. [15:58:59] ok launching clean jessie without any project's puppet role or hiera [15:59:12] +1 [15:59:45] now applying the role role::puppetmaster::standalone [16:00:34] \o/ ok [16:00:39] volans: what's thename of the instance? [16:00:54] puppetmaster [16:00:59] * volans brb phone [16:02:36] volans: instance names are still global unfortunately, so next time do prefix instance names with something more unique :) [16:03:05] unique per project? [16:03:34] volans: nope, unique across all of labs [16:03:51] so af-puppetmaster or something like that [16:03:58] ohhh, sorry didn't know, I can restart :D [16:04:23] volans: that'd be useful :D sorry to be a pain [16:05:22] no problem yuvipanda, maybe add a note in red in the launch instance popup if we own that code ;) [16:06:18] 06Labs, 10Labs-Infrastructure: Add a note to the launch instance window about instance names being global - https://phabricator.wikimedia.org/T147859#2705964 (10yuvipanda) [16:06:23] volans: ^ filed [16:06:33] great! [16:06:59] thanks [16:08:49] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:09:35] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:12:26] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Prepare and check production and labs-side filtering for olowiki - https://phabricator.wikimedia.org/T147302#2706000 (10jcrespo) a:05jcrespo>03chasemp Nominatively assigning to Chase, as the production part is done, but of course feel free to reassign... [16:13:42] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:14:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:15:06] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:16:06] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:17:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:18:38] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.80 ms [16:20:18] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:23:36] volans: any luck? [16:24:03] 06Labs, 10Beta-Cluster-Infrastructure: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2706048 (10Andrew) a:03Joe [16:24:43] (03CR) 10Lokal Profil: [C: 032] Lint SQL files using php-sqllint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/315226 (https://phabricator.wikimedia.org/T132641) (owner: 10Jean-Frédéric) [16:25:35] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [16:26:11] yuvitest: sorry side-tracked... getting there ;) [16:27:58] (03Merged) 10jenkins-bot: Lint SQL files using php-sqllint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/315226 (https://phabricator.wikimedia.org/T132641) (owner: 10Jean-Frédéric) [16:31:24] yuvitest: do you suggest that the puppetmaster is managed by the labs puppetmaser and the other instances that i will create will be managed by my puppetmaster? [16:32:51] 06Labs, 10Beta-Cluster-Infrastructure, 15User-Joe: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2706069 (10Joe) p:05Triage>03Normal [16:33:44] volans: yeah, I think that's a sane way to go for things like this [16:34:10] volans: or rather, let's start there, and then afterwards convert all instances in the project to use your puppetmaster (including the puppetmaster)? [16:35:35] ok [16:36:12] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms [16:36:56] yuvitest: I've got an error from /usr/bin/geoipupdate during the first puppet run suggested [16:37:07] although puppet has continued and finished the run [16:37:40] and in the second one I got a deprecation warning, I think is expected [16:37:40] Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations [16:39:26] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.63 ms [16:39:59] volans: can you paste the error? [16:40:13] volans: yes, the deprecation warning is expected [16:41:10] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [16:42:53] yuvitest: https://phabricator.wikimedia.org/P4195 [16:43:57] a second run was just fine (apart the deprecation warning) [16:44:54] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [16:48:08] volans: interesting, seems to be a intermittent network issue [16:48:15] volans: ok, so try setting up new instance and making it a client? :) [16:49:42] yep, adding few details in the wiki in the meanwhile [16:50:35] \o/ awesome [16:51:30] I think in Step 2 between sub-step 1 and 3 we should add a puppet run on the client, to get the hiera update [16:52:06] oh yeah, totally [16:52:25] I missed that, changking puppet.conf manually ;) [16:52:45] and puppet.conf.d/10-main.conf [16:54:13] and for the check of the fingerprint.. on the master running puppet cert sign doesn't show it ;) [16:55:58] just a puppet cert list before should do that , adding it [16:56:46] volans: ok! \o/ [17:03:22] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [17:03:34] volans: am gonna go afk in a bit to shower, anything you want from me before I go? [17:04:34] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [17:04:54] yuvitest: not for now, all looks good, updating the docs [17:08:48] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:09:36] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [17:10:04] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [17:11:06] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [17:12:13] yuvitest: all good, when you've time review the wiki page to see if what I've added sounds good [17:12:16] thanks for the help :) [17:12:20] RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [17:13:42] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:15:18] volans: awesome. looks great to me [17:15:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:16:17] volans: do you think you wanna try switching the puppetmaster itself to itself? [17:16:51] doing that will stop the sync from upstream puppet? [17:18:29] volans: nope [17:18:47] volans: there's a script (git-sync-upstream) that syncs from upstream puppet [17:18:51] ok that I can do that [17:23:31] afk now [17:27:06] I think I broke my puppetmaster... :) the rm -rf /var/lib/puppet/ssl is too destructive on the puppetmaster [17:27:15] hi there [17:27:21] better find /var/lib/puppet/ssl -name $FQDN.pem -delete [17:32:52] hi [17:33:31] i m feet artist and need wikipedia help [17:34:09] hello any body here [17:34:59] how can i create my profile here ? [17:39:22] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [17:42:53] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:04] yuvitest: ping me when you're back, no hurry, but I'm seeing some strange stuff [18:02:33] 10Tool-Labs-tools-Xtools: Adminstats is showing non-admin users too - https://phabricator.wikimedia.org/T145677#2706548 (10Matthewrbowker) Can someone provide a link that I can use to reproduce the issue? I'm going to work on fixing it this weekend. [18:06:47] !log tools.stewardbots Terminate stewardbot and sulwatcher jobs to restart them again [18:06:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [18:08:47] 10Tool-Labs-tools-Xtools: Adminstats is showing non-admin users too - https://phabricator.wikimedia.org/T145677#2706581 (10FriedhelmW) @Matthewrbowker: Link: https://de.wikipedia.org/w/index.php?title=Spezial%3ALogbuch&type=delete&user=Berihert&page=&year=&month=-1&tagfilter=&subtype=delete [18:12:15] 10Tool-Labs-tools-Xtools: Adminstats is showing non-admin users too - https://phabricator.wikimedia.org/T145677#2706616 (10Matthewrbowker) Cool, thank you much. That put me on the right path. https://tools.wmflabs.org/xtools/adminstats/?project=de.wikipedia.org&begin=2016-07-03&end= [18:12:39] !log tools.stewardbots Bots restarted [18:12:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [18:15:39] volans: uh oh. am back [18:19:22] yuvitest: so the change on the puppetmaster was a bit too destructive and I was starting from scratch, but when depoying a new instance with the same name of before it already get the puppetmaster::standalone applyied :( [18:19:36] and I cannot ssh [18:19:55] at least not immediately, now I can [18:20:18] but I was wondering if this is "expected" ,looks like a cache of roles applied was not cleanup when I destroyed the previous instance [18:20:42] andrewbogott: ^ [18:20:43] interesting [18:21:00] volans: in general, re-using same instance names is going to cause you a lot of problems and should be avoided [18:21:08] volans: I'm gonna file a bug [18:21:09] I didn't had the role in the project's puppet config, was only for that instance [18:21:15] ah good to know :D [18:21:16] yuvitest: yeah, those things are currently cleared when an instance is deleted. I couldn't decide if that was a bug or a feature [18:21:31] obviously everything else (project and prefix) persist [18:21:40] andrewbogott: I deleted the instance and recreated it [18:21:44] and it survived [18:22:01] ugh, typo [18:22:04] *aren't currently cleared [18:22:13] ah ok [18:22:18] why? [18:22:38] I'm pretty sure they should be [18:22:41] I'm gonna file a bug [18:23:44] or at least ask upon re-creation, if they should be re-applied or cleared [18:23:45] PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:23:52] having it cached is not bad per se [18:24:08] could be useful to "reimage" an instance for example [18:24:36] for the ssh access was just probably the slower puppet to install the puppetmaster before adding my key [18:25:03] 06Labs, 10Labs-Infrastructure: Horizon puppet roles not cleared when instance is deleted - https://phabricator.wikimedia.org/T147878#2706690 (10yuvipanda) [18:25:10] ^ [18:25:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [18:30:33] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [18:32:03] * volans stepping out for a bit ttyl, thanks for the support, I'll keep you posted and update the wiki for the puppetmaster registration to itself [18:36:42] volans: ok! [18:39:24] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [18:43:45] RECOVERY - Puppet run on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [18:46:35] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:01:04] 10Labs-project-Wikistats: Add olo.wikipedia to wikistats - https://phabricator.wikimedia.org/T147613#2706856 (10Dzahn) MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang) values ("olo", "Livvi-Karelian", "livvin kieli");Query OK, 1 row affected (0.01 sec) MariaDB [wikistats]> update wikipedias... [19:01:42] 10Labs-project-Wikistats: Add olo.wikipedia to wikistats - https://phabricator.wikimedia.org/T147613#2706861 (10Dzahn) 05Open>03Resolved [19:04:26] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2706900 (10leila) [19:04:37] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2706902 (10leila) a:05leila>03bd808 [19:05:12] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2706909 (10leila) @bd808 Please review the survey linked from the description, if you are happy, let's set a date and run it. I'll share the email we sent out last year shor... [19:11:35] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [19:16:27] 10Labs-project-Wikistats: wikistats (labs project): convert (all) mediawikis to use API instead of parsing old Special:Statistics - https://phabricator.wikimedia.org/T142766#2706984 (10Dzahn) 98.4% of all wikis in mediawikis table are fixed. The rest will either be upgraded at some point or stop working. I won'... [19:16:32] 10Labs-project-Wikistats: wikistats (labs project): convert (all) mediawikis to use API instead of parsing old Special:Statistics - https://phabricator.wikimedia.org/T142766#2706985 (10Dzahn) 05Open>03Resolved [19:22:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:37:19] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:42:27] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2707045 (10leila) [19:43:03] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Research-and-Data, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2689440 (10leila) [20:00:17] PROBLEM - Puppet run on tools-exec-1203 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:06:06] yuvitest: so I've updated it with https://wikitech.wikimedia.org/w/index.php?title=Standalone_puppetmaster&type=revision&diff=893483&oldid=892944 [20:06:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:06:36] although on the master itself I had to also do puppet cert clean FQDN [20:07:12] PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:08:44] volans: nice [20:08:49] volans: can you add that too? [20:09:26] PROBLEM - Puppet run on tools-exec-1210 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:09:44] yeah I was looking at the right way to saying it :D [20:10:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:12:28] PROBLEM - Puppet run on tools-exec-1214 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:12:40] PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:13:36] PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:13:52] PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:14:01] PROBLEM - Puppet run on tools-exec-1201 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:14:13] yuvitest: ok, this should cover all cases I got so far: https://wikitech.wikimedia.org/w/index.php?title=Standalone_puppetmaster&type=revision&diff=893508&oldid=893483 [20:14:36] hopefully will be useful for others [20:14:48] q! [20:15:15] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:16:05] volans: \o/ awesome [20:16:21] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:17:01] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:17:11] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:17:49] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:18:13] PROBLEM - Puppet run on tools-exec-1212 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:19:02] yuvitest: so now if I want to test puppet changes for my clients where should I made them? [20:19:36] directly in the puppetmaster local disk? [20:25:48] volans: usually, you make them and cherrypick on /var/lib/git/operations/puppet [20:25:57] you can also do it locally, yeah, but there's an autosync script that runs every 10min [20:26:06] it tries to stash and autorebase, and might cause issues [20:29:46] yuvitest: mmh, what do you mean with "you make them"? /var/lib/git/operations/puppet is a checkout of the production branch [20:30:18] RECOVERY - Puppet run on tools-exec-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [20:31:56] volans: ah, I mean - you make them locally, push to gerrit, then cherry pick [20:31:59] that' swhat I do [20:32:03] but yeah, you can do it locally too [20:32:12] but in general try to keep the tree clean [20:32:25] ok, thanks [20:37:49] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [20:46:48] PROBLEM - Puppet run on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:48:18] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:48:50] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:49:24] RECOVERY - Puppet run on tools-exec-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [20:50:12] PROBLEM - Puppet run on tools-grid-master is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:51:23] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:13] RECOVERY - Puppet run on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:27] RECOVERY - Puppet run on tools-exec-1214 is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:27] PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:52:37] RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [20:54:01] RECOVERY - Puppet run on tools-exec-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [20:55:15] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [20:55:34] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [20:56:22] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [20:57:00] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [20:58:14] RECOVERY - Puppet run on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [20:58:35] RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [20:58:51] RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [21:02:13] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [21:02:35] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [21:03:51] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [21:06:49] RECOVERY - Puppet run on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [21:08:51] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:09:49] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:30:13] RECOVERY - Puppet run on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [21:33:18] RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [21:43:51] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:54:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:02:23] RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [23:00:26] PROBLEM - Puppet run on bdsync-deb is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]