[00:50:13] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:55:14] RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [01:55:44] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-extensions-DynamicSidebar, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Hook OpenStackNovaUser::DynamicSidebarGetGroups has invalid call signature; ldap_count_entries() expects parameter 2 to... - https://phabricator.wikimedia.org/T119159#1819699 [02:00:13] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-extensions-DynamicSidebar, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Hook OpenStackNovaUser::DynamicSidebarGetGroups has invalid call signature; ldap_count_entries() expects parameter 2 to... - https://phabricator.wikimedia.org/T119159#1819714 [02:00:15] 6Labs, 10wikitech.wikimedia.org: Account creation success but shows error page - https://phabricator.wikimedia.org/T118916#1819715 (10Krenair) [02:24:49] 6Labs, 10wikitech.wikimedia.org: DB error when trying to create an account on wikitech - https://phabricator.wikimedia.org/T117553#1819736 (10Legoktm) This is caused by {ca2840b5a573fb948f6638f733e9cbfc068d9be9}, which checks for the string 'expects parameter' in a warning, and assumes it's an issue with the f... [02:25:11] 6Labs, 10wikitech.wikimedia.org: DB error when trying to create an account on wikitech - https://phabricator.wikimedia.org/T117553#1819739 (10Legoktm) p:5Triage>3Unbreak! [02:25:28] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-General-or-Unknown, 10wikitech.wikimedia.org: DB error when trying to create an account on wikitech - https://phabricator.wikimedia.org/T117553#1777456 (10Legoktm) [05:43:33] which box is the deployment puppetmaster these days? [05:43:38] PROBLEM - Puppet failure on tools-worker-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [05:43:45] deployment-puppetmaster seems to be very much out of date [05:46:18] ohh, nm; didn't scroll far enough to get past all the stale WIP ones [05:53:33] RECOVERY - Puppet failure on tools-worker-03 is OK: OK: Less than 1.00% above the threshold [0.0] [06:18:20] Niharika: anything happen? [06:44:42] PROBLEM - Puppet failure on tools-webgrid-generic-1405 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [06:59:09] PROBLEM - Puppet failure on tools-exec-1408 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [07:24:00] Earwig: Frances is at Wikisource conference, so she's unavailable mostly today and over the weekend. Shall I encourage her to send you a mail with her questions when she has time? [07:24:41] RECOVERY - Puppet failure on tools-webgrid-generic-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:10] RECOVERY - Puppet failure on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [09:26:51] Niharika: that'd be ideal [12:24:16] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [0.0] [12:59:09] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:19] 6Labs, 10Labs-Team-Backlog, 3ToolLabs-Goals-Q4: Labs NFSv4/idmapd mess - https://phabricator.wikimedia.org/T87870#1820424 (10coren) The package is built and now lives in the Wikimedia repo. Testing on labstore2001 shows that nfsd succesfully starts with the shim invoked and no visible issues as expected, bu... [16:39:05] 6Labs, 10Labs-Infrastructure: labcontrol2001: What is it? What is it for? - https://phabricator.wikimedia.org/T118591#1820788 (10Andrew) As best I can tell, labcontrol2001 serves as a secondary for public labs dns (ldap-backed pdns), and nothing else. We can kill it off if either: - We switch to all-design... [17:11:08] YuviPanda: do you use paramiko these days? and if not, what do you use instead? [17:25:43] andrewbogott: we can skip the meeting today I think, or briefly on irc, I have another changeset I would like to push out today hopefully post ops session [17:26:05] chasemp: ok, works for me [17:26:09] kk [17:26:20] Coren: ^ and ^^ [17:27:18] andrewbogott: https://gerrit.wikimedia.org/r/#/c/254426/ [17:27:58] chasemp: multi-tasking, but will look soon [17:28:06] sure thing [17:36:25] andrewbogott: ok, noted. [17:39:34] chasemp: does setting "%{::ipaddress_eth0}” in hiera really work? [17:39:43] yes [17:40:03] that's working now from yesterday [17:40:29] for our weird scope reasons $::site does not, I assume because where site is set confuses hiera [17:40:37] but $::facts work fine and are expected [17:40:48] or interpolation anyway [17:55:56] andrewbogott: re: the realm check point for wikitech...yeah not sure, I was just translating into the accepted idiom [17:56:08] unsure if we wikitech in labtest or not but I figured we can sort it out when it comes? [17:56:55] chasemp: I want to kill wikitech but in the meantime I have a bunch of coding I need to do there and I have not having a test box [17:57:06] so I’m thinking, yeah, probably will want something running OSM for the test cluster [17:57:08] unfortunately [17:57:09] I'm all for it just explaining why I didn't ditch the realm check yet [18:10:28] !log tools disabling puppet on the grid nodes listed at https://phabricator.wikimedia.org/P2337 so that the /tmp change in https://gerrit.wikimedia.org/r/#/c/252506/ do not apply early and break services [18:10:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:23:08] PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [18:24:11] !log tools Beginning draining web nodes; -lighttpd-1401 -lighttpd-1201 -generic-1401 [18:24:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:24:32] PROBLEM - Puppet failure on tools-exec-1218 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:24:56] PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:25:06] PROBLEM - Puppet failure on tools-exec-1203 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:25:07] PROBLEM - Puppet failure on tools-exec-1408 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:25:14] PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [18:25:18] PROBLEM - Puppet failure on tools-exec-1219 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:25:24] PROBLEM - Puppet failure on tools-exec-1405 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:25:30] Coren: ^ those are all on purpose, right? [18:25:39] Yes, as noted in the SAL [18:25:43] ah, so I see [18:25:43] ok :) [18:28:26] * Reedy thinks andrewbogott needs a bigger screen ;) [18:28:37] maybe [18:28:56] but also I’m in a meeting and was only noticing the email flood [18:29:05] heh [18:29:23] PROBLEM - Puppet failure on tools-exec-1205 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:30:09] PROBLEM - Puppet failure on tools-exec-1404 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:30:23] PROBLEM - Puppet failure on tools-exec-1204 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:30:57] PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:31:53] what we want is a bot trigger to enable-disable the notifications.. the shell script part is also there, but needs bot to authenticate users [18:34:01] PROBLEM - Puppet failure on tools-exec-1403 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:34:23] PROBLEM - Puppet failure on tools-exec-1213 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:34:23] PROBLEM - Puppet failure on tools-exec-1402 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:34:56] PROBLEM - Puppet failure on tools-exec-1221 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:35:33] PROBLEM - Puppet failure on tools-exec-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:38:06] PROBLEM - Puppet failure on tools-exec-1206 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:38:58] PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:39:06] PROBLEM - Puppet failure on tools-exec-1202 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [18:40:12] PROBLEM - Puppet failure on tools-exec-1201 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [18:40:18] PROBLEM - Puppet failure on tools-exec-1407 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:43:38] PROBLEM - Puppet failure on tools-exec-1214 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:44:10] PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:44:22] PROBLEM - Puppet failure on tools-exec-1410 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:44:32] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:44:40] PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:44:59] PROBLEM - Puppet failure on tools-exec-1217 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:45:01] PROBLEM - Puppet failure on tools-exec-1409 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:45:03] o.O [18:45:14] ah, ok [18:45:17] PROBLEM - Puppet failure on tools-exec-1212 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:45:37] I should read the log before I comment next time ;) [18:49:39] PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [18:50:15] PROBLEM - Puppet failure on tools-exec-1220 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [18:55:38] !log tools Putting -lighttpd-1401 -lighttpd-1201 -generic-1401 back in rotation, disabling the others. [18:55:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:00:16] RECOVERY - Puppet failure on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [19:04:26] RECOVERY - Puppet failure on tools-exec-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [19:04:29] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [19:04:29] RECOVERY - Puppet failure on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [19:04:58] RECOVERY - Puppet failure on tools-exec-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [19:05:00] RECOVERY - Puppet failure on tools-exec-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [19:05:10] RECOVERY - Puppet failure on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [19:05:10] RECOVERY - Puppet failure on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [19:05:16] RECOVERY - Puppet failure on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [19:05:24] RECOVERY - Puppet failure on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [19:09:10] Coren: when you do think would be a good tiem for me to push out some changes? are you done w/ your window? [19:10:25] RECOVERY - Puppet failure on tools-exec-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [19:10:53] RECOVERY - Puppet failure on tools-exec-1211 is OK: OK: Less than 1.00% above the threshold [0.0] [19:11:03] chasemp: Unless you changes are directly fiddling wit the tools manifest, it should be okay. [19:11:17] Chances of interference are pretty much zero. [19:14:00] RECOVERY - Puppet failure on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [19:14:12] RECOVERY - Puppet failure on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [19:14:22] RECOVERY - Puppet failure on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [19:14:23] RECOVERY - Puppet failure on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [19:14:56] RECOVERY - Puppet failure on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [19:15:02] !log tools done, and putting back in rotation: tools-webgrid-lighttpd-1402 tools-webgrid-lighttpd-1202 tools-webgrid-generic-1402 [19:15:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:15:20] RECOVERY - Puppet failure on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [19:15:34] RECOVERY - Puppet failure on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [19:18:09] RECOVERY - Puppet failure on tools-exec-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [19:18:49] RECOVERY - Puppet failure on tools-exec-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [19:19:11] RECOVERY - Puppet failure on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [19:20:19] RECOVERY - Puppet failure on tools-exec-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:40] RECOVERY - Puppet failure on tools-exec-1214 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:24] RECOVERY - Puppet failure on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:40] RECOVERY - Puppet failure on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:46] RECOVERY - Puppet failure on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:58] RECOVERY - Puppet failure on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:00] RECOVERY - Puppet failure on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:22] RECOVERY - Puppet failure on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:40] !log tools -lighttpd-1403 wants a restart. [19:25:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:27:46] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 383 bytes in 0.006 second response time [19:28:04] RECOVERY - Puppet failure on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [19:30:16] RECOVERY - Puppet failure on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [19:37:51] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 930207 bytes in 5.252 second response time [19:49:40] !log tools done, and putting back in rotation: tools-webgrid-lighttpd-1403 tools-webgrid-lighttpd-1203 tools-webgrid-generic-1403 [19:49:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:54:34] 6Labs, 10Labs-Infrastructure, 6operations: rename holmium to labservices1002 - https://phabricator.wikimedia.org/T106303#1821505 (10Andrew) a:3Andrew [20:28:34] Coren, chasemp, YuviPanda, whoever — I’ve temporarily broken new instance creation :( Working on it. [20:28:42] !log tools tools-webgrid-lighttpd-1404 tools-webgrid-lighttpd-1204 tools-webgrid-generic-1404 done and back in rotation. [20:28:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [20:29:32] andrewbogott: kk [20:41:46] !log tools tools-webgrid-lighttpd-1405 tools-webgrid-lighttpd-1205 tools-webgrid-generic-1405 done and back in rotation. [20:41:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [20:53:18] !log tools tools-webgrid-lighttpd-1406 tools-webgrid-lighttpd-1206 done and back in rotation. [20:53:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [20:58:27] !log tools tools-webgrid-lighttpd-1407 tools-webgrid-lighttpd-1207 done and back in rotation. [20:58:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:00:57] Coren, YuviPanda, chasemp, instance creation is working again, thanks to a sad revert [21:01:14] sad revert is sad [21:01:15] :( [21:01:26] Sad indeed. I take it you haven't figured out which part had the boo-boo. [21:01:49] no, although I have a ton of logs to read through now [21:01:59] nice friday evening reading material [21:02:35] I'm going to merge https://gerrit.wikimedia.org/r/#/c/254124/3 [21:02:43] puppet compiler spotted issues I fixed [21:03:24] YuviPanda: I also have a large change here [21:03:44] chasemp: conflicting change or? [21:04:03] this is just renames so shouldn't tecnically change anything [21:04:05] (file renames) [21:04:08] well could break the same shizzle [21:04:17] oh? link? [21:04:35] by that I mean it could break anything https://gerrit.wikimedia.org/r/#/c/254426/ [21:04:36] who knows [21:05:01] ah [21:05:10] yeah but those breakages should be distinctly differently visible [21:05:26] chasemp: mind if I merge and verify? should be less than 5min [21:05:35] sure thing [21:05:46] just saying, let's not overlap if we can help it [21:06:05] I think it's ok since my patch only touches things running *on top of* labs rather than labs infra itself [21:06:18] oh [21:06:21] i do touch glance and shit [21:06:24] and designate [21:06:27] you're right [21:06:32] but I'll be quick! [21:09:48] chasemp: everything seems ok [21:13:44] !log tools tools-webgrid-lighttpd-1408 tools-webgrid-lighttpd-1208 done and back in rotation. [21:13:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:20:56] 6Labs, 10wikitech.wikimedia.org, 7Security-General: Add password requirements for wikitech accounts - https://phabricator.wikimedia.org/T118751#1821827 (10Nemo_bis) It would also be nice if strong passwords worked at all: {T58114} [21:24:42] chasemp: in hindsight we did conflict, sorry about that - it was introduced in a latter patchset since I didn't realize that openstack was already in role/labs [21:25:09] it seems to have rebased appropriately I'm kinda checking now [21:25:22] !log tools tools-webgrid-lighttpd-1409 tools-webgrid-lighttpd-1209 done and back in rotation. [21:25:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:26:07] chasemp: yeah, I should've let you go first since my rebasing would've probably been easier [21:26:09] oh well [21:27:41] tbh I was kicking around going module/role already [21:27:52] it could mess up some scope stuff tho [21:27:58] idk yet [21:28:15] YuviPanda: does the role keyword work fine for things in module role? [21:28:24] I don't know why it wouldn't but I haven't tested it [21:28:32] chasemp: yeah there's a lot of other stuff in the role module already [21:28:36] that's using the role keyword [21:28:43] but are they also using hiera [21:28:47] yeah [21:28:50] k [21:28:50] lvs for example [21:30:24] !log tools tools-webgrid-lighttpd-1410 tools-webgrid-lighttpd-1210 done and back in rotation. [21:30:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:35:03] YuviPanda: could you hold on https://gerrit.wikimedia.org/r/#/c/254495/ until I through here [21:35:08] mainly as I've disabled puppet to rollout [21:36:02] (discussed on the other channel, I'm backing off from that now) [21:36:19] andrewbogott: you about? going to roll on https://gerrit.wikimedia.org/r/#/c/254426/ [21:36:21] who knows what happens [21:36:22] :) [21:36:43] * YuviPanda shall look at things that clearly have no conflict now [21:45:59] !log tools tools-webgrid-lighttpd-1411 tools-webgrid-lighttpd-1211 done and back in rotation. [21:46:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:48:02] chasemp: I’m here [21:50:08] chasemp: looks like instance creation isn’t working — possibly not my fault this time :) [21:50:41] hm I rolled out to labvirts now but no changes seen like at all [21:51:03] yeah it's been totally benign so far [21:52:02] something is pretty broken [21:52:04] don’t know what yet [21:52:22] YuviPanda: what did you do to test? did you spin up a vm or anything? [21:52:32] idk why it would appear this way [21:52:40] I just checked puppet and saw no diffs [21:53:03] yeah so far my change hasn't had an actual change yet [21:53:12] only gone on teh labvirt and labstore's tho [21:53:16] chasemp: actually, maybe that was a false alarm… stay tuned [21:54:27] I think I must have leaked something during my failed tests earlier. Building with a new different name seems to work [21:54:29] so, ignore me [21:54:41] chasemp: I see unrelated failures in puppet log about apt being broken because of signature mismatches [21:55:27] ok it's had apt broken for a while (labvirt1005) [21:55:29] not sure how to fix? [21:56:08] apt is broken on all the labvirts or just that one? [21:56:18] I just checked 1008 [21:56:22] broken there too [21:56:40] you mean this? Failed to fetch http://ubuntu.wikimedia.org/ubuntu/dists/trusty-updates/main/binary-amd64/Packages Hash Sum mismatch [21:57:22] yeah [21:58:25] YuviPanda: that’s also happening on labcontrol1001 and mw1160 [21:58:33] so, probably everywhere [22:00:18] that is apparently this https://bugs.launchpad.net/ubuntu/+source/apt/+bug/972077 [22:00:39] ah [22:00:43] * YuviPanda goes into a meeting [22:00:45] so not just us [22:00:53] andrewbogott: I dunno if that breaks puppet-run causing puppet to not run [22:00:59] shouldn't [22:01:59] !log tools tools-webgrid-lighttpd-1412 tools-webgrid-lighttpd-1413 tools-webgrid-lighttpd-1414 tools-webgrid-lighttpd-1415 done and back in rotation. [22:02:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [22:08:37] something is missing config wise on labservices1001 so I disabled puppet again, hacked back the missing params and am looking at why they weren't pulled correctly [22:12:26] I blame myself [22:18:45] maybe you could help me figure out why [22:18:51] --- /etc/powerdns/recursor.conf 2015-11-04 23:27:26.502174284 +0000 [22:18:51] +++ /tmp/puppet-file20151120-39773-j64pwm 2015-11-20 22:02:59.377923287 +0000 [22:18:52] @@ -11,7 +11,7 @@ [22:18:53] # [22:18:55] # allow-from=127.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fe80::/10 [22:18:57] -allow-from=127.0.0.0/8, ::1/128, 91.198.174.0/24, 208.80.152.0/22, 2620:0:860::/46, 198.35.26.0/23, 185.15.56.0/22, 2a02:ec80::/32, 10.0.0.0/8, 208.80.155.118 [22:18:59] +allow-from=127.0.0.0/8, ::1/128, 208.80.155.118 [22:19:01] happened [22:19:03] idk yet if scope change or what [22:25:49] chasemp: I can never remember in puppet diffs, is + the before or the after? [22:25:59] - is what was [22:26:02] + is what is being done now [22:26:03] ok [22:26:13] so my change...or maybe YuviPanda's? changs the allowd from [22:26:18] now it's defined at top level manifest [22:26:28] so maybe moving it into modules changes teh scope for lookup [22:26:30] that's a guess [22:26:31] https://gerrit.wikimedia.org/r/#/c/254581/ [22:26:35] so it’s ipresolve(hiera('labs_recursor'),4) that changed [22:26:37] probably [22:26:43] hmm [22:27:13] the role name didn't change so I am not sure why it would be affected [22:27:20] andrewbogott: that should still work this is pulling an array from manifests/network.pp [22:27:48] it's super wonky scope wise modules/role and manifests/role/ how they play together [22:27:52] I'm not sure here honestly [22:27:52] I must be on the wrong track [22:27:55] this isn’t from role::labsdnsrecursor? [22:29:01] oh yes [22:29:02] it hasn't moved [22:29:07] well bully for that idea [22:30:15] chasemp: you’re thinking the culprit is $network::constants::all_networks changing, right? That’s how it looks to me... [22:30:35] yes for soem reason when it hits labservices it's +allow-from=127.0.0.0/8, ::1/128, 208.80.155.118 [22:30:45] instead of allow-from=127.0.0.0/8, ::1/128, 91.198.174.0/24, 208.80.152.0/22, 2620:0:860::/46, 198.35.26.0/23, 185.15.56.0/22, 2a02:ec80::/32, 10.0.0.0/8, 208.80.155.118 [22:33:05] hi, can't make this work: http://tools-static.wmflabs.org/mpaatools [22:33:12] returns forbidden [22:33:36] could someone check if mpaatools/www/static is accessible? [22:34:13] chasemp: include ::network::constants isn’t in that file at all now is it? [22:35:20] oh, wait [22:35:26] I was looking at an old version where it wasn’t there [22:35:27] but it is now [22:35:32] so, somehow including that made things worse? [22:35:34] Coren: are you around and able to help out Mpaa-irc? [22:36:07] andrewbogott: yeah I think you are right [22:37:42] Mpaa-irc: your tool's homedir doesn't have rx bits set [22:38:35] YuviPanda, thanks, I thought r was enough :-( [22:38:42] now it is OK [22:38:59] Mpaa-irc: yeah, you need 'x' for directories, it means 'allow subdirectories / files to be accessed / listed' [22:39:10] the 'r' only lets you read the directry entry itself which isn't of much use [22:39:25] ah ... OK thanks [22:39:37] basically you have to "execute" the directory to list the contents [22:57:05] fun fact I think w/o x you can still open teh file you just can't list it [23:10:09] chasemp: It's the other way around. [23:10:22] chasemp: execute on directories is POSIX ACL "traverse" [23:10:46] yep you're right