[01:11:43] anyone around? [01:12:31] nope [01:19:27] :-/ [01:20:28] what's the issue Magog_the_Ogre? [01:21:27] on the previous servers, it was not possible to hit http://tools.wmflabs.org directly [01:21:37] one had to use tools-webproxy [01:21:40] now it seems to be reversed [01:21:58] should I permanently update my code or is this something someone needs to fix? [01:23:24] I will send an email to labs-l [02:20:50] 10Tool-Labs-tools-Other: Non technical: "Database reports" - status query - https://phabricator.wikimedia.org/T92353#1111890 (10MZMcBride) Database reports are in a state of disrepair. :-( [03:40:20] * ^d stabs nfs [03:54:05] hi ^d [03:54:07] why stabby [03:54:24] <^d> can't mount /home or /data/project bs again [03:56:47] ^d: ah. reboot [03:56:49] to fix [03:56:55] <^d> I did like 4 times each [03:58:05] oh [03:58:09] ^d: which instance is this? [03:58:16] I wonder if we should build staging to not rely on NFS [03:58:23] <^d> I wouldn't mind [03:58:26] <^d> staging-mc[1-3] [03:58:28] I’m getting rid of the only point I can think of where it is used [03:58:32] <^d> (are precise now) [03:58:34] which is userkeys [03:58:40] ^d: why do we need NFS on those anyway? [03:58:44] <^d> We don't. [03:59:16] ah, right [04:01:49] labstore.svc.eqiad.wmnet:/project/staging/home on /home type nfs (rw,noatime,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,nofsc) [04:01:53] ^d: i see that on mount [04:01:58] and I do see NFS [04:03:22] <^d> mc1 and 3 look ok now [04:03:28] wheeeee [04:03:33] we need to fix this shit, of course. [04:03:33] <^d> mc2 fails on puppet run still [04:03:51] can you give me its IP [04:03:54] I can check the server [04:04:07] nvm got it [04:04:07] <^d> sec [04:04:18] there’s an export [04:04:25] so server is fine [04:04:36] <^d> Probably busted ldap entry like before :\ [04:04:56] if there was a busted LDAP entry, the IP wouldn’t have made it to the server export... [04:05:18] <^d> Ah nvm then [04:06:36] ^d: reboot fixed it \o/ [04:07:13] <^d> If at first you don't succeed, try (and try ){3,} again [04:12:12] <^d> fuck but mc2 and 3 are trusty because chad didn't pay attention [04:12:20] * ^d shouldn't drink and labs [04:12:38] <^d> time to start over again! [04:56:28] (03CR) 10Yuvipanda: [C: 032 V: 032] Use ssh::userkey for root as well [labs/private] - 10https://gerrit.wikimedia.org/r/196019 (owner: 10Yuvipanda) [04:57:29] ZOMG SELF MERGE [04:57:51] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [04:58:25] Deskana: :D [04:58:43] Deskana: :D [04:59:01] Deskana: +2 in ops means ‘I am going to babysit this’, so most things are self merged [04:59:57] YuviPanda: Dmitry once tricked me into self-merging. [05:00:02] YuviPanda: https://gerrit.wikimedia.org/r/#/c/186862/ [05:00:03] heh [05:00:16] PROBLEM - Puppet failure on tools-webgrid-generic-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:00:31] Deskana: hah :) [05:01:43] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [05:02:05] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:02:51] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [05:02:53] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [05:04:03] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [0.0] [05:05:45] PROBLEM - Puppet failure on tools-redis-slave is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [05:06:01] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:06:57] PROBLEM - Puppet failure on tools-webproxy-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:07:21] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [05:07:50] PROBLEM - Puppet failure on tools-webgrid-07 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [05:07:58] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:08:22] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [05:09:17] Deskana: I remember you saying you wanted to write a simple tool sometime :) [05:09:18] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [05:10:34] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [0.0] [05:10:45] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [05:25:18] RECOVERY - Puppet failure on tools-webgrid-generic-02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:25:28] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [05:27:00] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [05:27:58] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [05:27:58] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [05:29:06] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [05:31:26] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [05:31:44] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:31:56] RECOVERY - Puppet failure on tools-webproxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:32:25] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [05:32:51] RECOVERY - Puppet failure on tools-webgrid-07 is OK: OK: Less than 1.00% above the threshold [0.0] [05:32:53] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [05:32:53] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [05:33:25] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [05:34:23] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:35:53] RECOVERY - Puppet failure on tools-redis-slave is OK: OK: Less than 1.00% above the threshold [0.0] [05:35:53] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [05:36:03] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [05:56:28] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [07:47:05] Tool is down: https://tools.wmflabs.org/kmlexport/?project=de&article=Nationalparks_in_Deutschland&redir=bing [07:49:53] ewrgerherh: kmlexport has been dying a lot... [07:49:58] * YuviPanda starts it back up [07:50:41] ewrgerherh: I’ve started it again [07:50:48] Thanks [07:51:32] isn't there a way to restart automatically. The tool is heavily used in German Wikipedia articles [07:56:03] ewrgerherh: yeah, I set that up too [07:56:50] thanks a lot [07:57:32] ewrgerherh: yw! [08:47:47] 6Labs, 10Tool-Labs: Fix Labs' PAM config mess - https://phabricator.wikimedia.org/T85910#1112308 (10yuvipanda) Any takers for getting this done? :) [10:52:58] hi guys! I was wondering if I could create a database in tool labs with my user. [12:10:00] webservcie broken? [12:10:03] can't restart... [12:10:10] no ws job in grid :/ [12:23:21] uh [12:23:23] which one? [12:23:25] he’s gone [12:44:36] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1112724 (10scfc) 3NEW [12:45:15] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1112732 (10yuvipanda) I wonder if that's an apt clash with the puppet-run apt run. [12:45:21] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1112733 (10scfc) ``` From: root@tools.wmflabs.org (Cron Daemon) Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) To: root@tools.wmflabs.org Date... [12:45:49] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1112734 (10scfc) ``` From: root@tools.wmflabs.org (Cron Daemon) Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) To: root@tools.wmflabs.org Date:... [13:00:02] marcmiquel: You can, with your service group user. Its name must start with your (database) username. [13:00:58] aha. is there any limit with the data I can store in it? i want to put id,page_titles, basicly for many languages. [13:01:49] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1112758 (10scfc) >>! In T92491#1112732, @yuvipanda wrote: > I wonder if that's an apt clash with the puppet-run apt run. That's what I thought of as well (but haven't confirmed), but my assumption is that `... [13:02:35] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1112760 (10yuvipanda) True, it seems less likely after seeing the actual error messages. [13:06:17] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1112781 (10coren) Apt tools indeed use proper locking, but do so to ensure exclusive runs not concurrency. But Yuvi is correct that those error messages are not it. [13:55:00] 10Tool-Labs: Install ipython3 on tools - https://phabricator.wikimedia.org/T92495#1112954 (10Sitic) 3NEW [13:57:47] 10Tool-Labs, 5Patch-For-Review: Fix and clean up generation of ssh_known_keys - https://phabricator.wikimedia.org/T92379#1112964 (10scfc) I learnt: - If a host isn't listed verbatim in `ssh_known_hosts`, it isn't recognized by `ssh`. Inter alia this means no "[]" around host names. - If a host name is listed... [13:59:08] 10Tool-Labs: Install ipython3 on tools - https://phabricator.wikimedia.org/T92495#1112967 (10yuvipanda) I'm going to highly reccomend a virtualenv, and asking for any system packages that that would need. [14:03:04] 10Tool-Labs: Install ipython3 on tools - https://phabricator.wikimedia.org/T92495#1112975 (10Sitic) Ah thanks, I had completely forgotten that it's also in pip, I'm too used to install such things with apt-get. [14:07:05] Coren, getting flooded again. [14:07:21] Cyberpower678: Don't touch anything so I can catch it in the act. [14:07:57] Actually. This happeed 6 hours ago, but the messages are coming in just now. [14:07:59] Sorry. [14:08:17] Ah, no worries. I was hoping I could debug it live. [14:08:28] 10Tool-Labs: Remove redundant ssh host keys from users' known_hosts - https://phabricator.wikimedia.org/T92497#1112987 (10scfc) 3NEW a:3scfc [14:08:32] I don't expect you to be able to travel in time. :-) [14:08:34] But it's the usual, restarting, failed, restarting, failed, resarting, failed, throttling [14:08:51] And throttling repeated over and over. [14:10:23] Coren, BTW how do I opt out of bigbro? [14:10:55] 10Tool-Labs: Remove redundant ssh host keys from users' known_hosts - https://phabricator.wikimedia.org/T92497#1112998 (10scfc) [14:10:59] In theory, just having blanked or deleted the .bigbrotherrc should have done the trick. I'm trying to figure out why that didn't work. [14:11:31] Maybe you can have a look to make sure I'm not going insane, or am just blind? [14:11:34] :p [14:11:54] It works when I test it, so there is something in the code that apparently fails for some users and not others. [14:12:01] so, any thoughts on the crazy idea of modifying a restricted part of Puppet configuration from MediaWiki? :P [14:12:16] I tried creating a setuid executable, but I still get permission denied [14:12:21] (I'm just editing Hiera) [14:13:18] Cyberpower678: In the meantime, I've removed your tools by hand from the scoreboard so at least you won't get the email anymore. [14:14:25] :-) [14:48:38] Coren: any concerns with this? https://gerrit.wikimedia.org/r/#/c/196225/ [14:49:22] andrewbogott: Shouldn't be too bad. It's a bit on the expensive side, but not ridiculously so. [14:49:41] 10Tool-Labs: Install ipython3 on tools - https://phabricator.wikimedia.org/T92495#1113078 (10yuvipanda) 5Open>3declined a:3yuvipanda :D cool [14:49:41] andrewbogott: I wish we had a reliable way to have that pushed instead of pulled. [14:50:05] Coren: it’s not impossible, we can add a hook to instance creation. But it would have to invoke something on labstore [14:50:12] …which means a rest service, etc. etc. [14:50:34] 10Tool-Labs, 5Patch-For-Review: Fix and clean up generation of ssh_known_keys - https://phabricator.wikimedia.org/T92379#1113083 (10scfc) [14:50:37] 10Tool-Labs: Remove redundant ssh host keys from users' known_hosts - https://phabricator.wikimedia.org/T92497#1113082 (10scfc) [14:50:40] Also Ryan was really against it - not unreasonably - for isolation reasons. [14:58:36] (03PS1) 10Petrb: Inserted a link to git repository [labs/toollabs] - 10https://gerrit.wikimedia.org/r/196227 [15:01:52] ^d, a sample size of one suggests that that tiny patch should resolve most cases of having to reboot an instance to get NFS mounting. Please let me know if you encounter that bug again. [15:02:08] <^d> okie dokie [15:02:09] <^d> thx [15:02:29] Sorry that I let that issue linger for so long :( [15:10:33] The only reliable method would be to have that not poll. It's probably worth considering seriously. [15:10:55] (fwiw, anything that polls ldap to make a change is suspect) [15:11:42] Well… this is dumb, but if we have an explicit sleep in the firstboot script that is > than the polling time we should be guaranteed success. [15:11:56] which, the jessie firstboot does that already :) [15:28:37] 10Tool-Labs: PHP abort()s during execution of a script on an exec node - https://phabricator.wikimedia.org/T78010#1113190 (10coren) Is this still an issue? [15:36:33] 10Tool-Labs: PHP abort()s during execution of a script on an exec node - https://phabricator.wikimedia.org/T78010#1113202 (10Magnus) 5Open>3Resolved a:3Magnus Works normally AFAICT. Originally opened it because of potential "black hole" server. [15:38:25] 10Tool-Labs: Provide source/repository link on https://tools.wmflabs.org - https://phabricator.wikimedia.org/T86431#1113211 (10scfc) a:3Petrb [15:38:37] (03PS2) 10Tim Landscheidt: Insert link to repository [labs/toollabs] - 10https://gerrit.wikimedia.org/r/196227 (https://phabricator.wikimedia.org/T86431) (owner: 10Petrb) [15:43:26] (03CR) 10Tim Landscheidt: [C: 032] Insert link to repository [labs/toollabs] - 10https://gerrit.wikimedia.org/r/196227 (https://phabricator.wikimedia.org/T86431) (owner: 10Petrb) [15:43:47] (03CR) 10Tim Landscheidt: [V: 032] Insert link to repository [labs/toollabs] - 10https://gerrit.wikimedia.org/r/196227 (https://phabricator.wikimedia.org/T86431) (owner: 10Petrb) [15:45:00] 10Tool-Labs: Tools crontab replacement must check whether run as root - https://phabricator.wikimedia.org/T87527#1113247 (10coren) [15:46:36] 10Tool-Labs: Tools crontab replacement must check whether run as root - https://phabricator.wikimedia.org/T87527#1113255 (10coren) 5Open>3Resolved The applied patch goes full-paranoia and forcibly invokes /usr/bin/crontab for non-managed users/ It's mostly unnecessary because puppet (at least) invokes cront... [15:46:37] 10Tool-Labs, 5Patch-For-Review: tools-trusty uses local crontab instead of remote (tools-cron?) crontab - https://phabricator.wikimedia.org/T86445#1113257 (10coren) [15:47:45] 10Tool-Labs, 5Patch-For-Review: tools-trusty uses local crontab instead of remote (tools-cron?) crontab - https://phabricator.wikimedia.org/T86445#1113268 (10coren) 5Open>3Resolved All bastions now use /usr/local/bin/crontab rather than the (now obsolete) xcrontab and symlinks. [15:48:11] 10Tool-Labs, 5Patch-For-Review: Provide source/repository link on https://tools.wmflabs.org - https://phabricator.wikimedia.org/T86431#1113271 (10scfc) 5Open>3Resolved [16:01:09] 6Labs, 10hardware-requests, 6operations: Hardware for Designate - https://phabricator.wikimedia.org/T91277#1113310 (10RobH) server holmium is now allocated for this task. I'll create the linked tickets for its setup. [16:04:38] 6Labs, 10Wikimania-Hackathon-2015: iPython for Labs: call for an interactive coding plattform - https://phabricator.wikimedia.org/T92506#1113324 (10daniel) 3NEW [16:06:15] 6Labs, 10Wikimania-Hackathon-2015: iPython for Labs: call for an interactive coding plattform - https://phabricator.wikimedia.org/T92506#1113335 (10yuvipanda) YESSSSSSSSSSSSSSSSS [16:06:32] 6Labs, 10Wikimania-Hackathon-2015: iPython for Labs: call for an interactive coding plattform - https://phabricator.wikimedia.org/T92506#1113336 (10yuvipanda) I've a preliminary test running on jupyter.wmflabs.org, with login via SUL. Needs more work.. [16:06:40] 6Labs, 10Wikimedia-Hackathon-2015: iPython for Labs: call for an interactive coding plattform - https://phabricator.wikimedia.org/T92506#1113337 (10daniel) [16:08:28] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1113340 (10RobH) 3NEW a:3RobH [16:08:49] 6Labs, 10hardware-requests, 6operations: Hardware for Designate - https://phabricator.wikimedia.org/T91277#1113347 (10RobH) [16:08:50] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1113340 (10RobH) [16:09:00] 6Labs, 10hardware-requests, 6operations: Hardware for Designate - https://phabricator.wikimedia.org/T91277#1078794 (10RobH) [16:09:01] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1113340 (10RobH) [16:09:13] 6Labs: Investigate replacing our custom DNS code with Designate - https://phabricator.wikimedia.org/T87280#1113357 (10RobH) [16:09:14] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1113340 (10RobH) [16:09:15] 6Labs, 10hardware-requests, 6operations: Hardware for Designate - https://phabricator.wikimedia.org/T91277#1113354 (10RobH) 5Open>3Resolved a:3RobH [16:09:36] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1113340 (10RobH) [16:09:37] 6Labs: Investigate replacing our custom DNS code with Designate - https://phabricator.wikimedia.org/T87280#987813 (10RobH) [16:14:06] 10Tool-Labs: Tools crontab replacement must check whether run as root - https://phabricator.wikimedia.org/T87527#1113378 (10scfc) Eh, no, Puppet uses `crontab` without an explicit path, which is the whole point of this task … [16:50:08] andrewbogott: ah, thanks for the email :) [16:50:20] andrewbogott: I forgot about that. [16:50:26] YuviPanda: it was kind of half-assed, I didn’t bother to track down what change actually caused it [16:50:39] andrewbogott: yeah, that was me. [16:51:08] YuviPanda: was there a corresponding change in the really-private repo? [16:51:27] andrewbogott: I went off trying to make tin and deployment-prep close enough, and fell into this huge rabbit hole and ended up with several yaks shaved, one of which is that user keys installed by puppet are not on NFS anymore now [16:51:28] andrewbogott: yeah [16:51:42] that’s why anyone not using a self hosted puppetmaster isn’t complaining :D [17:17:35] hi, what's the process to run update.php on betalabs? [17:18:51] it runs on a cronjob [17:18:56] people in -releng would know [17:21:24] legoktm: it runs in a jenkins job... [17:21:31] a job that I broke a few hours ago and have no idea how to fix >_> [17:23:28] YuviPanda: which job? [17:24:04] legoktm: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/44937/console and others [17:24:10] I know exactly why and how it is broken... [17:24:29] oh [17:24:30] no idea [17:38:36] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1113740 (10RobH) [17:42:56] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1113761 (10RobH) [18:06:17] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1113867 (10RobH) [18:06:32] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1113340 (10RobH) os install in progress, all previous steps complete [18:14:17] 10Tool-Labs, 5Patch-For-Review: meta_p.wiki's column size is 1 for all wikis - https://phabricator.wikimedia.org/T90084#1113899 (10scfc) 5Open>3Resolved ``` scfc@tools-login:~$ for host in s{1..3}.labsdb; do mysql --defaults-file=replica.my.cnf -h"$host" -e 'SELECT size, COUNT(*) FROM meta_p.wiki GROUP BY... [18:15:24] 10Tool-Labs, 5Patch-For-Review: meta_p.wiki's column size is 1 for all wikis - https://phabricator.wikimedia.org/T90084#1113901 (10scfc) 5Resolved>3Open Ooops, I missed that the change was still open. [18:24:55] anyone know why I have an erro 503 on http://tools.wmflabs.org/xtools/blame/?project=fr.wikipedia.org&article=Dieudonn%C3%A9&text=http%3A%2F%2Ffr.altermedia.info%2Fgeneral%2Fdieudonne-en-iran-pour-la-conference-sur-la-palestine_8906.htmll%0D%0A [18:38:16] YuviPanda: this is probably you: https://phabricator.wikimedia.org/P391 [18:38:39] We can fix it by building a fresh image, but let’s not do that until I figure out a proper way to do https://gerrit.wikimedia.org/r/#/c/196233/ [18:38:57] andrewbogott: which host is that? [18:39:06] andrewbogott: ooooh, is the ubuntu key in the fresh image? [18:39:21] YuviPanda: dunno [18:39:31] andrewbogott: which instance are you seeing this in? [18:39:41] I ran a salt command that should’ve gotten rid of this in most [18:39:50] YuviPanda: right, but that doesn’t work on fresh instances [18:39:52] which this is [18:39:56] right [18:40:06] so I guess the ubuntu key is in the fresh image... [18:40:13] * YuviPanda goes to see [18:40:20] the instance that’s producing it is doing other stuff so you can’t easily mess with puppet there. But if you make another one it should happen again [18:40:52] right [18:41:18] andrewbogott: I’m going to file a bug then doze off... [18:41:31] YuviPanda: ok — it’s clearly not urgent [18:42:00] 6Labs: Get rid of default ubuntu key on labs VMs - https://phabricator.wikimedia.org/T92538#1114053 (10yuvipanda) 3NEW [18:42:12] andrewbogott: cool. [19:03:19] 10Tool-Labs, 5Patch-For-Review: meta_p.wiki's column size is 1 for all wikis - https://phabricator.wikimedia.org/T90084#1114183 (10coren) 5Open>3Resolved Yes but, as you have noticed, tested as working with a cherry-pick. :-) [19:03:27] 10Tool-Labs: meta_p.wiki's column size is 1 for all wikis - https://phabricator.wikimedia.org/T90084#1114185 (10coren) [19:23:02] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1114295 (10RobH) [19:23:24] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1114296 (10RobH) p:5High>3Normal [19:24:18] 6Labs, 10hardware-requests, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1114303 (10RobH) a:5RobH>3Andrew @Andrew, System OS installed and awaiting service implementation. puppet/salt keys have NOT been accepted at this time. [19:24:25] 6Labs, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1114306 (10RobH) [20:31:13] 6Labs, 10Wikimedia-Labs-wikitech-interface: Use a Puppet ENC to define which classes are included in which nodes (in Labs) - https://phabricator.wikimedia.org/T85279#1114638 (10yuvipanda) Alright, so how about we make this a small webservice with pluggable backends? Early backend would be file based (yay for b... [20:42:43] 6Labs, 10Wikimedia-Labs-wikitech-interface: Use a Puppet ENC to define which classes are included in which nodes (in Labs) - https://phabricator.wikimedia.org/T85279#1114655 (10yuvipanda) Or we could go completely low-fi, and just have it be a simple script that's called by the ENC. This is the CGI model, and... [20:46:29] 6Labs, 10Wikimedia-Labs-wikitech-interface: Use a Puppet ENC to define which classes are included in which nodes (in Labs) - https://phabricator.wikimedia.org/T85279#1114688 (10yuvipanda) And no, it is only *like* CGI. Puppet ENC works by startying a script with a param (the name of the node) and expecting bac... [20:50:02] (03PS1) 10Legoktm: jenkins job validation, do not submit [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/196417 [20:51:18] (03PS1) 10Legoktm: jenkins job validation, do not submit [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196418 [20:52:38] (03PS2) 10Legoktm: jenkins job validation, do not submit [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196418 [20:52:53] (03CR) 10jenkins-bot: [V: 04-1] jenkins job validation, do not submit [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196418 (owner: 10Legoktm) [20:53:03] (03Abandoned) 10Legoktm: jenkins job validation, do not submit [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/196417 (owner: 10Legoktm) [20:53:15] (03Abandoned) 10Legoktm: jenkins job validation, do not submit [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196418 (owner: 10Legoktm) [21:03:30] 10Tool-Labs, 10Living-Style-Guide, 6Mobile-Web: npm version on tools-login.wmflabs.org is incompatible with MobileFrontend package.json used by the KSS styleguide - https://phabricator.wikimedia.org/T89093#1114776 (10Jdlrobson) [21:27:55] someone can grant me access to the bastion hosts? [21:47:20] (03PS1) 10Legoktm: tox: Rename channels env to standard py34 [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196447 [21:47:26] (03CR) 10jenkins-bot: [V: 04-1] tox: Rename channels env to standard py34 [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196447 (owner: 10Legoktm) [21:50:37] (03PS1) 10Legoktm: tox: Rename tests env to generic py27 [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/196449 [21:50:46] (03CR) 10jenkins-bot: [V: 04-1] tox: Rename tests env to generic py27 [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/196449 (owner: 10Legoktm) [22:19:02] (03CR) 10Legoktm: "recheck" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196447 (owner: 10Legoktm) [22:19:06] (03CR) 10Legoktm: "recheck" [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/196449 (owner: 10Legoktm) [22:19:31] (03CR) 10Legoktm: [C: 032] tox: Rename tests env to generic py27 [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/196449 (owner: 10Legoktm) [22:19:37] (03Merged) 10jenkins-bot: tox: Rename tests env to generic py27 [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/196449 (owner: 10Legoktm) [22:19:39] (03CR) 10Legoktm: [C: 032] tox: Rename channels env to standard py34 [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196447 (owner: 10Legoktm) [22:19:52] (03Merged) 10jenkins-bot: tox: Rename channels env to standard py34 [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196447 (owner: 10Legoktm) [23:39:36] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, 6operations: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1115326 (10Dzahn) related to the DNS work on labs i would suspect. https://phabricator.wi... [23:40:07] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, 6operations: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1115328 (10Dzahn) @Coren ^ is that possible ? [23:40:59] 6Labs, 10Wikimedia-Labs-Infrastructure, 5Patch-For-Review: Internal DNS look-ups fail every once in a while - https://phabricator.wikimedia.org/T72076#1115332 (10Dzahn) could this have caused T92351 ?