[00:08:58] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, 6operations: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118444 (10Krinkle) ``` [16:03 CET] krinkle at KrinkleMac in ~ $ host saucelabs.com saucel... [00:16:11] YuviPanda: https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#Set_up_project-wide_puppetmaster anything about `sudo puppet agent -t` vs /usr/local/sbin/puppet-run ? [00:21:45] 10Wikimedia-Labs-Other, 6operations, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1118454 (10coren) [00:21:46] 10Tool-Labs-tools-Other, 7Tracking: merl tools (tracking) - https://phabricator.wikimedia.org/T69556#1118455 (10coren) [00:22:49] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, 6operations: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118458 (10Dzahn) I found the root cause to be this option in /etc/resolv.conf ``` option... [00:28:59] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118464 (10coren) ndots:2 is necessary for something else, the actual bug is that the dnsma... [00:38:21] 6Labs: Ability to handle mail (mailman project) - https://phabricator.wikimedia.org/T91576#1118471 (10coren) 5Open>3Resolved a:3coren MX record added to LDAP [00:44:34] 10Wikimedia-Labs-General: tables views missing for idwiki_p (source s3) on s3.labsdb - https://phabricator.wikimedia.org/T75975#1118488 (10Krenair) [00:44:35] 10Wikimedia-Labs-Other, 6operations, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1118490 (10Krenair) [00:44:36] 10Tool-Labs-tools-Other, 7Tracking: merl tools (tracking) - https://phabricator.wikimedia.org/T69556#1118489 (10Krenair) [00:44:47] 10Wikimedia-Labs-Other, 6operations, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#530760 (10Krenair) [00:44:47] 10Wikimedia-Labs-General: tables views missing for idwiki_p (source s3) on s3.labsdb - https://phabricator.wikimedia.org/T75975#787463 (10Krenair) [00:44:54] 10Wikimedia-Labs-General: tables views missing for idwiki_p (source s3) on s3.labsdb - https://phabricator.wikimedia.org/T75975#787463 (10Krenair) [00:45:38] 10Tool-Labs: Provide namespace IDs and names in the databases similar to toolserver.namespace - https://phabricator.wikimedia.org/T50625#1118495 (10coren) As I noted in T50625#488761 the correct view to use is named toolserverdb_p and it lives on tools.labsdb As far as I know, this is the authoritative view. [00:47:38] 10Wikimedia-Labs-Infrastructure: missing database entries at categorylinks table on dewiki db - https://phabricator.wikimedia.org/T72711#1118497 (10coren) Is this still an issue after the changes effected by @Springle? [00:49:38] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118500 (10scfc) The error doesn't seem to lie with dnsmasq. On `tools-login`, the look-up... [00:49:57] Coren, I think https://phabricator.wikimedia.org/T75975 was also resolved at some point [00:50:46] Krenair: Yeah, I'm working through the backlog to find those. [00:50:59] 10Tool-Labs-tools-Other, 7Tracking: merl tools (tracking) - https://phabricator.wikimedia.org/T69556#1118514 (10coren) [00:51:02] 10Wikimedia-Labs-Other, 6operations, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1118515 (10coren) [00:51:02] 10Wikimedia-Labs-General: tables views missing for idwiki_p (source s3) on s3.labsdb - https://phabricator.wikimedia.org/T75975#1118510 (10coren) 5Open>3Resolved a:3coren Fixed after unification of the databases. [00:52:55] Coren, is an MX record all it takes for a labs instance to technically be able to handle mail? [00:55:56] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118537 (10coren) No, it's just that the Precise libresolv seems to be a little more forgiv... [01:03:15] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118547 (10coren) To wit: ``` marc@tools-trusty:~$ host notexist Host notexist.eqiad.wmflab... [01:03:43] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118555 (10Krinkle) I suspect this error got introduced when I switched over the CI pool fr... [01:03:51] Krenair: Well, as long as it has a public IP and actually runs an MTA, yeah. [01:04:14] But Legal doesn't /like/ random projects being able to receive mail. [01:04:30] Coren, xTools is peaceful. Have you found the bug yet, btw? [01:05:23] Cyberpower678: No, and my attempts to reproduce it failed. Apparently, it likes xtools so much it won't let go of it without a restart. :-) [01:05:28] Coren, right, so you only set it up where necessary for trusted users? fair enough [01:05:58] Krenair: Basically. [01:07:09] Coren, It and the global community of Wikimedia. :p We've implemented web watcher which does more than big brother, and since bigbro was dragged out by force, web watcher hadn't had the need to kick in yet. :p [01:07:09] * Coren farts in dnsmasq's general direction. [01:07:27] Speaking of farting... [01:07:31] * Cyberpower678 ate onions [01:09:19] Coren: Is there a short term fix? Since we don't need labs db in CI afaik, I can just cherry-pick it for now until a solution arises. [01:10:01] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118565 (10Krinkle) [01:10:42] Krinkle: I'm trying to figure out if there is a way to restore the more forgiving Precise behaviour on Trusty to wallpaper over the issue. [01:13:15] okay :) [01:13:19] Krinkle: Using www.whatever.com would also work, if that is feasible [01:13:36] Coren: It isn't in this case since it's behind many layers of abstraction in user code. [01:13:44] e.g. composer and npm packages fetched at run time [01:14:00] Took me a week to even get to this piont [01:14:07] (it wasn't a high priority at the time) [01:14:12] * Coren understands. [01:19:28] I can't seem to find the patch or changeset that made libresolv change behaviour. :-( [01:21:27] Why in the name of Baal would dnsmasq even return SERVFAIL instead of NXDOMAIN is beyond me. [01:28:07] Krinkle: I can't find a solution for this in time to be useful. Cherry-pick the change for now; I'll investigate further on Monday. I'm braindead past 9pm on Friday. :-) [01:29:21] Coren: Did you say... Ba'al? https://www.youtube.com/watch?v=qO5_Te12Iys [01:31:23] I use random deity names when I curse to avoid showing favoritsm. :-) [01:35:34] That's it? Just... Ba'al? As in... bocce? [01:36:27] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118579 (10scfc) @Coren: But there are you querying the Labs server, and (I think) dnsmasq... [01:39:24] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118580 (10scfc) And: ``` scfc@tools-login:~$ dig @10.68.16.1 tools-login.eqiad.wmflabs ;... [01:41:03] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118589 (10scfc) (Or a host name that does not exist.) [01:41:38] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118590 (10coren) >>! In T92351#1118579, @scfc wrote: > So (from a distance) it appears as... [02:00:17] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118610 (10scfc) http://www.linuxquestions.org/questions/linux-networking-3/powerdns-servfa... [02:02:14] Coren: Do you know why puppet is so bad at actually erroring? Of the last 100 puppet errors I encountered over the past month, only 1 actually resulted in a failed puppet run. Most are just stepped over and ignored (other than syslog entries) [02:02:29] Is there some flag to toggle to make it actually fail hard and refuse to continue? [02:02:46] I feel like we're always running after the errors when it could've just failed properly and result in pings / signals [02:07:05] 10Wikimedia-Labs-wikitech-interface, 6operations: wikitech.wikimedia.org SSL certificate considered "outdated security" in Chrome - https://phabricator.wikimedia.org/T92709#1118619 (10Krinkle) 3NEW [02:08:34] 10Wikimedia-Labs-wikitech-interface, 6operations, 7HTTPS: wikitech.wikimedia.org SSL certificate considered "outdated security" in Chrome - https://phabricator.wikimedia.org/T92709#1118626 (10Krenair) [02:14:29] 6Labs, 10Continuous-Integration, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1118636 (10Krinkle) 3NEW [02:19:45] 6Labs, 10Continuous-Integration, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1118645 (10scfc) What do you mean by "puppet failures and random regressions" in this case? [02:38:39] 6Labs, 10Continuous-Integration, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1118670 (10Krinkle) >>! In T92710#1118645, @scfc wrote: > What do you mean by "puppet failures and random regressions" in this case? Integrity errors or quality issues... [03:59:39] 10Wikimedia-Labs-wikitech-interface, 6operations, 7HTTPS: wikitech.wikimedia.org SSL certificate considered "outdated security" in Chrome - https://phabricator.wikimedia.org/T92709#1118701 (10Dzahn) [04:21:15] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118720 (10Dzahn) about workarounds: /etc/nsswitch says: hosts: files dns so to... [04:30:44] 10Wikimedia-Labs-wikitech-interface, 6operations, 7HTTPS: wikitech.wikimedia.org SSL certificate considered "outdated security" in Chrome - https://phabricator.wikimedia.org/T92709#1118731 (10Dzahn) this should be T73156 (SHA1 needs to be replaced with a SHA256 cert) Chrome 39 will warn users if SHA1 certif... [04:31:14] 10Wikimedia-Labs-wikitech-interface, 6operations, 7HTTPS: wikitech.wikimedia.org SSL certificate considered "outdated security" in Chrome - https://phabricator.wikimedia.org/T92709#1118734 (10Dzahn) [06:31:03] 6Labs, 10Beta-Cluster, 10Staging, 5Patch-For-Review: Provide option to autosign puppet certs for self hosted puppetmasters - https://phabricator.wikimedia.org/T92606#1118769 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Bam, done! [06:46:46] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [06:53:01] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:55] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:57:53] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 752639 bytes in 3.245 second response time [07:16:51] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [07:21:55] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:46:00] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:56] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 752654 bytes in 4.769 second response time [10:37:58] mutante: i got access (i received the email that confirm that) [10:38:24] mutante: but i thought that i also need someone to actually deploy the key or something? [13:51:29] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118965 (10scfc) Has someone looked at whether there is an SOA record in LDAP? If that is... [13:53:20] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1118966 (10scfc) ``` From: root@tools.wmflabs.org (Cron Daemon) Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) To: root@tools.wmflabs.org Date:... [13:53:43] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1118967 (10scfc) ``` From: root@tools.wmflabs.org (Cron Daemon) Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) To: root@tools.wmflabs.org Da... [13:53:59] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1118968 (10scfc) ``` From: root@tools.wmflabs.org (Cron Daemon) Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) To: root@tools.wmflabs.org Date: S... [14:01:57] 6Labs, 10Continuous-Integration, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1118969 (10scfc) There are two aspects to this: # Whether the Git failure caused Puppet to fail. This seems to have been the case. # Whether the Puppet failure trigge... [16:07:16] how do I stop lighttpd from starting the php-cgi processes? [16:09:49] YuviPanda: Are you there ? [16:10:35] YuviPanda: does bigbrother handle webservice2 restarts properly? I see this in the log: "unrecognized arguments: -lighttpd" [16:11:11] para: I'm on mobile and out so can't help much. However, try 'webservice --release trusty' instead? [16:11:54] Vivek: hey! On my phone and not here for long.... [16:11:55] alright. what should go in .bigbrotherrc? [16:12:13] para: whatever I put in quotes [16:12:19] ah. thanks [16:12:26] 'webservice --release trusty' [16:13:10] YuviPanda: I wanted to know if I could volunteer ? [16:13:31] some that can help with access to bastion? [16:13:46] I had to leave abruptly yesterday. [16:28:45] Vivek: yes! Definitely! However now is bad timing :( [16:29:17] einyx: if your shell access was approved then you can just pur your key in the preferences page on wikitech and it should give you access [16:33:17] Ok. [16:45:33] para: this stops the 6x0.4GB php-cgi processes: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Performance#Default_php-fcgi [17:16:08] YuviPanda: after how long the key get deployed? [17:18:01] einyx: It shouldn't be very long. It gets added to Nova which is then checked when you SSH [17:22:51] 10Gerrit-Patch-Uploader: Allow amended patch sets without rebase with gerrit patch uploader - https://phabricator.wikimedia.org/T92736#1119123 (10Umherirrender) 3NEW [17:32:47] Hey all, I'm an editor at the Wikipedia Signpost and we're currently working on rolling out a fairly complex technical initiative that we'd like to get more advice with and developer time in producing. [17:33:00] Thought I'd stop by and see if I can solicit opinions or volunteers here. [17:34:02] Also, is there an edit counter for the total amounts of Commons uploads on Labs? [17:40:19] Well that wasn't much of a response...hmm. [17:41:42] ResMar: It is saturday. Just give me a min. :) [17:42:25] Sorry, IRC rooms confuse me when there's a f-ton of people on and no one actually, well, on. [17:43:03] Most people are marked as away (through /away) it makes them look present when they actually aren't [17:43:19] Its cleaner than changing nicks [17:43:31] Hardly most, a few at best. [17:44:07] * YuviPanda would love to answer but it’s 11:30 PM on saturday and I’m on my phone [17:44:10] Anyway that's not the point. [17:44:14] ResMar: you would have better luck emailing labs-l [17:44:14] Yuvi! [17:44:20] Yuvi! [17:44:20] hi :D [17:44:22] We emailed you! [17:44:25] I got an email about it as well [17:44:25] I think. [17:44:26] yeah [17:44:33] been swamped with too many things :( [17:44:40] Same here [17:44:49] :/ [17:44:55] * YuviPanda apologies to ResMar [17:44:59] It [17:45:01] I’ll try to respond on Monday [17:45:03] It's ok* [17:45:17] So what is this "technical initiative" [17:45:23] I love the signpost [17:45:25] We've been wanting to get the Android app restarted and were also thinking about applying for a grant to get an iOS app. [17:45:38] Nice [17:45:51] There are basically three initiatives right now, this is one of them. [17:45:51] but [17:46:21] isn't it wiki based so people would most likely use the Wikipedia app and read their talk page? [17:46:42] The Wikipedia App is not optimized for the Signpost. [17:47:03] Besides we're making a lot of visual changes right now that'll need close integration with any application for reading. [17:47:26] That's a pretty cool initiative [17:48:14] Our threadbare team isn't going to get direct funding from the Foundation because of COI but we could at least get them to bankroll our platforms, maybe. [17:48:19] Mobile is such a big focus right now... [17:49:21] But I was more referencing the other two. [17:49:26] Have you tried #wikimedia-tech? [17:49:48] I was not aware of wikimedia-tech [17:50:08] Well I guess that is more of a support channel [17:50:17] Ah. [17:50:22] Well, we're here now. [17:50:24] (It's a very very quiet channel, though!) [17:50:42] It's not like you techs don't have every single technical IRC client open at once anyway ;) [17:50:48] (so many channels ;_; ) [17:50:52] client -> channel [17:51:39] quid I've been floating the idea of notifications tied to the editing of specific pages, like super-watchlists. [17:51:43] But I can tell you're also super-buried. [17:51:48] Try running /msg alis list #wikimedia* [17:52:01] Just to give you an idea :) [17:52:11] You're not kicking me out are you? :L [17:52:17] No [17:52:22] Good! [17:52:23] verily! and yes, I'm also massively in favour of a significant upgrade to the complexities available in our watchlists. [17:52:59] I need multiple watchlists [17:53:05] We have an IRC room where a bot sniffs edits to Signpost subpages for the Signpost. [17:53:10] but yes, mega-buried. However, you might be interested in https://phabricator.wikimedia.org/T76199 [17:53:56] That's almost useful enough for people to use the channel...but not really ideal, because, uh, we still can't be bothered with it. [17:54:03] heh [17:54:25] yup, gotta make tools that suit the demographics who use them. :/ [17:54:46] I was thinking of quietly linking little-known pages I've created in the templates we use for creating new story proposals, so that if I check the linked article box I'd get the notifs. [17:56:11] But Ed's been here longer than I have and he tells me he'd get buried in spam, I mean, uh, the way the notification trigger is *suposed* to work :) [17:56:20] supposed* [17:56:55] My personal hack at the moment, is to use a second account (that never edits), which has enotifwatchlist turned on (send email, when pages on watchlist are edited), and use that account to watchlist a few critical pages on various wikis. [17:57:37] Whoa! Haha. [17:57:50] I want to bring this to everyone's notice: https://phabricator.wikimedia.org/T5525 There is a student interested in pursuing this as a GSoC project, but no mentors. :( [17:57:56] but yes. we need lots more power and control, over the various multiple watchlists. The central hub for collecting ideas/links, is https://www.mediawiki.org/wiki/Watchlist_wishlist [17:58:20] I guess it also needs to wait for SUL finalization. [17:58:29] legoktm: Do labs shell accounts require password authenticated keys? [17:58:32] Which is the lead in this week's N&N, incidentally. [17:59:08] :) [17:59:40] yup, the SUL folks have been doing damn good stuff. [17:59:48] Eventualism! it works for everything, damnit. [17:59:48] well there is some code that does watchlist grouping https://gerrit.wikimedia.org/r/#/c/16419/ [17:59:59] so old now that it's very hard to rebase [18:00:02] It's not my domain so I've messaged Keegan to check the facts. [18:00:21] It really ought to be covered in tech but we've not had regular tech reports since Jarry left, unfortunately. [18:00:58] ya :/ [18:01:03] Here's another thing we're working on, historical links: https://en.wikipedia.org/wiki/User:Resident_Mario/SandBox [18:03:12] But it's the last thing we're working on that's the most technically complicated and IMO exciting, which I came here for advice on, really, which is story-tagging. [18:04:16] We're trying to automate our archives so that series can be easily generated for all the endless crazy things that the Signpost has covered over multiple issues for the past...decade. [18:04:58] Mr. Stardivarius cooked up a Lua script that's done a lot of that work: User_talk:Resident_Mario#Signpost index [18:05:03] https://en.wikipedia.org/wiki/User_talk:Resident_Mario#Signpost%20index* [18:05:05] https://en.wikipedia.org/wiki/User_talk:Resident_Mario#Signpost%20index * [18:06:01] Now we're trying to figure out the how and where of how we're going to get this rolling. Labs? User scripts? Lua? Wikidata? [18:09:02] The only solid idea we have so far is that we want the UI to look something like the Stack Overflow one: http://stackoverflow.com/tags [18:12:48] Very interesting :) reminds me of #projects in phabricator, and much of the brainstorming that is going into Flow's long-term searching & filtering plans [18:13:14] *flails arms impotently* [18:13:56] (I'm not a dev, so obviously can't help advise. Also I need to leave the house today. ttfn o/ ) [18:14:28] Alright, I'll try the other tech channels too. [18:14:46] Hello [18:14:55] Penny for my though---oh, goodbye. [18:16:17] (some people have bad connections. Some people get disconnected when they briefly close their laptops to move to another room. Don't put too much weight to people joining/quitting. ;) [18:16:54] ResMar: I think labs-l will get you volunteers :) [18:17:15] There are apparently so many sub-wikimedia IRC rooms that I can't get to tech because the maximum that is listed is 60. [18:18:39] Although I maybe should I've not yet put the time into untangling the byzantine system that is the mailing list, so I haven't joined them yet. [18:19:47] \join wikimedia-tech [18:19:57] \join #wikimedia-tech [18:20:36] Uh, don't mind that. [18:56:44] How do you even null-quit? [18:59:12] 6Labs, 10Wikimedia-Labs-Infrastructure: No init script for idmapd on labs jessie instances - https://phabricator.wikimedia.org/T87309#1119193 (10faidon) This bandaid has been merged about a month ago. Is there anything left here? (I think not) [18:59:23] 6Labs, 10Wikimedia-Labs-Infrastructure: LVM failures on Debian Jessie labs VMs - https://phabricator.wikimedia.org/T87310#1119194 (10faidon) Is this still an issue? [18:59:34] 6Labs, 10Wikimedia-Labs-Infrastructure: Debian Jessie image for Labs - https://phabricator.wikimedia.org/T75592#1119195 (10faidon) What's left to do here? [19:09:23] 6Labs, 10Wikimedia-Labs-Infrastructure: LVM failures on Debian Jessie labs VMs - https://phabricator.wikimedia.org/T87310#1119198 (10coren) That depends on how you define //issue//. There is still a problem with Jessie refusing to boot properly if any of its bootstrap partitions are on LVM volumes, but it's n... [19:10:23] 6Labs, 10Wikimedia-Labs-Infrastructure: Debian Jessie image for Labs - https://phabricator.wikimedia.org/T75592#1119202 (10coren) [19:10:24] 6Labs, 10Wikimedia-Labs-Infrastructure: No init script for idmapd on labs jessie instances - https://phabricator.wikimedia.org/T87309#1119199 (10coren) 5Open>3Resolved a:3coren No, this has proven to work well. [19:11:08] 6Labs, 10Wikimedia-Labs-Infrastructure: Debian Jessie image for Labs - https://phabricator.wikimedia.org/T75592#763305 (10coren) [19:11:09] 6Labs, 10Wikimedia-Labs-Infrastructure: LVM failures on Debian Jessie labs VMs - https://phabricator.wikimedia.org/T87310#1119204 (10coren) [19:11:29] Coren: heh, thanks [19:11:34] I wasn't expecting a reply today :) [19:11:38] sorry! [19:12:34] 6Labs, 10Wikimedia-Labs-Infrastructure: Debian Jessie image for Labs - https://phabricator.wikimedia.org/T75592#1119205 (10coren) 5Open>3Resolved a:3coren This is complete, as far as I know. Whether we decide to pursue the LVM issue or monitor it upstream is no longer blocking, and indeed we have a work... [19:14:10] paravoid: I happened to be reading email when they popped in, so I had a choice between (a) take a minute or two and do it now or (b) set that aside for Monday then promptly forget about it. :-P [19:14:43] * Coren knows his own foilibles. :-) [19:15:13] foibles*