[01:47:26] <bd808>	 !log tools.morebots qa-morebots awol from #wikimedia-qa
[01:47:31] <labs-morebots>	 Logged the message, Master
[06:58:15] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<22.22%)  
[08:18:15] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%)  
[09:03:16] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[09:33:29] <wikibugs>	 3Tool-Labs, Labs-Team: Fix Labs' PAM config mess - https://phabricator.wikimedia.org/T85910#957108 (10faidon) 3NEW
[09:51:11] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Can not sudo on deployment-cache-mobile03 - https://phabricator.wikimedia.org/T78720#957143 (10hashar)
[09:52:34] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Can not sudo on deployment-cache-mobile03 - https://phabricator.wikimedia.org/T78720#851871 (10hashar) Andrew, Marc, Yuvi, would you mind looking at deployment-cache-mobile03.eqiad.wmflabs please? I can not sudo  on it anymore and puppet is broken :-(   If it is fixa...
[09:55:57] <andrewbogott>	 hashar: I'm running puppet on deployment-cache-mobile03 now.  It doesn't look all that broken to me, but puppet is still running...
[09:56:31] <andrewbogott>	 ok, I maybe see the problem (although not how it got like this)
[09:58:31] <hashar>	 andrewbogott: welcome back :)
[09:58:54] <hashar>	 good to know puppet is working fine
[10:00:04] <andrewbogott>	 hashar: sudo-ldap was uninstalled, and puppet can't reinstall it (since apt is twitchy about messing with sudo)
[10:00:11] <andrewbogott>	 I'm installing it by hand now, we'll see if that fixes things.
[10:00:15] <hashar>	 \O/
[10:00:21] <andrewbogott>	 I don't know why it was removed, unless you have a manifest that installs sudo?
[10:00:36] <andrewbogott>	 (which might uninstall sudo-ldap since the two are mutually exclusive)
[10:00:39] <hashar>	 I have no idea :-/
[10:00:48] <hashar>	 that is a varnish cache for the beta cluster mobile traffic
[10:00:57] <andrewbogott>	 can you sudo now?
[10:00:58] <hashar>	 I don't think there is any sudo magic there
[10:01:07] <hashar>	 attempting
[10:01:10] <hashar>	 works!
[10:01:13] <hashar>	 you are magic!
[10:01:27] <andrewbogott>	 cool, I'll return to my nap attempt then :)
[10:01:35] <andrewbogott>	 btw, is there a CI meeting later?
[10:01:47] <hashar>	 andrewbogott: yup it is a weekly one
[10:01:57] <hashar>	 andrewbogott: though given the late time for you, you can probably skip it
[10:02:11] <andrewbogott>	 ok.  We'll see if I can stay awake… fighting a cold today.
[10:02:27] <andrewbogott>	 I'm running puppet again now to see if that system breaks itself again...
[10:02:36] <hashar>	 greg-g might be interesting in having a status update about new hardware being added to wmf lab
[10:02:42] <hashar>	 which you can do by email I guess
[10:02:55] <andrewbogott>	 puppet seems to alter LVS_SERVICE_IPS on each run, clearing, filling, clearing, filling...
[10:03:20] <hashar>	 yeah that always has been the case on the beta cluster varnish boxes. I have never bothered to investigate it though
[10:03:25] <andrewbogott>	 ok
[10:03:26] <hashar>	 there is no lvs supports on labs for sure
[10:03:33] <hashar>	 but it does not seem to cause any har
[10:03:34] <hashar>	 m
[10:04:16] <andrewbogott>	 ok, looks like puppet is fairly stable and you should have access.
[10:05:00] <hashar>	 I do :-] Thank you !
[10:05:02] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Can not sudo on deployment-cache-mobile03 - https://phabricator.wikimedia.org/T78720#957154 (10Andrew) 5Open>3Resolved a:3Andrew Puppet was choking on an attempt to install sudo-ldap.  I do not know why it wasn't installed already, since it comes stock on labs...
[10:05:19] <hashar>	 yeah yeah that is working properly
[10:05:25] <hashar>	 \O/
[10:07:55] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Continuous-Integration: Create labs project for CI disposables instances + OpenStack API credentials - https://phabricator.wikimedia.org/T84988#957158 (10hashar)
[10:07:57] <wikibugs>	 3Wikimedia-Labs-General: setup labs project for continuous integration jobs - https://phabricator.wikimedia.org/T55978#957159 (10hashar)
[10:08:11] <wikibugs>	 3Continuous-Integration, Wikimedia-Labs-General: setup labs project for continuous integration jobs - https://phabricator.wikimedia.org/T55978#578885 (10hashar)
[10:08:39] <hashar>	 andrewbogott_afk: thank you!
[10:08:51] <valhallasw`cloud>	 hashar: disposable CI instances <3
[10:08:59] <hashar>	 valhallasw`cloud: yeah eventually one day :]
[10:09:17] <hashar>	 valhallasw`cloud: the tracking ticket is https://phabricator.wikimedia.org/T47499  
[10:09:42] <valhallasw`cloud>	 subscribed!
[10:11:41] <hashar>	 valhallasw`cloud: there are multiple tasks slightly overlapping each others.  I should probably close them and reopen fresh tasks :-]
[15:05:40] <wikibugs>	 3Tool-Labs, Labs-Team: Fix Labs' PAM config mess - https://phabricator.wikimedia.org/T85910#957381 (10coren) All of this sounds sane.  As for the access.conf override, I'm very much not a fan of a global variable that silently changes security settings without a very visible and very glaring changeset that has h...
[15:13:45] <T13|mobile>	 Coren: how do I stop a specific job number in bash?
[15:14:35] <Coren>	 T13|mobile: qdel <jobno>
[15:15:32] <T13|mobile>	 Thanks
[15:34:19] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<66.67%)  
[15:37:12] <hashar>	 Coren: good morning!
[15:37:30] <hashar>	 Coren: thanks for your reviews yesterday (contint tmpfs stuff)
[15:37:35] <hashar>	 I guess they can be merged in now ?
[15:37:49] <Coren>	 hashar: Your call, if you feel ready for them.
[15:37:49] <hashar>	 https://gerrit.wikimedia.org/r/#/c/173511/ && https://gerrit.wikimedia.org/r/#/c/173512/
[15:37:53] <hashar>	 ready!
[15:41:25] <hashar>	 Coren: ready I am :-]
[15:41:46] <Coren>	 Ah, you want me to do it?  Sure.  Gimme a sec
[15:43:08] <Coren>	 Merged in.
[15:51:00] <hashar>	 Coren: great thanks!
[16:01:21] <wikibugs>	 3Continuous-Integration, Wikimedia-Labs-Infrastructure: CI labs instances can't start on reboot: tmpfs: Bad value 'jenkins-deploy' for mount option 'uid' - https://phabricator.wikimedia.org/T76250#957507 (10hashar) I have also updated lanthanum and gallium so they now have /var/lib/jenkins-slave/tmpfs belonging...
[18:44:15] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<33.33%)  
[18:59:14] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%)  
[19:09:18] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[19:23:27] <grrrit-wm>	 (03PS2) 10Awight: Add more repos maintained by Fundraising Tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/177378 
[21:15:16] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%)  
[21:25:17] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[21:54:37] <grrrit-wm>	 (03Abandoned) 10Awight: add Extension:FundraisingChart; notify wikimedia-dev about FR commits [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/119643 (owner: 10Awight)
[22:36:09] <Dispenser>	 Curiosity Question: How does MediaWiki handle leap seconds?  Are there any revisions ending in "%60" (my query was killed), does a leap second just last twice as long, or are our clock just 5 seconds off?
[22:39:00] <valhallasw`cloud>	 Dispenser: whatever php does, and that's whatever libc does, I think?
[22:39:46] <valhallasw`cloud>	 Dispenser: also, it's probably easier to check for specific examples instead: you know when the leap seconds were, so you can just use those dates
[22:40:51] <Dispenser>	 Dec 31, 2005 & 2008 + Jun 30 2012 & 2015
[22:42:37] <valhallasw`cloud>	 Dispenser: there's multiple revisions with rev_timestamp=20120630235959 but zero with 20120630235960
[22:42:45] <valhallasw`cloud>	 sorry, two with 20120630235959
[22:42:48] <valhallasw`cloud>	 so that's not conclusive
[22:43:53] <valhallasw`cloud>	 no 20051231235960 or 20081231235960 either
[22:44:02] <valhallasw`cloud>	 (used select rev_id, rev_timestamp from revision where rev_timestamp like '200812312359%' order by rev_timestamp desc limit 10;)
[22:45:54] <valhallasw`cloud>	 Dispenser: which doesn't answer the question of what happens in that second, though :/
[22:47:46] <Dispenser>	 It seems that it might be a longer second if its implemented with POSIX rules
[22:59:40] <wikibugs>	 3Labs-Team: Setup per-instance configurability for labs in hiera - https://phabricator.wikimedia.org/T1356#958407 (10chasemp) p:5Triage>3Normal
[23:00:29] <wikibugs>	 3Beta-Cluster, Labs-Team: Setup multimaster salt for large projects using salt-syndic - https://phabricator.wikimedia.org/T78466#958410 (10chasemp)
[23:00:37] <wikibugs>	 3Beta-Cluster, Labs-Team: Setup multimaster salt for large projects using salt-syndic - https://phabricator.wikimedia.org/T78466#958411 (10chasemp) p:5Triage>3Normal
[23:02:31] <wikibugs>	 3Wikimedia-Labs-wikitech-interface, Labs-Team: Use a Puppet ENC to define which classes are included in which nodes (in Labs) - https://phabricator.wikimedia.org/T85279#958417 (10chasemp) p:5Triage>3Normal
[23:09:47] <wikibugs>	 3Wikimedia-Labs-wikitech-interface, operations: Enable irc feed for wikitech.wikimedia.org site - https://phabricator.wikimedia.org/T36685#958446 (10chasemp) p:5High>3Volunteer?
[23:10:39] <wikibugs>	 3Wikimedia-Labs-wikitech-interface, operations: Enable irc feed for wikitech.wikimedia.org site - https://phabricator.wikimedia.org/T36685#391269 (10chasemp) This issue is three years old now, and has seen random attention.  It has no assignee.  It needs someone to volunteer for it (could be a WMF person) but it...