[00:01:31] <andrewbogott>	 I see… “emmanuel@emmanuel-Inspiron-1525” and “rsa-key-20140902” as the two registered keys.  Is one of those the one you’re using?
[00:01:58] <Baccanal>	 If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
[00:01:58] <Baccanal>	 Permission denied (publickey,hostbased).
[00:01:59] <Baccanal>	 If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
[00:01:59] <Baccanal>	 Permission denied (publickey,hostbased).
[00:01:59] <Baccanal>	 yes
[00:02:03] <Baccanal>	 sorry
[00:02:05] <Baccanal>	 yes
[00:04:29] <andrewbogott>	 Baccanal: do you see those “no such identity: /home/emmanuel/.ssh/id_dsa: No such file or directory” messages in your -vvv output?
[00:04:38] <andrewbogott>	 Seems like you can’t read your own keys, maybe.
[00:04:43] <andrewbogott>	 matanya: what do you think?
[00:05:34] <wikibugs>	 3Wikibugs, Tool-Labs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042643 (10Krinkle) 5stalled>3Invalid a:3Krinkle Bazinga!
[00:05:35] <wikibugs>	 3Wikibugs: wikibugs should notify on dependency changes - https://phabricator.wikimedia.org/T77006#1042646 (10Krinkle)
[00:06:48] <Baccanal>	 adrewbogott: yes but I have a file id_rsa instead
[00:07:28] <andrewbogott>	 Can you run with -vvv again and paste output?
[00:08:27] <Baccanal>	 https://dpaste.de/2P6J
[00:14:47] <andrewbogott>	 Baccanal: what is your username on wikitech?
[00:15:15] <Baccanal>	 andrewbogott: Automatik
[00:15:31] <andrewbogott>	 ok.
[00:15:50] <andrewbogott>	 I don’t know what’s happening… everything in the logs is consitent with just having a mismatched key :(
[00:26:10] <Baccanal>	 ok, so it's weird
[00:29:31] <andrewbogott>	 Baccanal: someone else might have new ideas to help sort this
[00:41:36] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<25.00%)  
[00:46:55] <Baccanal>	 I go, thanks andrewbogott for your help
[00:51:39] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[00:54:29] <wikibugs>	 3Labs: Missing record in replica - https://phabricator.wikimedia.org/T89689#1042701 (10Phe) 3NEW
[00:57:59] <wikibugs>	 3Labs: Missing record in replica - https://phabricator.wikimedia.org/T89689#1042709 (10scfc) a:3Springle @Springle, could you look into this as well, please?
[01:47:11] <wikibugs>	 3Labs, Labs-Vagrant, Wikimedia-Labs-Other, Wikimedia-Labs-wikistats, Wikimedia-Labs-wikitech-interface, Wikimedia-Labs-General, Beta-Cluster, Project-Creators, Wikimedia-Labs-Infrastructure, Tool-Labs-tools-Article-request, Tool-Labs, Wikimedia-Labs-extdist, MediaW... - https://phabricator.wikimedia.org/T89270#1042736
[04:22:21] <wikibugs>	 3Tool-Labs: Tools crontab replacement must check whether run as root - https://phabricator.wikimedia.org/T87527#1042850 (10scfc)
[04:22:22] <wikibugs>	 3Tool-Labs: tools-trusty uses local crontab instead of remote (tools-cron?) crontab - https://phabricator.wikimedia.org/T86445#1042851 (10scfc)
[04:24:57] <wikibugs>	 3Tool-Labs: tools-trusty uses local crontab instead of remote (tools-cron?) crontab - https://phabricator.wikimedia.org/T86445#1042853 (10scfc)
[04:24:58] <wikibugs>	 3Tool-Labs: Tools crontab replacement must check whether run as root - https://phabricator.wikimedia.org/T87527#1042852 (10scfc)
[04:28:54] <wikibugs>	 3Tool-Labs: tools-trusty uses local crontab instead of remote (tools-cron?) crontab - https://phabricator.wikimedia.org/T86445#1042855 (10scfc)
[04:28:56] <wikibugs>	 3Tool-Labs: Tools crontab replacement must check whether run as root - https://phabricator.wikimedia.org/T87527#1042854 (10scfc)
[04:29:08] <wikibugs>	 3Tool-Labs: Tools crontab replacement must check whether run as root - https://phabricator.wikimedia.org/T87527#993597 (10scfc)
[04:29:42] <wikibugs>	 3Tool-Labs: tools-trusty uses local crontab instead of remote (tools-cron?) crontab - https://phabricator.wikimedia.org/T86445#968396 (10scfc) (This "blocking" vs. "blocked by" business is absolutely frustrating :-(.)
[06:03:41] <wikibugs>	 3Tool-Labs-tools-Other: tmg/articlemedia tool not working - https://phabricator.wikimedia.org/T89695#1042897 (10Phoebe) 3NEW
[06:37:44] <valhallasw`cloud>	 legoktm i think noone has issues with it as long as its readable
[06:38:36] <valhallasw`cloud>	 But going back to all green is also fine with me
[06:43:28] <legoktm>	 okay :/
[06:48:11] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[06:54:49] <valhallasw`cloud>	 legoktm: the colors look cool and help to distinguish tag types, but they are maybe too inconsistent between clients...
[06:55:23] <valhallasw`cloud>	 (it looks great in irssi, less so in ircloud)
[07:18:12] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:53:41] <wikibugs>	 3Tool-Labs: Java jobs stop working - https://phabricator.wikimedia.org/T88799#1042987 (10dnaber) p:5Triage>3High
[07:56:57] <wikibugs>	 3Tool-Labs: Java jobs stop working - https://phabricator.wikimedia.org/T88799#1042988 (10dnaber) I've set this to priority 'high' now because the jobs need to run 24 hours per day to be really useful. Also, I have to log in about 5 times or so per week to restart the jobs. If we cannot solve this issue, I'll nee...
[08:27:01] <wikibugs>	 3Tool-Labs: Java jobs stop working - https://phabricator.wikimedia.org/T88799#1043022 (10valhallasw) Please check the output of   ```qacct -j fr-feedcheck```  to see what 'max vmem' is. Does it get up to 7500m?   In addition, is the job still running or not? If it's still running, it's not a memory issue.  Last,...
[12:04:51] <Baccanal>	 Hi
[12:05:16] <Baccanal>	 I've a problem: I'm not in the same groupe that my tool account. What should I do?
[12:05:21] <Baccanal>	 group*
[12:13:31] <valhallasw`cloud>	 Baccanal: I'm not sure what you mean by that. Do you mean 'groups' doesn't return the tool?
[12:13:40] <valhallasw`cloud>	 in that case: have you tried logging out and in again?
[12:17:43] <Baccanal>	 == {{langue|fr}} ==
[12:17:43] <Baccanal>	 === {{S|étymologie}} ===
[12:17:43] <Baccanal>	 : {{cf|administrateur|système}}.
[12:17:44] <Baccanal>	 === {{S|nom|fr}} ===
[12:17:44] <Baccanal>	 {{fr-rég|ad.mi.ni.stʁa.tœʁ sis.tɛm|p=administrateurs systèmes}}
[12:17:44] <Baccanal>	 '''administrateur systèmes''' {{pron|ad.mi.ni.stʁa.tœʁ sis.tɛm|fr}} {{m}}
[12:17:44] <Baccanal>	 # {{informatique|fr}} Personne responsable des [[serveur]]s d’une organisation.
[12:17:45] <Baccanal>	 ==== {{S|traductions}} ====
[12:17:45] <Baccanal>	 {{trad-début|}}
[12:17:45] <Baccanal>	 * {{T|en}} : {{trad-|en|system administrator}}
[12:17:46] <Baccanal>	 * {{T|nl}} : {{trad+|nl|systeembeheerder‎}}
[12:17:46] <Baccanal>	 {{trad-fin}}
[12:17:47] <Baccanal>	 [[Catégorie:Métiers du secteur tertiaire en français]]
[12:17:47] <Baccanal>	 {{clé de tri|administrateur systemes}}
[12:18:20] <Baccanal>	 sorry for this unfortunate copy and paste
[12:31:04] <Baccanal>	 for more clarity:
[12:31:15] <Baccanal>	 valhallasw`cloud: in fact, I logged in with SSH, and copied a file from /home/botomatik to /data/project/botomatik. But when I 'become botomatik', I can't write the file copied from /home/botomatik
[12:32:18] <valhallasw`cloud>	 Baccanal: right, because the group is your user group, not the tool group.
[12:32:58] <valhallasw`cloud>	 Baccanal: either take the file as tool ( 'take <filename>')  or chown user:group <filename> as user
[12:34:35] <Baccanal>	 ok take works!
[14:43:49] <wikibugs>	 3Labs, Wikimedia-Labs-wikitech-interface: Wikitech may fail to add the project-bastion group when shell access is granted - https://phabricator.wikimedia.org/T89667#1043590 (10coren) I've spent most of the last day trying to catch this bug "in the act" with little success beyond observing that jobs set to update...
[15:49:10] <Coren>	 andrewbogott_afk: Ping me once you're around?
[16:15:14] <paravoid>	 Coren, andrewbogott_afk: https://phabricator.wikimedia.org/T88923
[16:15:34] <paravoid>	 it's a failed disk, no response for over a week
[16:17:15] <Coren>	 paravoid: I never saw that one go by.  I'll sync up with Chris asap to swap it.
[16:17:22] <paravoid>	 thx
[16:17:28] <paravoid>	 needs to be tagged with #ops-eqiad
[16:17:51] <paravoid>	 it's not yet, it should be investigated first to make sure it's a HW problem indeed
[16:22:42] <wikibugs>	 3Gerrit-Patch-Uploader: Gerrit Patch Uploader does not reliably clean up after itself in /tmp - https://phabricator.wikimedia.org/T88517#1043719 (10Fomafix) 5Resolved>3Open REOPEN  The error occurs again: ``` Result from uploading patch: /data/project/gerrit-patch-uploader/git/bin/git clone --depth=1 ssh://g...
[16:23:35] <Coren>	 paravoid: That's what I'm looking at now.
[16:23:47] <paravoid>	 awesome
[16:24:33] <Coren>	 hostbyte=DID_NO_CONNECT is 99% sure to be a dead disk.  Updating ticker now.
[16:28:48] <paravoid>	 yup
[16:30:11] <wikibugs>	 3Labs: virt1002 broken disk? - https://phabricator.wikimedia.org/T88923#1043738 (10coren) Disk taken out of array because: [943491.790588] scsi 6:0:0:0: [sda]  Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK [943491.790590] scsi 6:0:0:0: [sda] CDB: Write(10): 2a 00 00 00 00 47 00 00 01 00 [943491.790595] en...
[16:30:27] <wikibugs>	 3Labs, ops-eqiad, operations: virt1002 broken disk? - https://phabricator.wikimedia.org/T88923#1043740 (10coren)
[16:30:45] <wikibugs>	 3Labs, ops-eqiad, operations: virt1002 broken disk? - https://phabricator.wikimedia.org/T88923#1043744 (10coren) p:5High>3Unbreak!
[16:31:00] <wikibugs>	 3Gerrit-Patch-Uploader: Gerrit Patch Uploader does not reliably clean up after itself in /tmp - https://phabricator.wikimedia.org/T88517#1043745 (10valhallasw) Odd. I cleared out /tmp again (which was again filled by g-p-u... or at leat 3.5GB of it) and restarted the webservice; maybe the fixed code wasn't loaded?
[16:33:39] <wikibugs>	 3Gerrit-Patch-Uploader: Gerrit Patch Uploader does not reliably clean up after itself in /tmp - https://phabricator.wikimedia.org/T88517#1043757 (10valhallasw) ```   File "/usr/lib/pymodules/python2.7/flup/server/fcgi_base.py", line 558, in run     protocolStatus, appStatus = self.server.handler(self)   File "/u...
[16:34:37] <Phawkes>	 [13gerrit-patch-uploader] 15valhallasw pushed 1 new commit to 06master: 02http://git.io/NANO
[16:34:37] <Phawkes>	 13gerrit-patch-uploader/06master 14e90e5e3 15Merlijn van Deen: import shutil for great glory
[16:35:17] <wikibugs>	 3Gerrit-Patch-Uploader: Gerrit Patch Uploader does not reliably clean up after itself in /tmp - https://phabricator.wikimedia.org/T88517#1043762 (10valhallasw) 5Open>3Resolved Deployed with import. Should *really* be OK now :-p
[16:35:43] <wikibugs>	 3Labs: Labs NFSv4/idmapd mess - https://phabricator.wikimedia.org/T87870#1043767 (10coren) After investigation and testing with labstore2001, disabling idmap entirely will be //approximately// a noop but will require a reboot and restart of labstore.  (I plan on using the restart required by the shelves being mo...
[16:39:06] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webgrid-02 is OK: OK: All targets OK  
[16:54:44] <GerardM->	 problems ?
[16:55:42] <shinken-wm>	 PROBLEM - Host tools-webproxy is DOWN: PING CRITICAL - Packet loss = 100%  
[16:56:26] <shinken-wm>	 PROBLEM - Host tools-exec-cyberbot is DOWN: CRITICAL - Host Unreachable (10.68.16.39)  
[16:57:02] <shinken-wm>	 PROBLEM - Host tools-exec-03 is DOWN: CRITICAL - Host Unreachable (10.68.16.32)  
[16:57:02] <shinken-wm>	 PROBLEM - Host tools-webgrid-04 is DOWN: CRITICAL - Host Unreachable (10.68.17.174)  
[16:57:03] <quiddity>	 labs seems to be down. http://tools.wmflabs.org/ etc
[16:57:12] <shinken-wm>	 PROBLEM - Host tools-exec-09 is DOWN: CRITICAL - Host Unreachable (10.68.17.64)  
[16:57:35] <GerardM->	 yep
[16:58:40] <shinken-wm>	 PROBLEM - Host tools-exec-07 is DOWN: CRITICAL - Host Unreachable (10.68.16.36)  
[17:00:13] <quiddity>	 Coren, legoktm ^
[17:00:23] <petan>	 meh
[17:00:28] <petan>	 Yuvi|Vacation: proxy is down?
[17:00:35] <quiddity>	 he's on vacation
[17:00:40] <petan>	 labs are having vacation as well
[17:00:44] <quiddity>	 heh
[17:01:39] <petan>	 ok so who do we have here who isn't on vacation
[17:01:49] <petan>	 Coren: are you here? it seems that web proxy is not working
[17:01:54] <T13|mobile>	 wm-bot...
[17:02:00] <petan>	 !ping
[17:02:01] <wm-bot>	 !pong
[17:02:13] <T13|mobile>	 Not on vacation. ^^ see.
[17:02:14] <petan>	 wm-bot is superbot is survives every outage
[17:02:19] <petan>	 too well written program
[17:02:26] <T13|mobile>	 Not every...
[17:02:31] <petan>	 if all programs were so epic as wm-bot world would be a better place
[17:02:38] <T13|mobile>	 +1
[17:03:05] <petan>	 ok so icinga just died
[17:03:12] <petan>	 houston WE GOT A PROBLEM
[17:03:58] <chasemp>	 andrewbogott: or Coren if you are about labs seems to be having some kind of issue :)
[17:04:32] <andrewbogott>	 chasemp: I’m in a call… will try to catch up
[17:04:49] <chasemp>	 andrewbogott: no worries just hadn't seen anyone ping you yet 
[17:04:58] <petan>	 andrewbogott: ping
[17:05:10] <petan>	 andrewbogott: did you notice that labs just blown up?
[17:05:31] <andrewbogott>	 Can y’all be more specific?  I’m multitasking, haven’t had a chance to read the backscroll
[17:05:46] <petan>	 for beginning, webproxy doesn't work
[17:05:49] <petan>	 it times out
[17:05:52] <andrewbogott>	 ok
[17:05:59] <petan>	 but other vm's are down as well
[17:06:02] <petan>	 per icinga
[17:06:06] <petan>	 lot of boxes died
[17:06:31] <petan>	 you see all these people are joining ^ because they are confused why their boxes just died
[17:06:35] <petan>	 they are all too shy to speak
[17:06:44] <petan>	 they just silently watch you and wait for you to save us
[17:08:25] <Chameleon_>	 nah, we're just trying to read the channel logs before re-asking if the world has died
[17:11:02] <andrewbogott>	 andrew@virt1005:/$ free
[17:11:03] <andrewbogott>	 -bash: /usr/bin/free: Input/output error
[17:11:20] <andrewbogott>	 So one of the virt hosts is dying.  I’m investigating :(
[17:11:33] <Qcoder00>	 Hi
[17:11:47] <marktraceur>	 Hi Qcoder00, issue already reported, seems to be recovery time
[17:11:48] <Qcoder00>	 Why does labs havign an issue slow down other sites?
[17:12:00] <marktraceur>	 I stand corrected
[17:12:05] <Qcoder00>	 I thought labs was on a seperate circuit?
[17:14:43] <marktraceur>	 It is, but if you have gadgets that access labs then you sort of bridge the gap :)
[17:14:49] <marktraceur>	 But only for your machine
[17:15:04] <Krenair>	 andrewbogott, chasemp I noticed that Special:NovaProxy shows no entries for deployment-prep
[17:15:05] <Krenair>	 but I was pretty sure we had a few just a couple of days ago
[17:15:06] <Krenair>	 maybe related, I don't know
[17:15:16] <Krenair>	 Qcoder00, separate circuit to what? the rest of production?
[17:15:21] <Qcoder00>	 Yeah
[17:15:25] <Krenair>	 it is
[17:15:30] <Qcoder00>	 Hmmm
[17:15:38] <Qcoder00>	 SO Wikisource shouldn't be slowing down?
[17:15:42] <Krenair>	 no
[17:15:46] <Qcoder00>	 Strange
[17:18:48] <Qcoder00>	 Hmm seems a Gadget is brdiging the gap
[17:18:58] <Qcoder00>	 Turned them alll off and it's running much faster
[17:19:36] <Krenair>	 You had a gadget on pulling data from labs every page view? :p
[17:20:50] <Qcoder00>	 Yeah...
[17:20:53] <Qcoder00>	 Not sure which though
[17:21:23] <L235>	 Oh dear
[17:22:35] <legoktm>	 hi
[17:23:15] <Krenair>	 Qcoder00, enwikisource?
[17:23:20] <Qcoder00>	 Yeah
[17:23:51] <Qcoder00>	 Still not sure which Gadget would do that?
[17:23:59] <Krenair>	 one of these: MediaWiki:BookMaker.js, MediaWiki:Gadget-BugStatusUpdate.js, MediaWiki:Gadget-WSexport.js, MediaWiki:Gadget-WhatLeavesHere.js, MediaWiki:Gadget-ocr.js, MediaWiki:Mobile.js, MediaWiki:OCR.js, MediaWiki:TemplateScript/pagetools-config.js
[17:24:19] <andrewbogott>	 Here is a list of hosts affected by the current outage:  https://phabricator.wikimedia.org/P305
[17:24:30] <Krenair>	 probably one prefixed with Gadget-, though a gadget could include any of them
[17:26:40] <Qcoder00>	 MediaWiki:Gadget-WSexport.js, (snickers...)
[17:26:58] <Qcoder00>	 (Sorry)
[17:27:10] <shinken-wm>	 PROBLEM - Host tools-submit is DOWN: CRITICAL - Host Unreachable (10.68.17.1)  
[17:27:34] <shinken-wm>	 PROBLEM - Host tools-webgrid-tomcat is DOWN: CRITICAL - Host Unreachable (10.68.16.29)  
[17:30:20] <wikibugs>	 3Wikimedia-Labs-Infrastructure: source group field is confusing - https://phabricator.wikimedia.org/T69759#1044008 (10chasemp) 5Open>3declined a:3chasemp meh, I made this report ages ago in bugzilla because I found this confusing. I'm closing it myself as I see no real actionable.
[17:37:58] <andrewbogott>	 good news, virt1005 survived a reboot so we may be able to rescue these VMs.
[17:38:11] <andrewbogott>	 It’ll be a bit before they’re up and running though
[17:40:12] <Coren>	 Gah!  Fine time I took to get lunch.
[17:41:24] <Coren>	 More precisely, fine time virt1005 took to die.
[17:41:35] <Coren>	 andrewbogott: Point me at work I can do to speed things up.
[17:42:08] <andrewbogott>	 Coren: ok.  I’m going to cold-migrate everything to virt1012.
[17:43:11] <Coren>	 andrewbogott: Giving up on virt1005?  Do we know what died?
[17:43:37] <andrewbogott>	 disk controller
[17:44:29] <Coren>	 dmesg
[17:46:50] <andrewbogott>	 Coren: https://phabricator.wikimedia.org/P304
[17:46:52] <Coren>	 andrewbogott: afaict, the box is fully up right now; or is a controller entirely missing?
[17:47:03] <andrewbogott>	 Also, review please?  https://gerrit.wikimedia.org/r/#/c/191079/
[17:47:15] <andrewbogott>	 I cycled power and it came back up but now I’m very suspicious.
[17:47:25] <andrewbogott>	 (backscroll about all this in #security)
[17:48:45] <Coren>	 I just read the backscroll and I agree with the evactuation - that's a nasty thing even if it was a fluke and I'd make sure it was before trusting it again.
[17:49:38] <andrewbogott>	 yeah
[17:49:50] <andrewbogott>	 Next task after this fire is out is a req for some new HPs
[18:01:52] <Qcoder00>	 Found the first gadget that's problematic with labs...
[18:02:01] <Qcoder00>	 MediaWiki:OCR.js
[18:10:23] <GerardM->	 question how hard is it to have the web proxy on multiple systems like WDQ ?
[18:13:48] <Coren>	 GerardM-: With the current setup, fairly hard - it'd need a HA redesign.  That said, the current outage does suggest that it might be worth the trouble.
[18:31:43] <Coren>	 GerardM-: The webproxy should be back around 15-20 min from now.
[18:37:36] <Coren>	 GerardM-: It should be up now, actually.
[18:38:30] <shinken-wm>	 RECOVERY - Host tools-webproxy is UP: PING OK - Packet loss = 0%, RTA = 0.51 ms  
[18:42:52] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[18:46:33] <wikibugs>	 3Tool-Labs, Wikimedia-Labs-Infrastructure: Make ar_content_format and ar_content_model available on ToolLabs - https://phabricator.wikimedia.org/T89741#1044220 (10Umherirrender) 3NEW
[18:46:40] <shinken-wm>	 RECOVERY - Host tools-submit is UP: PING OK - Packet loss = 0%, RTA = 0.70 ms  
[18:56:05] <Qcoder00>	 Hmm OCR is still down, but that's not a big inconnvenience
[19:03:27] <Coren>	 Qcoder00: Does it live in tools or on its own project?
[19:03:35] <Qcoder00>	 Tools I think
[19:03:59] <Qcoder00>	 MediaWiki:OCR.js is the script, but it uses lab resource I think
[19:07:52] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0]  
[19:15:51] <chasemp>	 webproxy seems not up? https://phab-01.wmflabs.org/
[19:16:00] <chasemp>	 mentioning because it seemed like it was meant to be back 
[19:16:59] <andrewbogott>	 chasemp: there are two different proxies, one for tools and one for everything else.
[19:17:05] <andrewbogott>	 the tools proxy is back, the other one, not yet.
[19:17:10] <chasemp>	 ah shows me lack of knowledge :)
[19:29:41] <GerardM->	 Hoi, WDQ is still down ... it could be up but is not
[19:30:43] <GerardM->	 given that it is load balanced ....
[19:43:00] <wikibugs>	 3Labs: New hp servers for labs - https://phabricator.wikimedia.org/T89752#1044517 (10Andrew) 3NEW
[19:46:37] <wikibugs>	 3Labs: New hp servers for labs - https://phabricator.wikimedia.org/T89752#1044537 (10Andrew) Specs and pricing here:  https://rt.wikimedia.org/Ticket/Display.html?id=8390
[20:12:08] <andrewbogott>	 chasemp: proxy is back, finally
[20:12:57] <chasemp>	 tx
[20:13:49] <chrismcmahon>	 do we have an ETA for beta cluster being functional again? 
[20:21:26] <seth>	 Hi!
[20:22:07] <seth>	 I'm getting an error when using jstart. Am I right here?
[20:24:24] <seth>	 Coren, petan: ?
[20:24:52] <Coren>	 seth: We're at reduced capacity atm because of a hardware failiure - that is almost certainly the issue.
[20:25:08] <seth>	 the message is "libgcc_s.so.1 must be installed for pthread_cancel to work" and "/var/spool/gridengine/execd/tools-exec-05/job_scripts/8221399: line 4: 17637 Aborted                 (core dumped) [...]"
[20:25:24] <Coren>	 Ah, no - that's just an out-of-memory.  :-)
[20:25:56] <seth>	 if I don't use jstart, everything works fine.
[20:26:00] <Coren>	 https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Why_am_I_getting_errors_about_libgcc_s.so.1_must_be_installed_for_pthread_cancel_to_work.3F
[20:28:18] <seth>	 Oops, sorry for not having read the faq sufficiently. Thanks for the link, I'll try that. :-)
[20:47:55] <Betacommand>	 Coren: is web proxy back up?
[20:48:15] <Coren>	 Betacommand: It is, though not all webgrid nodes are back yet (they are being evacuated now)
[20:49:17] <ArashPT>	 I can't restart or stop my webservice. screenshot: http://img42.com/DEp5h+
[20:49:44] <ArashPT>	 what should I do ?
[20:50:15] <Coren>	 ArashPT: We're at reduced capacity atm due to hardware failure.  You can ^C this - your webservice will remain queued and will be able to restart once the nodes return to availability
[20:50:28] <ArashPT>	 thanks
[20:51:20] <Krinkle>	 petan: "[Wikitech-l] New recent changes library", broken urls (and no 404 handlers on wikitech apparently)
[20:52:49] <qchris>	 Coren: A new user cannot (and never could) log in to gerrit. It seems his LDAP entry does not have the mail property set.
[20:52:50] <qchris>	 Can I verify somewhere/somehow that the user has set an email address on wikitech? Or how would the user get his LDAP "mail" set?
[20:54:10] <Coren>	 qchris: Almost certainly related to https://phabricator.wikimedia.org/T89001
[20:54:55] * qchris reads
[20:54:59] <Coren>	 qchris: There is little that can be done retroactively except patch the LDAP entry by hand.  Can do, if you need it; but please add to the phab task with the symptom
[20:55:38] <qchris>	 k. I'll add it to the ticket, then ping you about manually adding the mail.
[21:00:06] <wikibugs>	 3Labs: Wikitech creates broken LDAP entry for new instances and users - https://phabricator.wikimedia.org/T89001#1044752 (10coren)
[21:03:15] <wikibugs>	 3Labs: LDAP user without "mail" property got created - https://phabricator.wikimedia.org/T89760#1044759 (10QChris) 3NEW
[21:14:24] <shinken-wm>	 RECOVERY - Host tools-exec-03 is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms  
[21:17:37] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<12.50%)  
[21:18:31] <Betacommand>	 Coren: Im getting a 504 for my tool
[21:20:24] <wikibugs>	 3Labs: Wikitech creates broken LDAP entry for new instances and users - https://phabricator.wikimedia.org/T89001#1044814 (10QChris)
[21:20:25] <wikibugs>	 3Labs: LDAP user without "mail" property got created - https://phabricator.wikimedia.org/T89760#1044811 (10QChris) 5Open>3Resolved a:3QChris Coren added the mail property by hand for the user.  Thanks!
[21:25:45] <shinken-wm>	 RECOVERY - Host tools-exec-07 is UP: PING OK - Packet loss = 0%, RTA = 0.55 ms  
[21:27:06] <shinken-wm>	 RECOVERY - Host tools-exec-09 is UP: PING OK - Packet loss = 0%, RTA = 0.78 ms  
[21:27:38] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[21:30:48] <Betacommand>	 Coren: task 7741127 isnt dieing
[21:33:07] <Coren>	 Betacommand: It "lived" on a server that is currently down.  qdel -f did the trick.
[21:33:30] <Betacommand>	 Coren: thanks
[21:34:14] <shinken-wm>	 RECOVERY - Host tools-webgrid-04 is UP: PING OK - Packet loss = 0%, RTA = 0.74 ms  
[21:34:33] <Coren>	 Ah, and there is returns.
[21:35:04] <shinken-wm>	 RECOVERY - Host tools-exec-cyberbot is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms  
[21:35:40] <wikibugs>	 3Tool-Labs: Java jobs stop working - https://phabricator.wikimedia.org/T88799#1044847 (10dnaber) In about 90% the jobs are not running anymore. What's the correct behavior if the DB cannot be connected? Just crash and assume the process will automatically be restarted? Or manually try reconnecting after a wait p...
[21:41:27] <shinken-wm>	 RECOVERY - Host tools-webgrid-tomcat is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms  
[22:31:43] <grrrit-wm>	 (03PS1) 10Legoktm: Send AutoWikiBrowser to #autowikibrowser [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/191199 
[22:31:47] <legoktm>	 Reedy: ^
[22:31:55] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Send AutoWikiBrowser to #autowikibrowser [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/191199 (owner: 10Legoktm)
[22:32:05] <grrrit-wm>	 (03Merged) 10jenkins-bot: Send AutoWikiBrowser to #autowikibrowser [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/191199 (owner: 10Legoktm)
[22:32:18] <grrrit-wm>	 (03CR) 10Reedy: "Thanks! :D" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/191199 (owner: 10Legoktm)
[22:32:53] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 8ed0a167e287b7c3374f8b9b7e556e9b4b6180d6 Send AutoWikiBrowser to #autowikibrowser
[22:32:58] <labs-morebots>	 Logged the message, Master
[23:10:42] <Krenair>	 andrewbogott, is everything supposed to be back to normal now?
[23:11:05] <andrewbogott>	 Krenair: yep
[23:12:55] <Krenair>	 btw andrewbogott, why do I still get referenced to labsconsole when logging into stuff?
[23:13:01] <Krenair>	 references*
[23:13:48] <andrewbogott>	 Krenair: can you be more specific?
[23:13:58] <Krenair>	 If you are having access problems, please see:https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
[23:14:39] <andrewbogott>	 hm, interesting, I will look
[23:15:14] <andrewbogott>	 Krenair: looks to me like that’s an instance with an obsolete puppet applied to it.  what box?
[23:15:31] <Krenair>	 deployment-bastion
[23:15:47] <andrewbogott>	 yeah, that has its own puppet host.
[23:15:50] <Krenair>	 also, beta is broken because:
[23:15:50] <andrewbogott>	 Managed by… not me.
[23:15:51] <Krenair>	 krenair@deployment-db1:~$ sudo service mysql status
[23:15:51] <Krenair>	  * MySQL is not running, but PID file exists
[23:17:10] <Coren>	 Krenair: That's not surprising; all the affected instances have crashed violently.
[23:17:23] <andrewbogott>	 Krenair: hm, probably a side-effect of an unexpected reboot on deployment-db1 
[23:17:30] <Krenair>	 I started it
[23:17:34] <andrewbogott>	 I don’t know immediately how to cheer it up
[23:17:34] <Krenair>	 beta appears much less broken than normal
[23:17:46] * Coren chuckles.
[23:17:58] <Coren>	 Less broken than normal when it crashes.  :-)
[23:18:17] <Krenair>	 !log deployment-prep Started mysql on deployment-db1; beta now appears much less broken than before
[23:18:21] <labs-morebots>	 Logged the message, Master
[23:19:53] <andrewbogott>	 Coren: I would like to murder tools-webproxy-jessie and tools-webproxy-test since they were built using an obsolete image type.  Can you assure me that they’re OK to kill, or do I need to wait for Yuvi?
[23:20:02] <chrismcmahon>	 ? the api on beta has been returning "DB connection error" but now returns "The wiki is currently in read-only mode". progress?
[23:20:21] <Krenair>	 chrismcmahon, I turned on the DB server
[23:20:27] <Krenair>	 so it's less broken.
[23:20:35] <chrismcmahon>	 that would account for it
[23:20:41] <Coren>	 andrewbogott: I have confidence level 99% that they are useless; I wouldn't hesitate to shut them down.
[23:20:55] <Krenair>	 Isn't morebots supposed to log stuff to a page on wikitech?
[23:20:56] <andrewbogott>	 cool, thanks
[23:21:01] <Krenair>	 It does not appear to have worked.
[23:22:23] <Krenair>	 chrismcmahon, although I don't see it being read-only... details?
[23:22:29] <Krenair>	 http://en.wikipedia.beta.wmflabs.org/w/index.php?title=User:Krenair&action=history
[23:23:28] <andrewbogott>	 I see it… https://wikitech.wikimedia.org/wiki/Labs_Server_Admin_Log
[23:23:28] <chrismcmahon>	 Krenair: seems to be 100% back for me now also, thanks
[23:23:48] <andrewbogott>	 Krenair: ‘Started mysql on deployment-db1’ that, you mean?
[23:24:01] <Krenair>	 andrewbogott, aha, there it is
[23:24:41] <Krenair>	 didn't show on RC because labslogbot uses the bot flag
[23:27:04] <shinken-wm>	 PROBLEM - Host tools-webproxy-jessie is DOWN: CRITICAL - Host Unreachable (10.68.17.147)  
[23:27:30] <andrewbogott>	 hm, useless but monitored
[23:27:59] <Coren>	 That's automagical from the puppet manifest.
[23:28:12] <andrewbogott>	 Oh, ok.
[23:28:21] <andrewbogott>	 So, I don’t need to unregister?
[23:28:55] <shinken-wm>	 PROBLEM - Host tools-webproxy-test is DOWN: CRITICAL - Host Unreachable (10.68.16.113)  
[23:31:16] <Coren>	 I mean the monitoring was added automagically; it won't go away that easy.  :-P
[23:34:16] <andrewbogott>	 OK, if nothing else is breaking at this exact minute, I may go and get some lunch.
[23:34:22] * andrewbogott counts to ten with crossed fingers
[23:36:20] <andrewbogott>	 I’ll send a proper outage email when I return.
[23:36:58] <andrewbogott>	 Coren, I wasn’t much good at delegating during this one but I nonetheless appreciate your support :)
[23:41:19] <Coren>	 andrewbogott: Go lunch.  I'm glad I could be of moral support if nothing else; at least I was able to baby the tools instances as they came back up.