[00:14:53] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-dev.diskspace._var.byte_avail.value (10.00%) [00:22:31] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [06:11:05] no doubt people have seen this.. https://medium.com/@MrJamesFisher/wikipedia-needs-an-ide-not-a-wysiwyg-editor-7acd85b582c8 [07:48:06] !log started staggered upgrade os salt minion on all labs instances except beta (those were done earlier): completed are i-000000* and i-000001[0-6]* with a few excptions which will be dealt with later [07:48:07] started is not a valid project. [07:48:14] oh come on [07:48:22] it's for all of labs whaddya want, silly bot [07:49:50] fine, when someone shows up who knows where 'all labs' stuff should be logged, they can tell me [07:49:56] in the meantime, back to the upgrades [09:38:26] first round of salt minion updates complete. there re about 15 that must be dealt with manually (apt conf points to brewster, or dpkg is broken, or fs is broken, etc) [11:04:15] Coren: labs-vmbuilder-lucid is busted (maybe not needed any longer? hint hint): [11:04:16] apt-get update [11:04:17] 0% [Working]*** glibc detected *** /usr/lib/apt/methods/http: munmap_chunk(): invalid pointer: 0x00007f8e946a8470 *** [11:04:24] and then a nice stacktrace follows [11:34:15] it looks like a version of the 2.6.32 kernel tht was vulerable to the 208 days bug... going to reboot it (who can possibly be using it) [11:44:59] red herring, prolly a glibc bug [13:44:51] No doubt it can be axed by now. [14:38:06] well I worked around it (what are you doing on line now?? ah you are east coast...) [14:40:06] I think I am down to three instances that are a concern (there are a number that are shutoff, but I will ignore those, also a few nova error statuses, I don't know if you want to hear about those)... you know what, I'll just send an email [14:40:06] meh [14:40:16] and you can do whatever you want with the info including 'meh' ;-D [14:41:24] unless you would rather it be in a ticket? Coren [14:41:57] Nah, I'll tell andrewbogott_afk and unless he has a sudden need for it I'll just delete. [14:48:02] oh. no I mean the list of things that have nova errors... or the few that don't but aren't shut off, aren't pingable, dns fails.. [14:48:58] everything that responds to salt is updated, I'm now only down to something with oneiric on it, the logstash instances (left bd 808 a message), and these bad dns/ping/whatever/error instances [14:49:23] those are the ones I would rt or email... choose one (or choose irc + pastebin :-P ) Coren [14:50:07] email labs-l, I think, would be best. Easy to track, and interrested people can pipe up. [14:56:04] is there a reason why there is no logrotate for the web logs? [14:56:29] just noticed an access.log with > 300MB [14:56:33] ok, doing so now... err... not on that list >_< [15:14:00] ireas: There is no provision, atm, for per-user logrotate; that's been on my wishlist for some time but something else always took precedence. [15:16:30] Coren: okay. I agree it is not that important, but it would be nice to have before the log files get larger then 1 GB ;-) [15:17:29] It's not not important, it's just that there always turns out to be important-er things on my todo. :-) [15:39:15] Coren: emil sent, if the list is moderated it will be sitting in the queue. [15:39:34] kk [16:56:43] Change on 12mediawiki a page OAuth/For Developers was modified, changed by 82.9.34.145 link https://www.mediawiki.org/w/index.php?diff=1240076 edit summary: /* Ruby: OmniAuth strategy */ adding link to gem [17:57:53] 3Wikimedia Labs / 3wikistats: Make a stats table for Wikivoyage - 10https://bugzilla.wikimedia.org/44194#c8 (10Robert Hanke) 5RESO/FIX>3REOP Update Routine does not seem to work.