[00:17:52] !log wikistats test log [00:18:00] morebots stopped logging things [00:18:11] i already restarted the production copy but to no avail [00:18:34] !log no logging [00:18:34] no is not a valid project. [00:18:39] what [00:18:43] !log wikistats but this is [00:18:45] ... [00:18:56] !log morebots !log morebots [00:18:57] Did you mean tools.morebots instead of morebots? [00:18:57] morebots is not a valid project. [00:19:14] !log git how about this [00:19:39] !log tools is a valid project [00:26:02] 10Labs-Other-Projects: all morebots stopped listening to !log lines - https://phabricator.wikimedia.org/T137377#2366626 (10Dzahn) [00:27:56] 10Labs-Other-Projects: all morebots stopped listening to !log lines - https://phabricator.wikimedia.org/T137377#2366626 (10Legoktm) The last time I see it sucessfully logging was: ``` [15:44:06] !log tgr@tin Synchronized wmf-config/InitialiseSettings.php: enable AuthManager on group1 for reals T13550... [00:31:34] mutante: ^ [00:32:48] oh [00:32:54] interesting [00:51:06] I grafana was supposed to be working by now. [00:51:42] *I thought [00:51:46] yuvipanda: ^ [00:55:48] bd808: grafana is a catastrophic right now. The data is wrong and it completely fails 75% of the time now. [01:05:35] 10Labs-Other-Projects: all morebots stopped listening to !log lines - https://phabricator.wikimedia.org/T137377#2366626 (10scfc) `production-logbot.err` says: ``` […] 2016-06-09 00:15:27,273 ERROR: Failed to log message: LoginError(, {u'reason': u'You have m... [01:19:43] 10Labs-Other-Projects: all morebots stopped listening to !log lines - https://phabricator.wikimedia.org/T137377#2366716 (10Dzahn) So its really possible that the AuthManager change broke login for the bot? [01:36:34] 10Labs-Other-Projects, 10Adminbot: all morebots stopped listening to !log lines - https://phabricator.wikimedia.org/T137377#2366757 (10Peachey88) [02:21:11] 10Labs-Other-Projects, 10Adminbot, 13Patch-For-Review: all morebots stopped listening to !log lines - https://phabricator.wikimedia.org/T137377#2366626 (10Anomie) ``` 2016-06-09 02:08:27 [f4333b421af5d3f5880e6372] silver labswiki 1.28.0-wmf.5 exception ERROR: [f4333b421af5d3f5880e6372] /w/api.php DBQueryEr... [03:53:56] 06Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2366887 (10MZMcBride) >>! In T59617#2361443, @jcrespo wrote: > After some thinking, and my thoughts on why labs breaks so easily T136618#2356834, I think this is one of th... [04:16:38] So... I can't use the "take" command anymore. that intentional? [04:17:03] Tool labs that is. [04:18:03] NVM false alarm. Having six terminal windows and I tried to do it in the wrong one. [04:18:28] Crisis averted. [04:18:49] Yep. [08:37:22] !log ores rebooting ores-compute-01 [08:38:01] I can't connect to ores-compute-01.eqiad.wmflabs [08:38:14] I even tried rebooting [08:38:25] channel 0: open failed: connect failed: No route to host [08:38:25] stdio forwarding failedeapps [08:38:26] ssh_exchange_identification: Connection closed by remote host [08:42:17] it seems that's only ores-compute-01 [11:05:30] hello, is graphite back and running? [12:48:11] 10Labs-Other-Projects, 10Adminbot, 13Patch-For-Review, 05WMF-deploy-2016-06-14_(1.28.0-wmf.6): all morebots stopped listening to !log lines - https://phabricator.wikimedia.org/T137377#2367670 (10Tgr) 05Open>03Resolved a:03Anomie ``` 08:33 < logmsgbot> !log tgr@tin Synchronized php-1.28.0-wmf.5/extens... [13:56:47] 06Labs, 10Labs-Infrastructure, 06Operations, 06Release-Engineering-Team, and 2 others: Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2367839 (10hashar) @Dzahn thanks, though all those rules are indeed present on ho... [13:59:23] 06Labs, 10Labs-Infrastructure, 06Operations, 06Release-Engineering-Team, and 3 others: Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2367841 (10Dzahn) [14:19:53] Hey folks. I just found ores-compute-01.eqiad.wmflabs shut down and I can't start it up. [14:20:16] Last thing I did on the machine was enable "/srv" disk space. [14:20:29] That was um... ~2 days ago [14:20:39] Any suggestions? [14:20:50] andrewbogott or chasemp: ^ [14:21:18] halfak: I'll look in a minute [14:21:24] Thanks [14:26:05] halfak: for reasons that I'm not totally clear on, the host that instance is on won't allocate memory to start new VMs. I'm going to migrate you to a different host, will take a few minutes. [14:26:33] OK great. Do I get to keep all the data and configs in that VM? [14:27:01] yeah, nothing should change from your point of view, it'll just take a bit [14:27:11] yay! Ops magic [14:27:15] :) [14:39:26] halfak: is it working properly now? [14:43:12] yes. Thank you! [14:59:02] 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#2368011 (10Bmueller) @bd808, thanks for all your efforts and for coming up with an interim solution! [15:01:10] Is http://graphite.wmflabs.org/ still down? [15:02:39] I can't access labs graphite and I can't find a task for the problem. [15:04:52] halfak: I'm pretty sure graphite-in-labs was deprecated ages ago... [15:05:02] But you shouldn't take my word for it [15:05:04] huh. [15:05:16] That can't be right... [15:05:47] Maybe you are thinking about grafana.wmflabs.org? [15:06:05] maybe [15:09:54] halfak: I can debug if you have any how graphite works, where it's hosted, etc... [15:10:06] otherwise best to reach out to someone who is actually involved :) [15:10:14] OK. Maybe chasemp ? [15:10:23] Since I imagine this as a labs resource [15:10:51] yuvipanda, would probably know what's up too. [15:11:03] chase is moving, won't be online for a week or so [15:15:21] I don't remember what the status of the graphite host is. That was the one where yuvipanda tried to get the raid configuration changed and everything went badly. [15:17:22] bd808: that's what I'm confused about… if it's .wmflabs.org that refers to something running on a labs VM [15:17:30] so probably unrelated to labmon? [15:20:36] FWIW, I'm able to view stats from the graphite-labs on grafana. See https://grafana.wikimedia.org/dashboard/db/ores [15:20:42] Woops https://grafana.wikimedia.org/dashboard/db/ores-labs [15:21:09] Looks like maybe just the graphite UI is 'sploded [15:23:01] DNS maps graphite.wmflabs.org to the IP of the http proxy server (208.80.155.156) but I can't find an entry for it in the data dump I just made there [15:23:27] I wonder if that was setup strangely at some point and got lost when we had the redis issues? [15:28:52] !log mediawiki-vagrant Fixed broken iso image builder by manually running apt for T136146 [15:28:53] T136146: "Latest" hhvm package has lower version number and breaks mw-vagrant puppet - https://phabricator.wikimedia.org/T136146 [15:28:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-vagrant/SAL, Master [15:46:28] 06Labs, 10Labs-Infrastructure, 06Operations, 06Release-Engineering-Team, and 3 others: Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2368164 (10hashar) 05Open>03stalled From a quick chat with @mark we dont want... [16:13:13] 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#2368268 (10Umherirrender) ApiFeatureUsage at: https://de.wikipedia.org/wiki/Spezial:API-Funktionsverwendung?wpagent=Mozilla%2F5.0+MaboMwFramework... [16:53:19] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: I/O on labmon1001 is very slow - https://phabricator.wikimedia.org/T127957#2368458 (10scfc) This task is marked as blocked by T136227, but it looks to me as if instead T136972 is being pursued. Should the blocking tasks be changed? [18:58:51] hmm, puppet::self is a toast [19:02:27] MaxSem: I've used it recently… what are you seeing? [19:16:23] 06Labs: confirm that new base labs base image is adequate for kubernetes &c. - https://phabricator.wikimedia.org/T134944#2369166 (10Andrew) p:05Normal>03High It's a real bummer that the current Jessie image throws puppet errors out of the gate. Yuvi, can you invest some time in this so we can update the bas... [19:20:19] MaxSem: I just tried it on a new Jessie instance and it worked OK. I had to do "sudo service puppetmaster restart" in between puppet runs [19:30:44] !log commtech Rebuilding mw-vagrant vm on commtech-1 [19:30:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech/SAL, Master [19:35:55] andrewbogott, thanks - it helped [19:40:03] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: mediawiki-extensions-qunit failing "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2369227 (10hashar) The Nodepool instances receives their DNS configuration from DHCP.... [19:45:32] 06Labs, 10MediaWiki-Vagrant: Add default spam prevention tools or settings to wmflabs instances - https://phabricator.wikimedia.org/T131459#2369283 (10bd808) The hard part with making roles/configuration default for all MediaWiki-Vagrant instances setup in Labs is that these wikis are used for a really wide va... [19:53:16] 10Tool-Labs-tools-Other, 10Phabricator, 15User-bd808: Stashbot shouldn't subscribe itself to tasks - https://phabricator.wikimedia.org/T135790#2369299 (10bd808) Phabricator seems to have changed the default behavior related to the bot account flag. @gerritbot is now getting subscribed to tasks on comment as... [20:10:46] 06Labs, 10MediaWiki-Vagrant, 13Patch-For-Review: Add default spam prevention tools or settings to wmflabs instances - https://phabricator.wikimedia.org/T131459#2369363 (10Quiddity) https://www.mediawiki.org/wiki/Extension:QuestyCaptcha is often recommended for this. Could that (and E:ConfirmEdit) be a part o... [20:27:25] 10Tool-Labs-tools-Other, 10Phabricator, 15User-bd808: Stashbot shouldn't subscribe itself to tasks - https://phabricator.wikimedia.org/T135790#2311255 (10Paladox) @bd808 this is known upstream. [21:05:05] 06Labs, 10MediaWiki-Vagrant, 13Patch-For-Review: Add default spam prevention tools or settings to wmflabs instances - https://phabricator.wikimedia.org/T131459#2369786 (10bd808) You are trolling me by suggesting reCaptcha right? ({T129936}) [21:16:34] 06Labs, 10MediaWiki-Vagrant, 13Patch-For-Review: Add default spam prevention tools or settings to wmflabs instances - https://phabricator.wikimedia.org/T131459#2369828 (10Quiddity) >>! In T131459#2369786, @bd808 wrote: > You are trolling me by suggesting reCaptcha right? ({T129936}) Whoops, I didn't read th... [21:17:48] 06Labs, 10MediaWiki-Vagrant, 13Patch-For-Review: Add default spam prevention tools or settings to wmflabs instances - https://phabricator.wikimedia.org/T131459#2167691 (10Dzahn) T87598 [21:29:39] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2027631 (10Blahma) Just noticed this bug just before opening a new bug report on cswiki_p missing virtually all revisions and categorylinks from between 2016-03-08 18:00 and 21:00 UTC, which spoils the results... [22:09:41] andrewbogott, https://phabricator.wikimedia.org/T137347 can be closed? [22:09:57] I hope so! [22:10:21] 06Labs, 10Labs-Infrastructure: Instance creation results in nodes in ERROR state - https://phabricator.wikimedia.org/T137347#2370057 (10Andrew) 05Open>03Resolved a:03Andrew This was a not-fully-understood side effect of some maintenance work I was doing at the time. [22:32:07] (03PS1) 10Jean-Frédéric: Add testing infrastructure with npm/grunt [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293633 [22:32:56] (03CR) 10jenkins-bot: [V: 04-1] Add testing infrastructure with npm/grunt [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293633 (owner: 10Jean-Frédéric) [22:44:22] 06Labs, 10Labs-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2370138 (10Paladox) Jenkins is getting really slow now. 15-20mins for testing and merges. Could you up the priority pleas... [22:45:21] (03PS2) 10Jean-Frédéric: Add testing infrastructure with npm/grunt [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293633 [22:50:09] (03PS1) 10Jean-Frédéric: Add CSS linting via stylelint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293635 [22:57:33] (03CR) 10Lokal Profil: Add testing infrastructure with npm/grunt (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293633 (owner: 10Jean-Frédéric) [23:36:08] (03PS1) 10Jean-Frédéric: Do not import Intuition from toolbox PHP files [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651 [23:39:00] (03Abandoned) 10Jean-Frédéric: Do not import Intuition from toolbox PHP files [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651 (owner: 10Jean-Frédéric) [23:40:08] (03Restored) 10Jean-Frédéric: Do not import Intuition from toolbox PHP files [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651 (owner: 10Jean-Frédéric) [23:40:18] (03PS2) 10Jean-Frédéric: Do not import Intuition from toolbox PHP files [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651