[00:26:53] anyone know if puppet was set up recently (yesterday ish perhaps) to clobber any apache configs on labs installs? [00:27:27] jamesofur: yes it was [00:27:31] :-/ [00:27:34] ori refactored apache module [00:27:43] is there a way for me to stop this? [00:27:51] and I wish someone had said something :( [00:27:52] not sure yet, i saw andrew bogott talk to him [00:28:02] yes [00:36:02] Betacommand: the webservice start command didn't change the error at that URL [00:47:17] jamesofur: hey [00:47:22] only got a minute, but [00:47:34] first, sorry for the incovenience [00:47:55] i've set aside time later tonight (i'm in pst) to try and rectify it [00:48:17] the long and short of it is that files in the sites-enabled directory ought to be declared to puppet [00:48:33] do you have a copy of the files that were purged? [00:48:53] if not, they may be recoverable via puppet's filebucket functionality [00:50:12] jamesofur: in the interim [00:50:20] is there a way for me to stop this? [00:50:35] if you have sudo rights on the labs instance, the answer is yes: run puppetd --disable [00:51:01] that's not a viable long-term strategy, but if you need a quick fix it's a possibility [00:52:38] i'll be back in about three hours and then i'll get cracking on documenting a proper way to adapt to this change [00:52:47] sorry again for the inconvenience and ttyl [00:52:50] ori: aye, that sounds like the short term, it also looks like you may not be clobbering apache2.conf ... which means my horrible hack of "include this config from my home directory" appears to be working [00:52:57] at least so far... [00:53:15] jamesofur: yes, that works too [00:53:33] the apache2 module is not managing apache2.conf [00:53:45] and probably won't, ever, precisely to allow for these kinds of customizations [00:53:48] sadly no, I don't have a copy of the files that were purged :( trying to recreate it now [00:53:51] so if that works for you that's a fine solution [00:54:21] puppet should have "bucketed" the files it purged, meaning they're accessible via the puppet-filebucket tool [00:54:29] * jamesofur looks [00:54:34] if you note here what the instance name is i'll try to do that when i get back [00:54:40] but if you want to give it a shot yourself check out http://docs.puppetlabs.com/references/3.5.1/man/filebucket.html [00:54:54] also since i have to run i ran out of time before managing to send an email about this or updating bugzilla [00:55:02] if you know anyone else that is affected and in distress could you let them know? [00:55:13] gotta run, bbl [00:55:18] I will do, thank you :) [01:22:54] ori: thanks for the pointers, I grabbed it out of the bucket and have set myself up again (running off the project space for now rather then my personal directory). I understand time issues but in the future it would be helpful if we could adjust the order of operations, breaking changes like this should be announced before (even, when necessary, right before) the change itself precisely because you will inevitably run into " [01:22:55] ran out of time and didn't notify" after the change itself. [02:26:01] Interestingly its also a more descriptive system for describing the weather: below zero=Stay in, 10s=Really really cold, 20s=freezing cold, 30s=things will melt in the sun, 40s=Take a jacket, 50s=dress up outside, 60s=alright, 70s/80s=Open Windows, 90s=Turn on the AC, above 100=No working outside [02:26:18] wrong channel [10:50:28] I put in a project request onmedia labs [10:51:19] oops - on Wikimedia labs, a couple of days ago. How long does it usually take to get approved & set up? [13:54:48] 3Tool Labs tools / 3[other]: Migrate https://toolserver.org/~daniel/WikiSense/* to Tool Labs and provide redirect - 10https://bugzilla.wikimedia.org/58869#c5 (10nosy) Set the redirect. [15:18:49] 3Wikimedia Labs / 3wikidata: install AbuseFilter and SpamBlacklist on wikidata jenkins - 10https://bugzilla.wikimedia.org/65727#c2 (10Aude) this would still be nice to do, but a workaround is to ensure the hook gets run regardless of if it is registered. [15:19:48] 3Wikimedia Labs / 3wikidata: run phpunit (on jenkins) on EditFilterMergedContent handling code in EditEntity - 10https://bugzilla.wikimedia.org/65727 (10Aude) [15:19:53] bah [16:13:51] 3Wikimedia Labs / 3tools: Old version of webservice script on tools-dev - 10https://bugzilla.wikimedia.org/66845 (10Peter Schlömer (dapete)) 3NEW p:3Unprio s:3minor a:3Marc A. Pelletier The /usr/bin/webservice script on 'tools-dev' is not up-to-date; it is dated March 9, while the version on 'tools-l... [17:10:39] !log local-heritage Fixed database_statistics.py after notification on https://commons.wikimedia.org/wiki/Commons_talk:Monuments_database/Statistics#Bug_in_the_URL . Still have to commit it [17:10:40] Logged the message, Master [18:42:50] hi andrewbogott [18:42:56] something really strange happened [18:42:59] sudo pupetd -tv [18:43:09] sudo: puppetd: command not found [18:44:33] milimetric: puppet agent -tv [18:44:45] yep, you've been upgraded to puppet 3 [18:44:46] it's puppet3 that changed syntax [18:44:49] puppetd is now 'puppet agent' [18:44:54] AH! [18:44:56] :D [18:45:12] that's funny that must have JUST changed [18:47:40] thanks! [20:33:13] ori, have a moment to look at a vagrant issue? Or, possibly, a mediawiki issue that's invisible except in vagrant? [20:33:34] andrewbogott: probably not, they're debugging a cluster outage right now [20:33:43] YuviPanda: Ah, thought that was settled. [20:33:53] Actually, YuviPanda, do you want to try? Just to verify what I'm seeing? [20:33:53] andrewbogott: nope, postmoderm / digging through logs, etc [20:33:59] andrewbogott: sure [20:34:24] In vagrant (either labs or local, either way) install an extension that includes a special page, and visit that special page. [20:34:33] For example, Echo and Special:Notifications [20:34:48] andrewbogott: I just visited Special:UrlShortener (from role urlshortener) [20:35:08] * andrewbogott tries [20:35:14] andrewbogott: seems ok [20:35:26] andrewbogott: this is local vagrant, but sugarfrosties.wmflabs.org/wiki/Special:UrlShortener is labs-vagrant [20:35:36] can yo do a fetch and rebase in your mediawiki dir? [20:35:42] andrewbogott: sure. [20:35:56] andrewbogott: doing a fetch and reset --hard now... [20:36:39] andrewbogott: done, still works ok [20:36:46] Ah, interesting, it works for me too! [20:36:48] Mind installing echo? [20:36:53] andrewbogott: not at all. moment [20:36:54] meanwhile, I will compare the two... [20:37:03] andrewbogott: provisioning now [20:38:17] something happened with uploaded images on wikitech [20:38:35] mutante: link? [20:38:42] since before a certain time , they all became uploaded by 127.0.0.1 [20:38:50] so like my own uploads are not in my history [20:39:12] mutante: probably because I migrated from virt0 to virt1000? [20:39:18] I thought I preserved the history, but… seems not :( [20:40:09] https://wikitech.wikimedia.org/wiki/Special:ListFiles it changes on Nov 20 2012 [20:40:15] before that date they are [20:40:28] (recovered file, missing upload log entry) [20:40:35] is that the day of eqiad switch? [20:41:06] They're marked (recovered file, missing upload log entry) [20:41:09] that seems clear enough... [20:41:14] no, the switch was early 2014 [20:41:30] andrewbogott: http://127.0.0.1:8080/wiki/Special:Notifications shows me that I've no notifications [20:41:32] But, anyway… looks like whatever went wrong is no longer going wrong, yes? [20:41:37] eh, yea, besides if i go to my own contributions the files look like they disappeared [20:41:49] YuviPanda: but it doesn't spew a thousand error messages? [20:41:55] andrewbogott: nope [20:42:00] Like, um, this does? http://wikitech-test.wmflabs.org/wiki/Special:NovaProject [20:42:05] Because I get that same behavior on my local vagrant. [20:42:19] andrewbogott: right, but that's a Nove issue [20:42:25] No [20:42:28] It happens with echo [20:42:30] on local vagrant [20:42:35] which I can't show you because it's local. [20:42:41] Notice: Did not find alias for special page 'NovaProject'. Perhaps no aliases are defined for it? [Called from SpecialPageFactory::getLocalNameFor in /mnt/vagrant/mediawiki/includes/specialpage/SpecialPageFactory.php at line 592] in /mnt/vagrant/mediawiki/includes/debug/Debug.php on line 303 [20:42:46] andrewbogott: ^ that's what I see [20:43:14] Yes, but that alias problem is not local to OpenStackManager. [20:43:20] It happens to me for most special pages. [20:43:34] andrewbogott: but does it always point to NovaProject? [20:43:40] no. [20:44:14] Well, hm, on my local box I'm seeing Found alias defined for Userlogin when searching for special page aliases for UserLogin. Case mismatch? [20:44:18] andrewbogott: hmm. I also see no aliases for NovaProject at https://github.com/wikimedia/mediawiki-extensions-OpenStackManager/blob/master/OpenStackManager.alias.php [20:44:21] which is weird, not in reference to the page I'm viewing [20:44:36] YuviPanda: true, but adding one does not help :( [20:46:32] YuviPanda: I think that https://gerrit.wikimedia.org/r/#/c/59522/ is causing a million new warnings to print in vagrant. [20:46:37] But if not for you then… I'm baffled. [20:46:50] I suspect that that's catching and warning about cases that happened before without comment... [20:47:10] But also, there's the thing where adding an alias doesn't help :( Want to try? [20:47:14] andrewbogott: hmm, what if you revert just that change? [20:47:27] andrewbogott: probably, but not today :( 2:20 AM and surprisingly knackered... [20:47:50] andrewbogott: did you put up the wikitech role somewhere? [20:48:08] Hm good point, reverting that patch I still gets me ugly warnings [20:48:22] YuviPanda: I will, as soon as I convince my VM to talk to gerrit [20:48:31] andrewbogott: hah :) [20:48:44] key forwarding, usually. this is why I try to not develop directly on the machines... [20:55:20] andrewbogott: I'm heading off now. do email me if anything new comes up? [20:55:25] andrewbogott: and thanks for working on the wikitech role! :D [20:55:45] g'night! [21:03:15] <`fox`> hello, i have an issue with labs migrations. I would need libGeoIP installed for a project (it was installed on toolserver). As toolserver is shutting down very soon I would like to have that library installed as soon as possible [21:03:39] <`fox`> I opened a bug some months ago but nobody is in charge of it https://bugzilla.wikimedia.org/show_bug.cgi?id=62649 [21:04:16] `fox`: adding a patch now [21:05:12] <`fox`> YuviPanda, thank you very much! :D [21:05:46] <`fox`> please let me know when the patch will be up and running :) [21:06:26] `fox`: someone else will have to merge it, though. [21:08:18] `fox`: https://gerrit.wikimedia.org/r/140820 is the patch [21:08:33] andrewbogott: ^ trivial patch, blocking some toolserver migrations. merge when you have time [21:09:11] YuviPanda: can you add the bug # in comments? [21:09:13] Also, go to sleep! [21:09:38] andrewbogott: I did, I think? [21:09:55] andrewbogott: yeah, there's a bug # [21:10:00] andrewbogott: oh, wait, in comments you mean [21:10:01] andrewbogott: usre [21:10:02] <`fox`> YuviPanda, got it! oh there's even python-geoip :) even better than toolserver [21:10:13] YuviPanda: yes please :) [21:11:28] andrewbogott: updated [21:11:37] `fox`: :) [21:11:57] thanks [21:12:08] andrewbogott: :D [21:12:12] `fox`: has been merged, packages should be available in about 30minutes [21:13:14] <`fox`> YuviPanda, super efficient :) thanks! [21:13:25] <`fox`> thanks andrewbogott [21:13:30] `fox`: yw :) can you close the bug once you've verified it? [21:13:35] np, sorry we missed your bug [21:13:45] <`fox`> YuviPanda, sure [21:14:22] `fox`: also, file bugs under 'tools', rather than infrastructure :) that's why I didn't see it [21:14:35] 3Wikimedia Labs / 3tools: Install libGeoIP - 10https://bugzilla.wikimedia.org/62649 (10Yuvi Panda) a:3Marc A. Pelletier [21:15:09] `fox`: see, I re-assigned and now it shows up here. makes it easier to notice :D [21:15:14] for next time! [21:15:48] * YuviPanda|zzz goes to sleep [21:15:49] cya [21:15:50] guys [22:40:28] !log deployment-prep Deleted /var/log/diamond/diamond.log on deployment-jobrunner01 because /var was full [22:40:31] Logged the message, Master [22:46:57] !log deployment-prep Trebuchet runs on deployment-videoscaler01 are succeeding but not showing up in the `git deploy report` output [22:46:59] Logged the message, Master [22:47:18] !log deployment-prep Updated scap to 792a572 [22:47:20] Logged the message, Master [23:03:58] Betacommand: do you have a moment? [23:11:41] Anyone else, I'm having an issue with my tool at http://tools.wmflabs.org/mono/ [23:12:08] The error is "No webservice" but when I SSH in, it says the webservice is running