[00:41:05] !log deployment-prep cherry-picked scap change https://gerrit.wikimedia.org/r/#/c/155677/ for testing [00:41:07] Logged the message, Master [01:22:59] YuviPanda: how to exclude one specific repo from the operations channel in grrrit-wm [01:23:14] some kind of negation ? [02:16:30] 3Wikimedia Labs / 3wikistats: Make a stats table for 85 W3C wikis - 10https://bugzilla.wikimedia.org/41023#c10 (10Daniel Zahn) https://gerrit.wikimedia.org/r/#/c/155680/ [05:49:29] 3Wikimedia Labs / 3deployment-prep (beta): obtain access to deployment-bastion.eqiad.wmflabs - 10https://bugzilla.wikimedia.org/69435#c1 (10jeremyb) gwicke and thedj already have access. In gwicke can resolve this himself. :) [05:50:17] 3Wikimedia Labs / 3deployment-prep (beta): obtain access to deployment-bastion.eqiad.wmflabs - 10https://bugzilla.wikimedia.org/69435 (10jeremyb) a:3Gabriel Wicke [11:59:18] 3Wikimedia Labs / 3deployment-prep (beta): Logging out of Commons beta: Cannot contact the database server: Unknown database 'incubatorwiki' - 10https://bugzilla.wikimedia.org/69898 (10Andre Klapper) 3NEW p:3Unprio s:3normal a:3None [Splitting this off from bug 69096] Logging out of http://commons.w... [11:59:59] 3Wikimedia Labs / 3deployment-prep (beta): Cannot log in to Beta commons: Infinite redirect - 10https://bugzilla.wikimedia.org/69096#c7 (10Andre Klapper) 5NEW>3RESO/WOR No reply; closing as WORKSFORME. Moved issue in comment 5 to bug 69898. [12:47:57] Hi again. Nearly got the lalm to work, see http://tools.wmflabs.org/lalm/. Any idea why http://tools.wmflabs.org/lalm/navigate/WAY/16765070 seems not to be handled by tomcat at all? [12:48:36] wait... it is handled by tomcat, but returns a 500... where do I see more details about the error? [14:20:54] (03PS1) 10Yuvipanda: Port code to python [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155717 [14:21:09] (03CR) 10Yuvipanda: [C: 032] Port code to python [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155717 (owner: 10Yuvipanda) [14:21:16] (03CR) 10Yuvipanda: [V: 032] Port code to python [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155717 (owner: 10Yuvipanda) [14:22:04] (03PS1) 10Yuvipanda: Actually add the python build script [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155718 [14:22:20] (03CR) 10Yuvipanda: [C: 032 V: 032] Actually add the python build script [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155718 (owner: 10Yuvipanda) [14:31:54] YuviPanda: I guess labs/tools/wikipedia-android-builds.git needs some jenkins jobs :D [14:34:07] hashar: yeaaah :) [14:34:09] python [14:34:43] tox!!!!!!!!!!!!!!! [14:35:06] which make the CI configuration fairly trivial ( https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Test_your_python ) [14:36:54] (03PS1) 10Yuvipanda: Add minor debug logging [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155722 [14:36:56] (03PS1) 10Yuvipanda: Symlink latest build directory to 'latest' [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155723 [14:37:04] hashar: yeah, will do when I'm done with it [14:40:24] (03PS2) 10Yuvipanda: Add minor debug logging [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155722 [14:40:26] (03PS2) 10Yuvipanda: Symlink latest build directory to 'latest' [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155723 [14:57:36] (03PS1) 10Yuvipanda: Write out meta.json with metadata about each build [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155730 [14:59:45] (03PS2) 10Yuvipanda: Write out meta.json with metadata about each build [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155730 [15:01:50] (03PS3) 10Yuvipanda: Write out meta.json with metadata about each build [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155730 [15:18:26] (03PS4) 10Yuvipanda: Write out meta.json with metadata about each build [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155730 [15:26:01] (03CR) 10Dbrant: [C: 032 V: 032] Add minor debug logging [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155722 (owner: 10Yuvipanda) [15:30:44] (03CR) 10BearND: [C: 032] Symlink latest build directory to 'latest' [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155723 (owner: 10Yuvipanda) [15:33:13] (03CR) 10BearND: [C: 032] Write out meta.json with metadata about each build [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155730 (owner: 10Yuvipanda) [15:49:48] (03PS1) 10Yuvipanda: [WIP] Make the package name be .master [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155742 [15:55:15] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c4 (10Aude) broke again... Fatal error: Argument 3 passed to Wikibase\\EntityContentFactory::getPermissionStatusForEntityId() must be an instance of Wikibase\\Ent... [16:02:32] (03PS2) 10Yuvipanda: Make the package name be .master [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155742 [16:03:12] (03CR) 10Yuvipanda: [C: 032] Make the package name be .master [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155742 (owner: 10Yuvipanda) [16:03:32] (03CR) 10Yuvipanda: [V: 032] Symlink latest build directory to 'latest' [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155723 (owner: 10Yuvipanda) [16:03:43] (03CR) 10Yuvipanda: [V: 032] Write out meta.json with metadata about each build [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155730 (owner: 10Yuvipanda) [16:03:51] (03CR) 10Yuvipanda: [V: 032] Make the package name be .master [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155742 (owner: 10Yuvipanda) [16:05:58] legoktm: valhallasw`cloud how do I silence cron from sending me stupid 'your job has been submitted' emails? [16:06:06] -quiet I think [16:07:33] legoktm: ty [16:08:22] (03PS1) 10Yuvipanda: Reduce cronspam [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155746 [16:09:01] (03CR) 10Legoktm: [C: 031] Reduce cronspam [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155746 (owner: 10Yuvipanda) [16:09:59] (03CR) 10Yuvipanda: [C: 032 V: 032] Reduce cronspam [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155746 (owner: 10Yuvipanda) [16:23:05] YuviPanda: or >/dev/nul [16:23:06] L [17:51:11] *sigh* Looks like the new dump server is finally working right. Sorry about the delay with bz & gerrit; I really wanted to get that [bleep] thing working. [17:58:55] * valhallasw`cloud cheers Coren on [17:58:58] whoooooooo! [18:03:48] * Coren does not yet count it's chicken. [18:03:59] his* chickens* [18:15:10] (03PS1) 10Yuvipanda: Add status page that's static HTML + JS [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155767 [18:17:11] (03PS2) 10Yuvipanda: Add status page that's static HTML + JS [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155767 [18:27:46] 3Wikimedia Labs / 3deployment-prep (beta): HTML Tags are stripped from extension output - 10https://bugzilla.wikimedia.org/66516#c8 (10physikerwelt) 5PATC>3RESO/FIX Fixed. See http://deployment.wikimedia.beta.wmflabs.org/wiki/User:Anomie/Sandbox [18:30:31] (03PS3) 10Yuvipanda: Add status page that's static HTML + JS [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155767 [18:36:39] (03PS4) 10Yuvipanda: Add status page that's static HTML + JS [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155767 [18:37:38] (03PS5) 10Yuvipanda: Add status page that's static HTML + JS [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155767 [18:38:34] (03CR) 10Yuvipanda: [C: 032 V: 032] Add status page that's static HTML + JS [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155767 (owner: 10Yuvipanda) [18:39:39] bd808|BUFFER, Coren: Beta labs seems to be broken again: http://en.m.wikipedia.beta.wmflabs.org/wiki/Main_Page [18:40:10] nevermind, now it's working again :P [18:40:25] was getting 503 errors from bits [18:40:40] bits.beta.wmflabs I mean [18:42:31] (03PS1) 10Yuvipanda: Don't specify extra bootstrap theme [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155775 [18:44:38] (03CR) 10Yuvipanda: [C: 032] Don't specify extra bootstrap theme [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155775 (owner: 10Yuvipanda) [18:52:16] (03PS1) 10Yuvipanda: SLightly more human readable date formatting [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155777 [18:55:51] (03PS2) 10Yuvipanda: Slightly more human readable date formatting [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155777 [18:57:20] (03CR) 10Yuvipanda: [V: 032] Don't specify extra bootstrap theme [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155775 (owner: 10Yuvipanda) [18:57:33] (03CR) 10Yuvipanda: [C: 032 V: 032] Slightly more human readable date formatting [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155777 (owner: 10Yuvipanda) [19:13:37] guys [19:13:43] i think i thrashed the server [19:15:10] now i cant login to kill the script [19:19:14] Aviator_: which server crashed? [19:19:25] tools-login [19:20:27] Aviator_: still works for me [19:20:31] i'm on it now [19:20:46] thats weird [19:20:48] looks like it's working fine [19:20:50] afaict [19:20:57] why is it timing out on me [19:21:54] Aviator_: i also see other users logging in.. networking issue somewhere in between? [19:24:22] ok thats a relief [19:24:34] otoh i may have to reconnect [19:53:41] Are people aware that bits.beta.wmflabs.org is down? [19:53:51] E.g. http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load.php?debug=true&lang=en&modules=ext.visualEditor.viewPageTarget.noscript&only=styles&skin=vector&* [19:54:02] So no JSS/CSS resources are being served. [19:58:32] 3Wikimedia Labs / 3deployment-prep (beta): bits.beta.wmflabs.org down with 503 - 10https://bugzilla.wikimedia.org/69921 (10Matthew Flaschen) 3NEW p:3Unprio s:3normal a:3None http://bits.beta.wmflabs.org is entirely down with 503. E.g.: http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load... [19:58:59] ^ hashar [19:59:05] !log deployment-prep bits yielding 503 [19:59:07] Logged the message, Master [19:59:16] !log deployment-prep restarting udp2log-mw service on deployment-bastion [19:59:18] Logged the message, Master [19:59:44] superm401: yeah no clue what is happening [19:59:54] hashar, thanks for looking into it, though. [20:00:15] oh I am not [20:00:34] hashar, heh. Well restarting it is more than I did. [20:00:38] I know it's really late there. [20:00:50] everything is terribly broken :/ [20:00:56] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: $service_ips["hhvm_appservers"] is :undef, not a hash or array at /etc/puppet/modules/lvs/manifests/configuration.pp:626 on node i-0000007f.eqiad.wmflabs [20:00:58] that is on varnish [20:02:03] I asked ori to come by. [20:02:45] hey [20:02:45] is the issue that bits is down? [20:02:58] yeah yielding 503 [20:03:10] i've seen this before, the console on wikitech showed the OOM killer going crazy [20:03:11] let's see [20:03:16] and puppet is broken on varnish cache with : Error 400 on SERVER: $service_ips["hhvm_appservers"] is :undef, not a hash or array at /etc/puppet/modules/lvs/manifests/configuration.pp:626 on node i-0000007f.eqiad.wmflabs [20:03:26] due to some lvs service ip being added apparently https://gerrit.wikimedia.org/r/#/c/152908/5/manifests/role/lvs.pp [20:03:56] ah, right [20:03:57] let me see if i can fix [20:04:51] hashar: how has this been handled before? [20:04:54] by adding it for labs as well? [20:05:14] the prod manifests for role::cache::configuration refers to LVS services_ip in production [20:05:24] but on labs / beta we fill the app server IPs directly [20:05:39] I guess some hash is just missing the hhvm_appservers entry under labs [20:06:31] na it is around [20:06:43] maybe puppet master is out of date :D [20:07:50] yeah, i see it too [20:08:24] cache.pp line 429 [20:08:55] !log deployment-prep ran 'git pull' on deployment-salt:/srv/var-lib/git/operations/puppet [20:08:57] Logged the message, Master [20:10:02] oh, i see what's wrong [20:10:09] patch incoming [20:14:43] (03PS1) 10Yuvipanda: Use .alpha and not .master [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155801 [20:15:45] whoooooooo! [20:15:47] err [20:15:52] that was not the right screen [20:15:59] still, whoooooooo! is always in order [20:21:17] !log deployment-prep udp2log are back in /data/project/logs . The udp2log-mw service went stall for some reason. [20:21:19] Logged the message, Master [20:26:27] YuviPanda: is there something in grrrit-wm1 to exclude/negate things? [20:26:31] ori: seems something got solved [20:26:48] YuviPanda: like if it currently says "operations/*" and then i want an exempt from that rule [20:27:55] hashar: yes; https://gerrit.wikimedia.org/r/155806 [20:28:14] ori: you can reference Bug: 69921 [20:28:14] :-D [20:28:17] https://bugzilla.wikimedia.org/show_bug.cgi?id=69921 [20:28:32] sorry about that, that's my oversight. [20:28:56] the thing I dont understand [20:29:12] is that the patch that introduced hhvm_appservers was mentioning it has been tested on beta :D [20:29:17] can't wait for hiera [20:29:40] also ideally, on beta we should never have to reference lvs::configuration [20:29:47] since there is no LVS support on labs [20:29:53] +1 ori thank you verymuch [20:30:03] 13 'labs' => $::site ? { [20:30:03] 14 'pmtpa' => [ "i-00000051" ], [20:30:03] 15 default => undef, [20:30:04] superm401: ori fixed bits!! [20:30:09] sounds like that part should get eqiad [20:30:27] superm401: (also: ori broke bits!) :/ [20:30:35] shit happens :D [20:30:41] Thanks for the quick fix, ori. [20:30:51] that is from lvs::config, so does that mean it once ha LVS support for labs, but not anymore? [20:30:52] ori, you are entitled to update the #wikimedia-labs topic in triumph. [20:31:02] ori: also I noticed earlier in the week that apache on mediawiki{01,02} write some debug log to /var/log/apache2/ [20:31:35] ori: on labs /var/ is only a 2GB partition so that debug log ended up filling the disk. Would be better to send them to /data/project/logs/apache-debug.log or something [20:31:44] or over syslog [20:31:47] !instance i-00000051 [20:31:47] need help? -> https://labsconsole.wikimedia.org/wiki/Help:Instances want to manage? -> https://labsconsole.wikimedia.org/wiki/Special:NovaInstance want resources? use !resource [20:31:55] !resource i-00000051 [20:31:55] https://labsconsole.wikimedia.org/wiki/Nova_Resource:i-00000051 [20:32:10] This page has been deleted. [20:32:11] heh [20:32:20] lol [20:32:23] drool [20:32:24] or whatever [20:32:38] hashar: ah, there's an easy way to do that [20:32:53] declare an rsyslog::conf resource for labs with a lower priority than mediawiki's [20:33:03] if we had a new instance name in eqiad, we could add that to config [20:33:04] and have a filter rule that ends with '~', which means "don't apply further rules" [20:33:18] hashar: i can submit a patch for that if you like [20:33:38] every time someone says "there is an easy way to do that" I think about Fermat's Last Theorem ( http://en.wikipedia.org/wiki/Fermat%27s_Last_Theorem ) [20:33:59] heh [20:34:01] margin too narrow [20:34:11] a french mathematician that wrote a theorem, and next to it stated "I have a proof for it but it is too large to fit in the margin" [20:34:16] took 350 years to solve :D [20:34:18] lol [20:34:54] hashar: To be fair, the consensus now is that Fermat couldn't possibly have had a correct proof, though he may have thought of a flawed one. [20:35:05] hashar: I still think that was an Epic nerd snipe. :-) [20:35:06] Wiles proof can't really be understood by common mortals :-/ [20:35:25] Coren: yeah he probably had a wrong proof [20:35:44] Coren: I am sure the proof we have right now is over complicated and that one day we will find a simpler one [20:38:51] mutante: yeah, mostly just special cased :) I remove 'mediawiki/*' as a prefix, can add operations/* [20:42:54] bla bla re computer shut off [20:54:23] (03CR) 10Yuvipanda: [C: 032 V: 032] Use .alpha and not .master [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/155801 (owner: 10Yuvipanda) [21:06:04] hey, back in the past wm-bot could read RSS feeds [21:06:08] then at some point it broke [21:06:17] any newer bots that have RSS features yet? [21:07:03] well, i suppose i could use the same we use on irc.wikimedia [21:07:17] but it'd be RSS feeds from planet , not mw [21:34:10] !log limn Syntax error at '{'; expected [21:34:10] limn is not a valid project. [21:34:32] !resource limn1 [21:34:33] https://labsconsole.wikimedia.org/wiki/Nova_Resource:limn1 [21:35:02] Error 400 on SERVER: Could not parse for environment production: Syntax error at '{'; expected [21:35:05] '}' at /etc/puppet/manifests/role/mariadb.pp:86 on node limn1.eqiad.wmflabs [23:55:20] 3Wikimedia Labs / 3tools: Install package: python-pygments (syntax highlighting) - 10https://bugzilla.wikimedia.org/69050#c3 (10Daniel Zahn) 5PATC>3RESO/FIX Aug 22 23:52:57 tools-login puppet-agent[10419]: (/Stage[main]/Toollabs::Exec_environ/Package[python-pygments]/ensure) ensure changed 'purged' to 'l...