[00:06:52] 3Tool Labs tools / 3X!'s tools: Xtools offline - 10https://bugzilla.wikimedia.org/72104#c9 (10TParis) (In reply to Diego Queiroz from comment #3) > https://tools.wmflabs.org/supercount seems to not be working as well. In > respect to your Wikibreak, is there someone else working on this tool? > > I've cont... [00:24:52] 3Tool Labs tools / 3X!'s tools: Xtools offline - 10https://bugzilla.wikimedia.org/72104#c10 (10Daniel Zahn) fwiw, yesterday i also checked this and it seemed to me it was really just xtools at that time because other tools appeared to be working [00:30:55] 3Wikimedia Labs / 3tools: logging table appears to have different indexes from what vanilla MW does (e.g no type_action index) - 10https://bugzilla.wikimedia.org/72167 (10Bawolff (Brian Wolff)) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier I was looking at https://tools.wmflabs.org/tools-info/optimiz... [04:32:16] umm [04:32:20] how do I disable bigbrother [04:32:28] like I deleted .bigbrotherrc but it's still doing its thing [04:34:31] (poke for Coren) [05:12:09] 3Wikimedia Labs / 3tools: WMFLabs: Gzip text/javascript responses from fcgi (just like text/html) - 10https://bugzilla.wikimedia.org/71995 (10Tim Landscheidt) a:5Marc A. Pelletier>3Yuvi Panda [08:16:39] 3Wikimedia Labs / 3tools: Queue is mostly dead, which is slightly alive. - 10https://bugzilla.wikimedia.org/67347#c6 (10Yuvi Panda) 5UNCO>3RESO/FIX They seem ok now, and diamond no longer fills up /var... [08:19:07] 3Wikimedia Labs / 3(other): role::mediawiki-install::labs can't run phpunit tests - 10https://bugzilla.wikimedia.org/41797#c2 (10Yuvi Panda) 5NEW>3RESO/WON I'm going to mark this WONTFIX, please use labs-vagrant. [08:20:37] 3Wikimedia Labs / 3Infrastructure: autostart / puppetize snmtrapd on labs nagios - 10https://bugzilla.wikimedia.org/36470#c3 (10Yuvi Panda) 5NEW>3RESO/WON Yup, we won't be using that. Even prod's puppet freshness doesn't use snmp traps anymore. [08:20:56] 3Wikimedia Labs / 3deployment-prep (beta): Diamond logstash monitor fills /var/log/apache2 access log - 10https://bugzilla.wikimedia.org/72175 (10Antoine "hashar" Musso (WMF)) 3NEW p:3Unprio s:3normal a:3None deployment-logstash1.eqiad.wmflabs ended up with a filled /var/ /var/log/apache2/other_vhos... [08:22:07] 3Wikimedia Labs / 3deployment-prep (beta): Automatic mobile redirects for MobileFrontend on Betalabs - 10https://bugzilla.wikimedia.org/46620 (10Yuvi Panda) [08:29:52] 3Wikimedia Labs / 3deployment-prep (beta): Diamond logstash monitor fills /var/log/apache2 access log - 10https://bugzilla.wikimedia.org/72175#c1 (10Antoine "hashar" Musso (WMF)) + Yuvi Panda: seems the diamond collector for logstash spams the apache access log with garbage requests such as ""\x16\x03\x01"... [08:47:00] (03CR) 10Gilles: "How can I find out who has +2 on this repo?" [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/165438 (owner: 10Gilles) [08:57:07] gi11es: https://www.mediawiki.org/wiki/wikibugs?redirect=no :) [08:57:18] gi11es: also https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL for people who have deployed in the past [08:57:27] (03CR) 10Yuvipanda: [C: 032 V: 032] Add ImageMetrics to the -multimedia channel [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/165438 (owner: 10Gilles) [09:03:38] !log tools.wikibugs deployed to get {{gerrit|165438}} in [09:03:48] gi11es: I deployed, do test [09:03:57] YuviPanda: thanks [09:04:03] gi11es: yw! [09:05:03] YuviPanda: works [09:14:37] 3Wikimedia Labs / 3deployment-prep (beta): Diamond logstash monitor fills /var/log/apache2 access log - 10https://bugzilla.wikimedia.org/72175#c2 (10Antoine "hashar" Musso (WMF)) That might be due to monitor fatal using an HTTPS connection: modules/beta/files/monitor_fatals.rb:27:https://logstash-beta.wmfla... [09:17:22] 3Wikimedia Labs / 3Infrastructure: Allow login using mosh as an alternative to plain ssh on bastion - 10https://bugzilla.wikimedia.org/52693#c16 (10Yuvi Panda) Still not merged, so we can't really do much. [09:17:53] 3Wikimedia Labs / 3Infrastructure: Allow login using mosh as an alternative to plain ssh on bastion - 10https://bugzilla.wikimedia.org/52693 (10Yuvi Panda) [09:18:26] 3Wikimedia Labs / 3tools: qa-morebots log message bot is no more in #wikimedia-qa - 10https://bugzilla.wikimedia.org/72179 (10Antoine "hashar" Musso (WMF)) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier #wikimedia-qa usually have a 'qa-morebots' bot to update https://wikitech.wikimedia.org/wiki/Releas... [09:18:37] 3Wikimedia Labs / 3tools: Install ogc on tools - 10https://bugzilla.wikimedia.org/60449#c2 (10Yuvi Panda) 5ASSI>3RESO/WON npm modules should be installed locally (with npm install), not globally. [09:19:37] 3Wikimedia Labs: /srv won't mount on new 'large' instance - 10https://bugzilla.wikimedia.org/69161#c3 (10Yuvi Panda) 5NEW>3RESO/FIX Seems to be fixed now... [09:20:06] YuviPanda: do you have morebots access by any chance ? qa-morebots disappeared from #wikimedia-qa [09:20:33] hashar: I don't, but I've root! [09:21:50] YuviPanda: don't waste too much time on it though :d [09:21:56] i filled a bug for it already https://bugzilla.wikimedia.org/72179 [09:21:56] :) ok [09:22:04] hashar: I can just give you access... [09:24:08] 3Wikimedia Labs / 3Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - 10https://bugzilla.wikimedia.org/31828#c15 (10Yuvi Panda) Is this still relevant? I'll echo Ryan's comments about legal approval being required. [09:26:41] (03CR) 10Gergő Tisza: "> How can I find out who has +2 on this repo?" [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/165438 (owner: 10Gilles) [09:34:08] 3Wikimedia Labs / 3tools: Enable OpenJDK 8 - 10https://bugzilla.wikimedia.org/66171#c1 (10Yuvi Panda) https://bugs.launchpad.net/ubuntu/+source/openjdk-8/+bug/1341628 says no openjdk8 package for even the latest LTS, so this is going to be a while :( [09:35:29] I will let you know when I see andrewbogott around here [09:35:29] @notify andrewbogott [10:45:29] I am having issues with using [[File:...]] on wikitech wiki. [10:45:35] The image at the bottom of [10:45:38] https://wikitech.wikimedia.org/w/index.php?title=Analytics/Cluster/Oozie&oldid=131336#Current_submitted_Refinery.27s_Oozie_jobs [10:45:47] rendered for a few seconds (minutes?), and now [10:45:54] it's only a link ... [10:46:23] wikitechwiki and Commons don't get along very well [10:46:50] Previously, one could upload files to wikitechwiki. Now one has to use commons. [10:47:08] And lately we have many reports of remote repo being unstable https://lists.wikimedia.org/pipermail/mediawiki-l/2014-October/043445.html [10:47:09] Any idea how could I use images on wikitechwiki then? [10:47:16] Ok. [10:47:18] Reading thi therad. [10:47:27] .... Reading the thread :-) [11:10:37] 3Wikimedia Labs / 3tools: Enable OpenJDK 8 - 10https://bugzilla.wikimedia.org/66171 (10Andre Klapper) p:5Unprio>3Lowest [11:42:15] Earwig|away: bigbrother only checks for .bigbrotherrc every other minute or so; is it still being stubborn? [11:50:31] PROBLEM - ToolLabs: Puppet freshness check on labmon1001 is CRITICAL: CRITICAL: tools.tools-mail.puppetagent.time_since_last_run.value (10.00%) WARN: tools.tools-webgrid-tomcat.puppetagent.time_since_last_run.value (100.00%) tools.tools-submit.puppetagent.time_since_last_run.value (100.00%) tools.tools-exec-12.puppetagent.time_since_last_run.value (100.00%) tools.tools-exec-13.puppetagent.time_since_last_run.value (100.00%) tools.tools-dev. [11:51:36] Coren: ^ we should probably not disable puppet on all the things. [11:51:53] YuviPanda: Needed for a while still while I beat up puppet. [11:52:11] Coren: why? can't we test on toolsbeta and merge when it's stable? [11:52:41] or do it one patch at a time... [11:53:21] YuviPanda: Perhaps, but given how likely this is to break all of the grid I absolutely want to check with the real config and values. [11:54:07] And toolsbeta has the same general setup but almost none of the exceptions which are the important one to check. :-( [11:54:27] Coren: how would you test it with the real config when puppet is dissolved? [11:54:45] Coren: also, an email to ops@ about this plan would be nice :) I've no idea what's happening. [11:54:51] YuviPanda: By carefully running puppet agent --enable && puppet agent -tv && puppet agent --disable. :-) [11:55:33] 23:39‧ !log tools Puppet disabled for a couple of days while debugging gridengine puppetization. [11:55:41] Though I now realize that this never made it to the SAL [11:56:59] YuviPanda: I felt having puppet disabled was less disruptive than puppet errors on all the nodes. YVVM; the changes-in-progress are specifically nondestructuve though so I *could* enable puppet. [11:57:26] But it's arguable whether puppet not running is worse or better than puppet running and failing. [11:59:02] Coren: heh :) still, 'debugging' isn't detailed enough, I think. why/what, etc would be nice. [11:59:36] Coren: not just for me but for others as well... [12:03:15] Coren: can you also acknowledge the icinga alert? [12:06:30] Yeah, going to ack (well, set downtime). As for why/what: puppetizing the actual gridengine config, which is a "fun" challenge given how unreceptive gridengine is and how amusing puppet can be without resource collectors. [12:06:44] ah, heh. should be fun :) [12:06:50] and yay to puppetizing OGE [12:08:50] Coren: also, what's your TZ? [12:09:12] That was on my long list of "when I got time" but I really really wanted to have that done before we bring codfw up. [12:09:53] Ostensibly UTC-4 but my actual work hours are... not that predictable. I'm generally around between 10h and 17h but also sometiimes before and after. [12:10:11] ah, not bad :) [12:10:37] * Coren returns to his Manifests of Evil. [12:16:19] !ping [12:16:19] !pong [12:17:13] ok [12:22:00] (03CR) 10Yuvipanda: [V: 032] Update build script for Gradle [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/165375 (owner: 10BearND) [12:22:05] (03CR) 10Yuvipanda: [V: 032] Need JAVA_HOME for Gradle [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/165376 (owner: 10BearND) [12:22:12] (03CR) 10Yuvipanda: [V: 032] Expand wildcards when copying apk [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/165377 (owner: 10BearND) [12:22:17] (03CR) 10Yuvipanda: [V: 032] Output the current time stamp to stdout and stderr [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/165520 (owner: 10BearND) [13:43:45] YuviPanda: I'm here, what's up? [14:02:54] Coren: have a minute for an interesting puzzle? (Well, it's interesting to me at least) [14:03:30] Sure. I'm in a coding bit so a distraction might be just what the doctor ordered. :-) [14:04:55] On virt1000, look in /a/backup/public [14:04:58] and then look at root's crontab [14:05:17] labswiki-20141016.xml.gz (big!) was generated by my running the crontab command by hand [14:05:27] labswiki-20141017.xml.gz (tiny) was generated by the actual cron [14:06:27] different dates / different contents ? :D [14:07:07] it should be nearly indentical, it's a dump of the labswiki database [14:07:25] Clearly the cron is failing, but I don't understand how it's different from what happens when I run it by hand [14:08:03] andrewbogott: I don't see any backup-like entry in virt1000's root's crontab. [14:08:18] Oh, /usr/local/sbin/db-bak.sh? [14:08:22] oh, sorry, it's a script. um... [14:08:40] /usr/local/sbin/mw-xml.sh, that's the one [14:11:49] You're invoking 'mwscript' with no explicit path. the cwd isn't in the default $PATH. For that matter, the cwd should never ever be in root's $PATH [14:12:10] Chaning it to ./mwscript will probably make it work. [14:12:19] ah, you're right, that's surely it. [14:12:35] thanks! [14:12:53] no worries. [14:20:11] andrewbogott: https://gerrit.wikimedia.org/r/#/c/166902/ [14:24:58] YuviPanda: does that tox file apply to everything in that dir? [14:25:05] andrewbogott: yup. [14:25:13] ok [14:25:16] It should, I think. 80char limits are stupid [14:30:12] (03PS1) 10Yuvipanda: Add sh.py into the repository to avoid dependency issues [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/167205 [14:39:24] (03PS2) 10Yuvipanda: Add sh.py into the repository to avoid dependency issues [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/167205 [14:39:35] (03CR) 10Yuvipanda: [C: 032] Add sh.py into the repository to avoid dependency issues [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/167205 (owner: 10Yuvipanda) [14:39:46] (03CR) 10Yuvipanda: [V: 032] Add sh.py into the repository to avoid dependency issues [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/167205 (owner: 10Yuvipanda) [14:57:12] (03PS1) 10Yuvipanda: Move everything to be based off /srv [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/167212 [15:15:15] !ping [15:15:15] !pong [15:15:47] are you aware of http://tools.wmflabs.org/catscan2/notice.html? any ideas to fix it? [15:42:42] uurgh... http://tools.wmflabs.org/catscan2/notice.html [16:02:10] 3Wikimedia Labs / 3deployment-prep (beta): Automatic mobile redirects for MobileFrontend on Betalabs - 10https://bugzilla.wikimedia.org/46620#c1 (10Greg Grossmeier) a:5Ryan Lane>3None Resseting assignee (from Ryan Lane). Kaldari: Is this still needed? (this was in the general WMF Labs queue, not Beta Cl... [17:44:58] (03PS2) 10Yuvipanda: Move everything to be based off /srv [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/167212 [17:49:53] (03PS3) 10Yuvipanda: Move everything to be based off /srv [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/167212 [18:04:35] (03PS4) 10Yuvipanda: Move everything to be based off /srv [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/167212 [18:07:47] (03PS5) 10Yuvipanda: Move everything to be based off /srv [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/167212 [18:09:06] (03CR) 10Yuvipanda: [C: 032 V: 032] Move everything to be based off /srv [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/167212 (owner: 10Yuvipanda) [20:37:37] 3Wikimedia Labs / 3tools: qa-morebots log message bot is no more in #wikimedia-qa - 10https://bugzilla.wikimedia.org/72179#c1 (10Antoine "hashar" Musso (WMF)) Poked labs-l about it https://lists.wikimedia.org/pipermail/labs-l/2014-October/003015.html [20:42:07] 3Wikimedia Labs / 3tools: qa-morebots log message bot is no more in #wikimedia-qa - 10https://bugzilla.wikimedia.org/72179#c2 (10Andrew Bogott) 5NEW>3RESO/FIX This is fixed now. Our code still doesn't handle netsplits perfectly. If you like, I can add you to the 'morebots' tool so that you can restart... [20:50:22] 3Wikimedia Labs / 3tools: qa-morebots log message bot is no more in #wikimedia-qa - 10https://bugzilla.wikimedia.org/72179 (10Antoine "hashar" Musso (WMF)) 5RESO/?>3VERI