[02:08:28] Krinkle, did you do something with stream.wmflabs.org? [02:08:56] it's no longer associated with the labs instance [04:20:20] 10Tool-Labs-tools-wikiloves: Desenvolver aplicativo básico em Flask para a ferramenta - https://phabricator.wikimedia.org/T129712#2125248 (10Danilo) Iniciei o código baseado no ptwikis, [[ https://github.com/ptwikis/wikiloves | coloquei no github ]], clonei no Tool Labs e coloquei para funcionar: http://tools.wm... [06:51:06] PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:29:11] RECOVERY - Puppet run on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:37:02] does 'configuring' an instance on wikitech work for you? my POST never returns from Special:NovaInstance [10:26:06] 10Tool-Labs-tools-Other: Crash of svgtranslate - https://phabricator.wikimedia.org/T118146#2125667 (10Andrei_Stroe) Here is the error more properly formatted: ``` Notice: Undefined offset: 0 in /data/project/jarry-common/public_html/peachy/Includes/Image.php on line 181 Call Stack: 0.0093 668816 1. {m... [11:35:56] 6Labs: unable to change puppet groups via Special:NovaInstace - https://phabricator.wikimedia.org/T130104#2125830 (10fgiunchedi) [11:36:11] 6Labs: unable to change puppet groups via Special:NovaInstace - https://phabricator.wikimedia.org/T130104#2125843 (10fgiunchedi) p:5Triage>3High [11:36:24] reported as ^ [12:51:57] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2125964 (10Shizhao) I added {{Graph:PageViews}} to MediaWiki:Pageinfo-footer on Zh WP, see https://zh.wikipedia.org/wiki/MediaWiki:Pageinfo-footer Example: http... [14:22:51] 6Labs, 6Operations: overhaul labstore setup [tracking] - https://phabricator.wikimedia.org/T126083#2126125 (10Papaul) [14:22:54] 6Labs, 6Operations, 10ops-codfw: Figure out what labstore hardware is viable in codfw - https://phabricator.wikimedia.org/T128083#2126121 (10Papaul) 5Open>3Resolved Closing this since this is resolved in T128764 [14:56:49] 6Labs: Enforce true multi-tenancy for labs public DNS - https://phabricator.wikimedia.org/T130032#2126208 (10Andrew) [14:59:09] andrewbogott: (hint) [ ] and [x] in phab markup render in nice html checkboxes [14:59:41] paravoid: I know, it started out as a paste from a generated report [14:59:52] ok :) [15:07:57] Is there anyone around who works on the ‘megacron’ project? Or even anyone who knows what it is? [15:12:04] andrewbogott: I would blame Coren https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Megacron&diff=102661&oldid=102659 :D [15:12:25] but maybe he just added himself as a user before adding other people [15:12:36] andrewbogott: You know, it's obsolete by now and can be explodinated. [15:12:39] I think that just means he created the project [15:12:57] Coren: how confident are you of that? [15:13:07] I emailed all the (non-you) admins yesterday, no responses [15:13:16] andrewbogott: Very. This was a Facebook open source Academy project. [15:13:23] fantastic [15:13:25] * andrewbogott deletes [15:13:28] thanks! [15:13:35] mass cleanup going on ? ;) [15:13:47] but first I’m going to shout “MEGACRON!” a few times [15:13:57] hashar: it’s this: https://phabricator.wikimedia.org/T130032 [15:14:23] | table formatting | you should use | :-} [15:14:35] omg, everyone has issues with my internal note-taking :) [15:14:52] !log megacron deleting instances and project [15:14:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Megacron/SAL, dummy [15:15:19] @seen [15:22:00] 6Labs: Enforce true multi-tenancy for labs public DNS - https://phabricator.wikimedia.org/T130032#2126285 (10Andrew) [15:37:49] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2126355 (10Yurik) @Shizhao , please update your example link above - if someon's user interface is not "zh", they won't see it. The link should have `&uselang=zh... [15:46:29] andrewbogott: btw if you want to take a break from spring cleaning, the hanging POST to novainstance is back :( https://phabricator.wikimedia.org/T130104 [15:46:39] don't know how widespread it is though [15:59:31] (03PS2) 10MarcoAurelio: Centralize db credentials config file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276205 (owner: 10Glaisher) [16:01:29] (03CR) 10MarcoAurelio: [C: 032] Centralize db credentials config file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276205 (owner: 10Glaisher) [16:03:39] !log tools.stewardbots Merging gerrit change #{{gerrit|276205}} [16:03:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [16:07:31] (03Merged) 10jenkins-bot: Centralize db credentials config file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276205 (owner: 10Glaisher) [16:16:12] (03PS1) 10Glaisher: Cleanup CSS and JS [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277789 [16:26:23] (03PS2) 10Glaisher: Cleanup CSS and JS [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277789 [16:39:58] (03PS1) 10Youni Verciti: Rev 0.2 gnu-gpl defined [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/277797 [16:48:38] 6Labs: unable to change puppet groups via Special:NovaInstace - https://phabricator.wikimedia.org/T130104#2125830 (10Dzahn) I tested this by adding myself to the same project ("monitoring"), making myself project admin, clicking configure on that same instance "filippo-test-jessie2". I could remove the role, sa... [16:50:16] (03PS1) 10Glaisher: Fix replica.my.cnf file directory [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277802 [16:52:57] (03CR) 10MarcoAurelio: [C: 032] Fix replica.my.cnf file directory [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277802 (owner: 10Glaisher) [16:55:36] (03CR) 10MarcoAurelio: [C: 032] Cleanup CSS and JS [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277789 (owner: 10Glaisher) [17:08:03] (03Merged) 10jenkins-bot: Cleanup CSS and JS [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277789 (owner: 10Glaisher) [17:08:18] \o/ [17:09:32] hey bd808. I'm waiting for a hold on my calender on Tuesday morning to release or get confirmed. I'll respond to your email about meeting next week then (Tuesday should work for me). [17:10:02] lzia: awesome [17:17:15] 6Labs: unable to change puppet groups via Special:NovaInstace - https://phabricator.wikimedia.org/T130104#2126710 (10Dzahn) Filippo got the issue again when he tried to add the puppet::self role to instance test-prometheus4. I tried the same and could configure it without that issue. "Modified instance (test-pro... [17:22:16] 6Labs: unable to change puppet groups via Special:NovaInstace - https://phabricator.wikimedia.org/T130104#2126721 (10fgiunchedi) p:5High>3Low a:5Andrew>3None thanks @dzahn! Interestingly I can **remove** the role but times out when adding it back in. Anyways low now since I am unblocked and there's a wor... [17:24:13] (03PS1) 10MarcoAurelio: Deleted duplicate/redundant file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277807 [17:24:47] [13nagf] 15Krinkle pushed 1 new commit to 06master: 02https://github.com/wikimedia/nagf/commit/f3ce0237f16971e19fa91cbdcedb55d05be09005 [17:24:48] 13nagf/06master 14f3ce023 15zhuyifei1999: graphs: Add "Used" memory... [17:25:08] (03CR) 10MarcoAurelio: [C: 032] Deleted duplicate/redundant file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277807 (owner: 10MarcoAurelio) [17:26:02] [13nagf] 15Krinkle 04force-pushed 06docker from 1475e62bd to 14ff20842: 02https://github.com/wikimedia/nagf/commits/docker [17:26:02] 13nagf/06docker 1477498bf 15YuviPanda: Support local dev with Docker... [17:26:03] 13nagf/06docker 144b6448d 15YuviPanda: Move pid file to /tmp to avoid permission issues [17:26:03] 13nagf/06docker 14ff20842 15YuviPanda: Do not install recommended packages... [17:26:44] wikimedia/nagf#33 (master - f3ce023: zhuyifei1999) The build was broken. - https://travis-ci.org/wikimedia/nagf/builds/116436757 [17:27:53] wikimedia/nagf#34 (docker - ff20842: YuviPanda) The build was broken. - https://travis-ci.org/wikimedia/nagf/builds/116437054 [17:27:55] 10PAWS: PAWS public will not allow for downloading a whole TSV file - https://phabricator.wikimedia.org/T130132#2126755 (10Halfak) [17:28:59] [13nagf] 15Krinkle pushed 5 new commits to 06docker: 02https://github.com/wikimedia/nagf/compare/ff2084297fbf...4b9939e12b7f [17:28:59] 13nagf/06docker 142cacfef 15YuviPanda: Support local dev with Docker... [17:29:00] 13nagf/06docker 142961b2c 15YuviPanda: Move pid file to /tmp to avoid permission issues [17:29:00] 13nagf/06docker 1475e62bd 15YuviPanda: Do not install recommended packages... [17:30:51] wikimedia/nagf#35 (docker - 4b9939e: Timo Tijhof) The build is still failing. - https://travis-ci.org/wikimedia/nagf/builds/116437848 [17:30:58] (03Merged) 10jenkins-bot: Deleted duplicate/redundant file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277807 (owner: 10MarcoAurelio) [17:31:54] [13nagf] 15Krinkle pushed 1 new commit to 06master: 02https://github.com/wikimedia/nagf/commit/3109c09d8c0b2a6b7d724752924a47bebf036820 [17:31:54] 13nagf/06master 143109c09 15Timo Tijhof: graphs: Fix phpcs style violation [17:32:09] [13nagf] 15Krinkle pushed 1 new commit to 06docker: 02https://github.com/wikimedia/nagf/commit/1ec1972103b888dca3c7272ac08f525cd19ec688 [17:32:09] 13nagf/06docker 141ec1972 15Timo Tijhof: Merge branch 'master' into docker [17:33:58] wikimedia/nagf#36 (master - 3109c09: Timo Tijhof) The build was fixed. - https://travis-ci.org/wikimedia/nagf/builds/116438557 [17:36:31] (03PS1) 10Glaisher: Fix regression in delete.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277811 [17:37:28] (03PS2) 10MarcoAurelio: Fix regression in delete.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277811 (owner: 10Glaisher) [17:41:04] (03CR) 10MarcoAurelio: [C: 032] Fix regression in delete.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277811 (owner: 10Glaisher) [17:49:06] 6Labs: unable to change puppet groups via Special:NovaInstace - https://phabricator.wikimedia.org/T130104#2126881 (10fgiunchedi) a:3fgiunchedi assigning back to me since I suspect it is a local problem, with iceweasel I couldn't reproduce and I can't reproduce with chromium either now [17:58:15] (03Merged) 10jenkins-bot: Fix regression in delete.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/277811 (owner: 10Glaisher) [18:17:42] 6Labs, 10Labs-Infrastructure: Install pdf2djvu for Wikisource DjVu aid - https://phabricator.wikimedia.org/T130138#2126974 (10Nemo_bis) [18:22:46] 6Labs, 10Labs-Infrastructure, 10Tool-Labs: tools-k8s: Nagf down after rolling-update - https://phabricator.wikimedia.org/T130140#2126998 (10Krinkle) [18:23:01] 6Labs, 10Labs-Infrastructure, 10Tool-Labs: tools-k8s: Nagf down after rolling-update - https://phabricator.wikimedia.org/T130140#2127010 (10Krinkle) p:5Triage>3Unbreak! [18:23:59] 6Labs, 10Labs-Infrastructure, 10Tool-Labs: tools-k8s: Nagf down after rolling-update - https://phabricator.wikimedia.org/T130140#2126998 (10Krinkle) [18:29:54] (03PS6) 10MarcoAurelio: Adding .lighttpd.conf file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276209 [18:35:07] (03PS5) 10MarcoAurelio: Continuous Integration Python config for labs/tools/stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) [18:36:54] 6Labs, 10Labs-Infrastructure, 10DBA: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#2127049 (10Superyetkin) When can this happen? [19:35:04] 6Labs, 6WMF-Legal: Ensure that Terms of Use document restrictions on third-party web interactions - https://phabricator.wikimedia.org/T129936#2127308 (10tom29739) [19:40:52] 6Labs, 7Tracking: designatedashboard monkeypatch for proxy records - https://phabricator.wikimedia.org/T130151#2127329 (10Andrew) [20:00:29] andrewbogott: Are you familar with tools-k8s? [20:00:50] Krinkle: I haven’t worked with k8s much — just theory, no practice [20:00:51] A deployment earlier today took it down and I'm unable to get it back up [20:02:41] Krinkle: the instance itself, or a service on the instance? [20:02:45] Alternatively, I'd be open to having it torn off k8s and back to plain tools for the moment. [20:03:05] Yuvi turned it into a tools-k8s project as working example, except it's not working at the moment. [20:03:29] andrewbogott: The k8s pod is 'up' but nginx serves 500 and I have no information or know-how to debug. [20:03:40] ah [20:03:44] I don't know whether the proxy or my container is throwing the error [20:03:50] chasemp, valhallasw`cloud, any ideas about k8s? [20:04:00] uuuh. [20:04:00] Thx for checking [20:04:24] I'm not entirely sure. It should just pass through tools.wmflabs.org, right? [20:05:19] Yeah, and then it gets (somehow?) routed to the k8s pod instead of the main tools grid router. [20:05:39] which tool? [20:05:43] nagf [20:05:52] https://phabricator.wikimedia.org/T130140 [20:06:44] the 500 is super weird on itself [20:08:56] Krinkle: have you tried setting the X-Wikimedia-Debug header? I think it's an issue in nagf itself [20:09:05] curl http://192.168.0.224:8080 [20:09:05] Error - Nagf
Exception: Unable to write to cache file
[20:09:05] 	  in /data/project/nagf/inc/WebCache.php:57
[20:10:59] 	 6Labs, 10Labs-Infrastructure, 10Tool-Labs: tools-k8s: Nagf down after rolling-update - https://phabricator.wikimedia.org/T130140#2126998 (10valhallasw) ``` valhallasw@tools-proxy-01:~$ redis-cli hgetall prefix:nagf 1) ".*" 2) "http://192.168.0.224:8080" valhallasw@tools-proxy-01:~$ curl http://192.168.0.224:...
[20:12:46] 	 valhallasw`cloud: Hm.. interesting
[20:13:11] 	 6Labs, 10Tool-Labs, 13Patch-For-Review: Puppet errors on tools-web-static-01 and tools-web-static-02 - https://phabricator.wikimedia.org/T128411#2127423 (10scfc) a:3scfc
[20:13:41] 	 valhallasw`cloud: Why doesn't that surface by default? 
[20:14:10] 	 Krinkle: because we serve pretty error pages by default (which clearly failed here as well)
[20:14:23] 	 But my tool has its own error page
[20:14:30] 	 https://phabricator.wikimedia.org/T103662
[20:14:41] 	 not surprisingly filed by you ;-)
[20:15:34] 	 Krinkle: basically, what needs to be done is to move the error page to each lighttpd instance instead of the central nginx proxy
[20:15:57] 	 Was it an nfs issue or something else that unable to write to cache?
[20:20:43] 	 Hm.. I have a hunch
[20:20:52] 	 probably something went wrong when building the docker image
[20:23:35] 	 Ok fixed
[20:23:47] 	 I built the image with a non-empty cache, which then gets chowned wrong
[20:23:55] 	 so it wasnt able to write to those files
[20:30:46] 	 6Labs, 10Labs-Infrastructure, 10Tool-Labs: tools-k8s: Nagf down after rolling-update - https://phabricator.wikimedia.org/T130140#2127446 (10Krinkle) 5Open>3Resolved a:3Krinkle Thanks. I pushed a bad Docker image (it had a non-empty cache directory on my dev machine, which apparently becomes inaccessibl...
[20:31:16] 	 valhallasw`cloud: andrewbogott: Thanks!
[20:45:49] 	 !log tools.heritage jsubbed populate_image_table.py for https://phabricator.wikimedia.org/T130107 (see crontab -l for exact command)
[20:45:53] 	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master
[23:56:08] 	 (03PS1) 10Jean-Frédéric: Keep alive connection to second MySQL database using Ping [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/277925 (https://phabricator.wikimedia.org/T117045) 
[23:58:39] 	 (03CR) 10Jean-Frédéric: [C: 032] Keep alive connection to second MySQL database using Ping [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/277925 (https://phabricator.wikimedia.org/T117045) (owner: 10Jean-Frédéric)