[00:44:11] (03PS1) 10BryanDavis: Add 502, 503 and 504 error pages [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/308305 (https://phabricator.wikimedia.org/T144040) [00:57:01] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by 94.197.121.8 link https://www.mediawiki.org/w/index.php?diff=2229602 edit summary: [+353] /* Development and sysadmin tasks */ [00:57:53] volans fixed it now :). [01:02:54] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Krenair link https://www.mediawiki.org/w/index.php?diff=2229605 edit summary: [-353] Reverted edits by [[Special:Contributions/94.197.121.8|94.197.121.8]] ([[User talk:94.197.121.8|talk]]) to last revision by [[User:Shirayuki|Shirayuki]] [02:06:30] Amir1: are you around by any chance? [02:18:39] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [02:19:17] 10Labs-project-Wikistats, 10Internet-Archive, 13Patch-For-Review: Add link to archive.org dump - https://phabricator.wikimedia.org/T41370#2606406 (10Dzahn) @Nemo_bis There is an archive column now http://wikistats.wmflabs.org/display.php?t=mw Check it out [02:19:38] 10Labs-project-Wikistats, 10Internet-Archive, 13Patch-For-Review: Add link to archive.org dump - https://phabricator.wikimedia.org/T41370#2606407 (10Dzahn) 05Open>03Resolved [02:50:29] leila: hey, I was afk doing the dishes [02:50:31] sup? [03:00:50] I'm going to be afk for "Entehkhab vahed" leila [03:01:04] You can send me an email or telegram [03:01:24] Amir1: I'll do that. thanks. [03:01:38] good luck with entekhaab vaahed. do the right thing. ;) [04:14:54] Heyyyyyy... so will we be getting icinga monitoring for Tool Labs? [05:09:58] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mooeypoo was modified, changed by Mooeypoo link https://wikitech.wikimedia.org/w/index.php?diff=819339 edit summary: [05:31:28] PROBLEM - Puppet run on tools-exec-1211 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:11:28] RECOVERY - Puppet run on tools-exec-1211 is OK: OK: Less than 1.00% above the threshold [0.0] [06:56:17] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:10:13] paladox: great! thanks a lot for the fix [07:36:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [07:40:49] PROBLEM - Free space - all mounts on tools-services-01 is CRITICAL: CRITICAL: tools.tools-services-01.diskspace.root.byte_percentfree (<11.11%) [09:30:43] 06Labs, 10DBA, 06Editing-Analysis, 05Security: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2606533 (10jcrespo) > Better? Yes. [10:00:53] volans your welcome, your on earley :) [10:22:04] [Sat Sep 3 10:09:57 2016] /data/project/commons-delinquent/demon.php: not an executable file <-- what is wrong with jsub? [10:22:13] jsub won't execute the file *sigh* [10:26:55] yuvipanda O_O [10:29:21] oh, chmod +x . fixed :) [11:12:13] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Krenair link https://www.mediawiki.org/w/index.php?diff=2230020 edit summary: [+20] outdated [11:18:55] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Mainframe98 link https://www.mediawiki.org/w/index.php?diff=2230025 edit summary: [+4] Usage of [[Template:TNT]] [11:51:25] Alphos: if you think you can keep that speed that's great, just make sure you are able to verify the tools do what expected :) [11:51:32] Nemo_bis : pointing out the fact that rewriting from ext/mysql to mysqli is non-trivial ; and as long as we're doing things right, we might as well do things the easy way if we want them secure ;-) [11:51:54] and i'll say PDO is easier than mysqli til the day i die :D [11:52:41] lots of tools may need a modicum of scrapping, but i'd rather be on the safe side and stick to prepared statements and errorless code ;-) [11:53:29] think it would be possible to use php 5.6 ? php 5.3 was a good start, but it's definitely feeling its old age (and end of life) [11:58:32] on that note, maybe i should grab a bite [12:30:03] 06Labs, 10Labs-Infrastructure, 10media-storage: Provide Swift object store(s) for the labs projects - https://phabricator.wikimedia.org/T114998#2606650 (10hashar) 05Open>03declined Last year some of the discussion was to eventually provide an object storage system for all labs project. Not sure whether t... [16:49:25] (03PS1) 10MarcoAurelio: Update main page [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/308351 [16:56:07] (03CR) 10MarcoAurelio: [C: 032] Update main page [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/308351 (owner: 10MarcoAurelio) [16:56:29] (03Merged) 10jenkins-bot: Update main page [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/308351 (owner: 10MarcoAurelio) [17:01:38] !log tools.stewardbots [[gerrit:308351|Updated main page]] [17:01:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [19:07:24] PROBLEM - Free space - all mounts on tools-prometheus-02 is CRITICAL: CRITICAL: tools.tools-prometheus-02.diskspace.root.byte_percentfree (<11.11%) [19:08:11] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mooeypoo was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=819344 edit summary: [19:37:18] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:14:00] (03PS2) 10BryanDavis: Add 502, 503 and 504 error pages [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/308305 (https://phabricator.wikimedia.org/T144040) [20:25:27] (03CR) 10BryanDavis: "Cherry-picked to striker-deploy03 for testing." [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/308305 (https://phabricator.wikimedia.org/T144040) (owner: 10BryanDavis) [20:42:25] PROBLEM - Free space - all mounts on tools-prometheus-02 is CRITICAL: CRITICAL: tools.tools-prometheus-02.diskspace.root.byte_percentfree (<11.11%) [21:11:00] Alphos: on the trusty exec nodes we have PHP 5.5.9. The debian jessie kubernetes webservices have PHP 5.6.24. We won't ever have jessie exec nodes for OGE because the OGE package was removed from Debian. So PHP 5.6 support depends on where the code is running. [21:12:54] bd808 well, from what i saw on tools.erwin85, 5.3 seems on [21:13:19] definitely would love 5.6, or at default 5.5 [21:13:29] s/at default/at least/ [21:13:44] but i think we have to handle old tools still somewhat working ^^' [21:14:26] the default OGE exec nodes are precise. You can run on trusty by adding "-l release=trusty" to the command [21:14:28] bd808 : on another note, is there a list of all available *_p somewhere ? [21:14:43] * Alphos is having another attack of lazy ^^' [21:15:03] there's a quarry query for it somewhere... [21:16:00] https://quarry.wmflabs.org/query/4031 [21:16:38] yay, thanks :) [21:16:57] that's linked from https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Naming_conventions [21:17:11] (and a bit hard to find if you don't know it is there) [21:17:21] indeed :D [21:17:56] I wish I had 3 tech writers to redo the docs for Labs/Tool Labs from scratch :/ [21:27:07] now for the naughty : assuming i want to pass a query to all wiki_p every friday between 3am and 4am UTC (and it takes less than 3 minutes for the presumed biggest, enwiki), would it be better to have one cron job to do them all, or to divide into multiple cron jobs ? [21:28:07] (dewiki : real 0m27.200s ; enwiki : real 2m28.489s ) [21:28:34] it's probably nicer to use one job to run them in series rather than N jobs to run them in parallel. [21:28:56] iirc it's a common misconception that multiple jobs will run in parallel [21:29:21] well it depends on the capacity of the grid at the time [21:29:23] if you have 3 lines in your crontab, and all are triggered "at the same time", the second will only start once the first is complete [21:29:36] (and the third once the second is complete) [21:29:55] that would depend on the cron system in use I suppose [21:30:13] and whether the jobs executed inline or forked [21:30:14] so your job that should be triggered at :05 is in fact triggered at :05 + execution time of all the previous ones [21:30:30] dunno about how jsub handles that kind of things though [21:30:59] but i'm definitely fine with all in one job ^^ [21:31:43] jsub dispatches the jobs to be run asynchronously on the grid by default, so even with serial exec of jsub the jobs would likely overlap if the runtime is non-trivial [21:32:33] yah [21:32:46] so going for a sequential, all in one job then :) [21:33:38] resultset is less complete than i wanted (it lacks the item of the target page of a redirect), but time needed is orders of magnitude less, so i guess it'll have to do ^^ [21:34:48] orders of magnitude faster is always nice :) [21:34:50] arwiki took a little over an hour to process when i tried a few months ago ; roughly 8 seconds with this query. [21:34:54] IKR ! [21:35:13] oh wow. that's a great improvement [21:35:34] well, the issue is with the old query working 1) on two dbs 2) with REPLACE() on an index [21:35:45] didn't have a choice, "_" vs " " [21:36:30] now just three tables instead of (if memory serves) five (four including one twice), and a single db [21:37:25] now to review my loop-fu in sh, always a pleasant experience >_> [21:42:25] PROBLEM - Free space - all mounts on tools-prometheus-02 is CRITICAL: CRITICAL: tools.tools-prometheus-02.diskspace.root.byte_percentfree (<11.11%) [22:07:49] bd808 : i don't suppose i could ask you to have a tidbit of code review ? ^^ 7 lines of sh, 9 lines of sql (pretty sure it works as intended though) [22:17:24] PROBLEM - Free space - all mounts on tools-prometheus-02 is CRITICAL: CRITICAL: tools.tools-prometheus-02.diskspace.root.byte_percentfree (<11.11%) [22:54:15] (03CR) 10Alex Monk: [C: 032] Add 502, 503 and 504 error pages [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/308305 (https://phabricator.wikimedia.org/T144040) (owner: 10BryanDavis) [22:54:21] (03Merged) 10jenkins-bot: Add 502, 503 and 504 error pages [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/308305 (https://phabricator.wikimedia.org/T144040) (owner: 10BryanDavis) [22:58:57] bd808 : whole process for all wikipedias took 12 minutes and 53 seconds, i think it's a keeper ^^ [22:59:20] that's a good time :P [22:59:55] results : https://tools.wmflabs.org/wikidata-redirects-conflicts-reports/reports/2016-35/ [23:00:41] there's tenwiki in the mix, which throws an error because it doesn't have a wbc_entity_usage table, but i can live with that :D [23:01:46] i *think* it'll work from cron as it's intended to, but i'm afraid there's only one way to find out :D [23:17:25] PROBLEM - Free space - all mounts on tools-prometheus-02 is CRITICAL: CRITICAL: tools.tools-prometheus-02.diskspace.root.byte_percentfree (<11.11%) [23:29:42] welp, let's pray for the best for next friday at 3am UTC [23:32:24] PROBLEM - Free space - all mounts on tools-prometheus-02 is CRITICAL: CRITICAL: tools.tools-prometheus-02.diskspace.root.byte_percentfree (<11.11%)