[00:10:18] 6Labs, 10Tool-Labs, 10CirrusSearch, 6Discovery: "Search Tool Labs Documentation" doesn't work - https://phabricator.wikimedia.org/T128758#2090533 (10EBernhardson) It looks like the problem is those pages are indexed to labswiki_general, but the query is being performed against labswiki_content. Not sure ex... [00:11:42] PROBLEM - Puppet failure on tools-worker-1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [00:15:12] 6Labs, 6Operations, 10ops-eqiad: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2090540 (10chasemp) [00:19:27] 6Labs, 10Tool-Labs, 10CirrusSearch, 6Discovery: "Search Tool Labs Documentation" doesn't work - https://phabricator.wikimedia.org/T128758#2090576 (10bd808) >>! In T128758#2090533, @EBernhardson wrote: > Not sure exactly whats going on here ... perhaps help was added to the list of content indices at some p... [00:53:32] 6Labs, 10Tool-Labs, 10CirrusSearch, 6Discovery: "Search Tool Labs Documentation" doesn't work - https://phabricator.wikimedia.org/T128758#2090664 (10EBernhardson) 5Open>3Resolved a:3EBernhardson the quick reindex didn't work, but i did a full reindex and the queries look to be giving results again. [01:15:52] 10Tool-Labs-tools-stewardbots: hat-web-tools import for stewardbots - https://phabricator.wikimedia.org/T128743#2090716 (10Rschen7754) @MarcoAurelio I've given you access to hat-collector on toollabs so you can copy it over. [02:10:37] 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitechwiki has 4xx responses to requests for some static assets inc. poweredby_mediawiki_88x31.png and WikiEditor's button-sprite.svg - https://phabricator.wikimedia.org/T128747#2090838 (10Krinkle) >>! In T128747#2089235, @Krinkle wrote: >... [02:38:28] andrewbogott, should I file a task about deployment-tin? [03:17:43] PROBLEM - Puppet failure on tools-worker-1001 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [04:20:30] Krenair: I don’t think it’s really a bug, it’s just the way our virt setup works. It’s only if it’s trivial to rebuild from scratch that we can do anything about it. [08:08:03] 6Labs, 10Datasets-General-or-Unknown: Labs dumps mirror directories in /public/dumps/public/ no longer world-readable - https://phabricator.wikimedia.org/T128945#2090963 (10Nemo_bis) [08:34:53] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [09:12:19] 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2090986 (10Glaisher) >>! In T128742#2090272, @MarcoAurelio wrote: > I guess we need a cache folder too? It's not necessary and can work without it. I only attempted to unbreak the scrip... [09:14:19] PROBLEM - SSH on tools-worker-1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:30] (03PS1) 10Glaisher: Add .gitreview [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275161 [10:01:50] (03PS1) 10Glaisher: Set up php parallel-lint [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275163 (https://phabricator.wikimedia.org/T128503) [10:02:48] (03PS2) 10Glaisher: Set up php parallel-lint [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275163 (https://phabricator.wikimedia.org/T128503) [10:11:54] 10Tool-Labs-tools-stewardbots, 10Continuous-Integration-Config, 13Patch-For-Review: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#2091003 (10Glaisher) I think that would set up PHP testing. But I cannot help with Python tests unfortunately as I don't have the k... [11:39:10] RECOVERY - SSH on tools-worker-1002 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u1 (protocol 2.0) [12:16:10] PROBLEM - SSH on tools-worker-1002 is CRITICAL: Server answer [12:17:10] !log rcm deleted rcm-3 (Spring cleaning) [12:17:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL, Master [12:17:31] andrewbogott: ^ 20 GB more for you ;) [12:41:10] RECOVERY - SSH on tools-worker-1002 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u1 (protocol 2.0) [13:19:38] 6Labs, 10Datasets-General-or-Unknown: Labs dumps mirror directories in /public/dumps/public/ no longer world-readable - https://phabricator.wikimedia.org/T128945#2090963 (10scfc) bbc31547eca805c21fe9de0664ce14dd18a17b43 was merged on Thursday, so maybe that's related. [14:34:02] (03PS65) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [14:36:58] (03CR) 10Ricordisamoa: "PS65 updates grunt-jscs from ~2.7.0 to ~2.8.0" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [15:11:59] 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2091215 (10MarcoAurelio) >>! In T128742#2090986, @Glaisher wrote: >>>! In T128742#2090272, @MarcoAurelio wrote: >> I guess we need a cache folder too? > > It's not necessary and can wor... [15:14:31] 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2091216 (10Glaisher) It'd be nice if you could migrate all of it to the repo, if possible. It'd make it much easier to clean up once it starts to use Git. [15:15:15] Glaisher: I'll merge https://gerrit.wikimedia.org/r/#/c/275001/ [15:15:44] cool [15:16:26] (03CR) 10MarcoAurelio: [C: 032 V: 032] "Merge." [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275001 (https://phabricator.wikimedia.org/T128742) (owner: 10Glaisher) [15:17:56] !log tools.stewardbots Merged https://gerrit.wikimedia.org/r/#/c/275001/ [15:17:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [15:20:01] 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2091222 (10MarcoAurelio) >>! In T128742#2091216, @Glaisher wrote: > It'd be nice if you could migrate all of it to the repo, if possible. It'd make it much easier to clean up once it sta... [15:21:45] 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2091224 (10Glaisher) >>! In T128742#2091222, @MarcoAurelio wrote: > Patch above seems to work, but page lacks some CSS stuff I think :) Yep, the CSS/JS files are not in the public repo,... [15:23:09] (03PS2) 10MarcoAurelio: Add .gitreview [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275161 (owner: 10Glaisher) [15:23:28] (03CR) 10MarcoAurelio: [C: 032 V: 032] "Merge" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275161 (owner: 10Glaisher) [15:24:49] !log tools.stewardbots tools.stewardbots@tools-bastion-05 merged https://gerrit.wikimedia.org/r/275161 [15:24:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [15:28:07] (03PS3) 10MarcoAurelio: Set up php parallel-lint [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275163 (https://phabricator.wikimedia.org/T128503) (owner: 10Glaisher) [15:28:28] (03CR) 10MarcoAurelio: [C: 032 V: 032] Set up php parallel-lint [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275163 (https://phabricator.wikimedia.org/T128503) (owner: 10Glaisher) [15:28:56] !log tools.stewardbots Merged https://gerrit.wikimedia.org/r/#/c/275163/ [15:28:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [15:31:24] Thanks for your patches Glaisher :) [15:31:34] do we have now jenkins tests on PHP? [15:31:51] glad to help :) [15:32:08] no, jenkins will start to run them only after the integration/config patch is merged [15:32:18] but now you can run composer test to run the tests locally [15:32:57] I'll try to figure out how to make the pywiki tests work [15:33:08] * mafk is no informatic engineer [16:18:54] (03PS1) 10MarcoAurelio: Continuous Integration config for labs/tools/stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) [16:19:50] (03PS2) 10MarcoAurelio: Continuous Integration Python config for labs/tools/stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) [16:20:53] 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitechwiki has 4xx responses to requests for some static assets inc. poweredby_mediawiki_88x31.png and WikiEditor's button-sprite.svg - https://phabricator.wikimedia.org/T128747#2084612 (10scfc) When I "Edit Source" a page, the buttons hav... [16:21:12] 10Tool-Labs-tools-stewardbots, 10Continuous-Integration-Config, 13Patch-For-Review: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#2091276 (10MarcoAurelio) @hashar Like https://gerrit.wikimedia.org/r/275190 ? [17:58:09] !log ores Shut down the uwsgi service and started up the uwsgi-ores-web service on web-01 and -02. [17:58:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [18:46:30] What's with the load on s1.labsdb? Was writing a query last night and saw it balloon from 5 minutes to 55 minutes [19:10:44] 6Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 6Operations, and 2 others: copy wikitech dumps to dumps server ? - https://phabricator.wikimedia.org/T128680#2091569 (10ArielGlenn) The copy works but the cleanup of the autogenerated index.html files, also copied, did not. Patch is in, will check... [19:36:46] Dispenser: I'm not sure what you mean with 'what's with the load'. The two remaining servers are more heavily loaded because c2 is down, but the 5<->55 minutes issue sounds like data leaving a cache. [19:37:11] PROBLEM - SSH on tools-worker-1002 is CRITICAL: Server answer [19:39:43] Actually it was 5 min, 10 min, 22 min, and finally 55 minutes. I'm thinking some big script started up [20:02:10] RECOVERY - SSH on tools-worker-1002 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u1 (protocol 2.0) [20:06:46] "IF YOU SEE THIS, THINGS HAVE GONE VERY WRONG!" :p lol [20:07:06] ~/flickr2commons/~ [20:58:04] 6Labs, 10DBA: s52490 created 338 simultanous conections to the same server executing the same query - https://phabricator.wikimedia.org/T128355#2091698 (10mahmoud) Hey again @jcrespo, we made some code changes and would request that you or another DBA throttle the number of database connections up to 20 or so... [21:20:13] 6Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 6Operations, and 2 others: copy wikitech dumps to dumps server ? - https://phabricator.wikimedia.org/T128680#2091707 (10ArielGlenn) https://gerrit.wikimedia.org/r/#/c/275215/ added the dumps to the 'other' index page where they are now available. [21:56:15] (03PS18) 10Ricordisamoa: Initial commit [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 [21:58:00] (03CR) 10Ricordisamoa: "PS18 updates grunt-contrib-jshint from ~0.12.0 to ~1.0.0" [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 (owner: 10Ricordisamoa) [21:59:02] (03PS19) 10Ricordisamoa: Initial commit [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 [22:00:25] (03CR) 10Ricordisamoa: "PS19 updates grunt-jscs from ~2.7.0 to ~2.8.0" [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 (owner: 10Ricordisamoa) [23:15:10] PROBLEM - SSH on tools-worker-1002 is CRITICAL: Server answer [23:20:12] RECOVERY - SSH on tools-worker-1002 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u1 (protocol 2.0) [23:31:13] PROBLEM - SSH on tools-worker-1002 is CRITICAL: Server answer [23:36:11] RECOVERY - SSH on tools-worker-1002 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u1 (protocol 2.0)