[00:54:55] tgr: yeah, I had to basically go in and patch them :( [01:50:42] yuvipanda: so now that they are patched, is there a way suppress the URL prefix? [02:35:59] Change on 12www.mediawiki.org a page OAuth/For Developers was modified, changed by Tgr (WMF) link https://www.mediawiki.org/w/index.php?diff=2414810 edit summary: made some slides for the WDDC hackathon. They are not great but definitely less horrible than this page. [04:34:58] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [04:52:54] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [05:09:58] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [05:27:52] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [06:45:39] PROBLEM - Puppet run on tools-exec-gift-trusty-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:25:41] RECOVERY - Puppet run on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:25:03] 06Labs, 10DBA: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3074934 (10jcrespo) a:03jcrespo [10:48:52] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:28:51] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [12:47:37] 06Labs, 10DBA: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3075232 (10jcrespo) p:05Triage>03Normal [12:58:36] 06Labs, 06Operations, 07kubernetes: docker-engine pulled into our repositories only keeps the latest version - https://phabricator.wikimedia.org/T153416#3075281 (10akosiaris) Do we have a verdict on this one ? [13:04:51] PROBLEM - Puppet run on tools-elastic-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:19:19] PROBLEM - Puppet run on tools-elastic-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:20:35] 06Labs, 06Operations, 07kubernetes: docker-engine pulled into our repositories only keeps the latest version - https://phabricator.wikimedia.org/T153416#3075343 (10MoritzMuehlenhoff) Some higher level plans for structuring the repos are now collected at https://phabricator.wikimedia.org/T158583, input welcome. [13:22:14] PROBLEM - Puppet run on tools-elastic-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:32:20] (03PS1) 10Tarrow: include blank htpasswd needed for role::toollabs::elasticsearch [labs/private] - 10https://gerrit.wikimedia.org/r/341326 [14:41:46] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3075706 (10Marostegui) My idea would be as follows: - stop labsdb1006 -> copy `/srv/postgres` to dbstore1001 -> reimage -> copy the data back. If that works fine, repea... [14:44:54] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3075715 (10jcrespo) The most important step, and why we need to copy that data away in case something goes wrong is the postgres upgrade from 9.1 (precise) to 9.4 (jessie... [14:49:59] 06Labs, 06Operations: openstack instance creation sometimes takes >480s - https://phabricator.wikimedia.org/T159459#3075772 (10chasemp) cleaned up and restarted leaked instances from the small fiasco of rolling out https://gerrit.wikimedia.org/r/#/c/340986/ which due to a bug in nova requiring 20m to return ca... [14:54:24] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#3075804 (10Andrew) [14:54:26] 06Labs: Request creation of Wikimedia Incubator labs project - https://phabricator.wikimedia.org/T159068#3075801 (10Andrew) 05Open>03Resolved a:03Andrew Sorry for the delay in creation, I was sick most of last week. This project has been created, with @Hydriz as the projectadmin. [15:02:37] 06Labs: Request creation of Wikimedia Incubator labs project - https://phabricator.wikimedia.org/T159068#3075855 (10Hydriz) Awesome. As always, health over work. Thanks for your help! [15:20:19] 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3075915 (10jcrespo) [15:20:21] 06Labs, 10DBA: Possible labs imagelinks drift - https://phabricator.wikimedia.org/T159023#3075912 (10jcrespo) 05Open>03Resolved a:03jcrespo This is fixed now, but again I encourage you to try the new servers soon, where this problem wasn't there in the first place. ``` labsdb1001> SELECT * FROM imagelin... [15:28:10] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries: nlwiki Labs replica with problems - https://phabricator.wikimedia.org/T138927#3075954 (10jcrespo) 05Open>03Resolved ``` root@labsdb1003[nlwiki]> select * from page where page_title='Aline_fobe' and page_namespace=0; Empty set (0.01 sec) ``` [15:38:22] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3076003 (10akosiaris) >>! In T157359#3075706, @Marostegui wrote: > My idea would be as follows: > > - stop labsdb1006 -> copy `/srv/postgres` to dbstore1001 -> reimage -... [15:44:08] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3076026 (10jcrespo) > There is also this days pg_upgrade which with --link mode which should in theory help avoid that problem, but I 've never tested it in a 9.1 => 9.4... [15:46:21] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3076028 (10akosiaris) >>! In T157359#3076026, @jcrespo wrote: >> There is also this days pg_upgrade which with --link mode which should in theory help avoid that problem... [15:49:17] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3076041 (10jcrespo) Hey, I am not saying it is going to work 100% sure- I am just suggesting to try it first, and then go the slow route, which is basically what you sugg... [15:50:34] why am i unable to reboot one of my instances (https://wikitech.wikimedia.org/wiki/Nova_Resource:Mw-expt.wikitextexp.eqiad.wmflabs) .. it is showing up as SHUTOFF in the dashboard @ https://wikitech.wikimedia.org/wiki/Special:NovaInstance [15:51:27] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3076044 (10Marostegui) >>! In T157359#3076003, @akosiaris wrote: > > Actually I had a different plan in mind. So, labsdb1007 is a read-only slave of labsdb1006. My propo... [15:52:22] subbu it showed on the 14 of february 2017, the instance put in stop mode [15:52:23] https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource%3AMw-expt.wikitextexp.eqiad.wmflabs&type=revision&diff=1514784&oldid=994386 [15:53:05] was it stopped automatically? or did someone stop it? [15:53:36] but anyway, can it be restarted? [15:53:47] i don't see any option on the dashboard besides delete or reboot. [15:53:58] Im not sure if shut of instances can be started. [15:54:02] hmm ... [15:54:06] subbu: I'm looking at details now [15:54:25] I'm not sure why it is in shutdown state but I'll try to restart it, nothing seems obviously wrong here [15:54:31] ok. thanks. [15:57:29] subbu: give it a try [15:58:09] 06Labs, 10Labs-Infrastructure: labvirt1001 can't launch new VMs - https://phabricator.wikimedia.org/T159721#3076053 (10Andrew) [15:59:01] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3076066 (10akosiaris) >>! In T157359#3076041, @jcrespo wrote: > Hey, I am not saying it is going to work 100% sure- I am just suggesting to try it first, and then go the... [16:00:26] chasemp, ty works now [16:00:53] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3076077 (10Marostegui) >>! In T157359#3076066, @akosiaris wrote: > Yeah I can do that if it makes you two happier. That would be appreciated. Not happier per se, but a... [16:01:22] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3076082 (10jcrespo) @akosiaris Do you know if labsdb1007 is actively in use? If not at all, we could start doing it now, ahead of the maintenance window... [16:27:06] !log wikispeech Deploy latest from Git master: cc15420 (T151887) [16:27:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL [16:27:09] T151887: Make player buttons keep screen position during scroll - https://phabricator.wikimedia.org/T151887 [16:36:16] 06Labs: Request creation of GLAMpipe labs project - https://phabricator.wikimedia.org/T159309#3063738 (10chasemp) +1 ok [16:42:38] 06Labs, 06Operations, 07kubernetes: docker-engine pulled into our repositories only keeps the latest version - https://phabricator.wikimedia.org/T153416#3076211 (10yuvipanda) There's a thread on ops-l now because docker is now docker community edition. [16:44:50] RECOVERY - Puppet run on tools-elastic-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:46:20] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#3076226 (10Andrew) [16:46:22] 06Labs: Request creation of GLAMpipe labs project - https://phabricator.wikimedia.org/T159309#3076223 (10Andrew) 05Open>03Resolved a:03Andrew I'm afraid that the camelcase name isn't supported, but I've created the 'glampipe' project with Zache-tool as the project admin. @Zache, you can add additional use... [16:54:21] RECOVERY - Puppet run on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:47] 06Labs: Request creation of GLAMpipe labs project - https://phabricator.wikimedia.org/T159309#3076248 (10Zache) Thank you! [17:02:16] RECOVERY - Puppet run on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:12] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3076308 (10akosiaris) It is not. It is a read-only slave not really being used by anyone currently so we are free to start the process on it well ahead of the maint window. [17:41:13] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3076313 (10jcrespo) Thanks, I will start "breaking" it tomorrow Tuesday during EU morning- do not worry, I can take care of this- you probably have more urgent things to... [17:49:52] (03CR) 10Addshore: [C: 031] include blank htpasswd needed for role::toollabs::elasticsearch [labs/private] - 10https://gerrit.wikimedia.org/r/341326 (owner: 10Tarrow) [18:01:28] (03CR) 10Dzahn: [C: 032] include blank htpasswd needed for role::toollabs::elasticsearch [labs/private] - 10https://gerrit.wikimedia.org/r/341326 (owner: 10Tarrow) [18:01:32] (03CR) 10Dzahn: [V: 032 C: 032] include blank htpasswd needed for role::toollabs::elasticsearch [labs/private] - 10https://gerrit.wikimedia.org/r/341326 (owner: 10Tarrow) [18:22:02] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Static user testing - https://phabricator.wikimedia.org/T157682#3076466 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:22:06] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Install Python - https://phabricator.wikimedia.org/T157465#3076468 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:22:13] 10Tool-Labs-tools-LTA-Knowledgebase: Add discussion area - https://phabricator.wikimedia.org/T157223#3076471 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:22:25] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Add WYMeditor - https://phabricator.wikimedia.org/T156566#3076475 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:22:30] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Localization - https://phabricator.wikimedia.org/T155738#3076479 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:22:33] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Add tag functionality - https://phabricator.wikimedia.org/T155343#3076481 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:22:39] 10Tool-Labs-tools-LTA-Knowledgebase: Read directly from replica.my.cnf - https://phabricator.wikimedia.org/T156352#3076484 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:22:48] 10Tool-Labs-tools-LTA-Knowledgebase: Limit the path of session id cookies - https://phabricator.wikimedia.org/T155940#3076486 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:22:56] 10Tool-Labs-tools-LTA-Knowledgebase, 07Vuln-CSRF: Add CSRF tokens - https://phabricator.wikimedia.org/T155938#3076488 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:23:03] 10Tool-Labs-tools-LTA-Knowledgebase: Create search function - https://phabricator.wikimedia.org/T157464#3076491 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:23:30] 10Tool-Labs-tools-LTA-Knowledgebase: Write user guide - https://phabricator.wikimedia.org/T156353#3076493 (10Samtar) 05Open>03Invalid I've done some of this, just needs [18:23:36] 10Tool-Labs-tools-LTA-Knowledgebase: Recent changes should link to LTA entry - https://phabricator.wikimedia.org/T157684#3076495 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:23:54] 10Tool-Labs-tools-LTA-Knowledgebase: Add option for tool admins to change write access - https://phabricator.wikimedia.org/T156181#3076498 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:24:13] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4 - https://phabricator.wikimedia.org/T149870#3076500 (10madhuvishy) > > Two things we have noticed that need to happen to close this: > > - we need to use flock or something to ensur... [18:24:26] 10Tool-Labs-tools-LTA-Knowledgebase: Add history - https://phabricator.wikimedia.org/T157725#3076502 (10Samtar) 05Open>03Invalid Project ended, tool deleted [18:25:05] !log tools set complex_values slots=300,release=trusty for tools-exec-gift-trusty-01.tools.eqiad.wmflabs [18:25:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:25:57] gifti: I think I fixed the issue with your trusty queue. can you try again? [18:46:47] Hello? [18:48:48] Can anyone help me with a request to get write abilities in the wikipedia api? [18:49:51] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:51:43] tomthirteen, this is Jonathan (from email). Still there? [18:52:06] Hey, yes. Thank you for responding [18:52:25] Would you be able to help me? I would like to have write abilities in the api. [18:52:34] I presently only have read abilities. [18:53:03] I never received a replica.my.cnf [18:53:16] labs admins (bd808, chasemp, yuvipanda) for reference, Tom's access request: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Tomthirteen [18:53:35] looks similar to: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/S.desabbata [18:53:56] related phab ticket: https://phabricator.wikimedia.org/T135931 [18:53:59] Yes [18:54:12] No one has responded yet, though [18:55:17] background: tomthirteen would like to have a user database with access to replicas for strategic purposes [18:55:38] tomthirteen: it may take a little patience on your part here :) [18:55:47] Ok, np. [18:55:55] Is there an ETAish? [18:56:02] I've mentioned the names of several people who can help in the channel, and they received notifications. [18:56:15] I'm not able to provide an estimate, no [18:56:18] just helping any way I can [18:56:19] Ok, thank you! [18:56:27] I appreciate your help. [18:56:56] So I should just sit tight to they notify me on the access request? [19:03:04] !log tools Stopping webservice running on tool tree-of-life on author request [19:03:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:03:59] !log deployment-prep mwscript sql.php --wiki=aawiki "CREATE DATABASE cognate_wiktionary" [19:04:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [19:19:48] 06Labs, 10Labs-Infrastructure: Labs instance utrs-primary is running Ubuntu Precise and must be rebuilt. - https://phabricator.wikimedia.org/T159737#3076800 (10Andrew) [19:21:05] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2963025 (10Andrew) [19:29:52] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [19:49:49] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, and 2 others: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3077048 (10bd808) [19:55:42] 06Labs, 10Huggle: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted - https://phabricator.wikimedia.org/T157710#3077094 (10Andrew) If no one lays claim this week I'll probably shut this instance off next Monday, just to see if anyone notices and/or cares. [19:57:09] !log tools.precise-tools Deployed 79875de94698 [19:57:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.precise-tools/SAL [20:19:47] (03PS1) 10Legoktm: Add a COPYING file [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341381 [20:19:59] (03CR) 10Legoktm: [V: 032 C: 032] Add a COPYING file [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341381 (owner: 10Legoktm) [20:20:05] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3077244 (10madhuvishy) @Cmjohnson Moving to row B is fine, but we'd have to move both servers to row B then. We need both servers t... [20:23:53] (03PS1) 10Legoktm: Fix nearly all flake8 issues [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341383 [20:26:26] !log ores restarted precached and uwsgi-ores on ores-web-03 [20:26:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [20:45:12] (03CR) 10Jean-Frédéric: [C: 032] Change database docker image to mariadb:10.0 [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/339549 (https://phabricator.wikimedia.org/T158911) (owner: 10Jean-Frédéric) [20:50:52] (03PS1) 10Legoktm: Drop redis caching, it was probably useless anyways [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341393 [20:50:53] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:50:54] (03PS1) 10Legoktm: Set better requests user-agent [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341394 [20:50:56] (03PS1) 10Legoktm: Update my local path [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341395 [20:50:58] (03PS1) 10Legoktm: Add data.json to gitignore [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341396 [20:51:11] (03CR) 10Legoktm: [V: 032 C: 032] Fix nearly all flake8 issues [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341383 (owner: 10Legoktm) [20:51:24] (03CR) 10Legoktm: [V: 032 C: 032] Drop redis caching, it was probably useless anyways [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341393 (owner: 10Legoktm) [20:51:33] (03CR) 10Legoktm: [V: 032 C: 032] Set better requests user-agent [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341394 (owner: 10Legoktm) [20:51:43] (03CR) 10Legoktm: [V: 032 C: 032] Update my local path [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341395 (owner: 10Legoktm) [20:51:54] (03CR) 10Legoktm: [V: 032 C: 032] Add data.json to gitignore [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341396 (owner: 10Legoktm) [20:52:48] (03PS1) 10Legoktm: Document requirements [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341397 [20:52:58] (03CR) 10Legoktm: [V: 032 C: 032] Document requirements [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/341397 (owner: 10Legoktm) [20:53:11] (03Merged) 10jenkins-bot: Change database docker image to mariadb:10.0 [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/339549 (https://phabricator.wikimedia.org/T158911) (owner: 10Jean-Frédéric) [20:56:57] (03CR) 10jenkins-bot: Change database docker image to mariadb:10.0 [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/339549 (https://phabricator.wikimedia.org/T158911) (owner: 10Jean-Frédéric) [21:01:47] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Output data for new XTools: Articleinfo - https://phabricator.wikimedia.org/T157706#3077409 (10MusikAnimal) [21:02:11] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Output data for new XTools: Articleinfo - https://phabricator.wikimedia.org/T157706#3013867 (10MusikAnimal) a:03MusikAnimal [21:13:28] yuvipanda: seems to have cleared up [21:21:37] i'll allow myself to close the corresponding task [21:24:31] 06Labs, 10Tool-Labs: Investigate alternatives to dedicated exec node for gifti's tools - https://phabricator.wikimedia.org/T99130#3077565 (10Giftpflanze) [21:24:34] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#3077566 (10Giftpflanze) [21:24:36] 06Labs, 10Tool-Labs, 07Epic: Find a solution for tools-exec-gift on Trusty - https://phabricator.wikimedia.org/T156981#3077562 (10Giftpflanze) 05Open>03Resolved a:03Giftpflanze < yuvipanda> !log tools set complex_values slots=300,release=trusty for tools-exec-gift-trusty-01.tools.eqiad.wmflabs This see... [21:30:10] 06Labs, 10Huggle: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted - https://phabricator.wikimedia.org/T157710#3077610 (10Petrb) Don't instances in huggle project host some essential services [21:30:51] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [21:34:51] 10Tool-Labs-tools-Pageviews: Add Mediaviews to Pageviews suite - https://phabricator.wikimedia.org/T149642#3077617 (10harej-NIOSH) For whatever reason, the cron job I had set up to run the ingest script on a daily basis did not work. So I just run the script every time I need data, based on the last time the rep... [21:50:48] annika: good deal, let us know how it's going [22:09:56] 10Wikibugs: "jerkins-bot" - https://phabricator.wikimedia.org/T159759#3077729 (10Luke081515) [22:10:11] 06Labs, 10Huggle: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted - https://phabricator.wikimedia.org/T157710#3077744 (10Andrew) @petrb I can't tell what you're saying due to lack of punctuation... maybe "Don't! Instances in huggle project host some essential services!" or maybe "Don't ins... [22:10:42] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#3077746 (10chasemp) [22:12:25] 10Wikibugs, 06Operations: "jerkins-bot" - https://phabricator.wikimedia.org/T159759#3077768 (10greg) I noticed this recently, too. I presume it's a hack to make #operations team feel better about their -1s :) Only happens in their channel, afaict. [22:13:34] 10Wikibugs, 06Operations: "jerkins-bot" - https://phabricator.wikimedia.org/T159759#3077729 (10yuvipanda) I built the first version of this, and it didn't work. Someone fixed it later :D [22:14:15] 10Wikibugs, 06Operations: "jerkins-bot" - https://phabricator.wikimedia.org/T159759#3077729 (10Dzahn) i assumed jerkins-bot is just the alternative nickname it falls back to when jenkins-bot is taken, for example by its own ghost in case of netsplits or so.. [22:15:30] 10Wikibugs, 06Operations: "jerkins-bot" - https://phabricator.wikimedia.org/T159759#3077801 (10Luke081515) Hm, but I think "jerkins-bot" in this case means the gerrit-user, not the IRC-Nick? The IRC nick of the bot is currently wikibugs_, so I don't think this is related to nick-conflicts at IRC...? [22:15:49] 10Wikibugs, 06Operations: "jerkins-bot" - https://phabricator.wikimedia.org/T159759#3077802 (10greg) >>! In T159759#3077778, @yuvipanda wrote: > I built the first version of this, and it didn't work. Someone fixed it later :D where is "it"? [22:16:21] 10Wikibugs, 06Operations: "jerkins-bot" - https://phabricator.wikimedia.org/T159759#3077804 (10Dzahn) Yea, it's not related. Ignore that comment (deleted). [22:17:57] 10Wikibugs, 06Operations: "jerkins-bot" - https://phabricator.wikimedia.org/T159759#3077810 (10greg) >>! In T159759#3077802, @greg wrote: >>>! In T159759#3077778, @yuvipanda wrote: >> I built the first version of this, and it didn't work. Someone fixed it later :D > > where is "it"? nvm, see it: https://phab... [22:18:37] 10Wikibugs, 06Operations: "jerkins-bot" - https://phabricator.wikimedia.org/T159759#3077813 (10greg) 05Open>03declined On purpose. Humor is good. Moving along. :) [22:51:52] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:52:54] !log tools migrating tools-webgrid-lighttpd-1411 to labvirt1011 to give labvirt1001 a break [22:52:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:53:20] Whats wrong with labvirt1001 [22:55:29] PROBLEM - Host tools-webgrid-lighttpd-1411 is DOWN: CRITICAL - Host Unreachable (10.68.17.51) [23:06:53] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [23:14:46] Zppix: nothing much, it's just very busy [23:15:30] RECOVERY - Host tools-webgrid-lighttpd-1411 is UP: PING OK - Packet loss = 0%, RTA = 2.86 ms