[00:19:53] bd808: TParis getting user rights is not that hard. I do it with utrs nowl. Im at work tho so i cant talk much now [01:36:56] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Output data for new XTools: Top edits - https://phabricator.wikimedia.org/T160139#3096985 (10Samwilson) PR created: https://github.com/x-tools/xtools-rebirth/pull/7 [02:21:42] 10Tool-Labs-tools-Other: Add Read me to repository tool-editathonstat - https://phabricator.wikimedia.org/T159818#3097012 (10Ranjithsiji) Actually I am trying to create a repo in phabricator for the tool tools.wmflabs.org/editathonstat but I have no access to do anything. That is why created this. Any Idea how t... [03:36:12] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Build new front-end for xtools-articleinfo - https://phabricator.wikimedia.org/T159395#3097041 (10MusikAnimal) [03:36:56] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Build new front-end for xtools-articleinfo - https://phabricator.wikimedia.org/T159395#3066378 (10MusikAnimal) a:03MusikAnimal [03:37:22] 10Tool-Labs-tools-Xtools, 06Community-Tech: [Epic] Rewrite XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3097045 (10MusikAnimal) [03:37:26] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Build new front-end for xtools-articleinfo - https://phabricator.wikimedia.org/T159395#3066378 (10MusikAnimal) [03:37:47] 10Tool-Labs-tools-Xtools, 06Community-Tech: [Epic] Rewrite XTools: Top edits - https://phabricator.wikimedia.org/T160137#3097046 (10Samwilson) a:03Samwilson [03:38:23] 10Tool-Labs-tools-Xtools, 06Community-Tech: [Epic] Rewrite XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3010525 (10MusikAnimal) a:03MusikAnimal [03:47:05] 10Tool-Labs-tools-Xtools, 06Community-Tech: [Epic] Rewrite XTools: Top edits - https://phabricator.wikimedia.org/T160137#3097052 (10MusikAnimal) [03:47:09] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Output data for new XTools: Top edits - https://phabricator.wikimedia.org/T160139#3097049 (10MusikAnimal) 05Open>03Resolved All looks good to me, PR merged. [03:52:34] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Output data for new XTools: Articleinfo - https://phabricator.wikimedia.org/T157706#3097053 (10MusikAnimal) The code I worked on included a LOT of stuff outside articleinfo, so I kept merging into master so it can be used elsewhere and also to avoid edit confl... [04:13:26] 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3097074 (10Andrew) Are there still pending tasks here, or is this resolved? [07:17:45] 06Labs, 06Operations: labtestcontrol2001: cron-spam from invoke-rc.d atop _cron - https://phabricator.wikimedia.org/T159532#3097164 (10elukey) [07:22:56] 06Labs, 06Operations, 10Traffic, 07Puppet, 07Technical-Debt: Convert all of our site.pp/roles to the role/profile paradigm - https://phabricator.wikimedia.org/T159412#3097172 (10Joe) @Ciencia_Al_Poder care to explain why did you remove the "easy" tag? In general, I'd like to see a comment explaining act... [10:43:14] 06Labs, 06Operations, 10Traffic, 07Puppet, 07Technical-Debt: Convert all of our site.pp/roles to the role/profile paradigm - https://phabricator.wikimedia.org/T159412#3066827 (10Ciencia_Al_Poder) @Joe I found the easy tag is not suitable/applicable to this task, that's why I removed it Feel free to poke... [10:45:13] 06Labs, 10Analytics, 10DBA: Discuss labsdb visibility of rev_text_id and ar_comment - https://phabricator.wikimedia.org/T158166#3097467 (10JAllemandou) I discussed this with @ArielGlenn. He told me he wouold investigate. Ping @ArielGlenn? [10:47:03] 06Labs, 06Operations, 10Traffic, 07Puppet, 07Technical-Debt: Convert all of our site.pp/roles to the role/profile paradigm - https://phabricator.wikimedia.org/T159412#3066827 (10MoritzMuehlenhoff) I'd say let's add a few gerrit links of style conversions which have already landed to the task description,... [11:20:27] 06Labs, 10MediaWiki-User-login-and-signup, 10wikitech.wikimedia.org: Fatal exception when attempting to log into Wikitech - https://phabricator.wikimedia.org/T160171#3097519 (10Aklapper) [12:40:02] Cyberpower678: URGENT please reply when your able [12:42:50] 06Labs, 06Operations: Remove linux kernel 3.16 from the jessie image on labs - https://phabricator.wikimedia.org/T159990#3097660 (10faidon) 05Open>03Invalid Sounds a lot like an [[ http://xyproblem.info/ | XY problem ]] in general, please avoid opening tasks like that :) In addition to what has been menti... [12:52:28] Zppix: what's so urgent [13:47:16] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team: Commons shortlinks not supported - https://phabricator.wikimedia.org/T157434#3097829 (10Tobi_WMDE_SW) @jakob_wmde @wmde-leszek @wmde-fisch any idea how much effort it would be to add this support? is this something that could just be done by adjusting some r... [13:50:59] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team: Commons shortlinks not supported - https://phabricator.wikimedia.org/T157434#3097835 (10WMDE-leszek) Should be fairly simple I believe @Tobi_WMDE_SW ! [14:09:42] 06Labs, 10Horizon, 07Developer-notice, 13Patch-For-Review: Upgrade Openstack Horizon to Mitaka - https://phabricator.wikimedia.org/T158099#3097855 (10Andrew) 05Open>03Resolved This is done on Californium and seems fine. [14:13:52] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:22:38] 06Labs: ldap userkeys broken on labtest - https://phabricator.wikimedia.org/T152518#3097908 (10Andrew) 05Open>03Invalid I just tinkered with my .ssh/config and now this works fine. [14:23:42] 06Labs, 06Operations: Add lock_wait_timeout to maintain_views and maintain-meta_p - https://phabricator.wikimedia.org/T160412#3097911 (10chasemp) [14:23:54] 06Labs, 06Operations: Add lock_wait_timeout to maintain_views and maintain-meta_p - https://phabricator.wikimedia.org/T160412#3097923 (10chasemp) p:05Triage>03Normal [14:49:32] 06Labs, 10DBA: page_lang column of the page table is not replicated to Labs - https://phabricator.wikimedia.org/T154355#3098013 (10chasemp) a:05chasemp>03TTO As far as I know this is all deployed as intended now, please validate. [14:53:53] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [15:03:34] !log tools Shutting down webservices running on Precise job grid nodes [15:03:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:03:48] AmandaNP: ping (think you're heading up UTRS development?) [15:03:54] failing that, any UTRS devs aboot? [15:04:58] bd808: *taps plays* [15:06:09] and... the script did not work :/ [15:06:18] * bd808 digs in to find out why [15:07:39] and the answer is... `webservice shutdown` is not the proper command. `webservice stop` is [15:07:49] * bd808 tries again [15:07:55] :D [15:11:52] only 4 things still active on Precise exec nodes :) [15:12:49] chasemp: we are probably ready for https://gerrit.wikimedia.org/r/#/c/342161/ to be merged in ops/puppet [15:13:09] (03CR) 10BryanDavis: [C: 032] jsub: Remove support for release=precise [labs/toollabs] - 10https://gerrit.wikimedia.org/r/341666 (https://phabricator.wikimedia.org/T94792) (owner: 10BryanDavis) [15:13:17] ok I'll do that today if I can [15:13:26] may be thu as I'm prepping for the maint tomorrow + meetings etc [15:14:04] ok. it's going to start blowing up in about 15 minutes when I roll out the new jsub and webservice packages [15:14:17] no worries I'll silence quick [15:14:22] *nod* [15:14:34] (03Merged) 10jenkins-bot: jsub: Remove support for release=precise [labs/toollabs] - 10https://gerrit.wikimedia.org/r/341666 (https://phabricator.wikimedia.org/T94792) (owner: 10BryanDavis) [15:23:09] !log tools Installed jobutils 1.21 on tools-bastion-02 [15:23:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:25:16] !log tools Installing jobutils 1.21 across cluster using clush [15:25:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:32:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:35:00] 06Labs, 10Recommendation-API: Request increased quota for recommendation-api labs project - https://phabricator.wikimedia.org/T160344#3095525 (10chasemp) +1 , though we should come up with some way to circle back on these adhoc big allocations to reclaim them down the line. Good luck with your work :) [15:36:29] !log tools Upgraded toollabs-webservice to 0.36 on tools-bastion-02.tools [15:36:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:38:29] 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#3098215 (10Andrew) [15:38:31] 06Labs, 10Recommendation-API: Request increased quota for recommendation-api labs project - https://phabricator.wikimedia.org/T160344#3098212 (10Andrew) 05Open>03Resolved a:03Andrew I've increased your quotas to allow one additional 'bigram' instance. Let me know if I missed anything. [15:39:23] 10Labs-project-other: Successful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#3098222 (10Nemo_bis) 05Open>03Invalid The instance has been inactive for several months now (cf. [[https://web.archive.org/web/2017031... [15:40:34] !log tools Installing toollabs-webservice 0.36 across cluster using clush [15:40:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:44:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:47:34] bd808: Has the tools elasticsearch been set up in a special way? I.e. deviated much from the configuration in puppet? I asked you for the logs from it because I was getting error 502's back (and it seemed like I was exhausting TCP sockets). After much faffing and slowness on my side I set up an instance of role:toollabs:elasticsearch on labs and I can't seem [15:47:35] to replicate the issue despite hitting it with queries for >12hrs straight. [15:49:26] tarrow: it is all setup by the ops/puppet role. nothing special tuned by hand [15:49:49] there are other users though (mostly Stashbot) [15:50:21] ah, very interesting. I'll keep poking then; I wonder what the difference could be [15:51:28] Did you figure out how to get your program to do batch inserts? [15:52:37] yep; but it didn't solve it. I also looked at the client a bit and I think it should be using http keepalive for each connection but it doesn't seem to be [15:54:56] I should really have got back to you about it sooner but I ended up charging off and trying to host my own instance on labs to debug the problem which took a lot longer to set up than I had expected [15:56:05] At the moment things seem to be working fine using my small 1 node cluster on my labs project but it would be nice to know why it isn't working on tools [16:07:15] RECOVERY - Puppet run on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [16:24:08] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [16:56:06] bd808: so apparently my tools died because the precise instances no longer exist. A little warning wouldn't have hurt. XD [16:56:14] jk. [16:56:24] * bd808 is not amused [16:56:32] :( [16:56:47] bd808: long day? [16:57:33] often sarcasm and trolling are indistinguishable. [16:58:00] bd808: It wasn't supposed to be either. [16:58:27] bd808: It was supposed to poke fun that I didn't switch over until the instances were taken down. [16:59:07] I guess my sense of humor is not compatible with yours. [16:59:08] TheresNoTime: [17:00:47] anomie: are available to chat? It's not really urgent. [17:01:08] AmandaNP: hi, I've emailed the utrs email address [17:01:34] But tl;dr is after rename OAuth has done a heck [17:01:48] yep, just saw it now [17:02:00] there is an issue open for that, we just haven't gotten to it yet [17:02:05] * AmandaNP pulls up the db [17:02:13] Guessing email is a key in the db? [17:02:39] yep [17:02:45] Pre and post account having the same email = bad times [17:03:03] ya cause there are no duplicates set [17:03:21] so I just have to add a SQL statement to update email instead of try new account [17:03:45] Hmmm. I get the feeling I've been unknowingly pissing off everyone. :( [17:03:49] Sounds good :) thanks for looking at it [17:04:05] * TheresNoTime gives Cyberpower678 a toasted sandwich [17:04:15] Or my sense of timing is the worst [17:04:42] TheresNoTime: thanks. Is it from your sandwhich maker. :) [17:05:19] No I just found it [17:05:23] Only lightly nibbled [17:05:26] Should be fine [17:05:50] Cyberpower678: I'm somewhat busy at the moment. What's it about? [17:05:54] I'm full. [17:06:00] anomie: OAuth stuff. [17:06:24] anomie: if you can I would like to PM [17:07:13] TheresNoTime: try now [17:07:39] Cyberpower678: It might be best if you post your questions to an appropriate mailing list. [17:08:36] anomie: well one of them is somewhat private which I would like to communicate privately over. The other one is that edits that take longer than 10 seconds timeout and return an OAuth error on the API. [17:09:18] What exactly is the OAuth error? [17:10:31] I believe it's the nonce already used error. [17:11:03] If the edit takes longer than 10 seconds to commit, it will time out at the 10 second mark with said error. [17:11:57] Cyberpower678: Are you sure your library isn't timing out after 10 seconds and trying to resend the request? [17:12:28] The library is set to timeout after 300 seconds. [17:12:38] But let me double check. [17:14:33] Oh derp. It looks like it was changed back to 10. I feel like an idiot. [17:15:21] anomie: ^ sorry for wasting your time. [17:15:40] regarding the other matter, we can discuss that when you have more time. [17:23:47] !log tools Remove role::toollabs::precise_reminder from tools-bastion-03 [17:23:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:41:45] !log tools Hand fix tools-puppetmaster by removing the old mariadb submodule directory [17:41:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:54:10] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-webgrid-lighttpd-12* hosts - https://phabricator.wikimedia.org/T160442#3098820 (10bd808) [17:56:30] AmandaNP: all working, thank you! :D [17:56:50] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-webgrid-lighttpd-12* hosts - https://phabricator.wikimedia.org/T160442#3098845 (10bd808) ``` tools-bastion-02.tools:~ bd808$ sudo qmod -d '*@tools-webgrid-lighttpd-1201.eqiad.wmflabs' Queue instance "webgrid-lighttpd@tools-webgrid-lighttpd-1201.eqiad.... [18:07:18] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-webgrid-lighttpd-12* hosts - https://phabricator.wikimedia.org/T160442#3098884 (10bd808) Removed `tools-webgrid-lighttpd-12*.eqiad.wmflabs` from `qconf -mhgrp @webgrid` Also verified that none of these hosts were listed directly in any queues listed b... [18:08:14] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-webgrid-lighttpd-12* hosts - https://phabricator.wikimedia.org/T160442#3098904 (10bd808) ``` tools-bastion-02.tools:~ bd808$ sudo qconf -de tools-webgrid-lighttpd-1201.eqiad.wmflabs root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-webgrid-ligh... [18:14:21] 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: overhaul labstore setup [tracking] - https://phabricator.wikimedia.org/T126083#3098923 (10madhuvishy) [18:14:24] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4 - https://phabricator.wikimedia.org/T149870#3098921 (10madhuvishy) 05Open>03Resolved Looks like the backup jobs are running fine. Closing this. [18:44:10] 10Tool-Labs-tools-Xtools, 06Community-Tech: Output data for new XTools: Top edits - https://phabricator.wikimedia.org/T160139#3099123 (10DannyH) [19:09:22] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-exec-12* hosts - https://phabricator.wikimedia.org/T160457#3099266 (10bd808) [19:09:43] 06Labs, 10DBA, 10wikitech.wikimedia.org: SemanticMediaWiki tries to create temporary tables, but can't as wikiuser is restricted - https://phabricator.wikimedia.org/T110981#3099283 (10Bawolff) 05Open>03declined > If this was me, I would close it as won't fix You are the DBA, if the status of this ticket... [19:10:29] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-exec-12* hosts - https://phabricator.wikimedia.org/T160457#3099289 (10bd808) Disable queues: ``` tools-bastion-02.tools:~ bd808$ sudo qmod -d '*@tools-exec-1217.eqiad.wmflabs' root@tools-bastion-02.tools.eqiad.wmflabs changed state of "continuous@tool... [19:17:06] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-exec-12* hosts - https://phabricator.wikimedia.org/T160457#3099322 (10bd808) Kill running jobs: ``` tools-bastion-02.tools:~ bd808$ sudo qdel $(qhost -j -h tools-exec-1217.eqiad.wmflabs^C tools-bastion-02.tools:~ bd808$ sudo qdel $(qhost -j -h tools-e... [19:20:57] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-exec-12* hosts - https://phabricator.wikimedia.org/T160457#3099340 (10bd808) Remove from hostgroups using `sudo qconf -mhgrp @general`. Also verified that none of these hosts were listed directly in any queues listed by `qconf -sql`. [19:23:52] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-webgrid-lighttpd-12* hosts - https://phabricator.wikimedia.org/T160442#3099383 (10bd808) ``` tools-bastion-02.tools:~ bd808$ sudo qconf -ds tools-webgrid-lighttpd-1201.eqiad.wmflabs root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-webgrid-ligh... [19:24:58] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-exec-12* hosts - https://phabricator.wikimedia.org/T160457#3099385 (10bd808) Remove nodes from grid engine: ``` tools-bastion-02.tools:~ bd808$ sudo qconf -de tools-exec-1217.eqiad.wmflabs root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-exec-... [19:38:00] yuvipanda, chasemp: hey! thanks for the k8s help the other day. made a lot of progress [19:38:23] i'm hitting another small snag if you're around to help [19:39:13] kube-apiserver fails to start and `journalctl -u kube-apiserver` tells me "Invalid Authentication Config: open /etc/kubernetes/tokenauth: no such file or directory" [19:40:24] i see where tokenauth is referenced in the puppet k8s templates, but not where it's contents are sourced or how it might be generated [19:44:39] ah, toollabs::maintain_kubeusers looks to hold clues [19:44:42] * marxarelli investigates [19:54:39] 06Labs, 10wikitech.wikimedia.org: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3099461 (10Bawolff) Ok, so current uses of SMW+friends on Wikitech is: For querying ------- * **Analytics/EventLogging**: Calls `{{#ask: [[Category:EventLogging/Incident... [20:06:26] 06Labs, 10Tool-Labs, 15User-bd808: Decommission tools-exec-gift.eqiad.wmflabs - https://phabricator.wikimedia.org/T160461#3099497 (10bd808) [20:13:48] 06Labs, 10Tool-Labs, 15User-bd808: Decommission tools-exec-gift.eqiad.wmflabs - https://phabricator.wikimedia.org/T160461#3099545 (10bd808) ``` tools-bastion-02.tools:~ bd808$ sudo qmod -d '*@tools-exec-gift.eqiad.wmflabs' root@tools-bastion-02.tools.eqiad.wmflabs changed state of "giftbot@tools-exec-gift.eq... [20:24:49] 06Labs, 10DBA: page_lang column of the page table is not replicated to Labs - https://phabricator.wikimedia.org/T154355#3099624 (10TTO) `metawiki_p.page` now contains the page_lang column; however, `user_groups` view still dos not contain the `ug_expiry` column. Shall I open a new task for that? [20:27:08] !log tools Disassociated floating IPs from tools-exec-12* nodes (T160457) [20:27:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:27:12] T160457: Decommission all tools-exec-12* hosts - https://phabricator.wikimedia.org/T160457 [20:27:41] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Review OpenStack monitoring options w/out Mirantis packages - https://phabricator.wikimedia.org/T157760#3099633 (10Andrew) 05Open>03Resolved I moved the related tests to labnet and nrpe -- they seem to be working fine. [20:28:58] PROBLEM - Host tools-exec-1218 is DOWN: CRITICAL - Host Unreachable (10.68.18.19) [20:29:08] !log tools Deleted tools-exec-12* nodes (T160457) [20:29:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:29:36] PROBLEM - Host tools-exec-1217 is DOWN: CRITICAL - Host Unreachable (10.68.18.20) [20:29:57] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, and 2 others: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3099647 (10bd808) [20:30:00] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-exec-12* hosts - https://phabricator.wikimedia.org/T160457#3099646 (10bd808) 05Open>03Resolved [20:30:59] PROBLEM - Host tools-exec-1220 is DOWN: CRITICAL - Host Unreachable (10.68.16.38) [20:31:00] PROBLEM - Host tools-exec-1219 is DOWN: CRITICAL - Host Unreachable (10.68.18.40) [20:31:24] These shinken alerts are expected ^ [20:31:38] PROBLEM - Host tools-exec-1221 is DOWN: CRITICAL - Host Unreachable (10.68.16.84) [20:31:39] they will clear when shinken figures out that the hosts are gone [20:37:27] PROBLEM - Host tools-webgrid-lighttpd-1205 is DOWN: CRITICAL - Host Unreachable (10.68.18.48) [20:38:02] :D congratulations, bd808 [20:38:52] yuvipanda: thanks. As you know it has been a group effort :) [20:39:03] :D [20:39:08] congratulations, group of people! [20:39:18] bd808: expecting any pitchforks? [20:39:27] not at this point [20:39:32] * valhallasw`cloud hands yuvipanda a torch [20:39:42] Hurray, no more Precise \o/ [20:39:46] we only had 4 non-webservice jobs still running this morning [20:39:51] nice! [20:40:09] and the ~45 webservices look to be mostly abandoned [20:40:56] I should check to make sure that some bigbrother script isn't flipping out. Where does that run? I forget. [20:41:02] PROBLEM - Host tools-webgrid-lighttpd-1209 is DOWN: CRITICAL - Host Unreachable (10.68.17.152) [20:41:18] PROBLEM - Host tools-webgrid-lighttpd-1201 is DOWN: CRITICAL - Host Unreachable (10.68.18.45) [20:41:28] PROBLEM - Host tools-webgrid-lighttpd-1203 is DOWN: CRITICAL - Host Unreachable (10.68.18.47) [20:41:29] tools-services-XX I think, but it might be tools-cron-XX as well [20:41:30] PROBLEM - Host tools-webgrid-lighttpd-1210 is DOWN: CRITICAL - Host Unreachable (10.68.17.163) [20:41:40] PROBLEM - Host tools-webgrid-lighttpd-1204 is DOWN: CRITICAL - Host Unreachable (10.68.18.49) [20:41:42] PROBLEM - Host tools-webgrid-lighttpd-1208 is DOWN: CRITICAL - Host Unreachable (10.68.16.239) [20:42:35] PROBLEM - Host tools-webgrid-lighttpd-1207 is DOWN: CRITICAL - Host Unreachable (10.68.16.215) [20:42:35] modules/role/manifests/toollabs/services.pp: class { '::toollabs::bigbrother': [20:42:42] so I vote for tools-services :-) [20:42:49] looks like the live one is on services-02 [20:43:01] PROBLEM - Host tools-webgrid-lighttpd-1206 is DOWN: CRITICAL - Host Unreachable (10.68.18.54) [20:45:08] !log tools Deleted tools-webgrid-lighttpd-12* nodes (T160442) [20:45:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:45:13] T160442: Decommission all tools-webgrid-lighttpd-12* hosts - https://phabricator.wikimedia.org/T160442 [20:45:38] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, and 2 others: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3099700 (10bd808) [20:45:40] 06Labs, 10Tool-Labs, 15User-bd808: Decommission all tools-webgrid-lighttpd-12* hosts - https://phabricator.wikimedia.org/T160442#3099699 (10bd808) 05Open>03Resolved [20:47:49] the bigbrother process hasn't logged anything for 10 days. slightly surprising, but I don't know how many things are still using it [20:47:57] PROBLEM - Host tools-webgrid-lighttpd-1202 is DOWN: CRITICAL - Host Unreachable (10.68.18.46) [20:52:06] 06Labs, 10Tool-Labs: Decommission tools-precise-dev - https://phabricator.wikimedia.org/T160466#3099719 (10bd808) [20:54:28] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, 15User-bd808: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3099756 (10bd808) [21:02:54] PROBLEM - Host tools-exec-gift is DOWN: CRITICAL - Host Unreachable (10.68.16.40) [21:02:57] !log tools Deleted tools-exec-gift (T160461) [21:03:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:03:02] T160461: Decommission tools-exec-gift.eqiad.wmflabs - https://phabricator.wikimedia.org/T160461 [21:03:28] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, 15User-bd808: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3099847 (10bd808) [21:03:30] 06Labs, 10Tool-Labs, 15User-bd808: Decommission tools-exec-gift.eqiad.wmflabs - https://phabricator.wikimedia.org/T160461#3099846 (10bd808) 05Open>03Resolved [21:11:01] 06Labs, 10wikitech.wikimedia.org: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#545071 (10demon) >>! In T53642#3099461, @Bawolff wrote: > * **Help:MediaWiki-Vagrant in Labs/Hosts**: Doesn't even seem to work Killed this. Generally: let's make this... [21:11:34] 06Labs, 10Tool-Labs: Decommission tools-precise-dev - https://phabricator.wikimedia.org/T160466#3099875 (10bd808) ``` tools-bastion-02.tools:~ bd808$ sudo qconf -ds tools-precise-dev.eqiad.wmflabs root@tools-bastion-02.tools.eqiad.wmflabs removed "tools-precise-dev.eqiad.wmflabs" from submit host list ``` [21:13:16] !log tools Removed non-existent tools-submit.eqiad.wmflabs from submit hosts list [21:13:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:17:52] 06Labs, 10wikitech.wikimedia.org: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3099909 (10Bawolff) [21:21:29] PROBLEM - Host tools-precise-dev is DOWN: CRITICAL - Host Unreachable (10.68.16.31) [21:24:20] !log tools Deleted tools-precise-dev (T160466) [21:24:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:24:25] T160466: Decommission tools-precise-dev - https://phabricator.wikimedia.org/T160466 [21:40:30] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Output data for new XTools: Articleinfo - https://phabricator.wikimedia.org/T157706#3100071 (10kaldari) Looks good except that the Bugs and Assessments sections are missing. The bugs should be pulled from http://tools.wmflabs.org/checkwiki/ and the assessments... [21:42:41] Does ENWP not have OAuth? I have to authenticate to Meta? [21:43:21] TParis: it's fine to authenticate wherever, as long as the application is authorized for that wiki [21:44:26] TParis: you have to create the grant on meta, then your app can auth against any wiki in the farm. [21:44:28] Enwp does have oauth but since centralauth is used it will go to meta [21:44:39] Zppix: not quite right [21:44:53] Not centralauth my bad [21:45:07] I meant auth manager i believe [21:46:04] auth manager is the generic auth framework in MediaWiki, CentralAuth is the specific global auth handler that that Wikimedia uses. Neither have anything of substance to do with OAuth [21:46:53] Oh? I thought they did [21:46:59] We have centralized management of OAuth grants on metawiki, but all wikis can authorize a grant request [21:47:28] use of metawiki is a common convention for things that are cross-wiki [21:49:48] bd808: Thanks. I'll try to figure out what else could be wrong then [21:50:36] bd808: Can I PM you? [21:50:47] TParis: sure [22:38:45] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3100170 (10chasemp) [22:40:39] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3100177 (10chasemp) https://etherpad.wikimedia.org/p/T152399 [22:55:36] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#3100244 (10bd808) [23:01:07] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3100260 (10EddieGP) [23:02:39] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#3100288 (10bd808) [23:02:41] 06Labs, 10Tool-Labs, 15User-bd808: Decommission tools-precise-dev - https://phabricator.wikimedia.org/T160466#3100286 (10bd808) 05Open>03Resolved a:03bd808 [23:03:34] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#1172717 (10bd808) [23:03:37] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, 15User-bd808: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3100294 (10bd808) 05Open>03Resolved [23:04:48] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#3100297 (10bd808) [23:17:11] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#3100336 (10bd808) a:03yuvipanda We are `{{done}}`! I'm assigning to @yuvipanda so that he can have the satisfaction of closing this task that once looked like more work than we could possibly... [23:39:28] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Plan for rewriting XTools - https://phabricator.wikimedia.org/T154551#3100374 (10DannyH) [23:40:20] 10Tool-Labs-tools-Xtools, 06Community-Tech: Investigation: Plan for rewriting XTools - https://phabricator.wikimedia.org/T154551#2915501 (10DannyH) [23:41:55] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3100382 (10DannyH) [23:45:36] 10Tool-Labs-tools-Xtools, 06Community-Tech: Have Edit Counter use same architecture and front-end as the other pieces that have been re-written (articleinfo and topedits) - https://phabricator.wikimedia.org/T160481#3100408 (10kaldari) [23:46:08] 10Tool-Labs-tools-Xtools, 06Community-Tech: Have Edit Counter use same architecture and front-end as the other pieces that have been re-written - https://phabricator.wikimedia.org/T160481#3100408 (10kaldari) p:05Triage>03Normal [23:48:13] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Have Edit Counter use same architecture and front-end as the other pieces that have been re-written - https://phabricator.wikimedia.org/T160481#3100424 (10kaldari)