[00:03:15] 10Tool-Labs-tools-Xtools, 06Community-Tech: Ensure xTools Rebirth is fully responsive - https://phabricator.wikimedia.org/T165706#3274440 (10Matthewrbowker) [00:09:48] 10Tool-Labs-tools-Xtools, 06Community-Tech: Fix "Notice: Undefined index: allusers" in Adminstats when the wiki is unreachable - https://phabricator.wikimedia.org/T165707#3274469 (10Matthewrbowker) [00:11:07] 10Tool-Labs-tools-Xtools, 06Community-Tech: Convert xtools intuition to its own repository - https://phabricator.wikimedia.org/T165708#3274486 (10Matthewrbowker) [00:13:58] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewrite Xtools: RfX Analysis - https://phabricator.wikimedia.org/T165709#3274500 (10Matthewrbowker) [00:15:13] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewrite Xtools: RfX Vote Calculator - https://phabricator.wikimedia.org/T165710#3274514 (10Matthewrbowker) [06:33:16] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:37:25] PROBLEM - Puppet errors on tools-exec-1441 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:49:30] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:08:16] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [07:12:25] RECOVERY - Puppet errors on tools-exec-1441 is OK: OK: Less than 1.00% above the threshold [0.0] [07:24:30] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [08:13:14] 10Quarry: No data after 20170517193000 available via Quarry from tables (recentchanges, revision, logging) for several Mediawiki databases (svwiki_p, fiwiki_p, nowiki_p, ...) - https://phabricator.wikimedia.org/T165705#3274765 (10Larske) [09:12:46] 06Labs, 06Project-Admins, 15User-bd808: Create wmcs-team project and kanban milestone - https://phabricator.wikimedia.org/T165703#3275051 (10Peachey88) [09:31:50] http://i.imgur.com/LiaQ4U2.png [09:32:35] What does that mean? [09:42:36] What does that even mean? Why can't I even find a manual about it? How do I keep encounting this problem? Please help me, it's already 10 minutes say something ORZ [09:42:42] http://i.imgur.com/LiaQ4U2.png [09:46:37] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team, 15User-Tobi_WMDE_SW: Commons shortlinks not supported - https://phabricator.wikimedia.org/T157434#3275129 (10WMDE-leszek) a:03WMDE-leszek [09:48:48] 10Quarry: No data after 20170517193000 available via Quarry from tables (recentchanges, revision, logging) for several Mediawiki databases (svwiki_p, fiwiki_p, nowiki_p, ...) - https://phabricator.wikimedia.org/T165705#3275136 (10jcrespo) 05Open>03Resolved a:03jcrespo You can check the replication lag at... [09:51:14] 10Quarry: No data after 20170517193000 available via Quarry from tables (recentchanges, revision, logging) for several Mediawiki databases (svwiki_p, fiwiki_p, nowiki_p, ...) - https://phabricator.wikimedia.org/T165705#3275144 (10jcrespo) Note that only 1 server (c1) was affected. c2 was unaffected, and that co... [09:51:41] 06Labs, 10wikitech.wikimedia.org: Can we search namespaces on wikitech? - https://phabricator.wikimedia.org/T165725#3275146 (10Andrew) [09:52:03] Any human here?Why can my question stay there aleast 20 minutes and no one see that? [09:52:05] 06Labs, 10wikitech.wikimedia.org: Can we search namespaces on wikitech? - https://phabricator.wikimedia.org/T165725#3275160 (10Andrew) [09:53:34] http://i.imgur.com/LiaQ4U2.png [09:54:44] Can someone tell me what does that mean? It's already 20 minutes, any real keyboard-using creatrue here ? [09:59:08] Can someone aleast tell me why is side bar shows a tons of people but anyone say something to me? [10:02:21] http://i.imgur.com/LiaQ4U2.png [10:02:30] Again, can someone tell me what does that mean? [10:02:38] r96340: hello. [10:03:06] That error means that your ssh client and the server are not agreeing on who you are. [10:03:48] have you just recently setup your account, or did it work before and is now broken? [10:03:48] How can I fix it? [10:04:42] r96340: often that error is if your local username is different from your labs shell name. I'm not familiar with putty in particular but there's probably a 'username' field you can specify someplace [10:04:49] (That's just a guess though, could be lots of other issues) [10:05:00] r96340: have you created an ssh key pair and uploaded you public key using either wikitech or toolsadmin? [10:05:32] I believe that the intro docs for access via putty are https://wikitech.wikimedia.org/wiki/Help:Putty (maybe you're reading that page already) [10:05:51] r96340: if you are by chance at the hackathon then we can help you in person :) [10:06:07] You haven't added the private key to putty, I think. [10:06:28] I setup this account at eleven days ago, and I did had upload a ssh key. [10:06:42] https://wikitech.wikimedia.org/w/images/7/7f/20130526_2133_Putty_Login_Connection_SSH_Auth.png [10:08:42] *Sigh* [10:09:33] I did forgot to adding my priviate key to putty. Thanks everyone! [10:10:50] Taiwanese here. Sorry for bad English XD. Bye! [10:32:11] is there a walltime limit on tools labs? [10:33:47] 06Labs, 10wikitech.wikimedia.org: Can we search namespaces on wikitech? - https://phabricator.wikimedia.org/T165725#3275504 (10Andrew) From a quick conversation with @EBernhardson @dcausse, it sounds like wikitech search is already tuned in a few specific ways: Things under the "nova resource" namespace have... [10:34:28] bug off [10:36:48] gry: there is no limit on a job launched on the job grid. There are some limits on things running interactively from the bastion servers, but I don't recall that there is a strict wall/cpu throttle. [10:37:22] thanks; would you be able to check why 'gpy' tool keeps stopping? [10:39:59] gry: I can look at the logs in the tool's directory and see if anything stands out [10:40:23] yes, please; I've had it working properly until the migration to a new ubuntu version and I'm not on the top of how to work that out [10:40:31] it seems to work for a day or a few days and then quit [10:41:45] I don't see anything obvious in gpy.err. One thing that can abruptly kill a grid job is attempting to allocate more memory that the reservation that the job has on the grid. [10:41:59] how do i monitor memory ? [10:42:12] that's a fine question :) [10:42:25] there is a page that can give some info ... let me get the link [10:42:28] i think it should say 'Killed' in .err file in this case; it doesn't do that, does it? [10:42:33] thanks, bd808 :) [10:43:21] if you ctrl-f search in https://tools.wmflabs.org/?status for your job name you can see a point in time report of what the grid thinks is reserved and used for each job [10:43:57] the way that grid engine kills the jobs doesn't always show will in error logs [10:44:15] I think it just sends the process a SIGTERM or maybe even SIGKILL signal [10:45:15] gry: it looks like gpy is perl. anomie may know what to look for when a perl proc is killed by the grid [10:45:17] yes, SIGKILL; i can see why it is called a grid now :D; i don't know these units, what does "vmem 253/0" mean? [10:46:16] * bd808 has to read the source to remember if that is kilobytes or not [10:47:44] anomie: hi, i have gpy.nongnu.org sources running as 'gpy' project on tools labs and it stays up for about 2 days max but should stay forever; if you're willing to investigate i can add you to the tool (and/or would be glad to follow your advice). if I'm not on irc by then, you can email svetlana@members.fsf.org by then, or wait for me to come back, although it's rather chaotic [10:49:39] bd808: ta [10:49:45] brb [10:49:46] so grid engine tracks h_vmem in bytes internally. It looks to me like the display on ?staus is in megabytes [10:49:53] ok [10:49:55] rebooting now [10:50:55] gry: Sorry, I don't have time to debug your project for you. [10:52:15] anomie: do you gernally have any advice about how to tell if a perl job is being killed for hitting the grid memory limit? [10:53:01] When my bot gets killed, the standard error log gets "Out of memory" printed. You can also use qacct to see some useful info. [10:53:38] that's helpful. thanks [10:55:11] it means it's using 253 bytes of random access memory and 0 bytes of swap memory? [10:55:17] 10Quarry: No data after 20170517193000 available via Quarry from tables (recentchanges, revision, logging) for several Mediawiki databases (svwiki_p, fiwiki_p, nowiki_p, ...) - https://phabricator.wikimedia.org/T165705#3275551 (10Larske) Thanks for the prompt response with explanation on what was ongoing. The c... [10:56:32] gry: running `qacct -j gpy` is slow but will give you some reports about past execution of the job [10:56:59] I see most (all?) jobs ending with exit_status : 130 [10:57:33] err that's a bit cryptic [10:57:58] 130 is the bash exist status for Control-C [10:58:02] *exit [10:58:18] ooh bash.. i didn't ctrl+c it though [10:59:25] More specifically, it's the INT signal. Probably gridengine sent the signal for its own reasons. [11:00:15] hey anomie.. maybe i can log memory usage somehow in case the script is leaking, or kill and restart it automatically once every day? [11:00:17] looks like the maxvmem is consistently just over 500M. I think it's leaking memory and getting killed for that [11:00:42] 10Quarry: No data after 20170517193000 available via Quarry from tables (recentchanges, revision, logging) for several Mediawiki databases (svwiki_p, fiwiki_p, nowiki_p, ...) - https://phabricator.wikimedia.org/T165705#3275557 (10jcrespo) I highly recomend your code to integrate some kind of check for the heart... [11:00:45] ideally i think it would be nice to know where it is leaking from; 500MB you mean? that'd be a bit excessive [11:01:13] so the first thing you could try is asking for more max memory and see if it continues to grow or if it just needs a bit more space to become stable [11:01:23] in the job script? [11:01:47] i don't remember the synax, although i think 1GB of memory would be a bit excessive too :) [11:01:50] syntax [11:02:21] yes. jsub -m 1024M would ask for 1G [11:02:25] ok [11:02:38] you could also look into https://www.perl.org/about/whitepapers/perl-profiling.html [11:03:06] my perl skillz are old and rusty [11:04:04] thanks, i'll look at its behaviour with 1GB first and come back to troubleshooting if this doesn't fix it [11:04:30] good plan [11:14:50] iirc the grid memory thing is quite broken [11:16:01] it doesn't do limit via any sort of stuffs like cgroup or ulimit [11:16:24] as long as it knows that my job is leaking, it is good enough for me :) [11:24:16] bd808: no, -m is for email; maybe -M is for memory? [11:24:43] actually, -mem is it, i think :) [11:27:09] thanks again; laters :) [11:56:33] 06Labs, 10Beta-Cluster-Infrastructure, 07Puppet, 06Release-Engineering-Team (Next), 15User-Joe: Re-think puppet management for deployment-prep - https://phabricator.wikimedia.org/T161675#3275656 (10greg) [12:45:35] chasemp, andrewbogott: we've setup a test instance on http://wikitech-relforge.wmflabs.org/ to tune search [12:46:44] "creating a new node" (whihout quotes) now rank Nova Resource:Tools/Admin first by setting ns weight to 1, if you have other examples we can test them [12:49:57] dcausse: now I'm searching on the phrase 'building a new image' — the result seems useful in the old index and bad in the new index. [12:50:11] meh.. [12:50:14] In part because the SAL pages are polluting search now (and I can't decide if that's good or bad but probably bad) [12:50:30] looking [12:51:04] oh, except, the page that I'm looking for is also a page I just edited and renamed so maybe this is a bad example :( [12:51:58] andrewbogott: yes, I imported may 15 dumps so it might explain the problem [12:52:31] Yeah, I think this a bad test case. [12:52:44] But I also think that those SAL pages should probably be downranked. [12:54:45] sure, if we have a template for them we can problably add them to onwiki template boosting: https://wikitech.wikimedia.org/wiki/MediaWiki:Cirrussearch-boost-templates [12:56:33] hm.. no templates apparently but Category:SAL, sadly we don't have category boosting ready out of the box :/ [12:56:45] * andrewbogott was just about to say 'Category:SAL' [12:57:20] I think this chance is probably still an improvement so maybe we should just go with it for now and see how things shake out. chasemp, thoughts? [12:57:36] *this change [12:58:15] I'm good with seeing how the change fairs :) [12:58:19] fares? [12:59:04] sure, will prep a wmf-config change, we can always revert if it's desastrous [12:59:12] thank you! [13:10:10] On labs instances we're allowed to do cron arent we? [13:13:44] 06Labs, 10Labs-Infrastructure: labvirt1006 super busy right now - https://phabricator.wikimedia.org/T165753#3275963 (10Andrew) [13:33:03] 10Striker, 15User-bd808, 06wmcs-team (Kanban): Deploy striker on labtestweb2001 - https://phabricator.wikimedia.org/T156276#3276043 (10bd808) [13:33:47] 06Labs, 06Project-Admins, 15User-bd808, 06wmcs-team (Kanban): Create wmcs-team project and kanban milestone - https://phabricator.wikimedia.org/T165703#3276044 (10bd808) [13:34:28] 06Labs, 10Striker, 10Tool-Labs, 15User-bd808, 06wmcs-team (Kanban): Error saving OAuth credentials. [req id: f1a2370b1b8a4e1a8827de96b9bce144] bug - https://phabricator.wikimedia.org/T164847#3276049 (10bd808) [13:35:09] 06Labs, 10Tool-Labs, 15User-bd808, 06wmcs-team (Kanban): Modernize the admin tool's codebase - https://phabricator.wikimedia.org/T140254#3276051 (10bd808) [13:35:59] 06Labs, 10Tool-Labs, 15User-bd808, 06wmcs-team (Kanban): webservice stop says service not running but service.manifest not cleared - https://phabricator.wikimedia.org/T163355#3276055 (10bd808) [13:36:25] 06Labs, 15User-bd808, 06wmcs-team (Kanban): Consult with technical community on Cloud Services rebranding plan - https://phabricator.wikimedia.org/T165094#3276057 (10bd808) [13:36:51] 06Labs, 10Tool-Labs, 15User-bd808, 06wmcs-team (Kanban): Upgrade Tool Labs elasticsearch to 5.x - https://phabricator.wikimedia.org/T164842#3276058 (10bd808) [13:37:21] 06Labs, 10Tool-Labs, 15User-bd808, 06wmcs-team (Kanban): Broken unicode characters / invalid UTF-8 on Tool Labs index - https://phabricator.wikimedia.org/T164971#3276075 (10bd808) [13:52:20] 06Labs, 10Gerrit, 10wikitech.wikimedia.org: Request to rename LegoFan4000 to MacFan4000 on WikiTech - https://phabricator.wikimedia.org/T165624#3276168 (10demon) >>! In T165624#3274279, @bd808 wrote: > User `MacFan4000` already exists in LDAP. The account was created 2016-08-25T22:53:04Z. Both accounts are r... [13:56:46] (03CR) 10Jean-Frédéric: [C: 032] Prepare monument_tables for Wikidata and add tests [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342038 (owner: 10Lokal Profil) [13:58:32] (03Merged) 10jenkins-bot: Prepare monument_tables for Wikidata and add tests [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342038 (owner: 10Lokal Profil) [13:59:51] (03CR) 10jenkins-bot: Prepare monument_tables for Wikidata and add tests [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342038 (owner: 10Lokal Profil) [14:00:05] 06Labs, 10Tool-Labs: Adds a table with namespaces on Tools Labs DB replica - https://phabricator.wikimedia.org/T165763#3276214 (10Tpt) [14:06:45] Zppix: on instances within your own project yes, with Tool Labs no [14:06:48] use the grid for cron only [14:06:53] local cron is bad manners etc [14:13:08] 06Labs, 10Tool-Labs, 06translatewiki.net: update node.js on tools.telegrambot - https://phabricator.wikimedia.org/T159368#3276276 (10madhuvishy) @bd808 Amir and I just tried the tutorial and got telegrambot running on kubernetes, but it required an admin permission change for the .kube directory on /data/pro... [14:13:24] can someone point me at the documentation for generating a diffusion repo for a tools labs tool? I swear I've done it before.. [14:14:08] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team, 15User-Tobi_WMDE_SW: When generating attribution of image under a ported version of the licence, the unported license is shown in the attribution - https://phabricator.wikimedia.org/T136305#3276279 (10Jakob_WMDE) a:03Jakob_WMDE [14:14:25] tarrow: go to https://toolsadmin.wikimedia.org/tools/, find your tool, click the "create new repository" button [14:14:43] ah, perfect. Thanks! [14:15:45] * bd808 assumes he has not really documented this anywhere [14:42:59] 06Labs, 10Tool-Labs, 06translatewiki.net: update node.js on tools.telegrambot - https://phabricator.wikimedia.org/T159368#3276435 (10bd808) >>! In T159368#3276276, @madhuvishy wrote: > @bd808 Amir and I just tried the tutorial and got telegrambot running on kubernetes, but it required an admin permission cha... [14:44:16] 06Labs, 10Tool-Labs, 06translatewiki.net: update node.js on tools.telegrambot - https://phabricator.wikimedia.org/T159368#3276437 (10madhuvishy) @bd808 Okay thanks, I'll make a task to track that. @Amire80 Do you think we can close this task now? [14:48:40] 06Labs, 10Tool-Labs, 06translatewiki.net: update node.js on tools.telegrambot - https://phabricator.wikimedia.org/T159368#3276475 (10Amire80) 05Open>03Resolved a:03Amire80 Yes! Good enough for my needs till now. Thank you @bd808 and @madhuvishy! [15:36:45] 06Labs, 06Operations, 10ops-eqiad: rack/setup/install labnet100[34] - https://phabricator.wikimedia.org/T165779#3276633 (10RobH) [15:37:58] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team, 15User-Tobi_WMDE_SW: Add additional Text to the PD-Images hint - https://phabricator.wikimedia.org/T165332#3276656 (10WMDE-leszek) a:03WMDE-leszek [15:39:06] 06Labs, 06Operations, 10hardware-requests: Eqiad: (2) hardware access request for labnet1003/1004 - https://phabricator.wikimedia.org/T158204#3276658 (10RobH) 05Open>03Resolved These systems have been ordered on T163822 and installation will progress on T165779. Resolving this request, as its being gran... [15:52:21] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: rack/setup/install labcontrol100[34] - https://phabricator.wikimedia.org/T165781#3276700 (10RobH) [15:52:55] 06Labs, 06Operations, 10hardware-requests: Eqiad: (2) hardware access request for labcontrol1003/1004 - https://phabricator.wikimedia.org/T158207#3276717 (10RobH) 05Open>03Resolved These systems have been ordered on T163031 and will be setup on T165781. [15:58:26] 06Labs, 06Operations, 10ops-eqiad, 10procurement: rack/setup/install labmon1003 - https://phabricator.wikimedia.org/T165784#3276770 (10RobH) [15:59:41] 06Labs, 06Operations, 10ops-eqiad, 10procurement: rack/setup/install labmon1003 - https://phabricator.wikimedia.org/T165784#3276770 (10RobH) Please note that once the initial onsite-specific steps are done (steps up do the network port setup), I can handle the operations/puppet repo updates and install the... [15:59:55] 06Labs, 06Operations, 10ops-eqiad: rack/setup/install labmon1003 - https://phabricator.wikimedia.org/T165784#3276806 (10RobH) [16:02:44] 06Labs, 06Operations, 10hardware-requests: eqiad: (1) hardware access request for dedicated labmon1002 - https://phabricator.wikimedia.org/T161750#3276850 (10RobH) 05Open>03Resolved This has been ordered on T163808 and its setup will be tracked on T165784. [16:09:53] bd808 hi, sorry if i pinged the wrong person, but would you be able to register this #wikimedia-cloud channel just to prevent someone else doing that please? [16:19:00] (03PS1) 10Jean-Frédéric: Add debugging information at the start of processSource() [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354529 [16:19:02] (03PS1) 10Jean-Frédéric: Extract the geolocation from the UTF-8 decoded title page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354530 [16:21:33] 06Labs, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-LdapAuthentication, 10wikitech.wikimedia.org: Ldap auth extension vs. ldap vs. username Case - https://phabricator.wikimedia.org/T165795#3277001 (10Andrew) [16:22:53] 06Labs, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-LdapAuthentication, 10wikitech.wikimedia.org: Ldap auth extension vs. ldap vs. username Case - https://phabricator.wikimedia.org/T165795#3277016 (10Andrew) The wgLdapLoserCaseUsername may be meant to address this issue but that se... [16:34:31] (03CR) 10Lokal Profil: [C: 032] Add debugging information at the start of processSource() [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354529 (owner: 10Jean-Frédéric) [16:35:11] (03CR) 10Lokal Profil: [C: 032] Extract the geolocation from the UTF-8 decoded title page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354530 (owner: 10Jean-Frédéric) [16:37:14] (03Merged) 10jenkins-bot: Add debugging information at the start of processSource() [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354529 (owner: 10Jean-Frédéric) [16:37:34] (03Merged) 10jenkins-bot: Extract the geolocation from the UTF-8 decoded title page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354530 (owner: 10Jean-Frédéric) [16:39:35] 06Labs, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-LdapAuthentication, 10wikitech.wikimedia.org: Ldap auth extension vs. ldap vs. username Case - https://phabricator.wikimedia.org/T165795#3277033 (10Andrew) Presuming that the ldap auth check is case insensitive (which I think it i... [16:40:32] 06Labs, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-LdapAuthentication, 10wikitech.wikimedia.org: Ldap auth extension vs. ldap vs. username Case - https://phabricator.wikimedia.org/T165795#3277034 (10Reedy) ```lang=sql mysql:wikiadmin@silver [labswiki]> select user_name, count(user... [16:46:43] (03CR) 10jenkins-bot: Add debugging information at the start of processSource() [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354529 (owner: 10Jean-Frédéric) [16:47:51] (03CR) 10jenkins-bot: Extract the geolocation from the UTF-8 decoded title page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354530 (owner: 10Jean-Frédéric) [16:51:43] did labs get renamed? [16:55:49] (03PS1) 10Jean-Frédéric: Establish database connections for every source [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354540 [16:56:07] Zppix nope. It's a proposole [17:11:52] bd808: Can you approve Tool Labs for Denisa Rucaj? Helping at the Hackathon. [17:12:05] madhuvishy: ^ [17:13:17] Krinkle: let me see if I can log in from my phone... [17:15:44] bd808: I can do that :) [17:15:58] (if it's still on wikitech) [17:17:12] ah someone got to it before I did :) [17:17:25] I did it from my phone [17:18:42] (03CR) 10Multichill: [C: 032] Establish database connections for every source [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354540 (owner: 10Jean-Frédéric) [17:19:08] yuvipanda: did you find the ui for it on striker [17:19:33] bd808: nope, I just looked on wikitech [17:20:11] Ah. It's gone from wikitech. Request and approval happens in striker now. [17:20:21] Much nicer workflow [17:20:32] (03Merged) 10jenkins-bot: Establish database connections for every source [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354540 (owner: 10Jean-Frédéric) [17:22:13] (03CR) 10jenkins-bot: Establish database connections for every source [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/354540 (owner: 10Jean-Frédéric) [17:22:38] !log tools.heritage Deploy latest from Git master: ae1e775 (T138517), 2bd9781, 04a19e0 (T158911), e8dbe35, 4de8898, ea942d8, f956ab5, b63ebac, dd8c0c8, 63b3bc8, 8b15472 [17:22:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [17:22:42] T158911: Docker setup is broken because of InnoDB and FULLTEXT - https://phabricator.wikimedia.org/T158911 [17:22:42] T138517: mysqldump is timing out preventing all tables from being included in the dump - https://phabricator.wikimedia.org/T138517 [17:23:38] !log tools.heritage Deploy latest from Git master: 25023b6, d556d52, 56cd469, e15709d, 576a6d4, 550fb2d, 57d4f07, d2980f5 [17:23:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [17:24:10] bd808: nice! but I can still see list of users on wikitech, and saw this person was already added :D [17:24:16] will check out striker the next time :) [17:43:54] halfak|Mobile [17:45:05] halfak|Mobile: Is it possible to perform a binary search through a log dump for log entries of a given article? If so, how? [18:19:59] thedj: https://phabricator.wikimedia.org/T139859 [18:22:06] bd808: Thanks [18:39:01] !log deployment-prep: applying role::phabricator_server on instance deployment-phab01 (it had error, could not find role::phabricator::main and the name changed in role/profile conversion) [18:39:02] mutante: Unknown project "deployment-prep:" [18:39:16] !log deployment-prep applying role::phabricator_server on instance deployment-phab01 (it had error, could not find role::phabricator::main and the name changed in role/profile conversion) [18:39:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [18:40:46] !log deployment-prep deployment-phab01 still has puppet error "Could not find class role::phabricator::main" and that should simply be removed from it, but i can NOT find it in Horizon, i checked instance config, project config, the "Other" section, the "All classes" tab. Because it's gone. But how do i fix the instance config then? [18:40:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [18:42:11] paladox: ^ i just can NOT find the class in Horizon [18:42:17] so i cant remove it [18:42:42] if we could just make it forget about "role::phabricator::main" the rest should all be fine [18:44:08] Yeh, im trying to find out where it's being applied [18:44:09] mutante ^^ [18:44:30] i can go to the "All classes" tab and search for "True" with Ctrl+F [18:44:37] and there is ONLY my new class as it should [18:44:51] yet, running puppet on the instance.. could not find _old_ class name [18:45:29] i do remember how it was weird (shows up in "other" section, have to make sure to go to "all" and not "common" etc) last time too [18:45:36] but this is different. it's just not there [18:48:13] and regarding the second instance, deployment-phab-02, that shows us DOWN alert in Shinken.. well.. it just says "Shut Down" / Shutoff" in Horizon .. since 7 months(sic) [18:48:20] mutante: some projects also have puppet config on wikitech. e.g. https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep [18:48:55] andrewbogott: oh hi! it's not a Hiera setting though, it's the name of the role class applied on the instance [18:49:26] i need to make it forget about a role that doesnt exist anymore, but i cant remove it [18:49:53] what role? [18:50:25] instance: deployment-phab-01 project: deployment-prep error: Could not find class role::phabricator::main [18:50:38] correct role (applied in Horizon): role::phabricator_server [18:51:03] expected behaviour: role::phabricator::main shows up in Horizon, and i click remove, fixed [18:51:31] what i see: role::phabricator::main nowhere to be found in Horizon, so cant click "remove" [18:52:09] sorry, deployment-phab01 , no extra - [18:52:18] oh, ok, that helps :) [18:56:28] server = deployment-puppetmaster02.deployment-prep.eqiad.wmflabs [18:56:39] is this just a matter of waiting for that master to get the info? [18:57:02] mutante: Does this link work for you? https://horizon.wikimedia.org/project/prefixpuppet/?tab=prefix_puppet__puppet-deployment-phab [18:57:17] If so, scroll to the bottom and look at 'other classes' [18:57:39] yes, it does. and see "other" section has it ! [18:58:05] yep, it fell off of the table when the class was removed [18:58:12] since, no metadata from the puppet master [18:58:12] so i was at instance-level and at project-level but never at "prefix-level", that's it, right [18:58:27] i even knew the part about old classes moving to "other" , i learned that [18:58:32] If you look at the instance page [18:58:33] https://horizon.wikimedia.org/project/prefixpuppet/?tab=prefix_puppet__puppet-deployment-phab [18:58:38] but not the "prefix" [18:58:49] it has up top some links "This instance is also affected by…" [18:59:04] that's my attempt to make that less surprising, although it's still often surprising [18:59:14] thank you [18:59:17] ok, one more thing [18:59:25] when i click on "edit" in the "other" section now [18:59:39] should i just blank that and apply my new class from the list [18:59:45] yep [18:59:47] or should i actually put my new class name here [18:59:49] ok, cool [18:59:49] just blank out the 'other' section [18:59:53] alright [18:59:57] thanks:) [19:00:08] In theory if you add your class to the 'other' section it will be detected as in the list and yanked from 'other' and marked in the list [19:00:14] so the effect should be the same… in theory :) [19:00:37] thanks andrewbogott :) [19:00:44] " These puppet settings will affect all VMs in the deployment-prep project whose names begin with 'deployment-phab'. " gotha :) [19:01:18] yeh [19:01:25] Did they apply it globally? [19:01:47] well, that's the thing, it's kind of semi-global [19:01:54] not on instance, not on project [19:02:05] it's by project and prefix of the hostname [19:02:22] so it will apply to deployment-phab01 and 02 [19:03:09] o [19:03:10] oh [19:03:11] i just checked on instance and project level, that's why i did not see it before [19:03:20] never knew it could do that :) [19:04:51] !log deployment-prep: fixing role class config on deployment-phab* (remove role::phabricator::main, add role::phabricator_server in context prefix "deployment-phab. remove again from instance level for phab-01 [19:04:52] mutante: Unknown project "deployment-prep:" [19:05:06] i always do that because of commit messages :) [19:05:10] Lol [19:05:11] !log deployment-prep fixing role class config on deployment-phab* (remove role::phabricator::main, add role::phabricator_server in context prefix "deployment-phab. remove again from instance level for phab-01 [19:05:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [19:05:54] it says "Danger: There was an error submitting the form. Please try again." .. does so [19:08:52] it worked now (i logged in again) [19:09:09] puppet on the instance is working. it's fixed. nice [19:10:17] :) [19:11:21] paladox: so, i say 01 is fixed, and 02 is down since a long time. question is what about it. but it's not new breakage [19:11:41] ok [19:11:46] thanks for fixing it [19:11:54] could just click "start instance" but i have no idea what it was for, as opposed to -01 [19:12:21] probably when -01 wasn't puppetized and -02 was to make it so [19:12:27] oh [19:46:41] 06Labs: Tool videoconvert is using 1.034TB/8TB of tools NFS storage - https://phabricator.wikimedia.org/T165806#3277363 (10madhuvishy) [19:48:44] 06Labs, 10Tool-Labs: templatetiger is using 613G in Tools out of 8T - https://phabricator.wikimedia.org/T136192#3277367 (10madhuvishy) @Kolossos I noticed that this is back up to 415GB now, we are at pretty high tools NFS usage, it would be great if you could consider cleaning up some of this. Thank you! [20:25:39] 06Labs, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#3277451 (10Multichill) [20:42:59] 06Labs, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-LdapAuthentication, 10wikitech.wikimedia.org: Ldap auth extension vs. ldap vs. username Case - https://phabricator.wikimedia.org/T165795#3277479 (10Anomie) >>! In T165795#3277033, @Andrew wrote: > The step that actually creates th... [20:47:18] 06Labs, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-LdapAuthentication, 10wikitech.wikimedia.org: Ldap auth extension vs. ldap vs. username Case - https://phabricator.wikimedia.org/T165795#3277497 (10Anomie) >>! In T165795#3277479, @Anomie wrote: > It might do a lookup as you propo... [20:49:01] hm, can somebody help me with that error? Failed to connect to url-downloader.wikimedia.org port 8080: Connection timed out [20:49:13] get this when I'm trying to clone a github repo from one of my instances [20:49:17] from another one it works [21:33:40] Zppix apparently the rename is now in affect. https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction [21:52:23] (03PS5) 10Lokal Profil: [WIP]Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 [21:54:15] (03CR) 10jerkins-bot: [V: 04-1] [WIP]Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 (owner: 10Lokal Profil) [22:00:13] (03CR) 10Lokal Profil: "So Canada would need some more advanced SQL to work." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 (owner: 10Lokal Profil) [22:07:23] !log wikistats fixed puppet issues, puppetized db pass and grants to bootstrap instances, apt-get upgrading wikistats-petcow (jessie), testing stretch with separate instance -octopus .. [22:07:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [22:12:41] (03PS6) 10Lokal Profil: [WIP]Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 [22:13:47] (03CR) 10jerkins-bot: [V: 04-1] [WIP]Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 (owner: 10Lokal Profil) [22:29:23] (03PS7) 10Lokal Profil: [WIP]Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 [22:30:29] (03CR) 10jerkins-bot: [V: 04-1] [WIP]Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 (owner: 10Lokal Profil) [22:40:00] (03PS8) 10Lokal Profil: [WIP]Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 [22:41:38] (03CR) 10jerkins-bot: [V: 04-1] [WIP]Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 (owner: 10Lokal Profil) [22:45:13] (03PS9) 10Lokal Profil: [WIP]Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 [22:47:44] 06Labs, 10BetaFeatures, 06Collaboration-Team-Triage, 10Edit-Review-Improvements, 10wikitech.wikimedia.org: ERI requesting opt-in on wikitech but not available - https://phabricator.wikimedia.org/T165822#3277753 (10Quiddity) [23:23:50] (03PS10) 10Lokal Profil: Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 (https://phabricator.wikimedia.org/T165759) [23:25:03] (03CR) 10jerkins-bot: [V: 04-1] Build fill_table_monuments_all.sql from the json configs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 (https://phabricator.wikimedia.org/T165759) (owner: 10Lokal Profil) [23:50:43] 06Labs, 06Project-Admins, 15User-bd808, 06wmcs-team (Kanban): Create wmcs-team project and kanban milestone - https://phabricator.wikimedia.org/T165703#3277823 (10bd808) 05Open>03Resolved