[00:06:43] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:24:05] 06Labs, 10Pywikibot-core, 13Patch-For-Review: pywikipedia.org is not responding; pywikibot.org is not registered - https://phabricator.wikimedia.org/T106311#2879057 (10Dzahn) deployed . http://pywikibot.org/ now redirects to http://tools.wmflabs.org/pywikibot/ [00:25:04] 06Labs, 10Pywikibot-core, 13Patch-For-Review: pywikipedia.org is not responding; pywikibot.org is not registered - https://phabricator.wikimedia.org/T106311#2879058 (10Dzahn) 05Open>03Resolved a:03Dzahn @Siebrand all done here finally, right? [00:25:27] 06Labs, 10Domains, 06Operations, 10Pywikibot-core, 10Traffic: pywikipedia.org is not responding; pywikibot.org is not registered - https://phabricator.wikimedia.org/T106311#2879061 (10Dzahn) [05:41:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:07:40] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:48:21] PROBLEM - Puppet run on tools-exec-1412 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:28:18] RECOVERY - Puppet run on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [08:12:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:21:16] 10Tool-Labs-tools-Other: s51412__data reverted to last year state - https://phabricator.wikimedia.org/T150759#2879915 (10APPER) Hi. I'm not able to recover, because there are some errors in the database. The database exists, the tables exist, too. But if I want to query a table, I always get the following error:... [08:21:23] 06Labs: X! tools: Search Query Logical Error - https://phabricator.wikimedia.org/T153415#2879916 (10JustBerry) [08:26:25] 06Labs, 06Operations, 07kubernetes: docker-engine pulled into our repositories only keeps the latest version - https://phabricator.wikimedia.org/T153416#2879938 (10yuvipanda) [08:41:51] !log tools.hartman moved to k8s backend, is a mw installl [08:41:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.hartman/SAL [08:47:14] 06Labs, 10Labs-Infrastructure, 10DBA: fix m5 shard mysql issues with default encoding binary - https://phabricator.wikimedia.org/T112103#2880019 (10jcrespo) 05Open>03Resolved a:03jcrespo [09:08:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:07:05] 10Tool-Labs-tools-Other: s51412__data reverted to last year state - https://phabricator.wikimedia.org/T150759#2880250 (10jcrespo) a:03jcrespo > Maybe there are some permissions missing? Could you please have a look into this? The permissions were missing, but the main issue was that innodb tables had to be re... [10:27:31] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:36:41] 06Labs, 10Wikispeech: Request creation of Wikispeech labs project - https://phabricator.wikimedia.org/T153430#2880310 (10Lokal_Profil) [10:37:43] Hey yuvipanda, have you ever talked with Magnus about sunset of WDQ? [10:37:53] hey multichill [10:37:54] 06Labs, 10Wikispeech: Request creation of Wikispeech labs project - https://phabricator.wikimedia.org/T153430#2880310 (10Lokal_Profil) [10:37:56] nope I haven't [10:38:07] should I? is it dead again? [10:38:21] Not AFAIK [10:38:34] right [10:38:40] multichill: should I start an email thread? [10:38:43] But when a tool is still functioning it's usually a good time to prepare [10:38:55] Not when it's already burning ;-) [10:39:19] right :D [10:40:00] It depends on what Magnus thinks I guess. If he is in favour of shutting it down in the future, we can just make a bunch of phab tasks to do it gracefully [10:40:26] I'm sure WDQ is still used for a couple of use cases where the query service doesn't work. [10:40:37] multichill: I emailed him and you [10:40:38] But at least we can shift away most of the traffic [10:41:03] And that is probably also good for the reliability ;-) [10:41:06] ok, ty [10:41:44] multichill: np. thanks for bringing it up! [10:48:42] !log tools.tb-dev regenerate replica.my.cnf, was old style pNNNsNNN [10:48:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tb-dev/SAL [10:51:19] !log tools.hasteurbot regenerate replica.my.cnf, was using uNNNN style usernames [10:51:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.hasteurbot/SAL [10:52:56] 10Tool-Labs-tools-Other: s51412__data reverted to last year state - https://phabricator.wikimedia.org/T150759#2880405 (10jcrespo) The following tables could be recovered (in addition to the existing MyISAM tables): ``` +---------------------------+ | Tables_in_s51412__data | +---------------------------+ | b... [10:55:12] !log tools.enbssb was sharing replica.my.cnf with hasteurbot, regenerated [10:55:12] Unknown project "tools.enbssb" [10:55:26] !log tools.enbbsb was sharing replica.my.cnf with hasteurbot, regenerated [10:55:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.enbbsb/SAL [10:56:06] !log tools.danmicholobot regenerate replica.my.cnf, was using a uNNN style acct [10:56:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.danmicholobot/SAL [11:00:56] !log tools.xxn-tools regenerated replica.my.cnf, was using a uNNN style acct [11:00:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.xxn-tools/SAL [11:01:02] !log tools.wmve regenerated replica.my.cnf, was using a uNNN style acct [11:01:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmve/SAL [11:01:13] !log tools.hgztools regenerated replica.my.cnf, was using a uNNN style acct [11:01:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.hgztools/SAL [11:01:34] !log tools.boty regenerated replica.my.cnf, was using a uNNN style acct [11:01:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.boty/SAL [11:01:42] !log tools.botomatik regenerated replica.my.cnf, was using a uNNN style acct [11:01:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.botomatik/SAL [11:03:05] yuvipanda: since you're regenerating those, not sure if you might be interested in https://phabricator.wikimedia.org/T130469 [11:07:34] RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [11:52:37] 06Labs, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#2880526 (10Multichill) [11:52:49] yuvipanda: ^^ right projects etc? [11:53:05] multichill: yeah, that sounds good [11:58:42] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Bennylin was created, changed by Bennylin link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Bennylin edit summary: Created page with "{{Tools Access Request |Justification=Restarting stewardbot in case it gets flooded down. I got here from the description page in https://steward.wikimedia.org/wiki/Stewardbot..." [11:58:43] 06Labs, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#2880576 (10Multichill) [11:59:32] yuvipanda: I did the linking correct this time, right? I keep messing that up :P [12:01:10] multichill: in the body or betweent asks? [12:01:11] seems right to me :D [12:02:06] Parent/child/blocking things [12:02:42] yep [12:02:56] 06Labs, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#2880593 (10Multichill) [12:09:47] 06Labs, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#2880631 (10Multichill) [12:13:50] multichill: is there a WDQ to SPARQL translator? [13:52:18] Hi all, is anybody able to fulfil T153308? [13:52:18] T153308: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308 [14:12:05] Urbanecm: That's weird, are you the first user of pywikibot+oauth on the grid? I doubt that... Please add the versions of python and pywikibot and on what exec node you ran into the missing library [14:12:31] multichill, I don't think I'm the first. I only report it doesn't work for me :) [14:13:04] You might be using python3, rest python2 or old git version, etc etc [14:13:23] python pwb.py version.py [14:13:35] Or, jsub that -) [14:14:12] multichill, wait a moment please... Starting a Python takes a long time... [14:15:42] multichill, https://phabricator.wikimedia.org/P4634 without jsub [14:17:23] multichill, and https://phabricator.wikimedia.org/P4635 [14:17:50] BTW the error is shown when running version.py [14:19:32] Excellent. Throw that in the bug please. I tagged some people who are probably interested in getting that solved, but just dropped out of this channel (matrix down?) [14:20:09] multichill, going to do that. I'm also going to insert a sample error (login.py). [14:20:11] Urbanecm: Did you try PAWS? It's at http://paws.wmflabs.org/paws/hub/login , you might like it [14:20:35] The output on the pastebins isn't that big so just put everything in [14:21:07] multichill, yes. But with PAWS I've noticed another problem. I must insert [ and ] with Alt+number... [14:21:43] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2881002 (10Urbanecm) [14:21:57] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2876092 (10Urbanecm) [14:23:42] I played around with PAWS, but never really used it that much for actual bots. I mostly run my bots on the grid [14:24:27] multichill, I'd like to run them on the grid. But I can't. [14:31:17] multichill, interesting is I can run bots at dev bastion... [14:32:55] multichill, and with PAWS, I can't insert anything into the terminal... [14:32:59] multichill: just in case you didn't see, you have a talk page notification on wikitech [14:34:18] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2881023 (10Urbanecm) [14:35:16] Urbanecm: afaik, people usually go either with a virtualenv or setup.py install --user when they need packages [14:35:52] zhuyifei1999_, I know that. But when I use virtualenv I have another problem. Python can't reach standard modules as "datetime" :D [14:36:21] o.O seriously? [14:36:30] zhuyifei1999_, yes... I can try it again :D [14:37:16] I thought python inside virtualenvs have system python path in its own pythonpath be default [14:37:44] And I think important modules can be installed for all users. [14:39:15] Urbanecm: I just saw this in `virtualenv --help`: [14:39:16] --system-site-packages [14:39:16] Give the virtual environment access to the global [14:39:16] site-packages. [14:39:27] yeah I agree [14:39:38] I'm a problematic user... [14:39:39] :) [14:41:04] but I don't remember how pip packages are installed by default.. operations/puppet doesn't define any pip installs, afaik [14:42:09] stuffs usually gets packaged into a deb under operations/debs and installed via apt with puppet :( [14:42:16] For telling the whole truth. Venv works sometimes. At some point it breaks. The only advice I found is "reinstall it". But I don't want to reinstall venv on regular basis. I'd like to have it working... [14:44:03] you mean https://docs.python.org/3/library/venv.html [14:44:04] ? [14:44:32] zhuyifei1999_, what exact section do you mean at this page? [14:44:54] * zhuyifei1999_ is pretty much still a python 2 guy... haven't catched up with recent python3 developments [14:45:32] uh I mean by "venv" do you mean the builtin python 3 venv? [14:45:55] zhuyifei1999_: Thanks, I noticed a ping but didn't check it out yet [14:46:02] I use virtualenv [14:46:56] uh if it breaks there should be a reason [14:47:48] zhuyifei1999_, I don't know that reason... [14:48:04] do you have an error message? [14:48:13] Whats the issue [14:49:30] I'm just unsure if a pip package getup globally on tool labs has ever done :( [14:50:03] What do you mean [14:50:30] zhuyifei1999_, no. I must wait till it will break again. [14:50:57] oh btw Urbanecm ik when the os change (eg. from precise to trusty or trusty to jessie) virtualenvs can break [14:51:16] zhuyifei1999_, thanks for that info. [14:52:46] Zppix: https://phabricator.wikimedia.org/T153308 [14:54:39] Ill look into it assp [14:54:40] Assp [14:54:47] asap jesus i cannot spell [14:56:23] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2876092 (10zhuyifei1999) Do we have a process for setting up a pip package globally on exec nodes? I'm only aware of [[https://github.com/wikimedia/operations... [14:56:40] lol [14:59:56] zhuyifei1999_: you may be able to get the answer from operations [15:00:11] ... [15:02:09] ? [15:05:47] Zppix: I mean, there was unlikely ever such a practice of installing pip packages globally across exec nodes, that's why I suggested to use virtualenv [15:09:59] zhuyifei1999_: ah i dont see that happening considering usually only python users on labs use virtualenv usually [15:10:36] pywikibot is python [15:10:58] (and virtualenv is for python) [15:11:33] literally a virtual python environment [15:55:38] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2876092 (10bd808) >>! In T153308#2881079, @zhuyifei1999 wrote: > Do we have a process for setting up a pip package globally on exec nodes? I'm only aware of [... [16:03:21] Hi, perhaps: has a surplus ticket or a voucher for the 33C3 congress for me ? Then please send me a wikimail https://en.wikipedia.org/wiki/Special:EmailUser/Wikinaut [16:04:25] [corr] s/has/has someone/ [16:20:59] 06Labs: Update keystone hooks and monkeypatches for Mitaka - https://phabricator.wikimedia.org/T153469#2881423 (10Andrew) [16:39:27] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#2881494 (10Andrew) [17:21:26] Krinkle: wow T153468 is odd alright [17:21:27] T153468: Ferm/DNS library weirdness on deployment-mediawiki boxes - https://phabricator.wikimedia.org/T153468 [17:21:39] dammit, Krenair [17:21:50] :) [17:22:22] we expect an AAAA lookup for that domain returns a NOERROR with 0 actual AAAA responses [17:22:39] and I think ferm should treat that as a success and just record no addresses for it [17:32:03] /c/c [17:32:13] Ops, sorry :) [17:36:47] Krenair: I wanted to let you know so you can help spread the news (ill also put a calender event if i can on phab about this) but grrrit-wm will be down at 7:00pm CST utc-6 for maintainence if everything goes according to plan it should at most take 30 mins [17:37:03] maintenance? [17:37:09] Yes sorry [17:37:35] It has to do with nickserv stuff so no way around this [17:37:50] it's already got a nickserv account [17:37:55] I know [17:38:14] so what's the maintenance? [17:38:26] Im grouping the test bot into the same nickserv acct [17:39:19] And enabling some other things [17:40:19] why? [17:40:50] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2881023 (10yuvipanda) Can you tell me what browser you are using? and if it is chrome, can you try with Firefox? And if firefox, try with chrome? Also what keyboard layout are you using? [17:40:50] Easier then requesting a test bot cloak [17:44:43] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2881773 (10Urbanecm) I use Chrome, will try with FF. I use Czech QWERTZ layout, can try with english one. [17:45:22] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2881775 (10Urbanecm) @Dvorapa I think you reported the same problem on-wiki. [17:46:13] E424 [17:46:26] ^^ event id Krenair [17:56:20] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2881799 (10Dvorapa) >>! In T153457#2881775, @Urbanecm wrote: > @Dvorapa I think you reported the same problem on-wiki. Yes, i did, but I also reported it here on Phabricator and the issue was resolved. For... [17:59:33] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2881805 (10Urbanecm) It doesnt work for me... [18:01:12] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2881819 (10Dvorapa) @Urbanecm: T136118 [18:03:47] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2881838 (10Dvorapa) Maybe this is only a problem in Firefox [18:05:05] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2881841 (10Urbanecm) Thank you. [18:05:39] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2881843 (10Urbanecm) It cant be just a FF problem. I use Chrome. [18:12:59] I'm getting "server unexpectedly closed network connection" when trying to connect using PUTTY on Windows :/ any help? [18:15:36] Stryn: is it happening when you first connect or sometime later when you haven't typed anything for a while? [18:22:20] when I first connect [18:23:07] bd808: ^^ [18:23:23] Stryn: which server are you connecting to? I can try to watch logs from the server side and see if there is any corellation [18:23:27] Stryn: what server login or dev? [18:23:41] bd808: jinx [18:27:00] valhallasw`cloud: Matrix is down? [18:27:31] ? [18:28:49] tools-login [18:30:10] Stryn: ok. I'm watching the sshd logs there. can you try again? [18:30:19] ok [18:32:05] justs did and got the same error [18:32:27] I see "Failed publickey for stryn" in the logs [18:33:30] Stryn: I'll PM you the key fingerprint that the server sees [18:35:08] 06Labs: X! tools: Search Query Logical Error - https://phabricator.wikimedia.org/T153415#2881877 (10scfc) 05Open>03Invalid X! tools is not maintained in Phabricator. You can see the link for "Bugs" at the bottom of https://tools.wmflabs.org/xtools-ec/?user=Xx236&project=en.wikipedia.org. Please report the... [18:36:47] I have tried two different keys, and getting the same error on both [18:37:37] *nod* I do see two different keys in the logs [18:38:02] I just deleted the older one [18:38:29] I'll PM the public key that I see in LDAP [18:40:19] valhallasw`cloud: You and all the other matrix people left at the same time [18:40:32] multichill: ah. yeah, can be flaky at times [18:45:22] 06Labs, 10Labs-Infrastructure, 10Tool-Labs, 10Mail: Receiving spam from @toolserver.org address - https://phabricator.wikimedia.org/T153471#2881917 (10Dzahn) [18:46:50] 06Labs, 10Mail: Receiving spam from @toolserver.org address - https://phabricator.wikimedia.org/T153471#2881934 (10Krenair) [19:04:48] 06Labs, 10Mail: Receiving spam from @toolserver.org address - https://phabricator.wikimedia.org/T153471#2881989 (10Dzahn) So this was once setup by @coren. I found in puppet repo `puppet/modules/toolserver_legacy/templates/exim4.conf.erb` which has: ``` .. begin routers # Redirect using the alias file if i... [19:05:37] 06Labs, 10Mail: Receiving spam from @toolserver.org address - https://phabricator.wikimedia.org/T153471#2882004 (10Dzahn) 05Open>03Resolved a:03Dzahn [19:05:51] 06Labs, 10Mail: Receiving spam from @toolserver.org address - https://phabricator.wikimedia.org/T153471#2881451 (10Dzahn) please reopen if it hasn't stopped [19:06:55] !log toolserver-legacy removed toolserver mail alias by request (T153471) [19:06:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolserver-legacy/SAL [19:06:58] T153471: Receiving spam from @toolserver.org address - https://phabricator.wikimedia.org/T153471 [19:29:49] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2882128 (10Urbanecm) @yuvipanda It works with FF correctly. [19:30:42] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2882129 (10Urbanecm) [19:33:36] (03PS1) 10BryanDavis: Check and enforce OATH account protection [labs/striker] - 10https://gerrit.wikimedia.org/r/327786 (https://phabricator.wikimedia.org/T144712) [19:43:15] 06Labs, 10Striker, 06Security-Team, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org: Add user group to wikitech granting the oathauth-api-all right - https://phabricator.wikimedia.org/T153487#2882161 (10bd808) [20:27:24] 06Labs, 10Mail: Receiving spam from @toolserver.org address - https://phabricator.wikimedia.org/T153471#2882334 (10Pietrodn) Thank you. The spam has stopped. :) [21:25:43] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Bennylin was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1172588 edit summary: [23:11:16] !log tools.stashbot Restarted bot to change config for #wikimedia-releng channel [23:11:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [23:47:44] Attention: 1 hour until scheduled grrrit-wm maintainence [23:56:13] shall we pray?