[00:00:24] PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:40:22] RECOVERY - Puppet run on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [00:40:46] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [01:45:57] 10Tool-Labs-tools-Other: create tool to crunch metrics for views (play started) of video and audio files - https://phabricator.wikimedia.org/T116363#2784614 (10matmarex) For future reference, harej's tool's source code is at and 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, and 2 others: Provide public access to OpenStack APIs - https://phabricator.wikimedia.org/T150092#2784615 (10Andrew) Update -- documentation for keystone/oauth is pretty poor, and keystone has a habit of leaving partially-completed features to rot so I... [03:00:41] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, and 2 others: Provide public access to OpenStack APIs - https://phabricator.wikimedia.org/T150092#2784668 (10AlexMonk-WMF) >>! In T150092#2784615, @Andrew wrote: > It compiles properly for labtestcontrol but not for labcontrol, which I'm not yet clear... [03:20:07] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, and 2 others: Provide public access to OpenStack APIs - https://phabricator.wikimedia.org/T150092#2784739 (10Andrew) > Maybe it needs a require network::constants before the template() or something? Yep, fixed by changing the erb lookup. [03:54:07] 10Tool-Labs-tools-Other: Geohack should be mobile friendly - https://phabricator.wikimedia.org/T103409#1389066 (10Yurik) I think we should migrate GeoHack to - T137253 - which already has a good support for mobile devices. For example, see [[ https://www.mediawiki.org/wiki/Help:Extension:Kartographer#/... [05:43:48] 10Wikibugs, 06Wikisource: Post #wikisource bugs to #wikisource channel - https://phabricator.wikimedia.org/T150319#2781901 (10Legoktm) See , feel free to send a patch :) [06:08:27] (03PS1) 10Samwilson: Add Wikisource projects to the #wikisource IRC channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/320727 (https://phabricator.wikimedia.org/T150319) [06:09:18] 10Wikibugs, 06Wikisource, 13Patch-For-Review: Post #wikisource bugs to #wikisource channel - https://phabricator.wikimedia.org/T150319#2784877 (10Samwilson) Done so! Thanks. [06:35:04] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:59:38] PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:15:06] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [07:34:39] RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [09:40:57] 10Tool-Labs-tools-stewardbots, 10Continuous-Integration-Config, 13Patch-For-Review: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#2785079 (10hashar) Ahh the coupling with MediaWiki is due / related to T107529 [09:50:48] (03CR) 10Lokal Profil: [C: 032] Add some unit tests for database_statistics module [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/319990 (owner: 10Jean-Frédéric) [09:51:57] (03Merged) 10jenkins-bot: Add some unit tests for database_statistics module [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/319990 (owner: 10Jean-Frédéric) [13:21:51] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Wcam was created, changed by Wcam link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Wcam edit summary: Created page with "{{Tools Access Request |Justification=run bots on Chinese wikipedia |Completed=false |User Name=Wcam }}" [14:43:25] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 2.82 ms [14:56:09] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [15:42:52] Does the regular expression operations library come with python, or should I manually install it? [15:43:11] DatGuy: it comes with python [15:43:35] and how about "socket"? [15:49:08] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 1.63 ms [15:49:29] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2726255 (10Halfak) @jcrespo, I see that this is placed in the "Blocked external/Not db team" on the D... [15:52:43] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2785996 (10Halfak) Maybe we should be talking to @chasemp and @madhuvishy about getting this done... ? [15:53:45] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [15:54:07] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [16:06:29] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [16:27:33] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786145 (10jcrespo) I believe now that it is unblocked, the tables are actually replicated, so labs t... [16:29:40] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786150 (10jcrespo) I can confirm they are in labs: ``` $ mysql -h labsdb1001 wikidatawiki -e "SELE... [16:54:48] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786222 (10Halfak) Great! Thanks. Resolving [16:55:05] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2726255 (10Halfak) 05Open>03Resolved [16:56:10] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786227 (10jcrespo) Sorry I was not clear enough- the tables are replicated, but they are probably no... [16:56:17] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786228 (10jcrespo) 05Resolved>03Open [16:57:24] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786230 (10Ladsgroup) @jcrespo is right: ``` ladsgroup@tools-bastion-03:~$ sql fawiki "select oresm_i... [17:25:06] 10Labs-project-other, 06Developer-Relations, 10WikiApiary: move WikiApiary to Labs - https://phabricator.wikimedia.org/T149874#2786340 (10Kghbln) [17:33:32] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786420 (10Halfak) @jcrespo I see. Who should figure this out? [17:33:42] chasemp, ^ [17:33:55] We have tables replicated but not exposed in labsdb. [17:33:58] What do? [17:38:00] I checked repos but I couldn't find any configuration to do it. let me dig deeper [17:39:57] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786440 (10jcrespo) I think labs ops will know, but they are mostly unavailable/busy this week, so I... [17:40:35] halfak: Amir1 ^. chasemp is off this week at a conference [17:41:23] oh yeah! Kube, right? [17:41:52] yeah [17:41:57] I wonder who chasemp delegated to while he is away. [17:42:08] halfak: for what? [17:42:30] halfak: the rest of us are around, but the maintain_views stuff is part of the goal and nobody else is working on it rn :) [17:42:40] we also have a big migration on MOnday of NFS that we're prepping for [17:42:42] "the goal"? [17:43:01] halfak: 'the quarterly goal' [17:43:05] let me rephrase that actually [17:43:14] Gotcha. [17:43:18] halfak: it's in the process of being rewritten to completely suck, and the prev. one sucked totally. [17:43:29] it -> the process of exposing this info on labsdb [17:43:37] So I would expect an engineering manager to have a delegated contact when they are away. [17:44:19] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786444 (10Ladsgroup) I do the amending. [17:44:23] yuvipanda, gotcha. In the meantime, we're blocked on something that *seems* trivial. [17:44:39] halfak: indeed. but isn't actually trivial [17:44:48] no privacy and security concerns. Tables are already replicated. etc. [17:45:00] OK. Understood. [17:45:06] So how do we proceed? [17:45:13] halfak: we wait until next week [17:45:28] halfak: the views are there for Everything so in case it goes wrong you can have an informaiton leak [17:45:46] and since it's in the process of being redone I don't want to run it without chasemp being around [17:45:53] halfak: there is delegation, and that's why I'm talking to you ;) [17:46:03] Oh! great. [17:46:08] (not that I wouldn't talk to you otherwise) [17:46:20] So answer is "There's nothing to tell you until next week" [17:46:23] halfak: so I think Amir1 will amend it, and we'll try to merge and deploy that sometime next week [17:46:32] halfak: once we get over our NFS Migration [17:46:37] Which, to me, translates to "there's no way I can push this forward on my own" [17:47:08] Amir1, will amend what? [17:47:12] halfak: that's true [17:47:18] Amir1: he said on the ticket [17:47:22] err [17:47:26] halfak: Amir1 said on the ticket [17:47:34] halfak: indeed, there's no way you can push this forward on your own right now. [17:47:35] It seems like Amir1 couldn't find config. [17:47:48] Oh! I see that jcrespo pointed him to it. [17:47:52] halfak: there were two more comments since that. [17:48:06] Looks like amending that config is the way to push this forward. [17:48:11] Review waits until next week :) [17:48:12] * yuvipanda nods [17:48:54] Cool. It looks like we can just add ORES tables to "full views" [17:49:13] already made the patch [17:49:19] Nice [17:49:24] Cool. Thanks all :D [18:13:25] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2786574 (10Halfak) a:03Ladsgroup [18:15:37] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2726255 (10AlexMonk-WMF) >>! In T148561#2786440, @jcrespo wrote: > @Krenair may know, but it is not something he has to do... [18:41:17] (03CR) 10Legoktm: [C: 032] Add Wikisource projects to the #wikisource IRC channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/320727 (https://phabricator.wikimedia.org/T150319) (owner: 10Samwilson) [18:41:17] PROBLEM - Puppet run on tools-redis-1002 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:47:51] (03Merged) 10jenkins-bot: Add Wikisource projects to the #wikisource IRC channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/320727 (https://phabricator.wikimedia.org/T150319) (owner: 10Samwilson) [18:49:37] !log tools.wikibugs Updated channels.yaml to: 7a028caa9b6475ce5fd738a1a22edd1a6d00f6db Add Wikisource projects to the #wikisource IRC channel [19:06:26] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: labstore1003 - RAID fail - https://phabricator.wikimedia.org/T149156#2743689 (10Cmjohnson) Swapped failed disk [19:08:50] (03PS1) 10Paladox: Fix elseif for changing nick [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320821 [19:13:11] 06Labs, 10Tool-Labs: C# MONO bot runs OK but not with jsub - https://phabricator.wikimedia.org/T150099#2786886 (10Kotz) {meme, src=seal, above=sad, below="no help"} [19:16:17] RECOVERY - Puppet run on tools-redis-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:12] 06Labs, 10Labs-Infrastructure: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#2786971 (10Andrew) [19:31:44] 06Labs, 10Labs-Infrastructure: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#2787032 (10Andrew) [19:33:11] 06Labs, 10Wikimedia-Labs-General, 10DBA, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2787049 (10AlexMonk-WMF) [19:33:17] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2787048 (10AlexMonk-WMF) [19:35:31] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2787065 (10jcrespo) @AlexMonk-WMF, this is not a replication or Database issue, as I just demonstrated. [19:35:45] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2787066 (10jcrespo) [19:35:46] 06Labs, 10Wikimedia-Labs-General, 10DBA, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2787067 (10jcrespo) [20:01:17] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2787171 (10AlexMonk-WMF) >>! In T148561#2787065, @jcrespo wrote: > @AlexMonk-WMF, this is not a replication or Database is... [20:01:31] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2787186 (10AlexMonk-WMF) [20:01:32] 06Labs, 10Wikimedia-Labs-General, 10DBA, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2787187 (10AlexMonk-WMF) [20:04:03] (03CR) 10Zppix: [C: 032] Fix elseif for changing nick [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320821 (owner: 10Paladox) [20:04:29] (03Merged) 10jenkins-bot: Fix elseif for changing nick [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320821 (owner: 10Paladox) [20:51:23] (03PS1) 10Paladox: Reduce code duplication and also reply to the command nick if the nick does not need to change [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 [20:55:45] (03PS2) 10Paladox: Reduce code duplication and also reply to the command nick if the nick does not need to change [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 [20:57:50] 10Tool-Labs-tools-Zppixbot, 10Continuous-Integration-Config, 06Release-Engineering-Team, 07Jenkins: CI for #zppixbot - https://phabricator.wikimedia.org/T150487#2787353 (10Zppix) [20:59:00] (03CR) 10Zppix: [C: 032] Reduce code duplication and also reply to the command nick if the nick does not need to change [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 (owner: 10Paladox) [21:01:18] (03CR) 10jenkins-bot: [V: 04-1] Reduce code duplication and also reply to the command nick if the nick does not need to change [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 (owner: 10Paladox) [21:01:22] (03PS3) 10Paladox: Reduce code duplication and also reply to the command nick if the nick does not need to change [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 [21:02:16] (03PS4) 10Paladox: Reduce code duplication and also reply to the command nick if the nick does not need to change [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 [21:03:05] (03PS5) 10Paladox: Reduce code duplication and also reply to the command nick if the nick does not need to change [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 [21:03:32] (03CR) 10Paladox: [C: 032] "Per Zippix" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 (owner: 10Paladox) [21:03:38] (03CR) 10Paladox: "I mean Zppix" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 (owner: 10Paladox) [21:07:05] (03CR) 10Zppix: [C: 032] Reduce code duplication and also reply to the command nick if the nick does not need to change [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 (owner: 10Paladox) [21:07:08] 10Labs-project-Phabricator, 13Patch-For-Review: Stabilize vcs-user owned files and directories in Phab-02 - https://phabricator.wikimedia.org/T95982#2787412 (10Negative24) 05Open>03declined [21:07:25] 10Labs-project-Phabricator, 13Patch-For-Review: Stabilize vcs-user owned files and directories in Phab-02 - https://phabricator.wikimedia.org/T95982#1205584 (10Negative24) Phab-02 was deleted. [21:07:32] (03Merged) 10jenkins-bot: Reduce code duplication and also reply to the command nick if the nick does not need to change [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320841 (owner: 10Paladox) [21:16:45] (03CR) 10Hashar: "recheck" [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/317301 (owner: 10Zppix) [21:17:46] 10Tool-Labs-tools-Zppixbot, 10Continuous-Integration-Config, 06Release-Engineering-Team, 07Jenkins, 13Patch-For-Review: CI for #zppixbot - https://phabricator.wikimedia.org/T150487#2787435 (10hashar) 05Open>03Resolved a:03hashar Should be good now. Tested on https://gerrit.wikimedia.org/r/#/c/317301/1 [21:20:28] (03Draft2) 10Zppix: about.php addition [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/320859 [21:20:36] (03PS1) 10Paladox: Reduce code duplication [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320860 [21:20:54] (03CR) 10Paladox: [C: 04-2] "Haven't tested yet" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320860 (owner: 10Paladox) [21:35:51] (03PS2) 10Paladox: Reduce code duplication [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320860 [21:38:39] (03CR) 10Paladox: [C: 031] "This has been tested and works." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320860 (owner: 10Paladox) [21:39:20] (03CR) 10Zppix: [C: 032] Reduce code duplication [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320860 (owner: 10Paladox) [21:40:04] !log tools.lolrrit-wm merging https://gerrit.wikimedia.org/r/320860 [21:40:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL [21:40:58] (03Merged) 10jenkins-bot: Reduce code duplication [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320860 (owner: 10Paladox) [21:51:39] 10Wikibugs, 06Wikisource, 13Patch-For-Review: Post #wikisource bugs to #wikisource channel - https://phabricator.wikimedia.org/T150319#2787514 (10Legoktm) 05Open>03Resolved a:03Samwilson [22:25:23] yuvipanda, the module that you built for me at http://tools-docker-builder-01.eqiad.wmflabs/pyenchant-1.6.7-py2.py3.cp27.cp32.cp33.cp34.cp35.pp27.pp33-none-any.whl seems to have been taken down, and now my bot doesn't work. [23:14:26] (03PS1) 10Paladox: Support more approved wikimedia cloaks to run the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320931 [23:39:56] help [23:42:11] Server refused our key Using keyboard-interactive authentication. [23:42:28] in putty [23:42:58] Is there any solution [23:43:35] what server? [23:43:39] and username? [23:44:02] tools-login.wmflabs.org [23:44:36] Jarallah [23:47:31] Nov 10 23:42:00 tools-bastion-03 sshd[9426]: input_userauth_request: invalid user Jarallah [preauth] [23:48:04] root@tools-bastion-03:/var/log# ldapsearch -x uid=Jarallah dn -LLL [23:48:05] dn: uid=jarallah,ou=people,dc=wikimedia,dc=org [23:48:13] try it with a lowercase J? [23:48:38] where? [23:48:43] the username [23:48:52] use jarallah instead of Jarallah [23:49:11] root@tools-bastion-03:/var/log# id Jarallah [23:49:12] id: Jarallah: no such user [23:49:12] root@tools-bastion-03:/var/log# id jarallah [23:49:12] uid=13916(jarallah) gid=500(wikidev) groups=50062(project-bastion),50380(project-tools),52835(tools.jarallah),500(wikidev) [23:51:36] Thank you [23:54:56] 06Labs, 06Operations, 13Patch-For-Review: Set up monitoring for secondary labstore HA cluster - https://phabricator.wikimedia.org/T144633#2787910 (10madhuvishy)