[00:33:59] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:40:25] 06Labs, 10wikitech.wikimedia.org, 05Goal, 15User-bd808: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3250243 (10Paladox) [00:45:18] PROBLEM - Puppet errors on tools-exec-1432 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:08:57] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [01:20:17] RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [01:36:08] 06Labs, 10wikitech.wikimedia.org, 05Goal, 15User-bd808: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3250326 (10Yaron_Koren) Well done, guys! It took a lot of effort, but you've achieved "a slight decrease in usability". [02:22:21] PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:33:49] PROBLEM - Puppet errors on tools-exec-1431 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:57:22] RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [03:08:50] RECOVERY - Puppet errors on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0] [06:43:16] PROBLEM - Puppet errors on tools-exec-1412 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:21:52] 10Tool-Labs-tools-Other, 10DBA: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3225024 (10Framawiki) For your jstart problem, you can expressly set PWB path by using `python /shared/pywikipedia/core/pwb.py yourscriptname.py` as command. It's what I use for my bot. {T154011} will norma... [07:22:06] 10Tool-Labs-tools-Other, 10DBA: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3250682 (10Framawiki) [07:23:16] RECOVERY - Puppet errors on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [07:24:52] 06Labs, 10wikitech.wikimedia.org, 05Goal, 15User-bd808: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#1404878 (10Nemo_bis) >>! In T53642#3250326, @Yaron_Koren wrote: > Well done, guys! It took a lot of effort, but you've achieved "a slight decrea... [08:01:36] 06Labs: Request creation of labs project - https://phabricator.wikimedia.org/T164910#3250741 (10Acagastya) [09:33:33] 06Labs, 10wikitech.wikimedia.org, 05Goal, 15User-bd808: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3250928 (10Reedy) It would be perfectly reasonable to create other seperate tasks for the broken/missing functionality Baring in mind, the inte... [11:00:46] (03PS1) 10Alexandros Kosiaris: Empty keys for kubemaster.svc.$site.wmnet certs [labs/private] - 10https://gerrit.wikimedia.org/r/353041 [11:01:05] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] Empty keys for kubemaster.svc.$site.wmnet certs [labs/private] - 10https://gerrit.wikimedia.org/r/353041 (owner: 10Alexandros Kosiaris) [11:50:43] (03PS1) 10Alexandros Kosiaris: Fix typo in kubemaster key filenames [labs/private] - 10https://gerrit.wikimedia.org/r/353045 [11:50:46] (03PS1) 10Alexandros Kosiaris: Fix another typo with kubernetes master keys [labs/private] - 10https://gerrit.wikimedia.org/r/353046 [11:54:46] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] Fix typo in kubemaster key filenames [labs/private] - 10https://gerrit.wikimedia.org/r/353045 (owner: 10Alexandros Kosiaris) [11:54:51] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] Fix another typo with kubernetes master keys [labs/private] - 10https://gerrit.wikimedia.org/r/353046 (owner: 10Alexandros Kosiaris) [12:31:14] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team: Make it possible to use Esc key to close modal windows - https://phabricator.wikimedia.org/T163731#3251297 (10jhsoby) [12:44:33] !log git updating puppet-paladox3 instance using 'apt-get distupgrade' [12:44:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [13:53:18] PROBLEM - Puppet errors on tools-worker-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:54:17] PROBLEM - Puppet errors on tools-worker-1023 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:55:09] PROBLEM - Puppet errors on tools-worker-1013 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:55:15] (03PS1) 10Gehel: maps - renamed cassandra passwords for role / profile refactoring [labs/private] - 10https://gerrit.wikimedia.org/r/353068 [13:58:03] PROBLEM - Puppet errors on tools-worker-1018 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:59:25] PROBLEM - Puppet errors on tools-worker-1014 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:01:08] PROBLEM - Puppet errors on tools-worker-1016 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:02:46] PROBLEM - Puppet errors on tools-worker-1017 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:02:48] PROBLEM - Puppet errors on tools-worker-1010 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:02:54] PROBLEM - Puppet errors on tools-worker-1019 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:02:58] PROBLEM - Puppet errors on tools-worker-1009 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:03:24] PROBLEM - Puppet errors on tools-worker-1015 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:07:43] PROBLEM - Puppet errors on tools-worker-1011 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:07:47] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:07:55] PROBLEM - Puppet errors on tools-worker-1012 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:08:53] PROBLEM - Puppet errors on tools-worker-1025 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:09:29] PROBLEM - Puppet errors on tools-worker-1029 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:14:20] PROBLEM - Puppet errors on tools-worker-1003 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:14:26] PROBLEM - Puppet errors on tools-worker-1026 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:14:40] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:17:53] PROBLEM - Puppet errors on tools-worker-1027 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:18:01] PROBLEM - Puppet errors on tools-worker-1028 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:18:15] PROBLEM - Puppet errors on tools-worker-1005 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:21:03] PROBLEM - Puppet errors on tools-worker-1022 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:21:03] PROBLEM - Puppet errors on tools-worker-1002 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:22:37] PROBLEM - Puppet errors on tools-worker-1004 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:22:41] PROBLEM - Puppet errors on tools-worker-1008 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:23:16] PROBLEM - Puppet errors on tools-worker-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:24:48] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:54:24] That alert storm is understood and we're working on it [15:06:08] RECOVERY - Puppet errors on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:45] RECOVERY - Puppet errors on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:47] RECOVERY - Puppet errors on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:55] RECOVERY - Puppet errors on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [15:08:15] RECOVERY - Puppet errors on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [15:08:27] RECOVERY - Puppet errors on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [15:09:25] RECOVERY - Puppet errors on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [15:09:29] RECOVERY - Puppet errors on tools-worker-1029 is OK: OK: Less than 1.00% above the threshold [0.0] [15:12:46] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [15:12:58] RECOVERY - Puppet errors on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [15:13:54] RECOVERY - Puppet errors on tools-worker-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [15:17:40] RECOVERY - Puppet errors on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [15:17:56] RECOVERY - Puppet errors on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [15:19:27] RECOVERY - Puppet errors on tools-worker-1026 is OK: OK: Less than 1.00% above the threshold [0.0] [15:19:37] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [15:22:53] RECOVERY - Puppet errors on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0] [15:23:01] RECOVERY - Puppet errors on tools-worker-1028 is OK: OK: Less than 1.00% above the threshold [0.0] [15:24:20] RECOVERY - Puppet errors on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [15:26:00] RECOVERY - Puppet errors on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [15:26:02] RECOVERY - Puppet errors on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [15:27:38] RECOVERY - Puppet errors on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [15:27:42] RECOVERY - Puppet errors on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [15:28:16] RECOVERY - Puppet errors on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [15:29:49] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [15:33:17] RECOVERY - Puppet errors on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [15:34:14] RECOVERY - Puppet errors on tools-worker-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [15:35:08] RECOVERY - Puppet errors on tools-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [15:38:01] RECOVERY - Puppet errors on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [15:52:42] 06Labs, 10wikitech.wikimedia.org: Interacting with table generated by {{#ask}} throws Uncaught TypeError - https://phabricator.wikimedia.org/T101642#3252106 (10bd808) 05Open>03declined Obsoleted by {T53642} [16:08:01] Hi, im wondering why dosen't labs support ipv6? Is it because of openstack limitation? [16:11:38] 06Labs, 06DC-Ops, 06Operations, 10ops-eqiad: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3252214 (10Cmjohnson) @madhuvishy @chasemp Moved the labstores back to row C...labstore1001 is in c2 and labstore100... [16:13:39] 06Labs, 06DC-Ops, 06Operations, 10ops-eqiad: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3252239 (10madhuvishy) Woot thanks @Cmjohnson :D [16:23:49] paladox: yes, it won't change for awhile. [16:23:57] ok [16:24:00] thanks [16:25:41] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3252292 (10Cmjohnson) [16:25:49] 06Labs, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3051441 (10Cmjohnson) [17:29:44] !log tools Fixed broken puppet cert on tools-package-builder-01 [17:29:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:47:39] !log tools Signed and revoked puppet certs generated when our DNS flipped out and gave hosts non-FQDN hostnames [17:47:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:14:08] bd808: Hm.. unicode fluke? https://tools.wmflabs.org/?list (Jean Frederic) [18:14:29] ugh [18:15:01] Which phab component for this? [18:15:06] I recall there being a beta as well [18:15:09] but forgot its name [18:15:20] it's the "admin" tool [18:15:40] I have a completely rewritten version that has been sitting around for months [18:15:55] Krinkle: just file as #tool-labs [18:16:19] this really seems new [18:16:47] I think that data is read from a mysql table. I wonder if whatever loads that table went nuts [18:16:52] 06Labs, 10Tool-Labs: Broken unicode characters / invalid UTF-8 on Tool Labs index - https://phabricator.wikimedia.org/T164971#3252783 (10Krinkle) [18:17:34] it looks very much like a UTF-8 treated as latin1 problem [18:17:56] 06Labs, 10Tool-Labs: Broken unicode characters / invalid UTF-8 on Tool Labs index - https://phabricator.wikimedia.org/T164971#3252797 (10Krinkle) [18:18:44] bd808: yeah, note that it affects both the link and the label. Obvious, but important difference, which means the user page cannot be found either. [18:20:20] I wonder if an extension could be created that can hook into viewing https://wikitech.wikimedia.org/wiki/User:Jean-Fr%C3%83%C2%A9d%C3%83%C2%A9ric and redirect to the intended destination, based a known set of character converstion issues, given that those conversion isuses are determinstic. E.g. latin1->utf8 (reverse map), and a few others. [18:20:22] the data comes from the tools table in toollabs_p [18:20:42] Hm.. I didn't know that existed, interesting. [18:20:48] I honestly have no idea what populates that table [18:20:49] Populated based on ldap and scanning tools home dirs? [18:21:01] Maybe something in puppet? [18:21:56] I remember that populating thing being a hack [18:22:14] forgot what is actually is, though [18:23:13] * bd808 will look after lunch [18:23:34] github search is cool: https://github.com/wikimedia/puppet/blob/e959321aa620b77403cc9379db2e86080323c6e8/modules/toollabs/files/updatetools [18:49:11] 06Labs, 10DBA, 10wikitech.wikimedia.org, 07Blocked-on-schema-change: Drop Semantic Database tables from wikitech wikis - https://phabricator.wikimedia.org/T164887#3252920 (10Umherirrender) [18:51:28] hmmm... an LDAP search for uid=jeanfred returns garbage looking data too. Maybe it's the LDAP side that went nuts? [18:55:14] 06Labs, 10Tool-Labs: Broken unicode characters / invalid UTF-8 on Tool Labs index - https://phabricator.wikimedia.org/T164971#3252948 (10bd808) the beta is actually https://tools.wmflabs.org/admin-beta/tools but it shows the same problem. This doesn't look right either: ``` $ ldap 'uid=jeanfred' cn dn: uid=je... [19:04:01] 06Labs, 10Striker, 10Tool-Labs, 15User-bd808: Detachment of an account on Tool Labs from the Main wikimedia account. - https://phabricator.wikimedia.org/T164849#3252972 (10bd808) @FriyMan to be clear, the account that you created mistakenly and attached to your main Wikimedia account is [[https://wikitech.... [19:05:29] 06Labs, 10Striker, 10Tool-Labs, 15User-bd808: Detachment of an account on Tool Labs from the Main wikimedia account. - https://phabricator.wikimedia.org/T164849#3252986 (10FriyMan) Yes. This is the account. [19:11:01] !log tools Edited striker db record for user Stepan Grigoryev to detach SUL and Phab accounts. T164849 [19:11:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:11:05] T164849: Detachment of an account on Tool Labs from the Main wikimedia account. - https://phabricator.wikimedia.org/T164849 [19:12:06] 06Labs, 10Striker, 10Tool-Labs, 15User-bd808: Detachment of an account on Tool Labs from the Main wikimedia account. - https://phabricator.wikimedia.org/T164849#3253005 (10bd808) 05Open>03Resolved See also: {T164847} [19:26:46] 06Labs, 10Tool-Labs: Broken unicode characters / invalid UTF-8 on Tool Labs index - https://phabricator.wikimedia.org/T164971#3253079 (10bd808) Direct mysql query shows the encoding problem in the backing database: ``` (u3518@tools-db) [toollabs_p]> select * from users where name = 'jeanfred'\G ***************... [19:43:52] (03PS1) 10Dzahn: add private/files/releases/id_rsa.upload FAKE secret key [labs/private] - 10https://gerrit.wikimedia.org/r/353160 [19:44:56] (03PS2) 10Dzahn: add private/files/releases/id_rsa.upload FAKE secret key [labs/private] - 10https://gerrit.wikimedia.org/r/353160 [19:45:14] (03PS3) 10Dzahn: add private/files/releases/id_rsa.upload FAKE secret key [labs/private] - 10https://gerrit.wikimedia.org/r/353160 [19:46:53] (03CR) 10Paladox: [C: 031] add private/files/releases/id_rsa.upload FAKE secret key [labs/private] - 10https://gerrit.wikimedia.org/r/353160 (owner: 10Dzahn) [19:47:42] (03CR) 10Dzahn: [C: 032] "To fix puppet on deployment-tin/deployment-mira." [labs/private] - 10https://gerrit.wikimedia.org/r/353160 (owner: 10Dzahn) [19:48:20] (03CR) 10Dzahn: [V: 032 C: 032] add private/files/releases/id_rsa.upload FAKE secret key [labs/private] - 10https://gerrit.wikimedia.org/r/353160 (owner: 10Dzahn) [20:10:40] 06Labs, 10DBA, 10wikitech.wikimedia.org, 07Schema-change: Drop Semantic Database tables from wikitech wikis - https://phabricator.wikimedia.org/T164887#3253267 (10Reedy) [20:51:31] hello labs folks [20:51:54] i need to do a big wikitext grep across all wikis for https://www.mediawiki.org/wiki/Parsoid/Language_conversion/Preprocessor_fixups [20:52:14] can anyone help me set up a minimal instance with https://wikitech.wikimedia.org/wiki/Help:Shared_storage access to dumps? [20:59:07] cscott, tool labs has /public/dumps [20:59:26] If you need an instance, you'll have to wait for someone with the appropriate permissions to appear. [21:02:40] oddly enough, i have permissions needed to set up an instance, but not permissions needed to use tool labs ;) [21:02:58] That's unusual. [21:04:46] what I was told way back when was that tool labs was being deprecated in favor of labs. i'm also a WMF employee, that might have something to do with it. [21:08:39] It seems to be going the other way now. The labs people seem to want people to use tool labs in favour of labs. [21:10:20] 06Labs: Add /public/dumps access via NFS to dumpgrepper instance - https://phabricator.wikimedia.org/T164992#3253442 (10cscott) [21:10:26] and so the wheel turns. [21:10:57] anyway, i've applied for tool labs access, and i've also filed a ticket asking for dump access from my labs access. one of these should work. ;) [21:11:36] legoktm hi, extdist-02 seems to have puppet errors. Is that the host that host where to download extensions for mw from? [21:42:47] 06Labs, 10Tool-Labs: Broken unicode characters / invalid UTF-8 on Tool Labs index - https://phabricator.wikimedia.org/T164971#3253545 (10bd808) This looks ok: ``` $ getent passwd jeanfred jeanfred:x:3076:500:Jean-Frédéric:/home/jeanfred:/bin/bash ``` The python script is doing something similar via `pwd.getpw... [22:50:27] 06Labs, 10Tool-Labs: Broken unicode characters / invalid UTF-8 on Tool Labs index - https://phabricator.wikimedia.org/T164971#3253687 (10bd808) I live hacked the admin tool and found that it is connecting to mysql with a characterset of `utf8mb4`. If I change that to `utf8` things are still messed up in the sa... [22:50:58] !log tools.admin Live hacked for utf-8 character encoding problem (T164971) [22:51:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [22:51:01] T164971: Broken unicode characters / invalid UTF-8 on Tool Labs index - https://phabricator.wikimedia.org/T164971 [22:52:55] cscott: you are a tool labs user now! When you grep the dumps, don't try to uncompress them. That melts the NFS server [22:55:05] the meta question of should this be a Labs project or just use Tool Labs is sticky. Mostly the NFS servers got really sad with everyone using them and we made it trickier to get FNS access in an arbitrary project. (maybe too tricky) [22:55:16] s/FNS/NFS/ [22:57:53] Hello, Amitie 10g here [23:00:43] I'm currently running two tasks in HHVM and I want to know the exact ammount of memory them're consuming right now. [23:02:10] Hi all. I would like to host a copy of a wikimedia wiki on my vps for tools development. What is the best way to do that?