[00:59:36] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Armyboy2 was modified, changed by Armyboy2 link https://wikitech.wikimedia.org/w/index.php?diff=1063908 edit summary: [01:02:21] 06Labs, 10Tool-Labs: Remove modules/toollabs/files/host_aliases - https://phabricator.wikimedia.org/T109485#2840455 (10scfc) a:05scfc>03None [02:19:27] Any reason I can't send email with /usr/bin/jsub ? [02:35:02] 06Labs, 10Tool-Labs: /data/project/aaaaaa and /data/project/prometheus are owned by root - https://phabricator.wikimedia.org/T152170#2840634 (10scfc) [02:41:23] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Armyboy2 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1064197 edit summary: [02:43:54] 06Labs, 10Tool-Labs, 06Security-Team: Allow tools to have their own ".tools.wmflabs.org" subdomain - https://phabricator.wikimedia.org/T125589#1991911 (10Krinkle) >>! In T125589#1992011, @yuvipanda wrote: > The only problem would be that we already have advertised in some places login.tools.wmflabs.org, but... [04:32:37] andrewbogott, Krenair chasemp you know what .. i think i was mistaken .. my muscle memory just now typed ssh as '@promethium.wikitextexp.eqiad.wmflabs' which means that is how i must have logged on all the time ... not sure why i tried wmflabs.org that time in the afternoon. so, false alarm. [04:33:37] Ok :) [08:03:51] 06Labs, 10Tool-Labs: Missing php5-mcrypt module from tools-exec-14xx - https://phabricator.wikimedia.org/T149810#2840935 (10Incola) I'm sorry, I don't know how to discover on what host the job was executed. I have tried the following: ``` tools.lists@tools-bastion-03:~$ jsub -release=precise -o precise.out p... [10:19:19] PROBLEM - Puppet run on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:26:32] How to find out if a job was killed because of too high memory usage? I noticed one of my jobs not getting done and it probably used more memory than I set as limit. [10:27:01] Fixed it by raising the limit, now just wondering how I can confirm it was killed/crashed because of too high memory usage [10:51:48] 06Labs, 10Labs-Infrastructure, 10DBA: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2841178 (10Marostegui) [10:52:33] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2841193 (10Marostegui) [10:52:35] 06Labs, 10Labs-Infrastructure, 10DBA: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2841192 (10Marostegui) [10:57:41] 06Labs, 10Labs-Infrastructure, 10DBA: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2841202 (10Marostegui) [10:59:20] RECOVERY - Puppet run on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [11:00:09] 06Labs, 10Labs-Infrastructure, 10DBA: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2841205 (10Marostegui) [11:29:42] 06Labs, 10Recommendation-API: Request increased quota for recommendation-api labs project - https://phabricator.wikimedia.org/T152120#2841261 (10schana) [11:35:37] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2841276 (10Marostegui) Compression is done. s1 and s3 are now compressed (1.3T in total). I... [11:36:49] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2841278 (10jcrespo) Let's load the events and close this as resolved. [11:37:15] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2841279 (10jcrespo) a:03Marostegui [11:38:17] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2841280 (10Marostegui) Yes, I just wanted to load the events once the first data is copied ove... [11:43:07] 06Labs, 10Labs-Infrastructure, 10DBA: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2841317 (10Marostegui) The transfer from db1095 to labsdb1010 has just started [14:01:41] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2841520 (10jcrespo) [14:43:18] 06Labs, 10Tool-Labs: Missing php5-mcrypt module from tools-exec-14xx - https://phabricator.wikimedia.org/T149810#2841616 (10scfc) Thanks for the detailed report. I remembered reading something similar, and it turns out this issue had already been filed as T97857 which was closed because the user did not use `... [14:44:20] 06Labs, 10Labs-Team-Backlog, 10Tool-Labs: Tool Labs: Enable php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1253572 (10scfc) [14:44:22] 06Labs, 10Tool-Labs: Missing php5-mcrypt module from tools-exec-14xx - https://phabricator.wikimedia.org/T149810#2841629 (10scfc) [14:44:47] 10Tool-Labs-tools-Other: Croptool does not work on php 5.4 - https://phabricator.wikimedia.org/T103059#2841632 (10scfc) [14:44:49] 06Labs, 10Tool-Labs, 07Tracking: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#2841633 (10scfc) [14:44:51] 06Labs, 10Labs-Team-Backlog, 10Tool-Labs: Tool Labs: Enable php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1253572 (10scfc) 05declined>03Open a:03scfc [14:47:09] multichill: usually a program / script itself will clue you in to it being out of mem, there isn't an external solution [15:23:28] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2841740 (10Marostegui) The data has been copied over. events have been loaded. [15:23:39] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision with data the new labsdb servers and provide replica service with at least 1 shard from a sanitized copy from production - https://phabricator.wikimedia.org/T147052#2841744 (10Marostegui) [15:23:41] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2841743 (10Marostegui) [15:23:43] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2841742 (10Marostegui) 05Open>03Resolved [15:28:42] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2841770 (10Marostegui) I messed the data a bit when setting up the slave, I will transfer the... [15:30:18] chasemp: Does the grid engine keep track somewhere of return codes of past jobs? [15:31:00] I'm not entirely sure, there is an accounting of past jobs but I'm not sure what exactly exists there, it may very well be on start only [15:31:36] mark: Interested in getting some form of clarity about https://phabricator.wikimedia.org/T95714 and https://phabricator.wikimedia.org/T78167 ? Currently it is a mess...... [15:31:41] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision with data the new labsdb servers and provide replica service with at least 1 shard from a sanitized copy from production - https://phabricator.wikimedia.org/T147052#2841777 (10Marostegui) [15:31:43] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2841775 (10Marostegui) 05Resolved>03Open [15:33:04] chasemp: Python gave an interesting exception: KeyboardInterrupt. Quite special for a non-interactive job ;-) [15:33:21] multichill: it's possible you meant mafk who is not in the channel atm? [15:33:25] yes that is interesting [15:33:40] multichill: if you want to experiement, run teh job again and see it killed and drop me hte job id [15:33:47] and I"ll see what exists from SGE's perspectiv [15:33:48] e [15:33:53] but it probably won't be today [15:34:59] Don't bother, I was just wondering if I missed something in the documentation. [15:35:32] SGE is not too context heavy on what happens to a job after it's scheduled and running, it knows if it's dead or not afaict [15:36:59] It's easy to reproduce in python, just start a python script with jsub -once -mem 1m and have while True: add something to a list. [15:39:20] I meant mark, he commented on the task [15:40:54] 06Labs, 10Labs-Infrastructure, 10DBA: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2841786 (10jcrespo) These should be the ideal grants for new users: ``` GRANT USAGE ON *.*... [15:52:30] 06Labs, 10Horizon, 13Patch-For-Review, 07Upstream: Increase horizon session length - https://phabricator.wikimedia.org/T130621#2140896 (10scfc) Is this still an issue? I logged into Horizon yesterday 10:00Z, and when I revisited Horizon just now I'm still logged in, more than 24 hours later. [16:08:30] 06Labs, 10Quarry, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry is allowed to be hosted there - https://phabricator.wikimedia.org/T152212#2841828 (10yuvipanda) [16:14:41] 06Labs, 10Quarry, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry and PAWS are allowed to be hosted there - https://phabricator.wikimedia.org/T152212#2841853 (10yuvipanda) [16:18:02] 06Labs, 10Quarry, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry and PAWS are allowed to be hosted there - https://phabricator.wikimedia.org/T152212#2841856 (10yuvipanda) [16:28:55] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Implement a frontend failover solution for labsdb replicas - https://phabricator.wikimedia.org/T141097#2841887 (10jcrespo) ``` $ mysql -h labsdb-web -u u -p$PASS Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connect... [16:29:51] 06Labs, 10Labs-Infrastructure, 10DBA: Provide at least 2 separate service endpoints: one for slow, long running queries; and another for quick, web requests - https://phabricator.wikimedia.org/T147051#2841893 (10jcrespo) See progress here: T141097#2841887 [16:40:29] 06Labs, 10Labs-Infrastructure, 07Puppet: realm.pp: "Data retrieved from Toolsbeta is String not Hash" if not defined in Hiera - https://phabricator.wikimedia.org/T152142#2841923 (10scfc) The error message appears in: ``` [tim@passepartout ~/src/operations/puppet]$ git grep 'Reading data from' modules/admin/... [16:46:25] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, 10wikitech.wikimedia.org: Move novaobserver (and novaadmin) users out of ldap - https://phabricator.wikimedia.org/T152215#2841937 (10Andrew) [16:51:17] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, and 2 others: Provide instance level ro access to OpenStack APIs - https://phabricator.wikimedia.org/T150092#2841969 (10chasemp) [16:56:37] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, and 2 others: Provide read-only access to OpenStack APIs from WMF IP space - https://phabricator.wikimedia.org/T150092#2841976 (10chasemp) [17:10:38] 06Labs, 10Labs-Infrastructure, 10DBA: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2842008 (10yuvipanda) a:03yuvipanda [17:11:37] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2842026 (10Marostegui) [17:11:39] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision with data the new labsdb servers and provide replica service with at least 1 shard from a sanitized copy from production - https://phabricator.wikimedia.org/T147052#2842027 (10Marostegui) [17:11:41] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2842024 (10Marostegui) 05Open>03Resolved The data has been copied over and labsdb1010 is n... [17:11:50] chasemp: Took a bit of digging, but qacct -j 597187 returns it (takes a while to load). exit_status 143 (SIGTERM) maxvmem 1014.867M (and I set 1000 as the limit). So looks like the scheduler killed it for using too much memory :-) [17:12:06] ah [17:12:13] nice [17:12:54] One of the things with job schedulers is that usually the accounting is quite good because it would be used for billing ;-) [17:13:55] It's a bit weird to have two jobs with the same job id (597187) [17:14:16] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2842031 (10Marostegui) labsdb1010 has now data from db1095 and it is catching up. I have applied the events too: https://phab... [17:15:00] 06Labs, 10Tool-Labs, 10puppet-compiler: toolsbeta: set up puppet-compiler / temporary-apply - https://phabricator.wikimedia.org/T97081#2842033 (10hashar) > @hashar: how much work would it be to create an extra puppet-compiler job in Jenkins for this? I am not sure. The Jenkins job is not yet in the configur... [17:30:04] hey YuviPanda! is there a way to access PAWS (not internal) via ssh? [17:31:50] zareen: unfortunately not yet. what do you want to use it for? [17:36:14] YuviPanda: i need to upload a compressed dataset that's 193 MB. the manual upload is limited to 25 MB. i used the wget package to download the other datasets needed from my website, but this one is much larger and i'm having issues uploading to my site. was wondering if i could scp to move it back to PAWS. [17:37:00] zareen: ah, I see. do you have a tool labs account? if you do you can scp to your homedirectory and then I can 'behind the scenes' move it to your PAWS home [17:38:25] YuviPanda: i don't, but i can make one and do that :) i'll let you know once it's in my home directory. thanks [17:38:59] zareen: yw! am in a totally different tz now tho (+0530, for the next 3 weeks) so might have a bit different availability. [17:39:46] YuviPanda: no worries, it's not an urgent issue [17:39:56] zareen: ok :) [17:40:01] zareen: you can scp to paws-internal btw [17:40:46] YuviPanda: yah, i knew that but don't think it helps me for this since i want it to be a public dataset :) [17:41:07] zareen: \o/ awesome [17:41:08] ok [17:43:39] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Zareen was created, changed by Zareen link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Zareen edit summary: Created page with "{{Tools Access Request |Justification=PAWS |Completed=false |User Name=Zareen }}" [17:44:38] zareen: let me grant you access now [17:45:27] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Zareen was modified, changed by Yuvipanda link https://wikitech.wikimedia.org/w/index.php?diff=1066984 edit summary: [17:45:42] zareen: done [17:46:59] zareen: not done yet, the 'add user' is still spinin [17:48:10] zareen: ok done now [17:49:25] YuviPanda: okay, cool, i can ssh into tool labs from terminal right? [17:49:51] zareen: from terminal on PAWS? or your local terminal? [17:50:05] YuviPanda: local terminal [17:50:26] zareen: yup! you need to set an ssh key in wikitech under 'preferences' tho [17:51:34] YuviPanda: perfect, already done :) [17:51:46] zareen: ok! let me know when your scp is done [18:44:40] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: Find a method for `templatetiger` to sort large files - https://phabricator.wikimedia.org/T124822#2842294 (10chasemp) >>! In T124822#2082303, @chasemp wrote: > Is this still an issue? FWIW `scratch [18:46:08] 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: overhaul labstore setup [tracking] - https://phabricator.wikimedia.org/T126083#2842298 (10chasemp) [18:46:10] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: Find a method for `templatetiger` to sort large files - https://phabricator.wikimedia.org/T124822#2842296 (10chasemp) 05Open>03Resolved a:03chasemp [19:54:59] hey zareen [19:55:33] hey YuviPanda: could you please expire my old ssh keys to labs? i just updated it with a new one [19:56:26] zareen: so all keys are self-managed - you can see the ones you currently have in wikitech.wikimedia.org under 'preferences', and delete ones you don't want [19:56:52] oh perfect! [19:59:59] YuviPanda: hm, i'm still getting "Permission denied (publickey,keyboard-interactive,hostbased)." when trying to ssh into labs [20:00:30] zareen: what command are you using to ssh? [20:00:50] YuviPanda: ssh login.tools.wmflabs.org [20:02:40] zareen: am watching the logs now can you try again [20:03:18] zareen: > input_userauth_request: invalid user ZareenFarooqui [preauth] [20:03:25] YuviPanda: okay, trying now [20:03:27] zareen: you need to use ssh @login.tools.wmflabs.org [20:03:33] ahhhhh [20:03:46] your shell name that you picked when you created account, you can find it in preferences again if you forgot [20:05:39] YuviPanda: tried it again with my shell name and still got the same error [20:05:57] zareen: right, i see that. looking [20:06:36] hmm zareen is the correct username [20:07:00] zareen: can you add a '-v' flag to ssh and paste the output for me? [20:09:20] YuviPanda: OpenSSH_6.9p1, LibreSSL 2.1.8 [20:09:20] debug1: Reading configuration data /Users/ZareenFarooqui/.ssh/config [20:09:20] debug1: Reading configuration data /etc/ssh/ssh_config [20:09:21] debug1: /etc/ssh/ssh_config line 21: Applying options for * [20:09:22] debug1: Connecting to login.tools.wmflabs.org [208.80.155.163] port 22. [20:09:23] debug1: Connection established. [20:09:24] debug1: identity file /Users/ZareenFarooqui/.ssh/id_rsa type 1 [20:09:25] debug1: key_load_public: No such file or directory [20:09:26] debug1: identity file /Users/ZareenFarooqui/.ssh/id_rsa-cert type -1 [20:09:27] debug1: key_load_public: No such file or directory [20:09:28] debug1: identity file /Users/ZareenFarooqui/.ssh/id_dsa type -1 [20:09:29] debug1: key_load_public: No such file or directory [20:09:31] debug1: identity file /Users/ZareenFarooqui/.ssh/id_dsa-cert type -1 [20:09:31] debug1: key_load_public: No such file or directory [20:09:32] debug1: identity file /Users/ZareenFarooqui/.ssh/id_ecdsa type -1 [20:09:33] debug1: key_load_public: No such file or directory [20:09:34] debug1: identity file /Users/ZareenFarooqui/.ssh/id_ecdsa-cert type -1 [20:09:54] zareen: can you use phabricator.wikimedia.org/paste or dpaste.de or another pastebin service? :) IRC mangles large pastes :) [20:10:02] (and freenode might kick your user for 'flooding' ocassionally) [20:10:58] oops, sorry [20:11:17] np, IRC is ancient and doesn't have autopaste built in [20:12:11] sure seems like the key path is bad locally? [20:13:29] that could be it [20:13:33] yes, it does seem like something on my end [20:13:42] zareen: try adding a '-i ' to ssh command? [20:14:54] so `ssh -i `? [20:18:10] zareen: yeah [20:19:47] usage: ssh [-1246AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec] [20:19:47] [-D [bind_address:]port] [-E log_file] [-e escape_char] [20:19:48] [-F configfile] [-I pkcs11] [-i identity_file] [20:19:59] [-L [bind_address:]port:host:hostport] [-l login_name] [-m mac_spec] [20:20:00] [-O ctl_cmd] [-o option] [-p port] [20:20:00] [-Q cipher | cipher-auth | mac | kex | key] [20:20:08] [-R [bind_address:]port:host:hostport] [-S ctl_path] [-W host:port] [20:20:08] [-w local_tun[:remote_tun]] [user@]hostname [command] [20:21:30] zareen: uh, what's the full command you used? [20:22:29] ssh -i /Users/ZareenFarooqui/.ssh/wmf_tools [20:23:01] ah, try [20:23:09] ssh -i /Users/ZareenFarooqui/.ssh/wmf_tools zareen@login.tools.wmflabs.org [20:23:20] 06Labs: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#2842545 (10chasemp) [20:24:10] YuviPanda: it worked! :) [20:24:20] zareen: w00t! ok :) [20:25:00] YuviPanda: so i can scp my file to /home/zareen and you can access it, right? [20:25:11] zareen: yup [20:30:31] YuviPanda: thanks so much!! scp'ing now :) [20:31:10] zareen: np! I'm going afk for like 20min, I'll copy it over when I'm back. let me know of the name of the file when it's done scping [20:31:53] YuviPanda: sorry if i'm keeping you up :( it's enwiki_20161101_headings_2.tsv.bz2 [20:32:15] zareen: np. is it done scping? [20:32:21] YuviPanda: yes [20:32:28] ah ok [20:32:43] YuviPanda: i can wait till you're back though [20:33:43] zareen: nah, I'm just doing it now [20:33:53] should be done shortly [20:34:48] YuviPanda: sweeeet! have a good weekend :) [20:35:25] zareen: you too! copy done :) [20:35:31] I'll try to simplify this later [20:36:47] or raise the manual upload limit :) [21:04:12] 06Labs: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#2842545 (10scfc) I think this is a duplicate of T68623. Note that blocking that issue is not missing man-power, but technical impossibilities. [21:09:33] 06Labs, 10Labs-Team-Backlog, 10Tool-Labs, 13Patch-For-Review: Tool Labs: Enable php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#2842712 (10yuvipanda) Can someone test if this is an issue in the debian jessie kubernetes images we use? You can with the following: ``` webservice stop ``` `... [21:16:17] 06Labs, 10Labs-Team-Backlog, 10Tool-Labs, 13Patch-For-Review: Tool Labs: Enable php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#2842726 (10yuvipanda) Answered myself: yes, it works fine in the jessie images. So one way to get support for this would be to just move to k8s using the command... [21:40:44] chasemp: https://phabricator.wikimedia.org/T151570 Can you update meta_p as well per Krenair's comment? [21:41:08] https://tools.wmflabs.org/guc/?src=hr&by=date&user=Krinkle should include an edit on fi.wikivoyage.org but doesn't. [21:41:29] Krinkle: not at the moment, or at least that script as-is hasn't been run every by anyone here. I think jynus did the last upate for new wiki manually. [21:41:53] I'm looking at the maintain-meta_p script a bit but I'm not confident in it enough to jump right now [21:42:01] inserting a row manually with at least dbname, slice, family, lang and url would suffice [21:44:42] understood, I think we can do it but I prefer not to w/o asking jynus a few things and I don't want to muck around on his friday night when it could easily wait until monday, at least afaik [21:45:02] if you CC me and him on that task for it I'll make sure to talk to him [21:45:16] "Could not find data item k8s_infrastructure_users in any Hiera data file and no default supplied" <-- I can't find where that is supposed to be defined for Tools -- not in https://wikitech.wikimedia.org/wiki/Hiera:Tools , not in Horizon, and can't find it in operations-puppet either.... [21:45:26] OK. It does mean that until Monday, vandals may edit on that wiki without being seen by various tools. [21:48:10] Krinkle: I appreciate that, but we are only human. It's possible that wiki shouldn't have gone live otherwise w/o these components? I don't know, doing the best we can man. [21:48:58] ...because it's in private hieradata. Ok, that explains... [21:50:39] chasemp: No worries! I'm not blaming this on you :) - Ideally this is better coordinated and done at once when the wiki is created. This needs better streamlining and documentation. [21:51:16] But yeah, it's been 5 days and the wiki is still shaking on all sides and being pieced together bit by bit. [21:52:11] I'm sure the editors appreciate being able to put things in place ahead of time this way (with basic editing working etc.) - but it is probably better for next time if the wiki wasn't accessible until the backend is all sorted out. [21:53:31] agreed [21:53:44] there is an onwiki check list somewhere for this process including view generation "_p" etc [21:53:50] I don't know if meta_p is on that [21:53:56] maybe that would help make it a first class citizen [21:56:32] We don't block wiki creation on this stuff [21:56:34] We expect it to just work [21:57:11] (03PS1) 10Merlijn van Deen: Add tools hiera common.yaml [labs/private] - 10https://gerrit.wikimedia.org/r/325041 [21:58:24] * Krinkle edits the Add_a_wiki page [21:59:16] sure but nothing ever just works :) so the if we have a guide it should probably be procedural and single threaded [22:46:30] (03PS1) 10Merlijn van Deen: Add fake clushuser keypair [labs/private] - 10https://gerrit.wikimedia.org/r/325050 [22:50:43] what's the difference between "labs" and "tool labs"? [22:52:06] NOTNOTABLE: https://wikitech.wikimedia.org/wiki/Labs_labs_labs [22:55:58] do personal accounts also get replica access? [22:57:27] NOTNOTABLE: yes, but generally you should try to do as much as possible from a tool account. Alternatively, you can use quarry.wmflabs.org [22:58:14] oh, that'd probably do what I was looking for, thanks [23:05:22] 06Labs, 10Tool-Labs, 10puppet-compiler: toolsbeta: set up puppet-compiler / temporary-apply - https://phabricator.wikimedia.org/T97081#2843075 (10valhallasw) >>! In T97081#2840111, @scfc wrote: > CMIIW, but is the compiler really working? http://tools-puppet-compiler.wmflabs.org/324623000/ says "Hosts that... [23:30:12] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:33:29] 10Tool-Labs-tools-Xtools: xtools: remove circular (language) URLs - https://phabricator.wikimedia.org/T122326#2843111 (10Matthewrbowker) 05Open>03Resolved Fixed in https://github.com/x-tools/xtools-rebirth/commit/8f55f29738ef492544270b3dfd6e289d069a3735 - part of xTools Rebirth. [23:33:32] 10Tool-Labs-tools-Xtools, 07I18n: Update Intuition to no longer use deprecated functions - https://phabricator.wikimedia.org/T138527#2843114 (10Matthewrbowker) 05Open>03Resolved Fixed in https://github.com/x-tools/xtools-rebirth/commit/1f16d7399c396dbf929a2a4823a773aa8e7988bb - part of xTools Rebirth. [23:40:04] 06Labs, 10Tool-Labs, 10puppet-compiler: toolsbeta: set up puppet-compiler / temporary-apply - https://phabricator.wikimedia.org/T97081#2843122 (10hashar) That is an impressive number of patches Merlijn :) > What credentials does the master use to connect (there is no pubkey in /mnt/home/jenkins-deploy). T... [23:47:48] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 1.20 ms [23:52:07] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22)