[00:05:02] scfc replied with https://phabricator.wikimedia.org/T157744 (see bottom) [00:05:09] however, if I do webservice --backend=kubernetes python2 start [00:05:16] still get that error message of icu could not be imported stuff in my uwsgi.log (for more info on what exactly I did, just go a message or two up from scfc's latest reply) [00:37:43] zhuyifei1999_: ^^ [02:04:21] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [04:12:06] anyone around? i cannot connect to bastion - labs, i think my access got accidentally revoked during the "cleanup" [04:12:23] yurik: o/ [04:12:28] hi JustBerry [04:12:29] (I can login btw) [04:12:36] I know that doesn't help much [04:12:41] herm. [04:12:43] sure, i think its just my account that is ... misconfigured [04:12:47] o [04:12:54] can you make a phab ticket [04:13:04] makes sense [04:13:24] yurik: what's the error, etc. [04:13:25] yurik: bastion.wmflabs.org or one of bastions of tool labs? [04:13:35] zhuyifei1999_, the first [04:13:39] i haven't tested tools [04:13:54] zhuyifei1999_, actually both are blocked [04:14:00] connection closed by remote host [04:14:13] have you tried ssh -vvv? [04:14:23] i.e. very very verbose [04:14:31] zhuyifei1999_: !!! [04:14:40] 'ey ^^ [04:14:42] JustBerry: ? [04:14:50] zhuyifei1999_: so icu does work on the bastion [04:15:04] JustBerry: it may work on grid [04:15:06] zhuyifei1999_: true or false: I need to install icu in my local venv to get icu working for kubernetes? [04:15:15] true [04:15:18] 06Labs: Restore yurik's access to wmflabs - https://phabricator.wikimedia.org/T158028#3024078 (10Yurik) [04:15:27] I tried, more difficult than I thought [04:16:17] 06Labs, 10Tool-Labs: Restore yurik's access to wmflabs - https://phabricator.wikimedia.org/T158028#3024090 (10JustBerry) [04:16:21] zhuyifei1999_: yarp. [04:16:30] yurik: have you tried ssh -vvv? [04:16:50] 06Labs: Restore yurik's access to wmflabs - https://phabricator.wikimedia.org/T158028#3024091 (10zhuyifei1999) [04:16:58] zhuyifei1999_, sec, pasting gist [04:17:21] k [04:17:42] zhuyifei1999_: so I actually figured out that the issue is the lib48 thing [04:17:42] JustBerry: yurik's issue isn't specific to tool labs [04:17:45] did you get the same error? [04:17:51] I was just going to say that too zhuyifei1999_ [04:17:56] I realized afterwards ;p [04:18:00] ... [04:18:30] JustBerry: I have no time to investigate closely. i.e. with ptrace [04:18:33] zhuyifei1999_: I was thinking about it and you removed it [04:18:37] the tag [04:18:39] but regardless [04:18:49] zhuyifei1999_: I mean what error did you get when you tried to [04:18:57] "I tried, more difficult than I thought" [04:27:34] JustBerry: the .so one [04:28:07] I would ptrace if I have time [04:28:43] 06Labs: Restore yurik's access to wmflabs - https://phabricator.wikimedia.org/T158028#3024093 (10Yurik) [04:35:51] zhuyifei1999_: I looked at the service [04:35:58] zhuyifei1999_: issue with the .so is the version check [04:36:09] it looks for another version of the lib [04:36:13] when a later version is installed [04:36:25] hence the . not found [04:52:36] 10Tool-Labs-tools-Other: tools.spiarticleanalyzer: requesting installation of icu on bastion and kubernetes - https://phabricator.wikimedia.org/T157744#3024109 (10JustBerry) @scfc `webservice --backend=kubernetes python2 start` (kubernetes), though, yields the output posted earlier in the ticket (ImportError: No... [04:56:29] zhuyifei1999_: _icu.so, right? [05:02:55] 06Labs: Restore yurik's access to wmflabs - https://phabricator.wikimedia.org/T158028#3024078 (10scfc) (Interestingly, https://wikitech.wikimedia.org/wiki/Special:UserRights/Yurik shows `shell` still enabled which //should// correspond to whether an account is in the bastion project or not (and thus allowed to l... [08:48:31] 06Labs, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#3024250 (10Magnus) Yes, that redirect sounds sensible. Could someone do that on the VM config please, or do I have to change WDQ source code? [10:17:27] 06Labs, 10wikitech.wikimedia.org: Wikitech: rename 'shellmanagers' to 'shellmanager' for consistency - https://phabricator.wikimedia.org/T158039#3024432 (10MarcoAurelio) [10:19:50] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2054159 (10MarcoAurelio) Please, just don't. Wikitext talk pages are working just... [12:08:26] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3024759 (10hashar) About Zend 5.3 / CI: Context ============== Candidates were to use https://github.com/phpenv/phpenv | https://github.com/CHH/phpenv but they are not packaged for Deb... [12:49:16] 06Labs, 10Tool-Labs: /usr/local/bin/killgridjobs.sh is outdated - https://phabricator.wikimedia.org/T157792#3016568 (10chasemp) This has been superceded I believe by[[ https://github.com/wikimedia/operations-puppet/blob/35098e1e363e10696c2a19579fc141ff24480997/modules/toollabs/files/exec-manage | exec-manage... [13:04:46] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:13:43] 06Labs, 10Labs-Infrastructure: Investigate if --force should be the default for nfs-manage up - https://phabricator.wikimedia.org/T157478#3024919 (10chasemp) Since we have our own cautious check for anything displaying `Primary` I think possibly we could prompt the user if we find that string to acknowledge th... [13:44:47] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:06] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3025145 (10MoritzMuehlenhoff) I think we should simply drop 5.3 from the CI tests, then. I wasn't aware that the PHP versions had to be co-installable, which makes a custom 5.3 build for... [15:06:24] 10Tool-Labs-tools-Other: tools.spiarticleanalyzer: requesting installation of icu on bastion and kubernetes - https://phabricator.wikimedia.org/T157744#3025326 (10tom29739) PyICU requires the underlying C++ library to work, and this is in the `libicu52` apt package on Debian Jessie. When this is installed, PyICU... [15:09:07] 06Labs, 10Labs-Infrastructure, 10Tool-Labs, 10DBA, 10Wikimedia-Developer-Summit (2017): Labsdbs for WMF tools and contributors: get more data, faster - https://phabricator.wikimedia.org/T149624#3025336 (10jcrespo) 05Open>03Resolved I am going to resolve this as fixed, even if we have yet to finish th... [15:11:00] 06Labs, 10Labs-Infrastructure, 10DBA, 07Epic, 07Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#2475959 (10jcrespo) [15:11:02] 06Labs, 10Labs-Infrastructure, 10DBA: Having lots of accounts with separate grants makes auditing difficult. - https://phabricator.wikimedia.org/T141096#3025345 (10jcrespo) 05Open>03Resolved a:03jcrespo This is now done- users are handled with roles + accounts are handled on a database. Resolving mysel... [15:13:20] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Create a cronjob/check to run check_private_data data script and report back - https://phabricator.wikimedia.org/T153680#3025371 (10Marostegui) I have pushed that change and I have run puppet on labsdb1009 and the change is looking good. We will se... [15:19:50] 10Tool-Labs-tools-Other: tools.spiarticleanalyzer: requesting installation of icu on bastion and kubernetes - https://phabricator.wikimedia.org/T157744#3025395 (10zhuyifei1999) [15:19:54] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 07Tracking: Packages to be installed in Tool Labs Kubernetes Images (Tracking) - https://phabricator.wikimedia.org/T140110#3025396 (10zhuyifei1999) [15:23:32] 06Labs, 10DBA, 06Operations, 07Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#3025409 (10jcrespo) [15:23:35] 06Labs, 10Labs-Infrastructure, 10DBA, 07Chinese-Sites: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#3025407 (10jcrespo) 05Open>03declined labsdb1001 and labsdb1003 will be decommissioned at some point in the near future- this will be fixed, but no... [15:24:52] 06Labs, 10Labs-Infrastructure, 10DBA, 07Chinese-Sites: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#3025411 (10jcrespo) Also, the initial report was fixed, it was reopened because other issues tarted to get mixed there. [15:32:23] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Get HA db support for labs internal services - https://phabricator.wikimedia.org/T126251#3025423 (10jcrespo) a:05jcrespo>03None Blocked on performing a failover to the proxy by service owners, although I am not sure if failing over to codfw is a... [15:32:41] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Install Python - https://phabricator.wikimedia.org/T157465#3025426 (10Samtar) p:05Triage>03Lowest [15:33:07] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Add WYMeditor - https://phabricator.wikimedia.org/T156566#3025429 (10Samtar) p:05Low>03Lowest [15:33:10] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Add tag functionality - https://phabricator.wikimedia.org/T155343#3025430 (10Samtar) p:05Low>03Lowest [15:33:45] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Localization - https://phabricator.wikimedia.org/T155738#3025431 (10Samtar) p:05Normal>03Lowest [15:33:48] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Static user testing - https://phabricator.wikimedia.org/T157682#3025432 (10Samtar) p:05Normal>03Lowest [15:49:58] 06Labs, 10DBA: Lots of rows are missing from enwiki_p.`revision` - https://phabricator.wikimedia.org/T115207#3025476 (10jcrespo) 05Open>03declined I have given up on trying to resync production to the current labsdbs. It is almost impossible while they are in use. The soon to be setup servers wont have tha... [15:50:02] 06Labs, 10DBA, 06Operations, 07Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#3025478 (10jcrespo) [15:53:08] 06Labs, 10Tool-Labs-tools-Other, 10DBA, 13Patch-For-Review: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553#3025500 (10jcrespo) 05stalled>03Resolved [16:01:23] ALL: I'll be upgrading Horizon for the next little while, so best to stand back if you don't have urgent needs. [16:04:45] …if/when jenkins reviews my one-line patch [16:08:30] * paladox is wondering what changes this upgrade to horizon will do? [16:12:40] It moves a few workflows into JS so they're more responsive on the clientside. And in general the site should load more quickly (for unrelated reasons) [16:13:52] Oh thanks :), i see users can also change themes now. Thats according to http://docs.openstack.org/releasenotes/horizon/mitaka.html [16:18:59] andrewbogott: chasemp: good morning :-} Do we have any logs of compute/kvm booting instances? It seems the instances takes slightly longer to come up [16:19:29] hashar: chase is working on some metrics for that but we don't have much historical data so far [16:19:31] based on https://grafana.wikimedia.org/dashboard/db/nodepool?panelId=11&fullscreen&from=now-90d&to=now :) [16:19:34] yeah noticed that [16:19:42] the nova full stack metrics that is exciting [16:20:01] btw, chase, we should decide if that 'testing' image is good or not, and rename things. It's confusing to users to have two competing Jessie images on offer. [16:20:39] something I noticed, is whenever you get a new images, the first few instances take a while to boot. I presume for the image to be fetched on the compute node from glance [16:20:54] so your nova full stack metrics could have some slight bump whenever you produce new images [16:21:29] andrewbogott: yes sorry, I'll try to do that soon here [16:22:48] right, I haven't actually switched over to that image so I imagine there is no way ci could be using it? [16:22:55] PROBLEM - Puppet staleness on tools-docker-builder-04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [16:23:30] chasemp: I'm not sure if CI uses our images at all… hashar do you have your own base images for that? [16:23:36] partly [16:23:45] paladox: can you log in and see the horizon UI now? Look ok? [16:23:52] Yep [16:23:55] Though one problem [16:24:00] what is left running on the permanent slaves use whatever wmflabs image you provided at the point the image spawned. Which should be form 1+ year ago [16:24:03] i see the admin section [16:24:21] and it causes me to get logged out everytime i click on it. As i thought that was a new feature. [16:24:39] dammit, they fixed that bug in one place and clearly broke it in another [16:24:44] I'll have a look, thank you. [16:24:47] I like the new theme [16:24:51] Material [16:24:55] the nodepool images have their own images that are pushed to Glance ( image-ci-trusty and image-ci-jessie ) attached to the contintcloud project [16:25:11] hashar: ok, right, just making sure that there's no interaction between nodepool (e.g. nodepool slowdown) and new images we might install. [16:25:27] PROBLEM - Puppet run on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:25:30] will boot an instance and ci how long it takes to boot :) [16:27:00] andrewbogott still takes a while to load the puppet tab on the instance (horizion ui) [16:27:16] ALL: Horizon should be back up and fine. I'll be ironing out some clumsy GUI issues over the day today but everything should be useable in the meantime. [16:27:24] (7 seconds to boot a jessie one :D ) [16:27:41] paladox: yeah, as part of this upgrade I (intentionally) reverted a performance hack that was running on the old horizon. I may have to bring it back. [16:27:48] (18 for a trusty one) [16:27:52] oh [16:28:16] paladox: this one: https://gerrit.wikimedia.org/r/#/c/335593/ [16:28:22] everyone should try this material skin. Really nice. [16:28:28] oh [16:28:58] oh man, the puppet gui is WAY faster though. Still too slow, but much faster. [16:29:30] Oh [16:30:20] …unless I'm fooling myself. It seems faster to me but I haven't done time trials. [16:30:25] yeh faster, though i noticed its faster in the default ui. In the material one its slower. [16:31:06] andrewbogott i guess they applied the performance improvements to the default ui but not to the material skin. [16:31:46] It could me that the page download is just that much bigger with the other skin [16:31:51] *be [16:32:10] oh, i only noticed the speed problems with the puppet tab. Everything else is ok. [16:36:45] paladox: you got kicked out when you clicked to expand the 'Admin' disclosure? Or when you clicked on a particular menu? [16:36:51] (I'm trying to reproduce here) [16:37:09] andrewbogott when i clicked on instances under the admin tab [16:37:28] oh, yep! Boom [16:37:34] well that's impolite [16:37:52] Oh [16:37:53] yep [16:38:11] but when i click on Host Aggregates, it dosent log me out. [16:39:31] andrewbogott as horizion was upgraded to mitaka, does labs have to be upgraded to mitaka? [16:39:59] no, the APIs are versioned. So horizon knows to use the APIs that are supported by the Liberty services. [16:40:13] We probably won't upgrade the other services until next quarter, maybe in April or May. [16:41:39] ok [17:00:26] RECOVERY - Puppet run on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [17:05:41] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025811 (10chasemp) On all three new servers :) ```2017-02-14 17:03:54,237 DEBUG SQL: CREATE OR REPLACE DEFINER=vie... [17:08:40] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025834 (10jcrespo) gnwikibooks? [17:09:50] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025835 (10jcrespo) Try now on labsdb1009. [17:15:30] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025866 (10chasemp) >>! In T153743#3025835, @jcrespo wrote: > Try now on labsdb1009. nope > root@labsdb1009:~# maintain-views --databases enwiki --replac... [17:16:19] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025868 (10jcrespo) Then this is a different issue than the `FLUSH STATUS` one. Checking. [17:21:39] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025880 (10Marostegui) >>! In T153743#3025841, @chasemp wrote: >>>! In T153743#3025834, @jcrespo wrote: >> gnwikibooks? > > It seems order of creation for... [17:24:08] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025885 (10jcrespo) Maybe it lacks the grants for replace (drop). I have dropped `[ugwiki_p]> drop view abuse_filter_action;`. Can you run the same query o... [17:26:31] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025893 (10chasemp) Yes, it does exist but to update the views it's easier to say "update all of them to match canonical source replacing any that already... [17:27:09] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025895 (10Marostegui) Could be but the grants are ALL as per this (labsdb1009) so I would assume the replace should go through no?: ``` GRANT ALL PRIVILEG... [17:28:30] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025898 (10chasemp) >>! In T153743#3025885, @jcrespo wrote: > Maybe it lacks the grants for replace (drop). I have dropped `[ugwiki_p]> drop view abuse_fil... [17:33:20] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025952 (10Marostegui) Silly question, are you connecting through `localhost` specifically? I am thinking about the issue we had in September where it was... [17:35:25] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3025971 (10chasemp) >>! In T153743#3025952, @Marostegui wrote: > Silly question, are you connecting through `localhost` specifically? > I am thinking about... [18:07:57] 10Tool-Labs-tools-Other, 13Patch-For-Review: tools.spiarticleanalyzer: requesting installation of icu on bastion and kubernetes - https://phabricator.wikimedia.org/T157744#3026101 (10JustBerry) >>! In T157744#3025326, @tom29739 wrote: > PyICU requires the underlying C++ library to work, and this is in the `lib... [18:10:28] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3026114 (10jcrespo) I've added a workaround that makes no sense but that works for now ,we need to revisit it in the future. Maybe there is a bug on the la... [18:16:17] andrewbogott could this http://docs.openstack.org/developer/horizon/topics/settings.html#openstack-keystone-admin-roles be the setting for it showing the admin thing for normal users? [18:16:37] but they should really make it only log you out only if you press the log out button. [18:16:43] no, but I know what the issue is; I'm working on it [18:17:58] oh ok, thanks :) [18:21:12] andrewbogott: are merges automatically performed after +2? [18:21:42] JustBerry: it depends on repo [18:22:06] andrewbogott: operations/docker-images/toollabs-images [18:22:12] guessing not automatic [18:22:25] no idea [18:23:02] JustBerry does that repo have jenkins tests? [18:23:10] paladox: yup [18:23:17] this: https://gerrit.wikimedia.org/r/#/c/337603/ [18:23:18] Ok [18:23:22] andrewbogott: ^^ [18:23:51] oh isee [18:23:52] oh i see now [18:25:36] JustBerry try a c+2? [18:27:09] 10Tool-Labs-tools-Other, 13Patch-For-Review: tools.spiarticleanalyzer: requesting installation of icu on bastion and kubernetes - https://phabricator.wikimedia.org/T157744#3026276 (10JustBerry) @yuvipanda Since the gerrit patch above probably won't necessarily go live when it is merged, how are we planning to... [18:28:35] paladox: I can't. Perhaps andrewbogott or another WMF dev can take a look [18:28:44] ok [18:30:47] 'ello tgr o/ [18:50:13] 06Labs, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Replicate new pap_parent_id column in page_assessments_projects table on Labs - https://phabricator.wikimedia.org/T158097#3026399 (10kaldari) [18:50:22] 06Labs, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Replicate new pap_parent_id column in page_assessments_projects table on Labs - https://phabricator.wikimedia.org/T158097#3026413 (10kaldari) p:05Triage>03Normal [18:54:19] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 2 others: Create a "my first Python webservice" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134494#3026448 (10Ricordisamoa) >>! In T134494#3020414, @bd808 wrote: > The current tutorial does use mwoauth to do all the... [18:58:56] 06Labs, 10Horizon: Upgrade Openstack Horizon to Mitaka - https://phabricator.wikimedia.org/T158099#3026458 (10Andrew) [19:08:49] 10Tool-Labs-tools-Xtools: [PLAN] Move development for xtools from my repo to the project repo - https://phabricator.wikimedia.org/T158102#3026520 (10Matthewrbowker) [19:09:24] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3026537 (10Matthewrbowker) [19:09:27] 10Tool-Labs-tools-Xtools: [PLAN] Move development for xtools from my repo to the project repo - https://phabricator.wikimedia.org/T158102#3026536 (10Matthewrbowker) [19:10:31] paladox: can you still see the 'admin' tab? [19:10:55] nope [19:10:58] it's gone :) [19:11:31] 10Tool-Labs-tools-Xtools: [PLAN] Move development for xtools from my repo to the project repo - https://phabricator.wikimedia.org/T158102#3026548 (10Matthewrbowker) [19:12:22] 06Labs, 10Horizon, 13Patch-For-Review: Upgrade Openstack Horizon to Mitaka - https://phabricator.wikimedia.org/T158099#3026552 (10Andrew) [19:12:26] awesome, thanks for testing [19:13:15] 06Labs, 10Horizon, 07Developer-notice, 13Patch-For-Review: Upgrade Openstack Horizon to Mitaka - https://phabricator.wikimedia.org/T158099#3026458 (10Andrew) [19:13:51] Your welcome :) [19:14:11] andrewbogott, im just wondering why are we going with mitaka when there is newton? [19:14:20] 06Labs, 10Horizon, 07Developer-notice: Horizon Mitaka 'remember me' checkbox has bad tab order - https://phabricator.wikimedia.org/T158103#3026563 (10Andrew) [19:14:43] paladox: newton is not available on Ubuntu Trusty. so we need to upgrade our whole cluster to Debian before we can go to anything > Mitaka. [19:14:52] Oh [19:14:56] And, anyway, generally I like to do this stuff a version at a time since that's how the release notes are written :( [19:15:00] thanks for explaning :) [19:15:04] sure [19:15:29] 06Labs, 10Horizon, 07Developer-notice, 13Patch-For-Review: Upgrade Openstack Horizon to Mitaka - https://phabricator.wikimedia.org/T158099#3026584 (10bd808) [19:16:02] I think we can create themes in mitaka, unless im wrong and anyone can create themes? (By creating i mean an admin can install themes.) [19:17:19] andrewbogott: yuvi does the labs container rebuilding right? [19:18:46] JustBerry: I think so [19:19:19] Yuvi has done most of it to date, yes. But generally file a phab task explaining what you want and we will get to it as soon as we can [19:22:53] bd808: been done. https://phabricator.wikimedia.org/T157744 [19:23:03] patch uploaded and all [19:23:14] I just wasn't sure if I tagged the right before ;p [19:23:21] the right person* (whops) [19:25:13] 06Labs, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Replicate new pap_parent_id column in page_assessments_projects table on Labs - https://phabricator.wikimedia.org/T158097#3026611 (10kaldari) @Andrew: I heard you might be able to help with this. [19:25:32] that task is a bit of a train wreck to follow. :/ The current ask is just to have libicu added to the python base image? [19:27:34] bd808: yup. [19:27:48] bd808: I understand it's a trainwreck. tom gave a nice summary followed by zhuy's patch [19:29:45] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3026648 (10faidon) My personal order of preference would be (best -> worst): 1. Manually test 1.23 releases with PHP 5.3 (e.g. in a local VM or Vagrant or something); fix PHP 5.3 incompa... [19:30:55] 10Tool-Labs-tools-Other, 10Tools-Kubernetes, 13Patch-For-Review: tools.spiarticleanalyzer: requesting installation of icu on bastion and kubernetes - https://phabricator.wikimedia.org/T157744#3026654 (10bd808) [19:45:22] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3026704 (10MaxSem) I'm strongly in favor of #1. At this point, 1.23 is likely only to have security updates that need to be manually tested and skip the CI pipeline anyway. [19:54:07] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#3026733 (10Thgoiter) >>! In T92963#3020442, @bd808 wrote: > > nothing that can be done here until @Para licenses the code. This sounds pretty much like a [[ https://en.wikipedia.org/wiki/WTFPL | WTFPL license ]]: >>! In T92... [20:04:13] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#3026797 (10yuvipanda) We announced a while ago we're gonna do this on the 15th. [20:05:55] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#3026807 (10jcrespo) Let's use T157358 for this. Postgres is a different beast. [20:15:30] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2946760 (10greg) >>! In T143349#3026648, @faidon wrote: > My personal order of preference would be (best -> worst): > 1. Manually test 1.23 releases with PHP 5.3 (e.g. in a local VM or V... [20:21:21] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#3026879 (10bd808) @Thgoiter **I** can't license it as anything. Only @Para and his existing contributors can do that. [20:27:54] RECOVERY - Puppet staleness on tools-docker-builder-04 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:33:45] bd808 or yuvipanda, could you check - I still cannot connect to bastion.wmflabs or tools. I documented some of it in https://phabricator.wikimedia.org/T158028 [20:35:16] hey yurik, I think andrewbogott would be the best person to check (or fastest anyway), I'll ping him on the task [20:35:32] thx! [20:35:43] 06Labs: Restore yurik's access to wmflabs - https://phabricator.wikimedia.org/T158028#3024078 (10chasemp) @andrewbogott can you check into this? [21:19:14] chasemp: is that with the new image? [21:19:34] Because spinning up the first instance of a given image on a given labvirt is slow since it has to copy over the base first [21:19:58] andrewbogott: no it's w/ the old one [21:20:06] ok, not that then [21:20:32] andrewbogott: fwiw if look at admin-monitoring project they are being created and destroyed w/ the prepend fullstackd atm [21:20:49] and the logs are on labnet1002 at /var/log/upstart/nova-fullstack.log [21:23:14] andrewbogott: unf I think my fullstack timer there is uh cumulative in teh loop so not imipacting I don't think but not useful atm either [21:25:04] yuuup https://graphite.wikimedia.org/render/?width=760&height=544&_salt=1487105077.194&target=servers.labnet1002.nova.verify.fullstack&lineWidth=30&from=-3h [21:25:05] fixing [21:25:18] shoudln't effect creation timer in any way [21:26:04] I really love those new metrics [21:30:02] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#3027208 (10Thgoiter) He did. "Do what you want" = public domain. [21:30:27] yurik: can you try to connect to bastion-01 while I watch the log? [21:34:18] hashar: yeah neato https://graphite.wikimedia.org/render/?width=760&height=544&_salt=1487105077.194&target=servers.labnet1002.nova.verify.creation&lineWidth=30&from=-3h&target=servers.labnet1002.nova.verify.deletion&target=servers.labnet1002.nova.verify.puppet&target=servers.labnet1002.nova.verify.ssh&target=servers.labnet1002.nova.verify.dns&areaMode=stacked [21:34:35] \o/ [21:35:55] the unit is funky 3000.0 M :D [21:36:14] well uh, last result there is not good [21:36:22] :) [21:36:59] isn't OpenStack Ceilometer for the same purpose? [21:37:04] or is that to monitor the guests? [21:37:22] I think it's mostly used for billing purposes [21:37:24] not sure though [21:37:25] ceilometer is more geared towards say billing [21:37:26] yeah [21:37:46] there isn't actually a holistic project for meta heuristics I think [21:37:59] "hello CI you owe us 130493.33 dollars based on this month instances CPU" [21:42:51] $ openstack hypervisor list [21:42:52] ERROR: openstack Policy doesn't allow os_compute_api:os-hypervisors to be performed. (HTTP 403) [21:43:00] I am not privileged enough :] [21:44:42] andrewbogott, sorry, didn't see your message [21:44:47] tri can try it now [21:44:52] *i [21:44:52] sure, I'm still here [21:45:03] andrewbogott, just did [21:45:30] can you paste the message it gave you? [21:46:42] …because it sort of looks like it worked, on my end [21:47:07] yurik: ? [21:47:11] andrewbogott, sec [21:47:28] andrewbogott, it just worked! [21:47:31] i think it got fixed [21:47:37] ok... [21:47:47] I'll do what I just did on tools as well… and maybe you'll be all set. [21:47:48] * andrewbogott shrugs [21:47:50] andrewbogott, the tools still fail [21:47:58] ssh login.tools.wmflabs.org [21:48:06] Connection to login.tools.wmflabs.org closed by remote host. [21:48:35] so yes, it seems someone restored the access to bastion [21:50:44] yurik: try tools again? [22:02:27] andrewbogott, works, thanks!!! [22:02:42] i wonder why my irc notifications don't ping me any more :( [22:02:44] yurik: cool. are there other projects you need or is that it? [22:02:56] Bascially all I did was remove you and re-ad you, which forces a re-syncing of user groups. [22:03:05] So any project admin can do that in other projects. [22:03:11] andrewbogott, nah, i think this is good enough for now, thanks [22:03:21] great [22:03:35] thanks for your help! [22:03:45] 06Labs, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Replicate new pap_parent_id column in page_assessments_projects table on Labs - https://phabricator.wikimedia.org/T158097#3026399 (10chasemp) @kaldari I ran > maintain-views --all-databases --table page_assessments_projects --replace-all on... [22:03:59] 06Labs: Restore yurik's access to wmflabs - https://phabricator.wikimedia.org/T158028#3027408 (10Andrew) 05Open>03Resolved a:03Andrew I resolved this by just removing yurik from tools and bastion and re-adding him. The same should work for other projects if need be. [22:06:47] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Output data for new XTools: Articleinfo - https://phabricator.wikimedia.org/T157706#3027425 (10kaldari) [22:23:36] andrewbogott: the "Request a project" form on Wikitech isn't used anymore right? I can't find any links to it, seems we link to T76375 now? [22:23:36] T76375: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375 [22:24:40] RainbowSprinkles: Yeah, for a new project make a subtask of https://phabricator.wikimedia.org/T76375 [22:24:42] RainbowSprinkles: yeah, make a subtask (instructions are in the main task) [22:24:43] :) [22:24:55] Yeah, I don't want a new project [22:24:58] I want to delete the form ;-) [22:25:10] (trying to cleanup unused pageforms stuff) [22:25:21] So we have a better view as to what still uses it [22:26:52] I can't find any incoming links to it on wiki [22:31:19] fine with me [22:31:29] agreed [22:31:40] in fact, thank you :) [22:33:02] {{done}} [22:33:07] Down to just 4 forms :) [22:39:08] 06Labs, 10Horizon, 07Developer-notice: Horizon Mitaka 'remember me' checkbox immune to keyboard focus - https://phabricator.wikimedia.org/T158103#3027565 (10Andrew) [22:48:05] 06Labs, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Replicate new pap_parent_id column in page_assessments_projects table on Labs - https://phabricator.wikimedia.org/T158097#3027595 (10kaldari) 05Open>03Resolved a:03kaldari Looks like that did the trick! Thanks! [22:50:37] 10Tool-Labs-tools-Other, 10Tools-Kubernetes, 13Patch-For-Review: tools.spiarticleanalyzer: requesting installation of icu on bastion and kubernetes - https://phabricator.wikimedia.org/T157744#3027599 (10JustBerry) @bd808 Thanks for merging. Do(es) particular container(s) need to be rebuilt now? [23:25:41] 06Labs: Check exim4 versions on labs - https://phabricator.wikimedia.org/T158134#3027675 (10Andrew) [23:26:55] 06Labs: Check exim4 versions on labs - https://phabricator.wikimedia.org/T158134#3027692 (10Andrew) Hm, the exim config hasn't been touched since October, so something else may be happening. This still needs an audit though. [23:48:54] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2946796 (10demon) >>! In T143349#3026704, @MaxSem wrote: > I'm strongly in favor of #1. At this point, 1.23 is likely only to have security updates that need to be manually tested and sk...