[04:07:53] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Kevin py was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1755836 edit summary: [04:13:12] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ferdi2005 was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1755838 edit summary: [04:16:55] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Samuele2002 was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1755840 edit summary: [04:19:29] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/StanProg was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1755842 edit summary: [04:24:25] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Zoozaz1 was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1755844 edit summary: [04:43:35] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Saanina was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1755846 edit summary: [04:47:58] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Kobitate94 was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1755848 edit summary: [05:00:44] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Bgoncalves was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1755850 edit summary: [05:17:13] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/NeoAct was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1755852 edit summary: [05:20:28] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Yeryry was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1755854 edit summary: [06:49:58] PROBLEM - Puppet run on tools-exec-1418 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:50:26] whoop whoop [06:50:30] not that I know what any of that is [07:24:58] RECOVERY - Puppet run on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:28] 06Labs, 10Tool-Labs-tools-stewardbots, 06Discovery, 10Wikimedia-Portals, 10wikitech.wikimedia.org: Access changes for MarcoAurelio - https://phabricator.wikimedia.org/T162541#3167510 (10Deskana) [07:35:49] 06Labs, 10Tool-Labs-tools-stewardbots, 06Discovery, 10Wikimedia-Portals, 10wikitech.wikimedia.org: Access changes for MarcoAurelio - https://phabricator.wikimedia.org/T162541#3166612 (10Deskana) @MarcoAurelio Sorry to see you go. Best wishes for the future. [10:00:01] PROBLEM - Puppet run on tools-exec-1415 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:25:51] 06Labs, 06Operations: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3167816 (10akosiaris) >>! In T162462#3164242, @faidon wrote: > My suggestion, which needs a little more time to be fully tested is: > - Take the latest 3.8 jessie-backport (from snaps... [10:34:59] RECOVERY - Puppet run on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [11:03:40] 06Labs, 06Operations: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3167853 (10akosiaris) I 'll do the first task from above (get the 3.8 bpo and put it in jessie-wikimedia/main) and possibly do the puppet upgrade on the jessie hosts. [11:56:02] PROBLEM - Puppet run on tools-exec-1415 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:21:51] 06Labs, 06Operations: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3168039 (10akosiaris) The package has been uploaded on jessie-wikimedia/backports as of a while ago and some basic checks seem to be fine. For what is worth puppet-master is a new p... [12:36:01] RECOVERY - Puppet run on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [12:56:04] 06Labs, 10Tool-Labs, 06WMF-Legal: Install unrar on Tool Labs - https://phabricator.wikimedia.org/T151794#2828377 (10zhuyifei1999) The source code is available at http://www.rarlab.com/rar_add.htm and I got it compiled on tool labs. Due to the increasing complexity of abuse in T129845 I would very much like t... [12:59:09] 10Tool-Labs-tools-Other: A tool to discover groups of orphan pages - https://phabricator.wikimedia.org/T131702#3168181 (10Aklapper) [13:09:54] andrewbogott: regarding t162529 couldnt that api you mentioned just be a simple ping and if the reply to the ping takes more than x ms than it causes an alert? [14:09:39] 06Labs, 10MediaWiki-extensions-Linter, 07Wikimedia-log-errors: Table 'labtestwiki.linter' doesn't exist (208.80.153.14) - https://phabricator.wikimedia.org/T162605#3168367 (10jcrespo) [14:09:47] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3168379 (10hashar) 05Resolved>03Open Only noticed it now, but that happened again on 04/09 around 8:50, 21:00 and on 04/10 at 2:00. Each time two... [14:10:13] andrewbogott: chasemp: good morning! Seems a few more instances are stuck in deletion state :( [14:28:28] 06Labs, 10MediaWiki-extensions-Linter, 07Wikimedia-log-errors: Table 'labtestwiki.linter' doesn't exist (208.80.153.14) - https://phabricator.wikimedia.org/T162605#3168367 (10Legoktm) ``` legoktm@terbium:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=labtestwiki linter [92e48de0... [14:38:20] 06Labs, 10MediaWiki-extensions-Linter, 06Release-Engineering-Team, 07Wikimedia-log-errors: Table 'labtestwiki.linter' doesn't exist (208.80.153.14) - https://phabricator.wikimedia.org/T162605#3168469 (10jcrespo) I do not think mediawiki production should access that non-production db, ever. We should ask m... [14:42:01] 06Labs, 10MediaWiki-extensions-Linter, 06Release-Engineering-Team, 07Wikimedia-log-errors: Table 'labtestwiki.linter' doesn't exist (208.80.153.14) - https://phabricator.wikimedia.org/T162605#3168477 (10Legoktm) Disabling Linter there is also fine with me, I'm just not sure what the expectations for that w... [14:47:13] 06Labs, 10MediaWiki-extensions-Linter, 06Release-Engineering-Team, 07Wikimedia-log-errors: Table 'labtestwiki.linter' doesn't exist (208.80.153.14) - https://phabricator.wikimedia.org/T162605#3168510 (10jcrespo) > I'm just not sure what the expectations for that wiki are... I also don't know, and I don't... [14:53:47] hashar: I'll look in a few minutes. [15:05:55] (03PS1) 10BryanDavis: Make Tool model field naming match LDAP attribute naming [labs/striker] - 10https://gerrit.wikimedia.org/r/347391 [15:05:56] (03PS1) 10BryanDavis: Implement Tool Labs membership application and processing [labs/striker] - 10https://gerrit.wikimedia.org/r/347392 (https://phabricator.wikimedia.org/T162508) [15:07:52] andrewbogott: thanks! [15:08:16] andrewbogott: and sorry I gotta head out to get kids back home. On Saturday chase !log all the restarts he had to do [15:20:01] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Programming Geek was created, changed by Programming Geek link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Programming_Geek edit summary: Created page with "{{Tools Access Request |Justification=Using bots on enwiki under my account there, [[en:User:ProgrammingGeek|ProgrammingGeek]] and [[en:User:ProgrammingBot|ProgrammingBot]]. |..." [15:21:45] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Programming Geek was modified, changed by Programming Geek link https://wikitech.wikimedia.org/w/index.php?diff=1755962 edit summary: [15:22:01] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Programming Geek was modified, changed by Programming Geek link https://wikitech.wikimedia.org/w/index.php?diff=1755963 edit summary: [15:32:02] 06Labs, 10Labs-Infrastructure, 10WikiApiary: Requesting more disk space - https://phabricator.wikimedia.org/T162534#3168655 (10MarkAHershberger) >>! In T162534#3166414, @DeepBlue wrote: > Forgot to mention this but the server is the wikiapiary db1 server currently using ~30gb of memory 30GB of disk space, n... [15:36:09] 06Labs, 06Operations: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3168682 (10chasemp) [15:36:33] 06Labs, 06Operations, 10Ops-Access-Requests: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3161730 (10chasemp) [15:39:45] 06Labs, 06Operations, 10Ops-Access-Requests: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3168687 (10chasemp) Adding the group with initial roles: https://gerrit.wikimedia.org/r/#/c/346838/ Adding bd808 if that all makes sense... [15:48:04] (03CR) 10BryanDavis: Make Tool model field naming match LDAP attribute naming (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/347391 (owner: 10BryanDavis) [15:55:02] 06Labs, 10Horizon: keystonehooks: Figure out about member role removal - https://phabricator.wikimedia.org/T162615#3168699 (10Andrew) [16:27:40] 06Labs, 06Operations: Initial OpenStack Neutron PoC deployment in Labtest - https://phabricator.wikimedia.org/T153099#3168775 (10chasemp) We have neutron deployed initially but not functioning as a control plane for anything yet. The first part of this has been working through the keystone/ldap/neutron integr... [16:35:12] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: s51362 has been rate limited to 2 concurrent connections for creating hundreds of 1400-second queries to labsdb1001 and labsdb1003 every 10 seconds - https://phabricator.wikimedia.org/T162519#3168791 (10madhuvishy) Noting here that s51362 is the mysql us... [16:42:16] 06Labs, 06Operations, 10Ops-Access-Requests: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3161730 (10RobH) >>! In T162404#3168687, @chasemp wrote: > Adding the group with initial roles: > > https://gerrit.wikimedia.org/r/#/c/34... [17:01:05] 06Labs, 06Operations, 10Ops-Access-Requests: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3161730 (10faidon) Looks good to me and there were no objections in the ops meeting either. Bikeshedding a little bit: the "cs-roots" nam... [17:01:26] 06Labs, 06Operations, 10Ops-Access-Requests: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3161730 (10Dzahn) I would vote for **cloud-roots**. "cloud" seems the most obvious term for the team and of 17 root groups, 16 end in -ro... [17:04:26] 06Labs, 06Operations, 10Ops-Access-Requests: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3168881 (10chasemp) I'm open to what seems best :) `wmcs-roots` seems most popular, but `cloud-roots` is really clear. I'm punting to @b... [17:05:05] 06Labs, 06Operations, 10Ops-Access-Requests: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3161730 (10fgiunchedi) from the "naming is hard" bandwagon: I'd suggest an easily grep-able and unambiguous name like `wmcs` to be used ev... [17:08:12] 06Labs, 06Operations, 10Ops-Access-Requests: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3168886 (10chasemp) Good point @fgiunchedi -- i n purely searchable terms using `cs` is probably too diminutive and `wmcs` will be more se... [17:09:10] 06Labs, 06Operations, 10Ops-Access-Requests: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3168888 (10bd808) Let's do `wmcs-roots`. WMCS is our official short form written name for Wikimedia Cloud Services. Cookie licking `cloud`... [17:18:13] 06Labs, 10Tool-Labs-tools-stewardbots, 06Discovery, 10Wikimedia-Portals, 10wikitech.wikimedia.org: Access changes for MarcoAurelio - https://phabricator.wikimedia.org/T162541#3166612 (10bd808) I can handle the wikitech changes, but I will give a 7 day cool down period before doing so. No pressure on @Mar... [17:25:00] 06Labs, 10Tool-Labs, 06WMF-Legal: Install unrar on Tool Labs - https://phabricator.wikimedia.org/T151794#2828377 (10valhallasw) As far as I can see, the license of unrar is non-libre, so it would (unfortunately) not be allowed on Labs. However, for the purpose of detecting RAR files, uncompressing is maybe... [18:19:09] Is there any way to scale down a Labs VM? [18:19:48] One of my VMs is much bigger than it needs to be and I have found myself up against a limit [18:21:02] harej: there is not any easy/reasonable way to resize a VM up or down [18:21:05] harej: other than rebuilding there isn't [18:21:43] harej: but a quota extension is probably doable (although I'm not sure about the current policy there) [18:22:27] yeah, I was going to recommend a ticket to get a little more temporary quota so you can replace the larger vm with a smaller one [18:23:29] harej: check out T140904 for the quota change procedure [18:23:30] T140904: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904 [18:24:01] 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#3169180 (10bd808) [18:32:21] bd808: academic question, but what would happen if I actually did need more quota, to scale up something? [18:46:26] harej: you ask and explain why and 9 times out of 10 we grant it to you [18:46:57] the 1 time out of ten we talk it over and find a better way for you to do what you need to do [18:47:27] the quotas are mostly so that we can keep track of how fast we are growing and reclaim things that are not being used anymore [18:48:10] I don't know the numbers right now off the top of my head but I don't think we are stretched for ram or cpu across the project today [18:57:00] 06Labs, 06Operations, 10Ops-Access-Requests: create a 'root' group with bdavis strictly for labs/cloud services infrastructure - https://phabricator.wikimedia.org/T162404#3169334 (10chasemp) 05Open>03Resolved a:03chasemp >>! In T162404#3168888, @bd808 wrote: > Let's do `wmcs-roots`. WMCS is our officia... [19:11:38] bd808: so, considering I already have this overpowered VM, is there any reason why I couldn't just use it to do multiple things? Would be easier than tearing down and starting up again, especially since this Labs project is a group effort and my colleagues understand the stack better than I do. [19:13:03] harej: yeah, you can put several things on one VM if that works out ok. I guess I'd need more context to provide better guidance. [19:15:04] librarybase-reston-01 runs a MediaWiki with Wikibase. I have it maxed out, mostly in anticipation of it storing a lot of stuff. In terms of bandwidth, memory, and CPU, I don't think it's doing much. I'd like to set up another web server to run an API endpoint I plan on setting up that actually would get a lot of traffic (albeit in spurts – a handful of [19:15:04] people making a ton of requests, rather than many people making a few requests). [19:16:22] Also, does Wikimedia Labs support attaching volumes to instances, or am I confusing Labs with DigitalOcean again? [19:28:35] 06Labs: Create an inchinga instance for monitoring labs instances - https://phabricator.wikimedia.org/T162629#3169464 (10Paladox) [19:32:53] 06Labs: Create an inchinga instance for monitoring labs instances - https://phabricator.wikimedia.org/T162629#3169507 (10Paladox) [19:42:45] harej: if it's a largely independent endpoint, I would generally suggest another VM [19:43:05] It's totally independent. [19:43:11] And I agree, but I want to hear your reasoning. [19:44:48] Mostly just to clearly separate them. Then it's easier to determine dependencies, easier to find the cause of bottlenecks, etc. [19:45:10] Easier to rebuild/partially duplicate [19:45:44] Unfortunately, no support for attaching volumes. Then up/downgrading would have been easy :-) [19:52:46] someday we will have attachable storage. someday. [20:05:59] and resizable VMs?? :D [20:09:44] in many ways the idea of resizable VMs is flawed, no VM should be that special if things are done well [20:48:25] harej: if data storage was separated from the VM itself then most of the desire for resizable VMs would go away I think. The pain in building a new VM begger/smaller than the old is usually migrating the data it holds to the new vm [20:48:57] In an ideal world all the software/config is puppet managed already [20:49:39] having a nicer way for projects to manage custom puppet config is high on my list of wants though and related to that cattle not pets VM mentality [20:54:06] bd808: puppet does do a good job already at keeping things somewhat sync'd [20:55:35] the difficulty comes when you want to use Puppet to manage something that there is no existing WMF Puppet configuration for. All of us mortals (not WMF roots) have great pains in upstreaming new Puppet modules to the ops/puppet.git repo [20:55:58] this has been a struggle for many things I've worked on in the last 3.5 years [20:56:23] bd808: its a struggle just to get a simple patch through [20:57:29] that depends on the project, what the author thinks is simple, and the reviewers, but yeah CR is hard all over [21:03:46] chasemp: nodepool is screwed up again zuul has a job thats been queued over 3 hrs now ive already notified hashar [21:04:01] ^ andrewbogott I'm in a meeting can you take a look? [21:04:11] sure [21:08:31] Zppix: things look ok to me, might be worth doing a recheck (or retest? I can never remember which it is) [21:08:49] andrewbogott: its currently in postmerge it wouldnt change anything [21:09:12] but ill try [21:16:07] chasemp: andrewbogott please disregard that comment from Zppix [21:16:16] my bad :/ [21:16:45] ok :) [21:25:10] chasemp: andrewbogott: unrelated issue [21:25:45] some glitch happened on jenkins side. Nothing to worry about :] [21:26:11] looks like everything is fine ! kudos ! [21:26:41] sleep & [21:38:38] 06Labs, 10wikitech.wikimedia.org: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3169835 (10bd808) [21:38:40] 06Labs, 10Tool-Labs, 10wikitech.wikimedia.org, 05MW-1.29-release (WMF-deploy-2017-04-04_(1.29.0-wmf.19)), and 2 others: Replace SMW data on project instances with a tool using the OpenStack APIs - https://phabricator.wikimedia.org/T161662#3169834 (10bd808) 05Open>03Resolved [22:58:31] CUSTOM$ - ping4$ on $ is OK$: PING OK - Packet loss = 0%, RTA = 2.18 ms$ $ $ [22:58:36] mutante ^^ :) [23:01:26] CUSTOM$ - ping6$ on phabricator.phabricator.eqiad.wmflabs is OK: PING OK - Packet loss = 0%, RTA = 0.07 ms [23:01:37] twentyafterfour mutante ^^ [23:01:41] it shows the hostname now [23:03:37] CUSTOM$ - ssh$ on phabricator.phabricator.eqiad.wmflabs is CRITICAL: connect to address 10.68.20.111 and port 22: No route to host [23:03:54] ACKNOWLEDGEMENT$ - ssh$ on phabricator.phabricator.eqiad.wmflabs is CRITICAL: connect to address 10.68.20.111 and port 22: No route to host [23:07:47] ACKNOWLEDGEMENT$ - http$ on phabricator.phabricator.eqiad.wmflabs is CRITICAL: connect to address 10.68.20.111 and port 80: Connection refused [23:08:16] CUSTOM$ - http$ on phabricator.phabricator.eqiad.wmflabs is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 478 bytes in 0.003 second response time [23:08:45] shinken-wm: deployment-phab01 [23:08:46] generic-host [23:08:53] CRITICAL for Puppet run since 2w 6d [23:08:55] fwiw [23:10:42] paladox: i'd recommend figuring out how to have phab.phab and deployment-phab01 both work with the same roles [23:10:57] Ok. [23:11:56] paladox: btw, see on http://shinken.wmflabs.org/user/login it says the login is guest/guest on the page itself [23:12:05] just like for your icinga setup [23:12:07] oh [23:13:15] and one day the question will be shinken or icinga2 i suppose [23:13:42] vs some instances monitored by one system and some by the other [23:16:17] paladox: if you dont mind im stalking shinken's interface on wmflabs lol [23:16:21] 06Labs, 10DBA, 07Regression: Tool Labs: Add skin, language, and variant to user_properties_anon - https://phabricator.wikimedia.org/T152043#3170033 (10Krinkle) [23:16:25] lol [23:16:28] Zppix which one [23:16:40] the one that was linked above [23:16:46] oh [23:16:47] ok [23:16:56] CUSTOM$ - http$ on phabricator.phabricator.eqiad.wmflabs is CRITICAL: connect to address 10.68.20.111 and port 80: Connection refused [23:17:13] CUSTOM$ - http$ on phabricator.phabricator.eqiad.wmflabs is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 478 bytes in 0.002 second response time [23:20:50] wheres the icinga one your hosting [23:21:20] maybe better to use the bot test channel for a while until it's stable [23:21:39] #wikimedia-bots-testing i think is what it is [23:21:57] yea [23:22:16] its sad i cant even remember the channels i register lol [23:23:27] maybe chanserv can tell you :p [23:32:26] yep i will switch [23:59:59] hi, i'm trying to create a new tool called 'robokobot' using Special:NovaServiceGroup but it fails: "Failed to create service group." any idea?