[00:29:41] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/AlvaroMolina was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=457847 edit summary: [00:33:32] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/LegoFan4000 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=457850 edit summary: [00:52:11] 10MediaWiki-extensions-OpenStackManager: Special:NovaInstance should restrict project filter to projects where the current user is an administrator - https://phabricator.wikimedia.org/T106820#2233975 (10yuvipanda) 'We' would be the labs team, I guess. And yes, I don't think it makes sense for anyone else to use... [00:53:36] 10MediaWiki-extensions-OpenStackManager, 07Technical-Debt: Fatal error: Call to undefined method GoogleLoginAuth::connect() in nova/OpenStackNovaLdapConnection.php on line 21 - https://phabricator.wikimedia.org/T70718#2233976 (10yuvipanda) It's a practically abandoned extension, with a *ton* more tech debt tha... [01:05:46] 10MediaWiki-extensions-OpenStackManager, 07Technical-Debt: Fatal error: Call to undefined method GoogleLoginAuth::connect() in nova/OpenStackNovaLdapConnection.php on line 21 - https://phabricator.wikimedia.org/T70718#714295 (10Legoktm) The OSM extension depends upon the LdapAuth extension being used as the Au... [01:17:18] 06Labs, 10MediaWiki-extensions-OpenStackManager: Lists of users in Labs project pages should be sorted by wiki user name - https://phabricator.wikimedia.org/T128002#2233983 (10yuvipanda) 05Open>03Invalid We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - sw... [01:17:40] 10MediaWiki-extensions-OpenStackManager: Description should be a required field on Special:NovaSecurityGroup?action=create in OpenStackManager - https://phabricator.wikimedia.org/T116885#2233986 (10yuvipanda) 05Open>03Resolved a:03yuvipanda We (the labs team) plan to add no more new features / do any non-... [01:18:03] 10MediaWiki-extensions-OpenStackManager: Confirmation page after creation of service group should link to useful resources - https://phabricator.wikimedia.org/T59856#2233990 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - sw... [01:18:17] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Provide wikitech API for list of OpenStack regions - https://phabricator.wikimedia.org/T115237#2233993 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackMana... [01:18:41] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: https://wikitech.wikimedia.org/w/api.php?action=novaprojects&subaction=getall times out - https://phabricator.wikimedia.org/T115034#2233996 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do a... [01:19:24] 10MediaWiki-extensions-OpenStackManager: Weird parsing of openstackmanager-configureproject-serviceuserinfo message on Special:NovaProject&action=configureproject - https://phabricator.wikimedia.org/T89467#2233999 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any no... [01:20:30] 10MediaWiki-extensions-OpenStackManager: OpenStackManager: Instance actions via Special:NovaInstance should be protected by a nonce - https://phabricator.wikimedia.org/T58275#2234002 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackMan... [01:20:45] 10MediaWiki-extensions-OpenStackManager, 13Patch-For-Review: Improve OpenStackManager's Echo-based notification emails - https://phabricator.wikimedia.org/T58202#2234005 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - swit... [01:21:02] 10MediaWiki-extensions-OpenStackManager: Image names listed with image ID label - https://phabricator.wikimedia.org/T53648#2234008 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :) Other user... [01:21:15] 10MediaWiki-extensions-OpenStackManager: Ajax reboot link should ask for confirmation - https://phabricator.wikimedia.org/T52923#2234011 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :) Othe... [01:21:34] 10MediaWiki-extensions-OpenStackManager: Message echo-category-title-osm-instance-build-completed missing in Echo UI - https://phabricator.wikimedia.org/T49306#2234014 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch t... [01:21:44] 10MediaWiki-extensions-OpenStackManager: Set project filter with the jquery cookie module - https://phabricator.wikimedia.org/T48812#2234017 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :)... [01:22:01] 10MediaWiki-extensions-OpenStackManager: Create a combined javascript interface - https://phabricator.wikimedia.org/T46177#2234020 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :) Other user... [01:22:10] 10MediaWiki-extensions-OpenStackManager: Combine role and project membership into a single interface - https://phabricator.wikimedia.org/T46175#2234023 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is un... [01:22:29] 10MediaWiki-extensions-OpenStackManager: Combine queues with actions - https://phabricator.wikimedia.org/T46171#2234026 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :) Other users of OpenSt... [01:22:51] 10MediaWiki-extensions-OpenStackManager: Make current quotas and resources viewable - https://phabricator.wikimedia.org/T46170#2234029 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :) Other... [01:23:10] 10MediaWiki-extensions-OpenStackManager: Trim color codes in console output - https://phabricator.wikimedia.org/T42335#2234032 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :) Other users of... [01:23:19] 10MediaWiki-extensions-OpenStackManager: Upon initial registration keystone credentials aren't fetched - https://phabricator.wikimedia.org/T45372#2234035 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is... [01:23:38] 10MediaWiki-extensions-OpenStackManager: Domain creation doesn't properly invalidate domain cache in memcache - https://phabricator.wikimedia.org/T41777#2234038 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Hori... [01:23:48] 10MediaWiki-extensions-OpenStackManager: AJAXy autocomplete for adding members to a project - https://phabricator.wikimedia.org/T39486#2234041 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :... [01:24:03] 10MediaWiki-extensions-OpenStackManager: Associating and disassociating IPs doesn't update the instance's page - https://phabricator.wikimedia.org/T37998#2234044 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Hor... [01:24:05] YuviPanda: um [01:24:15] YuviPanda: can we decline all the tasks once everything has been switched over? [01:24:19] 10MediaWiki-extensions-OpenStackManager: When renaming users OpenStackManager should clear memcache key for user - https://phabricator.wikimedia.org/T36868#2234047 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #H... [01:24:39] 10MediaWiki-extensions-OpenStackManager: OpenStackManager special pages should link to in-wiki documentation - https://phabricator.wikimedia.org/T36500#2234050 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horiz... [01:24:45] 10MediaWiki-extensions-OpenStackManager: Special:NovaProject needs a table of contents - https://phabricator.wikimedia.org/T36196#2234054 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :) Oth... [01:25:01] 10MediaWiki-extensions-OpenStackManager: Renaming puppet classes, variables and groups should be possible - https://phabricator.wikimedia.org/T35315#2234057 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon... [01:25:21] 10MediaWiki-extensions-OpenStackManager: Support parameterized classes in puppet configuration - https://phabricator.wikimedia.org/T34693#2234060 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under wa... [01:25:34] 10MediaWiki-extensions-OpenStackManager: OpenStackManager should sign puppet certificates on instance creation - https://phabricator.wikimedia.org/T34555#2234063 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Hor... [01:25:57] 10MediaWiki-extensions-OpenStackManager: Delete project should delete all instances, security groups and volumes assocated with it. - https://phabricator.wikimedia.org/T34369#2234066 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackMan... [01:26:17] 10MediaWiki-extensions-OpenStackManager: Puppet classes and variables should have descriptions - https://phabricator.wikimedia.org/T34223#2234069 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under wa... [01:26:27] 10MediaWiki-extensions-OpenStackManager: All list views need pagination support - https://phabricator.wikimedia.org/T34158#2234072 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :) Other user... [01:26:45] 10MediaWiki-extensions-OpenStackManager: Deletion of domain names where dc doesn't match associated domain doesn't work - https://phabricator.wikimedia.org/T33872#2234075 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switc... [01:27:02] 10MediaWiki-extensions-OpenStackManager: Add more specific error messages for actions - https://phabricator.wikimedia.org/T29136#2234078 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :) Othe... [01:27:19] 10MediaWiki-extensions-OpenStackManager: Add a nova user special page interface for managing nova-specific user attributes - https://phabricator.wikimedia.org/T28586#2234081 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - sw... [01:27:29] 10MediaWiki-extensions-OpenStackManager: OpenStackManager extension needs better error checking - https://phabricator.wikimedia.org/T28582#2234084 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under w... [01:28:48] 10MediaWiki-extensions-OpenStackManager: Instance creation doesn't actually fail when hosts can't be added to LDAP - https://phabricator.wikimedia.org/T41776#2234087 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to... [01:28:58] 10MediaWiki-extensions-OpenStackManager: Special:NovaKey import should have file input option - https://phabricator.wikimedia.org/T34226#2234090 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way... [01:29:14] 10MediaWiki-extensions-OpenStackManager: Don't create or delete instance pages - https://phabricator.wikimedia.org/T45604#2234093 (10yuvipanda) 05Open>03declined We (the labs team) plan to add no more new features / do any non-critical work on OpenStackManager - switch to #Horizon is under way :) Other users... [01:49:55] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Write a k8s admission controller to enforce that all containers running come from our private repository - https://phabricator.wikimedia.org/T133515#2234095 (10yuvipanda) [01:57:43] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2234109 (10yuvipanda) >>! In T133252#2233064, @tom29739 wrote: > It would be nice to do everything in Phabric... [06:01:58] 06Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#2234205 (10Ato_01) 05Resolved>03Open p:05Triage>03Normal [06:03:26] 06Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#1989468 (10Ato_01) [06:45:25] PROBLEM - Puppet run on tools-webgrid-generic-1405 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:47:40] 06Labs, 10Labs-Sprint-100, 10Tool-Labs: Deploy new unified webservice code - https://phabricator.wikimedia.org/T98440#2234235 (10yuvipanda) [06:47:42] 06Labs, 10Tool-Labs, 13Patch-For-Review: Convert tomcat-starter to python - https://phabricator.wikimedia.org/T98442#2234232 (10yuvipanda) 05Open>03Resolved a:03yuvipanda ok, so webservice-new tomcat start works now with ^ 'hack'. There are only 6 of these running, so we can get rid of it after we move... [06:48:53] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:59:02] PROBLEM - Puppet run on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:00:02] PROBLEM - Puppet run on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:19:51] hmm [07:24:14] transient, just a race between apt-get update and puppet agent -tv [07:25:30] RECOVERY - Puppet run on tools-webgrid-generic-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [07:33:57] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:57] RECOVERY - Puppet run on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:10] 06Labs, 10Wikimedia-Apache-configuration, 10wikitech.wikimedia.org: Have a short link to the signup page - https://phabricator.wikimedia.org/T48175#2234264 (10Qgil) It's better, indeed. Maybe this is not as big of an issue as I thought it was back in the day. Feel free to decline. [07:40:09] RECOVERY - Puppet run on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [08:01:06] 06Labs, 10Tool-Labs, 10DBA: Provide replication lag as a database function - https://phabricator.wikimedia.org/T50628#2234278 (10jcrespo) @scfc Why declined? the functionality has been implemented on the database. Do you really need a "MySQL function" instead of a table? [08:07:25] 06Labs, 10DBA: Data missing from June 11/12 on s3.labsdb - https://phabricator.wikimedia.org/T115517#2234279 (10jcrespo) > So perhaps the issue is wider in scope than just s3, but also covering s7, and potentially also s5/s6 (which are also typically served by labsdb1003)? No, this only affected s3 shard- it... [09:13:34] 10MediaWiki-extensions-OpenStackManager: Description should be a required field on Special:NovaSecurityGroup?action=create in OpenStackManager - https://phabricator.wikimedia.org/T116885#2234356 (10scfc) 05Resolved>03declined [09:14:07] 06Labs, 10MediaWiki-extensions-OpenStackManager: Lists of users in Labs project pages should be sorted by wiki user name - https://phabricator.wikimedia.org/T128002#2234357 (10scfc) 05Invalid>03declined [09:47:30] 06Labs, 10Tool-Labs, 10DBA: Provide replication lag as a database function - https://phabricator.wikimedia.org/T50628#2234474 (10scfc) @jcrespo: Just trying to be accurate. If someone looks at this task "Provide replication lag as a database function" with the status "Resolved", they expect such a database... [09:56:56] 06Labs, 10Tool-Labs, 10DBA: Provide replication lag as a database function - https://phabricator.wikimedia.org/T50628#2234488 (10jcrespo) The important thing that worries me, @scfc, are you satisfied with that implementation? [11:08:22] 06Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs: shared Pywikibot code not available - https://phabricator.wikimedia.org/T125505#2234633 (10Ato_01) 05Open>03Resolved p:05Normal>03Triage [12:37:51] (03PS2) 10Jean-Frédéric: Add unit tests to test_update_database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/283108 [12:37:53] (03PS1) 10Jean-Frédéric: Rename processText to processPage and remove processTextfile [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/285174 [13:33:00] 06Labs, 06Discovery, 06Operations, 10hardware-requests, 03Discovery-Search-Sprint: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2234930 (10mark) [13:43:34] 06Labs, 10Wikimedia-Apache-configuration, 10wikitech.wikimedia.org: Have a short link to the signup page - https://phabricator.wikimedia.org/T48175#2234978 (10Krenair) 05Open>03declined Okay. Let's not start creating custom apache configs for specific mediawiki pages on a per-site basis, especially just... [13:48:46] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2235001 (10chasemp) A small bit of pile on due to the subject matter :) We have a very small amount of local... [14:46:47] 10Quarry: Various assortment of improvements - https://phabricator.wikimedia.org/T133545#2235207 (10Ankit-Maity) [15:10:25] 06Labs, 10Tool-Labs, 10DBA: Provide replication lag as a database function - https://phabricator.wikimedia.org/T50628#2235296 (10scfc) Yes, very much, it provides an easy, standardized and documented way to query the replication lag. ¡Muchas gracias! [15:11:13] 06Labs, 06Discovery, 06Operations, 10hardware-requests, 03Discovery-Search-Sprint: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2158516 (10chasemp) I think we were aware of it in a cursory manner. What is the Labs general outcome for this? Is this a service provided... [15:16:11] 06Labs, 06Discovery, 06Operations, 10hardware-requests, 03Discovery-Search-Sprint: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2235335 (10Gehel) a/b/c) The goal is to have a cluster to test improvements in search configuration, allowing to run tests against different op... [15:19:55] 06Labs, 06Discovery, 06Operations, 10hardware-requests, 03Discovery-Search-Sprint: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2235357 (10chasemp) If someone wanted to launch a bunch of queries against this cluster from Tools to analyze our search results would this be... [15:20:05] 06Labs, 06Discovery, 06Operations, 10hardware-requests, 03Discovery-Search-Sprint: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2235360 (10EBernhardson) The project will need to be accessible (read/write) from the search project in labs. These will have indices (not upda... [15:22:35] 06Labs, 06Discovery, 06Operations, 10hardware-requests, 03Discovery-Search-Sprint: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2235367 (10EBernhardson) @chasmp yes i think that would be reasonable. Since we are getting spinning disks they will only see good performance... [15:30:12] PROBLEM - Puppet run on tools-exec-1402 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:31:32] PROBLEM - Puppet run on tools-exec-1213 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:31:40] PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:36:07] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:36:16] andrewbogott: hi, http://wm-bot.wmflabs.org/browser/ is broken - what is going on? :p [15:36:28] I know you were doing something nasty with internal proxy :) [15:36:32] you sent me that e-mail [15:36:37] but I wonder if it's related [15:36:47] omg [15:36:51] it's no longer broken XD [15:36:58] some temporary proxy issue [15:38:27] *MAGIC* :D [15:38:38] same problem, same magic fix ;) [15:38:43] PROBLEM - Puppet run on tools-exec-1214 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [15:39:22] petan: No idea what that's about, but let me know if it happens again and you can tell what happened [15:39:29] and back down ... [15:39:49] ebernhardson: I was thinking it's just me :D [15:39:54] yep, it's back down [15:40:04] back up [15:40:37] it's consistently working for me. petan btw it's loading things from googlefonts and code.jquery.org and other places, which is a privacy policy violation and you should fix that :) [15:41:26] PROBLEM - Puppet run on tools-mail-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:41:27] YuviPanda: did something change in policies saying that external websites are no longer allowed to provide resources? [15:41:48] petan: that's always been the case from the beginning afaik. [15:42:01] can you link me to page which say that [15:42:18] seems DNS resolution is borked on labs [15:42:18] host deployment-puppetmaster.deployment-prep.eqiad.wmflabs [15:42:20] Host deployment-puppetmaster.deployment-prep.eqiad.wmflabs not found: 2(SERVFAIL) [15:42:20] :( [15:42:28] should I fill a task or is that looked at? [15:42:47] hashar: please file a task. [15:43:54] 06Labs, 10Labs-Infrastructure: labs DNS resolution fails - https://phabricator.wikimedia.org/T133552#2235517 (10hashar) [15:43:58] YuviPanda: done https://phabricator.wikimedia.org/T133552 [15:44:15] petan: it's at https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use#What_information_should_I_provide_to_users.3F [15:44:30] specifically, 'If you collect any Private Information from End Users, you must display this disclaimer to the End Users before you collect the Private Information:' [15:44:53] petan: in your case, you're sending user IP info to google / jQuery, and you shouldn't do that without an interstitial if you really want to do that. [15:45:17] YuviPanda: I really would like to see 1 person on this planet that uses internet and doesn't send their IP to google [15:45:32] petan: well, that's irrelevant to the labs terms of use. [15:46:05] yes and it's also a bit irrelevant to my original issue I was sorting out with andrewbogott :P [15:46:18] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:46:40] petan: indeed. Please do rectify the violation as soon as you can, though. thanks. [15:47:08] I really wish we focused on REAL issues and not some cosmetic stuff, which this indeed is. nobody cares about sending their IP's to google. [15:47:46] petan: I think I could find people to disagree with that statement [15:48:10] but yes this is orthogonal to the flapping proxy [15:48:16] bd808: yes, I would like to see a person who has internet and never visited google in their life :) [15:48:52] petan: please realize that these policies are long-considered and arrived at during ongoing discussions with lawyers. It's not like we Ops are just making shit up at random. [15:49:02] WMF Legal is working on an updated Labs TOU and privacy policy. Once that is done you will see more messaging about complicance [15:49:05] If you want the policies revised, you're taking it up with the wrong people [15:49:31] 06Labs, 10Labs-Infrastructure: labs DNS resolution fails - https://phabricator.wikimedia.org/T133552#2235547 (10scfc) [15:49:34] 06Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2235548 (10scfc) [15:49:36] no I didn't come here to sort out policies, I wanted to know what's wrong with proxy [15:49:42] that's fair :) [15:49:45] we're working on it [15:49:51] (although I'm at a conference) [15:49:52] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:50:22] 06Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#1962377 (10scfc) Started today ~ 15:28Z. [15:52:17] petan: for what it's worth, we have a couple of tasks in progress to provide better stability for DNS. Time will tell if we're targeting the right issues though. [15:53:51] we have redundant recursors but some (all?) of our instances are terrible at failing over when there's a slowdown. So going to move them behind lvs [15:54:54] 06Labs, 10Horizon: Switch dynamicproxy to point back to IP rather than domain names - https://phabricator.wikimedia.org/T133554#2235590 (10yuvipanda) [15:58:47] PROBLEM - Puppet run on tools-proxy-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:58:49] PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:01:03] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 323 bytes in 0.007 second response time [16:02:46] rip toollabs [16:03:10] That page was a load of rubbish anyway. [16:03:31] all pages are like that for me [16:03:51] (we're looking into the issue) [16:05:46] bd808, can you update the css file on Wikitech for that new template I put on the Main Page? [16:05:50] PROBLEM - Puppet run on tools-worker-1007 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [16:07:00] tom29739: what change needs to happen? [16:07:35] bd808, this needs to be put in: .mw-tpl-portal-name { [16:07:36] font-size:160%; [16:07:36] padding:1em; [16:07:36] text-align:center; [16:07:36] vertical-align:middle; [16:07:37] } [16:07:48] in the first section for portals. [16:09:04] andrewbogott: Proxy seems broken. all my instances throwing 502 [16:09:45] Might be something to with that problem with the Tool Labs homepage too. [16:10:09] magically working again [16:10:45] we're working on it [16:11:04] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [16:11:06] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 817571 bytes in 4.364 second response time [16:11:16] tom29739: https://wikitech.wikimedia.org/w/index.php?title=MediaWiki:Common.css&diff=459223&oldid=281177 [16:11:34] RECOVERY - Puppet run on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:47] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:14:08] bd808, the Suggested CSS section on this page: https://wikitech.wikimedia.org/wiki/Template:Portal_list_item_noimage/doc#Suggested CSS [16:14:18] https://wikitech.wikimedia.org/wiki/Template:Portal_list_item_noimage/doc [16:14:28] Has the CSS already merged in. [16:15:13] hm, best I can tell it was an accidental DOS. Luke081515, are things acting better for you now? [16:15:30] * YuviPanda got abunch of recoveries for things he monitors now [16:16:08] andrewbogott: but one that hits both resolvers? [16:16:23] RECOVERY - Puppet run on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:52] tom29739: why raise the padding to 1em? [16:18:04] it makes things wrap strangely for me [16:18:44] I just copied the stuff for .mw-tpl-portal-image and added that vertical-align attribute. [16:19:17] ah, probably stale css on the template doc then [16:19:22] bd808, I'm not too good at CSS, I don't know what it needs to look right. [16:19:33] I'll fix them up. Thanks for the help! [16:20:08] No problem. [16:21:20] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:45] Krenair: do you think you can spare some time for https://phabricator.wikimedia.org/T133554? [16:26:10] yes [16:26:14] assign to alexmonk-wmf [16:26:19] we would greatly appreciate it [16:26:40] Krenair: <3 thanks. [16:26:44] will take a look later today [16:27:10] 06Labs, 10Horizon: Switch dynamicproxy to point back to IP rather than domain names - https://phabricator.wikimedia.org/T133554#2235721 (10yuvipanda) a:03AlexMonk-WMF (Alex has graciously offered to take this on!) [16:29:59] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:46] RECOVERY - Puppet run on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:50] RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:45] RECOVERY - Puppet run on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [16:46:50] petan: ebernhardson it should be fine now, looks like another instance of https://phabricator.wikimedia.org/T124680 [16:49:34] ahh, fun. ok thanks! [17:20:54] PROBLEM - Host tools-worker-1011 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:05] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2236226 (10Andrew) Jaime, will you have time to work on this this week? I'm still hopeful that this will stamp out some of our existing DNS issues. [18:28:12] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2236265 (10jcrespo) Yes. [18:47:10] Is there are a query that list the page titles of articles, and the list is ordered by date of creation of the articles? [18:47:37] This could help to find the order of the article in a specific wikipedia. [18:48:01] 10Quarry: Quarry shows old username after user was renamed - https://phabricator.wikimedia.org/T73064#771640 (10Stefan2) >>! In T73064#2001109, @Krenair wrote: > * ... now we have two As? The username field is supposed to be unique according to tables.sql. This may also happen even if you do not implement user r... [19:01:01] Any help? [19:02:48] ASammour: for a single article, it's a trivial join with the revision table [19:02:58] for the full list... [19:07:25] ASammour: something like https://quarry.wmflabs.org/query/9278 ? [19:11:07] valhallasw`cloud: That is exactly what I need. Thank you very much!. [19:24:15] 06Labs, 06Operations: overhaul labstore setup [tracking] - https://phabricator.wikimedia.org/T126083#2236478 (10chasemp) [19:25:42] RECOVERY - Puppet run on tools-puppetmaster-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:33:53] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2236556 (10Andrew) \o/ [19:56:06] RECOVERY - Puppet run on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [20:45:38] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Decide on upgrade policy for Kubernetes - https://phabricator.wikimedia.org/T133598#2236884 (10yuvipanda) [20:46:19] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Decide on upgrade policy for Kubernetes - https://phabricator.wikimedia.org/T133598#2236897 (10chasemp) p:05Triage>03Normal [20:48:05] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Upgrade to Kubernetes 1.2.3 - https://phabricator.wikimedia.org/T133458#2236914 (10chasemp) p:05Triage>03Normal a:03chasemp [20:48:33] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Upgrade to Kubernetes 1.2.3 - https://phabricator.wikimedia.org/T133458#2232436 (10chasemp) I took this but would be glad to help someone else with what I know (which isn't much :) [21:52:59] 06Labs, 10Horizon, 13Patch-For-Review: Switch dynamicproxy to point back to IP rather than domain names - https://phabricator.wikimedia.org/T133554#2237077 (10AlexMonk-WMF) Patch above will point new backends to IPs instead of hostnames. Do you also want me to write a script to update all existing records @Y... [23:37:03] CP678: Around? If not, ping me when you are. I have an idea in regard to Peachy. [23:37:42] * CP678 lights a fire under the idea. [23:39:32] o.O [23:40:12] If you recall, you asked me to properly document Peachy correct? Is that project still under active development? If so, I'm looking into using a documentation generator to take care of that. [23:40:15] Matthew_, you better hurry before your idea catches fire. [23:40:28] I'm looking at https://www.phpdoc.org/ , [23:40:44] Matthew_, I do want to get back to it when I have more time available. [23:41:22] So I could create a labs project that basically scans for changes to the repository and runs phpdoc against the repo, and we can use the output on the web. [23:41:34] This is better than manually updating the documentaiton. [23:41:43] The only work that would need to be done is updating the docblocks. [23:41:55] How does it work? [23:42:59] Add a dependency to composer, then follow directions at https://www.phpdoc.org/docs/latest/getting-started/your-first-set-of-documentation.html#running-phpdocumentor , basically run "phpdoc" from the command line and point it at an output directory. It scans all the docblocks and generates documentation based on that. [23:43:42] Sure go ahead. [23:43:44] Then the only work is keeping the documentation blocks up to date. [23:43:49] Okay. [23:53:12] CP678: It's a start, the docblocks need to be expanded. But here you go: https://tools.wmflabs.org/peachy-docs/master/ [23:53:51] And options need to be tweaked. [23:56:03] Nice [23:56:17] That makes my life so much easier. [23:56:25] Yay! [23:57:01] I'll work on automating it with a quick webhook and I think I'll generate documentation by version as well.