[00:02:49] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Hall1467 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1533969 edit summary: [00:21:22] Reedy: It works! The C#-framework needs to be started from the home directory � for magic reasons. And I had to install (say: copy) an newer MySql.Data.dll to recognize SSL Mode=None [00:37:54] 06Labs, 10Tool-Labs: DNS resolution sometimes fails on tools-bastion-03 - https://phabricator.wikimedia.org/T143194#3035248 (10Samwilson) 05Open>03Resolved a:03Samwilson This problem seems to have now disappeared, without anything changing on the tool's end. This feed was failing before but now is built... [00:54:07] 06Labs, 10SyntaxHighlight, 10wikitech.wikimedia.org: Extension:SyntaxHighlight_GeSHi reports for pages with syntax highlighting errors are bogus - https://phabricator.wikimedia.org/T153616#3035280 (10scfc) 05Open>03Invalid I null-edited the remaining pages which left four in the category and fixed those... [00:54:30] 06Labs, 10Social-Tools, 10TopLists: Install TopLists on social-tools.wmflabs.org - https://phabricator.wikimedia.org/T158375#3035284 (10SamanthaNguyen) [01:03:42] 06Labs, 10Tool-Labs: Create developer environment using Docker images from Tool Labs Kubernetes - https://phabricator.wikimedia.org/T157733#3035320 (10scfc) p:05Triage>03Normal Downloading works: ``` [tim@passepartout ~/src/operations/puppet]$ sudo docker pull docker-registry.tools.wmflabs.org/toollabs-py... [01:19:27] 06Labs, 10SyntaxHighlight, 10wikitech.wikimedia.org: Extension:SyntaxHighlight_GeSHi reports for pages with syntax highlighting errors are bogus - https://phabricator.wikimedia.org/T153616#3035368 (10bd808) Thanks for following up on it @scfc! [01:24:04] Wurgl: Sweet [01:24:19] Did you rebuild it after you copied in a a newer mysql.data.dll? [01:42:57] 06Labs, 10Tool-Labs: deb.tools.wmflabs.org is not accessible from outside Tool Labs - https://phabricator.wikimedia.org/T158383#3035429 (10scfc) [01:43:07] 06Labs, 10Social-Tools, 10TopLists: Install TopLists on social-tools.wmflabs.org - https://phabricator.wikimedia.org/T158375#3035442 (10SamanthaNguyen) [03:16:19] (03PS1) 10Zppix: Addition of GNU license [labs/tools/quarrybot-enwiki] - 10https://gerrit.wikimedia.org/r/338308 (https://phabricator.wikimedia.org/T158388) [03:17:40] (03PS2) 10Zppix: Addition of GNU license [labs/tools/quarrybot-enwiki] - 10https://gerrit.wikimedia.org/r/338308 (https://phabricator.wikimedia.org/T158388) [03:17:48] (03CR) 10Zppix: [V: 032 C: 032] Addition of GNU license [labs/tools/quarrybot-enwiki] - 10https://gerrit.wikimedia.org/r/338308 (https://phabricator.wikimedia.org/T158388) (owner: 10Zppix) [03:25:34] PROBLEM - Puppet run on tools-worker-1027 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:27:02] 06Labs, 10Tool-Labs: replica.my.cnf does not exist for my tool labs account - https://phabricator.wikimedia.org/T158389#3035571 (10Hall1467) [03:28:22] 06Labs, 10Tool-Labs: replica.my.cnf appears to not exist for my tool labs account - https://phabricator.wikimedia.org/T158389#3035584 (10Hall1467) [04:00:35] RECOVERY - Puppet run on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0] [04:05:21] 06Labs, 10Tool-Labs: deb.tools.wmflabs.org is not accessible from outside Tool Labs - https://phabricator.wikimedia.org/T158383#3035617 (10bd808) I wonder if we should use an https proxy in front of it instead? Getting debs from an insecure http connection seems like a bad idea generally and especially outside... [06:28:20] 06Labs, 10Tool-Labs: deb.tools.wmflabs.org is not accessible from outside Tool Labs - https://phabricator.wikimedia.org/T158383#3035429 (10Pnorman) > Getting debs from an insecure http connection seems like a bad idea generally and especially outside of the tools project internal network. It should be fine. A... [07:37:51] 06Labs, 10Tool-Labs: giftbot webservice outages and/or issues - https://phabricator.wikimedia.org/T155494#3035776 (10Giftpflanze) 05Open>03declined [08:14:38] bd808: late reply but it's server side EL [08:15:54] (http://pastebin.com/WPWNwxFF for reference) [11:03:20] !log video git pulling v2c frontend to 6f329f3 and restarting webservice [11:03:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [11:07:23] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee, 10Tool-Labs-tools-Quentinv57's-tools: Consider publishing Quentinv57-tools' code on Diffussion and allow people to submit patches to them - https://phabricator.wikimedia.org/T158405#3036012 (10MarcoAurelio) [11:13:55] 06Labs, 06Operations: Can't create account "Trizek (WMF)" - https://phabricator.wikimedia.org/T158408#3036058 (10Trizek-WMF) [11:16:21] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee, 10Tool-Labs-tools-Quentinv57's-tools: Consider publishing Quentinv57-tools' code on Diffussion and allow people to submit patches to them - https://phabricator.wikimedia.org/T158405#3036012 (10zhuyifei1999) Side note: surprisingly, `/data/project/quenti... [11:19:01] 06Labs, 06Operations: Can't create account "Trizek (WMF)" - https://phabricator.wikimedia.org/T158408#3036058 (10MoritzMuehlenhoff) There's no shell account in labs for "trizek-wmf", did you mean "trizek"? [11:31:37] 06Labs, 06Operations: Can't create account "Trizek (WMF)" - https://phabricator.wikimedia.org/T158408#3036123 (10Trizek-WMF) "trizek" is my volunteer account, which has been very quickly created during a tech workshop. I don't use it for the moment and I prefer to have a separate account for my WMF work. Did... [11:32:11] 06Labs, 06Operations: Can't create account "Trizek (WMF)" - https://phabricator.wikimedia.org/T158408#3036058 (10scfc) I believe this is due to https://wikitech.wikimedia.org/wiki/MediaWiki:Titleblacklist denying account names that contain "(WMF)". So an account "Trizek (WMF)" would probably have to be create... [11:34:33] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee, 10Tool-Labs-tools-Quentinv57's-tools: Consider publishing Quentinv57-tools' code on Diffussion and allow people to submit patches to them - https://phabricator.wikimedia.org/T158405#3036128 (10MarcoAurelio) Before publishing the source code, if approved... [11:35:31] 06Labs, 06Operations: Can't create account "Trizek (WMF)" - https://phabricator.wikimedia.org/T158408#3036129 (10MoritzMuehlenhoff) Looks like the account blacklist indeed. So either choose a different name or check with Labs Admins whether there's a way to let them create it manually. [11:40:32] 06Labs, 10wikitech.wikimedia.org: Remove 'accountcreators' right on wikitech - https://phabricator.wikimedia.org/T158413#3036156 (10MarcoAurelio) [11:48:55] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:02:29] 06Labs, 10wikitech.wikimedia.org: migrateUserGroups.php after T158039 - https://phabricator.wikimedia.org/T158416#3036225 (10MarcoAurelio) [12:03:04] 06Labs, 10wikitech.wikimedia.org: migrateUserGroups.php after T158039 - https://phabricator.wikimedia.org/T158416#3036225 (10MarcoAurelio) [12:07:16] 06Labs, 10WikimediaMessages, 10wikitech.wikimedia.org: Group messages for Wikitech - https://phabricator.wikimedia.org/T158417#3036251 (10MarcoAurelio) [12:18:52] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitech: rename 'shellmanagers' to 'shellmanager' for consistency - https://phabricator.wikimedia.org/T158039#3036283 (10MarcoAurelio) [12:18:55] 06Labs, 10WikimediaMessages, 10wikitech.wikimedia.org, 13Patch-For-Review: Group messages for Wikitech - https://phabricator.wikimedia.org/T158417#3036282 (10MarcoAurelio) [12:19:24] 06Labs, 10WikimediaMessages, 10wikitech.wikimedia.org, 13Patch-For-Review: Group messages for Wikitech - https://phabricator.wikimedia.org/T158417#3036251 (10MarcoAurelio) [12:19:27] 06Labs, 10wikitech.wikimedia.org: Remove 'accountcreators' right on wikitech - https://phabricator.wikimedia.org/T158413#3036284 (10MarcoAurelio) [12:21:16] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee, 10Tool-Labs-tools-Quentinv57's-tools: Consider publishing Quentinv57-tools' code on Diffussion and allow people to submit patches to them - https://phabricator.wikimedia.org/T158405#3036288 (10MarcoAurelio) Also, whichever is easier to contribute to. I'... [12:24:03] PROBLEM - Puppet run on tools-exec-1218 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:24:53] !log tools mass apt-get clean and removal of some old .gz log files due to 30+ low space warnings [12:24:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:28:56] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [12:40:03] !log tools create tools-exec-gift-trusty [12:40:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:48:40] 06Labs, 10Tool-Labs, 07Epic: Find a solution for tools-exec-gift on Trusty - https://phabricator.wikimedia.org/T156981#3036343 (10chasemp) details on existing > tools-exec-gift ubuntu-12.04-precise (deprecated 2014-04-17) 10.68.16.40 m1.medium - Active nova None Running 2 years, 11 months ``` Name... [12:50:38] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#3036347 (10scfc) [12:50:40] 06Labs, 10Tool-Labs: replica.my.cnf appears to not exist for my tool labs account - https://phabricator.wikimedia.org/T158389#3036346 (10scfc) [12:51:15] 06Labs, 10Tool-Labs, 07Epic: Find a solution for tools-exec-gift on Trusty - https://phabricator.wikimedia.org/T156981#3036350 (10chasemp) I created `tools-exec-gift-trusty.tools.eqiad.wmflabs` but it seems to be having issues with puppet certificates. I haven't figured out why yet, but I imagine it's conne... [12:51:17] 06Labs, 10Tool-Labs: replica.my.cnf appears to not exist for my tool labs account - https://phabricator.wikimedia.org/T158389#3035571 (10scfc) p:05Triage>03Normal [12:51:57] !log tools create tools-exec-gift-trusty-01 [12:52:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:53:32] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#3036358 (10scfc) [12:53:43] !log tools.heritage Deploy latest from Git master: 68f3aaa, a0e053f [12:53:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [12:55:42] 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3036372 (10Petrb) or @bd808? [12:57:30] 06Labs, 10Tool-Labs, 07Tracking: Make maintain-dbusers.py create replica.my.cnf files for user accounts as well - https://phabricator.wikimedia.org/T158420#3036376 (10scfc) [13:02:22] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:02:41] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#2955587 (10chasemp) The fix here is actually a bit of a misnomer, and whi... [13:04:02] RECOVERY - Puppet run on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:06] PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:05:50] 06Labs, 10Tool-Labs, 07Epic: Find a solution for tools-exec-gift on Trusty - https://phabricator.wikimedia.org/T156981#3036402 (10chasemp) ok well, I created `tools-exec-gift-trusty-01` successfully but it has failed to migrate to the tools specific master :) ```Error: Could not retrieve catalog from remo... [13:08:15] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:10:14] PROBLEM - Puppet run on tools-precise-dev is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:11:53] 06Labs, 10Tool-Labs: deb.tools.wmflabs.org is not accessible from outside Tool Labs - https://phabricator.wikimedia.org/T158383#3036408 (10scfc) Currently inside Labs `deb.tools.wmflabs.org` resolves to `10.68.16.29` (i. e., floating IP for `tools-services-01`, no webproxy). So if a proxy would be used, that... [13:12:22] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [13:13:48] PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:19:52] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:23:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [13:27:29] 06Labs, 10Tool-Labs: Investigate OOMs in trusty webgrid nodes - https://phabricator.wikimedia.org/T91194#3036421 (10chasemp) 05Open>03Invalid I am going to close this as 'too little information / from days of yore' [13:29:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:30:43] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool quota bump - https://phabricator.wikimedia.org/T158320#3036425 (10chasemp) > This task is to raise the pool from 19 instances to 25 > Instances 29 > 102400 kB = 25 instances * 4 GB/inst * 1024kB/GB > 118784 kB = 29 i... [13:33:50] RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [13:34:43] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: Allow running cronjobs on k8s - https://phabricator.wikimedia.org/T158155#3028350 (10chasemp) There are no plans to support this in the near future in any serious capacity. [13:35:53] 06Labs, 10Huggle: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted - https://phabricator.wikimedia.org/T157710#3013987 (10chasemp) @Petrb friendly ping :) [13:37:14] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 05Prometheus-metrics-monitoring: Labs Promethius not recording k8s stats since 2017-01-24T06:00 - https://phabricator.wikimedia.org/T157355#3036431 (10chasemp) @fgiunchedi could you peak at this when you have a minute? [13:37:58] 06Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: add useful content to Wikitech:Shell - https://phabricator.wikimedia.org/T56697#3036435 (10scfc) [13:38:00] 06Labs, 10WikimediaMessages, 10wikitech.wikimedia.org, 13Patch-For-Review: Group messages for Wikitech - https://phabricator.wikimedia.org/T158417#3036434 (10scfc) [13:38:51] 06Labs, 10WikimediaMessages, 10wikitech.wikimedia.org, 13Patch-For-Review: Group messages for Wikitech - https://phabricator.wikimedia.org/T158417#3036251 (10scfc) (As T56697 is somewhat related, I've added it as a parent, so that it can be revisited after this task's patch has been merged.) [13:40:46] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool quota bump - https://phabricator.wikimedia.org/T158320#3036440 (10hashar) I gave too much details I guess. The request is to bump the quota of instances to 29. That will let us have 25 instances + 2 snapshots +2 extra for pot... [13:41:44] chasemp: good morning. I guess my task about Nodepool quota bump has too many info [13:42:20] the tldr is to bump quotas to Instances: 29 , RAM 118784 , Cores 58 [13:45:07] RECOVERY - Puppet run on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [13:45:13] RECOVERY - Puppet run on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [13:53:33] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#3036456 (10hashar) >>! In T155820#3036397, @chasemp wrote: > The fix here... [13:54:54] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [13:58:35] hashar: ok andrew and I will have to talk about it a bit [13:59:11] I don't mind lurking in the conversation to get some more context :-} [14:09:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [14:18:02] 06Labs, 06Operations, 10wikitech.wikimedia.org: Can't create account "Trizek (WMF)" - https://phabricator.wikimedia.org/T158408#3036496 (10MarcoAurelio) Not sure if it applies to wikitech but the global titleblacklist at Meta do also block such usernames from creation. This however can be bypassed by any use... [14:19:04] 06Labs, 10WikimediaMessages, 10wikitech.wikimedia.org, 13Patch-For-Review: Group messages for Wikitech - https://phabricator.wikimedia.org/T158417#3036500 (10MarcoAurelio) Yep that's helpful. [14:19:54] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:59:56] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [15:19:52] 06Labs, 06Operations, 06Release-Engineering-Team: contintcloud project thinks it is using 206 fixed-ip quota errantly - https://phabricator.wikimedia.org/T158350#3036573 (10Andrew) thanks for troubleshooting -- I'll dig in the source and try to see how it's computing that quota count. [15:55:25] (03CR) 10Zppix: [C: 031] Adding configuration for tool-quarrybot-enwiki [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/338265 (https://phabricator.wikimedia.org/T158355) (owner: 10Zppix) [16:05:43] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee, 10Tool-Labs-tools-Quentinv57's-tools: Consider publishing Quentinv57-tools' code on Diffussion and allow people to submit patches to them - https://phabricator.wikimedia.org/T158405#3036714 (10Huji) >>! In T158405#3036128, @MarcoAurelio wrote: > Before... [16:09:43] 10Tool-Labs-tools-Xtools, 06Community-Tech: [PLAN] Move development for xtools from my repo to the project repo - https://phabricator.wikimedia.org/T158102#3036719 (10Matthewrbowker) After looking at this further, the steps should probably be done in reverse order. Therefore, here is what I propose. @MusikAn... [16:11:58] 06Labs, 10Horizon, 06Operations, 13Patch-For-Review, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#3036720 (10scfc) AFAIUI, https://horizon.wikimedia.org/ has been updated to Mitaka which shows all roles, regardless of `filtertags`. Clicking on a Puppet tab no... [16:18:33] 06Labs, 10Horizon, 06Operations, 13Patch-For-Review, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#3036740 (10Paladox) Using th material skin still takes along time to load this tab. So some how the performance improvements weren't done for that skin. [16:20:18] 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3036760 (10bd808) You can enable `role::labs::lvm::srv` on your instance and force a puppet run via `sudo -i puppet agent --test --verbose`. This will create a partition that fills the remainder of your instance's disk... [16:20:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:22:28] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee, 10Tool-Labs-tools-Quentinv57's-tools: Consider publishing Quentinv57-tools' code on Diffussion and allow people to submit patches to them - https://phabricator.wikimedia.org/T158405#3036012 (10scfc) >>! In T158405#3036084, @zhuyifei1999 wrote: > Side no... [16:30:29] PROBLEM - Puppet run on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:37:16] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 05Prometheus-metrics-monitoring: Labs Promethius not recording k8s stats since 2017-01-24T06:00 - https://phabricator.wikimedia.org/T157355#3036789 (10fgiunchedi) Sure @chasemp! Looks like the tag `kubernetes_namespace` has been renamed to `kubernetes`, possibly fol... [16:39:49] 06Labs, 06Operations, 06Release-Engineering-Team: contintcloud project thinks it is using 206 fixed-ip quota errantly - https://phabricator.wikimedia.org/T158350#3036804 (10Andrew) Usually you can force quota recalculation with MariaDB [nova]> select * from quota_usages where project_id='contintcloud'; In... [16:41:29] 06Labs, 10Tool-Labs, 06Project-Admins: Migrate Tools access request process to Phabricator - https://phabricator.wikimedia.org/T72625#3036805 (10scfc) I haven't tested #Striker yet because I would need a second SUL account for that (?), but I think one of the motivations of T128158 was to make signing up for... [16:42:37] 06Labs, 06Operations, 10wikitech.wikimedia.org: Can't create account "Trizek (WMF)" - https://phabricator.wikimedia.org/T158408#3036058 (10bd808) @Trizek-WMF, I or any other Wikitech admin can make you an account that bypasses the title blacklist rules if you really want it. Typically we don't require or enc... [16:46:47] 06Labs, 06Operations, 10wikitech.wikimedia.org: Can't create account "Trizek (WMF)" - https://phabricator.wikimedia.org/T158408#3036838 (10Trizek-WMF) >>! In T158408#3036810, @bd808 wrote: > @Trizek-WMF, I or any other Wikitech admin can make you an account that bypasses the title blacklist rules if you real... [16:56:16] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 05Prometheus-metrics-monitoring: Labs Prometheus not recording k8s stats since 2017-01-24T06:00 - https://phabricator.wikimedia.org/T157355#3036855 (10fgiunchedi) [17:00:26] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [17:05:27] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [17:08:37] 06Labs, 06Operations, 10wikitech.wikimedia.org: Can't create account "Trizek (WMF)" - https://phabricator.wikimedia.org/T158408#3036895 (10bd808) >>! In T158408#3036838, @Trizek-WMF wrote: > I prefer to have separate accounts, like I've done for all other accounts. Why is it not encouraged? The technical co... [18:07:43] 06Labs, 06Operations, 06Release-Engineering-Team: contintcloud project thinks it is using 206 fixed-ip quota errantly - https://phabricator.wikimedia.org/T158350#3036994 (10Andrew) I restarted nova-network and it looks like nova is cleaning up those leaks now. I'll keep an eye out, but I've reduced the quot... [18:23:16] 06Labs, 10Tool-Labs, 06Project-Admins: Migrate Tools access request process to Phabricator - https://phabricator.wikimedia.org/T72625#3037006 (10bd808) >>! In T72625#3036805, @scfc wrote: > I haven't tested #Striker yet because I would need a second SUL account for that (?), Making a SUL account is cheap,... [18:23:19] 06Labs, 06Operations, 10netops: asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues - https://phabricator.wikimedia.org/T155875#3037010 (10faidon) 05Open>03Resolved a:03faidon The "Sanity Checks Failed" log messages continue to happen sporadically but we haven't had a switch failure in over 3 weeks no... [18:39:52] 06Labs, 10Tool-Labs, 07Epic: Find a solution for tools-exec-gift on Trusty - https://phabricator.wikimedia.org/T156981#3037043 (10chasemp) handled the cert issue via https://wikitech.wikimedia.org/wiki/Standalone_puppetmaster#Step_2:_Setup_a_puppet_client :) [18:42:42] PROBLEM - Puppet run on tools-exec-gift-trusty-01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [18:47:40] RECOVERY - Puppet run on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:22] 10Tool-Labs-tools-Xtools, 06Community-Tech: [PLAN] Move development for xtools from my repo to the project repo - https://phabricator.wikimedia.org/T158102#3037098 (10MusikAnimal) @Matthewrbowker Sounds good! No conflicts with our workflow, which we happily will adapt to whatever works best for you and us both... [19:02:44] 10Tool-Labs-tools-Xtools, 06Community-Tech: [PLAN] Move development for xtools from my repo to the project repo - https://phabricator.wikimedia.org/T158102#3026520 (10Matthewrbowker) {icon thumbs-up} Excellent. It's a plan. [19:03:17] 10Tool-Labs-tools-Xtools, 06Community-Tech: [PLAN] Move development for xtools from my repo to the project repo - https://phabricator.wikimedia.org/T158102#3037141 (10Matthewrbowker) [19:05:40] 06Labs, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: contintcloud project thinks it is using 206 fixed-ip quota errantly - https://phabricator.wikimedia.org/T158350#3037145 (10Andrew) 05Open>03Resolved I cleaned up about 100 leaks, like this: update fixed_ips a, instances b set a.inst... [19:07:00] aude: can you please comment on https://phabricator.wikimedia.org/T157708, or direct me to someone who knows something? [19:07:09] !log tools.ytcleaner Changed cleaner.sh cron from qsub to jsub so job will spawn on Trusty [19:07:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ytcleaner/SAL [19:11:29] !log tools.wiwosm Changed cron from qsub to jsub so job will spawn on Trusty [19:11:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wiwosm/SAL [19:16:09] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3037165 (10Capt_Swing) Thank you @tgr and @bd808 for addressing this issue so promptly! [19:21:51] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#3037178 (10Andrew) [19:21:53] 06Labs: Request creation of wikifactmine labs project - https://phabricator.wikimedia.org/T157385#3037175 (10Andrew) 05Open>03Resolved a:03Andrew @Tarrow, I've created this project. You are currently the only projectadmin, but you can add other members or projectadmins via the 'manage project' link on Wik... [19:23:06] !log tools.wikifactmine-api Deleted QLOGIN job submitted Wed Nov 16 13:45:47 2016 which had no running process on the associated exec node [19:23:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikifactmine-api/SAL [19:23:41] thanks andrewbogott :) [19:23:42] PROBLEM - Puppet run on tools-exec-gift-trusty-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:38:42] RECOVERY - Puppet run on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:18:04] PROBLEM - Host tools-exec-gift-trusty is DOWN: CRITICAL - Host Unreachable (10.68.21.65) [20:19:57] 06Labs, 10Tool-Labs, 07Epic: Find a solution for tools-exec-gift on Trusty - https://phabricator.wikimedia.org/T156981#3037318 (10chasemp) OK @Giftpflanze. > tools-exec-gift-trusty-01.tools.eqiad.wmflabs > qconf -Ae /var/lib/gridengine/etc/exechosts/tools-exec-gift.tools.eqiad.wmflabs > exechost "tools-exe... [20:20:59] 06Labs, 10Tool-Labs, 07Epic: Find a solution for tools-exec-gift on Trusty - https://phabricator.wikimedia.org/T156981#3037319 (10chasemp) [20:56:04] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: Allow running cronjobs on k8s - https://phabricator.wikimedia.org/T158155#3037342 (10yuvipanda) >>! In T158155#3034120, @Legoktm wrote: > To resolve this task I think we need documentation somewhere on wikitech mentioning the feature exists and w... [21:18:19] 10Tool-Labs-tools-Other, 06Operations: Jouncebot: Crashes when issued a command. - https://phabricator.wikimedia.org/T158448#3037357 (10Zppix) [21:19:15] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: Allow running cronjobs on k8s - https://phabricator.wikimedia.org/T158155#3028350 (10bd808) @Legoktm I think 'we' can create a page on wikitech that collects documentation about 'advanced' Kubernetes usage. There are some things I can contribute... [21:21:52] 10Tool-Labs-tools-Other, 06Operations, 10Stashbot: Jouncebot: Crashes when issued a command. - https://phabricator.wikimedia.org/T158448#3037387 (10Paladox) [21:23:29] 10Tool-Labs-tools-Other, 06Operations, 10Stashbot: Jouncebot: Crashes when issued a command. - https://phabricator.wikimedia.org/T158448#3037357 (10bd808) ``` ERROR:root:Unhandled exception. Terminating. Traceback (most recent call last): File "./jouncebot/jouncebot.py", line 281, in bot.start... [21:24:38] bd808: got a moment ref that eventlogging issue? [21:24:57] 10Tool-Labs-tools-Other, 06Operations: Jouncebot: Crashes when issued a command. - https://phabricator.wikimedia.org/T158448#3037394 (10Paladox) [21:35:11] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes: Unify k8s roles between prod and tools - https://phabricator.wikimedia.org/T158452#3037440 (10yuvipanda) [21:36:34] samtar: eh. kind of [21:36:45] do you have any error logs or just a lack of data? [21:37:02] eventlogging is not my specialty [21:37:56] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:39:43] bd808: nothing in logs/eventlogging.log or upstart/eventlogging-devserver.log :/ [21:39:52] (so no data, no errors) [21:40:42] I don't remember how the server side events work. Do they actually make a curl call to the collector? [21:41:09] or do they just write to some logging channel or something that ends up in the right place in production? [21:42:31] https://github.com/wikimedia/mediawiki-extensions-SpamBlacklist/commit/5910bfd7ba0164561f8b5f3d6e97197abe3436dd (GH) is the addition of the hook if that helps [21:43:57] ok, so that looks like it actually does make an http call [21:44:56] EventLogging::logEvent calls EventLogging::sendBeacon which then does an Http::post [21:45:18] the next question is what URL is it ending up posting to [21:45:47] easiest way to find that out would be for you to hack some logging into EventLogging::sendBeacon [21:45:51] 06Labs, 10Tool-Labs, 10Prod-Kubernetes, 10Tools-Kubernetes, and 2 others: Set-up live-restore for docker containers - https://phabricator.wikimedia.org/T157180#3037469 (10yuvipanda) This has been rolled out to prod as well. \o/ [21:47:14] bd808: just dump what it's up to to a file? [21:47:45] yeah or even jsut blow up with an error that will leave the url somewhere you can find it [21:47:58] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:48:32] either the collector doesn't work on mw-vagrant (which is possible) or its going to the wrong url (also possible) [21:48:52] I suppose you should also send some events by hand to the collector to see if it works [21:49:47] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: Allow running cronjobs on k8s - https://phabricator.wikimedia.org/T158155#3037473 (10scfc) 05Open>03Resolved a:03yuvipanda (This task was about allowing running `cron` jobs on Kubernetes, and that has been resolved; https://wikitech.wikimed... [21:56:18] FULLLLL LIKE [21:57:47] ممنون لطف عالی متعالی [21:57:53] تأیید [22:04:55] bd808: is there any way to find out how a specific role/module on vagrant gets enabled? I.e. I have varnish running there, but no idea which role created it [22:05:46] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:05:58] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:06:08] SMalyshev: no good way, no. We don't have any dependecy tree graphs or anything like that. `git grep` is your best bet [22:06:37] also it could have come from a role you had enabled at some point and then disabled [22:06:37] bd808: yeah grep yeilds a lot of result, all of which point to modules I didn't enable... [22:06:50] roles don't clean up after themselves [22:06:50] bd808: hmm so how I get rid of it then? [22:07:12] 10Tool-Labs-tools-Other, 07Epic: Toolserver.org tools that have not been migrated (tracking) - https://phabricator.wikimedia.org/T60865#3037488 (10scfc) [22:07:13] ssh in, stop it, and agpt-get remove the package? [22:07:14] 10Tool-Labs-tools-Other, 13Patch-For-Review: Migrate https//toolserver.org/~wiegels/wikipedia-termine.php to Tool Labs - https://phabricator.wikimedia.org/T62888#3037487 (10scfc) 05Open>03Resolved [22:08:01] bd808: hmmm what if it's not only the package? I mean some roles have pretty complex setups including messing with other roles' configs... [22:08:10] SMalyshev: or rebuild your VM :) cattle not pets [22:08:48] I personally recommend vagrant destroy; vagrant up about once a month [22:08:58] wait, you mean wipe the whole vagrant and rebuild it from scratch? [22:09:00] I've been slacking on that myself lately [22:09:35] the VM, yeah. The only time that ends up being hard is when you have lots of test pages that you need [22:09:42] that sounds kinds harsh and probably will require me moving a lot of files back and forth... [22:10:16] i have a bunch of configs, WIP scripts, etc. there [22:10:35] inside the VM iteself? [22:10:51] yeah, of course, that's where I need them since VM uses them [22:11:23] I tend to keep everything on my laptop and just mount it in, but obviously there are many ways to do things [22:12:21] hmm I do have /vagrant mounted on vagrant machine so maybe it's not inside VM? [22:12:32] mw-vagrant's puppet code and role management system assume that VMs are easily recreated and that doing so is easier than cleaning up [22:13:03] anything in /vagrant should be on your laptop too [22:13:18] certainly double check before you nuke it all :) [22:13:21] ah ok, most of things I need are on /vagrant [22:14:38] the databases are inside the VM and will be lost though, so if you have a lot of wiki customizations you probably want to mysqldump before destroying [22:15:00] same for elasticsearch indexes [22:15:50] elastic I don't care too much but wikis I'd like to keep. Do we have any scripts that just dump everything? [22:15:59] (and that can be reversed later?) [22:16:20] b/c last time I tried to import something I ended up with a completely broken wiki :( [22:17:26] SMalyshev: that's something we are really missing. :/ There are a couple bugs open about it but nobody has done the work to make it an easy process [22:17:51] in theory you can mysqldump everything and then load it back in the new vm [22:18:03] 10Tool-Labs-tools-Other, 07Epic: Toolserver.org tools that have not been migrated (tracking) - https://phabricator.wikimedia.org/T60865#3037495 (10scfc) [22:18:08] 10Tool-Labs-tools-Other, 06Commons: Move contests from toolserver to tool labs - https://phabricator.wikimedia.org/T63826#3037493 (10scfc) 05Open>03declined I believe there have been photo contests (WLM?) in the past three years with specialized software, so these tools do not seem to have been necessary a... [22:19:18] bd808: is there any script for it? there's 20+ databases now... I can write my own of course but it slowly turns into a project, which I am trying to avoid... [22:20:16] mysqldump --all-databases > all_databases.sql [22:20:34] 10Tool-Labs-tools-Other, 07Epic: Toolserver.org tools that have not been migrated (tracking) - https://phabricator.wikimedia.org/T60865#3037503 (10scfc) [22:20:37] 10Tool-Labs-tools-Other, 06Commons, 06Community-Tech, 10Internet-Archive: Create a new DerivativeFX after the Toolserver shutdown [AOI] - https://phabricator.wikimedia.org/T110409#3037502 (10scfc) [22:20:45] ahh that way... ok [22:20:52] that would probably work :) [22:21:09] its crude, but possible I think [22:21:55] It would be really neat to have `vagrant backup` and `vagrant restore` commands [22:22:04] yup :) [22:22:38] * bd808 declares this year the year of mw-vagrant on the desktop [22:38:52] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Make maintain-kubeusers run on first attempt - https://phabricator.wikimedia.org/T158453#3037512 (10yuvipanda) [22:40:00] 10Tool-Labs-tools-Other, 07Epic: Toolserver.org tools that have not been migrated (tracking) - https://phabricator.wikimedia.org/T60865#3037530 (10scfc) [22:40:47] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:01] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [22:42:43] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Make maintain-kubeusers run on first attempt - https://phabricator.wikimedia.org/T158453#3037531 (10chasemp) p:05Triage>03Normal [22:46:24] 10Tool-Labs-tools-Other, 07Epic: Toolserver.org tools that have not been migrated (tracking) - https://phabricator.wikimedia.org/T60865#3037537 (10scfc) [22:46:26] 10Tool-Labs-tools-Other: Migrate https://toolserver.org/~magnus/files_in_category.php - https://phabricator.wikimedia.org/T63181#3037535 (10scfc) 05Open>03declined AFAIUI, this tool produced a list of files in a category that then could be used to create a `` section. There seems to have been no de... [22:50:38] 06Labs, 10Tool-Labs: User sdesabbata has no replica.my.cnf - https://phabricator.wikimedia.org/T157176#2998125 (10chasemp) Hi @Sdesabbata we know that user credentials are not being generated atm. If you need to go ahead and make a tool to get fresh credentials and we'll clean that up later. {T158420} [22:51:24] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:53:21] 06Labs, 10Tool-Labs, 13Patch-For-Review: /etc/cron.daily/logrotate: gzip: stdin: file size changed while zipping - https://phabricator.wikimedia.org/T96007#1206058 (10yuvipanda) I've clushed this now. [22:54:41] 10Tool-Labs-tools-Other, 07Epic: Toolserver.org tools that have not been migrated (tracking) - https://phabricator.wikimedia.org/T60865#3037554 (10scfc) [22:54:44] 10Tool-Labs-tools-Other: Migrate https://toolserver.org/~magnus/image_pages_without_image.php - https://phabricator.wikimedia.org/T63180#3037551 (10scfc) 05Open>03Resolved a:03scfc This looks like a duplicate of https://en.wikipedia.org/wiki/Wikipedia:Database_reports/File_description_pages_without_an_asso... [22:56:08] 10Tool-Labs-tools-Other, 07Epic: Toolserver.org tools that have not been migrated (tracking) - https://phabricator.wikimedia.org/T60865#605796 (10scfc) [22:56:10] 10Tool-Labs-tools-Other: Migrate https://toolserver.org/~magnus/cgi-bin/duplicate_images_across.pl - https://phabricator.wikimedia.org/T63183#3037557 (10scfc) 05Open>03declined I can't find the source ATM, but https://www.mediawiki.org/wiki/Tool_Labs/Collection_of_issues_after_Toolserver_shutdown says: "http... [23:04:45] 06Labs, 10Tool-Labs, 13Patch-For-Review: /etc/cron.daily/logrotate: gzip: stdin: file size changed while zipping - https://phabricator.wikimedia.org/T96007#3037571 (10scfc) 05Open>03Resolved a:03scfc [23:04:53] 06Labs, 10Tool-Labs, 13Patch-For-Review: /etc/cron.daily/logrotate: gzip: stdin: file size changed while zipping - https://phabricator.wikimedia.org/T96007#1206058 (10scfc) a:05scfc>03None [23:31:23] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0]