[00:07:00] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Streetfog was created, changed by Streetfog link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Streetfog edit summary: Created page with "{{Tools Access Request |Justification=Creating tools that efficiently read spam and the spammer's style. This will be used to detect socking, and an overall anti-spam bot. |Co..." [05:22:04] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:35:48] 10Tool-Labs-tools-Xtools, 06Community-Tech: Add a server-side caching service for the new XTools - https://phabricator.wikimedia.org/T161057#3174123 (10kaldari) [05:36:11] 10Tool-Labs-tools-Xtools, 06Community-Tech: Build new front-end for xtools-articleinfo - https://phabricator.wikimedia.org/T159395#3174126 (10kaldari) [06:27:04] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [06:48:19] PROBLEM - Puppet run on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:28:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:05] 06Labs, 10wikitech.wikimedia.org: Disable 2FA for Freddy2001 on Wikitech - https://phabricator.wikimedia.org/T162772#3174229 (10Freddy2001) [08:42:29] PROBLEM - Puppet run on tools-exec-1430 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:42:36] 06Labs, 06Operations: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3174423 (10akosiaris) >>! In T162462#3172669, @jcrespo wrote: > I am commenting this here, please tell me if completely unrelated and I will create a new ticket: > > db1090 keeps fai... [09:08:51] 06Labs, 06Operations: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3174462 (10akosiaris) >>! In T162462#3174423, @akosiaris wrote: >>>! In T162462#3172669, @jcrespo wrote: >> I am commenting this here, please tell me if completely unrelated and I wil... [09:17:30] RECOVERY - Puppet run on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [09:25:55] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3174471 (10fantasticfears) Sure, I am perfectly fine with `discourse-for-wiki-asian-month`. [09:43:01] 06Labs, 06Operations: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3174556 (10Joe) >>! In T162462#3174462, @akosiaris wrote: > It has stopped happening since those last lines I 've pasted above (something by cron? logrotate?). I 'll keep an eye for i... [09:46:39] 06Labs, 06Operations: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3174557 (10akosiaris) >>! In T162462#3174556, @Joe wrote: >>>! In T162462#3174462, @akosiaris wrote: >> It has stopped happening since those last lines I 've pasted above (something b... [09:52:46] 06Labs, 10DBA, 06Operations: fstrim: Operation not supported on Labs DBs - https://phabricator.wikimedia.org/T151746#3174577 (10Marostegui) 05Open>03Resolved Closing this as that cronjob isn't there anymore on those labs hosts (which will be decommissioned soon) and it is not present on the new ones either. [10:08:29] 06Labs, 06Operations, 06Release-Engineering-Team, 07Zuul: Upgrade pbr for zuul - https://phabricator.wikimedia.org/T162787#3174639 (10Paladox) [10:08:44] 06Labs, 06Operations, 06Release-Engineering-Team, 07Zuul: Upgrade pbr for zuul - https://phabricator.wikimedia.org/T162787#3174651 (10Paladox) [10:48:20] 06Labs, 06Operations, 13Patch-For-Review: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3174739 (10akosiaris) >>! In T162462#3173260, @Andrew wrote: > Fixing the puppetmaster issue requires changing (well, removing) the pinning in the puppet manifes... [11:02:19] 06Labs, 10Wikidata: Puppet alert from wikidata-mobile labs VM - https://phabricator.wikimedia.org/T162430#3174782 (10hoo) 05Open>03Resolved The VM has been deleted. [11:21:02] 06Labs, 06Operations, 13Patch-For-Review: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3174807 (10akosiaris) 05Open>03Resolved Note that after merging https://gerrit.wikimedia.org/r/347825 removal of /etc/apt/preferences.d/puppet.pref is requir... [11:23:50] (03PS11) 10D3r1ck01: Add package.json and README.md [labs/tools/Wikimedia-Emoji-Bot] - 10https://gerrit.wikimedia.org/r/347304 [12:43:13] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3175038 (10fantasticfears) @AddisWang suggests to me that the project name may have a long url as a result. I would think it will be perfectly fine if we can assign the domain to the instance, e.g.... [13:12:14] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3153506 (10zhuyifei1999) The project name won't affect the HTTP hostname, as it can be specified [[https://wikitech.wikimedia.org/wiki/Help:Proxy|quite freely regardless of the project name]]. What... [13:23:06] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:58:07] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [14:08:36] 06Labs, 06Operations, 13Patch-For-Review: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3175218 (10Andrew) I'm getting clean puppet runs on labs instances with role::puppetmaster::standalone now. So, the labs case for this looks resolved -- anythin... [14:23:51] 06Labs, 06Operations, 13Patch-For-Review: Standalone puppet masters are broken (uninstallable packages) - https://phabricator.wikimedia.org/T162462#3163952 (10chasemp) > I'm getting clean puppet runs on labs instances with role::puppetmaster::standalone now. So, the labs case for this looks resolved -- an... [14:29:21] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3175253 (10chasemp) It makes a lot of sense to keep names abbreviated and concise and avoid alphanumeric complexity as much as possible. If I'm creating this for myself I would go for `discourse-w... [14:30:08] 06Labs, 10MediaWiki-extensions-Linter, 06Release-Engineering-Team, 07Wikimedia-log-errors: Table 'labtestwiki.linter' doesn't exist (208.80.153.14) - https://phabricator.wikimedia.org/T162605#3175255 (10jcrespo) @Legoktm we didn't get any response about this, so this is probably unmaintained. Can you use a... [14:32:41] 06Labs, 10wikitech.wikimedia.org: Disable 2FA for Freddy2001 on Wikitech - https://phabricator.wikimedia.org/T162772#3175259 (10chasemp) p:05Triage>03Normal Are you a member of the tools project? Our guidance on this is basically to sync up on irc with an admin and confirm you have your SSH key by editin... [14:36:51] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3175263 (10fantasticfears) It makes much more sense to me now. A short name `discourse-wam` would make it easier for future configuration. I edited the project name for clarity. Thank you for the e... [14:37:04] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3175264 (10fantasticfears) [14:40:04] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3175267 (10chasemp) >>! In T162134#3175263, @fantasticfears wrote: > It makes much more sense to me now. A short name `discourse-wam` would make it easier for future configuration. I edited the pro... [15:19:04] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:53:56] 06Labs, 10wikitech.wikimedia.org: Disable 2FA for Freddy2001 on Wikitech - https://phabricator.wikimedia.org/T162772#3175408 (10chasemp) 05Open>03Resolved a:03chasemp Ok I worked with @Freddy2001 to confirm they can access their Tools shell account and edit a file (`/home/freddy2001/unlock-my-2fa`) rest... [15:56:03] 06Labs, 10DBA, 07Regression: Tool Labs: Add skin, language, and variant to user_properties_anon - https://phabricator.wikimedia.org/T152043#3175411 (10Andrew) 05Open>03Resolved a:03Andrew ok, I applied https://gerrit.wikimedia.org/r/#/c/347854/ and then ran maintain-meta_p --all-databases --purge --deb... [15:59:05] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [16:52:24] 06Labs, 06Operations, 13Patch-For-Review: Instance creation fails before first puppet run around 1% of the time - https://phabricator.wikimedia.org/T160908#3114255 (10hashar) Since you mention DHCP leases. For the Nodepool images I have hunt slowness in the boot process and I got rid of a few ones related to... [16:59:15] 06Labs, 10Labs-Infrastructure, 10Icinga, 06Operations: remove/fix "Check for gridmaster host resolution" Icinga check for "labtest" - https://phabricator.wikimedia.org/T152024#3175600 (10Dzahn) p:05Low>03Normal just noticed this in Icinga again. still there. duration now 349 days. [17:00:59] 06Labs, 10Labs-Infrastructure, 10Icinga, 06Operations: remove/fix "Check for gridmaster host resolution" Icinga check for "labtest" - https://phabricator.wikimedia.org/T152024#3175605 (10Dzahn) a:03Dzahn [17:01:58] 06Labs, 10Monitoring, 10Shinken: Admin request for user paladox and Luke081515 in the project shinken - https://phabricator.wikimedia.org/T162629#3175607 (10Paladox) @chasemp Hi, any update on this please? I would like to start testing https://gerrit.wikimedia.org/r/#/c/347640/ :) [17:03:56] PROBLEM - Puppet run on tools-exec-1428 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:24:43] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: labvirt1002 ignoring some messages - https://phabricator.wikimedia.org/T162640#3175787 (10Andrew) 05Open>03Resolved [17:24:46] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3175788 (10Andrew) [17:29:38] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3175830 (10Andrew) 05Open>03Resolved This particular case, at least, is resolved. [17:32:09] 06Labs, 10Horizon, 07Developer-notice: Horizon Mitaka 'remember me' checkbox immune to keyboard focus - https://phabricator.wikimedia.org/T158103#3175839 (10Andrew) a:05Andrew>03None [17:36:53] (03CR) 10Dereckson: [C: 032] Add package.json and README.md [labs/tools/Wikimedia-Emoji-Bot] - 10https://gerrit.wikimedia.org/r/347304 (owner: 10D3r1ck01) [17:39:05] (03Merged) 10jenkins-bot: Add package.json and README.md [labs/tools/Wikimedia-Emoji-Bot] - 10https://gerrit.wikimedia.org/r/347304 (owner: 10D3r1ck01) [17:42:28] 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Set up LVS for labs dns recursors - https://phabricator.wikimedia.org/T119660#3175864 (10Andrew) a:05Andrew>03None [17:43:40] 06Labs, 10Striker, 10Horizon, 10wikitech.wikimedia.org, 07Documentation: Wikitech/horizon/striker 2fa documentation - https://phabricator.wikimedia.org/T155108#3175873 (10Andrew) a:05Andrew>03None [17:43:57] RECOVERY - Puppet run on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [17:46:16] 06Labs, 10Labs-Infrastructure: Wikitech error when adding users to projects - https://phabricator.wikimedia.org/T159021#3175883 (10Andrew) a:05Andrew>03None [17:47:22] 06Labs, 10Labs-Infrastructure: Investigate rabbitmq tcp_listen_options setting (and others) - https://phabricator.wikimedia.org/T140175#3175884 (10Andrew) a:05Andrew>03None [17:48:00] 06Labs, 10wikitech.wikimedia.org: wikitech: Tell people to log out and in again? - https://phabricator.wikimedia.org/T138373#3175887 (10Andrew) a:05Andrew>03None [17:53:34] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#3175896 (10Andrew) [17:53:37] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3175895 (10Andrew) 05Open>03Resolved [17:56:32] 06Labs, 10MediaWiki-extensions-Linter, 06Release-Engineering-Team, 07Wikimedia-log-errors: Table 'labtestwiki.linter' doesn't exist (208.80.153.14) - https://phabricator.wikimedia.org/T162605#3168367 (10Andrew) I think this was discussed on IRC, but, to follow up... this wiki is a test/dev version of wikit... [18:03:06] 06Labs, 10MediaWiki-extensions-Linter, 06Release-Engineering-Team, 07Wikimedia-log-errors: Table 'labtestwiki.linter' doesn't exist (208.80.153.14) - https://phabricator.wikimedia.org/T162605#3175944 (10jcrespo) Just see the errors- that is probably not a huge issue, but it clutters db mediawiki logging an... [18:58:21] 06Labs, 10MediaWiki-extensions-Linter, 06Release-Engineering-Team, 07Wikimedia-log-errors: Table 'labtestwiki.linter' doesn't exist (208.80.153.14) - https://phabricator.wikimedia.org/T162605#3168367 (10bd808) Table created: ``` labtestweb2001:~ bd808$ mwscript extensions/WikimediaMaintenance/createExtensi... [19:04:36] 06Labs, 10Tool-Labs: Template linking and transclusion check - Intuition warning message - https://phabricator.wikimedia.org/T162826#3176170 (10Reedy) [19:14:51] 06Labs, 10MediaWiki-extensions-Linter, 06Release-Engineering-Team, 15User-bd808, 07Wikimedia-log-errors: Table 'labtestwiki.linter' doesn't exist (208.80.153.14) - https://phabricator.wikimedia.org/T162605#3176192 (10Legoktm) 05Open>03Resolved a:03bd808 Thanks @bd808 . [19:49:55] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3041865 (10bd808) Reverted https://gerrit.wikimedia.org/r/347900 with https://gerrit.wikimedia.org/r/#/c/347905/ > We should pro... [19:57:00] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3176286 (10bd808) >>! In T143464#2569153, @aaron wrote: > Looks like the memcached client devs were too lazy to add ASCII support... [20:11:16] there is the IP address that was once icinga.wmflabs. all instances install nagios-nrpe for icinga checks from default node include [20:11:40] so that former Icinga IP gets added in local NRPE config as "allowed_hosts" to connect [20:12:19] but now that is a random pool IP, so it's some contint-cloud instance [20:14:24] this comes up because paladox wants to monitor an instance for testing, like disk space check via NRPE.. since that is all puppetized he cant just change the IP to allow his icinga2 instance IP.. puppet will change it back to that one above [20:14:49] mutante fixed it [20:14:57] so we probably want to change that to the IP that is now icinga(2) [20:14:58] i was editing wrong project [20:15:12] paladox: you could set it "per project" and in wiki? [20:15:19] paladox: that is a nice solution then [20:15:29] yep [20:15:57] alright, so for now this will work inside this project. and you know where to find it if you ever want to add more instances from other projects [20:16:50] separately we might not want that contint-cloud IP in there for all other projects.. but .. eh.. yea [20:17:20] we don't have a good default at this time [20:18:51] (03PS1) 10Thcipriani: Scap: Remove git_server from scap.cfg [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/347926 (https://phabricator.wikimedia.org/T162814) [20:20:32] 10Tool-Labs-tools-Other: Template linking and transclusion check - Intuition warning message - https://phabricator.wikimedia.org/T162826#3176366 (10Aklapper) CC'ing maintainer @Chameleon222 (assuming this is something to fix in the code of the tool itself and not in the Tool Labs infrastructure; please correct m... [20:41:58] Reedy: are you busy at the moment? [21:09:45] !log tools.meetbot Added Daniel Kinzler as maintainer [21:09:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.meetbot/SAL [21:10:26] 06Labs, 10Labs-Infrastructure, 05MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), 13Patch-For-Review: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#3176538 (10Andrew) [21:12:06] (03PS2) 10Thcipriani: Scap: Remove git_server from scap.cfg [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/347926 (https://phabricator.wikimedia.org/T162814) [21:19:19] 06Labs, 10Labs-Infrastructure, 05MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), 13Patch-For-Review: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#3176578 (10Andrew) [21:19:44] 06Labs, 10Labs-Infrastructure, 05MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), 13Patch-For-Review: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#2773808 (10Andrew) [21:20:06] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:26:59] (03CR) 10Thcipriani: [C: 04-1] "Needs I27ed2e8989db8b45e7b3397d37b961065f606bee to merge first" [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/347926 (https://phabricator.wikimedia.org/T162814) (owner: 10Thcipriani) [21:38:41] (03CR) 10Andrew Bogott: [C: 032] Implement Tool Labs membership application and processing [labs/striker] - 10https://gerrit.wikimedia.org/r/347392 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [21:42:16] (03CR) 10Andrew Bogott: [C: 032] Make Tool model field naming match LDAP attribute naming [labs/striker] - 10https://gerrit.wikimedia.org/r/347391 (owner: 10BryanDavis) [21:45:37] (03Merged) 10jenkins-bot: Make Tool model field naming match LDAP attribute naming [labs/striker] - 10https://gerrit.wikimedia.org/r/347391 (owner: 10BryanDavis) [21:46:26] (03Merged) 10jenkins-bot: Implement Tool Labs membership application and processing [labs/striker] - 10https://gerrit.wikimedia.org/r/347392 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [21:55:07] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [22:04:39] !log wikilabels ladsgroup@wikilabels-01:/srv/wikilabels/config$ sudo -u www-data /srv/wikilabels/venv/bin/wikilabels new_campaign trwiki "Değişiklik kalitesi (20,000 rastgele örnekleme, 2016)" damaging_and_goodfaith DiffToPrevious 1 50 (T161977) [22:04:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [22:04:44] T161977: Start v2 editquality campaign for trwiki - https://phabricator.wikimedia.org/T161977 [22:08:53] !log wikilabels ladsgroup@wikilabels-01:/srv/wikilabels/config$ less ~/trwiki.revisions_for_review.20k_2016.json | sudo -u www-data ../venv/bin/wikilabels task_inserts 54 (T161977) [22:08:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [22:35:50] RECOVERY - Host tools-exec-1433 is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms [22:47:02] PROBLEM - Host tools-exec-1433 is DOWN: CRITICAL - Host Unreachable (10.68.22.87) [23:35:14] (03PS1) 10D3r1ck01: Update README.md file [labs/tools/Wikimedia-Emoji-Bot] - 10https://gerrit.wikimedia.org/r/348010