[00:01:27] JustBerry: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/My_first_Flask_OAuth_tool is in pretty good shape now I think. It covers all the major parts of setting up a flask app using OAuth as a Kubernetes webservice [00:01:38] bd808: Truly--thanks. [00:02:13] there are more things I would like to add docs for in the "What next?" section, but that will come another day :) [00:03:17] The associated git repo at https://phabricator.wikimedia.org/source/tool-my-first-flask-oauth-tool/ adds a few things that the tutorial doesn't cover [00:03:57] namely multiple config files (defaults and local overrides) and using the shared Tool Labs CDN to get Bootstrap and jQuery [00:04:58] The git repo also shows some tweaks to the default Flask config to make session cookies more secure [00:05:31] https://phabricator.wikimedia.org/source/tool-my-first-flask-oauth-tool/browse/master/default_config.yaml;c53ca5002fbe942d5cc3031234c0732d3a3ff323$16-29 [00:34:03] 06Labs, 10Tool-Labs, 10Stashbot: Archive/delete tool morebots - https://phabricator.wikimedia.org/T157399#3003932 (10scfc) [00:35:50] 06Labs, 10Tool-Labs: Do not install package adminbot in execution nodes - https://phabricator.wikimedia.org/T157400#3003955 (10scfc) [00:36:30] 06Labs, 10Tool-Labs: Do not install package adminbot in execution nodes - https://phabricator.wikimedia.org/T157400#3003972 (10scfc) [00:36:32] 06Labs, 10Tool-Labs, 10Stashbot: Archive/delete tool morebots - https://phabricator.wikimedia.org/T157399#3003973 (10scfc) [00:52:10] Change on 12www.mediawiki.org a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Josephrmzv link https://www.mediawiki.org/w/index.php?diff=2388269 edit summary: [00:52:36] Change on 12www.mediawiki.org a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Josephrmzv link https://www.mediawiki.org/w/index.php?diff=2388270 edit summary: [00:52:51] Change on 12www.mediawiki.org a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Josephrmzv link https://www.mediawiki.org/w/index.php?diff=2388271 edit summary: [00:53:26] Change on 12www.mediawiki.org a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Josephrmzv link https://www.mediawiki.org/w/index.php?diff=2388272 edit summary: [00:53:42] Change on 12www.mediawiki.org a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Josephrmzv link https://www.mediawiki.org/w/index.php?diff=2388273 edit summary: Blanked the page [00:54:10] Change on 12www.mediawiki.org a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Matiia link https://www.mediawiki.org/w/index.php?diff=2388274 edit summary: Reverted edits by [[Special:Contributions/Josephrmzv|Josephrmzv]] ([[User talk:Josephrmzv|talk]]) to last revision by [[User:Ineuw|Ineuw]] [00:54:25] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Josephrmzv link https://www.mediawiki.org/w/index.php?diff=2388277 edit summary: [-1114] [00:54:35] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Matiia link https://www.mediawiki.org/w/index.php?diff=2388279 edit summary: [+1114] Reverted edits by [[Special:Contributions/Josephrmzv|Josephrmzv]] ([[User talk:Josephrmzv|talk]]) to last revision by [[User:Jianhui67|Jianhui67]] [00:55:18] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Josephrmzv link https://www.mediawiki.org/w/index.php?diff=2388280 edit summary: [-1410] [00:55:35] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Josephrmzv link https://www.mediawiki.org/w/index.php?diff=2388281 edit summary: [-1114] [00:55:48] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Josephrmzv link https://www.mediawiki.org/w/index.php?diff=2388282 edit summary: [-356] [00:56:05] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Josephrmzv link https://www.mediawiki.org/w/index.php?diff=2388283 edit summary: [-208] [00:56:14] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Matiia link https://www.mediawiki.org/w/index.php?diff=2388284 edit summary: [+3088] Reverted edits by [[Special:Contributions/Josephrmzv|Josephrmzv]] ([[User talk:Josephrmzv|talk]]) to last revision by [[User:Matiia|Matiia]] [00:56:15] that's a lot of removing [00:56:23] ah [00:59:51] 06Labs, 06Operations, 10Stashbot: Make morebots run on a production host - https://phabricator.wikimedia.org/T94638#3004040 (10bd808) Stashbot could be a candidate for running on the planned production Kubernetes cluster. The code would still need a security audit I'm sure. [01:02:37] 06Labs, 10Tool-Labs, 07Tracking: Tools that should get deleted (tracking) - https://phabricator.wikimedia.org/T133777#3004043 (10bd808) [01:02:39] 06Labs, 10Tool-Labs, 10Stashbot: Archive/delete tool morebots - https://phabricator.wikimedia.org/T157399#3004042 (10bd808) [01:04:15] 06Labs, 10Tool-Labs, 10Stashbot: Archive/delete tool morebots - https://phabricator.wikimedia.org/T157399#3003932 (10bd808) The configuration might be of minor historical interest with secrets removed. There are at least the cute custom messages for several users in there. [01:09:43] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 05Prometheus-metrics-monitoring: Labs Promethius not recording k8s stats since 2017-01-24T06:00 - https://phabricator.wikimedia.org/T157355#3004055 (10scfc) I don't know how the information flows, but: ``` [tim@passepartout ~]$ clush -g prometheus sudo ls -l /srv/p... [03:38:24] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Reguyla was created, changed by Reguyla link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Reguyla edit summary: Created page with "{{Tools Access Request |Justification=Mostly pulling SQL reports and other analysis of content to help facilitate the creation and improvement of content on the sites. I may a..." [05:00:07] 06Labs, 10Tool-Labs: tools-manifest cannot be build with pdebuild - https://phabricator.wikimedia.org/T157411#3004336 (10scfc) [05:02:20] 06Labs, 10Tool-Labs: tools-manifest cannot be build with pdebuild - https://phabricator.wikimedia.org/T157411#3004336 (10Legoktm) I'm not sure if this is what you're running into, but dh_clean is run outside of the chroot, so any packages that you need to run dh_clean with need to be installed on the host system. [05:05:04] 06Labs, 10Tool-Labs: Make debian-glue voting for labs/toollabs - https://phabricator.wikimedia.org/T157412#3004362 (10scfc) [05:05:29] 06Labs, 10Tool-Labs: Fix lintian errors in labs/toollabs - https://phabricator.wikimedia.org/T157413#3004377 (10scfc) [05:38:31] 06Labs, 10Tool-Labs: tools-manifest cannot be build with pdebuild - https://phabricator.wikimedia.org/T157411#3004410 (10scfc) I think https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786690 may be relevant here. I tried setting `DEBBUILDOPTS=-d` in `~/.pbuilderrc`, but that only confirmed your suspicion tha... [05:42:12] 06Labs, 10Tool-Labs: Puppet fails on tools-docker-builder-03 - https://phabricator.wikimedia.org/T157415#3004414 (10scfc) [05:50:42] 06Labs, 10labs-sprint-116, 10labs-sprint-117, 10labs-sprint-118, and 5 others: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#3004432 (10demon) [06:11:52] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Morebots-archived, 13Patch-For-Review: Do not install package adminbot in execution nodes - https://phabricator.wikimedia.org/T157400#3004467 (10Peachey88) [06:12:54] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Morebots-archived, 10Stashbot: Archive/delete tool morebots - https://phabricator.wikimedia.org/T157399#3004468 (10Peachey88) [07:03:04] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3004482 (10Marostegui) labsdb1011 has now commonswiki imported. I have restarted a couple of times MySQL there without any issues. Also I have done a SELEC... [08:56:05] PROBLEM - High iowait on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: tools.tools-webgrid-lighttpd-1413.cpu.total.iowait (>11.11%) [09:06:02] RECOVERY - High iowait on tools-webgrid-lighttpd-1413 is OK: OK: All targets OK [09:12:43] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:22:34] 06Labs, 10Tool-Labs, 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#3004755 (10hashar) a:03scfc @scfc made it happen :-} [09:22:39] 06Labs, 10Tool-Labs, 10Continuous-Integration-Config, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#3004757 (10hashar) 05Open>03Resolved [09:47:43] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [09:49:44] PROBLEM - Puppet staleness on tools-checker-01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [12:14:00] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team: Short Links - https://phabricator.wikimedia.org/T157434#3005045 (10Katja_Ullrich_WMDE) [12:45:27] !log wikispeech Updated logo (T156129) [12:45:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL [12:45:30] T156129: Minor adjustment to Wikispeech logo - https://phabricator.wikimedia.org/T156129 [12:54:54] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team: Short Links - https://phabricator.wikimedia.org/T157434#3005170 (10Tobi_WMDE_SW) p:05Triage>03High [12:55:34] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team: Commons shortlinks not supported - https://phabricator.wikimedia.org/T157434#3005045 (10Tobi_WMDE_SW) [12:57:29] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team, 07I18n: Internationalization (support translatewiki.net) - https://phabricator.wikimedia.org/T105345#3005179 (10Tobi_WMDE_SW) [12:59:01] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team: E-Mail in Feedback Form - https://phabricator.wikimedia.org/T157441#3005201 (10Katja_Ullrich_WMDE) [13:17:39] Sagan: ping [13:33:36] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Reguyla was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1468572 edit summary: [13:40:19] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3005360 (10Marostegui) All the tables have been imported to labsdb1010. Stopping and starting mysql worked fine, doing a SELECT over all the tables worked... [13:49:05] !log tools Deployed tools-manifest_0.11_all.deb. [13:49:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:49:32] !log tools Deployed toollabs-webservice_0.33_all.deb (T156605, T156626). [13:49:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:49:37] T156626: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626 [13:49:37] T156605: "webservice shell" fails with "No such file or directory" (with php5.6) - https://phabricator.wikimedia.org/T156605 [13:51:07] 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3005455 (10scfc) 05Open>03Resolved [13:51:39] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: "webservice shell" fails with "No such file or directory" (with php5.6) - https://phabricator.wikimedia.org/T156605#3005457 (10scfc) 05Open>03Resolved [13:57:22] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:07:23] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:58:38] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, and 2 others: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3005709 (10scfc) [14:58:42] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee: Create a wall for tools migration to trusty - https://phabricator.wikimedia.org/T156386#3005706 (10scfc) 05Open>03Resolved a:03zhuyifei1999 AFAICT this task has been done; thanks to @bd808 and @zhuyifei1999. Please file additional tasks for bugs, i... [14:59:11] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee: Create a wall for tools migration to trusty - https://phabricator.wikimedia.org/T156386#3005712 (10scfc) [15:35:39] thedj: o/ just a thought, may be worth linking to the actual CSS file at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Responsive_content_gadget - be nice to try it out before weighing in on enabling it as a gadget (which, to be fair, would be a good idea anyway) :-) [15:38:28] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: labsdb1005 (mysql) maintenance for reimage - https://phabricator.wikimedia.org/T157358#3005848 (10jcrespo) [15:40:25] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07User-notice: labsdb1005 (mysql) maintenance for reimage - https://phabricator.wikimedia.org/T157358#3002516 (10jcrespo) Adding user notice. In theory, no end users should be affected, but if some tools have not been properly programmed to reconnect, t... [15:42:32] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, and 2 others: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3005858 (10scfc) [15:42:37] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 13Patch-For-Review: Make a nag system to email maintainers of tools still running on precise grid hosts - https://phabricator.wikimedia.org/T149214#3005857 (10scfc) 05Open>03Resolved [15:44:24] samtar: done [15:45:14] thanks :) [15:51:51] samtar: i'll maybe add some screenshots later too, but i didn't get around to that yet. [16:02:20] 06Labs, 10Tool-Labs, 13Patch-For-Review: Please install mktorrent on tool labs - https://phabricator.wikimedia.org/T155470#3005960 (10Andrew) 05Open>03Resolved [16:02:54] (03PS1) 10Tim Landscheidt: Remove toolwatcher [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336425 [16:54:01] !log video git pulling v2c frontend to 2e55341 and restarting webservice [16:54:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [17:09:06] 06Labs, 10Tool-Labs: webservice/webservice2 have no man pages - https://phabricator.wikimedia.org/T95097#3006134 (10scfc) [17:09:09] 06Labs, 10Tool-Labs: Make lintian warnings voting errors in labs/toollabs repository - https://phabricator.wikimedia.org/T95098#3006133 (10scfc) [17:09:33] 06Labs, 10Tool-Labs: webservice and webservice-runner have no man pages - https://phabricator.wikimedia.org/T95097#1180355 (10scfc) [17:10:23] 06Labs, 10Tool-Labs: Make debian-glue voting for labs/toollabs - https://phabricator.wikimedia.org/T157412#3006138 (10scfc) [17:10:26] 06Labs, 10Tool-Labs: Make lintian warnings voting errors in labs/toollabs repository - https://phabricator.wikimedia.org/T95098#1180363 (10scfc) [17:10:57] 06Labs, 10Tool-Labs: Make lintian warnings voting errors in labs/toollabs repository - https://phabricator.wikimedia.org/T95098#1180363 (10scfc) [17:10:59] 06Labs, 10Tool-Labs: Make debian-glue voting for labs/toollabs - https://phabricator.wikimedia.org/T157412#3004362 (10scfc) [17:11:03] 06Labs, 10Tool-Labs: Fix lintian errors in labs/toollabs - https://phabricator.wikimedia.org/T157413#3006143 (10scfc) [17:15:12] !log video git pulling v2c frontend to bc77044 and restarting webservice [17:15:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [17:30:23] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3006177 (10MusikAnimal) Some basic documentation on how to set up a tool for the new XTools on Tool Labs: https://wikitech.wikimedia.org/wiki/Tool:Xtools [17:33:15] 10Tool-Labs-tools-LTA-Knowledgebase: Create search function - https://phabricator.wikimedia.org/T157464#3006195 (10DatGuy) [17:34:43] 10Tool-Labs-tools-LTA-Knowledgebase: Install Python - https://phabricator.wikimedia.org/T157465#3006210 (10DatGuy) [17:35:05] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Migrate XTools from Ubuntu Precise to Trusty - https://phabricator.wikimedia.org/T157123#3006213 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal I changed the jobs to run on Trusty, and without even starting them back up, I see that WikiHistory is work... [17:35:07] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, and 2 others: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#3006218 (10MusikAnimal) [17:36:03] 10Tool-Labs-tools-LTA-Knowledgebase: Install Python - https://phabricator.wikimedia.org/T157465#3006223 (10Samtar) >On labs thing I'm glad this ticket is well defined.. :P [17:36:52] 06Labs, 10Tool-Labs: Send "are you there?" email to tool labs members every 3 months to revalidate email address - https://phabricator.wikimedia.org/T148792#2732932 (10scfc) -1. I think it would be an unnecessary burden for tool maintainers while gaining no real value. Just because someone clicked on a link... [17:51:33] 06Labs, 10Tool-Labs: Virtualenvs slow on tool labs NFS - https://phabricator.wikimedia.org/T136712#3006257 (10zhuyifei1999) [17:51:36] 06Labs, 10Tool-Labs: Setup running uwsgi webservices on k8s - https://phabricator.wikimedia.org/T139783#3006254 (10zhuyifei1999) 05Open>03Resolved a:03yuvipanda With a lot of uwsgi+flask webservices already on k8s, this is done a few months ago most likely. (If not, feel free to reopen.) [17:56:10] 06Labs: Clean up backups of tools-mail on labvirt1009 - https://phabricator.wikimedia.org/T156160#3006266 (10Andrew) 05Open>03Resolved [17:58:13] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Move tools-db and tools-redis into DNS - https://phabricator.wikimedia.org/T139190#3006281 (10scfc) [18:14:25] PROBLEM - Free space - all mounts on tools-worker-1004 is CRITICAL: CRITICAL: tools.tools-worker-1004.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found)tools.tools-worker-1004.diskspace.root.byte_percentfree (<100.00%) [19:08:29] 06Labs, 10Labs-Infrastructure: Investigate if --force should be the default for nfs-manage up - https://phabricator.wikimedia.org/T157478#3006619 (10madhuvishy) [19:08:55] 06Labs, 10Labs-Infrastructure: Investigate if --force should be the default for nfs-manage up - https://phabricator.wikimedia.org/T157478#3006635 (10madhuvishy) p:05Triage>03Normal [19:11:25] 06Labs: labstore1004 is high load and periodic unavailability to icinga - https://phabricator.wikimedia.org/T155832#3006659 (10madhuvishy) 05Open>03Resolved a:03madhuvishy Incident report done - https://wikitech.wikimedia.org/wiki/Incident_documentation/20170119-Labstore. Also created https://phabricator.w... [19:29:57] !log mobile terminating unused instance 'zerotests' per discussion with dr0ptp4kt [19:30:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL [19:31:35] !log video git pulling v2c frontend to a651871 and restarting webservice [19:31:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [19:35:24] 10PAWS, 07Upstream: Hidden __pycache__/ directory prevents folders to be deleted from UI - https://phabricator.wikimedia.org/T147775#3006727 (10Abbe98) [19:58:24] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Add WYMeditor - https://phabricator.wikimedia.org/T156566#3006804 (10Samtar) [19:59:16] 10Tool-Labs-tools-LTA-Knowledgebase, 15User-Samtar: Add tag functionality - https://phabricator.wikimedia.org/T155343#3006806 (10Samtar) [20:04:28] !log video git pulling v2c frontend to 74a4ad8 and restarting webservice [20:04:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [20:07:48] (03PS1) 10Andrew Bogott: Add a bunch of dummy files for service-dev secrets [labs/private] - 10https://gerrit.wikimedia.org/r/336462 [20:07:51] (03PS1) 10Andrew Bogott: Add dummy passwords::mirrors [labs/private] - 10https://gerrit.wikimedia.org/r/336463 [20:09:32] (03CR) 10Dzahn: [C: 032] Add dummy passwords::mirrors [labs/private] - 10https://gerrit.wikimedia.org/r/336463 (owner: 10Andrew Bogott) [20:10:03] (03CR) 10Dzahn: [V: 032 C: 032] Add dummy passwords::mirrors [labs/private] - 10https://gerrit.wikimedia.org/r/336463 (owner: 10Andrew Bogott) [20:11:02] (03CR) 10Dzahn: [C: 031] Add a bunch of dummy files for service-dev secrets [labs/private] - 10https://gerrit.wikimedia.org/r/336462 (owner: 10Andrew Bogott) [20:14:32] (03CR) 10Andrew Bogott: [V: 032 C: 032] Add a bunch of dummy files for service-dev secrets [labs/private] - 10https://gerrit.wikimedia.org/r/336462 (owner: 10Andrew Bogott) [20:14:54] (03PS1) 10Andrew Bogott: Add docker/registry.yaml [labs/private] - 10https://gerrit.wikimedia.org/r/336465 [20:15:25] (03CR) 10Andrew Bogott: [V: 032 C: 032] Add docker/registry.yaml [labs/private] - 10https://gerrit.wikimedia.org/r/336465 (owner: 10Andrew Bogott) [20:17:15] 06Labs: labstore1004 is high load and periodic unavailability to icinga - https://phabricator.wikimedia.org/T155832#2956151 (10scfc) https://wikitech.wikimedia.org/wiki/Incident_documentation/20170119-Labstore#Actionables links with the text "Recable eth1 to be a direct crossover (to avoid eth0 and eth1 being on... [20:37:05] the error i reported yesterday on my instance is back [20:37:33] and again i have not changed it since yesterday when it was fixed by re-applying the correct role [20:37:46] again it looks like it's getting puppet::self or something [20:38:03] and then permission and depdency issues on .puppet/ stuff in my home [20:39:20] (03PS1) 10Andrew Bogott: Restbase secrets: Add more dummy files, hopefully all of them this time [labs/private] - 10https://gerrit.wikimedia.org/r/336472 [20:40:25] (03CR) 10Andrew Bogott: [V: 032 C: 032] Restbase secrets: Add more dummy files, hopefully all of them this time [labs/private] - 10https://gerrit.wikimedia.org/r/336472 (owner: 10Andrew Bogott) [20:46:09] (03PS1) 10Andrew Bogott: Move a misplaced dummy hiera file [labs/private] - 10https://gerrit.wikimedia.org/r/336474 [20:46:39] (03CR) 10Andrew Bogott: [V: 032 C: 032] Move a misplaced dummy hiera file [labs/private] - 10https://gerrit.wikimedia.org/r/336474 (owner: 10Andrew Bogott) [20:49:11] Hi - I'm interested in running my own MediaWiki site but I don't want to maintain the server infrastructure myself. Am I in the right place to do this? [20:51:14] Pinging chanops bd808 and madhuvishy [20:51:33] I would like an answer as soon as possible [20:51:39] my instances gets misconfigured by horizon somehow [20:52:10] or things break at intervals and then come back and break again later [20:52:21] Matthew123: the answer is a solid "maybe". :) Can you explain a bit more what your wiki will be used for? [20:52:37] mutante: what's up? [20:52:37] (03PS1) 10Andrew Bogott: Add more dummy private hiera settings for swift [labs/private] - 10https://gerrit.wikimedia.org/r/336478 [20:52:54] Bd808 I want to create a wiki on cities in the United States, including specific buildings, bridges, rivers, etc. [20:53:02] basically everything related to civil engineering [20:53:08] (03CR) 10Andrew Bogott: [V: 032 C: 032] Add more dummy private hiera settings for swift [labs/private] - 10https://gerrit.wikimedia.org/r/336478 (owner: 10Andrew Bogott) [20:53:18] and I also want to customize the MediaWiki features entirely myself [20:53:23] madhuvishy: on my instance yesterday i noticed puppet was broken, lots of dependency and permission errors for things in .puppet in my home dir, as if i had puppet::self or something [20:54:00] madhuvishy: then i checked in horizon for the configuration and noticed the normal role class it is supposed to have was not selected... so i added it back... and the error diseappared. don't know what changed it [20:54:14] Matthew123: hmmm... ok. Generally we are not in the business of providing wiki farm hosting for arbitrary projects. Civil engineering content seems like something that would fit into the wikipedia and/or wikiversity projects. [20:54:15] madhuvishy: today, i come back and even though it was fixed yesterday.. the same error is back now [20:54:40] Bd808 of course, that's only one topic [20:54:42] mutante: huh - is the role still applied? also - which instance [20:54:45] madhuvishy: this time my actual role class is configured in horizon but again it looks like i have puppet::self instead or so... and i never configured that [20:54:47] I'm open to others [20:55:13] I just want a place to run a personal MediaWiki site where I can manage all of the on-wiki configuration without maintaining the server infrastructure [20:55:22] madhuvishy: instance: wikistats-cowgirl project: wikistats class: role::wikistats::instance error: Error: Could not set 'directory' on ensure: Permission denied @ dir_s_mkdir - /home/dzahn/.puppet/var [20:55:30] And before you ask, all of the MediaWiki hosting services are no good [20:55:40] Matthew123: miraheze.org [20:55:48] Matthew123: We are in the business of hosting development and testing instances of MediaWiki that are used to create software that is destined for the Wikimedia production cluster. Content hosting is broadly out of scope however. [20:56:22] Mutante Miraheze doesn't offer what I want [20:56:30] I've checked everything [20:56:44] so it's more than "just a wiki" ? [20:56:54] what is it that you need [20:57:45] I need a wiki that I run entirely on my own without interference from staff and yet at the same time, someplace that I don't have to maintain the server [20:57:55] I would totally fail at the latter [20:57:55] mutante: i'm looking [20:58:00] madhuvishy: thank you :) [20:59:04] Matthew123: i'm afraid that doesn't exist because "without interference" and "interfere for me" are conflicting [20:59:48] Ok. I define interference as having staff groups that I can't access, staff putting limits on my configuration, etc. [20:59:49] Matthew123: even if we did content wiki hosting, we would not actually take care of the server for you. We would basically provide a VPS similar to what you could rent from AWS, Rackspace, or a large number of other providers. [20:59:49] mutante: try now? [21:00:05] Matthew123: why not just tell them what features you want enabled? they have a ticket system and all.. [21:00:13] the home directory permissions were messed up - i think i fixed [21:00:30] madhuvishy: is that possibly dormant fallout from old /home issues? [21:00:35] Mutante if you look closely, Miraheze is a Community farm [21:00:38] I don't want that [21:00:43] I want an independent wiki [21:00:56] I emailed the WMF mailing list yesterday with my request and they sent me to Labs [21:01:08] which WMF mailing list did you email? [21:01:20] madhuvishy: it looks like it's fixed, yes .. running :) [21:01:24] The generic one [21:01:25] madhuvishy: thank you [21:01:28] wikimedia-I [21:01:36] chasemp: yeah - i think mutante helped us test the initial versions of the script (might have been this instance) and /home existed there - but the permissions weren't fixed. but since /home already existed - the home fixing script didn't run there [21:01:38] Matthew123: you could pay somebody to be your admin? [21:02:09] Are you serious? [21:02:12] I want to be the admin [21:02:19] just not the server maintainer [21:02:23] there's a difference [21:02:29] i meant server admin [21:02:35] there is also this https://www.turnkeylinux.org/mediawiki [21:02:45] the issue is that I don't know how to fix the server should it crash or perform upgrades etc. [21:02:56] "Access denied" [21:03:26] Matthew123: I think we are asking teh same questions as https://lists.wikimedia.org/pipermail/wikimedia-l/2017-February/086313.html [21:03:58] The most holistic answer I think is we do not maintain mediawiki as a service, and don't have the ability to do so [21:04:01] bd808: I made a new tool on tool-labs called 'esbackup'. It dumps the content of the elasticsearch cluster (and keeps a 10 day rotation of it) so there is a backup. I've added you to it. I remember you saying that before it wasn't backed up. No promises it works but it might come in handy. [21:04:10] Matthew123: https://lists.wikimedia.org/pipermail/wikimedia-l/2017-February/086307.html right? [21:04:14] I'm not looking for a new WMF project [21:04:22] that's something different [21:04:28] Matthew123: here, i think this is as close as you can get https://www.hostvirtual.com/os/turnkey-linux-vps/mediawiki-appliance [21:04:40] In fact, I do not want to be attached to any global community like WMF [21:04:42] tarrow: nice. Somebody was asking about that in relation to the SAL data just yesterday. [21:04:43] you pay HostVirtual for VM and select the image with preinstalled mediawiki [21:04:50] tarrow: is taht tools source somewhere? [21:04:52] curious to see [21:04:53] then you have full root [21:04:59] Cost? [21:05:22] having said that, the 2013 timestamp there doesnt look great [21:05:58] Yeah [21:06:03] OUTDATED [21:06:03] Matthew123: "In fact, I do not want to be attached to any global community like WMF" -- that 100% ensures that Wikimedia Labs is not what you are looking for. [21:06:22] Matthew123: So your request on the mailing wasn't https://lists.wikimedia.org/pipermail/wikimedia-l/2017-February/086307.html ? Could you link to your message? [21:06:29] We provide computing resources for the Wikimedia movement [21:06:39] chasemp: it uses elasticdump (https://www.npmjs.com/package/elasticdump) which is open. The 3 line bash script to make and rotate the folders isn't actually published anywhere. [21:06:50] I guess that no one can offer what I want [21:06:52] great [21:07:02] I've been searching for three months [21:07:10] and I guess I've reached the end [21:07:24] * Matthew123 is very sad [21:07:34] AmandaNP: around? [21:07:37] It really sounds like you want a shared hosting environment that you can install MediaWiki on [21:07:38] chasemp: but I'm happy to put it somewhere if that would be good [21:07:46] i see a general conflict here between "i dont want a community" "i want to do it all myself" but "i don't want to pay for it" [21:08:02] sorry, but it's hard to find people who just do free server admin work [21:08:39] Bd808 define "shared" in the context that you used it [21:08:48] it's kind of like if you want to get a car repaired [21:09:15] Matthew123: https://en.wikipedia.org/wiki/Shared_web_hosting_service [21:09:29] Isn't there another way to "refund" a server admin other than money? [21:09:47] beer? [21:10:22] How is that physically possible? [21:10:24] It's not [21:10:42] if only there was a thing to exchange work for goods [21:11:01] Matthew123: have you tried Miraheze at all, I know it was mentioned above but that's as close as you're going to get - have you been to their IRC channel? #miraheze [21:11:10] full ack, miraheze is the best you can get [21:11:14] afaict [21:11:33] they even do custdom domain names [21:11:42] That's a Community environment [21:11:43] and no ads and it's free.. cant ask for much more [21:12:06] so? you are looking for people to do things, right [21:12:10] Whatever [21:12:13] I'm out of here [21:12:14] exactly [21:12:26] I've hit the end of the road and I'm not going any further [21:12:27] we tried to help you, you know [21:12:27] Matthew123: we can't provide what you are looking for. Best of luck. [21:14:01] tarrow: I'm going to say even a trivial tool like that would benefit from a repo as there are docs and strategies for keeping history to include if possible. Nothing on the Tools NFS should be considered safe long term and especially not versioned. [21:14:48] eventually all tools will have to be deployed through version control so there is that, but we are long ways away. It's a goal though. [21:14:59] chasemp: sure; I'll chuck it up on github. [21:15:46] tarrow: you are welcome to try diffusion as well https://toolsadmin.wikimedia.org/ should be integrated I think [21:16:05] https://phabricator.wikimedia.org/diffusion/ [21:16:23] ooh; I'll have a look. :) [21:17:15] Question, and probably the wrong channel to ask in - is there any way to get what percentage of enwp admins have enabled 2FA? [21:19:14] samtar: someone with prod database access could figure that out for you, but the needed data wouldn't be in the Labs replicas [21:20:01] I'll go *prod* someone in -tech (?) unless there's anyone around with prod db access? [21:20:27] definitely some kind of audit like that would be a one-off run by someoen with access but that's an attack vector in itself revealing that list [21:20:39] making a security issue with reasoning is probably teh only sane way to approach it [21:21:01] chasemp: even just a dry x/{total admins} sort of statistic? [21:21:25] ah, well it gets greyer :) Outside the realm of my ability to make the call I think [21:21:58] I guess what would be teh use of saying 75% of admins are 2factor other than to reveal it's worth your time to try to brute force 25% of admins [21:22:08] at least as far as public disclosure [21:23:06] A fair point [21:23:19] where would I log a task for that to be considered? [21:23:58] on phab home page there isa star drop down w/ a "report security issue" but really it's "file a sensitive task that may be a security issue" [21:24:09] I would start there but I don't know where it will go honestly [21:26:24] I would be surprised if that figure was revealed outside of an NDA actually [21:27:26] Oresrian: so my idea of having a line in the enwp admin newsletter of `only x% of us have enabled 2FA - please do it now` is waaay out? [21:28:00] well, I'm not WMF, but I'd not hold up hopes [21:28:15] there's no harm in asking though [21:28:37] samtar: poke the WMF security team folks. They can make the call and would probably like that you want to do something to encourage adoption. :) [21:29:04] ^ that [21:29:12] yup [21:29:37] Email for that or do they have an IRC channel? [21:31:00] I'll just open a ticket on their project :) thanks all! [21:53:21] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:00:49] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review, 07Puppet: mwyaml chokes on existing, but empty Hiera: pages on wikitech - https://phabricator.wikimedia.org/T152142#3007439 (10scfc) 05Open>03Resolved [22:05:26] the access for tools is really slow... it takes time to ssh, become, access files... [22:07:00] madhuvishy: puppet runs on the bastions seem to be hanging as well, probably related to ^ [22:07:01] any ideas? [22:07:23] dungodung: you're on tools-bastion-03? [22:07:25] bastion-03? [22:07:32] yes [22:07:40] madhuvishy: yeah, although I think I'm seeing the same hang on 02 [22:07:45] huh [22:07:48] I can kill my puppet run if it's in the way of you looking [22:08:03] oh, no, my mistake, -02 just finished [22:08:08] so it's just -03 that's messed up [22:08:16] aaaaaand it just finished too, finally [22:08:50] * andrewbogott looks at 'top' which is what he should've done in the first place [22:08:52] chasemp had to restart bastion-03 recently due to over-use [22:08:59] i can't even ssh [22:09:06] as in, it's just sloww [22:09:57] yeah it looks pretty high on load [22:09:59] bzip2 -d wikidata-20170130-all.json.bz2 [22:10:42] we were doing so well with his [22:10:49] is it the same person doing this all the time or something? [22:11:03] 6419 be/6 tools.im 0.00 B/s 15.13 M/s 0.00 % 0.00 % bzip2 -d wikidata-20170130-all.json.bz2 [22:11:11] hm [22:11:37] madhuvishy: well unf over a year ago this was common whackamole and restarts [22:11:55] using cgroups and some heavy handedness we have gotten to here w/o much incident [22:12:04] I wonder if priority for use of bzip2 is throttled [22:12:35] unpacking a full wikidata dump :/ [22:12:36] I am poised to kill -9 that process, any objections? [22:12:44] andrewbogott: oh yes pleae [22:12:51] nope [22:12:57] and we should send them a note this is not a good idea for the bastion [22:13:36] I cannot even get on their atm [22:13:46] um… ok, since when does kill -9 not actually kill a process? [22:14:40] since it's stuck in io probably [22:14:53] if top shows D wait state [22:15:00] even -9 won't take it out iirc [22:15:02] reboot [22:15:08] this has happened 2 times before I think [22:15:12] and I didn' thave time to dig in [22:15:20] what about killing the parent proc? [22:15:25] (the shell) [22:16:06] andrewbogott: we could just reboot [22:16:14] meanwhile https://gerrit.wikimedia.org/r/336544 [22:16:18] lemme try walling and killing the user first [22:16:28] samtar: at work on lunch. Pm will be open in a sec [22:16:36] proc seems to be dead now [22:16:46] oh, wait, that's not what I want [22:16:52] ok, yes [22:16:57] nice [22:17:01] dungodung: better? [22:17:12] iotop is pretty quiet [22:17:17] andrewbogott: oh indeed, thanks [22:18:23] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:30:22] 10Labs-project-Wikistats, 13Patch-For-Review: automatic import of new miraheze wikis - https://phabricator.wikimedia.org/T153930#3007572 (10Dzahn) 05Open>03Resolved Alright, there is a cron job now that runs the automated import once per week. (weekday 5, at 11:11). This should resolve the ticket :) [22:30:39] 10Labs-project-Wikistats: automatic import of new miraheze wikis - https://phabricator.wikimedia.org/T153930#3007575 (10Dzahn) [22:32:38] /11/8 [23:19:13] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3007690 (10Capt_Swing) @Reguyla that makes perfect sense. Thanks for clarifying. If some people can't use Quarry because of site-specific configurations, that should be a major issue and needs to be addressed... [23:20:36] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3002152 (10Capt_Swing) p:05Triage>03High [23:22:12] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3007694 (10Capt_Swing) Setting priority to high because this issue prevents people from accessing Quarry. [23:25:15] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3002152 (10Halfak) I've confirmed that Quarry's OAuth consumer registration says that it is applicable to all projects -- not just Meta. So I'm not sure what's going on here. [23:30:13] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3002152 (10bd808) Quarry uses OAuth for authorization (technically not really Authenication but that's a pretty nerdy detail). Since all Wikimedia accounts have been unified there are no longer per-wiki users... [23:30:48] halfak: this 'blocked on meta' think is tricky :/ [23:30:52] *thing [23:31:57] I suppose we could have a drop down of all 800 or whatever wikis that let you pick where you weren't blocked, but ... not sure thats really a good idea [23:34:18] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3007726 (10Capt_Swing) @bd808 have other tools that use Meta for authorization encountered this issue? Surprises me that this is the first time we've seen the problem. It's a bit of an edge case, granted, but... [23:37:44] 10Quarry: Allow Quarry to work on All Project vice only Meta - https://phabricator.wikimedia.org/T157342#3007731 (10bd808) >>! In T157342#3007726, @Capt_Swing wrote: > @bd808 have other tools that use Meta for authorization encountered this issue? Surprises me that this is the first time we've seen the problem.... [23:40:42] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3007735 (10bd808)