[00:11:11] 10Striker, 10Tool-Labs-tools-Zppixbot, 13Patch-For-Review, 15User-bd808: ToolsAdmin: Error 500 for ZppixBot repo listing - https://phabricator.wikimedia.org/T151409#2816716 (10Zppix) @bd808 Anything i need to do to fix that? [00:14:33] 10Striker: Striker error logs not getting into ELK cluster - https://phabricator.wikimedia.org/T151422#2816717 (10bd808) [00:15:35] 10Striker, 10Tool-Labs-tools-Zppixbot, 13Patch-For-Review, 15User-bd808: ToolsAdmin: Error 500 for ZppixBot repo listing - https://phabricator.wikimedia.org/T151409#2816729 (10bd808) >>! In T151409#2816716, @Zppix wrote: > @bd808 Anything i need to do to fix that? I've got a fix on Striker's side that rem... [00:48:13] (03PS1) 10Andrew Bogott: wikitechstatusconfig: Add some dummy entries [labs/private] - 10https://gerrit.wikimedia.org/r/323096 [00:49:59] (03CR) 10Andrew Bogott: [C: 032 V: 032] wikitechstatusconfig: Add some dummy entries [labs/private] - 10https://gerrit.wikimedia.org/r/323096 (owner: 10Andrew Bogott) [00:54:40] I seem to be getting : "Closing Link: internal-server-nat.wmflabs.org (Too many user connections (global))" on my IRC bot and it won't connect to IRC. [00:54:57] Is this a problem with my bot, or Labs? [00:55:39] labs [00:55:49] well, maybe not even labs [00:56:02] Does the labs have a freenode exemption? [00:56:13] it did at some point... [00:56:24] Apparently it's freenode because too many connections are coming from that IP address [00:56:55] bd808: Bets on IP change and not telling freenode? [00:57:04] seems not unlikely [00:57:45] there were (still are?) public ips on the gird engine exec nodes to help with this at some point too I think [00:59:32] tom29739: All I can tell you to do at the moment is to file a bug :/ [01:06:30] 10Tool-Labs-tools-Xtools: Adminstats is showing non-admin users too - https://phabricator.wikimedia.org/T145677#2816803 (10Matthewrbowker) 05Open>03Resolved I have fixed the page. While it does not filter non-admins, it does now show which users are admins and which aren't. @MusikAnimal Just so you know,... [01:06:56] (03PS1) 10BryanDavis: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/323100 [01:07:14] (03CR) 10BryanDavis: [C: 032] Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/323100 (owner: 10BryanDavis) [01:07:20] (03Merged) 10jenkins-bot: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/323100 (owner: 10BryanDavis) [01:14:51] 10Striker, 10Tool-Labs-tools-Zppixbot, 13Patch-For-Review, 15User-bd808: ToolsAdmin: Error 500 for ZppixBot repo listing - https://phabricator.wikimedia.org/T151409#2816808 (10bd808) 05Open>03Resolved https://toolsadmin.wikimedia.org/tools/id/zppixbot/repos/id/labs-tools-ZppixBot [03:07:34] 06Labs, 10Pywikibot-core, 13Patch-For-Review: pywikipedia.org is not responding; pywikibot.org is not registered - https://phabricator.wikimedia.org/T106311#1464523 (10Krinkle) Current state: * http://pywikipedia.org/ - redirect to http://tools.wmflabs.org/pywikibot/ * https://pywikipedia.org/ - SSL error... [06:47:16] PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:57:51] 06Labs, 10Pywikibot-core, 13Patch-For-Review: pywikipedia.org is not responding; pywikibot.org is not registered - https://phabricator.wikimedia.org/T106311#2817089 (10jayvdb) > http://pywikibot.org - "Domain not registered" (fixed by https://gerrit.wikimedia.org/r/243688) I am seeing "Domain not configur... [07:22:19] RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [07:32:27] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ninjastrikers was created, changed by Ninjastrikers link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Ninjastrikers edit summary: Created page with "{{Tools Access Request |Justification=I would like to run a bot for Burmese Wikipedia. |Completed=false |User Name=Ninjastrikers }}" [07:40:43] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2817114 (10Marostegui) As we spoke yesterday. I am using db1052 (which was depooled yesterday) to import S1's tablesspace to db109... [08:15:30] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 1.07 ms [08:39:01] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [09:02:34] 06Labs, 06Operations, 13Patch-For-Review: grafana-labs.wikimedia.org doesn't reflect grafana-labs-admin.wikimedia.org - https://phabricator.wikimedia.org/T143556#2817176 (10Volans) p:05Triage>03Normal [09:17:36] 10Tool-Labs-tools-Pageviews, 07I18n: pageviews-latest is probably a lego message - https://phabricator.wikimedia.org/T151439#2817183 (10Amire80) [09:49:39] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2817211 (10jcrespo) dbstore1001 doesn't use GTID, and it is a delayed slave that starts replication automatically, so it is not sim... [10:01:39] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.59 ms [10:04:09] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 1.03 ms [10:06:29] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [10:07:44] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [10:18:25] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2817274 (10jcrespo) a:03jcrespo I will move the reminder dbstore1001 replication channels to the right master, hopefully not brea... [10:30:56] 06Labs, 10Tool-Labs: BUB 503: AttributeError: 'module' object has no attribute 'python_2_unicode_compatible' - https://phabricator.wikimedia.org/T144554#2603370 (10Ricordisamoa) See also https://github.com/rohit-dua/BUB/issues/52 [11:00:25] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2817425 (10ArielGlenn) [11:02:07] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 07Beta-Cluster-reproducible, 07Puppet: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#2817428 (10hashar) Puppet provisions the Puppet_Internal_CA.crt file... [11:22:09] valhallasw`vecto: When uploading a patch in the gerrit patch uploader, I get redirected to https://gerrit.wikimedia.org/r/322648%20T150521:%20Don't%20crash%20on%20queries%20with%20optional%20values (Not found) [11:47:49] I guess Gerrits response after a push chabged slightly [11:55:01] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2817499 (10jcrespo) @ArielGlenn don't add yourself to this ticket, as it will be closed soon. Chec... [12:10:47] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#2817556 (10ArielGlenn) [12:33:46] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2817590 (10jcrespo) a:05jcrespo>03None db1052 and the others should be clear to be used. [13:02:40] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Dargasia was created, changed by Dargasia link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Dargasia edit summary: Created page with "{{Tools Access Request |Justification=To provide a message forwarding bot between #wikipedia-zh and its respective Telegram group (which has 611 members). |Completed=false |Us..." [13:03:53] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Dargasia was modified, changed by Dargasia link https://wikitech.wikimedia.org/w/index.php?diff=1014985 edit summary: Elaborate current situation. [13:10:29] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2817656 (10Marostegui) All done, db1052 is now master of db1095 which is replicating ROW-based rep... [13:26:58] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2817685 (10jcrespo) __wmf_checksums can be dropped at any time, it is the table I use for running... [13:49:31] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#2817707 (10MnemonicFlow) With this username I don't remember if it did work or not. I've previously used the 'cff' shell username to access the replica databases, to which I've lost access (can't re... [14:03:25] 10Tool-Labs-tools-stewardbots: Delete old data and/or stop logging to stewardbots' SULWatcher SQL DB - https://phabricator.wikimedia.org/T151113#2817762 (10MarcoAurelio) If no one objects, I'll clean the logging table in the next days. [14:37:32] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2817856 (10Marostegui) >>! In T150960#2817685, @jcrespo wrote: > __wmf_checksums can be dropped at... [15:04:47] !log deployment-prep fixed puppet on deployment-cache-text04 by manually enabling experimental apt repo, see T150660 [15:04:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [15:04:54] T150660: Post Varnish 4 migration cleanup - https://phabricator.wikimedia.org/T150660 [15:05:53] er, mostly [15:06:00] it certainly fixed some errors [15:09:46] okay, really fixed puppet this time [15:23:20] 10Quarry: Abuse Filter should be assign to someother person - https://phabricator.wikimedia.org/T151461#2817978 (10Sro43) [15:25:37] 10Quarry: Abuse Filter should be assign to someother person - https://phabricator.wikimedia.org/T151461#2817999 (10Samtar) 05Open>03Invalid p:05Triage>03Lowest [15:42:33] 06Labs, 10Tool-Labs: Reconfigure or delete toolsbeta-valhallasw-puppet-compiler - https://phabricator.wikimedia.org/T151462#2818024 (10scfc) [16:07:32] 10Tool-Labs-tools-Pageviews: Parse protocol of Massviews external links - https://phabricator.wikimedia.org/T151463#2818071 (10MusikAnimal) [17:09:17] 06Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review: Initial data tests for db1095 (temporary db1069 - sanitarium replacement) - https://phabricator.wikimedia.org/T150960#2818230 (10Marostegui) Script finished and now db1095 is trying to catch up with the master (I sto... [18:47:56] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Implement a frontend failover solution for labsdb replicas - https://phabricator.wikimedia.org/T141097#2818509 (10jcrespo) a:03jcrespo dbproxy1010 and dbproxy1011 are now serving as proxies for labsdb1009/10/11 on the labs-support network (they ju... [18:51:27] 06Labs, 06Community-Tech, 10DBA, 10MediaWiki-extensions-PageAssessments, and 2 others: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2818541 (10dpatrick) [18:52:01] 06Labs, 06Community-Tech, 10DBA, 10MediaWiki-extensions-PageAssessments, and 2 others: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2798143 (10dpatrick) @Bawolff, can you take at this? [19:00:05] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Implement a frontend failover solution for labsdb replicas - https://phabricator.wikimedia.org/T141097#2487045 (10jcrespo) [19:00:07] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, and 3 others: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2818586 (10jcrespo) 05Open>03Resolved a:05jcrespo>03Cmjohnson The servers are wo... [19:24:06] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 2 others: Developing community norms for vital bots and tools - https://phabricator.wikimedia.org/T149312#2818655 (10Capt_Swing) Related research: //[When the Levee Breaks: Without Bots, What Happens to Wikipedia’s Quality Control... [19:50:29] 06Labs, 10Tool-Labs: Reconfigure or delete toolsbeta-valhallasw-puppet-compiler - https://phabricator.wikimedia.org/T151462#2818681 (10valhallasw) 05Open>03Resolved Thanks for looking into it -- I have deleted the instance, and I will rebuild it if necessary. (I think everything that was on it was submitte... [20:02:13] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Implement a frontend failover solution for labsdb replicas - https://phabricator.wikimedia.org/T141097#2818714 (10chasemp) How is the haproxy layer failed over (between nodes) in prod atm? LVS or ucarp/VRRP or ? [20:04:24] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#2818719 (10chasemp) There appears to be some generic problem with generating replica.my.cnf files in /home for users. It may take a bit to untangle. @andrew is there a way to recover the `cff` she... [20:13:52] 06Labs, 06Community-Tech, 10DBA, 10MediaWiki-extensions-PageAssessments, and 2 others: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2818768 (10Bawolff) a:03chasemp This looks fine. +1 from security. [20:27:59] 06Labs, 06Operations, 13Patch-For-Review: Setting up grafana should also setup Anonymous read-only access for the default org - https://phabricator.wikimedia.org/T143556#2818812 (10fgiunchedi) 05Open>03stalled [20:29:09] 06Labs, 06Operations, 13Patch-For-Review: Setting up grafana should also setup Anonymous read-only access for the default org - https://phabricator.wikimedia.org/T143556#2571642 (10fgiunchedi) [20:41:14] 10Tool-Labs-tools-Pageviews, 07I18n: pageviews-latest is probably a lego message - https://phabricator.wikimedia.org/T151439#2817183 (10MusikAnimal) @Amire80 You are right, maybe... Try it out at https://tools.wmflabs.org/pageviews . It is the "Latest" dropdown above the date range picker. E.g. if you click on... [21:55:54] 10Tool-Labs-tools-Other, 06Release-Engineering-Team, 13Patch-For-Review: Jouncebot: Add functionality to change Nick from Jouncebot_ to Jouncebot automatically - https://phabricator.wikimedia.org/T150916#2819155 (10Zppix) @bd808 any news on this change (has it been deployed or not) [22:07:50] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:09:46] damn when did we get more then 3 bastions (or only 3 public) [22:23:01] Zppix, there's only 4 that I know of. [22:23:20] tools-bastion-03, tools-bastion-02 are public ones. [22:23:38] tools-bastion-04 and tools-bastion-05 are unused I think. [22:23:40] tools-bastion-05 exists too according to shinken-wm [22:23:46] weird [22:24:08] i only knew that 2 and 3 (and i think 1 which is now just labs now ) [22:25:05] There used to be tools-bastion-01, but yuvi created tools-bastion-03 because tools-bastion-01 was overloaded. [22:25:16] tools-bastion-01 has since been deleted. [22:25:25] oh, i thought labs took it over? [22:25:41] That might have happened with tools-bastion-05, I forget which. [22:25:59] I know one was deleted because of a security incident. [22:26:10] idk i know just as much about labs as i do about how to develop a vaccine jackshit [22:34:46] lego messages are the worst. [22:36:05] ^ that made 0 sense legoktm [22:36:19] that's alright [22:36:42] lol [22:38:40] legoktm, too much eggnog? [22:38:53] no [22:38:58] I think you have your holidays confused [22:39:06] legoktm whats wrong with eggnog on thanksgiving? [22:39:43] Whatever floats your boat [22:40:42] Zppix thats for christmas [22:41:56] paladox i know but being on enwiki you gotta have some form of alcohol or you'll drive yourself crazy [22:42:05] LOL [22:42:25] just see www.enwp.org/User:Zppix/enwikihell [22:42:48] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [22:45:26] 06Labs, 10Pywikibot-core, 13Patch-For-Review: pywikipedia.org is not responding; pywikibot.org is not registered - https://phabricator.wikimedia.org/T106311#2819347 (10Krinkle) >>! In T106311#2817089, @jayvdb wrote: >> http://pywikibot.org - "Domain not registered" (fixed by https://gerrit.wikimedia.org/r/24... [22:49:39] pywiki isnt responding O_o isnt that like used like in most bots that are used on enwiki and such