[03:08:18] <wikibugs>	 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries, 10Analytics, and 2 others: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3287995 (10kaldari)
[03:08:27] <wikibugs>	 06Labs, 10Tool-Labs-tools-Database-Queries, 10Analytics, 10DBA, 10MediaWiki-Page-deletion: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3288008 (10kaldari)
[03:09:29] <wikibugs>	 06Labs, 10Tool-Labs-tools-Database-Queries, 10Analytics, 10DBA, 10MediaWiki-Page-deletion: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3287995 (10kaldari)
[03:12:48] <wikibugs>	 06Labs, 10Tool-Labs-tools-Database-Queries, 10Analytics, 10DBA, 10MediaWiki-Page-deletion: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3288010 (10Tbayer)
[03:12:58] <wikibugs>	 06Labs, 10Tool-Labs-tools-Database-Queries, 10Analytics, 10DBA, 10MediaWiki-Page-deletion: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3288011 (10kaldari) FWIW, this doesn't seem to be a lag issue as all the pages af...
[03:20:29] <wikibugs>	 06Labs, 10Tool-Labs-tools-Database-Queries, 10Analytics, 10DBA, 10MediaWiki-Page-deletion: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3288012 (10kaldari) p:05Triage>03High Marking high priority since this is aff...
[05:58:20] <wikibugs>	 06Labs, 10Tool-Labs-tools-Database-Queries, 10Analytics, 10DBA, 10MediaWiki-Page-deletion: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3288072 (10jcrespo) 05Open>03Resolved a:03jcrespo This is a known issue, wa...
[06:21:41] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[06:54:47] <shinken-wm>	 PROBLEM - Puppet errors on tools-webgrid-lighttpd-1424 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[07:01:40] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:19:07] <wikibugs>	 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2415416 (10kaldari) There seem to be several pages on English Wikipedia which have been deleted but still appear on the Labs and Analytics Store replicas in the `page` table. For example: ```lang=sql MariaDB [enwiki_p]> sele...
[07:26:41] <wikibugs>	 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3288156 (10jcrespo) "several" is too vague- I have fixed the one given:   ``` root@neodymium:~$ ./sql.py -h labsdb1001.eqiad.wmnet enwiki -e "select * from page where page_namespace = 0 AND page_title LIKE 'BatissForever'" -...
[07:27:46] <wikibugs>	 06Labs, 10Tool-Labs-tools-Database-Queries, 10Analytics, 10DBA, 10MediaWiki-Page-deletion: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3287995 (10Marostegui) Just for the record of this ticket, Jaime kindly fixed it...
[07:29:47] <shinken-wm>	 RECOVERY - Puppet errors on tools-webgrid-lighttpd-1424 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:59:26] <r96340>	 How do I upload a code file to my tool account directory?
[08:01:33] <r96340>	 How do I upload a code file to my tool account directory? I'm usung PSCP, a PuTTY SCP tool, and I don't know if there is a way to become your tool account.
[08:07:23] <Xelgen>	 Hi all, I'm developing a tool, with backend in Python3, running on ToolLabs via Kubernetes. Tool uses a linux utility djvutoxml, which is available on ToolLabs instances. But it's not available in Python virtualenv.
[08:08:12] <Xelgen>	 I have pretty 0 expereince with virtualenv, and not much experience with Python in general. So I wanted to understand it there's any way I can make this utility accessible from virtualenv of Kubernetes
[08:22:52] <r96340>	 How do I upload a code file to my tool account directory? I'm usung PSCP, a PuTTY SCP tool, and I don't know if there is a way to become your tool account.
[08:23:30] <r96340>	 *using
[08:30:55] <Xelgen>	 r96340: Have you checked this? https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP
[08:34:50] <r96340>	 Okay, I didn't found that before.
[09:11:00] <harej>	 bd808: idea: MediaWiki as a service
[09:11:07] <harej>	 I fill out a form, it spawns a MediaWiki
[09:11:43] <harej>	 addshore and I are interested in this specifically in the context of wikibase as a service
[09:23:25] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[09:53:34] <r96340_>	 Okay. I tried https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP
[09:54:02] <addshore>	 harej: bd808 indeed :P
[09:54:12] <r96340_>	 And I become my tool account, and try to change it's permissoin
[09:54:35] <r96340_>	 But it just says "Operation not permitted"
[09:58:22] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:11:17] <mhashemirc>	 hmmm anyone here have any idea why the IFTTT channel can't access the database anymore?
[10:11:20] <mhashemirc>	 PermissionsError: (1045, "Access denied for user 'tools.ifttt'@'10.68.22.147' (using password: NO)", None)
[10:12:13] <mhashemirc>	 this wouldn't have anything to do with the password would it?
[10:20:26] <mhashemirc>	 yeah something must have gotten lost in a move
[11:55:19] <Zppix>	 O/ bd808 good morning
[11:56:59] <wikibugs>	 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#3288840 (10Dvorapa) Is this still an issue? I haven't seen this issue for a while.
[13:19:23] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:29:58] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[13:59:22] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:04:57] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:19:51] <wikibugs>	 06Labs, 10Tool-Labs: Cronspam from openstack-browser - https://phabricator.wikimedia.org/T166230#3289090 (10Andrew)
[14:31:29] <wikibugs>	 06Labs, 10Tool-Labs, 13Patch-For-Review: Update maintain-kubeusers to allow tool's to write to $HOME/.kube - https://phabricator.wikimedia.org/T165875#3279230 (10chasemp) >>! In T165875#3289112, @gerritbot wrote: > Change 354839 merged by Rush: > [operations/puppet@production] tools: have maintain-kubeusers...
[14:54:47] <bd808>	 !log tools.openstack-browser Added -quiet and stderr/out to /dev/null to hourly cache purge cron job
[14:54:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL
[15:14:19] <bd808>	 harej, addshore: I'd swear I recently archived a page on wikitech describing "mediawiki as a service" that Yuvi and some other people drafted at some point.
[15:14:33] <addshore>	 hahaha
[15:14:45] <bd808>	 there are a lot of things that someone would need to work through on the requirements for that
[15:15:08] <bd808>	 like is it about being easy to test code changes or about being easy to run a content wiki
[15:15:11] <harej>	 addshore and I discussed this and we figured the best approach is to containerize it because we also want to do things like have our own blazegraph instances
[15:15:23] <harej>	 and this is for wikis with actual content
[15:15:42] <bd808>	 blazegraph as a service sounds like something that might want a whole lot of ram
[15:16:56] <bd808>	 I think this is worth exploring, bot it also sounds like something that needs quite a bit of discussion before we start building out a product
[15:18:07] <bd808>	 addshore, harej: found it -- https://wikitech.wikimedia.org/wiki/Obsolete:Projects/mediawiki_Labs_project
[15:19:22] <bd808>	 I think that particular draft plan has a lot of issues
[15:19:48] <bd808>	 but the place to start is in the use cases: what would it do and who would it do that for
[15:29:38] <wikibugs>	 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3289267 (10Anomie) Here are a few more such deleted pages that I happen to know of:  Labs: ``` MariaDB [enwiki_p]> select count(*) from page where page_namespace in (10,11) and ( page_title like 'Cite_doi%' OR page_title lik...
[15:35:26] <wikibugs>	 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3289284 (10jcrespo) To clarify, the only reliable solution is the start using the new servers already (labsdb-web.eqiad.wmnet and labsdb-analytics.eqiad.wnet). See:   ``` root@labsdb1009[enwiki_p]> select count(*) from page...
[15:42:57] <harej>	 bd808: it looks like that plan addresses a different use case. what we want is to create wikibases to quickly set up and prototype data models, either to stage for wikidata import or to have as a standalone wiki in the spirit of the federated wikibase model
[15:45:09] <wikibugs>	 06Labs, 06Operations, 07Puppet: Update redis puppet class to support stretch - https://phabricator.wikimedia.org/T166233#3289296 (10Paladox)
[15:48:28] <wikibugs>	 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3289314 (10Anomie) Is there a task tracking the updating of the DNS entries such as enwiki.labsdb to point to the new servers?
[15:49:17] <Zppix>	 bd808: stashbot can be removed from #wikimedia-bot-gerrit considering I forward that channel as the bot it was made for was replaced by wikibugs
[15:49:45] <Zppix>	 I can also make a task if you want so you remember later
[15:50:17] <wikibugs>	 10Labs-project-Phabricator: Date problem on phab-01.wmflabs.org, comments timestamp was reset - https://phabricator.wikimedia.org/T165679#3289315 (10Paladox) Yep, seems to be working now though @Framawiki still show wrong. But everything else is correct.
[15:51:19] <wikibugs>	 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3289317 (10jcrespo) No, we are not ready yet because not all wikis are available yet, but you can follow the meta ticket at: T153058  We will be finishing the reimports first, then announcing it as beta first "opt-in", and l...
[15:52:45] <bd808>	 Zppix: a task would be helpful. thanks
[15:53:11] * bd808 is a busy fellow for the next couple of weeks and might forget
[15:53:38] <Zppix>	 bd808: alright ill cc you on it :) and dont worry its not urgent...
[15:54:49] <paladox>	 bd808 thanks for your reply on https://wikitech.wikimedia.org/wiki/User_talk:BryanDavis/Rebranding_Cloud_Services_products :)
[15:55:24] <paladox>	 Though #wikimedia-cloud seems to have a user in it now so im not sure how you will register the channel if it's decided we will use that channel.
[15:56:36] <Zppix>	 bd808: task created i went ahead and cc'd thanks again!!!
[17:09:23] <paladox>	 !log phabricator upgrading mariadb to 10.2
[17:09:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL
[17:36:55] <wikibugs>	 06Labs, 10Tool-Labs, 07LDAP: Archive user account purodha - https://phabricator.wikimedia.org/T152857#2862053 (10demon) Disabled in ldap (shell set to `/bin/false`) and disabled in Gerrit (set inactive bit). I dunno about wikitech (block? scramble password?)password.
[17:39:04] <wikibugs>	 06Labs, 07LDAP: Many (approx 265) labs instances in LDAP with associatedDomain: i-00000[0-9a-f]{3}\.(\b.*\b)?eqiad\.wmflabs - https://phabricator.wikimedia.org/T142449#2535532 (10demon) I don't see any of these anymore. Resolved?
[17:41:43] <wikibugs>	 06Labs, 10Tool-Labs, 07LDAP: Archive user account purodha - https://phabricator.wikimedia.org/T152857#2862053 (10Dzahn) Since wikitech uses LDAP to login, can it be disabled in LDAP so that it also doesn't work anymore for any ldap_auth web logins? I assume setting shell to /bin/false won't do that, right.
[17:44:51] <wikibugs>	 06Labs, 10Tool-Labs, 07LDAP: Archive user account purodha - https://phabricator.wikimedia.org/T152857#3289578 (10demon) Ah, good point. Scrambling the password in LDAP should do it.
[17:46:17] <bd808>	 !log tools.stashbot restarted to part from #wikimedia-bot-gerrit (T166234)
[17:46:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[17:46:21] <stashbot>	 T166234: Stashbot: Remove #wikimedia-bot-gerrit - https://phabricator.wikimedia.org/T166234
[17:46:50] <Zppix>	 Thanks brian bd808
[17:47:27] <bd808>	 "one of these days" I want to make those kinds of changes dynamic with the running bot
[17:47:40] <Zppix>	 bd808: stashbot is python isnt it?
[17:48:25] <bd808>	 yes. the current config is a static yaml file which isn't awesome for tracking runtime changes
[17:48:26] <Zppix>	 If so iirc tom29739  did something like that before
[17:49:06] <tom29739>	 ?
[17:49:26] <bd808>	 its not technically hard but not trivial either to refactor things to allow reloading the config file and reconciling it with running state.
[17:49:28] <Zppix>	 tom29739: with piaget didnt you have the bot update while still running
[17:49:48] <tom29739>	 To a point.
[17:50:06] <tom29739>	 The bot has a command that can restart it while its running.
[17:50:18] <tom29739>	 So it restarts, but it doesn't disconnect from IRC.
[17:50:25] <tom29739>	 wm-bot does too.
[17:50:30] <Zppix>	 bd808: ^
[17:54:40] <wikibugs>	 06Labs, 07LDAP: Many (approx 265) labs instances in LDAP with associatedDomain: i-00000[0-9a-f]{3}\.(\b.*\b)?eqiad\.wmflabs - https://phabricator.wikimedia.org/T142449#3289635 (10Andrew) 05Open>03Resolved a:03Andrew Yep, I did a few big cleanups and these should be all gone.
[18:06:43] <wikibugs>	 06Labs, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Update maintain-kubeusers to allow tool's to write to $HOME/.kube - https://phabricator.wikimedia.org/T165875#3289712 (10bd808) a:03bd808 I'll clean up the permissions on the existing directories manually from labstore1005.eqiad.wmnet
[18:08:04] <wikibugs>	 06Labs, 10Tool-Labs, 15User-bd808: Cronspam from openstack-browser - https://phabricator.wikimedia.org/T166230#3289720 (10bd808) 05Open>03Resolved a:05Andrew>03bd808 Fixed by adding `-quiet` and stderr/out redirection to `/dev/null` in the crontab.
[18:19:14] <wikibugs>	 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3289757 (10jcrespo) Labs now:   ``` root@labsdb1001[enwiki_p]> select count(*) from page where page_namespace in (10,11) and ( page_title like 'Cite_doi%' OR page_title like 'Cite_pmid%' )\G *************************** 1. ro...
[18:30:32] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[18:46:51] <wikibugs>	 06Labs, 10Tool-Labs, 07LDAP: Archive user account purodha - https://phabricator.wikimedia.org/T152857#3289900 (10Dzahn) https://wikitech.wikimedia.org/wiki/LDAP#Editing_info_about_a_user  says this can be done with "sudo ldapvi -b ou=people cn=Purodha" but when actually trying on seaborgium:      [seaborgium...
[19:05:33] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:14:43] <wikibugs>	 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3290004 (10kaldari) @jcrespo: I don't have any way to identify all the extra pages on Tool Labs. All I can say is that there are 1032 extra there:  Tool Labs: ```MariaDB [enwiki_p]> select count(*) from page; +----------+ |...
[19:19:58] <wikibugs>	 06Labs, 10Tool-Labs, 07LDAP: Archive user account purodha - https://phabricator.wikimedia.org/T152857#3290052 (10Dzahn) figured it out, done:   ``` [seaborgium:~] $ sudo  ldappasswd -H ldap://ldap-eqiad.wikimedia.org -x -D "cn=admin,dc=wikimedia,dc=org" -W -S "uid=Purodha,ou=people,dc=wikimedia,dc=org" New p...
[22:12:43] <wikibugs>	 06Labs, 10Tool-Labs-tools-Database-Queries, 10Analytics, 10DBA, 10MediaWiki-Page-deletion: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3290582 (10Tbayer) >>! In T166194#3288072, @jcrespo wrote: > This is a known issu...
[22:20:23] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[22:55:23] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:10:43] <wikibugs>	 10Tool-Labs-tools-Xtools, 06Community-Tech: [Epic] Rewrite XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3290736 (10kaldari)
[23:10:45] <wikibugs>	 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: If revisions are revdel'd, articleinfo compares the surrounding edits as if it were one edit - https://phabricator.wikimedia.org/T148857#3290734 (10kaldari) 05Open>03Resolved Code looks reasonable. Haven't been able to test since XTool-dev has been down al...