[00:18:57] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, 10Core Platform Team Backlog (Watching / External): Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10CCicalese_WMF)
[06:30:10] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) pc1008, pc1009 and pc1010 look good!
[06:33:41] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) @Cmjohnson Actually pc1008 needs to get the RAID rebuilt - it has strip size 64. The other two pc1009 and pc1010 are ok and have 256.
[06:34:01] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui)
[07:13:28] <wikibugs>	 10DBA, 10Cloud-Services, 10cloud-services-team: Prepare and check storage layer for shnwiki - https://phabricator.wikimedia.org/T206916 (10Marostegui)
[07:38:31] <wikibugs>	 10DBA, 10Analytics, 10Analytics-Kanban: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) p:05Triage>03High
[08:18:08] <banyek>	 what are our plans with T210478?
[08:18:09] <stashbot>	 T210478: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478
[08:19:29] <elukey>	 o/
[08:19:46] <banyek>	 \o
[08:20:23] <elukey>	 banyek: I already had a chat with marostegui to start making a high level plan with some (soft) deadlines possibly, just to have a path to April 2019 that is compatible with our work schedules priorities etc..
[08:20:37] <banyek>	 great
[08:21:27] <elukey>	 I have little idea about the next steps from the puppet/mysql side, so if you have time and you want to drop some ideas in the tasks that would be really great
[08:21:49] <elukey>	 I am 100% available to help but I'd need some direction since I am fairly ignorant about the subject :)
[08:24:27] <banyek>	 I have idea about the puppet side, but I am more curious about the data layer
[08:31:22] <banyek>	 but anyway, my plan is today for proceeding with T85757
[08:31:23] <stashbot>	 T85757: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757
[08:55:51] <banyek>	 First I'll downtime s2 replication on db2095 - it will break when I execute the schema change. Then will execute the schema change on db2035 (codfw master for s2) with replication. All the schema change will replicate around, and replication will break in db2095. I'll fix the triggers on db2095, and restart replication there
[09:11:37] <marostegui>	 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/475951/
[09:11:39] <marostegui>	 I need a review for that
[09:11:55] <elukey>	 banyek: sure I didn't mean "now", during the next days though :)
[09:12:19] <banyek>	 marostegui: checking
[09:21:51] <banyek>	 btw I have a thought that I have to do the same steps every time, eg. downime all hosts even no replication lag is expectable, etc
[09:22:10] <banyek>	 if we have a good screenplay, it is easier to apply it everytime
[09:23:20] <banyek>	 triggers fixed:
[09:23:24] <banyek>	 ```root@cumin2001:/home/banyek# T85757/fix_triggers.sh db2095:3312 s2
[09:23:25] <banyek>	 fixing triggers on bgwiki
[09:23:25] <banyek>	 fixing triggers on bgwiktionary
[09:23:25] <banyek>	 fixing triggers on cswiki
[09:23:25] <banyek>	 fixing triggers on enwikiquote
[09:23:25] <banyek>	 fixing triggers on enwiktionary
[09:23:25] <banyek>	 fixing triggers on eowiki
[09:23:25] <stashbot>	 T85757: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757
[09:23:26] <banyek>	 fixing triggers on fiwiki
[09:23:26] <banyek>	 fixing triggers on idwiki
[09:23:27] <banyek>	 fixing triggers on itwiki
[09:23:27] <banyek>	 fixing triggers on nlwiki
[09:23:28] <banyek>	 fixing triggers on nowiki
[09:23:28] <banyek>	 fixing triggers on plwiki
[09:23:29] <banyek>	 fixing triggers on ptwiki
[09:24:01] <marostegui>	 banyek: please, use paste
[09:24:23] <marostegui>	 banyek: Please get used to use paste instead of pasting lots of lines here
[09:29:54] <godog>	 on production hosts you can pipe to "phaste" to paste on phabricator
[09:30:09] <godog>	 #lifeprotip
[09:34:26] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek)
[09:35:59] <wikibugs>	 10DBA, 10Toolforge, 10Wikimedia-Portals: Pagecounts.json Tool reports zero pages for all wikis - https://phabricator.wikimedia.org/T210481 (10MaxSem) Something's wrong with some labs replicas: ` MariaDB [enwiki_p]> select * from site_stats \G ERROR 1356 (HY000): View 'enwiki_p.site_stats' references invalid...
[09:37:50] <wikibugs>	 10DBA, 10Toolforge, 10Wikimedia-Portals, 10cloud-services-team: Pagecounts.json Tool reports zero pages for all wikis - https://phabricator.wikimedia.org/T210481 (10Marostegui) p:05Triage>03High a:03Bstorm >>! In T210481#4776984, @MaxSem wrote: > Something's wrong with some labs replicas: > ` > Maria...
[09:55:09] <banyek>	 codfw is good, I am proceeding with eqiad one-by-one
[09:55:51] <banyek>	 marostegui: yesterday you fully  recovered s2, right?
[09:56:06] <marostegui>	 banyek: yesterday I finished db1122 and today I finished db1095
[09:56:37] <banyek>	 thanks. Then I continue
[10:12:37] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek)
[10:13:17] <moritzm>	 I'd merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/467264/ in about 15 mins unless there are objections? (note that there might be a handful of puppet failures caused by https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=910787, but those are resolved by a second puppet run, I'll keep an eye on icinga)
[10:42:33] <marostegui>	 banyek: ^
[10:43:23] <banyek>	 I had none
[10:43:30] <banyek>	 (objections I mean)
[10:47:51] <banyek>	 moritz ^
[10:50:39] <moritzm>	 ack, will merge in a bit
[10:51:46] <jynus>	 strangely, hit ratio has only gone down to 52%
[10:51:52] <jynus>	 for parser caches
[10:52:02] <marostegui>	 you were expecting more?
[10:52:11] <jynus>	 lower
[10:52:43] <marostegui>	 it should go 1/3 more or less, no?
[10:53:05] <jynus>	 I guess it was at its peak
[10:53:11] <jynus>	 77%
[10:53:21] <marostegui>	 that'd make more sense
[10:54:22] <jynus>	 once the new hosts are in place, we can increase the retention
[10:54:33] <marostegui>	 yep, I had that in mind
[10:54:36] <jynus>	 see if we can go to around 85% hit ratio
[10:56:29] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek)
[11:01:02] <banyek>	 I check db1070
[11:01:41] <marostegui>	 thanks
[11:03:35] <banyek>	 https://phabricator.wikimedia.org/P7851
[11:04:01] <banyek>	 I have no idea what this is supposed to be
[11:04:28] <marostegui>	 jmm is moritzm user I think
[11:04:36] <banyek>	 I just checked it, yes, he is
[11:05:25] <banyek>	 moritzm? 
[11:05:49] <banyek>	 In the meantime I proceed the schema change in db1095 (I try to push this before the train)
[11:07:11] <moritzm>	 I was logged into the db1070 servers as I had seen a systemd error and was checking whether it was caused by the diamond bug linked above
[11:07:15] <moritzm>	 jmm is my shell user
[11:08:56] <banyek>	 the same shows up on all the hosts one be one, but they're greening out
[11:09:09] <banyek>	 db2071 so far, now db2082, db1123, etc
[11:14:05] <moritzm>	 yeah, that's expected, it's caused by the bug in the Debian package I linked above
[11:14:41] <moritzm>	 there's no good way to fix this, as the bug is triggered on removals (and every update of a package is also implicitly a removal), so we can't deploy a fixed deb
[11:21:27] <jynus>	 I wonder if we should force a couple of puppet runs on all dbs
[11:23:49] <banyek>	 I am not sure if we should, the errors appear, and they're gone
[11:25:00] <moritzm>	 the error vanishes with the next puppet run, I'm triggering puppet runs as they come up (it's some kind of race, doesn't affect every host)
[11:27:56] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek)
[11:28:19] <banyek>	 I run one last schema change on s2 before the train
[11:28:42] <banyek>	 (db1103)
[11:51:12] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek)
[11:52:03] <marostegui>	 banyek: did you check the schema change needs on wikitech?
[11:52:19] <marostegui>	 whether it needs to be applied, I mean
[11:52:54] <banyek>	 not yet, but I noticed your comment about it
[11:53:29] <marostegui>	 sure, I ask because I tend to forget, I don't want you to make the same mistakes I do :)
[11:54:04] <banyek>	 Thanks for bringing this up anyways
[13:10:55] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) >>! In T85757#4767175, @Marostegui wrote: > Make sure to check whether wikitech (labswiki and labtestwiki) need this sch...
[13:12:23] <banyek>	 I proceed with the schema change
[13:28:14] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui) >>! In T85757#4777377, @Banyek wrote: >>>! In T85757#4767175, @Marostegui wrote: >> Make sure to check whether wikit...
[13:39:54] <banyek>	 the alter broke on zhwiki with error of: `pymysql.err.IntegrityError: (1062, "Duplicate entry '<masked>' for key 'user_name'")`
[13:40:05] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Cmjohnson) @marostegui  it's fixed!  Sorry about that
[13:41:27] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) Thanks for the fast response @Cmjohnson! Will you re-install it or should I? Thanks!
[13:50:36] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Cmjohnson) @marostegui if you don't mind can you do the reinstall.   Thanks
[13:56:03] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) >>! In T207258#4777536, @Cmjohnson wrote: > @marostegui if you don't mind can you do the reinstall.   Thanks  Will do! Thank you!
[13:57:24] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` pc1008.eqiad.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/201811271357_m...
[13:58:00] <banyek>	 I re-run the schema change on zhwiki only, it finished without any problem this time
[13:58:05] <banyek>	 *re-ran
[13:58:57] <marostegui>	 I have pooled pc1010 in pc1 - it is a new host, I am monitoring logs, errors etc, but if you see something weird let me know. It replaces pc1004
[13:59:45] <marostegui>	 jynus banyek ^
[13:59:59] <banyek>	 👌
[14:02:39] <jynus>	 I was just looking at hit ratio
[14:02:45] <jynus>	 it went up to 69%
[14:03:02] <marostegui>	 yeah
[14:03:07] <marostegui>	 it had a nice bump
[14:03:25] <marostegui>	 it is funny how the disk usage went down hehe
[14:04:08] <jynus>	 what is the planned replication topology?
[14:04:34] <marostegui>	 same as it used to have but adding pc1010 and pc2010 to pc1 (for the sake of having them somewhere)
[14:04:41] <marostegui>	 so pc1007 -> pc2007, pc2010
[14:04:46] <marostegui>	 pc1008 -> pc2008
[14:04:48] <jynus>	 oh, I see that is already setup
[14:04:51] <marostegui>	 yeah
[14:04:52] <marostegui>	 :)
[14:10:08] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) pc1010 has been pooled into pc1 - T208383#4777571
[14:10:32] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui)
[14:17:01] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) Thanks @Cmjohnson it looks good now! ` RAID Level          : Primary-5, Secondary-0, RAID Level Qualifier-3 Size                : 4.364 TB Sector Size         : 512 Is VD emulated...
[14:17:50] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui)
[14:24:08] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['pc1008.eqiad.wmnet'] `  and were **ALL** successful.
[14:41:38] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Cmjohnson) Dell ticket information for pc1007 You have successfully submitted request SR983104667.
[14:52:07] <wikibugs>	 10DBA, 10Patch-For-Review, 10User-Banyek: Productionize dbproxy101[2-7].eqiad.wmnet - https://phabricator.wikimedia.org/T202367 (10Banyek) Then the mapping should be: `dbproxy1012:   3321: (same services as currently on dbproxy1001)   3322: (same services as currently on dbproxy1002) dbproxy1013:   3322: (sa...
[15:24:08] <banyek>	 jynus: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/473546/ 
[15:24:55] <banyek>	 ^this is a POC only for a multiinstance haproxy config, it has to be changed (even the naming, I guess) but I'd appreciate if you tell me if the direction is good
[15:25:16] <banyek>	 marostegui: I also like to hear your opinion ofc
[15:25:56] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Halfak) The second bullet point ("filter RC for edits which are ORES-nondamaging and JADE-damaging") seems like a product propos...
[15:40:43] <jynus>	 banyek: I don't think that scales for more than 2 ports
[15:40:52] <banyek>	 I am leaving soon - I have to bring my daughter from the kindergarten. I'll continue working after, but probably you won't be here then
[15:41:21] <jynus>	 maybe try to just create a single template with all services and no parameters
[15:41:21] <banyek>	 jynus: agree, that's just a proof of concept
[15:41:28] <jynus>	 and no hiera keys
[15:41:46] <jynus>	 so we just chose one template or another
[15:41:53] <jynus>	 rather than 20 different strings
[15:41:59] <banyek>	 makes sense
[15:41:59] <jynus>	 so it is less complex
[15:42:12] <jynus>	 and we only have 2 or 3 different kinds of proxies
[15:42:24] <jynus>	 but they can do many things each
[15:42:30] <jynus>	 but it looks nice
[15:42:42] <jynus>	 I would separate in 2 patches, one the genral implementation
[15:42:53] <jynus>	 and another , the actual role assignment so it is easier
[15:43:16] <banyek>	 I will work on those, thanks for the insights
[15:43:25] <jynus>	 the rest looks good
[15:44:25] <jynus>	 although more work will be needed on the firewall side
[15:44:54] <jynus>	 only the simplest cases for the firewall are already implemented
[15:45:46] <banyek>	 👍
[15:46:45] <banyek>	 I leave now, get my kid, and after I repool db1105
[15:49:15] <banyek>	 see you
[15:54:47] <hashar>	 hello
[15:54:58] <jynus>	 hi
[15:55:00] <hashar>	 ah banyek left for the kids :)
[15:55:13] <hashar>	 he merged a patch to repool db1105
[15:55:20] <hashar>	 but I guess it should not be deployed just like that :)
[15:55:28] <jynus>	 mmm
[15:55:44] <hashar>	 (context: I am about to switch group0 to the new mw version)
[15:55:46] <jynus>	 so it is merged but not deployed?
[15:55:49] <hashar>	 yeah
[15:55:57] <hashar>	 not a big deal, I can just revert it
[15:56:01] <hashar>	 and craft a new commit
[15:56:05] <jynus>	 let me check the status, I will see if I can deploy it as is or have to revert it
[15:56:31] <hashar>	 don't rush on pooling db1105 though!
[15:59:45] <jynus>	 I think it should be ok to deploy, no alter running, warmed up
[16:00:34] <hashar>	 up to you, I didn't mean to interrupt :)
[16:01:25] <jynus>	 there is actually 2 staged changes, marostegui?
[16:02:04] <jynus>	 oh, no just the one, and yours
[16:02:08] <jynus>	 which I just rebased
[16:02:17] <jynus>	 let me deploy the one from banyek
[16:02:35] <jynus>	 he should have not merged if he was going to be away soon
[16:02:58] <jynus>	 as a rule, you have to be around at leasy around an hour, or leave it for the next day
[16:03:13] <jynus>	 as mysql problems could take some time to show up
[16:03:54] <hashar>	 jynus: well maybe it is easier to just revert that
[16:04:00] <hashar>	 I can do the gerrit dance
[16:04:16] <jynus>	 no, I am deploying now
[16:04:20] <hashar>	 okkkk
[16:04:23] <jynus>	 I will be around anyway
[16:07:28] <jynus>	 hashar: he made a mistake but he will not repeat it
[16:07:40] <jynus>	 do you want me to reset your rebase or it is ok as it is?
[16:07:56] <jynus>	 (the rebase of your group0 roll)
[16:08:28] <hashar>	 jynus: let me check
[16:08:50] <hashar>	 ah
[16:08:52] <hashar>	 rebased good
[16:09:06] <jynus>	 one ping, not related to databases
[16:09:13] <jynus>	 there is some ongoing issue
[16:09:25] <hashar>	 so have you synced db1105
[16:09:44] <jynus>	 yes, I deployed the rollback
[16:09:55] <jynus>	 I am monitoring any issues, but there should be none
[16:10:00] <jynus>	 https://phabricator.wikimedia.org/T210499
[16:11:08] <hashar>	 awesome
[16:11:12] <hashar>	 I am going to sync group0 so
[16:11:28] <jynus>	 note that our deploys are not real deploys
[16:11:54] <jynus>	 they will soon disapear from mediawiki-config, hopefully
[16:12:23] <jynus>	 but I will tell banyek to not do that- not merge without deploying, or if he has a personal emergency, to ping us
[16:12:37] <jynus>	 and to leave at least 1 hour before deploy and leaving
[16:12:48] <jynus>	 as well as not interfere with reserved windows
[16:13:59] <jynus>	 hashar: sorry for the inconveniences
[16:14:28] <hashar>	 jynus: no worries :) I was just making sure nothing was ongoing 
[16:14:37] <hashar>	 our process is usually to just revert and move on
[16:14:43] <jynus>	 yes, communication is key
[16:14:48] <jynus>	 and you would have done well
[16:14:53] <hashar>	 team work!
[16:15:08] <hashar>	 would be way easier when all those config bits are moved out of mediawiki-config
[16:15:09] <jynus>	 I just happened to know it would have been safe
[16:17:34] <jynus>	 hashar: one last thing, do you know if greg or Jeena will be around today and when?
[16:19:18] <hashar>	 jynus: they should appear in roughly an hour from now
[16:19:22] <hashar>	 though greg often show up earlier
[16:19:51] <jynus>	 thanks, that is very useful
[16:27:12] <banyek>	 I am just back
[16:28:16] <banyek>	 hashar, jynus: thanks, I see the host is back in service
[16:29:29] <hashar>	 \o/
[16:29:46] <jynus>	 banyek: see my advice above
[16:31:03] <banyek>	 jynus: agree. I merged, as I wanted to deploy, but then m*rostegui told me that I was not aware of the train, that's why hold it back
[16:31:25] <banyek>	 if I were aware the trains (which I will from now on) then I wouldn't merge
[16:31:42] <jynus>	 to undeploy, merge the revert
[16:32:05] <jynus>	 also organize your breaks- if you think you won't be around, hold the merge
[16:32:24] <jynus>	 and if emergencies happen (which is ok) tell us and we can take over
[16:33:11] <jynus>	 "he jaime, I merged this but I have to go, can you take care" would be enough :-)
[16:33:27] <hashar>	 or revert in gerrit and rebase on deploy host :)D
[16:33:43] <jynus>	 yes
[16:33:52] <hashar>	 not for train on tuesday, it takes a while for us to run the scripts, so there is plenty of time in between to sneak in some mediawiki-config change :)
[16:33:58] <jynus>	 that is what I meant with "merge the revert"
[16:40:21] <banyek>	 👍
[16:55:52] <marostegui>	 banyek: I told you the train was running but I wasn't aware the change was already merged
[16:56:33] <marostegui>	 As I explained, I tend to avoid pushing changes while the train runs, and I do if it is really needed and I always ask whoever is deploying the train if I can sneak in
[16:57:17] <marostegui>	 I had no idea the change was merged, I gave you the heads up that the train was running in case you didn't see it
[16:57:35] <marostegui>	 If the change was already merged, it should've been either deployed or reverted, either way would have worked
[16:58:18] <marostegui>	 If in doubt, ask: hey I merged this, what should I do?
[16:58:34] <marostegui>	 But without knowing it, I wasn't able to advise on what to do
[16:59:05] <banyek>	 I thought you've seen the Gerrit log in IRC, because we were talking like a few seconds after the merge
[16:59:14] <banyek>	 but, yea, I agree, and lesson learned
[16:59:25] <marostegui>	 No, I didn't, I just saw the patch being pushed as in sent to gerrit
[17:02:28] <hashar>	 jynus: greg-g and jeena are around :)
[17:02:41] <jynus>	 cool
[17:02:50] <hashar>	 jynus: though jeena is a meeting right now.  Her irc nick is longma
[17:02:54] <hashar>	 but greg might be available
[17:02:55] <hashar>	 :)
[17:02:58] <jynus>	 ok
[17:03:02] <jynus>	 I will wait
[17:04:45] <greg-g>	 I'm in the tech-mgt meeting for the next hour :(
[17:05:05] <jynus>	 let's not do it here
[17:05:11] <jynus>	 lets go to -operations
[17:05:32] <jynus>	 or pms, if preferred
[17:27:14] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4777873, @Halfak wrote: > The second bullet point ("filter RC for edits which are ORES-nondamaging and JA...
[17:52:50] <wikibugs>	 10DBA, 10Data-Services, 10Patch-For-Review, 10User-Banyek, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Bstorm)
[17:54:00] <banyek>	 now I am leaving for today
[17:56:22] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek)
[18:06:11] <wikibugs>	 10DBA, 10Data-Services, 10Wikimedia-Portals, 10cloud-services-team (Kanban): site_stats tables available on Cloud replicas - https://phabricator.wikimedia.org/T210481 (10Framawiki)
[18:06:26] <wikibugs>	 10DBA, 10Data-Services, 10Wikimedia-Portals, 10cloud-services-team (Kanban): site_stats tables unavailable on Cloud replicas - https://phabricator.wikimedia.org/T210481 (10Framawiki)
[18:10:13] <wikibugs>	 10DBA, 10Data-Services, 10Wikimedia-Portals, 10cloud-services-team (Kanban): site_stats tables unavailable on Cloud replicas - https://phabricator.wikimedia.org/T210481 (10Bstorm) Rebuilt the views on all replicas.  Please check that this is working now.
[18:10:44] <wikibugs>	 10DBA, 10MediaWiki-Change-tagging, 10Operations, 10MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), and 3 others: Migrate tag_summary usage to change_tag and drop the table - https://phabricator.wikimedia.org/T209525 (10Ladsgroup) a:03Ladsgroup
[18:36:35] <wikibugs>	 10DBA, 10Data-Services, 10Wikimedia-Portals, 10cloud-services-team (Kanban): site_stats tables unavailable on Cloud replicas - https://phabricator.wikimedia.org/T210481 (10Marostegui) 05Open>03Resolved Works for me and the two reported queries now work - thanks @bstorm!: ` root@cumin1001:~# for i in la...
[18:44:39] <jynus>	 banyek|away: good work on labsdb1005
[18:46:21] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) >>! In T200297#4778378, @awight wrote: >>>! In T200297#4777873, @Halfak wrote: > Harej and I chatted about this yesterda...
[19:31:27] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4778731, @daniel wrote: >>>! In T200297#4778378, @awight wrote: >>>>! In T200297#4777873, @Halfak wrote:...
[19:35:59] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) > I think we can support filtering by adding an index on the summary data? I was planning to do this unless there's a te...
[19:44:53] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4778966, @daniel wrote: >> I think we can support filtering by adding an index on the summary data? I was...
[21:26:15] <wikibugs>	 10DBA, 10Data-Services, 10Wikimedia-Portals, 10cloud-services-team (Kanban): site_stats tables unavailable on Cloud replicas - https://phabricator.wikimedia.org/T210481 (10MaxSem) Thanks, everybody! pagecounts.json now has the correct numbers.
[21:33:38] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) > There's no prefix matching though, these are two tinyint fields holding a boolean each.   Ah right, summaries are to b...
[22:02:43] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) Thanks, this has been a helpful tangent!  >>! In T200297#4779339, @daniel wrote: >> If you feel that it's very likely pe...
[22:11:12] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) > There should be no impact on revision pager queries until we have UI to filter on the new index.  The danger with this...