[01:05:51] 10DBA, 10Wikimedia-Site-requests: Global rename of Tarawa1943 → Ontzak: supervision needed - https://phabricator.wikimedia.org/T206730 (10Nihlus) @Marostegui When would be a good time for one of us to complete this? [05:17:56] 10DBA, 10Wikimedia-Site-requests: Global rename of Tarawa1943 → Ontzak: supervision needed - https://phabricator.wikimedia.org/T206730 (10Marostegui) Ping me today from now till 16:00 UTC [05:19:50] 10DBA, 10Wikimedia-Site-requests: Global rename of Tarawa1943 → Ontzak: supervision needed - https://phabricator.wikimedia.org/T206730 (101997kB) Can we start? [05:20:42] 10DBA, 10Wikimedia-Site-requests: Global rename of Tarawa1943 → Ontzak: supervision needed - https://phabricator.wikimedia.org/T206730 (10Marostegui) Yes, please paste the rename progress URL once you have it [05:22:13] 10DBA, 10Wikimedia-Site-requests: Global rename of Tarawa1943 → Ontzak: supervision needed - https://phabricator.wikimedia.org/T206730 (101997kB) https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Ontzak [05:24:25] banyek|away: not sure why you said: "I am pretty sure they wont page" because even if you ran puppet on icinga, the notifications didn't get disabled ;-). They won't page because they are downtimed, not because notifications got disabled after you ran it [05:39:31] 10DBA, 10Wikimedia-Site-requests: Global rename of Tarawa1943 → Ontzak: supervision needed - https://phabricator.wikimedia.org/T206730 (101997kB) Rename successfully completed. Thanks. [05:40:06] 10DBA, 10Wikimedia-Site-requests: Global rename of Tarawa1943 → Ontzak: supervision needed - https://phabricator.wikimedia.org/T206730 (101997kB) 05Open>03Resolved [07:45:59] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [07:46:08] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) 05Open>03Resolved [07:46:13] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [08:09:59] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Marostegui) As a first approach and in order to advance some work, we can probably compare sanitarium masters with the master (or other slaves) and leave sanitarium themselves aside for now. At lea... [08:28:04] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10JAllemandou) [08:31:43] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10JAllemandou) ping @Bstorm for the above comment, as she's the one having setup views. Thanks [08:34:49] marosteguiL: that was the reason. I downtimed the hosts, manually, so I was pretty sure that those pages will spare us [08:35:15] then I don't get the relation of running puppet too :p [08:39:52] puppet tries to start pt-heartbeat [08:40:04] it can't start it, because if the lack of running database [08:40:18] What I am saying is that you mentioned running puppet on icinga [08:40:21] Anyways, not a big deal [08:49:44] 10DBA: Duplicate rows error in db2095 replication @s7 - https://phabricator.wikimedia.org/T208672 (10Banyek) sorry I didn't noted it here: yes, the process finished, and found no problems [08:58:33] marostegui, banyek: fyi the maintenance script i was running yesterday completed. however, it completed late in my working day so i didn't have time to write up my notes [08:58:37] thanks for your help :) [08:58:44] phuedx: thanks [09:04:47] 10DBA: Duplicate rows error in db2095 replication @s7 - https://phabricator.wikimedia.org/T208672 (10Marostegui) >>! In T208672#4741499, @Banyek wrote: > sorry I didn't noted it here: yes, the process finished, and found no problems Can this be resolved then? As the follow-ups will happen at {T209048} [09:21:05] 10DBA: Duplicate rows error in db2095 replication @s7 - https://phabricator.wikimedia.org/T208672 (10Banyek) 05Open>03Resolved a:03Banyek [09:33:32] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10daniel) >>! In T209031#4732318, @Nuria wrote: > @Krenair: we are looking how to best import the public dataset from labs, we h... [09:43:16] 10DBA, 10Operations: BBU Fail on dbstore2002 - https://phabricator.wikimedia.org/T208320 (10ArielGlenn) p:05Triage>03Normal [09:46:59] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10Krenair) >>! In T209031#4740771, @JAllemandou wrote: > - **Access to underlying tables** - We could query the underlying tab... [09:48:12] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) I started the comparison on s5 between db1070 and db1082 with ` for db in $(cat mediawiki-config/dblists/s5.dblist); do ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1070 db... [09:49:46] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Marostegui) a:03Banyek Assigning it to you to reflect the current status [09:56:25] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) s5 completed, going forward [09:56:38] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) [09:58:13] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) `for db in $(cat mediawiki-config/dblists/s6.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1061 db1085; done | tee s6_check.out` [09:58:58] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) The screen name in sanitarium_check and runs on cumin1001 [10:06:38] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10JAllemandou) >>! In T209031#4741603, @Krenair wrote: >>>! In T209031#4740771, @JAllemandou wrote: >> - **Access to underlyin... [10:11:13] banyek: I wil wait for your config patch before merging this: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/473176/ [10:11:14] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) [10:12:36] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) s6 ok, continuing with s7 `for db in $(cat mediawiki-config/dblists/s7.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1062 db1079;... [10:25:05] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) s7 completed, s8: `for db in $(cat mediawiki-config/dblists/s8.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1071 db1087; done | t... [10:25:05] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) [10:54:07] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) s8 completed, s2: `for db in $(cat mediawiki-config/dblists/s2.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1066 db1074; done |... [10:54:24] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) [11:20:26] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata, and 3 others: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) >>! In T203709#4697081, @Ladsgroup wrote: > It's not done in s8 eqiad causing this {T207313} on Wi... [12:09:48] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10daniel) >>! In T209031#4741603, @Krenair wrote: >>>! In T209031#4741574, @daniel wrote: >> In any case, I'm wondering: doen't... [12:26:19] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) [12:27:28] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) s2 done, s4: `for db in $(cat mediawiki-config/dblists/s4.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1068 db1121; done | tee s4... [12:32:35] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) [12:34:28] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) s4 done, s3: `for db in $(cat mediawiki-config/dblists/s3.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1075 db1077; done | tee s3... [13:18:42] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) s3 finished, one difference found: `Checking database tcywiki Starting comparison between id 1 and 226 DIFFERENCE on db1077.eqiad.wmnet:3306: WHERE ar_id BETWEEN 1 AND 226 Execution ended... [13:18:53] ^^^ s3 tcywiki has to be fixed [13:19:11] now I start on s1 and do the fix in the meantime [13:36:58] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10JAllemandou) @daniel : We were using sanitized comment view (named `comment`), not the compat one (named `revision_compat`). W... [13:53:31] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10daniel) >>! In T209031#4742129, @JAllemandou wrote: > @daniel : We were using sanitized comment view (named `comment`), not th... [13:54:59] 10DBA, 10MediaWiki-extensions-WikibaseMediaInfo, 10SDC Engineering, 10StructuredDataOnCommons, 10Wikidata: MediaInfo extension should not use the wb_terms table - https://phabricator.wikimedia.org/T208330 (10Addshore) >>! In T208330#4738627, @Marostegui wrote: > I would prefer if we don't enable more stu... [13:55:53] thanks for the +1 [13:56:03] I brew a coffee and merge it [13:56:38] 10DBA, 10MediaWiki-extensions-WikibaseMediaInfo, 10SDC Engineering, 10StructuredDataOnCommons, 10Wikidata: MediaInfo extension should not use the wb_terms table - https://phabricator.wikimedia.org/T208330 (10Marostegui) >>! In T208330#4742192, @Addshore wrote: >>>! In T208330#4738627, @Marostegui wrote:... [13:59:35] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) [13:59:52] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) pc2007 is now in production replacing pc2004 [14:00:18] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) [14:00:50] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) a:05Banyek>03Marostegui Assigning this to myself to reflect the current status [14:11:33] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata, and 3 others: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) [14:28:58] marostegui: I fix the tcywiki now [14:29:07] great [14:29:19] what was the issue? [14:29:45] have you checked if that drift exists on other hosts in eqiad or codfw? [14:36:34] cood call [14:36:40] I didn't yet [14:37:04] bit I do it before I start [14:41:41] ok [14:41:52] hm [14:41:54] "good" news [14:42:11] all the hosts seems to be the same except the master (db1077) [14:42:24] what about codfw? [14:42:36] that applies to codfw to [14:42:37] too [14:43:04] so what was the drift? [14:43:06] sorry db1075 is the master [14:43:37] can't say it, because of ಬೆದ್_ರ್_ರಾಜನ್_ಪೋಪುನು [14:43:54] I mean, it was a different row or a row that was not present somewhere? [14:44:28] with pager md5sum: [14:44:36] select * from archive where ar_id=72; [14:44:43] 781f61fe4c757a4e6227fb9d31e172c5 [14:44:50] 037b068b6717621bd4ecf280a7b949d7 [14:44:55] right, so it is not a missing row [14:45:13] so if all the hosts are the same I suggest you fix the master WITHOUT replication [15:06:21] 10DBA, 10MediaWiki-API, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 3 others: ApiQueryExtLinksUsage::run query has crazy limit - https://phabricator.wikimedia.org/T59176 (10Anomie) [15:06:39] 10DBA, 10MediaWiki-API, 10MediaWiki-Database, 10MW-1.32-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 2 others: ApiQueryExtLinksUsage::run query has crazy limit - https://phabricator.wikimedia.org/T59176 (10Anomie) [15:17:37] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Marostegui) >>! In T209048#4742064, @Banyek wrote: > s3 finished, one difference found: > > `Checking database tcywiki > Starting comparison between id 1 and 226 > DIFFERENCE on db1077.eqiad.wmne... [15:18:56] 10DBA, 10Operations: BBU Fail on dbstore2002 - https://phabricator.wikimedia.org/T208320 (10Marostegui) I just restored the original flags to sync_binlog=1 and trx_commit=1 as s3 caught up. [15:19:12] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Marostegui) [15:19:30] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Wikidata, and 3 others: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 (10Marostegui) [15:19:45] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) in the meanwhile the check finished on s1 too, and found no differences [15:19:58] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) [15:21:58] 10DBA, 10User-Banyek: Checking archive tables across the databases - https://phabricator.wikimedia.org/T209048 (10Banyek) 05Open>03Resolved tldr: all the sections were checked only one differences was found in s3 (tcywiki) and @Marostegui fixed it too. [17:52:00] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10Bstorm) > - **Specialized views** - Views for comments from each of revision, archive, and logging, separately. We have t... [18:02:49] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10Anomie) >>! In T209031#4743469, @Bstorm wrote: > Materialized views in mariadb requires a plugin that basically is adding some... [18:04:34] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10Bstorm) Note: I'm not done reading back yet--but yeah, that's what I was thinking of. There's a lot here. [19:52:37] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10JAllemandou) Thanks again a lot @Anomie, @daniel, @Bstorm for having chimed in, your questions and suggestions have helped us... [20:01:47] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10chasemp) I don't want to muddy the waters as I have not been involved here :). But worth noting there is more than the views a... [20:12:41] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10JAllemandou) Thanks @chasemp for raising the point. We are aware of the 2 steps for sanitization. For us, code for sanitariu... [20:48:16] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10chasemp) Makes sense, again sorry for the drive by comment. Let's me know if I can be helpful :)