[00:42:53] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10bd808) >>! In T202889#4746346, @aborrero wrote: > Also, @bd808 what do you think? >>! In T202889#4746655, @Marostegui wrote: > If I had to choose, I would move thi... [00:54:25] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10bd808) As noted in T202889#4541255, {T167973} is open as well. When we move the labswiki database off of m5 then it would be just a bit more than the OpenStack sche... [05:59:27] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) pc2008 is now online for pc2. [05:59:40] 10DBA, 10Operations, 10Patch-For-Review, 10User-Banyek: Implement parsercache service on pc[12]0(07|08|09|10) and replace leased pc[12]00[456] - https://phabricator.wikimedia.org/T208383 (10Marostegui) [06:16:08] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) [06:19:11] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10Marostegui) >>! In T202889#4748811, @bd808 wrote: > . As far as I know the only blocker for moving labswiki's db to the main wiki db cluster is DBA time. I think t... [06:25:33] 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Marostegui) >>! In T200297#4745149, @awight wrote: > @Marostegui Pinging for review of these two files, h... [07:06:44] 10DBA, 10MediaWiki-API, 10MediaWiki-Database: prop=revisions API timing out for a specific user and pages they edited - https://phabricator.wikimedia.org/T197486 (10Marostegui) >>! In T197486#4657581, @Marostegui wrote: > As discussed via email, this bug was fixed on 10.1.37: https://jira.mariadb.org/browse/... [07:38:57] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) [07:42:44] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) [08:02:07] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) [08:04:34] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) s5 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1124 [] db1113 [] db1110 [] db1102 [] db1100 [] db1097 [] db1096 [... [08:04:52] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) [08:04:53] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Dropping site_stats.ss_total_views on wmf databases - https://phabricator.wikimedia.org/T86339 (10Marostegui) [09:14:46] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [09:17:53] 10DBA, 10Cloud-Services, 10User-Banyek: Prepare and check storage layer for shnwiki - https://phabricator.wikimedia.org/T206916 (10Banyek) a:03Banyek doing sanitization [09:18:13] 10DBA, 10Cloud-Services, 10User-Banyek: Prepare and check storage layer for yuewiktionary - https://phabricator.wikimedia.org/T205714 (10Banyek) a:03Banyek doing sanitization [09:18:21] 10DBA, 10Data-Services, 10User-Banyek: Prepare and check storage layer for punjabiwikimedia - https://phabricator.wikimedia.org/T207584 (10Banyek) a:03Banyek doing sanitization [09:18:30] 10DBA, 10Cloud-Services, 10User-Banyek: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Banyek) a:03Banyek doing sanitization [09:24:36] I checked all the new wikis are in s3 (as expected.) [09:25:07] do they have existing users? [09:26:01] checking [09:26:05] does it matter? [09:26:26] well, how can you check if the sanitization was properly done? [09:26:44] my plan was to run the 'check_private_data.py' [09:27:07] how can you check if _new_ things will be sanitized? [09:28:21] check if there's the triggers exists - iirc the check_private_data.py will yell if not [09:28:33] not, it won't yell [09:28:42] why not checking the data itself on mysql to be double sure? [09:30:13] I checked now and except pujnjabiwikimedia there are users for sure [09:30:29] there were some fishbowl ones, no? [09:31:19] only one, the punjabiwikimedia [09:31:36] so before handling that one to cloud, ask them to create an user so you can check if it sanitized correctly [09:31:50] my notes ending with [09:31:51] for the other ones, create one yourself and check if your user is sanitized finely [09:31:54] ```That is all for the public ones [09:31:54] We can go thru the privates once we have a private wiki [09:31:54] ``` [09:32:02] I don't know if that applies [09:32:07] but we don't have a private one, only fishbowl [09:32:17] no, private ones do not get replicated at all [09:32:20] is there any private one? [09:32:30] no, 3 public, one fishbowl [09:32:35] cool [09:32:50] so for the 3 public use your user to login and then check it the sanitization works, if so, handle it to cloud for the views [09:32:56] for the fishbowl, ask them to create one [09:33:19] 👍 thanks [09:37:51] if you are unsure about anyting ask me, better to be sure everything is correctly sanitized before creating the views [10:00:05] ity will be fun to create an user here for checking the triggers; https://yue.wiktionary.org/wiki/%E9%A0%AD%E7%89%88 [10:00:24] you just need to log-in with your user [10:00:33] is that the fishbowl one? [10:00:45] no, that's the punjabi.wikimedia.org [10:01:00] I just logged in [10:01:03] check my user [10:01:36] ok, works [10:01:43] we are both there with no data [10:01:54] I check the others [10:02:05] cool [10:02:58] I am not sure yue is correctly sanitized [10:03:37] it has no triggers [10:03:50] and the data is there [10:04:08] because I did it in codfw first as a dry-check [10:04:10] check db2094 [10:04:15] ah ok [10:04:16] let me check [10:04:42] that looks good [10:04:47] please !log when you run it [10:04:57] even for codfw, so we have traces [10:06:52] ok [10:07:12] I logged in all three and they're seem good, I do the same on eqiad too [10:10:02] cool [10:17:06] https://phabricator.wikimedia.org/tag/blocked-on-schema-change/ update the dashboard when you can please [10:29:58] ok [10:30:19] I finished the sanitizing the wikis [10:30:48] \o/ [10:30:59] if you agree I'd re-assing the yuewiktionary, liwikinews, and shnwiki to cloud people (for the views) and ask for a user in punjabiwikimedia [10:31:08] +1000 [10:31:16] move them into "Done" for us [10:31:23] * banyek winning dance [10:34:43] I need to ask someone for logging into those to see the filters actually work [10:35:07] for the fishbowl one you mean? [10:35:10] no [10:35:32] what do you mean then? [10:37:03] I logged in all three when did db2094 and checked if the filters work. They worked. But I was already logged in so after I did the sanitizing on db1124 I was not able to log in again to check if the new rows will be filtered too. [10:37:26] I can loging [10:37:34] Where do you need me to log-in? [10:37:44] li.wikinews.org [10:37:46] And better to check on the labs hosts itself, in the end, those are the important ones [10:38:02] done [10:38:07] shn.wikiemdia.org [10:38:55] shn.wikimedia.org [10:39:01] that doesn't work [10:39:13] I assume shn.wikipedia? [10:39:21] shn.wikipedia.org sorry [10:39:33] done [10:39:33] I see you [10:39:37] ok [10:39:46] any other one? [10:39:54] I have done liwikinews and shn [10:39:58] it was the yuewiktionary [10:40:10] but your user exists there [10:40:17] let me use my other user [10:40:42] gr8 [10:40:56] done [10:41:07] perfect, thanks [10:41:09] seems good [10:41:47] I'll do a quick check on the labsdb and then administer in the phabricator [10:42:02] yeah, always checks labs [10:42:09] as that's where the views will go [10:42:54] normally you can directly check labs and not check sanitariums really [10:43:05] if labs works, it means the whole chain works [10:43:10] so no need to check sanitarium AND labs [10:44:32] 👍 [10:44:37] checking labs is done [10:44:40] it's good [10:44:46] good [10:44:52] * banyek goes and does some phabricator tasking [10:50:49] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Banyek) I'll work with you on this [10:51:22] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Marostegui) Take the opportunity to upgrade MySQL too! Thanks [11:02:15] 10DBA, 10Cloud-Services, 10User-Banyek: Prepare and check storage layer for shnwiki - https://phabricator.wikimedia.org/T206916 (10Banyek) a:05Banyek>03None ready for the views creation [11:09:14] 10DBA, 10Cloud-Services, 10User-Banyek: Prepare and check storage layer for yuewiktionary - https://phabricator.wikimedia.org/T205714 (10Banyek) a:05Banyek>03None ready for the views creation [11:10:27] 10DBA, 10Cloud-Services, 10User-Banyek: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Banyek) a:05Banyek>03None ready for the views creation [11:12:01] 10DBA, 10Data-Services, 10User-Banyek: Prepare and check storage layer for punjabiwikimedia - https://phabricator.wikimedia.org/T207584 (10Banyek) The sanitization is done, but before I route this to the cloud team we need to check if the triggers work. I need for somebody to log in and ping me to check if t... [11:12:09] brb 10min [11:18:02] 10DBA, 10foundation.wikimedia.org: Drop the petition_data table from production - https://phabricator.wikimedia.org/T208979 (10Marostegui) For the record, this table is only present on s3 for the following wikis: ` foundationwiki testwiki ` [11:18:47] 10DBA, 10foundation.wikimedia.org: Drop the petition_data table from production - https://phabricator.wikimedia.org/T208979 (10Marostegui) ` root@db1075.eqiad.wmnet[foundationwiki]> select count(*) from testwiki.petition_data; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.00... [11:28:18] 10DBA, 10Cloud-Services: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Banyek) [11:28:35] 10DBA, 10Cloud-Services: Prepare and check storage layer for yuewiktionary - https://phabricator.wikimedia.org/T205714 (10Banyek) [11:29:12] 10DBA, 10Cloud-Services: Prepare and check storage layer for shnwiki - https://phabricator.wikimedia.org/T206916 (10Banyek) [11:31:16] Manuel: I'll leave soon for about an hour [11:31:29] if you give me a +1 I'll continue with this after I am back: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/473546/ [11:31:33] ok, please, triage the tickets we said [11:36:37] T209480 -> it would be nice to follow up with arturo, is this still happening, what are the next steps, is there anything required from us etc [11:36:37] T209480: labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 [11:37:03] ok [11:37:15] I have to leave now, I'll pick up here when I am return [11:37:21] bye [11:37:31] ok [11:37:32] bye [11:48:20] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10aborrero) Would next **Monday 13:00 UTC 19th Nov** work for a couple of reboots? Let's say `labsdb1010.eqiad.wmnet` and `labsdb1011.eqiad.wmnet` since these weren't mentioned... [12:16:49] 10DBA, 10cloud-services-team (Kanban): labnet1001/labstore1004 combined alert on 2018-11-14 - https://phabricator.wikimedia.org/T209480 (10aborrero) 05Open>03Resolved a:03aborrero We can move the conversation to the other tasks. [12:57:38] 10DBA, 10Cloud-Services: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Marostegui) @Banyek did GRANT this wiki into the labsdb user role grants? If not, the views will fail to be created [12:57:47] 10DBA, 10Cloud-Services: Prepare and check storage layer for yuewiktionary - https://phabricator.wikimedia.org/T205714 (10Marostegui) @Banyek did GRANT this wiki into the labsdb user role grants? If not, the views will fail to be created [12:58:00] 10DBA, 10Cloud-Services: Prepare and check storage layer for shnwiki - https://phabricator.wikimedia.org/T206916 (10Marostegui) @Banyek did GRANT this wiki into the labsdb user role grants? If not, the views will fail to be created [13:24:45] back [13:25:14] I have sent a patch to make the parsercache regex a bit more readable and concrete, check it [13:39:29] on muting icinga replication lag check in codfw do we have more comfortable way to do than downtime the hosts one-by-one? [13:40:10] actually nm [13:40:20] it wont cause too much lag, I'll !log it [13:41:32] you can also mute the whole host for like 300 seconds [13:41:40] with icinga-downtime [13:42:57] the host is not a problem, but I'll run it on codfw master, so if I mute it, I need to mute all hosts [13:43:28] but the test revealad there won't be too much lag, imo we'll be good [13:43:32] I was saying the whole host because that way you can use icinga-downtime [13:44:45] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10akosiaris) > @akosiaris labsdb1006/7 are involved -- and if we do this with a failover, I want to make sure the additional tables for T201544 are replicated? Can we confirm t... [13:48:28] 10DBA, 10Data-Services, 10User-Banyek, 10User-Urbanecm: Prepare and check storage layer for punjabiwikimedia - https://phabricator.wikimedia.org/T207584 (10Banyek) a:05Banyek>03Urbanecm [13:50:25] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) The tests were successful. Now I am running this schema change against the s6 codfw master with replication enabled. Th... [14:04:03] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) replication broke on db2046 as the alter command was `DROP COLUMN` instead of `DROP COLUMN IF EXISTS` [14:40:09] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Banyek) >>! In T209517#4749730, @aborrero wrote: > Would next **Monday 13:00 UTC 19th Nov** work for a couple of reboots? Let's say `labsdb1010.eqiad.wmnet` and `labsdb1011.eq... [14:42:50] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) Replication fixed as adding back the column manually, and restart replication [14:43:11] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [14:43:14] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Anomie) [14:43:42] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Anomie) [14:43:46] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Anomie) [15:11:00] 10DBA, 10Cloud-Services: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Bstorm) That and create the _p db. Silly bugs. [15:34:38] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Bstorm) @aborrero I'd say it's worthy of notifying users for toolsdb/wikilabels (labsdb1004/5) and possibly osmdb (labsdb1006/7) masters but not the wiki replicas or the secon... [15:36:10] 10DBA, 10Cloud-Services: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Banyek) >>! In T205713#4749900, @Marostegui wrote: > @Banyek did GRANT this wiki into the labsdb user role grants? If not, the views will fail to be created No I did not. I'll look after [15:36:56] I leave now for the kids, bbl [15:45:16] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui) Why did db2095:3316 break? [15:48:22] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Jdforrester-WMF) [15:48:43] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Jdforrester-WMF) (`1.33.0-wmf.5` won't ever get cut, as there's no train next week.) [15:48:57] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Jdforrester-WMF) [15:48:59] 10DBA: Drop table image_comment_temp on all wikis - https://phabricator.wikimedia.org/T209591 (10Jdforrester-WMF) 05Open>03stalled [16:43:14] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Bstorm) [16:43:28] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) >>! In T85757#4750443, @Marostegui wrote: > Why did db2095:3316 break? The reason was the same (the row was nonexistent... [16:44:14] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Bstorm) So far it looks like replication is picking up where it left off nicely on labsdb1007 (done). [16:44:59] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Banyek) @Bstorm sure, I can do it all, I can do the OS on those hosts [17:01:11] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10aborrero) email announcement has been sent for labsdb1006 reboot **next Tuesday 2018-11-20 at 17:30 UTC**. [17:03:12] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10aborrero) [17:11:27] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10awight) Due to T209604, the #wikilabels web service needs to be manually restarted whenever labsdb1004 (pgsql.eqiad.wmnet) is rebooted. Please ping me here when the schedule... [17:20:45] 10DBA, 10Cloud-Services: Prepare and check storage layer for liwikinews - https://phabricator.wikimedia.org/T205713 (10Banyek) @Marostegui I found T178128 but it's still not clear how to GRANT this wiki (well new wikis) Could you show/tell me some more information? [17:43:22] well, I call this a week [17:43:26] see you on monday [20:40:50] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10Bstorm) Thanks @awight. Does 11/20 @ 17:15 UTC sound good? I can work on that reboot while @aborrero does 1006. @banyek, will you be around for mysql upgrades or whatever?... [21:19:05] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10Milimetric) It seems filtered_tables.txt are not updated to work with the new columns, I asked about it here: rOPUP188a50fa82a... [21:40:51] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Upgrade/reboot labsdb* servers - https://phabricator.wikimedia.org/T209517 (10awight) >>! In T209517#4751877, @Bstorm wrote: > Thanks @awight. Does 11/20 @ 17:15 UTC sound good? I can work on that reboot while @aborrero does 1006. That would be great... [22:04:18] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10bd808) >>! In T209031#4751986, @Milimetric wrote: > - Step 1: use filtered_tables.txt to generate sqoop selects, selecting a... [22:16:12] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10Milimetric) >>! In T209031#4752128, @bd808 wrote: >>>! In T209031#4751986, @Milimetric wrote: >> - Step 1: use filtered_tabl... [22:20:22] 10DBA, 10Cloud-Services, 10Cloud-VPS, 10Patch-For-Review: Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 - https://phabricator.wikimedia.org/T140452 (10bd808) [23:03:38] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services: Not able to scoop comment table in labs for mediawiki reconstruction process - https://phabricator.wikimedia.org/T209031 (10Bstorm) I'm aiming to write tests for this script shortly because it is too complex to not have them. Overloading the scripts...