[portfolio](https://geonwoo.kim/en/portfolio) › Biztalk

# AWS MSK 3.8 migration

https://geonwoo.kim/en/portfolio/msk-migration

Period: 2025.08 — 2025.09

Support for production MSK 3.5 was set to end in 2025.10, so I built a new 3.8 cluster using KRaft metadata and recreated the existing topics on it.

Configuration for about 160 topics recreated automatically on a new cluster

Team: Solo

Role: Architecture design · KRaft adoption in development and production · SASL authentication · Topic and replica configuration · Message sending tests · Kafka UI monitoring

AWS MSK 3.8 · Kafka KRaft · SASL/SCRAM · AWS Secrets Manager · Kafka UI

## Choosing the authentication method

Problem: The new cluster needed server authentication added. MSK supported IAM, TLS and SASL, among others, so the choice had to weigh compatibility with the internal environment against operating complexity.

Solution: After reviewing the options, I chose SASL/SCRAM. User accounts are stored in AWS Secrets Manager. Each server is wired to Secrets Manager and reads the account from it to authenticate.

## Automating topic recreation

Problem: Every topic on the old cluster had to be recreated on the new one with the same settings. Partition counts, replicas and options differed by topic, so copying them by hand risked omissions and errors.

Solution: Exported the topic settings of the old cluster to a CSV file in one pass and reworked the replica configuration for topics that needed changes. A script reads the CSV and creates every topic on the new cluster. Topic names, partition counts, replicas and options carried over as they were, which greatly cut the time spent copying them by hand.


[Shopify store features and integrations for brands](https://geonwoo.kim/en/portfolio/lukuku-clients) · [BGMS log masking module](https://geonwoo.kim/en/portfolio/log-masking)