Data migration from Azure CosmosDB to Atlas Cluster
# mongodump from Azure Cosmos
STG_CONNECTION_STRING="mongodb://global-cosmos-mongo-stg:nIY5ZkhxozGEjuuuj0OnfSz4rQJ9yV4PauFtA4Foak5d9L9BqY6hHnTuG1QDIRhClnbb9HQazCISjdKBFdCrlA==@global-cosmos-mongo-stg.mongo.cosmos.azure.com:10255/?ssl=true&replicaSet=globaldb&retrywrites=false&maxIdleTimeMS=120000&appName=@global-cosmos-mongo-stg@"
mongodump --ssl --uri="$STG_CONNECTION_STRING" -o ~/Workspace/Mo2C/data/stage
# mongorestore into Atlas
mongorestore "mongodb+srv://global-mongo-stage:sGB7SyjBSU1zc50H@cluster0.bodh8.mongodb.net/iam?retryWrites=true&w=majority" ~/Workspace/Mo2C/data/stage
# mongodump from Atlas
mongodump --uri="mongodb+srv://global-mongo-stage:sGB7SyjBSU1zc50H@cluster0.bodh8.mongodb.net/iam" -o ~/Downloads
Problems
- Global
Writes
Cluster requires M30+ tier Shard Compute, and it’s very expensive. - Destroy a Cluster taking very long time.
- CosmosDB/Mongo has some indexes not compitable with Atlas Mongo. Key starting with
$
or containing wildcard*
- See details: Wildcard indexes
- Atlas ConnectionString has variety DNS issues. Seeing from local and also inside K8s clusters, see below:
This will require extra
DevOps
effort to solve the DNS across all evnironments and personal work stations. DNS resolver needs to have public DNS resolver like Google’s: 8.8.8.8 primary and 8.8.4.4 secondary.- K8s pod error:
```bash
mongorestore “mongodb+srv://global-mongo-stage:sGB7SyjBSU1zc50H@cluster0.bodh8.mongodb.net/iam?retryWrites=true&w=majority” ~/tmp
STG_CONNECTION_STRING=”mongodb://global-mongo-stage:sGB7SyjBSU1zc50H@cluster0-shard-00-00.bodh8.mongodb.net:27016,cluster0-shard-00-01.bodh8.mongodb.net:27016,cluster0-shard-00-02.bodh8.mongodb.net:27016,cluster0-shard-01-00.bodh8.mongodb.net:27016,cluster0-shard-01-01.bodh8.mongodb.net:27016,cluster0-shard-01-02.bodh8.mongodb.net:27016/iam?ssl=true&authSource=admin&retryWrites=true&w=majority” mongodump –ssl –uri=”$STG_CONNECTION_STRING” -o ~/tmp/db/
2020-12-15T18:22:08.501+0000 error parsing command line options: error parsing uri: lookup _mongodb._tcp.cluster0.bodh8.mongodb.net on 10.0.0.10:53: no such host 2020-12-15T18:22:08.501+0000 try ‘mongorestore –help’ for more information
- Solution: Update K8s coreDNS configuration: ```bash mongodb.net:53 { errors cache 30 forward . 8.8.8.8 8.8.4.4 }
- Local error:
> mongorestore "mongodb+srv://global-mongo-stage:sGB7SyjBSU1zc50H@cluster0.bodh8.mongodb.net/iam?retryWrites=true&w=majority" ~/Workspace/Mo2C/data/stage/iam 2020-12-15T18:03:26.432+0000 error parsing command line options: error parsing uri: lookup cluster0.bodh8.mongodb.net on [::1]:53: cannot unmarshal DNS message 2020-12-15T18:03:26.432+0000 try 'mongorestore --help' for more information
- Check networking
telnet cluster0-shard-00-00.bodh8.mongodb.net 27017 OR nc -w 3 -v cluster0-shard-00-00.bodh8.mongodb.net 27017
FYI,
- https://docs.mlab.com/troubleshooting-atlas-connection-issues/
- https://stackoverflow.com/questions/61820151/minikube-kubernetes-pod-cant-connect-to-mongodb-atlas
- https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/
- https://coredns.io/2017/07/23/corefile-explained/
- https://forums.docker.com/t/docker-dns-change-etc-resolv-conf-file/40617
- K8s pod error:
```bash
-
Global Writing requires a compound index, which requires a field named as
location
field which our documents don’t have yet, and another secondary field. - There are restrictions over unique compound index against shard_keys
2020-12-17T20:06:32.668+0000 Failed: iam.identity: error creating indexes for iam.identity: createIndex error: (CannotCreateIndex) { atlas-wm7ojp-shard-0/atlas-wm7ojp-shard-00-00.bodh8.mongodb.net:27017,atlas-wm7ojp-shard-00-01.bodh8.mongodb.net:27017,atlas-wm7ojp-shard-00-02.bodh8.mongodb.net:27017: "cannot create unique index over { ownerId: 1, t: 1, email: 1, uname: 1, cid: 1 } with shard key pattern { location: 1.0, ownerName: 1.0 }", atlas-wm7ojp-shard-1/atlas-wm7ojp-shard-01-00.bodh8.mongodb.net:27017,atlas-wm7ojp-shard-01-01.bodh8.mongodb.net:27017,atlas-wm7ojp-shard-01-02.bodh8.mongodb.net:27017: "cannot create unique index over { ownerId: 1, t: 1, email: 1, uname: 1, cid: 1 } with shard key pattern { location: 1.0, ownerName: 1.0 }" }