lightweight database migration tools with python

in daily work, it's common to need to migrate data between different databases. here are some simple methods to consider:

copy data between databases

kettle's table copy wizard

previously wrote a blog post about this: a simple guide to using kettle for database migrations.

use csv as intermediary

requires time to process field data types and ensure data consistency.

utilize sqlalchemy

wrote a blog post about this too: a step-by-step guide to using sqlalchemy for database migrations. the process involves creating models and manually mapping field types.

step-by-step database migration

assuming you need to migrate the emp_master table from sql server to sqlite, follow these steps:

create the target database schema

use sqlacodegen to generate sqlalchemy models based on the source database:

sqlacodegen mssql+pymssql://user:pwd@localhost:1433/testdb > models.py --tables emp_master

adjust the generated code manually to match your needs:

# models.pyfrom sqlalchemy import Column, Integer, Stringfrom sqlalchemy.ext.declarative import declarative_baseBase = declarative_base()class EmpMaster(Base):    __tablename__ = 'emp_master'    emp_id = Column(Integer, primary_key=True)    gender = Column(String(10))    age = Column(Integer)    email = Column(String(50))    phone_nr = Column(String(20))    education = Column(String(20))    marital_stat = Column(String(20))    nr_of_children = Column(Integer)

create the database and table using sqlalchemy:

# create_schema.pyfrom sqlalchemy import create_enginefrom models import Baseengine = create_engine('sqlite:///employees.db')Base.metadata.create_all(engine)

migrate data using pandas

read data from source database to a pandas dataframe and write it to the target database:

# data_migrate.pyfrom sqlalchemy import create_engineimport pandas as pdsource_engine = create_engine('mssql+pymssql://user:pwd@localhost:1433/testdb')target_engine = create_engine('sqlite:///employees.db')df = pd.read_sql('emp_master', source_engine)df.to_sql('emp_master', target_engine, index=False, if_exists='replace')

advantages of using pandas for data migration

pandas provides a convenient way to handle data transformation and export to various database formats. its read_sql() function simplifies data extraction from databases, while to_sql() handles the insertion process.

why choose pandas for database migration

pandas is lightweight and efficient for data migration tasks. it allows for quick data visualization and manipulation before storage in the target database.