Data Migration is standard practice when your organization develops a new member portal or membership site, and this can be a daunting task. Your association is going to want to keep your data in order to continue tracking reports on member engagement, member retention, and the popularity of your membership benefits, especially if your organization relies heavily on data to manage its operations and achieve its mission.
However, with the right tools and strategies in place, data migration can be a smooth and successful process. This is where CiviCRM comes in – an open-source CRM software that can help organizations migrate their data effectively and efficiently. Whether your association changes its CMS, AMS, or LMS. This blog will guide you through the steps of setting your data’s broad objectives and cleaning it up to prepare it for successful data migration
Here are some tips for achieving data migration success using CiviCRM:
Step 1: Plan Ahead
Before going deep into the data, take a step back to think of what your association’s goals are for your data. How do you want to use it? How does it benefit you? What saved reports do you want to create with it? Which dynamic (liquid) fields are you going to want to create from the data? What automation will you want from the data?
Make a list of the goals you have for your data. From this point, make sure you have the data you want to reach those goals, and that you create fields to house that data in our new data management system.
Start by asking your association’s team the following questions:
1. What data do we want to migrate?
Your team will likely have inherited data from previous years and programs that are no longer relevant. You may need to update or delete these data altogether. Your data may contain everything and anything! Let’s take a closer look.
The Must-Haves
The main content types your associations likely need to transfer for a member portal are your contacts, staff, members, membership types, events, activities, donations, user profiles, relationships, and partners. There may be other content types that are specific to your association’s needs.
2. Does future work and reports require all the data?
There will be data that is not useful to your mission and reporting. Either it’s historic, not relevant, or incomplete. We encourage you to feel comfortable with letting go of the data your association doesn’t need. Avoid the what-if mentality of thinking, “We may use it one day!!” Focus on your association’s mission, how you serve your members, and the practical applications of the data.
How do we know if we will need the data in the future or not?
Use the following framework to decide what data to keep; ask yourself:
Does the data support your association’s mission?
Does this data help you better meet the needs of your membership?
Do you see practical applications for the data that you will use in the near future?
Saving a Backup of Historic Data
You do not need to throw out the data that your association doesn’t need. It is reasonable to assuage your what-if fears by saving a backup in a location other than your association’s new CRM or database. This way you can access it if ever the need arises and it won’t clutter up your new database and reports.
Housekeeping
Before your association rushes to shuttle all your existing data into the new member portal, this is a welcome opportunity to clean up your data. You’ll want to make sure your association only carries over relevant and up-to-date data.
It’s like moving to a new home; you’ll want to go through the basement and ask yourself, “Do we really want to clutter up our new home with all the stuff we haven’t used in ten years?” Your task here would include deduplication, normalization, and other data-cleaning task.
Step 2: Getting a Closer Look at the Data Fields
Now that we know the data content types we want to move forward with, let’s look at the data fields of each content type. The fields are the titles of your columns in your data; and the categories so you can make sense of the date.
1. Identify Fields to Migrate
Look at all the fields in the data. First name, last name, email, location, donation amount, relevant tags — there are plenty of important fields. There may also be granular data that your system stores and isn’t relevant to your association’s work and mission — for example, time stamps on an event from years back or the food preferences of people who passed away.
Delete the data fields you do not need. Not sure if you should delete them? Go back to the questions above: does the field support your association’s mission? Is there a practical application for the data?
2. Identify the Field Type
Confirm the field type options for each field, i.e. is the field a radio button, checkboxes, text field, numeric value, etc? The person responsible for the data migration will need this data to make sure no errors occur.
Don’t worry about remembering all the steps! Get our Data Migration Checklist & Worksheet.
Step 3: Map Out The Fields
Go through each field to identify which fields of the old data will correspond to which field on the new platform. Not all databases are equal, and it’s common for data fields from one application not to map one-to-one to the new one. For example, one system may store data as an activity, while another system stores it as an event. Your data will still be there, just stored differently.
If we store data differently, can our association produce the same reports?
All the functionality is available in your new CRM and database (assuming you are using a powerful, detailed one such as Drupal, WordPress, and CiviCRM). You just need to find the functionality. It may not be the same steps as the previous platform, but you’ll learn the ropes on the new platform quickly. For support and tips, use the CiviCRM and Drupal forums to ask for help on how to do something.
Identify any missing fields in the new platform and create those fields.
Step 4: How are you going to do the migration?
The platform you use (Drupal, WordPress, or CiviCRM) determines how you perform the migration.
CiviCRM already has an import mechanism feature for the main content types, such as contacts, membership, activities, contributions, and participation.
Drupal and WordPress do have an import mechanism but it’s not automatic. You need coding. Drupal has a migration mechanism called Migrate API, which is the module you will use to migrate the data batch by batch. You need custom coding as well.
WordPress has several migration plugins to migrate data from old WordPress sites. Custom coding will be required as well if you want to migrate from a different CMS.
Provide ALL your Data to the Person Who Will Perform the Migration
Your files may be big. Like, really big with tens of thousands of rows. Provide all of your data to the person performing the migration to review the data.
Providing a sample of the data isn’t enough because there will always be surprises and inconsistencies hidden in the data. It’s best to identify these issues sooner rather than later.
Double and triple-check that the data files you are working with are the most up-to-date and cleaned up you can get. Changing the contents of the data files midway through can cause delays to the point of bringing you back to the beginning of the data migration planning process.
Step 5: Test Drive the Data aka The Deep Clean
1. Preliminarily Test the Migration of ALL The Data Through The Migration Process. In the planning phase, we review the fields manually to get a sense of the data. Your data, however, may be tens or hundreds of thousands of rows and it’s impossible to go through them manually. With a migration test, the Drupal, WordPress, or CiviCRM system will identify errors in the data.
Run all the data through the migration process to identify any errors in the data. Error messages and hiccups are to be expected. For example, the system will identify duplicate fields or when a piece of data doesn’t match the data type. For instance, if two organizations have been labeled as having a parent/child relationship. You’ll have the opportunity to clean up these issues and correct, add, or remove data.
2. Complete a Full Data Migration as a Test to Simulate the Launch. Now that all the errors in the data have been addressed, test the migration again. We want to see all the data transferred with zero error messages.
3. Identify the Batch Size Limitation for the Data Files
Depending on the size of your data files, it is likely that the system will not allow you to migrate the data all at once. The limitation depends on file size and number of rows. First, try with 100 rows of data. If this has no issue, increase it to 500 rows. Keep increasing gradually until the system prompts you that the size limit has been reached.
Step 6: Perform the Final Migration
When all data has been cleaned through a trial migration run, use the maximum migration batch size you identified to perform the final migration. The process will go smoothly because there will be no errors.
Final Test Drive aka Perform Quality Assurance Testing in the New Platform
After cleaning all the data through a trial migration run, use the identified maximum migration batch size for the final migration. If the results of the reports match the results in the old system, you are golden!
For good measure, spot-check a few records in the data and see if they match the old data.
Step 7: Train Your Team
Train your team: Once the data migration process is complete, it is important to train your team on how to use CiviCRM effectively. This may involve providing documentation, training sessions, and ongoing support to ensure that everyone is comfortable using the new system.
By following these tips, your organizations can migrate their data to CiviCRM with confidence, knowing that they have the right tools and strategies in place to achieve data migration success. With its robust features, flexibility, and ease of use, CiviCRM can help organizations manage their data effectively and achieve their goals more efficiently.
Need a Little Extra Help?
Book a call with us.
