The rule of 100 and 1000

One of the steps during the database implementation process is data conversion. Part of the data conversion process is identifying which data sets you want to convert (e.g., membership types) and which you don't (e.g., payment details for events from four years ago).

One of my simple rules of thumb for data conversion is the rule of 100 and 1000. If you have fewer than 100 records to convert for some given subset of data (e.g., committees), it is typically easier to re-key this data after the new database is set up, rather than trying to convert the data into the new system via some script.

On the other hand, if you have more than 1000 records of a given subset of data, it's typically least painful to write a conversion script and convert this data that way, rather than keying the data by hand.

So this leaves us with the "magical middle," those data subsets that have a count between 101 and 999. What do we do with these?

In most cases, if it's closer to 100 than 1000 (i.e., 250 records or fewer) it's still probably cheaper and easier to re-key the data by hand. This also provides the opportunity to clean up the data, correcting spelling or other errors that may be in the current data.

Once you start to push over 500 records, unless you think there is going to be an opportunity to clean up the data, re-keying by hand may not be the best choice. After all, re-keying actually introduces more room for human error.

As with all things in life, there are no hard and fast rules. There may be exceptions where re-keying hundreds of records makes sense for some reason. But applying the Pareto principle, using the rule of 100 and 1000 will help you quickly decide for the vast majority of data sets should be converted by hand and which by script.

About Wes Trochlil

For over 30 years, Wes has worked in and with dozens of associations and membership organizations throughout the US, ranging in size from zero staff (all-volunteer) to over 700. In that time Wes has provided a range of consulting services, from general consulting on data management issues to full-scale, association-wide selection and implementation of association management systems.

Scroll to Top