Subscribe to my newsletter EDM News
The rule of 100 and 1000
17 January 2008, by , in Data Management, 2 comments

One of the steps during the database implementation process is data conversion. Part of the data conversion process is identifying which data sets you want to convert (e.g., membership types) and which you don't (e.g., payment details for events from four years ago).

One of my simple rules of thumb for data conversion is the rule of 100 and 1000. If you have fewer than 100 records to convert for some given subset of data (e.g., committees), it is typically easier to re-key this data after the new database is set up, rather than trying to convert the data into the new system via some script.

On the other hand, if you have more than 1000 records of a given subset of data, it's typically least painful to write a conversion script and convert this data that way, rather than keying the data by hand.

So this leaves us with the "magical middle," those data subsets that have a count between 101 and 999. What do we do with these?

In most cases, if it's closer to 100 than 1000 (i.e., 250 records or fewer) it's still probably cheaper and easier to re-key the data by hand. This also provides the opportunity to clean up the data, correcting spelling or other errors that may be in the current data.

Once you start to push over 500 records, unless you think there is going to be an opportunity to clean up the data, re-keying by hand may not be the best choice. After all, re-keying actually introduces more room for human error.

As with all things in life, there are no hard and fast rules. There may be exceptions where re-keying hundreds of records makes sense for some reason. But applying the Pareto principle, using the rule of 100 and 1000 will help you quickly decide for the vast majority of data sets should be converted by hand and which by script.

About author:
  • I have never heard anyone formalize the rule of 100-1000, that’s excellent! I subconciously ask about dataset sizes, but never thought there was a rule about it.

    Garry

  • Thanks Garry. People often need rules to guide them, so here’s a rule!

Archives

Sign-up for EDM News

 

 

Testimonials

“Wes was able to come in and offer tangible, relevant advice that made us more productive immediately. I value his understanding of databases but more so, his understanding of how nonprofits work. There was no lost time educating him about how membership organizations are “different.” Wes recommended changes in processes as well as tips and tricks that were easy to implement made an immediate positive impact.”

Mary Pat Paris, Executive Director
International Registration Plan

“We came to Wes because we were very frustrated with our existing AMS and we wanted to improve our capabilities as soon as practicable. Wes very quickly helped us through a process of identifying our needs, identifying potential vendors, and selecting a new system that we’ll be able to move into very quickly. I especially appreciated Wes’s candor about our processes as well as the systems we were looking at. He was a great resource to have in a period of high anxiety for our organization. I would highly recommend Wes for any similar project.”

Jack Chiasson, CMP Executive Director
National Association of Life Brokerage Agencies

“Wes was able to come in and offer tangible, relevant advice that made us more productive immediately. I value his understanding of databases but more so, his understanding of how nonprofits work. There was no lost time educating him about how membership organizations are “different.” Wes recommended changes in processes as well as tips and tricks that were easy to implement made an immediate positive impact.”

Mary Pat Paris, Executive Director
International Registration Plan

Mary Pat Paris
International Registration Plan

“This is the second database implementation we’ve done since I have been at Western Arts Alliance (WAA). The first I did on my own. This time we engaged Wes Trochlil as our database planning consultant. Let me tell you, this process is a whole lot easier having Wes on your team! For a small association like WAA, it’s tempting for board and EDs to question the justification and expense of a database planning consultant. But it’s the small associations that need Effective Database Management the most. Wes strengthened our planning process, clarified our needs requirements, helped us steer around solutions that couldn’t meet our objectives, and saved us money in the long haul.”

Tim Wilson, Executive Director
Western Arts Alliance

Tim Wilson
Western Arts Alliance
%d bloggers like this: