One of the questions I hear often is, “Where can I get data?” I wish I heard it a lot more often.
The question means different things to different people. Some are on a quest for information that will drive business decisions. Others want practice to develop technical skills. Still others are interested in furthering social causes or understanding science. While some need the kind of detailed data that fuels statistical analysis, many are better off if they can find a source that has already done some of the data analysis for them, providing reports, data in aggregate form or even just specific facts.
Nearly all can obtain useful data to help meet their goals. Loads of data is available today, both privately within businesses, and through public sources. A little effort can yield a wealth of information. What worries me is knowing that many people who ought to be looking for data aren’t. They’re making decisions based on just personal opinions, or something in the news, or using some data, but neglecting data types or sources that would add value for them. What a waste.
The key to getting the data you need is to have well-defined goals and a clear sense of purpose. The better than you can define what information you need and what you’re going to do with it, the more easily you will be able to locate appropriate resources.
Use these four major types of data sources to guide you to the best data resources you can obtain.
Internal
Internal data sources, the information that your organization already has, are always the first resources to consider. Here you can find data that’s detailed, uniquely relevant to your organization, and unavailable to your competitors.
But getting data from internal sources isn’t always easy. You’ll have to figure out which functional areas collect and maintain the data, how to get access and what uses are permitted. That’s where your groundwork, defining exactly what you’re seeking and why, becomes very important. You may need to take additional steps, from making formal requests to obtaining permission from management, and your success will depend on having specific goals and a clear business case.
Resist the urge to work around the proper channels. Your IT department’s rules may sometimes seem like a pain in the neck, but they’re there to make sure that your organization stays within the law, among other things.
Government and nonprofit
If you must go outside your own organization for data, make every effort to get what you need from a government or nonprofit source. Every government agency collects data, and each has legal obligations to share at least some of it. An unbelievable amount of this information is as near as your computer, your phone or your library, so take full advantage of it.
Government agency data may be transactional, that is, a record of government activity, or statistical, which is data collected specifically for analysis purposes. Property transfers and voting records are transactional data. The census is statistical data, as is the Consumer Price Index. While transactional data is usually available in detailed form only (records of individual transactions), statistical data is normally aggregated to protect privacy of individuals.
Some agencies are more helpful about sharing data than others, but you’ve got to focus on finding the right agency first. Need data about average Americans? Probably the US Census Bureau. Need data about hog belly prices? Maybe the
Many nonprofit organizations are good sources of data in their specialty areas. For business information, investigate relevant trade associations. A great resource for finding these (there are thousands of them in the US alone) is the Encyclopedia of Associations, which is available at most public and university libraries. You can also use it to find organizations devoted to social causes and research. Keep in mind that these organizations usually share information in report form, and many don’t think of what they offer as data, so phrase your request with care.
Don’t use data that you find online if the source is not clear and appropriate. Datasets floating around the internet may be fine for data analysts looking for practice, but if you’re using data for decision making, you’d better know exactly where it came from.
Commercial
If the data you need is not available to you internally, or from a government or nonprofit source, you should consider buying it. You can find some major sources listed in my article, “When and Where To Buy Consumer Data (And 12 Companies Who Sell It).” Another reason to go with a commercial source is to save costs when the effort of gathering and formatting data from government sources is significant.
But buyer beware, not all commercial data is of good quality. Ask questions about how the data is obtained and managed, and investigate some samples before you plunk down a lot of money to any vendor.
New data collection
Finally, when you can’t get data because it doesn’t exist, consider going out and collecting it. Depending on what data you need, you might conduct a survey, install sensors or send people out into the world to observe and measure things. This is usually a last resort because it can be time-consuming and expensive, but it gives you the chance to get data that is tailored to your own needs, and keep it to yourself.
