Workflow Management#
A key part of being a successful data scientist is keeping track of all the data that goes into a project, the various manipulations that are performed on that data, and the final results. Having a well-thought-out and documented workflow is important for ensuring your project is intelligible to you, to your colleagues, and to any future researchers who may wish to revisit your work. Moreover, a well-designed workflow also institutionalizes practices that protect against errors, like corrupting your source data or relying on datasets whose provenance cannot be determined.
This reading introduces a set of principles and practices I use in my own work. There are many different approaches to workflow management, but any successful workflow should respect a few principles:
Establish a clear provenance for all results and data.
Protect source data from corruption.
Allow for easy replicability by a third party.
With that in mind, here is the workflow that I use to achieve these goals. Over time, you will surely develop different strategies that work for you. Regardless of what specific choices you make, however, be mindful of these three principles.
Folder Structures#
My workflow is primarily organized around a folder structure. I start any project I do with five folders:
00_source_data: Raw inputs to this project, taken from other sources. Data in this folder should never be modified by code in this project.10_code: All code for analysis.20_intermediate_data: Data files created by code in this repository (presumably code that processes data found in00_source_data).30_results: Tables, figures, maps, etc. that will later be used in papers.40_docs: The actual writeup of the analysis, which dynamically pulls in tables and figures from30_resultswhen my.texor.typfile is compiled.
The core idea of this folder structure is that everything flows from the 00_source_data folder into 20_intermediate_data and 30_results as you run the code in 10_code.
00_source_data only contains the data received from third parties (or raw data collected in the field, etc.), and everything in that folder is associated with a .txt file that says where it came from. This is where data from outside the project enters the project pipeline.
10_code contains all the code for the project. All of it. Indeed, if you’ve done everything right, then you should be able to:
Delete everything in your
20_intermediate_datafolder and everything in your30_resultsfolder,Re-run all the code in your
10_codefolder (which starts by reading in data from00_source_data)
And have all the results you want for your paper reappear in your 30_results folder.
And if not all your results reappear (i.e. aren’t generated when you run your code), you know that you have some files in there that aren’t actually being created by code you currently have in the project (meaning those results aren’t reproducible).
Never Edit Source Data#
One idea crucial to this workflow is that you NEVER modify data in the 00_source_data directory and save it back into the 00_source_data directory. The direction of travel for data should always be from 00_source_data into 20_intermediate_data, and never back the other way. The way you “save” modifications is by keeping the code that makes the modifications to your source data.
Source data, in other words, is “Read Only.”
A big part of the reason for this is that if you were to open a file in 00_source_data, change it, and replace the old version of the data with the modified version, there’s no longer any way to roll those changes back, and there’s no record of the changes you’ve made. Yes, if it comes from a third party you could go re-download it (and you should if you do this by mistake!), but (a) someone needs to know a modification took place to be aware they should go get the original data again, and (b) not all source data can be easily replaced. If you get data from the field and don’t have a read-only-never-modified version of the source data, then there’s no way to roll back modifications.
Read results into paper, don’t copy#
As a reminder, you want the paper in 40_docs to read-in all results and figures on build (assuming you’re using LaTeX or Typst). Don’t try to transcribe or copy-paste results into your paper explicitly, or you’ll inevitably end up with typos and out-of-date figures in your paper.
File Numbering#
A key feature of data science workflows is that they are often sequential: code for importing data is meant to be run before code for cleaning data, and code for cleaning is meant to be run before analysis.
There are a few ways to document how files are meant to be used in a workflow. Some people, for example, like to keep a README document with directions on the order in which files should be run. Others rely on a makefile (a file that calls all code in a project) to specify the sequence in which files should be run.
Personally, I’m an advocate of prefixing files meant to be run in a specific sequence with a number. For example, a small project might have three files of the following structure:
10_import_data.py
20_clean_data.py
30_analyze_data.py
Why 10, 20, 30 and not 1, 2, 3? Because another key feature of data science is that projects never end up looking the way we expect them to when we start, and numbering by tens makes it easy to add files between existing files without having to re-number everything.
For example, suppose I realize that before cleaning my data, I need to export some summary statistics about cleanliness issues. With this file structure, I can easily add a new file in the middle of the existing files:
10_import_data.py
15_export_cleanliness_report.py
20_clean_data.py
30_analyze_data.py
Note I only use number prefixes for files that are meant to be executed sequentially. Sometimes I’ll have a file with functions that get imported into the files meant to be run sequentially — in that case, I put said file in the directory with the numbered files that import it but don’t prefix it. For example, if I had a function to verify that my unique identifier always had certain properties, I might put it in a file called verify_ids.py and put it alongside the files that import and use it.
Use Lots of Small Files#
OK, you might ask, but why do I need lots of files at all? Why not just one big “project” file? The answer is twofold.
The first reason is that it’s easier to debug small files. The only way to really know if your code works is to run it from start to finish in a clean session of whatever program you’re using (R, Stata, Python, Julia). That way you know that if the code works, it works because everything you need is in the file, and not because you manually created an important variable that’s not in the code, but is already in memory. But if your entire project is in one many-thousand-line file, running that file takes forever. Moreover, when a problem occurs in that giant file, it could have lots of sources. Working with smaller files (a couple hundred lines at most) makes it much easier to test the file quickly, and makes it easier to find the source of problems when they occur. Small files with descriptive titles also make it easier to navigate your project. That means if at some point you realize there’s a problem with your data imports (wait, why am I missing three days of data??), it’s very clear where you need to make changes.
The second reason is perhaps even more important. In my experience, we aren’t really smart enough to work with files that do more than one or two things. Your computer is more than happy to save thousands of lines of code in a single file called my_data_science_project.py. The problem is that it’s very hard for you to look at a file with thousands of lines of code and answer the question “is the code in this file accomplishing what I want this file to accomplish?”. If, by contrast, you have a smaller file that does one thing in much less code, it’s easier to evaluate that small chunk of code and decide if it accomplishes the goal set out by the file’s title (e.g., importing your data).