Senate majority vs. national debt – Getting at public data is a pain

Last weekend, my friend Ian and I were talking, and we got to the topic of trying to find data online to fact check newspapers.  As newspapers get squeezed due to readership moving online, they have less money to do in-depth research.  We hear more stories about Britney Spears and her ilk not just because of her hot pants, but because she’s cheaper to cover.  Sometimes, when newspapers put out something, we find out later that their facts weren’t exactly right. 

One of things he wanted to fact-check was the national debt, and how it related to our politicians.  I imagine he was incensed at the state of financial affairs and wanted to know who were the fools that did it.  He wanted to know what the senate majority was plotted against the national debt. 

After that, I took some time in the last couple of days to find the data, and then write a ruby script to scrape it and plot it on a javascript webpage.  Here’s the resulting graph:

The first surprising thing was how fast the national debt has grown in the last 30 years.  It’s been an exponential growth.  I think that while increased spending have happened, the interest on the debt has a significant effect on that growth. 

The second surprising thing was how much democrats dominated the era between 1930 and 1980, with a sliver of Republican majority in the 50’s.  And before that, it was dominated by the Republicans.  Apparently, political ideology shifts back and forth. 

One of the things I have to note about the graph is that the first set of Red in the early 1800’s is not the modern Republican party we know today.  It was another party of the same name, I think also called Jeffersonian-Republicans.  The gaps before and after were times when there were no Democrats or Republicans.  There were Pro-administration or Anti-administration, or Pro-Jackson, Anti-Jackson, or Whigs and others.  In addition, there weren’t 100 seats in the Senate at the very beginning, so we see some instances where Senate Majority wasn’t more than 50 seats.

Oh, and because I’m lazy, I didn’t label the axis.  The left y-axis is the # of senate seats held by the majority party.  The right y-axis is the national debt in dollars.

We can see from the graph that the explosive rise in national debt occurred in the last two or three decades.  In addition, both parties had Senate majority at the time.  Not only that, but the Senate Majority party only had a slight majority, which meant that it could tip in favor of the other party from congress to congress.

Seeing how it was exponential, I plotted it as a log-plot.  Ian quipped that “it’s terrifying that it makes sense to plot the national debt in log scale.”

You can see more details here.  Remember, a line in a log plot means exponential growth.  We can see that there are times in history that the US debt dropped or rose at a significantly rapid rate.  I was surprised to see that in the mid 1830’s, it looked like the US cleared itself of its debt.  I don’t know enough about history to know whether the US just defaulted or it paid the debt back.  The significant rises in debt seem to correlate with the major wars.  1860’s for the US Civil War, the 1915’s for WWI and 1940’s for WWII.

Now the last graph is not for the faint of heart.  It’s a graph of the rate of change of the national debt on a log scale.

Because the changes have been so dramatic in the last couple of decades, it dwarfs any changes in earlier periods on a linear scale.  Here, we can see that for most of the history of the US, the debt change fluctuated up and down.  There were periods of lots of spending, but then also periods of cutting back.  Sometimes, one party was responsible for doing that, and sometimes, it was another.  But in the recent years, we’ve just been accruing debt.  See that little dip in the late 90’s/early 2000?  That was the big savings from the Clinton era.  Note that it’s a log scale, so that a little dip up high on the scale means that it’s huge when it’s lower down.  If we had the level of debt back in the 1920’s, we’ve almost have cleared it. 

So all this has been interesting, but is this what falls under the “all that jazz” category?  What does it have to do with the web and programming?  I’ve been thinking about all the public the data that’s out there, and how to get to it.  The conclusion was that it was pretty damn hard.  There were four steps to tell this story to you.  I had to find the data, scrape it, clean it, and then graph it.  Of the four steps, the hardest part was scraping and cleaning.  It took a good 4+ hours to do it, and I’m a programmer.  Most other people that were curious enough could use excel, but last I checked, excel didn’t do data scraping on web pages.  Hello cut and paste. 

I think it should be much much easier for citizens in a country where we elect government officials to be informed and see this data for themselves.  Before, we had relied on journalists to give me the straight dope on these facts.  But, as I mentioned before, the newspapers have been in decline.  As a result there’s less budget to pay for good reporting watching the government and what it’s doing.  Beyond watching the government, I expect that people generally have questions that can be best answered by graphs of public data–and those answers aren’t just yes or no. 

As an example, another friend of mine, Matt, is single and looking for the ladies.  However, living in Columbia, MD, it’s a tough dating environment–everyone’s under 18 or over 40.  So if he could move, which counties in Maryland has the highest number of single females? 

Do any of you find that you have similar questions that can be answered by public data in graphs?

Data sources:

14 thoughts on “Senate majority vs. national debt – Getting at public data is a pain

  1. According to the data source, it’s historical debt outstanding. I don’t think it’s inflation adjusted. For me, at least, the point of the exercise was to pull the data out and play with it. I tried not to draw any conclusions, and only point out things that I noticed, because I didn’t spend time doing things like adjusting for inflation.This is a hunch, but I’m guessing the average inflation is less growth than the debt. But until we are able to get our hands on that data easily, we won’t know for sure.

  2. Great work. You’re like a combination of and The Sunlight Foundation. Public companies have to report their financials to the SEC and those data are available on Edgars and soon will be available in XBRL. Probably the most important thing we can do as citizens is to demand that the government report it’s financials in XBRL so everyone has access to the data. It was also be nice if some sort of taxonomy was developed for new laws and those too were reported that way.

  3. The federal debt dropped in the 1830s because Andrew Jackson withdrew all federal funds from the national bank, then the Second Bank of the US.No federal debt, but it resulted in the Panic of 1837!

  4. This is brilliant. Access to public data is massively important — it will help put the truth in the hands of the people. Think of the data madness that happens at the hands of politicians even with unemployment data. A journalist friend was telling me recently about how California’s effective unemployment rate is probably much higher than it currently is (greater than 10%) because the statisticians only count people currently getting paid through unemployment — which is totally erroneous since lots of people have been unemployed so long they don’t qualify for unemployment anymore. It’s easy for the public to be misled by figures that are just plain wrong. But if we can crunch the numbers ourselves, the hope is that we’ll be able to better govern ourselves too.

  5. There's no "guys."  It's just me.  Well, I'm not here to endorse one party or another.  The point of the post was just an adventure in sucking down some public data, look and comment at the interesting parts, and then complain about how it should be much easier to get at this public data, for something as simple as that. I don't seem to have made that very clear, as most of the post is about the data itself, and only at the end was it about how hard it was to get at it.  I'll make a note of just talking about one thing next time.

  6. Yeah, Garry (I’ve been spelling your name wrong) hit it on the nose. I’d like to see easier data access so we can crunch that data ourselves, and be able to see what’s going on. Yet, it’s also easy to lie with statistics and graphs, and worse yet, lie to yourself, as you only look for data that supports your beliefs, not ones that go against it.Due to different formats and what not, it’s currently a bit of a pain–and I don’t think that the Semantic Web will come any time soon. We’ll see if it gets any easier

  7. Kuddos for crunching some hard-to-find data! I am also in total agreement that most websites need an overhaul, and that the government numbers should be easier to analyse and harvest. That said though, I think you noted some of the pitfalls of this graph; too many years; party changes too much; bulk of the debt occurring in last 30 years. But there are also some pitfalls like since FDR, the parties changed platforms, and thus the R’s were the liberals and the D’s were conservative. I also feel that firrst blush of the graph (due to the amount of years) looks like the dems made the majority of the recent debt. I too made a detailed database on the public debt that includes the House of Reps, President, Senate, and combinations of the 3. The results are shocking! Here is the link to the database without formulas (cleanest version in Editgrid)… and here is the same database with formulas… Cheers to finding the truth in numbers.

  8. Yes. thanks for sharing the debt history. I’ll check it out. I think it’s been pretty hard to find the numbers, and also be educated about it as well. I’d like to think my numerical literacy is better than most, but then again, who knows.Currently, I’m working on to help me import data I find and present them in a way that’s easy to find and download for others.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s