Data Science Specialization at Coursera

Last year, I took the nine online courses in the Data Science Specialization offered by Johns Hopkins University via Coursera. It could have been called “Data Science with R” since one whole course and a good part of the other courses were more about R programming than data analysis. I certainly learned more about R and the R ecosystem than about statistics and data science, but I already had some knowledge of the latter. The nine courses are:

  1. The Data Scientists Toolbox
  2. R Programming
  3. Getting and Cleaning Data
  4. Exploratory Data Analysis
  5. Reproducible Research
  6. Statistical Inference
  7. Regression Models
  8. Practical Machine Learning
  9. Developing Data Products

There’s also a “capstone” course/project, but it’s not offered as often, and the timing didn’t work for me.

Course Structure

Each course was four weeks, and the pattern for most weeks was about 30 minutes of video lectures and a multiple choice quiz. Some courses also had a project involving some data analysis, and those were the most educational parts of the series. There were also practice assignments and online discussion forums, but I found the Coursera site too sluggish to really use those casually.

As would be expected from courses like this, what you get out depends on the extra effort you put in. Someone completely new to the material would have to spend a lot of self-learning time because they’re not going to learn technical material like programming and statistics from a few video lectures. Unfortunately for me, the multiple choice quizzes made it a little too easy to get by with little effort. That’s partly a necessity of an online course, but being able to take each quiz three times seems to undermine any rigor. If you can eliminate just one of the four choices, you’re guaranteed to get each question correct by the third try.

Positives

Instructors. One thing that kept me in the series (besides my stubbornness to finish what I started) was the quality of the instructors, Brian Caffo, Jeff Leek and Roger D. Peng. They obviously have an enthusiasm for the subject and did a great job organizing the material. I especially like that they would often do live R coding during the lectures (I’m sure there were some edits, but still…). Sometimes that’s where the best R tips were learned.

Material. I liked that they spent a good amount of time just on programming and just on reproducible results, two topics that could be ignored or brushed over and still pass for data science. For the modeling, they made of point of avoiding linear algebra, which was a nice change from the standard approach.

Projects. While still constrained by the time available, these longer assignments forced you to apply the material to perform some basic analysis and publish the result. The projects used real data sets which often required some clean-up/preprocessing, which was a good lesson in itself.

Negatives

Peer grading. The longer assignments were better measures, but due to the compressed timing, they had to be kept pretty simple, and due to the large number of students, they had to be peer graded. For peer grading, each student had to grade four others. I can only hope they required some sort of agreement among duplicate graders because the grading guidance was minimal and coarse. (Is there a model provided? Yes/No. Is the model correct? Yes/No…)

The one course I didn’t get high marks in was the course I expected to be my best: Exploratory Data Analysis. I didn’t get good scores on the the long assignment from the peer grading. I knew I wasn’t strictly following the assignment guidelines, but I hoped the graders would be more flexible. The assignment called for a report with something like three pages of text and an appendix with two pages of graphs. I prefer to put the graphs inline with the text, but some graders stuck with the literal guidelines and gave my zeros for that part of the grading (there was often no way of giving partial credit).

Coursera. Besides the sluggish forums and grading limitations already mentioned, I would like to go back to review parts of the material (partly so I could be more specific in this review), but the archives are no longer available. Last month at least some of the course archives were available, but now none are. Given the scope of the material, providing later access seems essential.

Bottom Line

For someone new to the field, I think these courses will be too brief to learn the material well, but they provide a great tour from some great tour guides and will help frame further study.

Posted in Code | 1 Comment

Days in a Month Formula

Curtis McEnroe posted a “tongue-in-cheek” derivation of a formula for the number of days in a month, ignoring leap years.

His result, in JavaScript:

function f(x) { return 28 + (x + Math.floor(x/8)) % 2 + 2 % x + 2 * Math.floor(1/x); }

His derivation is quite logical. I made a shorter alternative using a different technique, which I will leave for the reader to puzzle out.

function f(x){return 28+(62648012>>x>>x&3); }

 

Posted in Uncategorized | Leave a comment

My year with Android via Nexus 5

A year ago, I was ready for a change. I was tired of my tiny iPhone screen and the off-contract cost of AT&T cell service, so I switched to a Google Nexus 5 with a nice screen and T-Mobile with friendly and cheaper month-to-month service. But now I’m back to a iPhone and AT&T. The size problem has been fixed with the iPhone 6, though it’s still a little smaller than the Nexus 5, and my AT&T service is now employer subsidized. AT&T has also adapted to offering more T-Mobile-like plans.

Reasons I was looking to change back to iPhone/AT&T:

  • Battery life was terrible for my Nexus 5. It would barely make it 10 hours in sleep mode. I can only imagine that my cell/wifi reception was so poor that the phone was constantly waking up and looking for a better connection.
  • Poor T-Mobile coverage in this area. Too many places, I had no data coverage.
  • Android was not as convenient to sync with my Mac (iTunes/iPhoto). However, now I see iTunes and iPhoto aren’t as convenient as they used to be if you don’t opt in to the cloud sharing.
  • I missed having a hardware mute switch for the entire phone. Android has several different volume settings and muting one (like ringer volume) doesn’t mute the others (like app noise volume).

Things I will miss from Nexus/Android:

  • Convenient integration with Google Now. I get most of the functionality from the Google iOS app, but it’s not as convenient or as tightly integrated as on Android. Surprisingly often, it would tell me what I needed to know before I asked.
  • T-Mobile customer service. The fact that it was easy to cancel is a good sign in itself. Also, when we went to Toronto last summer, T-Mobile prompted me to buy a short-term international data plan (about $20), while my wife’s AT&T phone silently accumulated $100s in international roaming charges, and she thought she had cellular data turned off.
  • Openness of the app store, including being able to use Firefox and ad blockers.

One bonus of the switch-back is that my old iPhone apps carried over from before. Perhaps the downside of the open Google Play store is there is more a focus on in-app ads instead of paid apps. The ad-based Scrabble was unusable (a fullscreen ad after every move), so I’m happy to get back to my paid, ad-free Scrabble app on iOS.

One more thing: for some reason the iOS Google Maps app doesn’t show traffic as well as the Android version. These two screen captures are showing about the same thing before my morning commute, but the iOS version on the left puts the blue “your route” line over the red “slow traffic” lines, making it hard to see the traffic along your route.

Posted in Uncategorized | Leave a comment

Charity Solicitations Visualized

Way back in 2006, I tallied my charity solicitations, and since then the situation has gotten comically worse. So over the past year, I’ve tried to keep all the solicitations I received from various charities, some of which I haven’t contributed to in years. This time, instead of a simple table, I’ve made a couple kitchen floor charts showing the actual pieces of mail received. I probably missed a few, and phone calls are not represented.

Here are my “charts” of charities sending the most and least amount of mail in 2014.

charitymost

charityleast

Care is easily the most annoying (I even found another piece after taking the pic). I didn’t even know I had donated to Care, but they handled a donation for typhoon relief.

It’s good to see that Public Citizen and Southern Poverty Law Center are doing better. They were the top offenders in 2006, but they’re now mostly honoring my request for one solicitation per year.

Perhaps the annoyingness is exacerbated by my giving pattern, which is to give toward the end of the year. Unfortunately for me, common practice is to accept the December gift and then send an “annual renewal” just a few weeks later in January.

The worst offenders have either been dropped or switched to reduced anonymous giving, but I expect the junk mail to continue for years. And anonymous giving is expensive as far as I can tell. Network for Good adds a 5% fee and JustGive adds 4.5%. Hopefully, that includes the credit card fees, but I’m not sure. It shouldn’t been so expensive just to move money. Fidelity Charitable with a flat fee of $100 per year may be another option, especially if the credit card fee is separate for the other options.

My truly least annoying charity (annoyance == 0) is one that sent me no mailings or online annoyances: the Online Encyclopedia of Integer Sequences. Thanks to Neil Sloane and many volunteers for that excellent resource. Wikimedia was also in the no-mailings camp, but they made Wikipedia pretty annoying to use for most of December, so I’m not sure where to rank them.

Posted in Graphs, Uncategorized | Leave a comment

Remaking ProPublica’s Blocked News Graph

Earlier this month, investigative newsroom ProPublica published an interactive graph showing the availability of various news web sites within China.

original

My initial reaction was that the missing (gray) and inconclusive (yellow) data are dominating the view without providing much information. Of course, the graph is precisely reflecting the data available, but I’d prefer a view that helps me see larger patterns. Which news sites were likely blocked all year? The Huffington Post stands out with a lot of red, but it’s not so obvious what other news sites were also never measured as available throughout the year.

Here’s a close-up of three sites with lots of inconclusive measurements that dominate the coloring. They have three different mixes of conclusive measurements, but that difference doesn’t stand out until you focus on it.

originalclose

So my main “improvement” is to interpolate the absent/inconclusive data. For instance, if a site is measured as blocked, then unmeasured, then measured as blocked again, relabel the middle no data region as presumed blocked. Same for open, and for mixed endpoints relabel no data as in transition. Inconclusive is a really tricky category; I leave it as different from no data but similarly tint inconclusive measurements according to the surrounding measurements.

My second improvement is to avoid the red/green colorblindness issue. I switched the open color from green to blue. The stoplight colors have strong connotations which help with the interpretation of the original graph, and so it’s a trade-off to balance a small benefit to the 97% with normal vision versus a large benefit to the few. Still, here’s what the close-up above looks like with the most common red/green colorblindness (using Color Oracle):

originalblind

My third improvement is to reduce the white space between rows. I didn’t realize it until I tried it, but I think the strong white banding is distracting.

Finally, here’s my view (data through December 19):

remakeshort

Is my improvement better? In its current form it looks a bit busy, but I think if a designer picked better colors, it would be quite functional, assuming you buy in to the interpolation idea.

With my remake, it’s easier to see the three sites that were never seen as blocked (CNN, ProPublica, and Washington Post), the six news sites that were never seen as open and especially the transitions, which might be the most interesting parts. Presumably, the late September transitions are related to the Hong Kong protests.

I haven’t tried to reproduce the nice axis labeling, reference lines or the interactivity of the ProPublica version, all of which I like.

Posted in Graphs | Leave a comment

Toronto Discs

Bonnie and I spent last week in Toronto, and the week ended with a Frisbee theme. Though most of the trip was unplanned tourism, I did preview the area Ultimate offerings before leaving and found a Friday evening pick-up game that was only a 30 minute walk from the hotel.

torontoultimate1

Here’s a pic of the game at Sir Winston Churchill Reservoir with the ever-present CN tower in the background. The game was recreational, which is good because my legs were a little out of sync from walking around the city all week.

By complete coincidence, our hotel was across from the home field of the Toronto Rush semi-pro Ultimate team and they had a play-off game that Saturday night. Here’s a pic from the hotel room of the team warming up for the game.

torontoultimaterush1I attended most of the game despite the steady light rain. The rules were a little different with active referees assessing yardage penalties for violations. Unfortunately the game was a little sloppy due to the rain, but there was still some nice play amplified by three on-field video cameras and a giant replay screen on the scoreboard.

The next day, we ran into a third disc connection: a full disc golf course on the tiny islands in Lake Ontario. I don’t think I’ve ever seen a disc golf warning sign.

torontodiscgolf1

Not sure how active the community is. I saw this flyer for a tournament on the bulletin board and thought I might enter until I noticed the date.

torontodiscgolf2

Posted in Sport, Ultimate | Leave a comment

Benchmarking resizable arrays versus linked lists in C++

The second half of Herb Sutter’s recent talk, “Modern C++: What You Need to Know,” explores the question of how many items do you need before it’s better to use a std::list instead of a std::vector for an insertion sort. The conventional wisdom is that eventually the cost of moving half or so of the vector to make room for a new element will overtake the overhead of the linked list traversal. However, testing showed the answer is never and underscored the importance of memory locality in modern computers (with lots of prefetched cache data).

I decided to try it out on my own machine, but before I got too far with the code I looked around to see if someone already had code I could use. Soon enough, I found C++ benchmark – std::vector VS std::list VS std::deque on Baptiste Wicht’s blog. He ran lots of tests (not just insertion and removal)  on lots of data types (not just integers). Like Herb Sutter, Baptiste also found arrays to be faster than linked lists for small objects, but lists were better for large objects. The meaning of large varied depending on the algorithm. For sorting, large was 16 bytes, for insertion, large was more like 64 bytes. He didn’t explore arrays of pointers to large objects.

In trying out his year-old code on my Mac, I encountered a couple compile errors and one logic error. Since the code was on GitHub, I thought it would be a good chance to try it out. I set up a GitHub account, forked his code into my account, fixed the issues, and made a pull request for my changes to be merged back into Baptiste’s original code. A couple days later, he accepted the changes — the internet works!

Here are some results from my runs. I went up to n = 1,000,000 for two operations and two data sizes.

vectlist

 

Looks like for my PowerMac, large is somewhere between 8 and 32. The real hero is deque, which is segmented vector that can easily grow at both ends.

 

Posted in Code | Leave a comment