Tim On Computer Science

You’re here because you can’t get the unit tests for the Coursera Improving Deep Neural Networks course, Week 3, exercise number 1, to pass. You are trying to correctly implement

def compute_total_loss(logits, labels):

And the unit test keeps failing with an off-by-a-large number (i.e. not the usual “shapes are incompatible” error).

Sorry, this post isn’t going to just give you the answer.

But, since Coursera (instructions and code) and Tensorflow (documentation) have come together to make an incredibly frustrating riddle, this will provide hints on where they went wrong, and thus where you went wrong.

Another guess on why you’re here: you entered “tf.keras.losses.categorical_crossentropy example” into a search engine, and followed the link tf.keras.losses.CategoricalCrossEntropy, and that page shows the first argument as “from_logits=False,”. And you just figured that was close enough.

Yet another guess on why you’re here: you read the instructions – “It’s important to note that the “y_pred” and “y_true” inputs of tf.keras.losses.categorical_crossentropy are expected to be of shape (number of examples, num_classes)”. And you noticed the code gives you “logits” and “labels”, in that order. So Coursera has said “y_pred/y_true”, then said “logits/labels”, and the first documentation link doesn’t even have the first two arguments. But even with all of these naming mis-matches, you still entered the arguments in the order given (implied?) by the instructions.

Final clue: use the link provided by the instructions – tf.keras.losses.categorical_crossentropy and pay attention to the order of the arguments.

Then, channel Jeff Atwood on the “hard things in computer science” and write your own post about how using poor names can cause a ton of problems…

This is a variation of “Be, Do, Have” (“Be, Do Have” == Be a photographer, Do take a bunch of photos, and then Have/Buy expensive equipment). It records my recent epiphany in Machine Learning. This variation is “Read, Do, Aha”.

I have read a lot of material on machine learning (linear regression, logistic regression, neural nets, supervised learning, CNNs, etc.) I have done some certifications (mainly from Coursera, from Andrew Ng, in the Machine Learning Specialization).

I have already completed the Convolutional Neural Networks course (fourth of the five of the Deep Learning Specialization) and recently decided to “go back” and start with the first course in this specialization – Neural Networks and Deep Learning.

In that course, one of the exercises is to build a simple neural net (simple = two layers total, no hidden layers). So simple it is equivalent to a Logistic Regression model. A very basic plot of a Logistic Regression model is:

In the example above, two inputs (plotted above as x and y) are given a binary classification (blue dot or orange dot), and then a single-line regression is computed, and the graph shaded into blue and orange areas. Thereafter, you can use that line to test the accuracy of known (training) data, and to make predictions on never-seen-before (test) data. In the above example, the training accuracy is quite good, and it seems safe to say the test accuracy will/would be quite good as well.

The course uses “Cat or not-Cat” as the binary class definitions, and flattens a 64x64x3 image into a an input of 12,288 values. After everything is implemented and run against the training data, you have a 12,000+ dimension hyperplane that divides the input space into Cat/not-Cat.

The “Aha!” moment looks like this:

1. Fact: It can be trained to 99% accuracy to correctly separate the Cats from the not-Cats, in a very small amount of time/iterations.
2. Fact: It has a terrible accuracy (70%) on the test set (aka never-seen-before data). Just guessing would have a 50% accuracy if the test set was split 50-50. i.e. the hyperplane is terribly overfit


3. Aha! This makes sense because you tried to use 12,288 pixels individually to decide between Cat/not-Cat. (i.e. you had no hidden layers, no “build up of knowledge”, no “higher level constructs”). It only used individual pixels. When looked at from that perspective, it is amazing that it can get 99% accuracy on the training data, and completely un-amazing that it doesn’t generalize to the test data.

While looking at various datasets from kaggle to do some experiments with Python and graph visualization with networkx, the arXiv dataset caught my attention. It has a (relatively) simple schema – authors, papers, and categories. It is also huge – there are 2,219,423 lines/records in its 3.5GB uncompressed .json file (1.2GB compressed).

The format of that .json file is also a practical example of an issue with JSON – you can’t start with a valid .json file, then add/write/append another record, and end up with valid JSON. (The CSV format is allows appending without rewriting the entire file, but CSV has many other issues that make it unattractive for most cases.)

There is a (semi-formal?) standard for the format used by arXiv (even though it is never mentioned by name) – “JSONL” or “JSON Lines”. See this 2022 entry from atatus.com and this jsonlines.org summary. The vocabulary is fun: “Each line has a valid JSON value”. Other vocabulary: “Line-delimited JSON” and ” Newline-delimited JSON” and “Concatenated JSON”. The suggested extension is “.jsonl”

Tool support for JSONL seems to be hit-or-miss. jq supports JSONL (surprise!). The Python library json does not (it throws a JSONDecodeError “Extra data: line 2 column 1 (char 1689)”. As the title says – this post is about pandas.

Based on many of the code examples, you could easily conclude that pandas does not support JSONL. For a naive implementations, see Arxiv Data analysis (loads every object into an array, then creates the DataFrame with pd.DataFrame.from_records(array). The people who run this naive code have either (a) trimmed the number of lines in their input .json to something reasonable, like 100k lines or (b) have access to machines with 24GB+ of RAM in order to have two copies in memory at the same time.

The github gist that inspired this post was from cj2001/load_arxiv_data.py, which creates the entire array of “metadata”, but with a limit on the number of lines/objects created. It does limit the number of lines, but it is still “the hard way”.

For a slightly more sophisticated implementation, see Scientific Article Recommendation – See In[2] extract_data with “yield line” and In[3] fetch_n_records with list comprehension to load the file into memory:

return [json.loads(record) for record in islice(data_gen, chunksize)]

In conclusion:

The correct pattern is to use pd.read_json( filename, lines=True, orient=’records’, nrows=nnn) (as can be seen in ArXiv) Note that leaving off the “nrows=” argument can create an out-of-memory situation even when using nrows=<a huge value> will work just fine (e.g. arXiv, nrows=5000000 works, since there are only 2219423 lines, but leaving off nrows crashed with OOM killer)

So – add pandas.read_json() to the list that supports JSONL natively:

df = pd.read_json("arxiv-metadata-oai-snapshot.json",
                  nrows = 5000000,
                  orient='records',
                  lines=True )