< Back to the archive

Like what you see? Subscribe here and get it every week in your inbox!

Issue #266 - April 14, 2024

Here are the top threads of the week, happy reading!

Top comment by ChicagoBoy11

I'm Brazilian and when I moved back home for a bit the country had recently inaugurated a national exam for all high-schoolers. About maybe a year after, they'd publish an anonymized data set for researchers with a nice question-by-question breakdown of how every student performed. I worked at a school at the time, and one thing we wanted to know was how WE had performed in the questions so we could gauge if there were some things our students were consistently unprepared for.

They didn't provide that data, but it turns out with a little bit of grokking and staring at that 10gb text file, you could reverse engineer it so you could extract all the kids of a given school, and aggregate all of their answers. I produced a nice little report for our admins, with the questions of the text next to how we had performed in the aggregate and state averages, as well as averages of our "competitor schools."

The best part, though, is how I remember it being a bit "bullshit" that we, a private school, could afford to do this, but since the data was actually valuable to inform practice, surely the department of education should do that for every school! Whelp, over the weekend, I computed that info for every school in the dataset, and just stored a CSV for every school in an S3 instance (this was my ridiculous caching strategy lol). Spun up a frontend where you could select your school, and nicely visually go through every question, as well as print a pdf summary of the whole thing. On Monday morning tweeted at an ed journalist, and in a few days had a spread of me in the country's top newspaper, and people emailing about jobs "at my company."

This was the most rewarding project I've ever done, and I'm sad to say nothing has come close since. It cost me $0, produced a public good that I could see was being accessed from every state in the whole country, was technically interesting, and I saw it through from start to finish over the weekend!

Top comment by hruk

Yes, we use it to back several services in production at an 8 figure ARR business, with Litestream as our streaming backup option. I'm quite happy with it because it's trivial to debug and test against a local copy of the production data if necessary, but there are a few things to think about.

* You need to have your code repopulate index statistics with ANALYZE or PRAGMA OPTIMIZE now and again, or you may get a confused query planner.

* It's best if your code defines meaningful types to scan SQLite values into, and you ideally avoid ever writing to the DB via the CLI. Alternatively, be aggressive in writing CHECK constraints and triggers to ensure data validity.

* You need to be working with a team of developers that are pretty comfortable with databases to get a lot of value out of SQLite. I've been surprised at how performant features can be if a single mind is designing the schema, constraints, triggers, and queries, examining the query plan to choose indexes, and writing the application code that accesses the database. SQLite minimizes the barrier to doing this with real snapshots of production data - but that still doesn't mean it's going to be easy.

* You need to think about how long your code holds a write transaction open. Individual statements in SQLite run very quickly (I've seen ~250 microseconds per insert on a EBS-backed EC2 instance), but if you have a Django app with @transaction.atomic everywhere, you're going to run into lock contention quite quickly because your Python functions take much longer to run. If possible, "organize" all the data for your writes outside the transaction and then hold the lock for as short as possible.

* Any transaction that may eventually write must start with BEGIN IMMEDIATE, otherwise SQLite may throw an error to keep it's promise of serialized isolation on writes.

All said, I'd do it again for any other services-based backend. It also incidentally enforces the good hygiene of "services can only access their own data." Happy to answer any questions you have.

Top comment by softwaredoug

Maybe fear of death?

You get older, you get more tired(?), serene(?), you can see the progress towards death as a natural process in your own life.

Also you're around more death. I was with my mother when she died. I can't say it was a pleasant process, but it was very natural. She accepted hospice, and seemed to be at peace with death. The last few months with her a lot of positive memories were made that will stick with me.

I don't _want_ to die, but its clearly something I can feel will naturally come and should plan for.

Top comment by HeyLaughingBoy

Every job in my 30-year career has been at companies that manufacture something. I have to say that what you're planning is a really big ask. I mean, you could make an entire career out of any one of those areas you mention. e.g., Supply Chain Management is a pretty popular Master's degree program. It might be better to choose one and learn what you can.

But first, why? What's your position at this company? It seems like you're trying to improve something, but perhaps first spend six months at the company and understand how and why they work before looking at areas to improve?

Top comment by jasonjayr

Our organization is using Office365.

Either by accident, or just defaults getting increasingly more tight, Outlook won't connect to the account unless I allow it to be a device administrator.

On my personal phone, that's a hard no. So I'm using the PWA for the occasions I NEED to check email.

But a TOTP app of my choice, implementing a standard RFC protocol? I think that's okay, on the condition that it does not mean my phone is in scope for any regulations the company is mandated to adhere to.

Top comment by paulgerhardt

I love how varied the responses are. I’ll play…

Walk every aisle of your local Best Buy or MicroCenter. Do a mental calculation of how would this hardware product be 10x better with (present day’s) latest software trend.

Eg can I make this blender 10x better by bolting it to an LLM? Can I make this coffee machine 10x better by adding on a coffee+distilled water+third wave salt subscription? Can I make this network router 10x better shipping it with an invisible-to-normies tailscale + Speedify + 5G SIM card for ultra low latency, highly reliable zoom calls from the middle of nowhere using fast open and MPTCP?

Why 10x? Because if it’s not a 10x better experience the switching costs (aka “activation energy”) will be too high to gain market traction before you run out of funding and die.

Stack rank your items by categories such as “will this increase usage from once every 9 months to 9 times a day (ie smoke detector to co2 environmental sensor)”. “Does this have have a viral k factor that the original didn’t (access control for package delivery in door locks)”. “Does this unlock a recurring revenue stream that previously wasn’t there?” (smart decoding of video feed events). Etc.

Narrow your list down from 220 to 5. Build some prototypes. Test your market amongst fellow founders. Find the one niche vertical that has surprising utility and becomes a runaway success which exceeds your ability to fabricate in your living room using arduinos and off the shelf parts until you’ve exhausted Best Buy, Amazon, Spark Fun, and Adafruit’s supply of said item and you need to make your own version.

Get partnered up with a second rate design firm in one of the flyover states that has weird connections for manufacturing in this space from a random guy you met a JS conference three years earlier that cold calls you. Launch a successful crowd funding campaign. Raise funding. Fly to China to meet some contract manufacturers.

Pick the wrong contract manufacturer based on your own inexperience and your design firm’s prior relationship. Have things go terribly wrong because one didn’t use glass reinforced abs instead of regular abs because injection molded tools are expensive and a lot of tooling adjustments boil down to “guess and check”.

Move to China. But like not the cool part like Shenzhen or Shanghai - like a semi-obscure part - like Yiwu. Have way too many hot pots over Mao Tai trying to find a T2 vendor that knows how to injection mold helical gears. Learn some gutter slang from an obscure dialect of Chinese like Gan. Fire half a dozen T2’s. Hire five more. Fire three more. Refine it down to one final T2 that’s giving you problems. Eg try to find a paint vendor that can match the white on your injection molded part to the white on your metal part.

Nope out of the first paint vendor who can actually do this because of sketchy work conditions like a half naked infant running around the factory floor wearing 开裆裤.

Almost rage quit.

Question why you’re pirating grad school text books on material science from Turkish warez sites instead of taking that aqui-hire to AirBnB when they were like 12 people. Make a futile effort to give out safety equipment to T2’s regarding hearing and eye protection because you begin to question how everything is done in this country.

Nearly burn out again.

Begin to see the light. Parts fit. Production is scaling. Find a critical flaw. Fail.

Then do it all again until you succeed or run out of money.

Top comment by Plnt

For Windows 10 and Windows 11, there is already a built-in app called Quick Assist. It's a bit bare-bones compared to Team Viewer but it's free and it works out of the box.

https://en.wikipedia.org/wiki/Quick_Assist

https://apps.microsoft.com/detail/9p7bp5vnwkx5?hl=en-US&gl=U...

Top comment by simonbarker87

Sound is much more important - even if you consider your podcast video first (in which case it’s not a podcast but that’s another topic) sound is more important. Take whatever smart phone you have and use that for the video and then spend any money you have for the camera and spend it on a decent mic.

People will watch lower quality video with good sound for far longer than they will watch good video with bad sound.

Source: listening to podcasts now for 15 years and grew my own to 5k listens a month across 100 episodes.

Top comment by Freak_NL

Literature is not really art appreciated through vision. It is a mental exercise. Reading by seeing the text is just one form of uploading the text into your brain; Braille and audio are two others. The layout and appearance of the text can add to the experience, but this is mostly¹ not required except to make reading the text as relaxed and comfortable as possible.

For a blind person sculpture can surely be as engrossing as for a sighted person.

1: Notable exceptions exist; e.g., House of Leaves.

Top comment by timmaxw

In Python job interviews, I think the interviewer will only judge your code on asymptotic complexity, not absolute speed. I think Python engineers generally aren't expected to know how to micro-optimize their Python code.

Some general tips for algorithmic complexity in Python:

- Python's list is equivalent to C++ std::vector. If you need to push/pop at the head of the list, use Python's "collections.deque" to avoid the O(N) cost. Python's standard library doesn't have a linked list implementation.

- Python's dict is a fast unordered hashmap. However, if you need order-aware operations like C++'s std::map::lower_bound(), you're out of luck; Python's standard library doesn't have a tree implementation.

- Python has a "bisect" module for binary searching in a Python list, and a "heapq" module for using a Python list as a heap. However, neither one is nicely encapsulated in a data type.

If your Python program is seriously CPU-bound, the normal solution is to use a C/C++/Rust extension. For example, if you're doing large numeric calculations, use NumPy; it can store a vector of numbers as a single contiguous array, which is much faster than a list-of-Python-floats.

If you want to parallelize across CPU cores, it's important to understand the Python GIL (global interpreter lock). Often you need to use multiple Python processes. See e.g. https://superfastpython.com/multiprocessing-pool-gil/

Maybe also worth reading about __slots__ (https://wiki.python.org/moin/UsingSlots); it's a micro-optimization that helps when allocating a lot of tiny Python objects.

Hope some of that is helpful! Good luck with your job interviews.