I have to say that https://student.mit.edu/catalog/index.cgi looks like a great site! It is all public, and when you open a category (a static site), all the information appears all at once and you can Ctrl-F for keywords. It might not solve that "unknown unknown" problem that the author mentions, but it is certainly much preferable to the solution that our university used (https://www.his.de/hisinone; one example: all courses are displayed in a tree and if you click to unfold a node, the server generates verbose HTML and sends it to the client. This takes at least 10 seconds on good days).
Yeah literally the only thing they need to do to fix it is get rid of the no-cache and Connection: close headers. Maybe make an "All" page for better CTRL-F? Surely their catalog doesn't change more than once per minute and could have some level of caching (at least with revalidation)? Keep-alive would cut out ~150 ms of page load time and letting at least something like nginx cache it seems like it would cut out another ~150 ms.
Lots of school infrastructure is decades old. I bet lots of students can better manage the IT than the IT department. The only issue is security and privacy. Not sure if there is a balanced solution for self-governance.
I can confirm this. I recently graduated (PhD) and was constantly submitting fixes for my department's servers. I'm not even a networking person or anything... I do ML.
And by fixes, I mean supplying the code and everything. One example is that the listing of classes generated a link as long as there was a folder on the server. Which the folders are automatically generated via the course offering of that term. This led to lots of links to nowhere when students were trying to see the course material and syllabus from previous offerings (so it was something people did quite a bit. Click, 404, back, click next, 404, repeat...). So the for my submit I found the file providing all the links, and the few lines around there. Provided a few alternatives such as "don't provide link unless there's an index.html inside" or "don't provide link if folder is empty". I even generated the one line `find` command that could go through and purge all empty directories that were older than 6mo (or any desired time). (That would clean up for anyone looking via the cli, which was more common than you'd expect) All the work was done, just someone with permissions needed to run.
There were tons of small fixes (as well as some much bigger ones...) like this that I and others submitted. Very few ever were done. Maybe the IT guy's hands were tied but every time I walked past his office he was sitting on his couch watching Netflix on his iPad[0]. I saw 3 different IT employees in my time and none of them took action on any of those types of issues.
I think the small things just get brushed off. Thinking "oh, well it's frustrating, but only happens a few times a year and doesn't cause much harm." Which, is true. But also, isn't the beauty of scaling in CS that even though 15 minutes of my work only saves a person 5 minutes (3 times a year, every year) that I'm saving that 5 minutes for hundreds or thousands of people? Saving 30 seconds doesn't sound like a lot but saving saving 100 people 30 seconds is saving an hour. If they have to do that once a day then you're saving a full work day every week. That's only 100 people too...
But I learned an important lesson: the little things matter.
[0] Not to bash him too much. When shit went down he usually responded pretty quickly. And grad students work some weird hours...
I was around when MIT had its first online catalog. Reportedly hacked together by a grad student over a summer (probably not a whole summer).
Anyway I think for the catalog there are no serious privacy issues, and there wouldn't be a problem having students work on it. Databases with student information (and that includes registering for classes) are a different story.
You probably were speaking in hyperbole, but there’s not a lot of school equipment out there still running windows XP and older. There’s some certainly but “lots”? Probably not except in the poorest of school systems.
I started at a university full-time in 2014 in the IT department. Same year a project to decommission an aging IBM mainframe that sat as a lynchpin in nearly every important operation on campus; course registration and thus billing and grades, transcripts, as well as a home-grown SSO from the early 2000s no one understood any more. The consulting costs were insane, as were the support costs, so that poor thing had to go (second prettiest rig in the datacenter after my Xserves).
When I left that job in 2022, I believe most of it had been offloadded, but I can't say for sure if they had actually shut the fucker down yet.
My understanding is you're allowed to automate the discovery of classes as much as you want. but as soon as you try to automate signing up for them you're gonna get in trouble.
I enjoyed reading this. Among the points discussed, the one about structures that can be read by machines (machine-readable formats) seems to hold important implications for the future of data structures and representation.
While there are many attempts being made today, at the current pace of change I can imagine us returning to a clean, well-structured, text-based origin. As for UI representation, I believe AI will likely handle that on its own.
I've heard of people making reminder bots for schools that give you a short time slot to accept getting off a waitlist before it moves to the next person
I have to say that https://student.mit.edu/catalog/index.cgi looks like a great site! It is all public, and when you open a category (a static site), all the information appears all at once and you can Ctrl-F for keywords. It might not solve that "unknown unknown" problem that the author mentions, but it is certainly much preferable to the solution that our university used (https://www.his.de/hisinone; one example: all courses are displayed in a tree and if you click to unfold a node, the server generates verbose HTML and sends it to the client. This takes at least 10 seconds on good days).
Yeah literally the only thing they need to do to fix it is get rid of the no-cache and Connection: close headers. Maybe make an "All" page for better CTRL-F? Surely their catalog doesn't change more than once per minute and could have some level of caching (at least with revalidation)? Keep-alive would cut out ~150 ms of page load time and letting at least something like nginx cache it seems like it would cut out another ~150 ms.
Lots of school infrastructure is decades old. I bet lots of students can better manage the IT than the IT department. The only issue is security and privacy. Not sure if there is a balanced solution for self-governance.
SIPB is the student run hacking club at MIT (https://sipb.mit.edu/)
At MIT, SIPB has managed IT better then the IT department for a long time.
Usually it's not a lack of will but just institutional entrenchment.
Students may volunteer but there's no path that would allow them to fix things.
I can confirm this. I recently graduated (PhD) and was constantly submitting fixes for my department's servers. I'm not even a networking person or anything... I do ML.
And by fixes, I mean supplying the code and everything. One example is that the listing of classes generated a link as long as there was a folder on the server. Which the folders are automatically generated via the course offering of that term. This led to lots of links to nowhere when students were trying to see the course material and syllabus from previous offerings (so it was something people did quite a bit. Click, 404, back, click next, 404, repeat...). So the for my submit I found the file providing all the links, and the few lines around there. Provided a few alternatives such as "don't provide link unless there's an index.html inside" or "don't provide link if folder is empty". I even generated the one line `find` command that could go through and purge all empty directories that were older than 6mo (or any desired time). (That would clean up for anyone looking via the cli, which was more common than you'd expect) All the work was done, just someone with permissions needed to run.
There were tons of small fixes (as well as some much bigger ones...) like this that I and others submitted. Very few ever were done. Maybe the IT guy's hands were tied but every time I walked past his office he was sitting on his couch watching Netflix on his iPad[0]. I saw 3 different IT employees in my time and none of them took action on any of those types of issues.
I think the small things just get brushed off. Thinking "oh, well it's frustrating, but only happens a few times a year and doesn't cause much harm." Which, is true. But also, isn't the beauty of scaling in CS that even though 15 minutes of my work only saves a person 5 minutes (3 times a year, every year) that I'm saving that 5 minutes for hundreds or thousands of people? Saving 30 seconds doesn't sound like a lot but saving saving 100 people 30 seconds is saving an hour. If they have to do that once a day then you're saving a full work day every week. That's only 100 people too...
But I learned an important lesson: the little things matter.
[0] Not to bash him too much. When shit went down he usually responded pretty quickly. And grad students work some weird hours...
I was around when MIT had its first online catalog. Reportedly hacked together by a grad student over a summer (probably not a whole summer).
Anyway I think for the catalog there are no serious privacy issues, and there wouldn't be a problem having students work on it. Databases with student information (and that includes registering for classes) are a different story.
You probably were speaking in hyperbole, but there’s not a lot of school equipment out there still running windows XP and older. There’s some certainly but “lots”? Probably not except in the poorest of school systems.
Windows XP? no
perl scripts written in the 90s controlling grades and course registration? Absolutely
I started at a university full-time in 2014 in the IT department. Same year a project to decommission an aging IBM mainframe that sat as a lynchpin in nearly every important operation on campus; course registration and thus billing and grades, transcripts, as well as a home-grown SSO from the early 2000s no one understood any more. The consulting costs were insane, as were the support costs, so that poor thing had to go (second prettiest rig in the datacenter after my Xserves).
When I left that job in 2022, I believe most of it had been offloadded, but I can't say for sure if they had actually shut the fucker down yet.
Advice: capitalize the S in Courseek, to emphasize the misspelling as intentional.
And, AWESOME WORK. This is truly "making things better", which is my favorite What Does An Engineer Do saying.
Unless the meaning was CourseEEK
OMG that .. is spot on.
I can understand why, too ... SOO many courses ... EEEEEEK! Amirite?
Seriously for a sec - it's a fantastic collection of quality MIT courses, in my view. They engage me, at least!
My understanding is you're allowed to automate the discovery of classes as much as you want. but as soon as you try to automate signing up for them you're gonna get in trouble.
I guess they don't want my money...
I enjoyed reading this. Among the points discussed, the one about structures that can be read by machines (machine-readable formats) seems to hold important implications for the future of data structures and representation.
While there are many attempts being made today, at the current pace of change I can imagine us returning to a clean, well-structured, text-based origin. As for UI representation, I believe AI will likely handle that on its own.
I just did something similar last week: https://varun.ch/posts/elective-list/
it’s almost funny how similar the scraping code ends up looking for these kinds of things
Nice write-up!
I've heard of people making reminder bots for schools that give you a short time slot to accept getting off a waitlist before it moves to the next person
tl;dr the app is at https://chuanqisun.github.io/courseek/
I also wrote a short piece reflecting on vibe coding this app and musing on the broader implication of building non-scalable software. https://stackdiver.com/posts/non-scalable-software/