Unless I’m misreading it which is possible it’s awfully late, he said he processed 60,000 rows didn’t find what he was looking for but his hard drive overheated on the full pass.
Discs don’t overheat because there was load. Even if he f***** up and didn’t index the data correctly (I assume it’s a relational database since he’s talking about rows) The disc isn’t just going to overheat because the job is big. It’s going to be lack of air flow or lack of heatsink.
I guarantee you he was running on an external NVMe, and one of those little shitty-ass Chinese enclosures. Or maybe one of those self immolating SanDisk enclosures. Hell, maybe he’s on a desktop and he slept a raw NVMe on his motherboard without a heatsink
There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they’ve never seen before and never done before.
Can’t be a relational database, Musk said the government doesn’t use SQL.
Lol he also said cybertrucks don’t suck ;)
He said many things.
Please remember that he is a genius. Only geniuses say a lot of things.
I rest my case.
deleted by creator
Somehow I feel over clicking without understanding of the consequences sounds like something a techbro would do
There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they’ve never seen before and never done before.
Honestly, any sweet, white-haired old lady who keeps pictures of her dogs and grandkids on her desk who’s been doing data entry for 15 years could do circles around these clowns.
But she might also have the wisdom and perception to know we’re not supposed to be doing this “work” at all, which is why he recruits naive teenagers and college kids who are still emotionally immature to think that this is going to be their “destiny” or their opportunity to get into the big leagues of business.
I keep hearing things about these hires he has, I don’t think they’re naive, At least not as such. They seem to be more power hungry trust fund babies.
But yeah, people with a few years in them would be a moral liability in that line of work.
Yeah if you read more of these guys tweets they are clearly in politics. One message tried to claim trump loves kids (to be clear: in the abstract sense, not in the he definitely fucked kids on an island with Epstein sense). Then they tried to twist the words to say “why don’t you love kids”. It was clumsy like you’d expect from someone who is practically a teenager, but the core is an attempt to follow the usual right wing playbook.
yes but also why say 60K when you could have literally said anything? I mean surely the fact that he thinks 60K rows a big number is already explaining alot lol.
It’s bait.
They probably have an explanation tweet at the ready to make more sense of it. They just want enough 'hurr durr these idiot" comments before they reverse Uno card this with more context.
Based on all that has been going on, I feel like they don’t really have the capacity to think more than one step ahead. They do sth stupid and then they usually follow up with “lol joke” or “lol you can’t understand”
“You don’t know what you’re talking about 💩💩💩”
Wait, seriously?
I don’t get how people don’t see this stuff. Yea the average trumpet isn’t out there planning Jack shit. But there are think tanks that are. Remember anti-smoking campaigns. Anti climate change campaign. The precision to purchase advertising into key areas and specific demographics that would spread a message. This is ancient knowledge with modern technology.
Imagine a room full of former Wall Street, quants, established experts from fields like behavioral science and psychology. All with the singular goal to decide where to dedicate a dragons horde of wealth to maximize effect in a world where we all have anonymous pipes directly into our eyes and ears. We never stood a chance. There’s no rich socialist funding think tanks. There’s no counter. We can laugh at the yokel all we want. But the yokel is being puppeteered by some scary fuckers with intention to seize power with the new shifting Zeitgeist. Soldiers don’t need to think. But their generals are. The left are like guerilla fighters going up against an imperial army full of Patons and Eisenhower’s.
Cambridge analytical, heritage foundation, international democracy Union. We’re fucked until we actually recognize why we’re fucked
music theaters also have rows, and they run on sql so logic checks out.
60k of rows is nothing. Fuck, where do you find these “geniuses”?
I do data analysis for a living, I reach out to tech and complain if I can’t open a file with a million+ lines.
Tbf we don’t know how many columns there are /s
The IRS just switched columns and rows. So there’s 60k rows and 330 million columns /s
There’s only one reason he wants kids and not experienced adults.
Wow.
I’ve been processing a couple of billion rows of data on my machine, the fans didn’t even come on. WTF are they teaching “experts” these days, or has Elmo only hired people who claim that they can “wrangle data” and say “yes” ?
Even if querying data was processing-heavy and even if somehow the ‘hard drive’ got warm during this, then there still would need to be a hardware defect in order for the drive to overheat.
Yes, but this may be a symptom of an issue I’ve been seeing with younger programmers; they’ve siloed themselves so specifically into whatever programming they “specialize” in, that they become absolutely useless at dealing with absolutely anything else related to their job. And exasperating this issue is the fact that they’ve grown up with systems that “just work”. Windows, iOS, and android are all at the point where fucking around with hardware issues is very uncommon for the average person.
Asking this guy to solve a hardware problem is like asking hime to tune a carburetor. He likely has not the slightest clue how to start.
In my experience, a lot of software dev degree paths basically don’t even have relevant classes on hardware at all. Classes on hardware are all in IT Helpdesk and Network Admin degree paths whereas the software dev students are dumped straight into Visual Studio right off the bat with no relevant understanding of the underlying hardware or OS.
My CS degree had a hardware/IT support class, but A) it was entirely simulation based. We never touched any actual hardware. We “built” PC’s or identified physical issues in 3d sim software, set up RAID arrays in software, etc. B) it was super hand holdy and you only ever go over a problem once, so nothing on the class has stuck. I know much more from having built, troubleshot and maintained my own computers and network than I ever learned from that class, then learned more by doing in an actual IT support position before becoming an engineer.
I mean to be fair the sheer amount of material most university engineering programs require these days makes spending significant time on specific problems almost impossible. They try to shove so much theory into your head they lose track of practical implementation. Basically everyone I went to school with complained about the lack of practical application relative to theory, and I studied mechanical engineering which is theoretically and literally chiefly concerned with hardware.
deleted by creator
My experience does not reflect yours. Computer Architecture, Discrete Math (logic gate math), and Operating System Concepts were all required classes in my CS degree from just a few years ago.
Honestly that’s good to hear. I’ve run into some devs who are completely mystified on how to connect to a remote database and couldn’t tell a socket from sandwich.
In my degree, we had to write kernel mods and device drivers
Can I have my socket with rye. I like rye.
You don’t teach a farmer how an internal combustion engine works. Computers are tools to software engineers. What they need to know is how to operate them, not how to maintain them.
Horseshit. Computers aren’t tools for a software engineer. Computers are tools to an administrator, an accountant. Computers are the sandbox you are building castles in as a software engineer. If you don’t understand the system upon which you build, its abilities and features, its limitations, it’s dependencies, you are going to make some stupid mistakes.
You need to understand discrete mathematics as a consequence of computer computation. You need to understand parallel processing and threading for muli-core processors. You need to understand networking, package management, security vulnerabilities, etc. from different architectures and protocols. And it ALWAYS helps to understand the very basics of a computer’s functioning, from hardware, CPU architecture, machine code, assembly/low level programming, memory management, etc.
print('Hello, World!) is day one shit for a reason. Programming language and logic is the basics. The real expertise comes from your 3rd and 4th year materials. Databases, architecture, theory of computation, discrete mathematics, networking, operating systems, compilers, etc.
computers are a tool to anybody who uses them?
If you’re using a tool, it goes without saying, you should probably have at the very least, a cursory understanding of it’s function. Lest you injure yourself gravely.
I’m not sure how well that analogy holds up. Farmers are usually pretty well versed in mechanical systems. To the point that now that John Deere has been screwing them over on right to repair that some farmers are even becoming versed in computer programming so they can flash the firmware on their tractors.
Farmers can build a two stroke from parts.
No, but if a farmer’s tractor is overheating (as in the gard drive conparison), I’m sure they could diagnose it.
I never said that it was impossible for a farmer to learn things outside their immediate field. Just like computer programmers often have knowledge of hardware and the general technology stack.
My point, to make it explicit to a few of the illiterates who’ve replied to my comment so far, is that it is not necessary to teach a web developer how a goddamn CPU works. They can gain nothing from that knowledge because there are at least 3 levels of abstraction between JavaScript and assembly.
Operating your tools and being able to maintain and repair your tools are the unequivocally essential skills for everyone in every single industry.
If you can’t, you are not a professional.
The concepts of machine logic, registers/lookups/etc are essential for every programmer. If you don’t have a clear idea about how the simplest CPU functions, you don’t have any basis of understanding the abstractions in front of you, scripting in JS. Not a professional.
And my point is that the example you used does not make the point you are trying to make, but rather the opposite. I get what you’re saying, it just doesn’t apply to farmers and mechanics.
no but a web dev should have some knowledge basis on what the ever living fuck their AIDs code fuelled by nothing but the cheapest source of caffeine and brain damage they have even does.
This is the entire reason why half of the internet is just broken, stupid developers who don’t know how anything works, but know how to code, making dogshit implementations of anything and everything they can get their hands on.
It doesn’t matter that the learning is segmented, you should STILL be learning about computer hardware and it’s architectural choices, it’s literally the reason why programming languages work the way that they do.
What the fuck
How is he going to fix his tractor? Wait days for John Deere to send somebody? Let the crop rot on the vine?
No, not really. Programming requires understanding of the underlying hardware, at least to a certain extent. Otherwise performance issues will look like dark magic and optimizing anything would be impossible.
Where do you start debugging if something goes wrong with the software and your information level is this low/ do you look at network stats? CPU utilization, paging/swapping? Is the hard disk bandwidth the bottleneck? Without at least some passable understanding of a computer architecture people like this just throw up their hands, or throw whatever tricks they know at the wall and see what sticks.
A lot of farmers are learning how they work cause the companies that sell them the equipment keep fucking them over. I would argue that farmers nowadays needs to know how that works along with basic programming to get past the anti-consumer bullshit companies put in to make it nigh impossible to fix things yourself.
doesnt matter if you know how to program, john deere is just going to put some autistic encryption and ID locking on their shit, what needs to happen is for john deere to stop fucking doing this.
Most tractors are walking computers anyway, farmers are genuinely the most multi talented people you will ever meet in your life.
Just keep trying to justify your own lack of competency I guess. ¯\_(ツ)_/¯
CS departments were doing poorly, but now they’re putting out farmers? No wonder all these new graduates can’t find a job.
Ooh wait 'til Musk realises he can improve US agricultural efficiency.
I mean every programmer says they intend to quit and pick up farming. Might as well give them the knowledge to be successful at their late career while they’re at it
You’ve never met a farmer in your life.
the only reason farmers are afloat financially is BECAUSE they can rebuild an engine if needed.
Just look at the john deere right to repair shit. It’s literally a huge problem.
It is good for the programmer to know how the computer operates, as well.
That’s the price of specialization. Don’t ask a software engineer to troubleshoot hardware. Don’t ask a backend dev to write a frontend. Don’t ask a proctologist to look at your cough.
You simply cannot be proficient at every sub-sub-specialty. That’s why we collaborate and hand the ‘my computer gets hot’ problems to the hardware people. The alternative would be only moderately useful generalist.
I’m not asking everyone to be able to become a hardware specialist, but if you can’t even figure out “my computer gets hot” I’m not going to be able to trust anything you do. Identifying a heat issue does not take a rocket surgeon.
If it was an nvme ssd i could almost believe it. Some come with totally underspecced heatsinks
I’ve read a story on the forbidden website where a “database” was a single table with a single column holding a single row that contained the actual data as a CSV blob. I’m willing to bet the muskies are not beyond such acts of genius.
It’s terrifying that this is plausible.
We call it New Redis!
He hired a bunch of 19-25 year old. Not experts
There is nothing wrong with being 19-25. There’s something wrong with being wholly incompetent.
There’s not really anything wrong with being incompetent, so long as you have the humility to admit it and learn from people who know better, and try not to cause harm. That’s not Musk’s minions though.
I think it’s important to differentiate incompetence from ignorance. Ignorance is not knowing. Incompetence is not being able to fulfill the requirements for your assigned task. If you cannot fulfill the requirements for your given task, then you should not be given said task.
Hey! Thats offensive to 19-25 year olds, there are many who just finished college/university and are more than aware.
They’re just role playing like in movies, with no idea of the consequences.
How on earth is it offensive to say they’re “not experts”? They’re not prodigies with PhDs. These specific young men are just technical enough and ideologically aligned.
Except they’re not, as you will know their tweet would be false after your first year of any technical (IT oriented) education.
Technical enough to be hired, is all I meant. 🙄
Apologies, if I came over as hostile. I did not get your meaning through text.
Even then, no. These were all obviously nepotism hires who would not have otherwise qualified.
Meh, some of them won some hackathons and scholarships, it’s pretty clear they’re otherwise at least somewhat bright but they don’t have any relevant domain knowledge.
In other words, the type of person most likely to be prone to hubris and catastrophic failures.
First year? That shit is like A+ cert level knowledge or below, and A+ is damn near worthless. They would know that in the first few hours of a study guide
I was being generous when you consider the people in school who somehow pass, even when they don’t know a thing 🥲
Your original comment was ambiguous as to if being an “expert” and “being 19-25” are mutually exclusive.
lol a 19-21 yo isnt going to have a degree lol,
If they went into uni straight out of high school, they could. A lot of Bachelor holders would be around that age, since they start at 18.
Bunch of 1337 hax0rs script kiddies who don’t understand anything but they suck elon’s balls or something idk.
These are the type of people that have deleted the French language from their GNU/Linux system.
You have to understand that the average Trump voter probably knows everything they know about computers from watching the ‘wacky-zaney hacker with personality issues/quirks’ “hack” into things by tippity tapping their fingies on a keyboard in your average copaganda performance.
This is something those types of people will believe.
You’re on the mark. I’m like Help Desk Level 2, I wouldnt even consider myself an actual wizard. The average person in my office thinks I’m Gandalf. Its scary how much these people dont know. And each one of them is out there on the internet.
has Elmo only hired people who claim that they can “wrangle data” and say “yes” ?
There’s two issues going on:
- Elmo’s sociopathic approach to laying people off is public knowledge, and top experts have the luxury of not even applying for his jobs.
- Elmo’s ability to judge engineering talent has likely been wildly exaggerated thanks to how he has successfully bought organizations full of talented people, in the past.
60k rows is generally very usable with even wide tables in row formats.
I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.
After 1M rows move on to something better like Dask, polars, spark, or literally any DB.
The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.
My go-to tool of late is
duckdb, comes with binaries for most platforms, works out of the box, loads any number of database formats and is FAST.
Yes, his Boy Harem.
From the same group that doesn’t understand joins and thinks nobody uses SQL this is hardly surprising .
Probably got an LLM running locally and asking it to get data which is then running 10 level deep sub queries to achieve what 2 inner joins would in a fraction of the time.
You’re giving this person a lot of credit. It’s probably all in the same table and this idiot is probably doing something like a for-loop over an integer range (the length of the table) where it pulls the entire table down every iteration of the loop, dumps it to a local file, and then uses plain text search or some really bad regex’s to find the data they’re looking for.
I think you’re still giving them too much credit with the for loop and regex and everything. I’m thinking they exported something to Excel, got 60k rows, then tried to add a lookup formula to them. Since you know, they don’t use SQL. I’ve done ridiculous things like that in Excel, and it can get so busy that it slows down your whole computer, which I can imagine someone could interpret as their “hard drive overheating”.
Considering that is nearly exactly some of the answers I’ve received during the technical part of interviews for jr data eng, you’re probably not far off.
Shit I’ve seen solutions done up that look like that, fighting the optimiser every step (amongst other things)
I have to admit I still have some legacy code that does that.
Then I found pandas. Life changed for the better.
Now I have lots if old code that I’ll update, “one day”.
However, even my old code, terrible as it is, does not overheat anything, and can process massively larger sets of data than 60,000 rows without any issue except poor efficiency.
They don’t understand joins? How…
60k isn’t that much, I frequently run scripts against multiple hundreds of thousands at work. Wtf is he doing? Did he duplicate the government database onto his 2015 MacBook Air?
60k is laughably, embarrassingly small. It’s still sqlite-sized.
Sqlite can easily handle millions of rows. Don’t sell it short
How about a 6.4TB sqlite database?
Should be enough to hold 60k rows
I’m not
I have an sqlite db that is a few GB in size, game saves using the format. Sadly almost all blob data, would love to play with it if it was a bit more readable
i mean its even excel sized depending on how many columns. This is seriously sad and alarming
Hey now that’s real close to the 65,535 16-bit limit (from 20 years ago)
Holy shit if this ids lm issue that’s too funny
60k is single json file
A TI-86 can query 60k rows without breaking a sweat.
If his hard drive overheated from that, he is doing something very wrong, very unhygienic, or both.
He probably mining crypto on top of running his SQL queries.
What? You don’t run your hard drives in the oven while baking brownies? It makes them zesty.
There must be more join statements than column names
I’ve run searches over 60k lines of raw JSON on a 2015 MacBook air without any problems.
Don’t know what Elmos minions are doing, but I’ve written code at least equally unefficient. It was quite a few years ago (the code was in written in perl) and I at least want to think that I’m better now (but I’m not paid to code anymore). The task was to pull in data from a CSV (or something like that, as I mentioned, it’s been a while) and it needed conversion to XML (or something similar).
The idea behind my code was that you could just configure which fields you want from arbitary source data and on where to place them on the whatever supported destination format. I still think that the basic idea behind that project is pretty neat, just throw in whatever you happen to have and have something completely else out of the other end. And it worked as it should. It was just stupidly hungry for memory. 20k entries would eat up several gigabytes of memory from a workstation (and back then it was premium to have even 16G around) and it was also freaking slow to run (like 0.2 - 0.5 seconds per entry).
But even then I didn’t need to tweet that my hard drive is overheating. I well understood that my code is just bad and I even improved it a bit here and there, but it was still so very slow and used ridiculous amounts of RAM. The project was pretty neat and when you had few hundred items to process at a time it was even pretty good, there was companies who relied on that code and paid for support. It just totally broke down with even a slightly bigger datasets.
But, as I already mentioned, my hard drive didn’t overheat on that load.
Seriously - I can parse multiple tables of 5+ million row each… in EXCEL… on a 10 year old desktop and not have the fan even speed up. Even the legacy Access database I work with handles multiple million+ row tables better than that.
Sounds like the kid was running his AI hamsters too hard and they died of exhaustion.
Excel have a limit of 2^20 rows, something more that 1M. Curious what version of excel are you using for that.
You’re correct - the standard tabs can only hold roughly 1.2 million rows.
The way to get around that limitation is to use the Data Model within Power Pivot:
It can accept all of the data connections a standard Power Query can (ODBC, Sharepoint, Access, etc):
You build the connection in Power Pivot to your big tables and it will pull in a preview. If needed, you can build relationship between tables with the Relationship Manager. You can also use DAX to build formulas just like in a regular Excel tab (very similar to Visual Basic). You can then run Pivot Tables and charts against the Data Model to pull out the subsets of data you want to look at.
The load times are pretty decent - usually it takes 2-3 minutes to pull a table of 4 million rows from an SQL database over ODBC, but your results may vary depending on datasource. It can get memory intensive, so I recommend a machine with a decent amount of RAM if you’re going to build anything for professional use.
The nice thing about building it out this way (as opposed to using independent Power Queries to bring out your data subsets) is that it’s a one-button refresh, with most of the logic and formulas hidden back within the Data Model, so it’s a nice way to build reports for end-users that’s harder for them to fuck up by deleting a formula or hiding a column.
Oh yes, I remember using power query for a few months once I started working with bigger databases, but I saw that moving to Python would be better carrer wise and never came back to excel to do actual work (but at the end everything get exported to excel)
I mean if we were to sort of steelman this thing, there sure can be database relations and queries that hit only 60k rows but are still hteavy as fuck.
I’d do that if I was given so much stupid access
My fucking events table of my synapse DB in postgres is nearly ten times as large, and I ported that from sqlite no long ago, in a matter of minutes. All of the data is on a 2*3 cluster of old 256GB SSDs, equaling about 1.5TB with Raid 0. That’s neither really fast, nor cool. But stable.
deleted by creator
No, its an external drive, appearently.
They even doubled down: https://xcancel.com/DataRepublican/status/1900565922831618202#m
Well thanks for trying to keep us from catching it I guess
thank god, i am going to be ok, i was worried it might be contagious (it’s not, he just has brain damage)
Can you screenshot or something? I can’t load that link
Does this one work? https://nitter.net/DataRepublican/status/1900565922831618202#m Its a bit too long to screenshot.
That works, thanks
So it was a probably shitty external HD… that’s a whole other can of worms
“I store my records on vinyl. You’ve probably never heard of them.”
IT guy checking in.
The only time I’ve even seen drive temp sensor alarms is on server raid arrays and other similar hard drives/SSDs… Never in my life have I seen one available on a consumer device, nor have I seen any alarm for and drive temp, go off. It just doesn’t happen.
IMO, this is one of those language barriers where people call their computer chassis (and everything in it) the “hard drive”.
Applying that assumption, their updated statement is: His computer over heated.
Idk what kind of shit system he’s running on that 60k rows would cause overheating, but ok.
Hard drive? How old is his computer?
Is this a real post? I can’t seemed to find it on that website “X, formerly known as Twitter.”
Yep, it’s real. It was a reply, so maybe that’s why you’re having trouble finding it. Here’s a nitter link to it.
Thanks. I’ve legit never understood how to navigate Twitter. Both Twitter and Snapchat gave me a good dose of what I imagine my grandmother feels when faced with operating a desktop computer…
Haha I totally understand :) Glad I could help!
some poeple have linked the discussion in other comment threads
When the only thing that is stopping kids from dismantling your government is an O(N^N) algorithm
Are you telling me there’s a difference between an inner and a cross join?
Cross join is obviously faster, I don’t even have to write “on”
deleted by creator
Doesn’t actually say that 60k overheated his drive. He says that he ran a run on 60k, and that he couldn’t do the whole database due to overheating. Two unrelated statements except that 60k is the lower bound for what he could process.
Doesn’t mean he knows what he’s doing though, as pretty huge datasets are processable on quite modest hardware if you do it right.
deleted by creator
I don’t want to take away from the valid point of your comment, but rocket science is almost exclusively math
deleted by creator
What in the fuck is this idiot doing? I’ve process datasets far larger than that and never once have I run into a hard drive “overheat”. I mean what level of incompetence do you have to have to get a hard drive to overheat processing a measley 60K rows of data?
I’ve met people who think their desktop PC is the “hard drive” but they’re all over 60.
just a lame-ass excuse for not finding whatever evidence they were looking for.
elsewhere, some seeding was done.
now they’ll do the ‘full’ data grab and ‘find’ what they were looking for.
Yeah. Hiring inexperienced children into government isn’t fraud, by itself. But I bet it makes fraud way easier.
especially if some of the children have a background in cyber crime . this is somehow not a joke , see this video (youtube.com)
60k rows of anything will be pulled into the file cache and do very little work on the drive. Possibly none after the first read.
Not if each row is pi!
You can put 60k rows in Excel 95.
