The entire second on chess engines is, from the view of the entire thesis of TFA, is incoherent. Let's assume, for sake of argument, that I agree with the section: that an idealized chess move predictor isn't a predictor — which is not a thing that exists, as the space of chess is enormous, but let's pretend! — that's not what LLMs are? Even if we just restrict ourselves to the space of written English prose, the space is quite literally infinite. So, hopefully obviously, no LLM is comparable to an idealized chess engine. Similarly, incoherently, we wave away the "make_more_likely", when, at least to me, the entire meat of that argument would be in the reward function, and we just gloss over that entirely.
(I would also agree with the parent commenter on that the writing smells like an LLM.)
The reward function seems indeed to be the protagonist there, still it stays in the shadows. One can only imagine that it is some kind of evaluator that scores the sequence based on grammar correctness, semantic consistency, etc.? To use the proposed chess analogy, maybe it could be a Stockfish engine that evaluates the submitted position that results from the move submitted by the LLM?
Yes, but also no. If I prompt a graphics artist, "draw me a picture, any style, of a calendar with this day circled in red", you do not have to tell them what a calendar looks like, what the days of the week are, or how many there are, or that days in a month increment sequentially.
Also, you probably don't need to tell an artist "don't violate other people's copyrights" while you're at it, though that's perhaps somewhat more debatable than "does the artist know that the day after Thursday is Friday".
This applies equally to all disciplines: AI generated code regularly contains wtfisms that a human would not need to be guided from, or at the very least (more similar to the copyright problems) that experience and knowledge would drive them away from — and permit me to entrust, particularly more experienced — humans with a vague outline of the idea, and trust that the details will get filled in sensibly. "Filling in details sensibly" is where AI hallucinates the hardest.
Some of us have been here longer than AI. "To be intellectually stimulated" would be the why, which there used to be more of prior to the frontpage becoming riddled with AI slop. "HN is for conversation between humans" as the guidelines say, and though the guidelines imply that's towards comments, I'd prefer it to be towards submissions too, for the same reasons it was added for comments.
I'm already at the point of "the reward/effort of HN is getting pretty low", but the question is, where to leave for?
> SHOULD — This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
You'll also need RFC 6919[1] for "OUGHT TO":
> The phrase "OUGHT TO" conveys an optimistic assertion of an implementation behavior that is clearly morally right, and thus does not require substantiation.
And the up-threads gut feel that "ought to" is slightly stronger comes across there, I think: ought to conveys that moral stance, where as "should" is more "there's a decision here, possibly".
[1]: https://www.rfc-editor.org/rfc/rfc6919.html; n.b., this is an April 1st RFC. I like it, though, and it isn't uncommon to see a stray "ought to" lacking a citation in a non-April-1st RFC.
My understanding is that they’re generally treated as homonyms in most cases but some philosophers like to use OUGHT in a moral sense and SHOULD as practical advice to achieve an OUGHT.
Hence OUGHT leads to SHOULD and the latter is meaningless without the former.
E.g.: You OUGHT to secure systems that handle sensitive customer data, so you SHOULD regularly patch your software.
Just telling someone they should do something without an initial moral motivation to do the “right thing” can fall on deaf ears.
I’ve seen an official patch management strategy in a government organisation that was simply: “Don’t”.
Their moral argument was that they OUGHT to save taxpayer money and so they SHOULD save time, money, and effort by not applying updates (which they felt weren’t necessary for security).
In other words: OUGHTs are like axioms that you can choose, and then the SHOULDs derive from them automatically.
my point is that as programmers we have to take into account that the average human today has a computer that likely struggles to open Win11 notepad. I don't buy random laptops from amazon, but an astonishingly high amount of people do and we have to work with that
> The article posits that they may just have centralized monitoring, not control.
… the article does posit this … nonsensically. TFA also establishes that "RMCS" stands for "Remote monitoring and control system". I would think the "and control" portion of the acronym would indicate that it is centralized monitoring … and control.
> I noticed this when a mozilla dev said I must use systemd+pulseaudio
I am 99% sure Firefox will run just fine without systemd, or OpenRC half of Gentoo would be with pitchforks and torches.
It isn't correct, either, that it requires PulseAudio; I'm running FF with PipeWire, for example. But perhaps let's steelman that to "does not work with ALSA". I'm not sure what the objection to PA would be. It appears you can run FF under an adapter that will backport FF's PA output to ALSA — so it is possible to do —, I'm just not fathoming what the objection to Pulse is. (Or for that matter, its successor, PipeWire.)
While I suppose this doesn't say the design isn't secure, system() is one of those calls that has no place in modern code. It is insecure by its design.
> Any user input that is employed as part of `command` should be carefully sanitized, to ensure that unexpected shell commands or command options are not executed. Such risks are especially grave when using system() from a privileged program.
The "system" function (and also "popen" function) are helpful, although they should be used carefully. You should only use it where the entire input (rather than merely a part of it) comes from a trusted source from the local user (and documented in a clear way that it does this), such as being entered interactively or from a user configuration file, or in some cases a entirely hard-coded string (although in such a case, often one of the exec functions works better), and to ensure that the security boundary is correct (e.g. you should probably avoid it if it is setuid). If it is necessary to pass additional data then you might use environment variables, pipes, temporary files, etc.
Strings passed to system or popen should not be constructed by combining other strings; they should be directly unchanged from whatever trusted source it comes from.
The specific use documented in the article is a situation where I should think that you should not call the shell (since the command includes untrusted input, and also because there might be a better way to display the error message).
There are additional possible security issues with such things though, whether you use the shell or execute directly, some of which are due to the use of text rather than binary data for communication (although changing that won't solve everything).
Well, so there's two other problems with system (and popen):
1. IME security teams want to run dumb linters over the code that look for such things. While uses such as the ones you describe are secure, the linter might be blunter than that. I'm not sure that's necessarily a bad thing: "this use of system(3) is secure" has a cognitive tax at review time, and as the code changes; often, I prefer the stance of "don't make me think" with regards to security: i.e., do the simple, trivially secure thing, not the complex, secure under the just-right conditions thing. Then we don't have to persuade (potentially non-technical) security teams, non-technical auditors, linters, etc. that are going over the code with blunt instruments.
2. (And much more minor) the entire execution of sh is often just wasted performance that an exec(2) removes.
(& yes, you are also right that even an exec(2), mishandled, can have its problems in some circumstances. But typically when the pattern is system("<shell that just runs what we'd exec(2)>"), then whatever those problems are, system(3) is going to share them.)
> My viewpoint is that we should eventually have AI be assisting in determining if a search is reasonable in the moment.
That is truly bananas. Whose AI? Trained on what data? We've had numerous studies show how AIs can be biased, and a police dept. is going to choose whatever AI "works best for them" — i.e., keeps most out of their way.
You can see Flock for examples here: despite requiring justification for searches, cops are entering things like "hehehe"[1]; the AI would similarly be given bad or falsified inputs.
> If there is an actual constitutional violation that that happens it can be resolved after the fact.
The entire point of the Constitution is to prevent exactly that. There is no resolution for violating someone's rights; you cannot merely go back in time and unviolate them. The point is to be free from unreasonable searches & seizures.
Your comments read like police have no ability to search without a warrant, which isn't true. If circumstances warrant it, a warrant from a judge is not required, and I'm sure a good number of people would argue that that's already true more often than it ought to be.
Part of the point of having a separate branch of government involved, too, is to prevent the first branch from just deciding "why, of course I'm right. What else would I be?", and to evaluate the facts separately (which the police cannot by definition do).
> I believe every part of the law enforcement and legal system needs to be systematically reevaluated finding ways to increase efficiency.
I'm all for increased efficiency, but AI is not that. "Slop, faster" is not the same "the correct action, while respecting the rights of the citizens, faster".
>Whose AI? Trained on what data?
The government. Trained on data to maximize accuracy and minimize constitutional violations.
>We've had numerous studies show how AIs can be biased, and a police dept. is going to choose whatever AI "works best for them"
Violating the constitution means that evidence gets thrown out in court and the victims get financially rewarded. It is not in the police's interest to violate people's rights.
>the AI would similarly be given bad or falsified inputs.
And then those cops should be severely punished for intentionally violating people's constitutional rights.
>There is no resolution for violating someone's rights
There is already resolution for having your constitutional rights violated. You sue for damages and to have them stop violating your rights.
>no ability to search without a warrant, which isn't true. If circumstances warrant it
There are other situations where it doesn't warrant it which is what we are talking about here. Refer to TFA.
>Part of the point of having a separate branch of government involved, too
And I'm pointing out that this involvement is not needed by the constitution. Branches of government is an arbitrary degree of separation.
> Violating the constitution means that evidence gets thrown out in court and the victims get financially rewarded. It is not in the police's interest to violate people's rights.
Victims get financially rewarded by the state and the officers involved receive no kind of punishment at all. It's not hard to see why incentives lead to this already being widespread, and that's just what we know about when it gets thrown out or overturned. There's a whole shady practice of parallel construction going on.
And you are not even addressing how easily a model could hide all kinds of special treatment - either way - at the whim of its creators, be they the government or some outside "trusted" party
what the judge sees is all public record, and can be appealed up the chain, and ultimately overridden by a pardon.
for a model, we would need all the training data, training method, fine tuning, etc to all be public record, along with all of the operations at inference time.
Indeed they can but they are in some ways like an individual model per judge, albeit with a sub-optimal interface and very slow, but at least they are hard to game in mass.
(I would also agree with the parent commenter on that the writing smells like an LLM.)
reply