Since getting laid off in May and failing to find any jobs for ML in healthcare, I am working with a friend I met during my MPH to start a boutique consultancy to help hospitals deploy AI / health technology.
You can still do that with AI. You hire 1 accountant to use AI to do the work of 20, require them to sign off on all of the work, and yell at them, before firing them, and then hiring an even less experienced one to manage the work of 50.
It's hard to describe, but it's felt like LLMs have completely sucked the entire energy out of computer vision. Like... I know CVPR still happens and there's great research that comes out of it, but almost every single job posting in ML is about LLMs to do this and that to the detriment of computer vision.
To me its totally obvious that we will have a plethora of very valuable startups who use RL techniques to solve realworld problems in practical areas of engineering .. and I just get blank stares when I talk about this :]
Ive stopped saying AI when I mean ML or RL .. because people equate LLMs with AI.
We need better ML / RL algos for CV tasks :
- detecting lines from pixels
- detecting geometry in pointclouds
- constructing 3D from stereo images, photogrammetry, 360 panoramas
These might be used by LLMs but are likely built using RL or 'classical' ML techniques, tapping into the vast parallel matmull compute we now have in GPUs / multicore CPUs, and NPUs.
I thought there been a lot of progress in last 2 years. (Video) Depth Anything, SAM2, grounding Dino, DFINE, VLM, Gaussian splats, Nerf. Sure less than progres in LLm but still I would say progress accelerated with LLM research.
You said :
"- detecting lines from pixels
- detecting geometry in pointclouds
- constructing 3D from stereo images, photogrammetry, 360 panoramas"
==> For me it is more something like :
Source = crude video-or-photo pixels (to) ===> Find simple many rectangle-surface that are glued together one another.
This is, for me, how you really go easily to detecting rather complexes geometry of any room.
I kind of did a version of what you suggest - I think I linked to a video showing plane edges auto-detected in a pointcloud sample.
Similarly I use another algo to detect pipe runs which tend to appear as half cylinders in the pointcloud, as the scanner usually sees one side, and often the other side is hidden, hard to access, up against a wall.
So, I guess my point is the devil is in the details .. and machine learning can optimize even further on good heuristics we might come up with.
Also, when you go thru a whole pointcloud, you have a lot of data to sift thru, so you want something fairly efficient, even if your using multiple GPUs do do the heavy matmull lifting.
You can think of RL as an optimization - greatly speeding up something like monte carlo tree search, by learning to guess the best solution earlier.
I feel like 3D reconstruction/bundle adjustment is one of those things where LLMs and new AI stuff haven't managed to get a significant foothold. Recently VGGT won best paper which is good for them, but for the most part, stuff like NERF and Gaussian Splatting still rely on good old COLMAP for bundle adjustment using SIFT features.
Also, LLMs really suck at some basic tasks like counting the sides of a polygon.
>but almost every single job posting in ML is about LLMs
not in the defense sector, or aviation, or UAVS, automotive, etc. Any proper real-time vision task where you have to computationally interact with visual data is unsuited for LLMs.
Nobody controls a drone, missile or vehicle by taking a screenshot and sending it to ChatGPT and has it do math while it's on flight, anything that requires as the title of the thread says, spatial intelligence is unsuited for a language model
It felt the same back in 2012-2015 when deep learning was flooding over computer vision. Yet 10 years later there is a net benefit for computer vision: a lot of tasks are now solved much better/more efficiently with deep learning including those that seemed "unfit" to deep learning like tracking.
I'm hopeful that VLMs will "fan out" into a lot of positive outcomes for computer vision.
That is fair. I think it is a case of just seeing a lot if great talent rush to the "in" thing. Other systems are still being developed and that isnt lost but there is just a feeling if being left out of it all while still doing great stuff.
There is nothing to productize vs LLMs right now. I would say robots could fix that but they have hard problems to solve in the physical sense that will bottle neck things
agreed about sucking the air out by LLM. The positive side is that its a good time to innovate in other areas while a chunk of ppl are absorbed in LLMs. A proven improvement in any other non LLM space will attract investment.
I'm a avid (hobbyist) photographer and I've noticed a TON of genuinely good 3rd party lenses (primarily Sigma and Tamron) and even 'fine' lenses at rock bottom prices (Viltrox, 7Artisans, TTArtisans, etc) for like $250. The conventional wisdom I've heard is that computer aided design has totally revolutionized this field.
I can only hope that projects like these help build better lenses for the future.
At the *very* least you should be following the administrative rules act (requiring you to solicit 45 days for comments by effected parties) before making such a dramatic change in policy.
Courts absolutely love striking down EOs (of both Dems and Reps Admins) when they should have been following the administrative rules act.
You can file an APA lawsuit about anything. Nobody really calls APA violations “illegal.” It’s a “show your work” and “don’t be drunk or crazy” procedural law.
Completely off topic to the actual article - but...
I really get irked a lot by seeing (overly obvious) ai generated images being used for stock photos. One of the reasons my wife and I still subscribe to a couple of newspapers is that the photography helps bring to life the story being told. Why where these photos taken and how do they impact the story. You don't get that same visceral emotional reaction with low quality cartoon images.
I mean tons of retrospective studies are literally "IRB exempt" (and highlighted as such in their methods) where no explicit consent is required or needed. Physicians doing case reviews don't need consent of the patient this work. Doing retrospective analyses for defining clinical phenotypes on patient data that has been aggregated isn't needed either. Collecting clinical MRIs to do deep learning doesn't require an IRB.
IRBs are only REALLY required when you are intervening in patient care or pose some theoretical risk to a human.
Some institutions still want you to submit approval for institutional data, but as a non-lawyer it seems that's much more of a CYA policy.
> Collecting clinical MRIs to do deep learning doesn't require an IRB
Yes it does, but typically the IRB will waive consent and waive notification for those sorts of studies if the images can be de-identified. There's also HIPAA involved which may or may not require establishing a BAA depending on what's being done if the images can't be de-identified. This is particularly an issue with brain MRI because it is usually trivial to generate an image that can be compared to full-face photographs (i.e. can be compared to drivers license/passport type photos to reestablish identify).
And longer term you're not getting anything even in the door at the FDA without an IRB and you're not selling anything without FDA approval.
Also please note that MRI is a Class II regulated device that deposits energy in subjects/patients so it doesn't qualify for a lot of the exemptions (early last year the FDA granted IRBs the ability to do things like wave full written consent for minimal risk research for FDA-regulated research).
I've worked in MRI AI both in academic and non-academic centers, including technology that has received 510K clearance.
I admit it sounds pedantic, but I'm not discussing IRB *exemptions* that are sometimes required by an institution nor am I discussing BAAs. I was specifically talking about the specific IRB applications (which I've submitted and signed before) that the blog author was talking about. Yes, HIPAA and other state and local regulations also govern what you can and cannot do with the data, but that's not what I argued.
Sorta off topic but the FDA doesn't care so much about SAR unless you're directly programming the MR machine's pulse sequence. If you're just doing quantification of some brain structure for monitoring a biomarker, they're primarily concerned if your product 1) matches an existing prerequisite and 2) functionality that your product achieves performance that you say it does. That is why the marketing around most of the early DL / AI based radiology startups were focused on language for "study prioritization" rather than more specific claims.
MRI are Class II devices and until last year exemptions for IDE applications and for not obtaining written informed consent specifically excluded devices that by design or intent deposit energy in humans (regardless of overall study risk. Last year the FDA finally extended some power to IRBs to harmonize with HHS research regulations which have allowed IRBs to waive consent for minimal risk studies for a long time. But FDA directing you to an IRB rather than both an IRB and the FDA itself doesn't mean you're not dealing with an IRB.
FDA and human subject protections come from different laws with different legislative authority. The regulations are not the same except to the extent that the agencies themselves work to harmonize things. If you are doing anything covered by FDA you must follow FDA's regulations in addition to any other applicable human subjects research regulations. And because MRI scanners are Class II regulated devices it means that people are being scanned with a doctor's permission, an IRB's permission or the FDA's direct permission.
FDA "doesn't care" about SAR to the extent that they have published guidance that if you operate an approved MRI scanner within normal operating mode (which are settings defined by IEC that do not necessitate medical supervision), then the FDA will not automatically consider use of the scanner itself to elevate a study's risk (in the way that using something like a CT scanner with ionizing radiation would). Risk determination goes beyond whether or not the MRI itself is a risk though. For example a research study that diverts patients to MRI in a way that delays care in an emergent situation (say testing experimental sequences for stroke detection) is unlikely to overall qualify as minimal risk even if the scanner operates in normal mode because of other non-MRI risks associated with the study procedures.
Retrospective use of de-identified or anonymized medical records that already exist are of course a different thing because the risks to the patient are primarily privacy risks.
And you are correct the actual FDA labels of all the AI crap that's coming out are jokes compared to what a lot of sales bullshitters promise. But you better believe all the data submitted to the FDA by the MRI manufacturers support their accelerated acquisitions that use deep learning recons follow FDA's clinical trials regulations.
I don't disagree with anything you're saying necessarily, but a lot of people are conflating my statement of IRB exemption with having an explicit IRB authorization. I guess more to my point is people seem to be failing to understand the role of retrospective research (and I will easily concede that different institutions have different legal interpretations) and how it's an important part of research. Don't get me wrong, you absolutely still have regulations about what you can and cannot do with that data, but saying you need an IRB for everything doesn't match the reality that I've seen first hand.
That said, there's plenty of buying and selling of radiological images for industry development on the second hand market. Now where the line of "research" vs. "industrial" work is, well that's something I would leave to legal council. But as you said any sort of "altering" of clinical outcomes is a clear IRB is required zone like DL based recon.
I have published multiple papers in peer review journals where we were not covered by IRBs. That is not to say other research ethics standards don't exist, but that IRBs are simply one portion of the healthcare research regulations.
Sure, there are also cases where it is not needed. But some called out in GP were not typically correct. And often it is opaque to the researchers only because someone did an IRB in an earlier step with usage that covers sharing it with you.
I’m a radiologist, and I’ve worked on these sorts of research studies before. You absolutely do need an IRB to do deep learning on clinical MRIs. I’ve had to write IRBs just to collect retrospective MRIs. Even for studies that are potentially IRB exempt, going through the exemption request takes hours of work on my part and weeks to months before the IRB grants the exception.
As other commenters mention, the “theoretical risk to a human” encompasses nearly all research. For what I do, any imaging studies that aren’t de-identified pose a theoretical risk to patient privacy. If you try to de-identify images, you learn that this is nearly impossible. Sure, you can try your best to scrub DICOM headers, but these headers are mis-used by vendors, so identifying information can appear almost anywhere. You could delete the headers entirely, but then you lose a lot of metadata that you may need to properly display the images. Further, people contend that you can identify individual peoples’ faces if you 3D-reconstruct CT/MR images, so then you have to expend resources to delete faces from all head/neck/brain imaging. Confirming that this was done properly requires manual review and limits the size of your dataset.
Edit: I think the disagreements here are partly due to institutional differences in IRB requirements and partly due to conflating “IRB exemptions” with the idea of not having to interact with the IRB at all. You always have to interact with the IRB—even just to obtain an exemption. While obtaining IRB approval is a cumbersome process, obtaining IRB exemption is only slightly less cumbersome. I’m sure this varies across institutions, but I’ve been at three different large urban academic centers, and obtaining exemptions has been a multi-month process at all three.
>IRBs are only REALLY required when you are intervening in patient care or pose some theoretical risk to a human.
In practice, sure, but that's not what the spaghetti of rules that one is compelled by law to follow state. This disconnect is the whole point of the article.
>IRBs are only REALLY required when you are intervening in patient care or pose some theoretical risk to a human.
Theoretical risk is a very broad category. IRB approval is required to download de-identified human genome sequences, in part because the genome sequence can be used to identify the individual. Drawing blood from an individual, or using blood that has been drawn for other purposes, requires IRB approval. And IRB approval is required for many sociology studies (or surveys), in part because publishing an individuals answers to a survey might put the subject at risk.
So the "theoretical risk" threshold can be very low. It's theoretical after all, so the harm need only be imaginable.
I am in a lab that does EEG research and have recently been doing research using self-supervised learning on EEG. We had to get approval for "secondary use of data" to use open datasets from https://openneuro.org/ and we had to jump through a bunch of hoops with our IRB. It was harder to get it approved than the EEG experiments we run ourselves for some reason.
My wife and I went to Tofino (on Vancouver) this last summer where you can rent a boat for a tour of the coastal black bears. Very highly recommend it.
(As a local) it sounds weird to say "on Vancouver" without the island part. Vancouver means the city. If you want to sound cool you can say "the Island".
Super cool! I love Vancouver island - normally visit Campbell River where I used to have family. Always wanted to make it to the west side for Tofino or the West Coast Trail.
