I share your frustration and agree with a lot of your points, and in fact I was motivated to solve a lot of these problems while in med school. My thought was that all you need is technical expertise on healthcare data to revolutionize the field. We have the technical expertise, but we just don't have the healthcare data for several reasons.
First, patient privacy laws (while a net good) scare institutions from sharing high quality data. The best you'll get is small batches of de-identified data released infrequently. Patient notes are unlikely to ever be released in large quantities since they can so easily pinpoint some patients.
Second, you need to coordinate thousands of physicians and/or healthcare facilities across the US (or world) to record data on their own performance in a standardized way. Many hospitals do this on some agreed upon metrics (30-day readmission rate, hospital-acquired pneumonia rate, average HbA1c level for a doctor's diabetic patients etc.) largely because they're used to determine government funding/penalties. But at the end of the day, there's no direct incentive for physicians or institutions to collect any other data on their own performance and release it publicly. In fact, there are more risks to doing this than benefits. To solve this problem you need to tie hospital funding with requirements to collect and publicly share performance data while also mitigating punishment.
To physicians' credit, many of us are actually motivated to at least privately collect data on our own performance so that we can improve. But this is incredibly difficult and time consuming—especially for those of us who come into contact with dozens and dozens of patients every day. Sure, better data collection tools would dramatically help us monitor our own metrics, but the only entity with the cash to purchase or create these tools is the hospital, and its reply is going to be, "What's the ROI?" And the answer is honestly probably negative. You may suggest buying/building small relatively inexpensive tools (as I've personally tried), but the hospital isn't interested. Like most large enterprises, hospitals want long-term contracts, dedicated support teams, and tried and true tools. Small tools pose too much of a security risk and maintenance headache.
First, patient privacy laws (while a net good) scare institutions from sharing high quality data. The best you'll get is small batches of de-identified data released infrequently. Patient notes are unlikely to ever be released in large quantities since they can so easily pinpoint some patients.
Second, you need to coordinate thousands of physicians and/or healthcare facilities across the US (or world) to record data on their own performance in a standardized way. Many hospitals do this on some agreed upon metrics (30-day readmission rate, hospital-acquired pneumonia rate, average HbA1c level for a doctor's diabetic patients etc.) largely because they're used to determine government funding/penalties. But at the end of the day, there's no direct incentive for physicians or institutions to collect any other data on their own performance and release it publicly. In fact, there are more risks to doing this than benefits. To solve this problem you need to tie hospital funding with requirements to collect and publicly share performance data while also mitigating punishment.
To physicians' credit, many of us are actually motivated to at least privately collect data on our own performance so that we can improve. But this is incredibly difficult and time consuming—especially for those of us who come into contact with dozens and dozens of patients every day. Sure, better data collection tools would dramatically help us monitor our own metrics, but the only entity with the cash to purchase or create these tools is the hospital, and its reply is going to be, "What's the ROI?" And the answer is honestly probably negative. You may suggest buying/building small relatively inexpensive tools (as I've personally tried), but the hospital isn't interested. Like most large enterprises, hospitals want long-term contracts, dedicated support teams, and tried and true tools. Small tools pose too much of a security risk and maintenance headache.