Defcon had a great talk on all the different navigational systems pilots can use and a note at the end that these shouldn't be decommissioned at the rate they're experiencing atm https://www.youtube.com/watch?v=wSVdfOn737o
From what I've seen in news reports, China has built a lot of tower blocks that are 10s of floors, rather than the 4-5-floor buildings I saw when I worked in and travelled around India.
India has ~4x the population of the US so the ratio of buildings isn't much of a surprise.
The US has only used 25% of its land. Overture published a land use dataset a while back that could go some way to verify how much land on earth is urban, covered in forest, etc.. That might only need a single SQL statement given how they structured their data.
That 5 TB of data will probably be 3-400 GB in Parquet. Try and denormalise the data into a few datasets or just one dataset if you can.
DuckDB querying the data should be able to return results in milliseconds if the smaller columns are being used a better if the row-group stats can be used to answer queries.
You can host those Parquet files on a local disk or S3. A local disk might be cheaper if this is exposed to the outside world as well as giving you a price ceiling on hosting.
If you have a Parquet file with billions of records and row-groups measuring into the thousands then hosting on something like Cloudflare where there is a per-request charge could get a bit expensive if this is a popular dataset. At a minimum, DuckDB will look at the stats for each row-group for any column involved with a query. It might be cheaper just to pay for 400 GB of storage with your hosting provider.
I would love to have an air conditioning / cooling solution that is directly linked to solar panels with no batteries involved. Like the sun shines, we get electricity, we do the work. My main goal for this thought experiment is to come up with uses of solar electricity that is resilient to the unpredictable and unreliable energy generation from solar. Thoughts?
Supposedly water heaters are an amazingly good target for variable power. Water heating takes 18% of home energy use[0] and is already a well insulated storage device. Just heat it up as you are able with the day's sun.
If you are looking for a cooling solution, you could go the other way and make water chillers through a dedicated water tank. You would tie the HVAC to pipe air through a heat exchanger. Seems like all of that is well established engineering.
> I would love to have an air conditioning / cooling solution that is directly linked to solar panels with no batteries involved.
I think that would be very viable with fridges, that represent a large share of electricity consumption among the poorest. Before electricity, people powered fridges by constantly buying ice blocks. They were just isolated boxes where food was stored together with the blocks. Perhaps it's just necessary to go back at the roots, and make fridges that take energy from solar panels and generate a lot of ice by day, and uses it to keep cold at night, with no need for batteries.
I love this idea but I don't know enough about the specifics. Isn't it really bad(TM) for the pump or compressor or something I don't know about for the input power to be variable like this? Like there might be an errant cloud somewhere.
The whole point of my thought exercise is to see if we can somehow make the cost go down. My understanding is that the panel can easily last twenty five years but the battery you'd be lucky to go beyond eight?
Edit: good news / bad news
Bad news: this is not an original thought
Good news: smarter people than me are already working on this. See solar Variable Frequency Drive (VFD). Basically, my thought is a pump is a pump. if you can build a pump to pump water to irrigate poppy fields, you can use the same pump to drive refrigerant in a refrigerator/ freezer / heat pump.
> I love this idea but I don't know enough about the specifics. Isn't it really bad(TM) for the pump or compressor or something I don't know about for the input power to be variable like this?
No, modern refrigerators and other white box appliances with variable speed motors use electrically commutated (aka EC or brushless) motors that allow for motor speed control. Larger three-phase induction motors can have their speed controlled by a variable frequency drive (VFD).
> Basically, my thought is a pump is a pump. if you can build a pump to pump water to irrigate poppy fields, you can use the same pump to drive refrigerant in a refrigerator/ freezer / heat pump.
You’re close, but it’s more ‘a motor with enough power can drive any pump (or fan)’ than ‘a pump is a pump is a pump’, as there are many different kinds of pumps for various working fluids (water, glycol, oil, refrigerant, etc)
Regarding the fully solar powered A/C, you can smooth out power generation and consumption using capacitors (aka batteries)
Though experiment? This is already a thing in the off grid community. In practice you need at least small battery to smooth out the power, but it doesn't take much home automation to kick on the mini split when the panels/inverters have power to spare.
There are ~15 GB of SAR imagery at the bottom being rendered as is from GeoTIFF files. On my 2020 MBP rendering that amount of data in QGIS would lag without building mosaics and tiles.
The Parquet pattern I'm promoting makes working across a wide variety of datasets much easier. Not every dataset is huge but being in Parquet makes it much easier to analyse across a wide variety of tooling.
In the web world, you might only have a handful of datasets that your systems produce so you can pick the format and schemes ahead of time. In the GIS world, you are forever sourcing new datasets from strangers. There are 80+ vector GIS formats supported in GDAL. Getting more people to publish to Parquet first removes a lot of ETL tasks for everyone else down the line.
70% of daytime RGB sat imagery is covered by clouds. I'm not sure how easy it would be to spot if clouds were covering a city's lights at night.
I've only seen Maxar publish one night time image and that was of Dubai. I suspect smaller buildings in not so well lit areas could end up getting missed out.
SAR imagery would work well for seeing at night and through clouds but I'm not sure what the state of AI building footprint detection is with SAR atm.
Amazon, Esri, Grab, Hyundai, Meta, Microsoft, Precisely, Tripadvisor and TomTom, along with 10s of other businesses got together and offer OSM data in Parquet on S3 free of charge. You can query it surgically and run analytics on it needing only MBs of bandwidth on what is a multi-TB dataset at this point. https://tech.marksblogg.com/overture-dec-2024-update.html
As someone who works with mapping data for HGV routing, I've been keeping an eye on Overture. I wonder do you know if anyone has measured the data coverage and quality between this and proprietary datasets like HEREmaps? Does Overture supplement OSM road attributes (such as max height restrictions) where they can find better data from other sources?
I haven't done any deep dives into their road data but there was ~80 GB of it, mostly from TomTom, in the August release. I think the big question would be how much overlap there is with HERE and how would the metadata compare.
If you have QGIS running, I did a walkthrough using the GeoParquet Downloader Plugin with the 2.75B Building dataset TUM released a few weeks ago. It can take any bounding box you have your workspace centred on and download the latest transport layers for Overture. No need for a custom URL as its one of the default data sources the plugin ships with. https://tech.marksblogg.com/building-footprints-gba.html
Thanks for the response. There must be value in benchmarking data coverage and quality for routing data such as speed limits, vehicle restrictions, hazardous cargo etc... . I guess the problem is what do you benchmark against.
Shapefiles shouldn't be what you're after, Parquet can almost always do a better job unless you need to either edit something or use really advanced geometry not yet supported in Parquet.
The Airport in Tartu, Estonia had a navigation upgrade last year in order to help mitigate navigation jamming that's taking place in the region. https://www.eans.ee/en/uudised/tanasest-saab-tartu-lennuvalj...
Defcon had a great talk on all the different navigational systems pilots can use and a note at the end that these shouldn't be decommissioned at the rate they're experiencing atm https://www.youtube.com/watch?v=wSVdfOn737o