[lug] Speed up osm2pgsql

[Alan Robertson]

I had a similar thought...
-- 
  Alan Robertson
  alanr at unix.sh

On Thu, Feb 8, 2018, at 1:17 PM, Steve Sullivan wrote:
> Hi Lee,
> 
> Not sure if you want to go this direction, but on ebay
> there are a lot of big servers for sale that could
> put the whole problem in main memory, ending swapping.
> 
> For example, Dell R910 6 core with 256 GB RAM for about $800.
> 
> Steve
> 
> On Thu, Feb 08, 2018 at 12:08:35PM -0700, Lee Woodworth wrote:
> > Populating a postgresql database for open street map uses 230GB
> > of virtual address space. The swap device is very active with
> > lots of writes at an average rate of 20MB/s.
> > 
> > The source parsing phase was going to take 2+ weeks using a
> > regular disk for swap. Changing to an ssd speeds things but
> > writes 1.4TB (actual device writes) out of a 75TB TBW. The
> > wear level reported by smartctl went from 0 to 10 just for
> > for this phase.
> > 
> > The post-processing phase is going very slow during a database
> > sort where the db is on a raid1. Does anybody know of knobs to
> > tweak to speed up larger-than-memory table sorts (100GB)? Or do we
> > just need to use an ssd and bite the bullet on having to replace
> > them frequently?
> > _______________________________________________
> > Web Page:  http://lug.boulder.co.us
> > Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> > Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
> 
> -- 
> 
> ========================================
> Steve Sullivan      steve.sullivan at mathcom.com
> 720-587-7498        http://www.mathcom.com
> ========================================
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety