2016-01-20

Author: MacHu-GWU

.gitignore

@@ -4,13 +4,15 @@
 
 # File
 *.pyc
+*.log
+*.csv
+*.txt
+*.sqlite3
 
 # Directory
 build
 dist
 uszipcode.egg-info
-dataset
-prepare_dataset
 
 # =========================
 # Windows image file caches

dataset/federalgovernmentzipcodes/about.rst

@@ -0,0 +1,43 @@
+The data is originally from http://federalgovernmentzipcodes.us/, last updated on ``2012-01-22``.
+
+About the data fields:
+
+- Updated often, but not regularly.
+- 81,831 rows of data.
+- All 41,891 active zipcodes + 634 decommisioned zipcodes from the recent past.
+	All 80673 active Primary(41885), Acceptable(13988), and Not Acceptable(24800) placenames. Some additonal- placenames for decommisioned codes.
+- 29,971 Standard, 9465 PO BOX, 2437 Unique, and 649 Military codes.
+- 50 States +
+- 361 AA Military - Americas
+- 38 AE Military - Europe
+- 164 AP Military - Pacific
+- 1 AS American Samoa
+- 290 DC Washinton DC
+- 4 FM Federated States Micronesia
+- 13 GU Guam
+- 2 MH Marshall Islands
+- 3 MP Northern Mariana Islands
+- 176 PR Puerto Rico
+- 2 PW Palau
+- 16 VI Virgin Islands
+
+Sources:
+
+- Current zipcodes, placenames, zipcode type(Standard, PO, Unique, Military), placename type (Primary, Acceptable, Not Acceptable): USPS
+- Military placenames (base or ship name): MPSA 2008 Election Ballot information
+- Tax returns filed, estimated population, total wages: IRS 2008
+- Latitude and Longitude; National Weather Service supplemented by Google Earth and Maps and occasionally other sources
+- Decommisioned zipcodes, Our old database--usually quality sources, but not verifiable.
+
+Other Sources of zipcode information:
+
+- Placenames (Cities, towns, geographic features) can be found at US Geological Survey GNIS Dataset
+- The IRS has additional data fields for 2008 and is reviewing their publication procedures for later years. see http://www.irs.gov/taxstats/indtaxstats/article/0,,id=96947,00.html
+- The Census publishes data, but they use Zipcode Tabulation Areas (ZCTAs) which 1) have changed areas between the 2000 census and the 2010 census 2) do not map well to USPS zipcodes well. If needed http://www.census.gov/geo/ZCTA/zcta.html
+- Social Security recipients by zipcode http://www.ssa.gov/policy/docs/statcomps/oasdi_zip/
+- For economic researchers and those who want tons of background on data sources by zipcode, University of Missouri OSEDA project
+
+Appendix:
+
+zipcode wiki: http://en.wikipedia.org/wiki/ZIP_code
+zipcode FAQ: http://www.zipboundary.com/zipcode_faqs.html

dataset/federalgovernmentzipcodes/federalgovernmentzipcodes.zip

@@ -0,0 +1,2 @@
+This is geocoded data for all address in free-zipcode-database-Primary.csv.
+Update on 2016-01-20

dataset/geocoded-data/about.rst

@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+extract federalgovernmentzipcodes.zip and zcta2010.zip, find 
+free-zipcode-database-Primary.csv and zcta2010.csv, put it with this script.
+
+And run step1, step2, step3, then you get the zipcode sqlite database for 
+uszipcode 0.0.8.
+"""
+
+from __future__ import print_function
+from geomate.tests import GOOGLE_API_KEYS
+from pprint import pprint as ppt
+import pandas as pd
+import geomate
+
+df = pd.read_csv("free-zipcode-database-Primary.csv", dtype={"Zipcode": str})
+todo = df["Zipcode"].tolist() # 42522 zipcode
+
+googlegeocoder = geomate.GoogleGeocoder(api_keys=GOOGLE_API_KEYS)
+googlegeocoder.set_sleeptime(0.1)
+batch = geomate.BatchGeocoder(googlegeocoder, db_file="geocode.sqlite3")
+batch.process_this(todo, shuffle=True)

dataset/step1_geocoding.py

@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+This script is to construct a zipcode sqlite database for uszipcode extension.
+"""
+
+from __future__ import print_function
+from pprint import pprint as ppt
+from sqlite4dummy import *
+import pandas as pd
+import json
+
+def titleize(text):
+    """Capitalizes all the words and replaces some characters in the string 
+    to create a nicer looking title.
+    """
+    if len(text) == 0: # if empty string, return it
+        return text
+    else:
+        text = text.lower() # lower all char
+        # delete redundant empty space 
+        chunks = [chunk[0].upper() + chunk[1:] for chunk in text.split(" ") if len(chunk) >= 1]
+        return " ".join(chunks)
+ 
+engine = Sqlite3Engine("geocode.sqlite3")
+metadata = MetaData()
+metadata.reflect(engine)
+geo_result = metadata.get_table("geo_result")
+ 
+# --- google geocoding result ---
+geocode_data = dict()
+for zipcode, json_text in engine.select(Select(geo_result.all)):
+    try:
+        zipcode = json.loads(zipcode)
+        json_dict = json.loads(json_text)
+        northeastbound_lat = json_dict["geometry"]["bounds"]["northeast"]["lat"]
+        northeastbound_lng = json_dict["geometry"]["bounds"]["northeast"]["lng"]
+        southwestbound_lat = json_dict["geometry"]["bounds"]["southwest"]["lat"]
+        southwestbound_lng = json_dict["geometry"]["bounds"]["southwest"]["lng"]
+        lat_google = json_dict["geometry"]["location"]["lat"]
+        lng_google = json_dict["geometry"]["location"]["lng"]
+        geocode_data[zipcode] = [northeastbound_lat, northeastbound_lng, 
+                            southwestbound_lat, southwestbound_lng, 
+                            lat_google, lng_google]
+    except Exception as e:
+        pass
+print("Got %s zipcode google geocoded." % len(geocode_data))
+
+# --- primary zipcode data ---
+primary_zipcode_data = dict()
+df = pd.read_csv("free-zipcode-database-Primary.csv", dtype={"Zipcode": str})
+for record in df.values:
+    (
+        Zipcode, ZipCodeType, City, State, LocationType, Lat, Long, Location, 
+        Decommisioned, TaxReturnsFiled, EstimatedPopulation, TotalWages,
+    ) = record
+    primary_zipcode_data[Zipcode] = [
+        ZipCodeType, City, State, LocationType, Lat, Long, Location, 
+        Decommisioned, TaxReturnsFiled, EstimatedPopulation, TotalWages,
+    ]
+print("Got %s zipcode primary zipcode." % len(primary_zipcode_data))
+
+# --- read zcta data, construct uszipcode data---
+uszipcode_data = list()
+df = pd.read_csv("zcta2010.csv", dtype={"ZCTA5": str})
+for record in df.values:
+    (
+        ZCTA5, LANDSQMT, WATERSQMT, LANDSQMI, WATERSQMI, 
+        POPULATION, HSGUNITS, INTPTLAT, INTPTLON,
+    ) = record
+    if (ZCTA5 in geocode_data) and (ZCTA5 in primary_zipcode_data):
+        (
+            northeastbound_lat, northeastbound_lng, 
+            southwestbound_lat, southwestbound_lng, 
+            lat_google, lng_google,
+        ) = geocode_data[ZCTA5]
+          
+        (
+            ZipCodeType, City, State, LocationType, Lat, Long, Location, 
+            Decommisioned, TaxReturnsFiled, EstimatedPopulation, TotalWages,
+        ) = primary_zipcode_data[ZCTA5]
+
+        # calculate derived field
+        try:
+            Density = POPULATION / LANDSQMI
+        except:
+            Density = None
+        try:
+            Wealthy = TotalWages / POPULATION
+        except:
+            Wealthy = None
+
+        if lat_google:
+            Lat = lat_google
+        if lng_google:
+            Long = lng_google
+
+        ZipCodeType = titleize(ZipCodeType)
+        City = titleize(City)
+        
+        uszipcode_data.append([
+            Zipcode, ZipCodeType, City, State, POPULATION, Density, 
+            TotalWages, Wealthy, HSGUNITS, LANDSQMI, WATERSQMI, 
+            Lat, Long, 
+            northeastbound_lat, northeastbound_lng, southwestbound_lat, southwestbound_lng,
+        ])
+print("Got %s zipcode for uszipcode database." % len(uszipcode_data))
+
+uszipcode_data = pd.DataFrame(uszipcode_data, columns=[
+    "Zipcode", "ZipcodeType", "City", "State", "Population", "Density", 
+    "TotalWages", "Wealthy", "HouseOfUnits", "LandArea", "WaterArea", 
+    "Latitude", "Longitude", 
+    "NEBoundLatitude", "NEBoundLongitude", "SWBoundLatitude", "SWBoungLongitude", 
+])
+uszipcode_data.to_csv("zipcode.txt", sep="\t", index=False)

dataset/step2_construct_dataset.py

@@ -0,0 +1,10 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+from __future__ import print_function
+from pprint import pprint as ppt
+from sqlite4dummy import *
+import pandas as pd
+
+engine = Sqlite3Engine("geocode.sqlite3")
+metadata = MetaData()

dataset/step3_makedatabase.py

@@ -0,0 +1,15 @@
+The data is originally from http://proximityone.com/cen2010_zcta_dp.htm
+
+About the data fields:
+
+- ZCTA5: zipcode
+- LANDSQMT: land square meters
+- WATERSQMT: water square meters
+- LANDSQMI: land square miles
+- WATERSQMI: water square miles
+- POPULATION: population
+- HSGUNITS: housing units
+- INTPTLAT: latitude
+- INTPTLON: longitude
+
+2010 census data

dataset/zcta2010/about.rst

@@ -1,6 +1,5 @@
 Welcome to the uszipcode Documentation
-====================================================================================================
-
+===================================================================================================
 ``uszipcode`` is the most powerful and easy to use zipcode information searchengine in Python. Besides geometry data (also boundary info), several useful census data points are also served: `population`, `population density`, `total wage`, `average annual wage`, `house of units`, `land area`, `water area`. The geometry and geocoding data I am using is from google map API on Oct 2015. To know more about the data, `click here <http://www.wbh-doc.com.s3.amazonaws.com/uszipcode/uszipcode/data/__init__.html#module-uszipcode.data>`_. `Another pupolar zipcode Python extension <https://pypi.python.org/pypi/zipcode>`_ has lat, lng accuracy issue, which doesn't give me reliable results of searching by coordinate and radius.
 
 **Highlight**:
@@ -11,7 +10,7 @@ Welcome to the uszipcode Documentation
 
 **Quick links**:
 
-- `Home page <https://github.com/MacHu-GWU/uszipcode-project>`_
+- `GitHub Homepage <https://github.com/MacHu-GWU/uszipcode-project>`_
 - `Online Documentation <http://www.wbh-doc.com.s3.amazonaws.com/uszipcode/index.html>`_
 - `PyPI download <https://pypi.python.org/pypi/uszipcode>`_
 - `Install <install_>`_