tag:blogger.com,1999:blog-83449072717561111272024-02-21T07:52:35.844-08:00I Lessen DataDanhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.comBlogger38125tag:blogger.com,1999:blog-8344907271756111127.post-49297952760914096592017-04-27T15:06:00.000-07:002017-04-27T15:06:10.510-07:00New Ubuntu EC2 Instance things to remember<div dir="ltr" style="text-align: left;" trbidi="on">
if you want to attach an existing EBS volume to the instance, make sure it's in the same Availability Zone (like "us-east-1c")<br />
<br />
To SSH at all:<br />
download that private key that they make you create as part of the setup console<br />
<span style="font-family: Courier New, Courier, monospace;">ssh -i (thatkey.pem) -l ubuntu</span><br />
(there's one user created at the start, called ubuntu)<br />
<span style="font-family: Courier New, Courier, monospace;">sudo adduser dantasse</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo vim /etc/group # and add dantasse to sudo group</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo vim /etc/ssh/sshd_config # and set PasswordAuthentication to yes</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo /etc/init.d/ssh reload</span><br />
Now you should be able to SSH.<br />
<br />
Attach the EBS volume on the AWS console. For example, attach it as /dev/sdf. Because it's Ubuntu, it will be attached as /dev/xvdf instead. (because reasons :P) You have to mount it still, though. Say you want to mount it at /data:<br />
<span style="font-family: Courier New, Courier, monospace;">sudo mkdir /data</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo mount /dev/xvdf /data</span><br />
<br /></div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-91279092341369063112016-12-16T13:49:00.003-08:002016-12-16T13:49:42.926-08:00Some things I learned in that adventure that I always forget<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Courier New, Courier, monospace;">dpkg --get-selections | grep -v deinstall </span><br />
shows all packages you have installed on Ubuntu<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">/etc/apt/sources.list</span> is where your repositories (for debian/etc packages) are stored<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">lsb_release -a</span><br />
shows what Ubuntu release you're running</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-81037660438876485212016-12-15T17:59:00.000-08:002016-12-15T17:59:05.262-08:00A hairy adventure in upgrading Ubuntu and Postgres<div dir="ltr" style="text-align: left;" trbidi="on">
(Detailing this more as a historical artifact than anything else, but it might help someone.)<br />
<br />
I had an EC2 Ubuntu 14.04 Trusty machine running PostgreSQL 9.3 and PostGIS 2.1 that I had to upgrade. For whatever reason.<br />
<span style="font-family: Courier New, Courier, monospace;">sudo do-release-upgrade</span><br />
<br />
but PostGIS was holding it back. I don't remember how I found this out but I think it was a simple google. So I did:<br />
<span style="font-family: "Courier New", Courier, monospace;">sudo apt-get remove postgresql-9.3-postgis-2.1 # </span><span style="font-family: inherit;">turns out this was a mistake</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo do-release-upgrade</span><br />
<br />
ok, now I'm on Ubuntu 16.04 Xenial. Now to upgrade postgres:<br />
<span style="font-family: Courier New, Courier, monospace;">sudo pg_upgradecluster 9.3 main /data/db/postgresql/9.5/main</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
ugh but wait, can't do it without PostGIS. hold on --<br />
<span style="font-family: Courier New, Courier, monospace;">psql tweet</span><br />
<span style="font-family: "Courier New", Courier, monospace;">tweet=#</span><span style="font-family: "Courier New", Courier, monospace;"> </span><span style="font-family: Courier New, Courier, monospace;">select * from tweet_pgh limit 1;</span><br />
(some kind of error because I don't have PostGIS installed anymore.)<br />
<br />
Huh.<br />
<span style="font-family: Courier New, Courier, monospace;">psql tweet</span><br />
<span style="font-family: Courier New, Courier, monospace;">tweet=# ALTER EXTENSION postgis UPDATE TO "2.3.1";</span><br />
(some kind of error about "no update path from 2.1.2 to 2.3.1")<br />
<br />
Huh. I guess I have to 1. get PostGIS back, 2. give it an "update path" to 2.3.1.<br />
Getting PostGIS back:<br />
<span style="font-family: Courier New, Courier, monospace;">sudo apt-get install postgresql-9.3-postgis-2.1</span><br />
(not found)<br />
<br />
Huh. I flailed here for a while and then <a href="http://download.osgeo.org/postgis/source/">downloaded 2.1.2 from source</a>, and did the old<br />
<span style="font-family: Courier New, Courier, monospace;">./configure</span><br />
<span style="font-family: Courier New, Courier, monospace;">make</span><br />
<span style="font-family: Courier New, Courier, monospace;">make install</span><br />
But of course it wasn't that easy! Got one compile error about json and using <a href="https://trac.osgeo.org/postgis/ticket/2539">this old ticket</a>, realized I had to go into liblwgeom/lwin_geojson.c and edit "<span style="font-family: Courier New, Courier, monospace;">#include <json json.h=""></json></span>" to "<span style="font-family: Courier New, Courier, monospace;">#include <json-c json.h=""></json-c></span>"<br />
Got another compile error and had to like #IFDEF out references to AggState and WindowAggState in lwgeom_accum.c (as in <a href="https://trac.osgeo.org/postgis/changeset/13797/branches/2.0">this old ticket</a>) and assume it'll work (sure did). Or rather, I was able to configure and make it, then I just symlinked it into the right place:<br />
<span style="font-family: Courier New, Courier, monospace;">sudo ln -s /home/dantasse/postgis-2.1.2/postgis/postgis-2.1.so /usr/lib/postgresql/9.3/lib </span><br />
<br />
THEN, I could do this:<br />
<span style="font-family: "Courier New", Courier, monospace;">tweet=# select * from tweet_pgh limit 1;</span><br />
<br />
but I couldn't do this:<br />
<span style="font-family: Courier New, Courier, monospace;">tweet=# ALTER EXTENSION postgis UPDATE TO "2.3.1";</span><br />
<div>
<span style="font-family: inherit;">Still getting the "no update path" error, until I found <a href="http://gis.stackexchange.com/questions/172872/setting-upgrade-path-to-postgis-2-2">this old post</a>, and then:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">cp /usr/share/postgresql/9.5/extension/postgis--2.1.2--2.3.1.sql /usr/share/postgresql/9.3/extension/</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">Finally:</span></div>
<div>
<span style="font-family: "Courier New", Courier, monospace;">sudo pg_upgradecluster 9.3 main /data/db/postgresql/9.5/main # </span><span style="font-family: inherit;">(took about 8 hours, b/c I think it has to dump and restore the whole DB?)</span></div>
<div>
<span style="font-family: inherit;">and we're all good. Ubuntu 16.04, Postgres server and client 9.5, PostGIS 2.3.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Can't believe that worked.</span></div>
</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-30709946684791360862016-09-26T18:45:00.001-07:002016-09-26T18:45:39.581-07:00Moving a PostgreSQL table from an Ubuntu machine to a RHEL machine<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Courier New, Courier, monospace;">pg_dump -Fc --file=(/.../pg_dump_file) --table=(tablename) --dbname=(database name)</span><br />
<div>
(sftp the pg_dump_file to the new machine)<br />
(install postgres by doing all of the below side bonus)</div>
<div>
<span style="font-family: Courier New, Courier, monospace;">pg_restore --clean --create --dbname=(database name) --jobs=3 (pg_dump_file)</span><br />
<br /></div>
<div>
the -Fc means "binary format, not text"</div>
<br />
<div>
Side bonus: installing PostgreSQL 9.4 on RHEL 6 (Red Hat Enterprise Linux,) (tips from <a href="https://wiki.postgresql.org/wiki/YUM_Installation">here</a>)</div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo yum localinstall https://download.postgresql.org/pub/repos/yum/9.4/redhat/rhel-6-x86_64/pgdg-redhat94-9.4-3.noarch.rpm # </span><span style="font-family: inherit;">figure out what rpm you need <a href="https://yum.postgresql.org/repopackages.php#pg94">here</a>; if you just yum install postgresql you'll get an old version</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo yum install postgresql94 postgresql94-server</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo service postgresql-9.4 initdb # one-time initialization</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo service postgresql-9.4 start # make the postgres server go</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo service postgresql-9.4 on # make the postgres server go every time I log on</span></div>
<div>
<br /></div>
<div>
Note: on RHEL 7, you have to do these 3 lines instead of those last 3:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo /usr/pgsql-9.4/bin/postgresql94-setup initdb</span></div>
<div>
</div>
</div>
<span style="font-family: Courier New, Courier, monospace;">sudo systemctl start postgresql-9.4.service</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;">systemctl enable postgresql-9.4.service # so it starts at boot</span></div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo yum install postgresql94-contrib # for hstore</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm # </span><span style="font-family: inherit;">to get the right EPEL rpm for RHEL 6; it's like adding a new APT repository in Ubuntu I think</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo yum install postgis2_94 # </span><span style="font-family: inherit;"><a href="http://www.postgresonline.com/journal/archives/362-An-almost-idiots-guide-to-install-PostgreSQL-9.5,-PostGIS-2.2-and-pgRouting-2.1.0-with-Yum.html">more help here</a> if you need</span><br />
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo su - postgres</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">psql</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">CREATE USER (myusername) WITH SUPERUSER;</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">CREATE DATABASE (dbname);</span><br />
(go to that new database)<br />
<span style="font-family: Courier New, Courier, monospace;">CREATE EXTENSION hstore;</span><br />
<span style="font-family: Courier New, Courier, monospace;">CREATE EXTENSION postgis;</span><br />
(quit postgres, e.g. with ctrl-d)<br />
(log out of being the postgres user; e.g. with ctrl-d again)</div>
<div>
<br /></div>
<div>
Double note: I have 2 cloud machines, one on RHEL 6 and one on RHEL 7. The RHEL 7 one killed me at the "install postgis" step; adding the EPEL rpm didn't make all the build errors go away. (when I poked around for other EPEL rpms, I eventually found one that fixed most of the build errors, but then gave me another error where it was missing libpoppler.so-46. I think I was the first person ever to have this error. In a turn of lunacy, I spent part of the day building GDAL from source. Surprisingly, it kinda worked. But didn't get me to PostGIS.) So, if you googled this and are looking for an answer, I'm afraid I can't help. Sorry!</div>
</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-47349453302362938152016-02-11T17:35:00.000-08:002016-02-11T17:35:14.661-08:00Useful PostgreSQL setup recipes<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Courier New, Courier, monospace;">CREATE USER foo;</span><br />
(or equivalently: <span style="font-family: Courier New, Courier, monospace;">CREATE ROLE foo WITH login;</span>)<br />
<span style="font-family: Courier New, Courier, monospace;">GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO foo;</span><br />
<span style="font-family: Courier New, Courier, monospace;">or</span><br />
<span style="font-family: Courier New, Courier, monospace;">GRANT SELECT ON ALL TABLES IN SCHEMA public TO foo;</span><br />
<br />
<span style="font-family: inherit;">I think these are all pretty self explanatory. Then the new user is up and running!</span><br />
<span style="font-family: inherit;">If you want the user to have a password:</span><br />
<span style="font-family: Courier New, Courier, monospace;">CREATE USER foo WITH PASSWORD 'bar';</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">To get rid of the user: </span><br />
<span style="font-family: Courier New, Courier, monospace;">REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM foo;</span><br />
<div>
<span style="font-family: Courier New, Courier, monospace;">DROP ROLE foo;</span></div>
</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-90824193880355762252016-01-14T14:14:00.001-08:002016-01-14T14:14:20.930-08:00ogr2ogr<div dir="ltr" style="text-align: left;" trbidi="on">
Dear Future Dan,<br />
I know you are trying to convert a shapefile to a GeoJSON file. I know this because you googled "site:ilessendata.blogspot.com ogr2ogr". Here is what you want to do:<br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">ogr2ogr -t_srs EPSG:4326 -f GeoJSON output.json input.shp</span><br />
<br />
Love, Past Dan<br />
<br /></div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-71445623803820645472015-11-09T12:12:00.004-08:002017-11-21T00:16:21.761-08:00Dump a PostgreSQL table to json<div dir="ltr" style="text-align: left;" trbidi="on">
This seems like it should be easy but it isn't, and I've had to look it up a couple times. Here's the magic juice:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">copy (select array_to_json(array_agg(row_to_json(t)))</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> from (</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> select * from tweet_pgh limit 5</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ) t</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">) to '/home/dantasse/data_dump/foo.json';</span><br />
<br />
gets 5 rows from the tweet_pgh table and dumps them to that file. (the directory has to be world-writeable I think, because it'll execute as the postgres user.)<br />
<br />
Edit:<br />
<span style="font-family: Courier New, Courier, monospace;">COPY (SELECT ROW_TO_JSON(t) FROM (SELECT ST_AsGeoJSON(coordinates), * FROM tweet_austin) t) TO '/data/austin.json';</span><br />
<span style="font-family: inherit;">or just</span><br />
<span style="font-family: "Courier New", Courier, monospace;">COPY (SELECT ROW_TO_JSON(t) FROM (SELECT * FROM tweet_austin) t) TO '/data/austin.json';</span><br />
<span style="font-family: inherit;">seems to work better when you're memory limited.</span></div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-30873593647916308822015-05-04T19:30:00.002-07:002015-05-06T16:04:49.958-07:00"Let's Hadoop Something" on Amazon EMR<div dir="ltr" style="text-align: left;" trbidi="on">
Edit: ugh, disregard this post maybe; still couldn't get the program to both run without errors and produce correct output. Started using Spark locally instead; it's better for my needs anyway. Leaving it up just in case this helps at all.<br />
<br />
That's the goal I set out with: to run *something* on Hadoop. In this case it was simple: in each tweet, get the frequency of each emoji. So if a tweet is:<br />
☃☃☃ it's cold ☃☃☃☹☹<br />
I want to get "[6, 2]" because there's one emoji that happens 6x and one that is 2x.<br />
Then I want to get the overall frequency of each frequency. Basically, I just want to know how many times people tweet an emoji once, and how many times they spam it 10x.<br />
<br />
(the clever among you would recognize that this doesn't require any hadoops. indeed, it takes about 10 seconds to just loop through a 500mb text file with a bunch of tweets and spit out the answer. so in the real world, I should not be map-reducing anything here. but it's a learning experience; bear with me.)<br />
<br />
Writing map-reduce code involves separating your code into mappers and reducers. For python, for example, this is easy: just make one python file for your mapper and one for your reducer, that both take in lines on sys.stdin and output lines to stdout. (this was unexpectedly simple for me. I like that they just use stdin and stdout, and one-data-point-per-line.)<br />
<br />
Upload everything to s3: your mapper script, your reducer script, and all your input data (in the form of a text file).<br />
<br />
Then create an EMR cluster (using all the defaults, but do add an SSH key and just take m1.medium instances if you're just testing, as they're the cheapest), and add a "step" that is a "streaming program". Terminate your cluster when you're done so you don't spend extra money.<br />
<br />
You'll find your output in the output location that you chose, in the form of a few files like "part-00000", "part-00001". Not sure why it doesn't keep reducing until you're down to one final output, but I guess that is for you to do on your own?<br />
<br />
Otherwise, good luck! This took me about a couple bucks to run, plus $15ish in debugging. It is frustrating when debugging costs real money. So it goes.<br />
<br />
More caveats!<br />
- as of May 5, 2015, Amazon EMR only supports Python 2.6.9. This is mildly frustrating. (here is <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/ami-versions-supported.html">a list of what it supports that will hopefully be kept up to date</a>) Test your code on python2.6 before you upload it; it'll save you some headaches.<br />
- make sure you set the "output location" to be a folder that doesn't exist yet. Otherwise it will just crash. You can figure this out from the logs, but each map-reduce job is ~20 min and probably a buck or two so better not to waste any.<br />
- sometimes the logs are hard to find. The most valuable ones will be available through the EC2 console. You can find them here:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsksD1aMOOTHBi1O6zY9Iez5LquVgyn4qsgVspi-VALZVTu2WpC1hPnBDUTmn6a2Mzt_AH70N8R9jG8bQh_YDDOk6vrvwod9r1mOFQ3sOPj-jKsOiMWeNXVWYA5fYHygHiRkRWUC9nLgk/s1600/Screen+Shot+2015-05-04+at+9.59.49+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsksD1aMOOTHBi1O6zY9Iez5LquVgyn4qsgVspi-VALZVTu2WpC1hPnBDUTmn6a2Mzt_AH70N8R9jG8bQh_YDDOk6vrvwod9r1mOFQ3sOPj-jKsOiMWeNXVWYA5fYHygHiRkRWUC9nLgk/s640/Screen+Shot+2015-05-04+at+9.59.49+PM.png" height="112" width="640" /></a></div>
Click "view logs" to see logs related to the whole job, or go into "view jobs" then "view tasks" then "view attempts" to see logs for each individual mapper or reducer. Sometimes the logs will not show up. This is frustrating. Wait a few minutes and try again. If they're still not there, then I am not sure why. Sorry about it.</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-27819738582346262482015-04-10T14:11:00.001-07:002015-04-10T16:37:00.248-07:00MongoDB vs PostgreSQL for geo queries: postgres is indeed pretty much across-the-board faster<div dir="ltr" style="text-align: left;" trbidi="on">
ok, I have some benchmark numbers. Two almost-identical databases, one in mongo, one in postgres, so I can actually compare apples to apples. In Mongo I have everything in one collection. In postgres, I have two tables: tweet_pgh, which is all the data, and tweet_pgh_small, which is just the data I need.<br />
<br />
These tables are pretty optimized: the mongo collection has indices on user.screen name, coordinates (geospatial), coordinates.coordinates.0 and coordinates.coordinates.1. The sql tables have btree indices on user_screen_name and gist indices on the coordinates (clustered). Oh, and of course I've got PostGIS installed.<br />
<br />
Every query here selects everything in a table, then iterates through it (just incrementing a counter) to simulate a semi-realistic use case.<br />
<br />
<div class="p1">
Count in mongo: 3653683</div>
<div class="p1">
<span class="s1">Count in postgres: 3653278</span></div>
<div class="p1">
<br /></div>
<div class="p1">
Searching for single user stuff. User A has 2942 items, User B has 1928 items, User C has 3499 items. (using A, B, and C here for their anonymity instead of their real twitter handles.) First three are mongo, the rest are postgres.</div>
<div class="p1">
Mongo: {'user.screen_name': '(user A)'} 11 sec</div>
<div class="p1">
Mongo: {'user.screen_name': '(user B)'} 6 sec</div>
<div class="p1">
Mongo: {'user.screen_name': '(user C)'} 24 sec</div>
<div class="p1">
Postgres: SELECT * FROM tweet_pgh WHERE user_screen_name = '(user A)'; 5 sec</div>
<div class="p1">
Postgres: SELECT * FROM tweet_pgh WHERE user_screen_name = '(user B)'; 6 sec</div>
<div class="p1">
Postgres: SELECT * FROM tweet_pgh WHERE user_screen_name = '(user C)'; 6 sec</div>
<div class="p1">
Postgres: SELECT * FROM tweet_pgh_small WHERE user_screen_name = '(user A)'; 1.9 sec</div>
<div class="p1">
Postgres: SELECT * FROM tweet_pgh_small WHERE user_screen_name = '(user B)'; 1.4 sec</div>
<div class="p1">
Postgres: SELECT * FROM tweet_pgh_small WHERE user_screen_name = '(user C)'; 1.9 sec</div>
<div class="p1">
<br /></div>
<div class="p1">
Huh. So for these it looks like postgres takes either the same amount of time, half the time, or 1/4 the time. But the real win comes from just having a smaller table.</div>
<div class="p1">
<br /></div>
<div class="p1">
<span class="s1">Searching for geographic areas. And these are *with geospatial indices* on both tables. Here I'm searching for a 0.01-degree lat by 0.01-degree lng box, which has 69k things in it.</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Mongo: {'coordinates': {'$geoWithin': {'$geometry': {'type': 'Polygon', 'coordinates': [[[-79.99, 40.44], [-79.99, 40.45], [-80, 40.45], [-80, 40.44], [-79.99, 40.44]]]}}}} - 53 min</span></div>
<div class="p1">
Postgres: SELECT * FROM tweet_pgh WHERE ST_MakeEnvelope(-80, 40.44, -79.99, 40.45, 4326) ~ coordinates; 33 sec</div>
<div class="p1">
Postgres: SELECT * FROM tweet_pgh_small WHERE ST_MakeEnvelope(-80, 40.44, -79.99, 40.45, 4326) ~ coordinates; 3 sec</div>
<div class="p1">
<br /></div>
<div class="p1">
Then try the same thing with a smaller box (0.001x0.001 degree) with way fewer things (~ 5):</div>
<div class="p1">
<div class="p1">
Mongo: {'coordinates': {'$geoWithin': {'$geometry': {'type': 'Polygon', 'coordinates': [[[-79.899, 40.44], [-79.899, 40.441], [-79.9, 40.441], [-79.9, 40.44], [-79.899, 40.44]]]}}}}; 7 min</div>
<div class="p1">
Postgres: SELECT * FROM tweet_pgh WHERE ST_MakeEnvelope(-79.9, 40.44, -79.899, 40.441, 4326) ~ coordinates; 0.16 sec</div>
</div>
<div class="p1">
Postgres: SELECT * FROM tweet_pgh_small WHERE ST_MakeEnvelope(-79.9, 40.44, -79.899, 40.441, 4326) ~ coordinates; 0.07 sec</div>
<div class="p1">
(still waiting on the mongo query for this)</div>
<div class="p1">
<br /></div>
<div class="p1">
Wow! So, wait. So the same query that took 53 minutes in Mongo took 33 seconds in postgres. And <i>three</i> seconds in a reduced-size table.</div>
<div class="p1">
<br />
It's a little unfair; in Mongo you can do a little better by using $gt and $lt on the coordinates instead of doing a $geoWithin - it's easier to compare numbers than coordinates, if you're just doing a box query. So we have the following:<br />
<br />
<div class="p1">
<span class="s1">{'coordinates.coordinates.1': {'$gt': 40.45, '$lt': 40.46}, 'coordinates.coordinates.0': {'$gt': -79.95, '$lt': -79.94}} 20 min</span></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p1">
<span class="s1">{'coordinates.coordinates.1': {'$gt': 40.46, '$lt': 40.461}, 'coordinates.coordinates.0': {'$gt': -79.97, '$lt': -79.969}} 2 min</span></div>
<br />
But this is still nowhere near the postgres time! So:<br />
<br />
- <b>use PostgreSQL with PostGIS for geo queries, not MongoDB</b></div>
<div class="p1">
- this is especially true if you can index and cluster your dbs</div>
<div class="p1">
- maybe postgres is a little better for simpler queries too</div>
<div class="p1">
- in postgres (maybe in mongo too) there is a lot of benefit to be had from cutting down the size of your tables as small as possible</div>
<div class="p1">
<br /></div>
</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com5tag:blogger.com,1999:blog-8344907271756111127.post-4490827892938492172015-01-24T11:29:00.000-08:002015-01-24T11:29:25.394-08:00Dealing with images on the Pebble watch<div dir="ltr" style="text-align: left;" trbidi="on">
My goal: a Pokemon watch, that shows a pokemon each minute instead of the numerical minute. I figured I'd get 144x144 images, and then just display one each minute. The challenges:<br />
<br />
<b>1. You need black and white (not grayscale) images.</b> This is easily solved with a tool like <a href="http://www.tinrocket.com/hyperdither/">HyperDither</a>, which uses "dithering" to create shades of gray using only black and white pixels.<br />
For example, changes this color Bulbasaur into this black and white one:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipZ89-eu-QnIq13E1mwyKsJPcbDil7PyBCfcJJnlp73o4q_ZoqMAqpl1d6Nd5Hmpzv0lzSHSdgnXAMy0EgAsgZDe_oXtDRE_u78LbdgQ4llgo0It4Bcge4Sxsz23yOQT7ZA5BAxKfRSP8/s1600/1.png.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipZ89-eu-QnIq13E1mwyKsJPcbDil7PyBCfcJJnlp73o4q_ZoqMAqpl1d6Nd5Hmpzv0lzSHSdgnXAMy0EgAsgZDe_oXtDRE_u78LbdgQ4llgo0It4Bcge4Sxsz23yOQT7ZA5BAxKfRSP8/s1600/1.png.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhE7rMD7X9PsaW6HmQCfM4gUPNfstOYM9_sx6-h8eUyKmW40JWNM1ATjipAUcczTu0MWJqN8wrtKUgWHIYdyor2PDA0hqZUYL6rWR2UYNNks_x-h0mZePC0X7nsKy5-FymcFfia9WTFyYU/s1600/1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhE7rMD7X9PsaW6HmQCfM4gUPNfstOYM9_sx6-h8eUyKmW40JWNM1ATjipAUcczTu0MWJqN8wrtKUgWHIYdyor2PDA0hqZUYL6rWR2UYNNks_x-h0mZePC0X7nsKy5-FymcFfia9WTFyYU/s1600/1.png" /></a></div>
<br />
This is nice. HyperDither has a batch mode, which works, but it actually makes slightly smaller PNGs if you do it one at a time.<br />
<br />
<b>2. Saving space</b>. You get 24kb RAM per app and 96kb storage for images/fonts per app. It's not easy to fit all the images you need in 96kb. I wanted 60 images, so I had to average ~1.5kb per image. Luckily, black/white 144x144 PNGs are not so big. Using HyperDither, I got pretty close to 96kb, but not quite: I'd have 90kb of PNGs locally, but when I uploaded them to the Pebble, it converted them to "pbi" files, which are almost double the size.<br />
<br />
(also note that if you include too many pictures that are over 96kb, you'll get a cryptic error: it'll compile fine, but then will say "Installation failed. Check your phone for details." but no real way to check your phone for details, and no hints about why it fails. You'll get a similar cryptic error if you upload an image but never reference it in your code (<a href="http://stackoverflow.com/questions/26996907/cloudpebble-installation-failed-check-your-phone-for-details">stack overflow related question</a>).<br />
<br />
<b>Spriting</b><br />
So I figured there was some overhead per-image, so I tried spriting, or combining multiple images into one big image, like so: (created with <a href="http://instantsprite.com/">InstantSprite</a>, which is awesome and free)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJs8Ph_IdpGVCr5y8fjwpzJAytdepF1cv-hZ4JRoy3p_a12nogfFJeqovM_pVfhalm88e96zdENAfFqkacjn4Mz7DLxJEg9F7n8KrDTc0EtmhxdhmWHvZYSzbp6bft7GgC_EQCEt9801w/s1600/pokemon_54_59.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJs8Ph_IdpGVCr5y8fjwpzJAytdepF1cv-hZ4JRoy3p_a12nogfFJeqovM_pVfhalm88e96zdENAfFqkacjn4Mz7DLxJEg9F7n8KrDTc0EtmhxdhmWHvZYSzbp6bft7GgC_EQCEt9801w/s1600/pokemon_54_59.png" height="65" width="400" /></a></div>
<br />
and then you just take a subset of that image at a time. Pebble provides a call, <a href="http://developer.getpebble.com/docs/c/group___graphics_types.html#ga5d86515990747e47a76c0a16ed6b2850">gbitmap_create_as_sub_bitmap</a>, to do exactly that. However, a few problems:<br />
- if you make an image that is too big (like 10 pokemon at a time, or 1440x144), then gbitmap_create_as_sub_bitmap just crashes the app with no feedback. I found that 6 pokemon (864x144) worked, while 1440x144 didn't. Not sure what the actual limit is.<br />
- it doesn't even make the images smaller! 10 PNGs with 6 pokemon each is not really any smaller than 60 PNGs with 1 pokemon each.<br />
<br />
<b>uPNG. Forget Spriting.</b><br />
So, forget spriting. I then found <a href="https://github.com/mhungerford/png_demo">Matthew Hungerford's port of uPNG</a>, which is a library that lets you use raw PNG files instead of converting them to PBI first. Just include the .c and .h files, use gbitmap_create_with_png_resource instead of gbitmap_create_with_resource, and then edit your appinfo.json file to change the type of each image from "png" to "raw".<br />
(editing your appinfo.json may require pushing your code from CloudPebble to Github, then cloning it locally, editing the file, committing and pushing your change. also, you might have to move your image files from resources/images/foo.png to resources/data/foo.png. anyway, this is nice, because you can then upload a bunch of images by editing a text file instead of uploading through the GUI a lot.)<br />
<br />
He also provides scripts to transform images to B/W or dithered first, which ended up saving a few kb in my case.<br />
<br />
In the end, thanks to uPNG, I had 80kb of images that actually stayed 80kb. (plus a little bit of overhead per image, but didn't matter.) Success! <a href="https://github.com/dantasse/PokemonWatch">Here's my watch code on Github</a>.<br />
<br />
<b>Other Tips</b><br />
- Use logging, like: APP_LOG(APP_LOG_LEVEL_DEBUG, "hello"); - this took me too long to discover.<br />
<div>
<br /></div>
<span id="goog_792820038"></span></div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com2tag:blogger.com,1999:blog-8344907271756111127.post-90815671095020327752014-10-19T12:07:00.003-07:002014-11-19T14:02:36.127-08:00Starting MongoDB on OS X<div dir="ltr" style="text-align: left;" trbidi="on">
When you install MongoDB on a Mac with Homebrew, it spits out this important info:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">To have launchd start mongodb at login:</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ln -sfv /usr/local/opt/mongodb/*.plist ~/Library/LaunchAgents</span><br />
<span style="font-family: Courier New, Courier, monospace;">Then to load mongodb now:</span><br />
<span style="font-family: Courier New, Courier, monospace;"> launchctl load ~/Library/LaunchAgents/homebrew.mxcl.mongodb.plist</span><br />
<span style="font-family: Courier New, Courier, monospace;">Or, if you don't want/need launchctl, you can just run:</span><br />
<span style="font-family: Courier New, Courier, monospace;"> mongod --config /usr/local/etc/mongod.conf</span><br />
<br />
This is important because the MongoDB docs tell you to run it by "sudo service mongodb start", which doesn't work on macs. Macs use "launchctl" instead of "service" (and they work differently). The things that they launch are defined by plist files in ~/Library/LaunchAgents.<br />
<br />
There used to be a thing called "brew services", but now if you use that, it will tell you that it's unsupported and will be removed soon, so I guess don't use it.<br />
<br />
Or if you just want to start it for a little while now, you could just start it with "mongod", but that doesn't work because you have to tell it where the config file is, which tells mongo where the database should be. Of course, you might not know where the config file or the database are. That's what the last line (mongod --config ...) is for.<br />
<br />
Another tip: MongoDB logs are stored by default in:<br />
<span style="font-family: Courier New, Courier, monospace;">/var/log/mongodb/mongodb.log</span></div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-78666525921380958462014-02-23T15:01:00.001-08:002014-02-23T15:02:04.575-08:00Making maps in python, step 1: installing dependencies<div dir="ltr" style="text-align: left;" trbidi="on">
I want to plot some things on a map. I got one guide from <a href="http://sensitivecities.com/so-youd-like-to-make-a-map-using-python-EN.html">here</a>, that uses Basemap.<br />
<br />
<a href="http://matplotlib.org/basemap/index.html">Basemap</a>:<br />
brew install geos<br />
brew install gdal<br />
brew install gfortran<br />
<br />
pip install matplotlib<br />
pip install numpy<br />
pip install pandas<br />
pip install shapely<br />
pip install basemap --allow-external basemap --allow-unverified basemap<br />
pip install scipy<br />
pip install pysal<br />
pip install Fiona<br />
pip install descartes<br />
<br />
Okay, geez, I had written a ton more in this Blogger window, it looked like it was saving my draft, but then it... somehow didn't? Jesus, nothing on computers ever works. All right, I'll try to recreate it:<br />
<br />
on Ubuntu, I wanted to pip install as much as possible (because of virtualenv) so I did stuff like this:<br />
sudo apt-get build-dep numpy scipy matplotlib<br />
pip install numpy scipy matplotlib<br />
I think I just used pip for pandas, shapely, pysal, descartes. Fiona required something else, like apt-getting gdal or something?<br />
<br />
<a href="http://kartograph.org/">Kartograph</a>:<br />
Ubuntu: http://kartograph.org/docs/kartograph.py/install-ubuntu.html<br />
OSX: pip install -r https://raw.github.com/kartograph/kartograph.py/master/requirements.txt worked, I think because I already had installed gdal via brew.<br />
<br />
<a href="http://scitools.org.uk/cartopy/docs/latest/index.html">Cartopy</a>:<br />
Ubuntu: https://github.com/SciTools/cartopy/issues/46<br />
OSX: https://github.com/SciTools/cartopy/issues/48</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com1tag:blogger.com,1999:blog-8344907271756111127.post-48310385940815792492014-02-03T08:19:00.002-08:002014-02-03T08:19:23.490-08:00Python sentence segmentation, kind of quick and mostly legit<div dir="ltr" style="text-align: left;" trbidi="on">
Sentence segmentation (splitting a big block of text into sentences) is not trivial. You can't just split on periods, for example, because you'll get tripped up on every Dr. and Ms. and etc. and so on! However, it's mostly solved and in libraries, so here's a quick way to do it in python.<br />
<br />
NLTK is a pretty general-purpose natural language processing toolkit. You could install the whole thing via instructions on <a href="http://nltk.org/">their website</a>. But that will also install a lot of other NLP tools. Also, a lot of these tools can be trained, which makes them more accurate if you have training data, but more difficult to get started if you don't have such training data. To get a pre-trained model:<br />
<br />
- download Punkt from <a href="http://nltk.org/nltk_data/">NLTK Data</a> (<a href="http://nltk.github.com/nltk_data/packages/tokenizers/punkt.zip">direct link to Punkt</a>)<br />
- unzip it and copy english.pickle into the same directory as your python file. This is the trained model, which has been serialized out to a file. (obviously, this assumes you're segmenting English text; if not, grab one of the other .pickle files.)<br />
- in your python code, unpickle it like so:<br />
<span style="font-family: Courier New, Courier, monospace;">import pickle</span><br />
<span style="font-family: Courier New, Courier, monospace;">segmenter_file = open('english.pickle', 'r')</span><br />
<span style="font-family: Courier New, Courier, monospace;">sentence_segmenter = pickle.Unpickler(segmenter_file).load()</span><br />
- then call:<br />
<span style="font-family: Courier New, Courier, monospace;">sentences = sentence_segmenter.tokenize(text)</span><br />
(where "text" is a string containing all your text).<br />
<br /></div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com4tag:blogger.com,1999:blog-8344907271756111127.post-6574759068799604252013-12-01T21:27:00.002-08:002013-12-01T21:27:36.555-08:00Sony Smartwatch 2 Programming<div dir="ltr" style="text-align: left;" trbidi="on">
Some tips:<br />
- the <a href="http://developer.sonymobile.com/reference/sony-addon-sdk/packages">documentation</a> seems to be, well, sparse. The most helpful thing I've found was code samples. They released five with the SDK (Sample{Control, Widget, AdvancedControl, Sensor, Notification}Extension) and two more <a href="http://developer.sonymobile.com/downloads/code-example-module/music-player-and-8-game-extensions-for-smartwatch-open-source/">here</a> (EightPuzzleExtension and OSS_MusicExtension). Load them up into Eclipse as Android projects, they mostly work out of the box.<br />
- well, mostly. For EightPuzzleExtension, I had to add<span class="s1"> </span><span style="font-family: Courier New, Courier, monospace;"><span class="s2"><</span><span class="s3">uses-permission</span><span class="s1"> </span><span class="s4">android:name</span><span class="s1">=</span>"android.permission.READ_EXTERNAL_STORAGE"<span class="s1"> </span></span><span class="s2"><span style="font-family: Courier New, Courier, monospace;">/></span> to the Android Manifest.</span><br />
<span class="s2">- also useful was <a href="http://xiangchen.me/blog/?p=2214">Xiang "Anthony" Chen's HelloWatch post</a>. All the example apps have 4 classes, and Anthony explains them better than I can. Almost all of my code went in the Extension app.</span><br />
<br />
Here's the app I've been working on: <a href="https://github.com/dantasse/MorseWatch">MorseWatch</a>. Mostly just takes in touch events. Also, I made a <a href="https://github.com/dantasse/HelloSmartwatch">HelloSmartwatch</a> app that uses accelerometers.<br />
<br />
Maybe more details later. In the meantime, better to get some info and helpful links out there as soon as possible.<br />
</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com1tag:blogger.com,1999:blog-8344907271756111127.post-76064510590303766992013-12-01T17:34:00.002-08:002013-12-20T21:25:33.900-08:00Excursions in rooting my Nexus 4<div dir="ltr" style="text-align: left;" trbidi="on">
Man, I never really wanted to dig into phone rooting, but I recently went to do a Portable Wi-fi Hotspot and found that T-Mobile now blocks all data when I tether through my phone. (which is insane. why does T-Mobile care whether I'm using my phone or computer? I'm not using a ton of data. but anyway.)<br />
<br />
I found <a href="http://forum.xda-developers.com/showthread.php?t=2512674&page=8">this post, which explains how to edit one sqlite database to allow tethering again</a>.<br />
<br />
Unfortunately, it's in /data/data/..., which is only editable (or even viewable) if you have root access. (this is I think the same as root on mac/linux, which is roughly the ability to "sudo".) For some reason, Android makes this difficult.<br />
<br />
Luckily, xda-developers to the rescue.<br />
<a href="http://forum.xda-developers.com/showthread.php?t=2018179">Nexus 4 Rooting Guide</a>.<br />
<a href="http://forum.xda-developers.com/showpost.php?p=34744848&postcount=4">Backing up your phone before you root it</a>.<br />
<br />
How'd it go? Pretty smoothly. Wiped my phone, but the two-part backup in the link above (both the "adb pull" and the "adb backup" part) meant I didn't lose much. Had to reconfigure a couple minor things, not bad. And now I can tether!<br />
<br />
The major steps, in my mind, are:<br />
- back up your stuff<br />
- unlock the bootloader (this is when your phone gets wiped)<br />
- flash (install) the custom <a href="http://www.androidcentral.com/what-recovery-android-z">recovery</a><br />
- install the SU app or something? a little unclear on this step, but it involves installing SuperSU.<br />
- restore your data<br />
<br />
A couple of gotchas:<br />
- as usual when reinstalling everything on your phone, if you use two-factor authentication (as you should), generating some backup codes beforehand makes everything easier.<br />
- on the "clockwork mod" site, all you need is the one in the "download recovery" column.<br />
- don't download/install over-the-air Android updates. It breaks your root. (which is not a huge deal, but it means you have to re-flash the recovery and re-install the SU app.)</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-59984422975533192942013-08-16T09:38:00.001-07:002013-08-16T09:38:24.303-07:00find | xargs rm, if your filenames have spaces<div dir="ltr" style="text-align: left;" trbidi="on">
I have a nested file structure, with 200 subdirectories, and I want to get rid of everything that ends with "Deriv.csv" (but keep all other files). Usually I would do this:<br />
<span style="font-family: Courier New, Courier, monospace;">find . -name "*Deriv.csv"</span><br />
to find all the files (find everything in the directory structure starting here (.) that has the name "anything that ends with Deriv.csv"), and then just rm them all, with the help of xargs. It would look like this:<br />
<span style="font-family: Courier New, Courier, monospace;">find . -name "*Deriv.csv" | xargs rm</span><br />
(this is a good pattern to know, though a little hard to wrap your head around at first; piping to <span style="font-family: Courier New, Courier, monospace;">xargs</span> makes the results of your first thing be the argument to the second thing. So if it finds <span style="font-family: Courier New, Courier, monospace;">file1Deriv.csv, file2Deriv.csv, file3Deriv.csv</span>, then the <span style="font-family: Courier New, Courier, monospace;">| xargs rm</span> makes it do "<span style="font-family: Courier New, Courier, monospace;">rm file1Deriv.csv file2Deriv.csv file3Deriv.csv</span>".)<br />
<br />
The added difficulty comes in because some of the filenames have spaces, so the find returns something like:<br />
<span style="font-family: Courier New, Courier, monospace;">file 1 Deriv.csv</span><br />
<span style="font-family: Courier New, Courier, monospace;">file 2 Deriv.csv</span><br />
and then just passing that to xargs rm makes it run:<br />
<span style="font-family: Courier New, Courier, monospace;">rm file 1 Deriv.csv file 2 Deriv.csv</span><br />
which of course makes it complain that "file" is not found, "1" is not found, "Deriv.csv" is not found, etc. (I'm lucky that I didn't have any stray things called "file" or "Deriv.csv" sitting around that I wanted to keep, or they would have been removed by this mistake!)<br />
<br />
One thing I found <a href="http://www.unix.com/shell-programming-scripting/129701-pipe-xargs-rm-filename-spaces.html">here (thanks!)</a> is:<br />
<span style="font-family: Courier New, Courier, monospace;">find . -name "*Deriv.csv" | xargs -I{} rm {}</span><br />
-I does two things:<br />
1. separate arguments by line, not by whitespace (great!)<br />
2. make everything after the -I be its own command that you can control however you want. In this case, we do something simple: take the argument (with the {}) and put it after an rm. But you could also do, for example:<br />
<span style="font-family: Courier New, Courier, monospace;">mkdir deriv_files</span><br />
<span style="font-family: Courier New, Courier, monospace;">find . -name "*Deriv.csv" | xargs -I{} mv {} deriv_files/</span><br />
<div>
which is pretty neat. (<a href="http://linux.die.net/man/1/xargs">reference</a>)</div>
</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com1tag:blogger.com,1999:blog-8344907271756111127.post-55681663452494623752013-06-24T17:50:00.000-07:002013-06-24T18:02:56.610-07:00Twitter -> LCD via python and arduino<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
I've got a little LCD screen. I thought it'd be neat if anyone could tweet and it'd show up on that screen. Kind of like that goofy thing at basketball games where you can send a text message somewhere and maybe it'll show up on the huge scoreboard.<br />
<br />
Code's <a href="https://github.com/dantasse/dans_office">on github</a>, if you're into that sort of thing.</div>
<div>
<br /></div>
<div>
<b>Getting the tweets from twitter onto my computer</b></div>
<div>
<br /></div>
<div>
I created a new account called @dansoffice, and I want to get every tweet that mentions @dansoffice into my twitter program. I want it real-time, and I don't want to get rate limited for polling a lot, so I looked to their streaming APIs.</div>
<div>
<br /></div>
<div>
What I want is the "<a href="https://dev.twitter.com/docs/api/2/get/user">user stream</a>", so I have to authenticate as @dansoffice and send a GET to: https://userstream.twitter.com/1.1/user.json</div>
<div>
<br /></div>
<div>
<a href="http://docs.python-requests.org/en/latest/">Requests</a> is a nice python library to make http easy. Pair it with <a href="https://github.com/requests/requests-oauthlib/tree/master/requests_oauthlib">requests_oauthlib</a> (installed via pip; don't confuse it with <a href="https://pypi.python.org/pypi/requests-oauth">requests_oauth</a>), and the authentication is easy too. (<a href="https://dev.twitter.com/apps">make a Twitter app</a>, go to the "OAuth tool", and it will tell you your access token and secret, and your consumer key and secret; use requests_oauthlib per their documentation. "consumer" = "client" in this case.)</div>
<div>
<br /></div>
<div>
And Requests even handles streaming requests via iter_lines... sort of. It's got a "chunk size" - a buffer that must fill up before it does anything. For some reason, this sometimes stops tweets from being read, until the <i>next</i> tweet comes in and clears out the buffer. <a href="https://github.com/kennethreitz/requests/issues/844">Known issue</a>.</div>
<div>
<br /></div>
<div>
So in my case, a terrible hack: also request that tweets are delimited by length, so the length comes in, followed by a newline, and clears the buffer; at this point, send another request (this one non-streaming, to <a href="https://dev.twitter.com/docs/api/1.1/get/statuses/mentions_timeline">mentions_timeline</a>) to get the actual text of the most recent @dansoffice mention. Ugh.</div>
<div>
<br /></div>
<div>
<b>Getting the tweets from my computer to the LCD</b></div>
<div>
<b><br /></b></div>
<div>
Python writes nicely to serial, via <a href="http://pyserial.sourceforge.net/">pyserial</a>. Found my serial port's name from Arduino (tools->serial port menu). Easy enough to write to it when I get a tweet.</div>
<div>
<br /></div>
<div>
The LCD screen I have is <a href="https://www.sparkfun.com/products/256">this 20x4 display</a>. I had to solder some header pins to it. Arduino has a <a href="http://www.arduino.cc/en/Tutorial/LiquidCrystal">nice LCD demo</a>, which was easy and straightforward after I fixed my shoddy solder job. At that point, it became just a matter of chopping a string into 4 lines. (Nothing is easy in C.)</div>
<div>
<br /></div>
<div>
<b>Fin</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHksY6liq00dNKedQ97JLoJA_sTuMOdky7L6jhxrHzlaBmFggBKSM-HCLX9NI5aCY-jwpW_pJj1wzO0lezTU6yyKz10yA0zBw8QGBDUSHCQLzBQqd1vY0ILTIGznJfvuwumlTB6uO3F7E/s1600/IMG_20130624_203917.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHksY6liq00dNKedQ97JLoJA_sTuMOdky7L6jhxrHzlaBmFggBKSM-HCLX9NI5aCY-jwpW_pJj1wzO0lezTU6yyKz10yA0zBw8QGBDUSHCQLzBQqd1vY0ILTIGznJfvuwumlTB6uO3F7E/s400/IMG_20130624_203917.jpg" width="400" /></a></div>
<div>
<b><br /></b></div>
<div>
Now, as long as my little twitter monitor is running, and the Arduino is connected, tweets mentioning @dansoffice show up in real time. Now all I have to do is get a long mini-USB cable to stretch to my window. All the tweet-publicizing excitement of the NBA!</div>
<div>
<br /></div>
<div>
(maybe my next thing should mimic the Skinner-style pizza/t-shirt bribes and exhortations to MAKE SOME NOISE.)</div>
</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com2tag:blogger.com,1999:blog-8344907271756111127.post-27503948195118007292013-06-19T14:21:00.002-07:002013-06-19T14:21:54.632-07:00Embedding a browser in a python application on a mac<div dir="ltr" style="text-align: left;" trbidi="on">
I heard about PyQtWebkit (e.g. <a href="http://www.rkblog.rk.edu.pl/w/p/webkit-pyqt-rendering-web-pages/">here</a>). Sounds great.<br />
<br />
First I had to get PyQt. Or maybe PyQt and Webkit, or Py and QtWebkit, or who knows. I wanted all those parts.<br />
<br />
Tried to install it from source (e.g. <a href="http://www.riverbankcomputing.co.uk/software/pyqt/download5">from Riverbank</a>) but that was a hopeless mess. Version 4 or 5? I don't know. I had to build SIP first, which eventually went fine. I tried to build PyQt, but had trouble because it couldn't find qmake, so I went looking for that. But then (as I often do when building things from source) I wondered if I was doing this whole thing wrong.<br />
<br />
Tried to install PyQt using MacPorts, but either didn't find the right port or something didn't work. So I tried Homebrew (which is like MacPorts but newer; says you shouldn't run both but it's been okay for me so far):<br />
<span style="font-family: Courier New, Courier, monospace;">brew install pyqt</span><br />
<br />
Turns out, I guess, PyQtWebkit is included in PyQt; I just ran the code at the top of <a href="http://www.rkblog.rk.edu.pl/w/p/webkit-pyqt-rendering-web-pages/">the above link</a>, and bam! Google.pl in a python Qt window.<br />
I wonder if I could have used pip to get PyQt; <a href="http://stackoverflow.com/questions/4010842/python-2-7-cannot-import-pyqt4">looks doubtful though</a>.<br />
<br />
Looks like instead maybe I could have downloaded <a href="http://sourceforge.net/projects/pyqtx/files/">PyQtX</a>. But I haven't tried that.<br />
<br />
Related: Selenium, but that drives an external browser, doesn't let you put one in your own app. Followed the instructions <a href="https://pypi.python.org/pypi/selenium/2.7.0">here</a>.</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-24563320139059598682012-11-12T10:55:00.004-08:002013-12-01T17:37:32.048-08:00Android accelerometer sampling rates<div dir="ltr" style="text-align: left;" trbidi="on">
An app I'm working on relies on quick spikes in accelerometer activity, like fractions of a second, so I have to have a pretty fast accelerometer. Android lets you specify accelerometer sampling rate, kind of: you can choose SENSOR_DELAY_NORMAL, SENSOR_DELAY_UI, SENSOR_DELAY_GAME, or SENSOR_DELAY_FASTEST. Then the phone will try to give you samples at a rate that works for your app.<br />
<br />
On NORMAL, my Nexus S gives me about 7Hz. Not at all fast enough. On UI, GAME, or FASTEST, I get about 49Hz, which is almost fast enough. So I gave some friends a little test app which determined their phone's FASTEST sample rate and here's what I found (readings in Hz):<br />
<br />
Motorola Droid X: 5.19<br />
Motorola Droid X2: 6.23<br />
ZTE Blade: 9.11<br />
HTC Nexus One: 24.86<br />
HTC Evo 4G: 37.54<br />
HTC Wildfire: 38.40*<br />
HTC Desire: 47.27*<br />
LG Nexus S: 49.44<br />
Samsung Galaxy S3: 93.98<br />
Motorola Atrix: 94.32<br />
Sony Ericsson x10 Mini: 94.77**<br />
Samsung Galaxy S2: 96.03, 97.5<br />
Samsung Galaxy Nexus: 99.8<br />
Samsung Note II: 100.0<br />
Samsung Galaxy S3: 100<br />
Samsung Galaxy Teos: 109.42*<br />
LG Nexus 4: 198<br />
<br />
(* from <a href="http://stackoverflow.com/questions/9358862/impossibility-to-change-the-rate-of-the-accelerometer">this stack overflow post</a>)<br />
(** from <a href="http://www.slideshare.net/paller/motion-recognition-with-android-devices">this slideshow</a>, slide 13)<br />
<div>
<br /></div>
Want to add a reading? Here's how:<br />
1. Enable installation of apps from unknown sources (Settings -> Security -> Unknown Sources)<br />
2. <a href="https://sites.google.com/site/dantasse/AccelerometerTest.apk">Download this app</a><br />
3. Tap "start", wait 5 seconds, and post (in a comment) which phone you have and what reading you get.<br />
Edit: thanks all for adding to this! If you're looking for some other phones, check out the comments below.</div>
Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com30tag:blogger.com,1999:blog-8344907271756111127.post-54912880910040918552012-05-20T09:29:00.001-07:002012-07-22T14:26:08.586-07:00vim: using the system clipboard<div dir="ltr" style="text-align: left;" trbidi="on">
It annoyed me that I couldn't yank (copy) something in vim using y and then paste it in something else with ctrl-v, or vice versa. I found that the following line (linux, x11, vim 7.3), makes it just work:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">:set clipboard=unnamedplus</span><br />
<br />
What's going on here?<br />
vim has "registers." Each register is like its own clipboard. You can choose your register with <span style="font-family: 'Courier New', Courier, monospace;">"</span>. So for example:<br />
<span style="font-family: 'Courier New', Courier, monospace;">"ayy</span><br />
means "using the <span style="font-family: 'Courier New', Courier, monospace;">a</span> register (<span style="font-family: 'Courier New', Courier, monospace;">"a</span>), copy/yank the current line (<span style="font-family: 'Courier New', Courier, monospace;">yy</span>)". But most of the registers only live within vim. Vim has two special registers though: * and +. On Windows and Mac, they are both the system clipboard. On X11 they are different and only the + is the system clipboard. So in vim on either system, you should be able to type:<br />
<span style="font-family: 'Courier New', Courier, monospace;">"+yy</span><br />
to copy the current line into the system clipboard, and then ctrl-v in a windowed application to paste it. Then the "<span style="font-family: 'Courier New', Courier, monospace;">set clipboard=unnamedplus</span>" line just makes the + register the default one.<br />
<br />
More info from vim tips <a href="http://vim.wikia.com/wiki/VimTip21">here</a> and <a href="http://vim.wikia.com/wiki/Accessing_the_system_clipboard">here</a>.<br />
<br />
Side note: I think that by saying "system clipboard" I am being a little vague because there are actually three system clipboards in X11, but the ctrl-c ctrl-v one is the one I always want. I think this is called, amusingly, the CLIPBOARD clipboard. <a href="http://www.cyberciti.biz/faq/xclip-linux-insert-files-command-output-intoclipboard/">More here</a>.<br />
<br />
Another side note: on my older Macbook (OS X 10.6.8 Snow Leopard), I couldn't easily get Vim 7.3.74+ (where <span style="font-family: 'Courier New', Courier, monospace;">clipboard=unnamedplus</span> is introduced). The <a href="http://code.google.com/p/macvim/">MacVim</a> version available is 7.3.53. However, as mentioned above, * and + are both the system clipboard on a Mac, so <span style="font-family: 'Courier New', Courier, monospace;">set clipboard=unnamed</span> works. (<a href="http://hynek.me/articles/macvim-and-the-clipboard/">be careful if using tmux</a>.) You can check if unnamedplus is available, so my .vimrc now looks like:<br />
<span style="font-family: 'Courier New', Courier, monospace;">if has('unnamedplus')</span><br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;"> set clipboard=unnamedplus</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">else</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> set clipboard=unnamed</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">endif</span><br />
</div>Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-90183924483589816512012-03-22T12:04:00.000-07:002012-03-22T12:04:13.324-07:00Nook Rooting and Mounting Drives in Unix<div dir="ltr" style="text-align: left;" trbidi="on">
I got a Nook Simple Touch, because you can put Android on it and then use the Nook app, the Kindle app, or whatever pdf reader you want to read all of the ebooks, not just Amazon's or B&N's. To put Android on it, I used <a href="http://forum.xda-developers.com/showthread.php?t=1343143">this guide</a>. (this is for Nooks with firmware 1.1; <a href="http://forum.xda-developers.com/showthread.php?t=1132693">here's the one for 1.0 or 1.0.1</a>. Find your firmware version in Settings->Device Info->About Your Nook)<br />
<br />
The process works as follows: download a disk image, put it onto a microSD card, put that card in the Nook, boot up the nook, and then do a bunch of things on the software to sign in to the Android Market and download apps. The tricky part for me was putting the image onto the SD card.<br />
<br />
The command to put the image on the SDcard (where touchnooter-2-1-31.img is the name of the disk image file) is:<br />
<span style="font-family: 'Courier New', Courier, monospace;">sudo dd if=touchnooter-2-1-31.img of=/dev/<sdcard></sdcard></span><br />
The tricky part is that you have to replace "/dev/<sdcard>" with the name of your SD card. (<a href="http://nookdevs.com/Nook_Simple_Touch/Rooting">this site</a> clarifies: "where <sdcard> is your sdcard (for example /dev/sdc or /dev/mmcblk0, not the mount point of the sdcard or an existing partition like sdc1 or mmcblk0p1)." What does this mean?</sdcard></sdcard><br />
<br />
If a volume (aka a partition of a drive) is "mounted", then it's got an alias in the big unix filesystem tree and you can access files on it. Usually, when you plug in an SD card or USB drive, the system mounts it automatically. You can just run the command "<span style="font-family: 'Courier New', Courier, monospace;">mount</span>" to see what's already mounted; most of it didn't mean anything to me when I ran it, but at least I saw this:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">/dev/sda1 on / type ext4 (rw,errors=remount-ro,commit=0)</span><br />
which is my main hard drive's main partition, so that's interesting. You can also (in Ubuntu) use the "Disk Utility" to see what's mounted.<br />
<br />
In Disk Utility, click on your card reader on the left, and you'll see in the top right "Drive" and in the bottom right "Volumes". Look for "Device:", which will tell you the name of the drive in the filesystem. For example, right now I've got a card in there that has one partition and a bunch of free space; the Drive is /dev/sdd and the Volumes are /dev/sdd1 (the existing partition) and /dev/sdd (the free space).<br />
<br />
When you're running that <span style="font-family: 'Courier New', Courier, monospace;">sudo dd</span> command above, you want to write to the <i>drive</i>, not a <i>partition</i> of that drive. I kept trying to dd the image to /dev/sdb1 (which was a partition of my SD card) instead of /dev/sdb (which represented the card itself). Once I wrote the file to /dev/sdb, it worked.<br />
<br />
Note that playing around with dd is dangerous: if you accidentally write files to the wrong drive (like /dev/sda in my case), you can erase your hard drive. So make sure (via Disk Utility) that you're writing to the SD card.<br />
<br />
Outstanding questions in my mind:<br />
- what's the command-line tool that will tell me the same things that Disk Utility tells me?<br />
- what is "/dev/sdb" and how does it get there? I guess it's just an identifier that refers to my SD card when I put it in, but why /dev/sdb? And <i>what is it</i>, is it a file descriptor or what? I think /etc/fstab and /etc/mtab are involved, but I'm not sure how; and who puts those two files there?<br />
- is "unmounting" exactly the same as "ejecting" your SD card/USB drive?<br />
- what does <span style="font-family: 'Courier New', Courier, monospace;">dd</span> do that just moving the disk image file to the SD card doesn't do? According to <a href="http://en.wikipedia.org/wiki/Dd_(Unix)">wikipedia</a>, dd is just a lower-level way to move raw data. So it can... cause some file to be run when you boot or something?<br />
</div>Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-58843821663150986912011-08-17T10:24:00.000-07:002011-08-17T10:24:55.778-07:00Double-UTFI've stored a bit of a snapshot of all the music I've liked by picking one song per album and putting them into a seasonal playlist on itunes. So I've got "2006 Spring", for example, which has about a dozen songs I liked to listen to in spring 2006.<br />
<br />
But I've just stored these in itunes. Not only does that mean they're locked within the Apple Empire, they're also vulnerable to me losing my hard drive. So I wanted to get them into real text files. Luckily, itunes lets you export playlists. Unluckily, it's in some bizarre janky format, when I really just want to extract the artist, title, and album for each song. Simple python script to the rescue.<br />
<br />
Ah, but even after deleting some of the crud, I was left with a file in a mash of file formats! See, I had pulled out artist, title, and album, then concatenated them with commas, then written that to a file. But I hadn't paid attention to encodings, so I had some UTF-16 characters, then some UTF-8 commas, then more UTF-16 characters. But Python has an easy answer: just read in the one file as UTF-16, specify that your output file is UTF-8, and within your script deal with strings and don't worry about encodings.<br />
<br />
Tim Bray <a href="http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF">explains UTF-8, UTF-16, and UTF-32 clearly</a>; this is something I probably should have thoroughly understood a while ago.<br />
Evan Jones has <a href="http://www.evanjones.ca/python-utf8.html">a nice overview of how to use unicode in Python</a>.<br />
<br />
And here's my script:<br />
<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">#!/usr/bin/env python</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">import codecs</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">for filename in open("filenames.txt"): # next time I'll learn</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # syntax for "for filename</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # in current directory"</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> filename = filename.strip()</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> outfilename = "output/" + filename.replace(" ", "_")</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> outfile = codecs.open(outfilename, "w", "utf-8")</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> for bigline in codecs.open(filename, "r", "utf-16"):</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> lines = bigline.split("\r")</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> for line in lines:</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> parts = line.split("\t")</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> if len(parts) < 4:</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> continue</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> song = parts[0]</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> artist = parts[1]</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> album = parts[3]</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> linetowrite = "%s, %s, %s\n" % (artist, song, album)</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> outfile.write(linetowrite)</span><br />
<div><br />
</div>Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com1tag:blogger.com,1999:blog-8344907271756111127.post-65592103917589235522011-07-06T16:31:00.000-07:002011-07-06T16:31:09.586-07:00vim: search and replace like breathingOne great thing about vim is how you can search/replace with just a few keystrokes. Being able to search and replace at light speed makes you feel very wizardly. Here are some commands I've been using frequently to do so.<br />
<br />
(Note that these are all regexes, and furthermore vim regexes, which differ slightly from other regex implementations, and I won't get into that.)<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">/foo(enter)</span><br />
(just type this, in insert mode, and you're instantly at the next instance of "foo". Then hit "n" to go to the next instance of "foo".)<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">/foo\c(enter)</span><br />
same as the above, but case insensitive. (I guess you could do <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">:set ic</span> first instead of the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">\c</span> there, but I don't like doing things that leave state lying around if I don't have to)<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">:s/foo/bar/g(enter)</span><br />
Replace all "foo"s with "bar" <i>on this line only</i>.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">:%s/foo/bar/g(enter)</span><br />
Replace all "foo"s with "bar" in the entire file.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">:%s/foo/bar/gc(enter)</span><br />
Replace all "foo"s with "bar in the whole file, but ask for confirmation at each one. I like this one a lot.Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-59611583092634435162011-06-29T15:50:00.000-07:002011-06-29T15:50:16.413-07:00Tracking users via cookiesDisclaimer: this is a "work in progress" or a "I don't know if this is good" post.<div><br />
</div><div>I have a simple web app called Sea Salt that serves a javascript game. I'd like to keep track of users at a very low level- just track them across requests, and maybe track if they come to the site again the next day but that's not super important. I don't want to make them log in or anything.</div><div><br />
</div><div>And it has a splash screen. I want the following behavior:</div><div>- first time you come to /, you get "welcome to this app, click to start"</div><div>- if you click that, you go to /play and I create a User entry for you.</div><div>- if you come to / again, you get "you've already started. click here to continue OR if that wasn't you, click here to restart."</div><div>- if you restart, I create a new User entry for you.</div><div><br />
</div><div>Very simplified App Engine python server code:</div><div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">application = webapp.WSGIApplication([('/', Intro),</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> ('/restart', Restart),</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> ('/play', Play), ...</span></div></div><div>Set up the URL mappings.</div><div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">class Intro(webapp.RequestHandler):</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> def get(self):</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> cookie_id = self.request.cookies.get('sea_salt_id')</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> if cookie_id and User.get_by_id(int(cookie_id)):</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> #(render already_started.html</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">)</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> else:</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> #(render index.html)</span></div><div>Pretty simple. If you go to /, first check the cookie. If your cookie corresponds to a real user, then you must have been here before. already_started.html contains links to /restart and /play. Otherwise, you haven't been here, so show you the splash screen, which has just a form that posts to /play.</div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">class Restart(webapp.RequestHandler):</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> def get(self):</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> self.response.headers.add_header(</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> 'Set-Cookie',</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> 'sea_salt_id=-1; expires=Thu, 01-Jan-1970 00:00:01 GMT')</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> self.redirect('/')</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: inherit;">If you to go to /restart, delete your cookie, and send you back to /.</span></div><div><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">class Play(webapp.RequestHandler):</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> def get(self):</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> cookie_id = self.request.cookies.get('sea_salt_id')</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> if not cookie_id or not User.get_by_id(int(cookie_id)):</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> self.redirect('/')</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> #(render game.html)</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> def post(self):</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> cookie_id = self.request.cookies.get('sea_salt_id')</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> if not cookie_id or not User.get_by_id(int(cookie_id)):</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> user = User.create()</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> user.put()</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> id = user.key().id()</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> self.response.headers.add_header(</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> 'Set-Cookie',</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> 'sea_salt_id=%d; expires=Fri, 31-Dec-2020 23:59:59 GMT' % id)</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> #(render game.html)</span></div><div><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: inherit;">This is the trickiest. If you go to /play via a GET (like typing it in the address bar), either let you keep playing (if you've already started) or redirect you to /. If you go to /play via a POST, either let you keep playing (if you've already started) or create a user for you and then let you play. I think this is right, because GETs should be read-only while POSTs can write, right?</span></div></div><div><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: inherit;">This all seems a little too complex for its own good, but it seems to work. If you have any better ideas (or if I've made any mistakes), I'd love to hear them. Thanks!</span></div>Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0tag:blogger.com,1999:blog-8344907271756111127.post-31876216657142795072011-06-01T16:29:00.000-07:002011-06-13T17:28:57.583-07:00.vimrc: colorcolumn80 character line? No problem!<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">" displays a red column at 80 characters</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">set colorcolumn=80</span><br />
<br />
Thanks to <a href="http://stackoverflow.com/questions/235439/vim-80-column-layout-concerns">this stack overflow post</a>.<br />
<br />
EDIT: you should probably surround it with an "if exists" to avoid annoyance if you port your .vimrc to another machine that has an older version of vim (colorcolumn is new in 7.3). Here's the syntax:<br />
<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">if exists('+colorcolumn')</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> set colorcolumn=80</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">endif</span><br />
<br />
<br />
Thanks to <a href="http://stackoverflow.com/questions/235439/vim-80-column-layout-concerns/3765575#3765575">this answer</a> on that same stack overflow post.Danhttp://www.blogger.com/profile/03312048754374286109noreply@blogger.com0